vendredi 4 septembre 2015

Error starting Spark in EMR 4.0

I created an EMR 4.0 instance in AWS with all available applications, including Spark. I did it manually, through AWS Console. I started the cluster and SSHed to the master node when it was up. There I ran pyspark. I am getting the following error when pyspark tries to create SparkContext:

2015-09-03 19:36:04,195 ERROR Thread-3 spark.SparkContext (Logging.scala:logError(96)) - -ec2-user, access=WRITE, inode="/user":hdfs:hadoop:drwxr-xr-x at

org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:271)

I haven't added any custom applications, nor bootstrapping and expected everything to work without errors. Not sure what's going on. Any suggestions will be greatly appreciated.




Aucun commentaire:

Enregistrer un commentaire