vendredi 17 avril 2015

Apache Spark EC2 Script launching slaves but no master

When using the Apache Spark EC2 script to launch a cluster I have found somewhat of a bug which is beginning to hit my pocket. When specifying the number of slaves: if you enter a number which is greater than or equal to your limit then the cluster is launched with your maximum number of slaves - but no master! This gives you no control over the slaves and thus cluster.


I have not found a way to launch just a master with the Apache Spark EC2 script. I have manually shut down 1 of the slaves to make space for a master; however, when then trying to re-launch the script it just says:



Searching for existing cluster my_cluster...
Found 0 master(s), 4 slaves
ERROR: There are already instances running in group my_cluster-master
or my_cluster-slaves


To overcome this, I have to log into the AWS console and terminate all the clusters and then restart. As Amazon charge per hour I am being charged for a full hours worth of my maximum number of clusters - all for nothing.


Is there a way to launch a master when slaves already exist?





Aucun commentaire:

Enregistrer un commentaire