lundi 20 avril 2015

spark-ec2 sporadic errors - Intial job has not accepted any resources

Sporadically, I get the following error when running Apache Spark on EC2:

WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory

It happens randomly and the only way to fix is it to shut down the cluster and re start everything. Clearly, this is inefficient. Why does this error randomly occur? I can see in the console all my instances are perfectly fine. I am trying to work with a 6MB file. It can't possibly be out of memory. Every time I attempt to do something on Apache Spark EC2 there seems to be some random error popping up. This recent error occurred when I'm running a program I have run 5'000+ times before. On the same cluster type. Why is there so many sporadic errors? And what does this one even mean considering my instances and master are working perfectly.




Aucun commentaire:

Enregistrer un commentaire