I have written a binary classifier using Python Theano Library. Based on different data files, I would like to parallelize the classifier based on Amazon Web Service (AWS) EC2 with one master node and several slave nodes using Apache-Spark. I've tested my code with "local" mode on the AWS-EC2 master node, and classifier works very well except the long running time. However, when I run my same code on AWS-EC2 cluster with one master node and two slave nodes (all three nodes are t2.micro type), I got the following error:
WARNING (theano.gof.compilelock): Overriding existing lock by dead process '17075' (I am process '17712')
15/08/08 00:10:12 ERROR Executor: Exception in task 0.3 in stage 0.0 (TID 6)
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:196)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.spark.api.python.PythonRDD$$anon$1.read(PythonRDD.scala:111)
at org.apache.spark.api.python.PythonRDD$$anon$1.<init>(PythonRDD.scala:179)
at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:97)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:63)
at org.apache.spark.scheduler.Task.run(Task.scala:70)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
I am sure that I correctly set-up my cluster on AWS-EC2 as well as the master node url in my code, and I'm also sure that all my codes have no bugs. But I have no idea about the problem. I really appreciate if anyone helps me solve the problem.
Aucun commentaire:
Enregistrer un commentaire