vendredi 7 août 2015

Apache-Spark with Python Theano

I have written a binary classifier using Python Theano Library. Based on different data files, I would like to parallelize the classifier based on Amazon Web Service (AWS) EC2 with one master node and several slave nodes using Apache-Spark. I've tested my code with "local" mode on the AWS-EC2 master node, and classifier works very well except the long running time. However, when I run my same code on AWS-EC2 cluster with one master node and two slave nodes (all three nodes are t2.micro type), I got the following error:

WARNING (theano.gof.compilelock): Overriding existing lock by dead process '17075' (I am process '17712')
15/08/08 00:10:12 ERROR Executor: Exception in task 0.3 in stage 0.0 (TID 6)
java.net.SocketException: Connection reset
    at java.net.SocketInputStream.read(SocketInputStream.java:196)
    at java.net.SocketInputStream.read(SocketInputStream.java:122)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
    at java.io.DataInputStream.readInt(DataInputStream.java:387)
    at org.apache.spark.api.python.PythonRDD$$anon$1.read(PythonRDD.scala:111)
    at org.apache.spark.api.python.PythonRDD$$anon$1.<init>(PythonRDD.scala:179)
    at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:97)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:63)
    at org.apache.spark.scheduler.Task.run(Task.scala:70)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

I am sure that I correctly set-up my cluster on AWS-EC2 as well as the master node url in my code, and I'm also sure that all my codes have no bugs. But I have no idea about the problem. I really appreciate if anyone helps me solve the problem.




Aucun commentaire:

Enregistrer un commentaire