I have a method createjob() to create a map reduce job definition, where I have my input data, my mapper and reducer, this files are upladed to an s3 bucket, and Im trying to use my method enqueuejob() is called to enqueues a mapreduce job request to an aws sqs queue but this queue part its not working. Do you see why?
def enqueuejob(datafile,mapper,reducer):
conn_sqs = boto.sqs.connect_to_region('us-east-1')
queuename = 'testqueue'
q = conn_sqs.get_all_queues(prefix=queuename)
msgdict = {'datafile': datafile, 'mapper': mapper, 'reducer':reducer}
msg = cPickle.dumps(msgdict)
m = Message()
m.set_body(msg)
def createjob():
bucket = conn_s3.get_bucket('myfolder')
key_mapper = bucket.new_key('mapper.py')
key_reducer = bucket.new_key('reducer.py')
key_mapper.set_contents_from_filename('mapper.py')
key_reducer.set_contents_from_filename('reducer.py')
mapper="s3n://myfolder/mapper.py",
reducer="s3n://myfolder/reducer.py",
input='s3n://myfolder/test.txt',
output='s3n://myfolder/output/wordcount_output',
enqueuejob(input,mapper,reducer)
Aucun commentaire:
Enregistrer un commentaire