Im getting 3 messages from sqs queue: mapper.py, reducer.py and test.txt ...and Im storing each message inside a list.
And I want to use the content of each message in my mapper, reducer and input variables(last 3 code lines).
But outside my while loop from line where i Have print "Connecting to EMR"
Im not seeing how can I use my list of filenames for the purpose I want.
Do you see any solution for this?
In my print item
inside for loop it shows only the first element of filenames_list.
conn = SQSConnection()
myqueue = conn.get_queue('myQueue')
while myqueue.count() != 0:
filenames_list = list()
message = myqueue.read(wait_time_seconds=10)
filename = ""
if message is not None:
myqueue.delete_message(message)
filename = message.get_body()
print "Got message %s from queue" % filename
else:
print "No message. Retrying in 10 seconds..."
filenames_list.append(filename)
for item in files_list:
print item
print "Connecting to EMR"
conn = boto.emr.connect_to_region('us-east-1')
print "Creating Streaming step"
step = StreamingStep(name='Example1',
# after myFolder/ I want to put here the content of my list relative to mapper.py
mapper="s3n://myFolder/HereIWantToPutTheListContentRelatedToMapper.py",
# after myFolder/ I want to put here the content of my list relative to reducer.py
reducer="s3n://myFolder/HereIWantToPutTheListContentRelatedToReducer.py",
# after myFolder/ I want to put here the content of my list relative to mapper.py
input='s3n://myFolder/HereIWantToPutTheListContentRelatedToTest.txt',
output='s3n://myFolder/output/wordcount_output')
Aucun commentaire:
Enregistrer un commentaire