samedi 28 mars 2015

use a list outside my loop to concatenate a url string with some list values

Im getting 3 messages from sqs queue: mapper.py, reducer.py and test.txt ...and Im storing each message inside a list.


And I want to use the content of each message in my mapper, reducer and input variables(last 3 code lines).


But outside my while loop from line where i Have print "Connecting to EMR" Im not seeing how can I use my list of filenames for the purpose I want.


Do you see any solution for this?


In my print item inside for loop it shows only the first element of filenames_list.



conn = SQSConnection()
myqueue = conn.get_queue('myQueue')
while myqueue.count() != 0:
filenames_list = list()
message = myqueue.read(wait_time_seconds=10)
filename = ""
if message is not None:
myqueue.delete_message(message)
filename = message.get_body()
print
print "Got message %s from queue" % filename
else:
print
print "No message. Retrying in 10 seconds..."

filenames_list.append(filename)


for item in files_list:
print item

print "Connecting to EMR"
conn = boto.emr.connect_to_region('us-east-1')
print "Creating Streaming step"
step = StreamingStep(name='Example1',

# after myFolder/ I want to put here the content of my list relative to mapper.py

mapper="s3n://myFolder/HereIWantToPutTheListContentRelatedToMapper.py",

# after myFolder/ I want to put here the content of my list relative to reducer.py

reducer="s3n://myFolder/HereIWantToPutTheListContentRelatedToReducer.py",

# after myFolder/ I want to put here the content of my list relative to mapper.py
input='s3n://myFolder/HereIWantToPutTheListContentRelatedToTest.txt',

output='s3n://myFolder/output/wordcount_output')




Aucun commentaire:

Enregistrer un commentaire