mardi 28 avril 2015

Properly handling context.succeed()/context.fail() with AWS S3 service calls in AWS Lambda

I've already searched through posts here (i.e. How do you structure sequential AWS service calls within lambda given all the calls are asynchronous?) and elsewhere, and can't seem to find that one little bit of information that will help me get past this annoying issue. When you have a Lambda function that iterates through a loop, and within that loop makes a call to say s3.putObject(), it runs into a short-circuit issue when trying to properly deal with context.succeed()/context.fail() or the older context.done(null, 'msg') way of closing the Lambda process.

I.E. the iteration needs to call the s3.putObject() with the current object to be uploaded, but still output to cloudwatch or possibly SQS/SNS the file that was successfully uploaded. However, all of my attempts at putting this type of closure into the function meets with random results of sometimes getting the file names, other times only getting some of the file names, etc.

What is the best way to do it? I've attempted to use Q and async but to be honest, I'm still learning all of this stuff..

Below is a rough example of what i'm attempting to do:

function output(s3Object){
     s3.putObject(s3Object, function(err, data){
          if (err) {
               console.log('There was an issue with outputting the object.', err);
          } else {
             // how do you properly close this if you have x number of incoming calls??
          // context.done(null, 'success');
}


// and later in the code where it actually calls the output function
// and NOTE: it should output all of the file names that the invocation uploads!
for (var a = 0; a < myRecords.length; a++){
     output(myRecords[a]);
}

But, as I said previously, any attempts I've made so far, get mixed results.

Successfully output object: myBucket/prefix/part_000000123432345.dat
Successfully output object: myBucket/prefix/part_000000123432346.dat

But another test of the function outputs:

Successfully output object: myBucket/prefix/part_000000123432346.dat

Argh.




Aucun commentaire:

Enregistrer un commentaire