samedi 14 mars 2015

Incrementally loaded data from DynamoDB to S3 using Amazon Data Pipeline

My scenario is based on 'DAT' (which contains date) column in DynamoDB, I need to incrementally load the data to S3 using Amazon Data Pipeline console.


To perform this I have used Hive Copy activity and added filtersql as DAT > unix_timestamp(\"2015-01-01 01:00:00.301\", \"yyyy-MM-dd'T'HH:mm:ss\"). When I use filtersql then getting an error message


Failed to complete HiveActivity: Hive did not produce an error file. Cause: EMR job '@TableBackupActivity_2015-03-14T07:17:02_Attempt=3' with jobFlowId 'i-3NTVWJANCCOH7E' is failed with status 'FAILED' and reason 'Waiting after step failed'. Step '@TableBackupActivity_2015-03-14T07:17:02_Attempt=3' is in status 'FAILED' with reason 'null'


If I use without filtersql statement then data moved from DynamoDB to S3 without any error. Please someone help me on this error.


Thanks, Dinesh Kumar





Aucun commentaire:

Enregistrer un commentaire