mercredi 28 janvier 2015

Variable to provide iteration number for Hive Query reading from Kinesis stream in an AWS Datapipline

I am trying to create an AWS Datapipeline that excutes a Hive Query whose output is written to an S3 Bucket. The data is then moved from the S3 bucket in an AWS Redshift Cluster.


The Hive Query is using a Kinesis Stream as its input. I'm trying to leverage Kinesis' checkpointing capability and want to pass a variable to my ShellCommandActivity so I can set the iteration number in the hive script. Is there anyway for me to have a variable that increments by 1 every time the datapipeline is run?


Any assistance would be great!





Aucun commentaire:

Enregistrer un commentaire