lundi 30 mars 2015

How can I get the most recent record from an Amazon Kinesis stream?

I would like to get the most recent record from an Amazon Kinesis stream. I intend to extract the timestamp from this record and compare it to the timestamp of the last record checkpointed by a consumer app in order to check whether the consumer is falling behind.


I cannot use the shard iterator type LATEST. This is because LATEST points to just after the most recent record, so it cannot be used to access the most recent record.


Is there a simple way to get the latest record?


An approach I am considering is to get the shard iterator for the sequence number of the record most recently processed by the consumer, make a GetRecords request using that shard iterator, get the next shard iterator from the result of the request, and repeat until a GetRecords request doesn't return any records.


This approach would involve reading all records since the consumer's checkpoint, which seems unnecessarily wasteful. Is there any way around requesting all these records?





Aucun commentaire:

Enregistrer un commentaire