mardi 30 juin 2015

Kinesis Client Library : multiple workers for a stream

I have a .war in which we have a Kinesis Application which processes a stream which contains a single shard. We deploy two instances of the war in production. As a result, I would end up with two workers working on a single stream with a single shard. What is the recommended way to deal with this issue? I tried deploying two wars on my dev machine locally, and it seems to be fine in the sense that each record is being processed only once. I know that AWS recommends one instance per shard. From their docs:

Typically, when you use the KCL, you should ensure that the number of instances does not exceed the number of shards (except for failure standby purposes). Each shard is processed by exactly one KCL worker and has exactly one corresponding record processor, so you never need multiple instances to process one shard.

In general, I'm not clear on the multiplicity of the relationship between shards, workers, and record processors. From the documentation, it sounds like it is: For n shards, we need 1 worker and n record processors, which would imply that there is only ever 1 worker processing a stream.

Any help is appreciated.




Aucun commentaire:

Enregistrer un commentaire