samedi 10 octobre 2015

Synchronizing in Spark Custom Receiver

Working on integrating Spark Streaming with Amazon SQS. In SQS, once received a message must be 'deleted' to mark as 'acknowledged'. But my question is, if I create a Spark Custom Receiver then Spark will trigger many instances of the Receiver on many nodes, correct? But then each receiver may possibly get the same message delivered, no? How do I prevent that? Simply using a 'synchronized' section in my receiver wouldn't solve this problem, correct?

Any pointers would be appreciated.




Aucun commentaire:

Enregistrer un commentaire