Working on integrating Spark Streaming with Amazon SQS. In SQS, once received a message must be 'deleted' to mark as 'acknowledged'. But my question is, if I create a Spark Custom Receiver then Spark will trigger many instances of the Receiver on many nodes, correct? But then each receiver may possibly get the same message delivered, no? How do I prevent that? Simply using a 'synchronized' section in my receiver wouldn't solve this problem, correct?
Any pointers would be appreciated.
Aucun commentaire:
Enregistrer un commentaire