jeudi 6 août 2015

Using a temp file for lookups?

I have a C# program that is executed on an EC2 cloud server and connecting to an RDS SQL DB. It collects some data, checks if that data exists an SQL DB table and if not it adds the item to a list to be bulk saved later. Now I'm struggling a bit on what the best approach is here.

I've tried these options so far:
a) before inserting make a DB make a call to see if it exists. Pro: less memory intensive. Can run on cheaper cloud servers. Cons: higher DB reads. Slower performance
b) Put a Unique Constraint on the tables and in code catch this Unique Constraint exception and move on. Pro: less memory intensive. Can run on cheaper cloud servers. Cons: higher DB reads. MUCH Slower performance. Id increment of the transaction don't get rolled back leading to an error down the road of making out the int Id column
c) At the start, make a bulk SQL call for all the lookup values and put them all into a hashed set. Reference this hashed set to see if the value exists or not. Pro: Much much faster. Much less DB reads Cons: Pay more for servers with more memory. Risk hitting Out of memory errors as it scales.

The reason I say making single SQL calls is slow is because of the latency of the RDS DB. It's minor but adds up when doing 100k calls in a row to see if something exists. Ideally I'd like as fast performance as possible. I've considered using DyanmoDb but the pricing is just too much and doesn't make much sense in my situation.

So is there some way to save lookups to a file on the local disk that can be quickly referenced later?

EDIT: Why am I being downvoted? If I am doing something insanely wrong please tell me what it is!




Aucun commentaire:

Enregistrer un commentaire