vendredi 7 août 2015

Strategies for optimizing the performance of IO intesive jobs on AWS

I wrote a script that analyzes a lot of files on an AWS cluster.

Running it on the cloud seems to be slower than I expected - the filesystem is shared via NFS, so the round-trip through the network seems to be the limiting step here. Bottom line - the processing power of the cluster is limited by the speed of the internal network which is considerably slower than the speed of the SSD the data is located in.

How would you optimize the cluster so that IO intensive jobs will run efficiently?




Aucun commentaire:

Enregistrer un commentaire