samedi 8 août 2015

Scalable solutions for python server side scripting?

I'm currently working on a web application that let's the user upload input files (to an AWS S3), and the upload completion triggers a python script that does some server side processing (it takes between 5 and 30 seconds, depending on the inputs), and returns the results to the client.

Currently the python script is being run as a separate script in the web server (Apache server running flask), but I'm looking for a production worthy solution that would let several users process their files simultaneously without blocking the server or having to wait minutes to get their results back. I'm also planning to add some functionalities to the script that will use multiprocessing.

I've been looking at AWS Elastic Map Reduce, and I have some experience using EC2's and AMI's for processing tasks, but from my little experience with them, it seems like spinning up an EC2 to do the processing when the user uploads a file would take too long (> 1 to 2 minutes) just to start up.

I've also looked at AWS Lambda, which sounds like would be perfect except for the fact that it doesn't support python (I've tried executing python from nodejs, but it's been impossible to get libraries like numpy to work in there).

what kind of infrastructure should I use for this kind of application?

thanks




Aucun commentaire:

Enregistrer un commentaire