mercredi 21 janvier 2015

Optimizing creating new classes into api

In one of my Rails applications I'm working with a huge amount of data from Amazon AWS S3. Currently this is my code where I am creating a new Batch and looping through a Hash created in my Batch class.



Class User < ActiveRecord::Base
...
def create_batch
batch = Batch.new
obj = batch.access_bucket
i = 0

while i < obj[:folder].length do
@batch = Batch.new
@batch.folder = obj[:folder][i]
@batch.batch_number = obj[:batch_number][i]
@batch.url = obj[:url][i]
@batch.save!
i = i+1
end
end
end


The Batch class looks as such:



class Batch < ActiveRecord::Base
before_create :access_bucket
Obj = {:folder => [], :batch_number => [], :url => []}
def access_bucket
s3 = AWS::S3.new
bucket = s3.buckets['curateanalytics']
bucket.objects.each do |obj|
if obj.key =~ /swipe batches/
if obj.key =~ /jpg/
Obj[:folder] << sort_objs(obj.key)[0]
Obj[:batch_number] << sort_objs(obj.key)[1]
Obj[:url] << sort_objs(obj.key)[2]
end
end
end
return Obj
end

def sort_objs(url)
swipe = url.split("swipe batches/").last.split("/").first
batch_id = url.split("swipe batches/").last.split("/")[1]
file = url.split("swipe batches/").last.split("/").last
return [swipe, batch_id, file]
end
end


The problem is each key of the Obj Hash contains 3402 elements, most of which are the same. While my initial sorting is parsing so I can see what the JSON will look like my two questions are


1) is there anyway to optimize this code so it will not need to loop through all 3402 elements?


2) Can I use .uniq on the @batch.folder so that each batch with the same name will be filed under that JSON subcategory?





Aucun commentaire:

Enregistrer un commentaire