mardi 23 juin 2015

DynamoDB - very slow write operations

I have a DynamoDB running in AWS cloud and I am populating it with data on regular (scheduled) bases. Basically, once every hour, I receive a file that needs to be processed and the results has to be saved in the database.

I am using the following class to handle the DB connection and perform the batch writes:

public class DynamoDBService {

  private final AmazonDynamoDB amazonDynamoDB = new AmazonDynamoDBClient();
  private final DynamoDBMapper mapper = new DynamoDBMapper(amazonDynamoDB);

  @Value("${aws_region}")
  private String region;

  @PostConstruct
  public void init() {
    log.info("Region: {}", region);
    amazonDynamoDB.setRegion(RegionUtils.getRegion(region));
  }

  /**
   * 
   * @param records
   */
  public void saveRecord(final Collection<Record> records) {
    log.info("Saving records...");

    // create table if necessary here

    List<BillingRecord> recordsToSave = new ArrayList<BillingRecord>(100);

    for (BillingRecord record : billingRecords) {

      recordsToSave.add(record);

    }

    // save the records
    List<FailedBatch> failedBatch = mapper.batchWrite(recordsToSave, new ArrayList<BillingRecord>());
    // process failed writes here

    log.info("All records have been saved.");
  }
}

The problem is that the writes are painfully slow. I read the documentation and increased the throughput capacity (so it should now support over 300000 writes/hour) but it takes over 15 minutes to process one List containing approx. 8000 records.

I read that the optimal number of writes in one batch operation is 25 and size of one record below 1kb. I tested it both on my local machine (which I know will be slower because of the traffic overhead) and in the AWS worker environment but the results were both quite slow. Is there any way in which this process can be optimized?




Aucun commentaire:

Enregistrer un commentaire