jeudi 21 mai 2015

Error with Kinesis when use a lot of connection

I'm using Kinesis to store records sent from Jmeter installed on EC2 server. The problem is that when I launch 7200 threads and use 1 shared on my Kinesis stream, all works fine. If I launch 9000 threads I receive this error

Rate exceeded for shard shardId-000000000001 in stream Jmeter under account 769870455028. (Service: AmazonKinesis; Status Code: 400; Error Code: ProvisionedThroughputExceededException; Request ID: 98f687d9-ffbe-11e4-a897-357ee8c24764)

So I increse the number of shard, setting it to 2 and 3 but it doesn't work. So I think that the problem isn't the shard number but my java code or i don't know what else. This is my code:

  public MyKinesisClient( String streamName, int partitionKey, String accessKey, String secretKey, String endpoint, String serviceName, String regionId ) {
        this.streamName=streamName;
        this.partitionKey=partitionKey;
        AWSCredentials credentials = null;
        credentials = new BasicAWSCredentials(accessKey, secretKey);
        kinesisClient = new AmazonKinesisClient(credentials);
        kinesisClient.setEndpoint(endpoint,serviceName,regionId);
    }

    /**
     * Metodo utilizzato per l'invio di un json a Kinesis
     * @param json: com.amazonaws.util.json.JSONObject da inviare a Kinesis
     * @throws UnsupportedEncodingException
     * @throws JSONException
     */
    public void sendJson(JSONObject json) throws UnsupportedEncodingException, JSONException {
        try{
        PutRecordRequest putRecordRequest = new PutRecordRequest();
        putRecordRequest.setStreamName(streamName);
        putRecordRequest.setData(ByteBuffer.wrap(json.toString().getBytes("utf-8")));
        //putRecordRequest.setData(ByteBuffer.wrap(String.format("testData-%d", createTime).getBytes()));
        putRecordRequest.setPartitionKey(String.format("partitionKey-%d", partitionKey));
        kinesisClient.putRecord(putRecordRequest);
        }catch(Exception e){
            System.out.println(e.getMessage());
        }

    }

There is an instruction to use more shard?Thanks in advance




Aucun commentaire:

Enregistrer un commentaire