jeudi 15 octobre 2015

Cassandra is randomly slow on two nodes

I'm using Cassandra in my .NET project to store some data. I have performance issue when I'm deploing my app from dev machine to test environment on Amazon AWS.

Below is attached sample log from my app. It shows only query times. On my dev machine these commands work everytime in a few milliseconds but on multi node setup I'm getting this random slowdowns. It's the same codebase. And all the reads are simple queries by primary key (ex. GetTableDataChangeCommand). This database is almost empty (only a few rows in each table).

My setup is two m3.xlarge machines with centos7, and I'm querying Cassandra from AWS from the same datacenter.

But I'm getting the same problem on my customer's infrastructure (on premise, not AWS).

What can be the issue here? Any hints?

GetTableDataChangeCommand: 930ms
GetTableDataChangeCommand: 962ms
GetTableDataChangeCommand: 2ms
GetUserDirectRolesCommand: 960ms
GetTableDataChangeCommand: 930ms
GetAllGroupsCommand: 7ms
GetUserGroupsCommand: 9ms
GetAllOrganizationUnitsCommand: 10ms
GetAllRolesCommand: 8ms
GetUserRolesCommand: 4ms
GetUserDirectRolesCommand: 2ms
GetUserCommand: 2ms
GetTableDataChangeCommand: 2ms
GetTableDataChangeCommand: 1ms
GetTableDataChangeCommand: 5ms
GetAllGroupsCommand: 4ms
GetUserGroupsCommand: 1ms
GetTableDataChangeCommand: 1ms
GetAllRolesCommand: 4ms
GetAllOrganizationUnitsCommand: 4ms
GetUserRolesCommand: 3ms
GetUserDirectRolesCommand: 2ms
GetUserCommand: 3ms
GetTableDataChangeCommand: 3ms
GetTableDataChangeCommand: 5ms
GetUserGroupsCommand: 1ms
GetTableDataChangeCommand: 1ms
GetTableDataChangeCommand: 7ms
GetAllRolesCommand: 6ms
GetAllGroupsCommand: 8ms
GetAllOrganizationUnitsCommand: 7ms
GetUserRolesCommand: 3ms
GetUserDirectRolesCommand: 2ms
GetTableDataChangeCommand: 1ms
GetUserCommand: 3ms
GetTableDataChangeCommand: 1ms
GetAllOrganizationUnitsCommand: 5ms
GetTableDataChangeCommand: 8ms
GetTableDataChangeCommand: 9ms
GetUserGroupsCommand: 2ms
GetAllGroupsCommand: 10ms
GetAllRolesCommand: 4ms
GetUserRolesCommand: 3ms
GetTableDataChangeCommand: 1ms
GetUserCommand: 2ms
GetTableDataChangeCommand: 1ms
GetUserGroupsCommand: 1ms
GetUserDirectRolesCommand: 2ms
GetAllRolesCommand: 3ms
GetTableDataChangeCommand: 1ms
GetTableDataChangeCommand: 1ms
GetAllGroupsCommand: 4ms
GetAllOrganizationUnitsCommand: 5ms
GetUserRolesCommand: 3ms
GetUserCommand: 2ms
GetTableDataChangeCommand: 2ms
GetTableDataChangeCommand: 3ms
GetUserDirectRolesCommand: 5ms
GetTableDataChangeCommand: 3ms
GetTableDataChangeCommand: 4ms
GetUserGroupsCommand: 1ms
GetAllGroupsCommand: 4ms
GetAllOrganizationUnitsCommand: 7ms
GetAllRolesCommand: 6ms
GetUserRolesCommand: 3ms
GetTableDataChangeCommand: 1014ms
GetUserCommand: 1014ms
GetTableDataChangeCommand: 999ms
GetTableDataChangeCommand: 984ms
GetUserGroupsCommand: 984ms
GetUserDirectRolesCommand: 983ms
GetAllOrganizationUnitsCommand: 984ms
GetTableDataChangeCommand: 2ms
GetAllGroupsCommand: 4ms
GetAllRolesCommand: 2ms
GetUserRolesCommand: 3ms




Aucun commentaire:

Enregistrer un commentaire