Elasticsearch cluster CPU maxing out at 100 request per second search queries

The Problem: I'm running a elastic search server cluster with four index's one has 4.5 million documents the other has 13 million. The other index's are the marvel and the kibana and are very small.

Whenever I get to about 150 queries per second with jmeter ("testing framework") The CPU will max out. The more I turn it up the more the CPU maxes out.

From everything I read online about performance tuning it usually has memory issues but our box is running out of cpu way before memory and causing 6 second response times with the test.

the setup:

3 x client nodes AWS m3.xlarge
4 cores 16gb
3 x master nodes AWS m3.medium
1 core 4gb <-i beleive
3 x data nodes   AWS c3.2xlarge
8 cores 30gb

Plugins:

  AWS
  Marvel

Document Count

account-index-v1.0   4.5 M
entry-index-v1.0     13.1 M

@160 query per second using jmeter to execute the following query:

            CPU  LOAD(1m) MEM  %FreeD   IOPS
Client Nodes
hidden:9300 0.0    0.0    7.3    n/a     n/a
hidden:9300 0.0    0.0    4.3    n/a     n/a
hidden:9300 0.0    0.1    8.3    n/a     n/a

Data Nodes
hidden:9300 99.0   10.2   11.7    69.7 GB  1.2
hidden:9300 71.0   3.0    15.0    69.6 GB  3.9
hidden:9300 16.7   0.3    12.7    69.8 GB  0.1

Master Nodes
hidden:9300 0.3    0.0    3.0     73.0 GB  0.2
hidden:9300 0.3    0.0    7.0     73.0 GB  0.1
hidden:9300 0.3    0.0    5.0     73.0 GB  0.1

queries

{
"match"{"event_id":"10000"},
"match"{"race_id_indexed":"10000"},
"match"{"is_test":"F"},
"match"{"status":"CONF"},
"must_not":[{"match"{"type":"TEAM"}}],
"query":{"match_all":{}}
}



marvel.agent.enabled: true
cluster.name: Vision
bootstrap.mlockall: true
http.enabled: true
index.number_of_shards: 3
index.number_of_replicas: 1

<%if node.has_key?("ec2") %>
plugin.mandatory: "cloud-aws"
discovery.type: "ec2"
discovery.ec2.groups: "<%= node["ec2"]["security_groups"][0] %>"
discovery.ec2.ping_timeout: "120s"
discovery.zen.ping.multicast.enabled: false
<% else %>
discovery.zen.ping.multicast.enabled: true
<% end %>