lundi 1 juin 2015

AWS Discovery is not working properly

I have the following config set up on two nodes in the same availability zone:

cluster.name: AWSelasticsearch
plugin.mandatory: cloud-aws
cloud.aws.access_key: [ACCESS_KEY]
cloud.aws.secret_key: [SECRET_KEY]
discovery.type: ec2
discovery.ec2.groups: "launch-wizard-1"
cloud.aws.region: us-west
discovery.zen.ping.multicast.enabled: false
discovery.ec2.host_type: private_ip
discovery.ec2.ping_timeout: 30s

When I launch Elasticsearch, I can see that the permission for the IAM role are fine (I changed the log level to TRACE) and the node actually finds himself on the list, but cannot find the other node.

In my security group, I have allowed ports 9200-9400.

Here's some of the log from Elasticsearch:

[2015-06-01 09:52:33,250][INFO ][node                     ] [node2] started
[2015-06-01 09:52:34,706][INFO ][cluster.service          ] [node2] new_master [node2][w3awzO63QZmOA3dFZyXeSA][ip-172-31-30-54][inet[/172.31.30.54:9300]], reason: zen-disco-join (elected_as_master)
[2015-06-01 09:52:34,739][INFO ][gateway                  ] [node2] recovered [0] indices into cluster_state
[2015-06-01 09:53:08,562][INFO ][node                     ] [node2] stopping ...
[2015-06-01 09:53:08,584][INFO ][node                     ] [node2] stopped
[2015-06-01 09:53:08,584][INFO ][node                     ] [node2] closing ...
[2015-06-01 09:53:08,590][INFO ][node                     ] [node2] closed
[2015-06-01 09:53:10,158][INFO ][node                     ] [node2] version[1.5.2], pid[16355], build[62ff986/2015-04-27T09:21:06Z]
[2015-06-01 09:53:10,159][INFO ][node                     ] [node2] initializing ...
[2015-06-01 09:53:10,174][INFO ][plugins                  ] [node2] loaded [cloud-aws], sites []
[2015-06-01 09:53:12,169][DEBUG][discovery.zen.elect      ] [node2] using minimum_master_nodes [-1]
[2015-06-01 09:53:12,173][DEBUG][discovery.zen.ping.unicast] [node2] using initial hosts [], with concurrent_connects [10]
[2015-06-01 09:53:12,175][DEBUG][discovery.ec2            ] [node2] using ping.timeout [30s], join.timeout [10m], master_election.filter_client [true], master_election.filter_data [false]
[2015-06-01 09:53:12,177][DEBUG][discovery.zen.fd         ] [node2] [master] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2015-06-01 09:53:12,180][DEBUG][discovery.zen.fd         ] [node2] [node  ] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2015-06-01 09:53:12,697][DEBUG][discovery.ec2            ] [node2] using host_type [PRIVATE_IP], tags [{}], groups [[launch-wizard-1]] with any_group [true], availability_zones [[us-west]]
[2015-06-01 09:53:13,708][INFO ][node                     ] [node2] initialized
[2015-06-01 09:53:13,718][INFO ][node                     ] [node2] starting ...
[2015-06-01 09:53:13,932][INFO ][transport                ] [node2] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/172.31.30.54:9300]}
[2015-06-01 09:53:13,954][INFO ][discovery                ] [node2] AWSelasticsearch/9oxwXKGnSr-O-xBSBaWFIg
[2015-06-01 09:53:13,955][TRACE][discovery.ec2            ] [node2] starting to ping
[2015-06-01 09:53:15,188][INFO ][com.amazonaws.http.AmazonHttpClient] x-amzn-RequestId: not available
[2015-06-01 09:53:15,265][INFO ][com.amazonaws.http.AmazonHttpClient] AWS Request ID: 4d97845d-6e06-47d2-81ad-5dd3995971c5
[2015-06-01 09:53:15,268][TRACE][discovery.ec2            ] [node2] building dynamic unicast discovery nodes...
[2015-06-01 09:53:15,268][DEBUG][discovery.ec2            ] [node2] using dynamic discovery nodes []
[2015-06-01 09:53:15,278][TRACE][discovery.zen.ping.unicast] [node2] [1] connecting to [node2][9oxwXKGnSr-O-xBSBaWFIg][ip-172-31-30-54][inet[/172.31.30.54:9300]]
[2015-06-01 09:53:15,346][TRACE][discovery.zen.ping.unicast] [node2] [1] connected to [node2][9oxwXKGnSr-O-xBSBaWFIg][ip-172-31-30-54][inet[/172.31.30.54:9300]]
[2015-06-01 09:53:15,346][TRACE][discovery.zen.ping.unicast] [node2] [1] sending to [node2][9oxwXKGnSr-O-xBSBaWFIg][ip-172-31-30-54][inet[/172.31.30.54:9300]]
[2015-06-01 09:53:15,397][TRACE][discovery.zen.ping.unicast] [node2] [1] received response from [node2][9oxwXKGnSr-O-xBSBaWFIg][ip-172-31-30-54][inet[/172.31.30.54:9300]]: [ping_response{node [[node2][9oxwXKGnSr-O-xBSBaWFIg][ip-172-31-30-54][inet[/172.31.30.54:9300]]], id[1], master [null], hasJoinedOnce [false], cluster_name[AWSelasticsearch]}, ping_response{node [[node2][9oxwXKGnSr-O-xBSBaWFIg][ip-172-31-30-54][inet[/172.31.30.54:9300]]], id[2], master [null], hasJoinedOnce [false], cluster_name[AWSelasticsearch]}]
[2015-06-01 09:53:30,333][INFO ][com.amazonaws.http.AmazonHttpClient] x-amzn-RequestId: not available
[2015-06-01 09:53:30,335][INFO ][com.amazonaws.http.AmazonHttpClient] AWS Request ID: be67124c-c57a-40e8-856d-ce45bd5d54c4
[2015-06-01 09:53:30,335][TRACE][discovery.ec2            ] [node2] building dynamic unicast discovery nodes...
[2015-06-01 09:53:30,335][DEBUG][discovery.ec2            ] [node2] using dynamic discovery nodes []
[2015-06-01 09:53:30,335][TRACE][discovery.zen.ping.unicast] [node2] [1] sending to [node2][9oxwXKGnSr-O-xBSBaWFIg][ip-172-31-30-54][inet[/172.31.30.54:9300]]
[2015-06-01 09:53:30,341][TRACE][discovery.zen.ping.unicast] [node2] [1] received response from [node2][9oxwXKGnSr-O-xBSBaWFIg][ip-172-31-30-54][inet[/172.31.30.54:9300]]: [ping_response{node [[node2][9oxwXKGnSr-O-xBSBaWFIg][ip-172-31-30-54][inet[/172.31.30.54:9300]]], id[1], master [null], hasJoinedOnce [false], cluster_name[AWSelasticsearch]}, ping_response{node [[node2][9oxwXKGnSr-O-xBSBaWFIg][ip-172-31-30-54][inet[/172.31.30.54:9300]]], id[3], master [null], hasJoinedOnce [false], cluster_name[AWSelasticsearch]}, ping_response{node [[node2][9oxwXKGnSr-O-xBSBaWFIg][ip-172-31-30-54][inet[/172.31.30.54:9300]]], id[4], master [null], hasJoinedOnce [false], cluster_name[AWSelasticsearch]}]
[2015-06-01 09:53:43,954][WARN ][discovery                ] [node2] waited for 30s and no initial state was set by the discovery
[2015-06-01 09:53:43,967][INFO ][http                     ] [node2] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/172.31.30.54:9200]}
[2015-06-01 09:53:43,967][INFO ][node                     ] [node2] started
[2015-06-01 09:53:45,402][INFO ][com.amazonaws.http.AmazonHttpClient] x-amzn-RequestId: not available
[2015-06-01 09:53:45,404][INFO ][com.amazonaws.http.AmazonHttpClient] AWS Request ID: 4a3b60a7-1f8a-4df4-8745-da4221ca6727
[2015-06-01 09:53:45,404][TRACE][discovery.ec2            ] [node2] building dynamic unicast discovery nodes...
[2015-06-01 09:53:45,404][DEBUG][discovery.ec2            ] [node2] using dynamic discovery nodes []
[2015-06-01 09:53:45,404][TRACE][discovery.zen.ping.unicast] [node2] [1] sending to [node2][9oxwXKGnSr-O-xBSBaWFIg][ip-172-31-30-54][inet[/172.31.30.54:9300]]
[2015-06-01 09:53:45,407][TRACE][discovery.zen.ping.unicast] [node2] [1] received response from [node2][9oxwXKGnSr-O-xBSBaWFIg][ip-172-31-30-54][inet[/172.31.30.54:9300]]: [ping_response{node [[node2][9oxwXKGnSr-O-xBSBaWFIg][ip-172-31-30-54][inet[/172.31.30.54:9300]]], id[1], master [null], hasJoinedOnce [false], cluster_name[AWSelasticsearch]}, ping_response{node [[node2][9oxwXKGnSr-O-xBSBaWFIg][ip-172-31-30-54][inet[/172.31.30.54:9300]]], id[3], master [null], hasJoinedOnce [false], cluster_name[AWSelasticsearch]}, ping_response{node [[node2][9oxwXKGnSr-O-xBSBaWFIg][ip-172-31-30-54][inet[/172.31.30.54:9300]]], id[5], master [null], hasJoinedOnce [false], cluster_name[AWSelasticsearch]}, ping_response{node [[node2][9oxwXKGnSr-O-xBSBaWFIg][ip-172-31-30-54][inet[/172.31.30.54:9300]]], id[6], master [null], hasJoinedOnce [false], cluster_name[AWSelasticsearch]}]
[2015-06-01 09:53:45,408][TRACE][discovery.ec2            ] [node2] full ping responses: {none}
[2015-06-01 09:53:45,408][DEBUG][discovery.ec2            ] [node2] filtered ping responses: (filter_client[true], filter_data[false]) {none}
[2015-06-01 09:53:45,418][INFO ][cluster.service          ] [node2] new_master [node2][9oxwXKGnSr-O-xBSBaWFIg][ip-172-31-30-54][inet[/172.31.30.54:9300]], reason: zen-disco-join (elected_as_master)
[2015-06-01 09:53:45,424][TRACE][discovery.ec2            ] [node2] cluster joins counter set to [1] (elected as master)
[2015-06-01 09:53:45,449][INFO ][gateway                  ] [node2] recovered [0] indices into cluster_state

I can also see that each server can connect to the relevant ports on the other server, so it doesn't seem to be a network connectivity issue.




Aucun commentaire:

Enregistrer un commentaire