I've got a Docker application up and running on Elastic Beanstalk, and have noticed that occasionally it will time out and return 504 Gateway Timeout to requests from the Internet. In order to debug this, I created a script that I'm running on my local machine that constantly sends the exact same request to my application and tells me if it received anything but a 200 response. This is an extract of the output:
[21:38:14] Success
[21:38:15] Success
[21:38:15] Success
[21:39:15] Non 200 code: 504
[21:40:15] Non 200 code: 504
[21:41:15] Non 200 code: 504
[21:42:15] Non 200 code: 504
[21:43:15] Non 200 code: 504
[21:44:15] Success
[21:44:15] Success
[21:44:16] Success
I was wondering if the 504 requests even hit my EC2 instance, so I checked its nginx logs:
172.31.8.21 - [01/Mar/2015:21:38:15 +0000] "GET /v1/jobs HTTP/1.1" 200 5
172.31.8.21 - [01/Mar/2015:21:38:15 +0000] "GET /v1/jobs HTTP/1.1" 200 5
172.31.8.21 - [01/Mar/2015:21:38:15 +0000] "GET /v1/jobs HTTP/1.1" 200 5
172.31.23.11 - [01/Mar/2015:21:39:15 +0000] "GET /v1/jobs HTTP/1.1" 400 0
172.31.23.11 - [01/Mar/2015:21:40:15 +0000] "GET /v1/jobs HTTP/1.1" 400 0
172.31.23.11 - [01/Mar/2015:21:41:15 +0000] "GET /v1/jobs HTTP/1.1" 400 0
172.31.23.11 - [01/Mar/2015:21:42:15 +0000] "GET /v1/jobs HTTP/1.1" 400 0
172.31.23.11 - [01/Mar/2015:21:43:15 +0000] "GET /v1/jobs HTTP/1.1" 400 0
Looking at the logs, it seems that the load balancer sends a request to my EC2 instance, which then times out after 60 seconds which makes the load balancer return 504 and logs 400 into my nginx logs. What's really strange is that the IP address of the 504 requests is different from the IP address of every other successful request.
As I understand it, this is the topology of how a request is handled in a Docker Elastic Beanstalk application:
request -> load balancer -> load balancer proxy -> EC2 instance -> EC2 instance proxy -> Docker image
So I checked the nginx logs inside my Docker image, and there are no access.log entries for the time period when the load balancer was returning 504s.
That means that for some reason, requests are getting to the EC2 instance proxy and then timing out. Has anybody experienced this problem and knows why? I'd also like to know why the timeouts only happen when the 172.31.23.11 IP address is the one doing the requesting.
It's also worth noting that nginx on my EC2 instance reports nothing in its error.log when the timeouts occur.
Aucun commentaire:
Enregistrer un commentaire