samedi 4 juillet 2015

HAProxy and AWS loadBalancer - 503 error

we've recently split our main web app(that runs on EC2 in https behind a Load balancer with autoscaling) in two separate web modules.

The main infrastructure has now one load balancer and n-server for the main module (main.elasticbeanstalk.com) and a load balancer with n-server for the secondary module (secondary.elasticbeanstalk.com)

We've created a HAproxy dedicated instance that is resolved by the domain www.mycompany.com and proxies the request as following:

-://www.mycompany.com/fancymodule -> secondary.elasticbeanstalk.com

-://www.mycompany.com/ -> main.elasticbeanstalk.com

We put it production and after ~12hours.. http://ift.tt/1hwAzlGfancymodule start getting 503 Service unavailable. If I manually restart HAproxy everything start working wonderfully.

I've managed to replicate the issue renewing the ip address associated to secondary.elasticbeanstalk.com (es: Converting from a load balancer to a single instance).

Seems like HAproxy is not renewing the dns resolving to the secondary.elasticbeanstalk.com, so it get stuck with the old ip and cannot reach correctly the web server.

And is not a short downtime! It doesn't route correctly until I restart the service!

Is it possible that the load balancer, being in elasticIp, get associated with a new ipaddress an therefore is no longer reachable?

Can someone give a look to this config and tell me if I'm doing something stupid?

global
  log         127.0.0.1:514 local2 info
  chroot    /var/lib/haproxy
  pidfile     /var/run/haproxy.pid
  maxconn     4000
  user        haproxy
  group   haproxy
  daemon

# turn on stats unix socket
  stats socket /var/lib/haproxy/stats
  tune.ssl.default-dh-param 2048

defaults
  retries                 3
  timeout http-request    10s
  timeout queue           1m
  timeout connect         10s
  timeout client          1m
  timeout server          1m
  timeout http-keep-alive 10s
  timeout check           10s
  mode    http
  option  httplog

frontend mydomain
  log global
  bind *:80
  bind *:443 ssl crt /etc/ssl/certificate.pem
  acl isSsl ssl_fc
  redirect scheme https if !isSsl
  option dontlog-normal
  mode http

  acl secondaryDomain url_beg /fancymodule

  acl liveDomain hdr_end(Host) -i www.mycompany.com

  use_backend live_secondary if secondaryDomain liveDomain
  use_backend live_main if liveDomain

  default_backend live_main

backend live_main
  rspadd Set-Cookie:\ module=main;Path=/
  server main main.elasticbeanstalk.com:80

backend live_secondary
  rspadd Set-Cookie:\ module=secondary;Path=/
  server secondary secondary.elasticbeanstalk.com:80

listen stats :1234
  mode http
  stats enable
  stats hide-version
  stats realm Haproxy\ Statistics
  stats uri /stats
  stats auth user:pswd




Aucun commentaire:

Enregistrer un commentaire