jeudi 15 octobre 2015

Stop traffic to unhealthy instances without replacing them by auto scaling

Using ELB for TCP protocol and AWS Auto Scaling I run into the following problem when scaling out.

  • three EC2 instances each with 2,000 connections
  • scaling out because that is my specified threshold
  • a new instance gets added by Auto Scaling

How can I stop now traffic going to the three EC2 instances which have too many connections?

  1. Removing it from ELB will mean that it will get terminated after a maximum of 1h using connection draining. Bad: TCP connections will get closed.

  2. Marking the EC2 instance as unhealthy using CloudWatch. Bad: Auto Scaling will detect and replace unhealthy instances

  3. Detaching EC2 instance from Auto Scaling group manually via AWS CLI. Bad: Detaching it from Auto Scaling will also remove it from ELB, see 1.

The only possible solution I can see here and I am not sure if it is feasilbe:

Using CloudWatch mark the EC2 instance as unhealthy. ELB will stop distributing traffic to it. At the same time update the EC2 health for Auto Scaling manually:

aws autoscaling set-instance-health --instance-id i-123abc45d –-health-status healthy

This should override the health in a way that ELB will continue to ignore the EC2 instance and AWS Auto Scaling will not try to replace the instance. Would that work or is there a better solution?




1 commentaire: