AutoScaling and Persistent TCP Connections

Normally when you work with web applications auto scaling is something that's easily solved, especially with AutoScaling, ELB, Route53 etc.

However, with persistent TCP connections I find it harder to determine when to actually scale down. Scale up is pretty straight forward: define when and how and follow that protocol. But scaling down in an system that has long lived tcp connection opened seems to be more complicated, especially if your tcp server holds a certain state of the connected client. Which leaves me to the following question:

How do you know when to scale down if you have live connections to your servers?

Consider a setup where you have state dependent TCP servers behind a ELB (maybe in an AutoScaling Group), do you move to a more complicated Load Balancer solution or is this possible to achieve with what AWS has to offer?