Secure distribution of private keys to Docker on CoreOS

We're starting to use Docker/CoreOS as our infrastructure. We're deploying on EC2. The CoreOS cluster is managed by an auto-scaling group so new hosts come and go. Plus, there's a lot of them. I'm trying to find a way to distribute a secret (private RSA key or a shared secret for a symmetric cypher) to all hosts so I can use that to securely distribute things like database credentials, AWS Access Keys for certain services, etc.

I'd like to obey "the principle of least privilege". Specifically, if I have 2 apps in 2 different containers running on the same host, each should only have access to the secrets that app needs. For example, app A might have access to the MySQL credentials, and app B might have access to AWS Access keys for Dynamo, but A can't access Dynamo and B can't access the MySQL.

If I had a secret on each server then this wouldn't be hard. I could use a tool like Crypt to read encrypted configuration data out of etcd and then use volume maps to selectively make credentials available to individual containers.

The question is, how the heck do I get the keys onto the host safely.

Here's some things I've considered and why they don't work:

• Use AWS roles to grant each host access to a encrypted S3 bucket. The hosts can then read a shared secret from there. But this doesn't work because S3 has a REST API, Docker doesn't limit the network access containers have, and the role applies to the whole host. Thus, any container on that host can read the key out of S3, then read all the values out of etcd (which also has an unrestricted REST API) and decrypt them.


• In my CloudFormation template I can have a parameter for a secret key. This then gets embedded in the UserData and distributed to all hosts. Unfortunately, any container can retrieve the key via the metadata service REST API.


• Use fleet to submit a global unit to all the hosts and that unit copies the keys. However, containers can access fleet via it's REST API and do a "fleetctl cat" to see the key.

Basically, it seems like the core problem is that everything has a REST API and I don't know of a way to prevent containers from accessing certain network resources.

Ideas?