How do I add or remove security groups from ENIs using Boto?

I'm using the Boto Python interface to manage my EC2 software-defined networking and I'm writing a method to manage Security Groups on Elastic Network Interfaces (ENIs).

I don't know how to tell EC2 to add or remove security groups to/from an ENI.

My approach thus far is essentially:

import boto
conn = boto.connect_ec2()

my_eni = conn.get_all_network_interfaces(['eni-xxxx1234'])[0]
my_eni_groups = my_eni.groups
my_eni_sg_ids = [ x.id for x in my_eni_groups ]

desired_sg_state = ['sg-xxxx1234','sg-xxxx5678']

# if present, do nothing, else, add it somehow..
for sg in desired_sg_state:
    if sg in my_eni_sg_ids:
        print('Okay: ', sg)
    else:
        # not sure what method to use here!

I searched through the documentation and couldn't find anything about association/disassociation of Security Groups in the boto.ec2.securitygroup or boto.ec2.networkinterface objects. I'm certain that there's a way to do that, but it's not obvious to me.