Limiting Access to AWS Network when Using Hardware VPN Connection

I have set up a "Hardware VPN Connection" between a remote customer and my AWS network (the VPN is to a "virtual private gateway" of mine). Assume that I have no control over the IPs that my customer uses and they could very well be "malicious".

I would like to limit the IP addresses on my VPC that my customer can access. How would I accomplish this?

One thought was to create two VPC with a peering relationship and then just allow traffic from the VPC with the VPN to the other VPC going to the IP addresses that I whitelist. Unfortunately, AWS explicitly does not allow this configuration.