How do I restrict access to a static s3 website to a VPN

I'm trying to secure access to an internal static website.

Everyone in the company is using a VPN to access our Amazon VPC so I would like to limit access to that site if you're using the VPN.

So I found out this documentation on AWS to use VPC endpoint which seems to be what I'm looking for.

So I created a VPC endoint with the folowing policy.

{
  "Statement": [
    {
        "Action": "*",
        "Effect": "Allow",
        "Resource": "*",
        "Principal": "*"
    }
  ]
}

On my S3 bucket, I verified that I could access index.html both from the regular Web and from the VPN.

Then I added the following bucket Policy to restrict to only the VPC Endpoint.

{
  "Id": "Policy1435893687892",
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1435893641285",
      "Action": "s3:*",
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::mybucket/*",
      "Principal": {
        "AWS": [
          "arn:aws:iam::123456789:user/op"
        ]
      }
    },
    {
       "Sid": "Access-to-specific-VPCE-only",
       "Action": "s3:*",
       "Effect": "Deny",
       "Resource": ["arn:aws:s3:::mybucket/*"],
       "Condition": {
         "StringNotEquals": {
           "aws:sourceVpce": "vpce-1234567"
         }
       },
       "Principal": "*"
     }
  ]
}

Now Regular Web gets a 403 but I also get a 403 when I'm behind the company VPN.

Am I missing something?