Canvas Video and CORS Issue in IE and Safari

I have a canvas element that uses getImageData() and putImageData() onto a canvas. My <video> element has a property of crossorigin="anonymous".

The video is coming from S3/Cloudfront. My bucket has this CORS config right now:

<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://ift.tt/1f8lKAh">
    <CORSRule>
        <AllowedOrigin>*</AllowedOrigin>
        <AllowedMethod>GET</AllowedMethod>
        <MaxAgeSeconds>3000</MaxAgeSeconds>
        <AllowedHeader>*</AllowedHeader>
    </CORSRule>
</CORSConfiguration>

This works great in Chrome and Firefox. I am successfully drawing video onto the canvas.

My issue is with Safari and IE (11 specifically, but from what I understand it's an IE10+ issue).

When I attempt this in IE or Safari, I get this error:

Unable to get image data from canvas because the canvas has been tainted by cross-origin data.

I understand the error because before I setup CORS, chrome and firefox was giving the same error. Adding CORS fixed it for chrome and firefox but not IE and Safari.

I noticed the response headers of the video were different from say chrome to IE.