Rails OpenID Connect Provider

Backstory:

We have a functioning SSO service which uses OAuth2. I need to add in the functionality to be part of an OpenID Connect workflow for Amazon STS.

I've implemented the /.well-known/openid-configuration endpoint which returns the following (as json of course)

{
  "issuer"                                => url_for,
  "authorization_endpoint"                => oauth_authorize_url,
  "token_endpoint"                        => oauth_access_url,
  "jwks_uri"                              => openid_jwks_url,
  "response_types_supported"              => ["code", "code id_token", "id_token", "token id_token"],
  "subject_types_supported"               => ["public"],
  "id_token_signing_alg_values_supported" => ["RS256"],
}

I've created an RSA key with the following command:

openssl genrsa 1024 -sha256

I've stored that and parse it into the following:

App.jwk[:private_key] = OpenSSL::PKey::RSA.new(PRIVATE_KEY)
App.jwk[:public_key]  = App.jwk[:private_key].public_key
App.jwk[:exponent]    = "AQAB"
App.jwk[:modulus]     = Base64.encode64(App.jwk[:public_key].n.to_s)

Amazon then successfully makes the connection back to retrieve the jwks, which contains the following:

{
  "keys" => [
    {
      "alg" => "RS256",
      "kty" => "RSA",
      "use" => "sig",
      "exp" => App.jwk[:exponent],
      "mod" => App.jwk[:modulus],
      "e"   => App.jwk[:exponent],  #Im not sure which of these values I'm supposed to return
      "n"   => App.jwk[:modulus],
    }
  ]
}

I then generate a jwt for the user, which I sign with the same private key from above, and send it to Amazon with the AssumeRoleWithWebIdentity API call, and amazon accepts the JWT with no problem, but it always returns the following error:

InvalidIdentityToken => Token signature invalid

I've tried many different permutations of things to try to get this to verify, and I'm out of ideas. I suspect it has something to do with the jwks, but from the docs I'm reading, I can't see what I'm doing wrong.