Authentication failed for user when deploying to Amazon EC2

I am trying to run cap production deploy under my project folder using Capistrano 3. I am able to ssh into my production server using the pem as login credential. But when I run cap production deploy, I get the following message:

cap aborted! SSHKit::Runner::ExecuteError: Exception while executing on host cityspade.com: Authentication failed for user ec2-user@cityspade.com

The EC2 server has already allowed my IP for SSH connecting. No idea how to fix this issue.