If you're using msysgit with the OpenSSH tools, you need to either create ~/.ssh/id_rsa, or create a Git configuration in ~/.ssh/config which points to your key.
Here's an example of a Git configuration for Bitbucket that will use the correct username, and a key other than the default key (in case you maintain one key for SSH connections, and another for Git accounts).
~/.ssh/config:
Host bitbucket.org
Hostname bitbucket.org
User git
IdentityFile /C/keys/yourkey.key
Once in Git Bash, you can run two commands to add your key to your current session's ssh-agent to avoid having to repeatedly type the key's password.
eval `ssh-agent`
ssh-add /C/keys/yourkey.key