How do you know what key a song is in by ear?
In a lot of metal, the tonic note will be the open 6th string (E, D, C, B, A, depending on the tuning). Once you can identify the tonic chord, based on this resolution feel, it's again just a case of finding the root and quality (major/minor) of that chord on the fretboard. This will tell you the song's key.
How do you learn to recognize notes by ear?
Pitch ear training: Train your ear to recognize notes by playing the same note over and over while singing or humming it, and associating the sound with its name in your mind. The more clearly you can hear a note in your head, the better you'll become at identifying pitches.