These days, you'll have to keep the plastic in some kind of metallic sleeve. The RFID chip in US Green Cards, and I imagine passport cards as well, is designed to be readable from across a room.
But not from the phone, if you put it the wrong way around.
An iPhone can read the RFID but you have to know that it is NOT where the icon is, but actually on the backside of the hard thick page (or something, I forget).