Right, this is something which I have noticed, but has never worried me in the slightest..
When I unscrew my fuel filler cap, pressure can be heard being released.. I'm not sure weather its being sucked in or pushed out though.
But it does it every time, even if I leave it for an hour...