Altimeter Setting Differences

Hello Everyone, my first post here... When I used to fly as a VFR rated pilot in real life, I started each flight by setting the altimeter by twisting the baro knob to set the altimeter to the field elevation. What I've found in X-plane is that there's a difference in the baro setting gained by setting the field elevation and the baro setting given by the ATIS of the field I'm located at. Shouldn't I fly with the altimeter setting of the field and not by the one given in the ATIS? I do change the altimeter setting to the destination airport before landing, so maybe it is a moot point. Thanks for your feedback.