That's actually fairly common. Two main causes, the head of the mill isn't trammed in accurately. Does it do the same thing both directions? How big of a cut are you taking? When cutting, something deflects, with fly cutters, it's the cutter. Let's say you're taking .015 off each pass. Because of deflection, it might only be taking .013 off the first time. On the second time it hits the material, it's taking .002 off, possible even .001. Making a couple dry passes( at the same setting) will help out.
Jamie