OK Bob was correct, the nut is causing the problem. Specifically the spring-loaded part is moving under load. Many lathes dispense with that and have a simple one piece nut, you have some backlash but you just learn to compensate for it. It's a constant amount that doesn't change.
Try this: re-install the nut in the opposide direction if possible, in other words, if the spring loaded portion was toward you before, install it with that portion away from you and see if the problem is reduced.
Alternately, remove the spring-loaded portion completely