Yea....I also did not realize that your vise is an angle vise plus a swivel vise. Personally, I would be very careful before you do any "permanent fix" to your mill just to make that vise properly trammed.
With a swivel and and angle...You just have too many variables in that set up. If you must use that vise...Then I agree with others who recommend shimming it to get everything trammed. Long term, I'd look for a good basic mill vise without a swivel and without the angle. As for as the vise you now have, just leave it under the bench until you need an angle vise. It's not like it won't ever come in handy.