That can be a number of things. Quality of drill bit or bent, uneven length cutting edges, tailstock alignment/tailstock chuck concentricity. You can sort of check this by after drilling a pilot hole, run your drill bit up to the hole with the tail stock & slowly enage the hole, look closely to see if the drill bit deflects.
Even if it doesn't deflect, that won't guarantee anything, still a drill bit doesn't make accurate size holes, it just one of the more efficient/quicker was to make a hole. If you need an accurate sized 17/64 hole, drill undersize then ream or use a boring bar for final dimension.