I sometimes ease the QCTP up to the bit just enough to steady it once it touches to prevent it from getting off to a wobbling start. When a cut is just starting, if the edges are not perfect, and the starting hole is not dead true (two difficult conditions to meet), it can easily grab a little and start an uneven cutting action that is impossible to correct without external measures.
If your bit is long, of course it's going to be more limber and this makes the problem worse. If it is sort of critical that your hole be straight, then start off about 1/32" under and go about 1 diameter deep. Then come back with a stubby boring bar and open the hole up to the drill size, maybe a few thousandths under. This will allow the drill to cut with very little deflection. But when you make contact with the point left by the undersized, and presumably out of true center, go very slow with the feed and it won't generally affect the full size drill much. If you just drill with normal or heavy feed, it will try to follow the pilot point and you will lose at least part of your improved concentricity.