If nothing else appears, you could make and harden a collet (you have one to copy) or make a split sleeve bushing to fit in the larger collet. The loads should be pretty small, just need it to have little runout. A bushing split from both ends at right angles to each other seems like a reasonable idea to me. You might be able to find one commercially made.