Thanks for everyone's comments. The quick answer is the mystery thread on my vintage device is 10-32.
Looking back at it now, I wouldn't call a 10-32 nut on an 8-32 stud a "good" fit. When I tried this again, the nut I tested does thread rather than slide, but it's pretty wobbly. By comparison, when I put the mystery nut on a 10-32 stud it's a very solid fit. And vice-versa for a 10-32 nut on the mystery stud.
In any case I appreciate the posts, and picked up a few ideas. BTW I do have several mikes and a thread pitch gauge, but what came in the handiest was when I was digging around in drawers I found a pair of inch and metric sets of "Thread Detective" testers. This allowed me to verify 10-32 and rule out at least the other common nearby sizes.