Wherever you are in this project, OR other "micro" projects, the Number One issue is holding then secure whilst doing machine (or) Handwork.
So many of us don't think of some otherwise Obvious concepts, we "deal with Machinery".
Try to Superglue your part, to what some term as a "pallet". Specifically, that is a larger Sacrificial BASE, which can be held via a Vice, or Hold Down Clamps. the Super Glue Works Best when there is "Zero Fit" between parts. In Other words, Flat on Flat, And of course CLEAN metal: Acetone then Alcohol. One Drop of CA per Square Inch of Surface.
After all Milling, Filling to remove, choices abound Heat (but that is Stinky) Freeze! the Dissimilar Metallurgical Properties will cause a difference in Shrinking, and Break your bond!
I hope that helps.
philip, from the Great Pacific NorthWET.