To the OP: I faced the same situation. I put up with a so-so quality drill press for years, using it as part of building two cars from scratch. A lot of chatter, and the spindle was always slightly loose, so when you'd apply any pressure, it would skew sideways slightly, but I made due. My "shop" consists of one half of a two-car garage, where the shop machines sharing space with the current project car, so space is always at a premium. Like the OP, I started thinking about retirement and how to set up the shop for the long term. The drill press was wearing out, so I used the chance to make an enormous upgrade and replace it with a proper mill (PM-935). That, in combination with buying a slightly shorter-bed lathe, meant that everything still fits. The mill does everything the drill press did and so much more. No chatter, and it was the first time I ever drilled a hole where the only sound I could hear was the slight hiss of the drill bit removing the metal. Accuracy of hole placement, variable speed, DRO, it's an amazing upgrade if you can swing it. So, I have no regrets and never looked back.
That said... "if" you have the space, then sure, have both, but if you have room for only one, it's a pretty clear choice.