The bearings might help with side loads, but you won't be able to hold the tool securely. End mills want to pull out of the chuck. That force will try to pull the spindle down, likely pulling whatever you have on the taper off. Once that happens, you have a tool and whatever held it flying around. That's assuming you have some way to lock the spindle so it doesn't get pulled down. You might be able to bore it out with a collet taper and hold down. You would probably need a lathe for that, but you might be able to rig a tool holder on the table and use the spindle to feed.
You might be able to tap the spindle taper to hold the drill chuck on with an internal screw. That doesn't solve the problem of holding an end mill in a drill chuck, but it would keep the chuck on. Drill chucks just don't have the grip required to hold an end mill though.
Perhaps build an end mill holder that will match the taper with a hold down screw and use a weldon shank end mill with a set screw to hold it in.
I looked into this when I was getting started. I don't have the same drill press, but what I found was that you will spend the same or less on a mini mill and not be likely to kill yourself. In the event it does work as you expect, you would be lucky to take cuts of 5 or 10 thou. That gets old real fast. Even a mini would do a lot better. You also haven't dealt with X/Y movement. If you intend to use a cheap X/Y vise, I wouldn't. The screws are not designed for the forces required and will shift around. I've seen some that I would bet would snap from the cutting forces.
Drill presses usually can't run slow enough for milling. You also need to deal with that.
This winds up being a lot like using a butter knife as a screwdriver. Sure, you can, sort of, for very limited applications. But an actual screwdriver works a LOT better.