Drill sharpness, drill tip/end geometry, etc are all important issue. It's already been covered.
Just a thought, hope it will be useful.
For this material & thickness, and the size of drill, I've eyeballed touch up (grinder) drills for sharpness, without this issue. Aluminum is/can be gummy, "sharp" drill major plus, and lube.
Seems that for 1/4" thick aluminum and a 1/4" drill 450 rpms is a bit slow.
If I have done my math correctly. I've only got 3 brain cells and I'm over 60 yrs, so reading, thinking and typing are really pushing the juice.
AL cutting spd is 350-400.
4 x 350 = 1400; 1400 divided by drill size .250 = 5600 rpms.
4 x 400 = 1600; 1400 divided by drill size .250 = 6400 rpms
Maybe in the CNC world with coolant feed - 5600+.
But at 1500-2000 and a bit of cutting oil, a 1/4 drill should go very nicely.
Try increasing the rpms, a little cutting oil, and avoid "busting" thru.
Side note: I have 2 lathes and a BP mill. Applying cutting oil. Get a small 1/2" best, 1" is good,
non-synthetic paint brush. A "small" can (fruit/juice type) pour in cutting oil. Use the brush to apply cutting oil to the drill, also works very good when threading (int/ext) in the lathe.
Best regards.....carl