I’ve taken way deeper cuts in aluminum, it’s all about how you approach it.
If I want a nice finish I’ll use a good mill (I’m spoiled for choice at work) and take a light cut with a fast spindle speed.
If I need to remove material I’ll use a ****ty mill and I’m not afraid to take upwards of .100 with power feed and a slow spindle speed.
Not going to get too far into specifics as I don’t want to give people bad ideas and help them ruin their machine.
There are a multitude of grades of carbide, some way tougher and more tolerant of abuse than others. But I do agree with using HSS over carbide, especially over an insert.
ETA: yes, having a flange is worlds different than what I had envisioned.