I never use collets for drilling. If I use a larger end mill, I use a holder. I think that most people who have problems with the tool being drawn out of there collet during work don't have something set up right, the tool is dull, or perhaps they are pushing a dull tool past its limits. A properly tightened collet is sufficient to drive an end mill. You will usually get better runout of the tool, which makes your work easier, and is easier on the tool as well.
Oh, and if you use a holder for an endmill, with the set screws locked on the flat, it won't spin; something will break.