I wouldn't depend on the "key" which is actually just a set screw to transmit torque to the collet or tool holder. The purpose of the setscrew is to locate the collet in the same orientation each time it's installed. I have seen a number of them damaged by not tightening the drawbar sufficiently. In some cases they trap the collet and make it extremely difficult to remove. In other cases a damaged setscrew can make it difficult to install and properly seat another collet.
Many professional shops remove the setscrew so they don't have downtime because a collet or tool holder spun in the spindle. The setscrew in my mill was damaged when I bought it. It was a pain to remove since the threads got damaged. I replaced mine and have been careful to properly tighten the drawbar and not overload the collet with too aggressive a cut or an oversize end mill.
If I recall correctly the setscrew has a 1/4-24 thread, so replacements aren't usually found at the local hardware store