It all depends on what you're using the collets for. I have R8, C5, and S2 collets by 1/64". They allow the use of number and letter drills, as well as end mills with different size shanks. I have some 3/16" end mills with 3/8", 5/16, 1/4", and 3/16" shanks. I also have many end mills under 1/8" where the shank and cutting end are the same diameter.
Most drill chucks are limited to a minimum size of 1/16", and end mills should always be used in a collet. Without the smaller size collets there would be no way to hold any of these tools. I wouldn't try single pointing any thread on a lathe without a collet. There are lots of screw sizes from size #0 (.060) to size #10 (.190) that require collet sizes smaller than 1/4"