Aside from the jaws hitting the bed ways (what good is a chuck if you can't use it with the jaws opened up all the way), I like having my primary chuck smaller. Spins up & down fast, less weight to lug around.
Seems like the norm is to have a scroll chuck smaller than the 4 jaw. Of course this depends on the work you do. Lathes these days usually come with a 3 jaw one size smaller than the 4-jaw, except when you get into a certain size lathe & both chucks are the same size.
My 12x36 came with a 6.25" 3-jaw & an 8" 4-jaw. I felt these sizes were perfect for my lathe. So when I replaced them I stuck with 6.25" 6-jaw & 3-jaw and for my 4 jaw I also got 8" again but I went with a direct mount. That eliminated the additional weight & stick out from the adapter like on my original 4-jaw.
I primarily use my scroll chucks. I break out the 4-jaw when I need to hold something larger or of course when I need the independent jaws. I vote for the larger 4-jaw as long as the jaws clear but if I primarily used a 4-jaw and didn't need the extra capacity I would have choosen the smaller size.