Here is what I practice.
If you are using a center drill for its intended purpose, (i.e. providing a bearing for turning on centers) then I would say smaller is better if you plan to use a dead center. The smaller bearing surface has less friction due to the smaller area and lower sfm. It does have to be large enough to support the forces generated in the turning process without distortion
.
If you are using a live center, then friction is not an issue but as WreckWreck stated, it becomes more difficult to work in close if the center drilled hole is large.
A helpful exercise would be to observe the center drilled holes in turned shafts such as motors, axles, drive shafts, etc. It will give you a visual as to what is practiced.
If using the center drill for spotting a drilled hole, I usually go smaller there as well with a hole large enough to seat the central web on the drill. I try not to drill so deep as to create a vertical wall as the following drill can bite into the sharp edge and deflect slightly, defeating the purpose of spotting the hole.
Bob