I've switched colors mid-print and haven't had much problem. If you heat the hot-end to extrusion temp, then pull the old filament and feed in the new you should have only specs of the old color and only briefly - certainly not enough to print a brim for even a small part.It's also pretty helpful when you switch colors and forget to purge the old color out of the nozzle.
except, on small prints the brim isn't quite long enough purge out all of the old color. being forgetful gets you some interest first layer color effects.
GsT