A yellowish brown is probably too dark for a cutting tool for steel. A barely perceptible color to a light straw color is probably more in line for a rotary broach. the steel should be polished to remove any color prior to tempering. View in a bright light. The color is due to surface oxidation of the steel. Different thicknesses of oxidation affect different wavelengths of light much like an oil film on water. If the process is too slow, a false reading may occur.
A Google searching for tempering colors will provide all the information you need.