I think a 3HP VFD will work fine. A VFD will normally run at least a 125% of it's rated power for some period of time. It would be rare that you would ever hit the FLA on you mill.
You can set the spindle speed in the G-code by a M3 S 1000 command, where M3 is spindle rotation CW, S 1000 tells it to run at 1000 RPM (M4 is CCW rotation, M5 is spindle off) There is a way to feed back the spindle speed to Mach3, but I'm not sure how to do it. Google will be your friend here, I also think there is a thread on that subject on H-M.
Most 17xxRPM rated motors are safe to run at 2x speed, or 120Hz. I would not do this with a 34xxRPM rated motor unless the motor is specifically rated for high speed operation, like a high speed spindle motor.