Disconnecting both output and input of the VSD should not really harm the VSD if all power is disconnected first. But doing so can introduce a myriad of other problems. One of the main reasons for not putting any device between the VSD and its associated motor, like a switch, circuit breaker or connector is the danger associated with disconnecting under load. When disconnecting a motor under load, the motor generates a back EMF (Voltage) that could be of a magnitude that would destroy the output devices (Transistors, IGBT's) of the VSD. The other reasons were already stated above by other respondents, namely the match required between the VSD and motor. Any items that could cause variations in impedance, capacitive, inductive or resistive, will cause mismatches in the tuning between the VSD and the motor. Typically contactors and switches can introduce resistive changes through contact resistance. changing cables can introduce capacitive and inductive changes. All these changes can cause a mismatch between the previously ideal matched VSD and Motor with resultant loss of ideal control. One would ideally want the connection between the VSD and the motor to be as "stiff" as possible. It is also very important to keep the earthing or grounding between the motor chassis and the dedicated earthing or grounding tab on the VSD as sound as possible. A separate earthing cable between the two is ideal. For lightning and surge protection, the ideal would be to have surge arrestors on the supply side of the VSD, together with well and proper earthing arrangements in the event of not being there when the lightning strikes. A double or triple pole isolator would be the best method to isolate the mains from the VSD during lightning storms. For single phase, use a double pole switch (life and neutral), for three phase, use a triple or quad pole switch (3 phases and neutral). Switching off only the life lines will cause lightning surges to still reach the VSD via the neutral and earth wires.
By the way, in industry some installations require multiple motors to be connected to a single VSD. In such cases it is normal for every motor to have its own motor protection relay. In the event that motor has a fault it would isolate or trip that motor. The other motors still in circuit will protect the VSD from back-EMF damage. The motor protection relay that tripped will then normally switch of the VSD to ensure total system integrity. So yes, one gets disconnecting circuits between VSD's and motors, but in this special case, the remaining motors protect the VSD.
My recommendation is that you do not regularly disconnect your motor from the VSD as it increases your chances of loosing the VSD.