So I removed the washer and picture 1 shows where it is, but before I removed the washer it was in picture 2 location. The problem is that the position can change depending on how I set the compound, and when it's in the wrong position it can crash.
It looks like previous owners used the compound as an anvil...
Another thing I have trouble with a 4 way is that it seems boring bars have different level compared to normal tool bits, I don't know if this is by design but say a 20mm tool bit requires another mm of shim, a 20mm boring bar on the other hand requires nearly 12mm of shim to be up to the center level. Not sure why.