Is there a set screw in either of these two holes that might retain the worm screw shaft?
View attachment 439878
If so, it may be a double set screw... two in one hole, so if you remove one, check the hole to see if there is a second one.
There may be a dowel pin in one of those holes with a tapped hole in the end of it to pull it out instead of a set screw.
I believe you said you removed the collar from the bottom... did you also remove the 4 flathead screws and the round plate under them?
-Bear
Edit: I went back and read your first post, I see you did remove the 4 screws...