I wouldn't try to hammer a broach. They are hardened and might break.
The pulley is aluminum and pretty easy to work. I used a small HSS bit in the lathe using the carriage as the press. You can get a little pack of HSS bits at Harbor Freight. They aren't great, but they are good enough for this if you don't have any little ones. I just used it directly, but it works better if you grind a little relief. There are a bunch of youtube videos showing how to do it.
There's always the small files as well.