Specifically, what are you using as a drill bushing?
McMaster Carr #8492A327
Press-Fit Drill Bushing with Head, 0.625" ID, 3/4" OD, 1" Long
So I use a regular 5/8" drill bit with that bushing and I made an aluminum piece which helps hold my vac hose in place (not needed)

The 5/8" bit will remove most of the material, leaving a hole that looks like this:

I don't plunge the router bit. I use a foot switch and turn it off, place it in the hole and then turn it back on to make the cut. This is what the hole likes like when done
