While lowes and HD have some metric bits, if you live near Menards, they carry a lot of them.
I would second mcmaster, they will have anything.
Also if you are in a decent size town, find a local machine tool/fastner supply shop, they will have all the metric drills you should need.
Far as the inch bits being a bit small, it will go. You aren't working in steel. That .004" interference is nothing. I've drilled with the inch bits and put 8 and 6mm dowels in, they go right in.