I would get one C6 type long bit and try - especially before "investing" in PB Swiss.
I think that will solve the problem.
But I would definitely also dispatch an e-mail to Fein, and ask about it. Is it supposed to lock "E6" type as secure as "C6" type bits - or not? Because I'd say, that the moment you leave the 1"/25mm range for 2"/50mm or 4"/100mm bits, it's the "E6" type that is most commonly found in those lengths. I guess you'd have to search thoroughly to source long "C6" type bits locally. (I honestly wouldn't know a single (brick and mortar) store here that could sell them to me over the counter upon asking for them.)
So there is this 1% left telling me, it might not be supposed to wobble like that, and maybe a warranty replacement for that Fein QuickIn bit holder is in order?
Besides that, other owners could chime in, if it's the same for them. Would be sort of an answer also.
Kind regards,
Oliver