of course having both is the best.
I have found that finishing sander (I have a festool ETS125) does not work very well with coarse sand paper (like 40 and 60), i.e. it does not sand faster than with 80 sandpaper (I use only abranet).
However, 5mm sander are in general heavier and less ergonomic to handle (more vibrations etc...). The wear on the sandpaper (specifically the edges) is also much quicker than with 3mm sanders.
So I use a (cheap) 5mm sander for 40-80 grid, and a 3mm sander (festool ETS) for 80-220.