Irrespective of how smooth you finish your timber before painting, the act of "whetting" the fibres of natural timbers with the first coat tends to raise the ends up sufficiently to roughen your lovely smooth finish. I've found it desirable to "denib" these raised fibre ends after the initial coat, although I personally prefer the very softest of papers in 240-400, most usually the coarsest of the range.
Subsequent coats are usually taken back a bit with 400 between coats, just to ensure adhesion & smoothness. The lightest of sands, particularly by hand, tends to highlight imperfections, foreign bodies, runs etc. that might otherwise be missed. One's fingertips seem particularly sensitive to the finest of lumps & undulations that might be felt through a folded sheet of abrasive.
I've personally never bothered with any more than an initial, single coat of primer, but then again I'm not using the same type or variety of finishes either.
