The easiest way to figure out how deep you can cut is using your chip load calculations. As any machinist should know this is the most important part of determining how clean the cut is and the life of your tools edge. For aluminum I prefer a chip load of .011 - .007. I have found the higher chip load produces a higher polished edge, however, the feed needs to be much higher. Here are a few of the formulas to figure out an endmill chip load.
Chip load = feed rate/(rpm x # of cutting edges)
Rpm =feed rate / (# of cutting edges x chip load)
Feed rate = rpm x # of cutting edges x chip load
Another very common rule to follow is to never make s deeper cut than the bit diameter at hand. Also, as stated earlier in the thread cutting lube is very important for your bit life not so much to cause melting but it will hear your tool up and round the cutting edge over.
Something else to think about is if there is any wobble since you are doing it by hand. A .01" wobble will cause a very odd cut. Lots of things can happen at that point since it would be affecting your feed rate, chip load, and in some cases aye of thinking your RPM.
Truly the best way to machine aluminum with an endmill would be on some type of mill or CNC. However, a great cut is obtainable through a hand router there is just a lot of variables. I am guessin when you tried to take more than a mm the reason for a unclean cut was the router having wobble. In my CNC I use a .25" single flute up cut aluminum bit by Onsrud and I make a .25" pass with it no problem. However my rpm is around 20000 and my feed rate will vary between 150 and 210 IPM. In other words a deeper cut is obtainable but not likely with a hand router.
I really like the work you have done with a hand router that's really inventive and cool.