There is no "disadvantage" - it's now the default. Wearing armor then gives you an advantage as it protects you from damage, the way it works in every other game 
If you want to move quicker, train Agility. It doesn't rely on any armor type, and is exactly what you're asking for.
You avoiding the answer

Well.. all right "every other game?"
Never thought this should be like every other game, where everyone runs around armored and armed like battle vehicle... but thats minor now.
How does the armor works? Did we say nonsense? I dont think so. The way explained in Hassadria's post is reasonable.
In my country was great war in medival ages, where heavy armored army lost battle against just "peasants in clothes" Why? Cause they was able to move way faster then armored knights. Why should it be in game different?
Light Armor -
Its protection is not great. The protection of such armors flows from the ability to move fast, unrestricted. Like there was said, the "move-ability and knowlidge how to move to avoid, or ease the hit" counts here. So it should count for no armor at all too.
Medium armor -
is combination between light and heavy
Heavy -
Here is the skill still used for ability to move and knowlidge about the hard points of armor. Fighter moves himself to try to avoid the hit, but its heavy, so its often not possible (noone can say that you move same fast with clothes and with 60Kg heavy plate armor on you) So the fighter need try to move, so the blow is delivered into hard point of the armor.
Point me if there is something wrong on this theory, and we accept it

If its reasonable, it will be nice to ponder about implementing in game. Every kind of protection should be usable, not only the "best" and nothing more.
Thanks for at least thinking about it. (armor might be not suitable word for this... "L,M,H protection", or "L,M,H defense" fits way better)