If you look at the system messages while fighting, you'll notice that the higher your armor skill, the more frequent the ">...but you doged", ">...but you blocked" and ">Your opponent failed to do any damage" messages are, which suggest you're more skilled at deflecting/parrying/evading attacks with the armor, all mostly related to moving in it with ease. Damage varies depending on what kind of weapon your opponent is using, if any, and what kind of armor you are wearing, if any, which kinda makes sense too.
Perhaps the problem has to do more with how many levels you need to train and the progression system rather than the logic behind the use or armor in-game.