Well, one of the most popular reasonings is that heavy armor simply lowers the caster\'s range of movement. If casting a spell is as simple as reciting an incantation or something else that doesn\'t require movement, then I\'d say its silly for armor to penalize. However, if a spell has a physical element, I think heavy armor really should penalize. Thusly, difficult spells may require the caster to wear less armour (although the more difficult spells would likely be more powerful, making up for the lack of armour). In such a system, it should be difficult to change armor in the midst of combat.
Well, just a thought.