I have to stand with Weltall and Tessra on this one.
It's a bit of a far cry to ask that mages not be able to wear HA. That being said, this goes back to my age-old outcry against the HA users. There's no downside to wearing plate If a mage wants to wear HA, and he or she is properly trained in the art, than I have no problem seeing them wear it, as far as realism goes. However, said mage should be forced to deal with the exact same physical impediments that HA poses to any wearers. Namely, very little endurance, limited mobility, and greatly slowed attacks. It's full plate, people.
So yes, there should be certainly some downside to wearing plate as a mage. But that same downside should be applied to everyone who wants to wear plate.