If you don't mind taking inspiration from other mmos...
Ryzom is another skill based mmo with the same kind of open/unrestricted character development that PlaneShift has.
One of their key mechanisms for maintaining 'balance' between magic and melee is equipment.
- Wearing heavier armour debuffs your magic abilities.
Equipping magic amplifiers (in the weapon slot) buffs magic abilities. The buffs are classed - i.e. applied to PlaneShift each amplifier would have a separate modifier for each of the 6 magic Ways.
This system allows unrestricted character development without classes, but also prevents any single character from becoming an overpowered tank/mage/healer (all at the same time). Their setup has an activation timer for equipping gear to prevent players using macros to quickly change all their gear to circumvent the restrictions.
An easier alternative to timered equipping might be a simple spell buff attached to magical staves.
ExampleA player is in a group and intends to act as a 'healer'. They equip a magical staff with modifiers for the 6 Ways.
- Crystal - 40%
Red - 10%
Brown - 20%
Azure - 20%
Blue - 30%
Dark - 10%
No bonus is given just on equipping the staff. The player must cast a spell which 'activates' the staff and applies those bonuses. That spell takes 30 seconds to cast.
The player's healing spells are now 40% stronger and they get small bonuses to their other spells. If the group changed and they needed a damage caster, the player would need to equip a different staff and activate it (taking 30 seconds).
________
This setup would still technically allow anyone to cast spells, but the potency of spells cast by players not wielding activated staves would be weak - i.e. like they are now.