Factions are only related to the chance of interacting with NPCs in quests, possibly later to prices or selections in trades as well, or the behaviour of animal tribes regarding your presence.
They don't affect the success of the skill, even though they may have the same name. A "Magic way" is a skill, but a faction is a different concept.
In brief: No reason to be afraid of.