The answer is: If you do nothing but go around killing or stealing from people, then people will kill you on sight or find someone that can.
Death won't do, death isn't negative enough for us players. However I'm with Duraza and saying, don't just throw them in jail. Execute them! A multiple murderer won't spend a few weeks in jail. As soon as s/he is arrested by guards or defeated by players who then bring them to the guards, tie the criminal to a Megaras and off to the Azure Sun with em!
Promise True Death, the ending of their character and their precious skills, to whomever breaks the law too much, along with other very frigthening punishments and then I might be able to live with open PvP.
As for the player guard thingy, it's easily monitored. In fact we already have the monitoring system implemented, called: Factions.
Any character with a chaos, thieves, .. or whatever other lawless faction will not be able to become a guard. Any normal character will not be able to become a guard. Only characters with a high law and guard faction will be able to become a guard.