Ouch...
I'm running 64 bit linux (gentoo), with an AMD 3200+, 1GB RAM, and a nvidia 6600GT. I also have 2 GB of swap, split between two hard drives. Planeshift normally runs fine, even with a server, npcclient, and two player clients.
I don't know if it's the same for you, as I have my planeshift binaries compiled in debug mode, which would make bigger binaries and use more memory... but I did a little test, "swapoff -a" first, and wait for all my swap partitions to be deactivated, then start one client and log into the test server. It was fine loading the login menu, but as I started loading (in the laanx dungeon) the memory usage went up to about 950 MB and slowed to a crawl, after I waited a while, psclient finished with the message "Killed".
So, I'm not sure if you have it already, but it would probably help to add a swap partition, or enlarge the one you have. I don't know if planeshift used more than 512 MB in my test, but when testing a while back with a debug client and a big map like bronzedoors I have seen the memory/swap usage go above 1 GB total

The memory issues are noted in the bug tracker, some of the developers are working on things that hopefully will improve it for future releases. One other thing you can do now, is turn off sound... the current sound code is buggy and uses lots of memory.