Point of interest, armies started using the spear because it didn't take a lot of skill to use and reduced the edge highly 'trained' men with swords and axes and such had.
That having been said, one of my primary beefs with any discussion about weapons in combat is people frequently forget that battles between armies require different training than duelling. If you had an armored knight against an unarmored man who used the florentine style of fencing, my money would be on the fencer, every time. He may not win every time, but I give him at least a 70%. You put a hundred fencers on the battlefield and put them up against a hundred knight on foot with their armor, my money would be on the knights. The reason why swords got lighter as time went on was that heavier blades were slower. When everyone ran around in armor, this wasn't a problem, but as armor began to loose its advantage (against crossbows and later, gunpowder) mobility was more of an equalizer. Fighting on a battlefield has way more dynamics than a duel. You've got formations, friends, enemies, differences in terrain, in numbers. All of these affect how you fight. In a duel, you've got you, and yoiu've got the other guy, you're both on the same terrain usually and you're usually in a controlled field that limits distractions. A lot of the random factors are cut down and you can concentrate on the fight.
Also a hundred peasants with spears have a good chance against cavalry, but I wouldn't give you twenty cents for the chances of one peasant with a spear against a cavalryman off his horse. Spears, axes, swords... whatever. Length matters, but not that much. The determiner is skill. I've had enough fights with different weapons to know that your chances of winning are most affected by TWO things, knowing how to use YOUR weapon, whatever it is and knowing what the enemy might do with his. There are plenty of tricks to mitigate the length of weapons in combat. An expert knife fighter is going to wipe the floor with someone of average skill with any weapon, spear, club, sword, dynamite.. whatever. I'm sorry, as far as the broomstick vs katana/shortsword analogy goes, if I'm better with the sword than you are with the broomstick, your first attempt to block me or hit me and you're going to pull away a much shorter broomstick. A spear is better than a broomstick, but well trained soldiers are not just given a spear, but also some kind of sword in order to use when things got too close for spears. The big disadvantage of any pole arm is that they are unweildy. If you're unskilled or not very well trained, you will not realize this until it's too late and you would have been better served fighting with the same skill level with some other weapon. Skill with a weapon is not simply being able to hit the other guy with the sharp end. It's knowing and using your advantages while trying to limit the advantages of the other guy. Whoever can do that better usually wins.