Swords were pretty useless against plate armor. Plate armor brought about a switch to blunt weapons, because pretty much all piercing and slashing weapons were rendered close enough to useless not to be worth it. Daggers still had a use, though. While no knight would charge into battle with knife extended, in the event combat turned to grappling, or in the event a plated warrior was taken off his feet, it was the dagger that was best suited to administering a death blow - but then, only once the opponent was fully subdued and pinned. If daggers need to be a waste of time leveling, so should swords and bows. Really, the only weapons that should be remarkably effective should be hammers, maces and the like. That's the realism if realism is the goal - not sure it should be.
If realism is the goal, and nigh invulnerable plate armor users (and they were neigh invulnerable until the advent of firearms), plate armor users should risk collapse from overheating if they don't take breaks from wearing the armor, and particularly if they fight in it for extended periods. The idea that plate armor rendered one immobile is largely a fallacy. Not a total fallacy - tournament jousting armor was indeed so specialized and heavy in some cases, that one did need help to mount a horse for example, but show arms and practical arms were two different things - same with sword weight. I have a replica zweihander from renaissance Landeskenechts and it weighs a good 15 pounds. Parade swords could sometimes get so unweildy, but combat swords never did.
So anyway, Real plate armor should be largely invulnerable to bladed weapons, reasonably vulnerable to blunt weapons, and the user should face penalties to endurance - passing out or worse, if he fights extended periods in any sort of warm weather. One idea about plate armor that isn't a fallacy is that the stuff was a microwave. Thick padding, heat conductive metal - it was a serious problem.
EDIT: In summary, if realism is the key, the tradeoff for practical invulnerability in combat to swords, daggers and the like should be my seeing random heavy armor wearers passed out here and there if they wear their armor everywhere they go, and passing out from protracted combat. To further the realism, the dagger users aught be able to take the opportunity to administer a near 100% chance death blow into a gap in the armor, such as the visor. THAT was the dagger's raison d'etre, and if its weaknesses are being fully recognized, so too should its strengths. Fair is fair.