PlaneShift
Gameplay => General Discussion => Topic started by: ChrisM on December 07, 2003, 12:59:42 pm
-
Hi,
I started working on a own multiplayer game and faced the choice to use either TCP or UDP.
First I started working with the TCP-protocol but than I saw your network engine. :)
I tried to unterstand your source, but just a few questions:
- When do you send those ACK-packets, always or just on high priority packets? And how do you guarantee that the ACK packet is coming through? I mean, otherwise the other connection endpoint would send the packet again and again until you ACK comes through.
- How does those multipackets work? Many unimportant pakets (like weather, which does not need to be sent immediately and can wait a few secods) are buffered and stored in a big packet sent when the buffer is full?
Sorry for my really bad English, I hope you understood my questions and could give me a brief answer. :)
ChrisM
-
most MMORPG\'s use UDP for lag and bandwidth issues, while FPS\'s, which generally have more bandwidth per player, is more dependant on accuracy, and have servers close to the players, can get away with TCP.
Don\'t know anything about PS\'s network code :p
-
Hi,
yes, but where is the advantage of UDP if you \"emulate\" TCP with UDP using ACK-packets and so on?
ChrisM
-
As i said, i have no idea how PS\'s network code is built up, but UDP gets routing priveleges and stuff, so even if you do emulate TCP, the latency should (atleast in my fantasy world) be lower.
-
TCP, generally speaking, is unsuitable for games, unless they are low bandwidth (like turn based games, such as chess). The reason being that TCP is guaranteed delivery and ordering, which on paper sounds like a great idea, but in practice causes many problems. If you start losing packets in a TCP system, because of the ordering, you actually cease all delivery until the next packet gets through. In fact, TCP will actually give up resending the packet after a while and will start sending small \"test\" packets to try to establish a path again. Once it gets the path back, only then will it try to resend the errant packet.
What it all comes down to is that TCP can actually lag by *minutes* at a time (occassionally), and it is not uncommon for it to lag by many seconds. For real time game use, this is unacceptable.
Therefore, most real time games use UDP with thier own reliability and ordering system on top of it. You may think this is re-inventing the wheel, but actually it is not. With UDP and your own system, you can selectively determine which packets need to be ordered and/or reliable.
I\'ve not looked at the PS code, but in my game, for instance, chat packets are neither ordered nor reliable. Item packets are reliable, but not ordered. Movement packets are both ordered and reliable. This means I can send chat data at any point without getting held up by a stray Item packet or movement packet like I would with TCP. I imagine PS uses a similar selective system.
-Ron
-
The problem with UDP is if you live far from your ISP, and the lines are bad, all the packets can be lost(and not resent), making the game unplayable for those people.
That said, I used UDP on a 2D game I made where you are a square and walk around on some tiles...hmm, not much replayability there :P
I also have no idea how PS\'s network code is built up, but I have combined speed.direction.inertia.xpos.ypos etc. into one packet to lower bandwidth requirements.
-
Originally posted by ChrisM
yes, but where is the advantage of UDP if you \"emulate\" TCP with UDP using ACK-packets and so on?
Because you use ACKs only for part of packets. For example if you send positions of people and loose a packet, there is already a newer packet going in on the wire so you don\'t need to resend the lost one. At the time you would detect and resend the lost packet again, the information it contains would already be out of date.
And this same problem exists in TCP too. TCP would also stop sending other packets because of the lost one.
-
Hi,
ok, thx for your very informative answers!
I think I will need to use UDP to get a more or less lagfree game... :)
Thanks!
ChrisM
-
Originally posted by RonHiler
Movement packets are both ordered and reliable.
But won\'t this stall the whole movement in case of packet loss until this is detected and the packet resent ? Why resend when following more up-to-date update will come faster and obsoletes the lost one ?
-
RonHiler that was an excellent post and very informative. That said, it is funny how your priorities are the opposite of ours. :-)
We have guaranteed delivery of chat packets, and non-guaranteed delivery of moving/position packets. The reason for this is that usually within a few msec the movement packet is obsolete and superceded by a new one.
A few more points:
- ACKs are sent to keep the source computer from resending guaranteed packets. Non-guaranteed packets are not ACKed and never resent.
- All messages can be merged and sent as one packet. The msging layer sends whatever it wants, and these items go into a queue per client. The networking thread wakes up every 50msec or so and merges whatever it can from a queue into 1 packet and sends it. It does this for each client, in the case of the server.
You could wait even longer to send VERY low priority packets but we pretty much assume that every client will be getting a fairly steady stream of data concerning other game events and sliding a weather message in there doesn\'t really affect much.
Hope that helps,
- Vengeance
-
Originally posted by tangerine
But won\'t this stall the whole movement in case of packet loss until this is detected and the packet resent ? Why resend when following more up-to-date update will come faster and obsoletes the lost one ?
Hmmm, well, it\'s a bit misleading what I said. In my game, there are movement packets and there are movement packets :) The type of movement packets that are reliable and ordered don\'t stall movement, and in fact it is possible to get them out of order, they are just reordered by the client as they get there (by server timestamp value as well as a incremental key value).
The reason they are reliably sent is that I don\'t send actual positions (well, actually I do, but only once they are *known* positions by the server at some given time, so they are always out of date, but that\'s getting more complex than needed for this discussion :) ). I\'ve found sending absolute positions every x ms to be a less than satisfactory method (perhaps you guys found a way to do this, but I couldn\'t make it work with any decent results, heh). I instead send velocities and facings. Thus, every single velocity change *has* to get to the client, there is no superceding one data packet with another, as each builds on the previous. If one gets lost and data comes out of order, the clients use interpolation to \"guess\" on where the player is based on the data they do have until the actual missing packet(s) gets there.
I\'ve found this works really well. There is still a good deal of \"popping\" in my game for 3rd party avatars, but that\'s for an entirely different reason (which will be fixed in A6 :) ).
Originally posted by Vengeance
That said, it is funny how your priorities are the opposite of ours. :-)
Hehe, yeah. It\'s odd how any given game\'s authors independently come up with many of the same solutions to some problems, yet entirely different solutions in other things.
The reason for this is that usually within a few msec the movement packet is obsolete and superceded by a new one.
Yeah, I\'d do it the same way if I was sending absolute positions every certain amount of time.
- ACKs are sent to keep the source computer from resending guaranteed packets. Non-guaranteed packets are not ACKed and never resent.
Ditto.
- All messages can be merged and sent as one packet.
I don\'t do this. I send each packet as a separate entity, even the really small ones. Yeah, I know there is some overhead with packet sending, but I found in my game the time spent trying to pull apart messages into individual packets just wasn\'t worth the effort :) I was spending more time checking packet lengths and pulling out the individual packets than I was saving by sending them together. By sending them as individual messages, I can just read the first bytes to see what kind of packet was received, then rip the individual data members for that kind of packet and I can move immediately into processing.
That\'s just what I found though, YMMV. :)
-
Yes, sending only positions won\'t work well, because you need the velocities (or even acceleration info) for client-side prediction of positions. But why not send both positions and velocities, unreliably ? If you send positions too, you don\'t need reliable&ordered delivery because now you don\'t need all of the packets.
It seems to me that you described very well the theoretical aspect of game networking but you don\'t follow this theory in your praxis :)
-
Hi,
another thanks for all your new answers! :)
I think I\'ve understood the usage of UDP packets and I\'ll think for my game it will be enough to not merge packets but send them all alone as RonHiler does.
But only one question: Do you really send absolute positions? Isn\'t this very insecure because everyone can manipulate his client to move much faster or implement a \"teleport cheat\"?
Isn\'t it better to send just acceleration und turning events and let the server move the player?
ChrisM
-
Originally posted by tangerine
Yes, sending only positions won\'t work well, because you need the velocities (or even acceleration info) for client-side prediction of positions. But why not send both positions and velocities, unreliably ? If you send positions too, you don\'t need reliable&ordered delivery because now you don\'t need all of the packets.
Well, understand, I\'m trying not to go over my entire movement packet methodology. It is by far the most complex server/client packet passing system in my game, heh. So you are only getting a little bit of the picture, there are actually 5 different packet types that can go back and forth just dealing with avatar movement, and each packet type carries 8-12 peices of data with it.
So to answer your question, I do send out absolute positions from the server, and if it turns out the time stamp on this position is later than a missing velocity change packet, the client ceases to request that packet, since it is not needed.
Yes, the velocity packets are reliable and ordered. But ordered only in the respect that they are re-ordered at the client end by timestamp, they can actually come out of order, and reliable only until they become obsolete, at which point they\'re quietly dropped :)
Which, actually now that I think about it, is a good answer to ChrisM\'s initial question of why UDP over TCP. TCP could never do that, while with UDP, since it\'s my own custom system that I can design on a per packet basis, I can do tricky things like that to make packets \"semi-reliable\" and \"semi-ordered\".
Originally posted by ChrisM
I think I\'ve understood the usage of UDP packets and I\'ll think for my game it will be enough to not merge packets but send them all alone as RonHiler does.
Whatever you find works best for you is what you ought to use. PS merges packets before they send them, and I don\'t. Either way has advantages and disadvantages. My packets have a tendancy to be pretty good size, because each one packs all the data that is needed to do a particular task. Perhaps PS sends one bit of data per packet instead (?? I don\'t know, I\'m just guessing), and therefore it makes sense for them to merge several small packets first.
Isn\'t it better to send just acceleration und turning events and let the server move the player?
Absolutely! Clients should never ever send out absolute positions. When we are talking about sending out absolute positions, we mean from the server to the players, not ever the other way around. My clients only send keystroke data to the server. The server handles calculating their positions and facings and velocities, and returns this to all the other players.
Never trust the client! :)
-
Hi,
Absolutely! Clients should never ever send out absolute positions. When we are talking about sending out absolute positions, we mean from the server to the players, not ever the other way around. My clients only send keystroke data to the server. The server handles calculating their positions and facings and velocities, and returns this to all the other players.
That\'s exactly what I expected and hoped to hear. :)
As far as I can see it in the PS source code, the client sends absolute coordinates but the position is first valided by the server meaning the server checks whether the client could get to this position since last known position in the elapsed time between the two packets.
I now started planning my network system und my packets are also relatively huge, except of course packets like special commands but the occur not very often and it wouldn\'t be profitable to cache them and wait for another packet to pack them together...
Hmm... PS has a very complex network engine. NetBase seems to be a lowlevel base class overloaded by client and server and MsgHandler a class also overloaded by client and server to distribute incoming packets to those iNetSubscriber-inherit classes who subscribed for these events.
But I don\'t unterstand what the NetThread class for the server is... :(
ChrisM
-
Originally posted by ChrisM
As far as I can see it in the PS source code, the client sends absolute coordinates but the position is first valided by the server meaning the server checks whether the client could get to this position since last known position in the elapsed time between the two packets.
Yeah, sanity check. That\'s another perfectly good way to do it. I don\'t do a sanity check because I only transmit keypress data, there is no way for clients to tell my server \"I am *here* now\", so there\'s no need for one. Different solutions to the same problem :)
I now started planning my network system und my packets are also relatively huge, except of course packets like special commands but the occur not very often and it wouldn\'t be profitable to cache them and wait for another packet to pack them together...
Don\'t get them too big. Your practical limit on UDP packets is about 1K. Beyond that and you risk routers splitting up the packets into chunks, which is a certain formula for many lost packets :) My packets tend to run in the 50-200 byte range, except for file transmition (patching) which max the packet sizes at 1K.
Hmm... PS has a very complex network engine.
As for all the rest of your post, you\'ll have to get info from the PS team (if they\'ll give it). I\'ve not looked at the PS code, nor do I intend to. I\'m sure it\'s very good code, don\'t get me wrong, but I\'m sure they wouldn\'t want me using their code concepts in my very non-open source codebase :) So I don\'t even give myself the temptation of looking at what they\'ve done by even downloading it, heh.
-
I haven\'t measured our average message size, but I would bet our average message size is around 25 bytes. Our dead reckoning system sends positions, velocities, angles and angular velocities and gets all that in 25-30 bytes.
Since many players are being updated each second, merging packets *really* helps network efficiency. By merging them, you only get per packet overhead on the merged version. So you get 10-20 bytes of overhead per 750 byte message, instead of 10-20 bytes of overhead per 20 byte message, which is 3% overhead instead of 50%.
In a smaller game, CPU utilization may be more of a factor but in an MMORPG, merging short packets is definitely worth it, and I really don\'t think it takes much CPU overhead anyway.
Just my $0.02,
Venge
-
Hi,
again thanks for your helpful answers!
I think, now it\'s time for me to start planning my UDP system, I\'ll start with a very simple std::vector or std::list for the clientlist and only one receive()-function with a short queue in its own thread.
But Planeshift source code is sometimes a bit irritating... for example I wonder why the overloaded network classes (NetBase & MsgHandler) for the server are in the server\'s directory, but those for the client are in the netlib?
So, I\'ll create the first source code line now...
Thanks for your help!!
ChrisM
-
Good luck, Chris! Let us know how it goes.
-
Hi,
argh, one question left:
Is there any limit under this there is no fragmentation or something? Or is it theoretically possible that a two byte packet is split up into two one byte packets? (yes, I know, this won\'t happen, but I am talking about the theory :) )
ChrisM
-
You prompted me to go look it up. To my surprise, it isn\'t 1K like I thought it was, it\'s 512 bytes.
Which means packets <=512 bytes won\'t fragment. Packets > 512 bytes *might* fragment (depending on the router hardware they pass through to get to their destination).
Note that fragmenting is not necessarily a disasterous thing. It just means there is more chance of losing a part of the packet in transit (in which case the entire packet will be discarded). You don\'t have to put the packet back together or anything. The UDP network protocol will do that for you. As I said, my patching packets are 1K, so they might very well be fragmenting without me having ever known it. And I probably won\'t change that. Even though I basically have twice the possibility of losing a packet, so what? Patch packets are reliable/ordered sent and it\'s not a time critical packet anyway (when players are patching they expect to have to wait a few minutes).
I better go back and make sure my play packets are under that limit though :) I could have sworn it was 1K rather than 0.5K, heh.
-
Hi,
wow, thanks for your cool post, where do you look up things like this? I tried to read in the MSDN for UDP, but there are not very much useful informations.
Did I understand the whole thing?
When a packet is fragmented by one router which routes the packet to its destination, the beginning and the end of the packet may get to the receiver and the middle is missing?
Could it for example be, that a 2048 byte large packet is fragmented and the two middle packets (assuming the packet is split in 512 byte large ones) are missing... will than recvfrom() return 1024 and return just the 512 byte large first packet and the 512 last byte?
ChrisM
-
Originally posted by ChrisM
where do you look up things like this? I tried to read in the MSDN for UDP, but there are not very much useful informations.
http://www.gamedev.net
Use it often. You can find *anything* there :)
Could it for example be, that a 2048 byte large packet is fragmented and the two middle packets (assuming the packet is split in 512 byte large ones) are missing... will than recvfrom() return 1024 and return just the 512 byte large first packet and the 512 last byte?
Not quite. UDP has no guarantee of delivery or ordering, but it does have one guarantee. That if you get a packet, it will be whole and in internal order with itself. Thus, if a packet fragments, the UDP layers of network protocol will put it back together on the other end before you ever see it.
What this means in terms of missing fragments though is that if one fragment gets lost, the *entire* packet is discarded. Thus, if you fragment into 10 segments, and you get 9 of them, it is the same as if you lost the entire packet. This is why large packets will get lost much more often than small ones.
Thus, to answer your example, recvfrom() will never give you a completion notification (or whatever is appropriate for the particular network model you are using, for MM games, you *should* be using completion notifications) for that particular packet, since you lost two of the four fragments. It is the same as if it never arrived at all.
-
The ones in the common directory are not in the client directory because they are shared by the client and the npc superclient.
-
Hi,
ok, now everything makes sense. :)
Thanks for your answer. I can\'t believe that just before a week I really wanted to use TCP for my network system. :)
Now I just hope, that the \"invalid\" packets (> 512 bytes) in my buffer, won\'t jam it but be deleted after a few seconds.
So, now it\'s really time to start implementing my network interface! ^^
Thanks again for all your answers!
ChrisM
-
In PS, the networking layer splits apart messages bigger than the allowed size into multiple packets and reassembles them on the receipt side.
Starting to understand why there is so much code and complexity in PS networking? ;-)
- Venge
-
Hi,
Starting to understand why there is so much code and complexity in PS networking? ;-)
Yes. :)
But why do you separate MsgHandler and ClientMsgHandler since there are no other MsgHandlers? However, I think it\'s for future expansion (to perhaps implement a ServerMsgHandler), isn\'t it? :)
And why do you store everything in those binary trees, even if a linked list would make much more sense, in my opinion (for example the list of the guilds).
ChrisM
-
MsgHandler is designed to work for any number of connections. ClientMsgHandler works for only 1 connection, so it is much easier/simpler.
Binary Trees are used because they are faster to search than linked lists. Thus they have O(log n) sorted insert time and O(log n) retrieval.
-
Hi,
great, I understood it! :) (I first didn\'t notice that you overload == and comparision operator for all classes that are stored in a tree, so I wondered how you sorted the objects in the tree *g*)
OK, just one last question (I promise it\'s the last ;) )...
Why do you use so many different Message structures? I mean, all those messages derived from MessageCracker are obviously, but what\'s about the other message classes?
I mean, what\'s the different between psMessage and psNetMessage and between MessageEntry and psNetMessageEntry?
ChrisM
-
Hi,
ok, I think I got it now.
netpacket.h is for a part of a message, meaning one UDP packet sent over the network.
message.h is for one merged packet, normally consisting of one netpacket (but can be more if message size > 512 and this packet was automatically splitted by the PS client on the source computer).
Does it work like this?
ChrisM
-
Yes.
-
Hi,
OK, thx for all your answers!
ChrisM