Author Topic: Network improvement  (Read 1446 times)

Kerol

  • Hydlaa Notable
  • *
  • Posts: 574
  • Assets > Asshats
    • View Profile
Network improvement
« on: July 26, 2006, 07:22:32 pm »
After the CS conference I thought about the structure of PS (thinking of the more or less detailed presentation by Talad) with the different layers and having a seperated psserver and psnpcclient. The fact that psserver needs only a tiny bit of ressources and that npcclient doesn't have to be on the main machine and also isn't limited on one npcclient makes the network the actual bottleneck.
The question wasn't answered how much network traffic PS causes, but I think it can't be little.

A few thoughts and possible solutions:

1. In the past I already suggested to rebuild the chatsystem by making it a client-to-client system (for instance with a modified jabber client built into PS), thus halving the latency for chat communication and also reducing lost chat packages caused by too heavy traffic on the line to the server.
The biggest problem there is the possibility to circumvent chatcontrol by GMs/devs and possible faking of chat messages (if /report sends the incoming chat to the server, for instance). This could be solved by signing the chatmessages and validating.
As I guess that (my favoured) solution will probably not be implemented, I suggest an alternative.
A QoS (quality of service) heading in message-type packages. As far as I know (I'm no network engineer) this can be implemented pretty easy with an IPv6 compliant client.

2. For the far future when different levels in yliakum will be implemented, I suggest to have a server for each level seperately. For "world-wide" communication, the message could be send from one server as broadcast to the other servers, that broadcast to the players.
This system actually could also be implemented for huge areas with a lot of NPCs and players. Communication that requires the servers to communicate inbetween will be slower, but the network load on one server will be lowered a lot.

3. One thing that could help now already is to have the client load all and evey char related data from the server on startup (everything that would require a server-query on event like opening a window like guildwindow, stats/skills) and make the server send updates only on changes of those data (only the diff). That would eliminate the latency when opening a window and minimize the need for server queries during runtime for those events. The initial loading of this non-essential data to the client could be done with a low priority in the background while the client already is up and running, thus also not requiring a longer startup for the client.
This could also be done with /who. Loading the list when the client fires up and after that, only update the clientside if anything changes, and not requiring the client to query the server. This would also eliminate the spam effect when repeating /who a dozen times while keeping the client always up to date.
Entering /who would show the list that already is on the client side, and would has no effect on the server.

A good implementation would work here with transactions and maybe with periodic checksum checks.

4. Make the client calculate space in which the graphics would actually not show an entity and send back that data to the server. If there are entities in that space, don't send location updates for those entities to the client. I would say to have this space-calculation and exclution of location updates only on space which is very far away (which won't be shown by the client's graphics at all for a long while) and when the player has not left a defined cube for a longer while (for instance when moving only minimal, in circles or standing about). The cube would be updated on the client every 5 sec and check if the player has left it. If not, do the calculation of excluded spaces and send it to the server. When the player moves out of the cube, usual location updates resume.
Actually this method would only need a little bit more CPU by the client, for the cube and the calculation of excluded space. Effort on the server to check if entities are in that space is minimal, the reduced network load shouldn't be underestimated, though. I always thought it stupid that I got lag by looking in a direction where many people must be standing, but I have not seen anyone. This method would minimize that lag.


retired GM leader

DaveG

  • Forum Addict
  • *
  • Posts: 2058
    • View Profile
Re: Network improvement
« Reply #1 on: July 26, 2006, 09:46:32 pm »
1)  This is not what the wish list is for.  Moved to dev stuff forum.
2)  Take an actual look at the client's bandwidth usage sometime.  It's nothing; under 1KB/s.  The server is the only thing we need to worry about, and it's really in fine shape.
3)  As I already tried to explain to you, outsourcing our chat is not only stupid but wouldn't help anything.  It'd hurt us just because it's a whole extra pile to worry about.
4)  Network lag is primarily due to having the server 1/3 second ping away in Singapore.
5)  For the multi-server setup, doing servers per region is probably better.  That way everyone has the lowest latency possible.
6)  The windows are being updated (some have been already) to send deltas.  That's the one thing here that's really an issue, and we've been working on it.  Sending everything at login, however, is a waste.
7)  You can't cache the /who list.  It's always changing, and since it's not used that often it'd be a waste to send updates in the background.  Also, it's tiny and doesn't affect squat.
8)  There's really no way for the client to tell what it can and cannot "see" at the moment, and it's probably not worth the extra calculations.  Again, the client is fine; it's the server that we might have to care about, and it's fine.

If you want some real optimizations, take a look at what I did to the DR for this release.  I lobbed off about 25% from our total bandwidth usage, in addition to other optimizations.  I have another method we may eventually use which could do that again.   (no, I'm not exaggerating)

You seem to be interested in this, so have some numbers from Laanx:
Quote
=================
Bandwidth profile
=================
Total time: 462 seconds
Total byte: 2108604
count=759   perc=19.0 byte=401491 avg-byte=528.974 max=1315.000 Name=MSGTYPE_PERSIST_ACTOR-sent
count=12108 perc=15.9 byte=335825 avg-byte=27.736 max=71.000 Name=MSGTYPE_DEAD_RECKONING-recv
count=43    perc=13.3 byte=279843 avg-byte=6507.977 max=32765.000 Name=MSGTYPE_GUIGUILD-sent
count=105   perc=11.7 byte=245823 avg-byte=2341.171 max=2883.000 Name=MSGTYPE_GUIINVENTORY-sent
count=871   perc=8.4 byte=177698 avg-byte=204.016 max=633.000 Name=MSGTYPE_NPCOMMANDLIST-recv
count=6499  perc=8.3 byte=175153 avg-byte=26.951 max=40.000 Name=MSGTYPE_DEAD_RECKONING-sent
count=5076  perc=5.1 byte=108428 avg-byte=21.361 max=49.000 Name=MSGTYPE_STATDRUPDATE-sent
count=159   perc=4.5 byte=94518 avg-byte=594.453 max=782.000 Name=MSGTYPE_ALLENTITYPOS-sent
count=17    perc=4.0 byte=83385 avg-byte=4905.000 max=4905.000 Name=MSGTYPE_MSGSTRINGS-sent
count=1376  perc=1.3 byte=26524 avg-byte=19.276 max=22.000 Name=MSGTYPE_COMBATEVENT-sent
count=81    perc=1.2 byte=25715 avg-byte=317.469 max=1340.000 Name=MSGTYPE_GUISKILL-sent
count=17    perc=0.9 byte=18488 avg-byte=1087.529 max=2015.000 Name=MSGTYPE_AUTHAPPROVED-sent
count=60    perc=0.8 byte=17890 avg-byte=298.167 max=1824.000 Name=MSGTYPE_GUIMERCHANT-sent
count=377   perc=0.8 byte=16618 avg-byte=44.080 max=687.000 Name=MSGTYPE_SYSTEM-sent
count=1     perc=0.7 byte=15499 avg-byte=15499.000 max=15499.000 Name=MSGTYPE_CHAR_CREATE_TRAITS-sent
count=14    perc=0.4 byte=8330 avg-byte=595.000 max=595.000 Name=MSGTYPE_MOVEINFO-sent
count=87    perc=0.4 byte=7507 avg-byte=86.287 max=1358.000 Name=MSGTYPE_PETITION-sent
count=159   perc=0.3 byte=7158 avg-byte=45.019 max=304.000 Name=MSGTYPE_CHAT-sent
count=300   perc=0.3 byte=6865 avg-byte=22.883 max=79.000 Name=MSGTYPE_NPCOMMANDLIST-sent
count=64    perc=0.3 byte=6059 avg-byte=94.672 max=114.000 Name=MSGTYPE_PERSIST_ITEM-sent
count=27    perc=0.3 byte=5364 avg-byte=198.667 max=261.000 Name=MSGTYPE_MOTD-sent
count=17    perc=0.2 byte=4608 avg-byte=271.059 max=695.000 Name=MSGTYPE_CHARACTERDETAILS-sent
count=10    perc=0.2 byte=4261 avg-byte=426.100 max=473.000 Name=MSGTYPE_GLPYH_REQUEST-sent
count=298   perc=0.2 byte=3652 avg-byte=12.255 max=66.000 Name=MSGTYPE_USERACTION-recv
count=18    perc=0.2 byte=3599 avg-byte=199.944 max=207.000 Name=MSGTYPE_MAPACTION-recv
count=722   perc=0.1 byte=2888 avg-byte=4.000 max=4.000 Name=MSGTYPE_REMOVE_OBJECT-sent
count=14    perc=0.1 byte=2799 avg-byte=199.929 max=207.000 Name=MSGTYPE_MAPACTION-sent
count=14    perc=0.1 byte=2510 avg-byte=179.286 max=276.000 Name=MSGTYPE_LOOT-sent
count=80    perc=0.1 byte=1991 avg-byte=24.887 max=178.000 Name=MSGTYPE_CHAT-recv
count=30    perc=0.1 byte=1681 avg-byte=56.033 max=69.000 Name=MSGTYPE_AUTHENTICATE-recv
count=43    perc=0.1 byte=1664 avg-byte=38.698 max=81.000 Name=MSGTYPE_GUIMERCHANT-recv
count=50    perc=0.1 byte=1400 avg-byte=28.000 max=28.000 Name=MSGTYPE_SLOT_MOVEMENT-recv
count=265   perc=0.1 byte=1325 avg-byte=5.000 max=5.000 Name=MSGTYPE_PING-recv
count=56    perc=0.1 byte=1087 avg-byte=19.411 max=43.000 Name=MSGTYPE_GUISKILL-recv
count=1     perc=0.1 byte=1058 avg-byte=1058.000 max=1058.000 Name=MSGTYPE_ADMIN-sent
Note that MSGTYPE_NPCOMMANDLIST, MSGTYPE_ALLENTITYPOS, and MSGTYPE_NPCOMMANDLIST are server-NPCclient communications and don't actually get sent anywhere but localhost.

From those numbers, Laanx was only going 4.5 KB/s there.  (not including headers and such)  This data shows some of the relative sizes of messages, but I probably should get one from running a bit longer to show more.  (this was just the most recent one I had saved, and it's actually a bit old)

No matter how you cut it, MSGTYPE_DEAD_RECKONING (movement messages) are the VAST majority of all networking traffic by volume and #2 by size.  (was more than twice what it is there, last version)  The info for actors is the current top, but it's only sent on entering an area with them.  (probably not much we can really save there)  The crap for the guild and inventory windows are the other tops, and it is a problem we already know of.  (the skill window was worse than what you see there, but Bereror allready killed that)  MSGTYPE_MSGSTRINGS, a one time hit on login, was 3 times what you see there, but I now compress that so it's much smaller.  From there down, the rest is nothing.

Oh, and if you want the relevent changes to the DR I did, they're here and here.  Basically, I'm storing an extra byte which has flags indicating what's packed inside.  This way, I don't send 4 bytes for a float when the number is just zero.  (which at any one instant, most are)
« Last Edit: July 26, 2006, 10:25:00 pm by DaveG »

::  PlaneShift Team Programmer  ::

Wired_Crawler

  • Hydlaa Citizen
  • *
  • Posts: 429
    • View Profile
Re: Network improvement
« Reply #2 on: July 27, 2006, 04:22:50 am »
I'm curious - if it is not secret - is Laanx server being used also for other purposes ? Like, for example, web server for other companies, ftp server... Some time ago (before reinstallation or upgrade?) you could see network statistics, which were showing constant average output around 400 kB/s with peaks exceeding 1.5 MB/s... What causes (or was causing) it, if server uses only 4.5 kB/s ? Or is it 4.5 kB/s per client ;) ? Traffic coming TO server was around 10 times smaller.
"Close the world, txEn eht nepO."

Kerol

  • Hydlaa Notable
  • *
  • Posts: 574
  • Assets > Asshats
    • View Profile
Re: Network improvement
« Reply #3 on: July 27, 2006, 07:16:15 am »
Quote
Take an actual look at the client's bandwidth usage sometime.  It's nothing; under 1KB/s.  The server is the only thing we need to worry about, and it's really in fine shape.
All the suggestions were proposed with the intention to lower the load on the server, not the client.

Quote
As I already tried to explain to you, outsourcing our chat is not only stupid but wouldn't help anything.
Well. "Outsourcing" isn't the word I would use for the system I had in mind. It's just jabber-like, not an actual jabber client, although one could take over code from jabber.
But as I said. That solution probably won't be implemented, which is why I proposed the usage of QoS for that instead.

Quote
Sending everything at login, however, is a waste.
I think it only seems like a waste. You need to take into consideration that the average user (who doesn't only RP, but also fights, trades and is in a guild) uses the inventory, the guildwindow, chat and stats/skills window at least once per session. There are many people who query /who at least once per session (at startup to see if there's something going on) and GMs use the petitionmanager and GMconsole more than once per session.
So you need to query the server anyway for those things. If that is done on startup or during the session isn't really a difference for the server. If it is done on startup however, the user doesn't notice the lag when opening a window  and also it doesn't require to query the server again and again, which was the big problem with /who, what also was the reason to cap it to 30 people (which is pretty stupid imo), to reduce the load on the server caused by /who spamming.
If it is loaded on startup and updated by the server when necessary (and only sending the diffs, not the whole stuff) there's no way to spam the server nor the user gets lag when querying the window data the first time as they are there already. Plus the load is reduced due to minimised package size because of only sending the diffs.

Quote
There's really no way for the client to tell what it can and cannot "see" at the moment.
I disagree. You can't see what is behind a wall or a small entity in great distance or entities hidden by fog or rain. The client knows what he _can_ see. Given an infinite space minus the space that can be seen is the space which can't be seen :)
An example. you have a candle as lightsource and a circle as obstacle. The shadow produced by the circle has a sort of infinite cone as volume.
Now turn it around. Take the camera or better your avatar as ray-source and calculate the "shadows" cast by walls and other obstacles and return the volumes you get as space that can't be seen. Reduce the spaces to very simple (but with infinite expansion) geometrical forms such as cones or cubes and return those to the server. The server only needs to check if entities are in those spaces which requires O(n) time. If an entity is in such a space, don't send the location updates to the client that couldn't see the entity.
The clue of this is that the server can circumvent that mechanism for entities that _were_ hidden, but because of whatever now can be seen. So it doesn't matter really what the client "thinks" it can't see, but there's big chance that this spares a lot of useless traffic.


retired GM leader

DaveG

  • Forum Addict
  • *
  • Posts: 2058
    • View Profile
Re: Network improvement
« Reply #4 on: July 27, 2006, 10:18:59 am »
Quote
Take an actual look at the client's bandwidth usage sometime.  It's nothing; under 1KB/s.  The server is the only thing we need to worry about, and it's really in fine shape.
All the suggestions were proposed with the intention to lower the load on the server, not the client.
Yes, I know.  I was just being thourough.

Quote
As I already tried to explain to you, outsourcing our chat is not only stupid but wouldn't help anything.
Well. "Outsourcing" isn't the word I would use for the system I had in mind. It's just jabber-like, not an actual jabber client, although one could take over code from jabber.
But as I said. That solution probably won't be implemented, which is why I proposed the usage of QoS for that instead.
Simply put, our chat system will always be part of our system.

Quote
Sending everything at login, however, is a waste.
I think it only seems like a waste. You need to take into consideration that the average user (who doesn't only RP, but also fights, trades and is in a guild) uses the inventory, the guildwindow, chat and stats/skills window at least once per session. There are many people who query /who at least once per session (at startup to see if there's something going on) and GMs use the petitionmanager and GMconsole more than once per session.
So you need to query the server anyway for those things. If that is done on startup or during the session isn't really a difference for the server. If it is done on startup however, the user doesn't notice the lag when opening a window  and also it doesn't require to query the server again and again, which was the big problem with /who, what also was the reason to cap it to 30 people (which is pretty stupid imo), to reduce the load on the server caused by /who spamming.
If it is loaded on startup and updated by the server when necessary (and only sending the diffs, not the whole stuff) there's no way to spam the server nor the user gets lag when querying the window data the first time as they are there already. Plus the load is reduced due to minimised package size because of only sending the diffs.
It might be usefull in some situations, but I really never noticed much of a delay.  The main problem is the plain text waste the systems have, and the fact that only the skills have been switched over to diffs.

Quote
There's really no way for the client to tell what it can and cannot "see" at the moment.
I disagree. You can't see what is behind a wall or a small entity in great distance or entities hidden by fog or rain. The client knows what he _can_ see. Given an infinite space minus the space that can be seen is the space which can't be seen :)
An example. you have a candle as lightsource and a circle as obstacle. The shadow produced by the circle has a sort of infinite cone as volume.
Now turn it around. Take the camera or better your avatar as ray-source and calculate the "shadows" cast by walls and other obstacles and return the volumes you get as space that can't be seen. Reduce the spaces to very simple (but with infinite expansion) geometrical forms such as cones or cubes and return those to the server. The server only needs to check if entities are in those spaces which requires O(n) time. If an entity is in such a space, don't send the location updates to the client that couldn't see the entity.
The clue of this is that the server can circumvent that mechanism for entities that _were_ hidden, but because of whatever now can be seen. So it doesn't matter really what the client "thinks" it can't see, but there's big chance that this spares a lot of useless traffic.
Um... you don't quite understand here.  Sure, everything is possible, but there is no method currently.  CS is working on a fancier culling system, which we theoretically might be able to tap for things like this.  If we were to create a hitbeams thing to calculate everything it could see, it'd murder things.  To use your lighting example, you need to know how it actually works.  All our lighting is done at once and cached.  The average client just shows the lightmaps, but does no actual lighting itself.  It takes about a 6 or 8 hours (or whatever it's up to now; depends on the comp; dfryer said it took him 8.5 hours just for Bronze Doors) to relight all the maps.
« Last Edit: July 27, 2006, 10:35:38 am by DaveG »

::  PlaneShift Team Programmer  ::

Kerol

  • Hydlaa Notable
  • *
  • Posts: 574
  • Assets > Asshats
    • View Profile
Re: Network improvement
« Reply #5 on: July 30, 2006, 05:29:43 pm »
Quote
Simply put, our chat system will always be part of our system.
I don't get what you want to express with that. I never suggested to make the chatsystem independant from psserver, did I?  ::|

Quote
To use your lighting example, you need to know how it actually works.
Well.. Again my "dangerous half-proficient knowledge" coming from the CS devs and the conference.. :)
They are working on shading systems and if I understood it correctly, one is working similiar to what I described. I don't know a lot about it, so I can't give you details, but that's how far I got it.

If it is too CPU intensive to calculate realtime hitbeams (I use your term although I don't know if that is really the correct word) for all items and walls, I can think of two tricks how to reduce that load.
First is to place entities with a relatively big expansion at locations that can be expected to be non-visible for a lot of people, like rooms, or areas behind walls and such. The entity itself is invisible for the player but included in the maps themselves. As an example such an entity is placed in a room, filling it nearly completely, leaving little space at the entrances, as the server needs to react quickly when someone walks out. The client now calculates whether it can "see" the entity or not. If it cannot "see" (i put it in parenthesis as the entity is only visible for the program, not the player) it sends to the server "entity 'exampleroom' is not visible for me". Now the server checks if other entities are actually in the space of entity 'exampleroom' or not. If there are other entities in that space, the client can't see them neither, so the movement data won't be sent to the client.
If any spell or other effect makes one or all entities in that room visible (for the player or relevant for proximity), although they are included in entity 'exampleroom', the server may send the relevant data, although the client doesn't expect any, but that doesn't should produce any problems.

The second idea I have to reduce the CPU load for the proposed system is to place a 3D-grid of "lighters", kind of lightsources but invisible (again). The client would need to check if it can receive light from a lighter (the positions and lightintensity is known to the client of course, they are included in the maps again). If light cannot be received by a defined lighter, because of rain, walls or anything, the client gives back this info to the server. The size and intensity of the lighters probably need to be extrapolated by the lighting maps. The clue is that, if light cannot be received from a defined lighter, it is impossible to see something behind a lighter that can't be seen, if it is of the same or smalller size and/or has a lower light intensity.

I think the first idea is best in cities and dungeons with many edges while the second could be used best in open areas.

That system will minimize the unnecessary packages sent about movement, leaving the problem with the number of necessary packages.
How do network-FPS handle the problem about movement packages?

Oh, an idean just popped up:
I try to explain:
Lag is a big problem in fights of all kinds, PvP and PvC, because some people have shorter ping, thus nearly no latency after hitting a movement button, while others have a long ping, and often don't see the movement of the opponent till it is too late. With mobs it's the same. Trying to evade mobs that actually aren't at the position anymore are really disturbing.
So the idea is to synchronize all movements (of all entities, included players), at the cost of ping. If that works as I imagine, everyone would get lag after hitting a movement key, no matter how much the actual ping time to the server is. The clue is that everyone has about the same latency, included PvC fights, thus no misjudgement of situations due to lag anymore. I think to have a fixed, relatively big overall lag for movements is a good price for fairer/less frustrating fights, more independant on ping times.

How I imagine it could work:
The server checks the ping to the clients, calculates the average ( := averageping).
Now the server adds a synthetic latency before sending movement packages to clients that have a better ping than averageping.
For mob-player synchronisation I think a time synchronisation NTP-like (don't think of outsourcing, think of just taking the code from NTPd and building it into psclient/server) is best. On startup, the client gets sync'ed with the server. The server adds timestamps to the outgoing movement packages and the client can reconstruct the movements.
That's the main principle of the whole idea. Make the clients reconstruct the events, but synchronised on everyone's PC, in logical order.
There's no way to cheat client-sided with this, as it has no effect to change the timestamps for (from the servers side) incoming movement packages, as the client doesn't know the "future" moves from the opponent. Actually I don't think it's of much use to implement timestamps for incoming movement packages at all, at least not for this idea. The unfair advantage given by a different geographical location thus lower latency is minimised this way, also the problem with the mobs (the server can make the mob react on movement of the player while the player has no data yet) should be solved with that.
« Last Edit: July 30, 2006, 05:31:16 pm by Kerol »


retired GM leader

DaveG

  • Forum Addict
  • *
  • Posts: 2058
    • View Profile
Re: Network improvement
« Reply #6 on: July 30, 2006, 07:37:05 pm »
1)  You need to make smaller posts...  You could've said all that in half the text.  ;)
2)  Yes, you suggested switching to Jobber and other stuff.  That'd be someone else's system for our chat.
3)  What I think you're suggesting is only sending DR for actors in sectors you can see, which we already do.  I'm honestly not sure what you're talking about there.
4)  I mean no offense, but you simply don't know enough of what's going on here to actually come up with something that would make any meaningfull difference.  Theoretically, based on concept alone, some of these sorts of things might lower network traffic, but in no real amount that would affect anything, and would simply not be worth the effort.
5)  Lag is based on having our server in a god awful location combined with message processing inefficiencies.  There is no dire need to lower bandwidth usage here.  The only real issue is the waste of non-cached window data, which as I said was already known.
6)  Artificially giving players lag is possibly the most ridiculous idea I've ever heard...  One of the many examples I could give:  Player with a bad connection interacts with a group, all others match the weakest link, everyone has a crappy connection, people start disconnecting because they can't stand the thing.
7)  As I already said, I have a real bandwidth optimization planned for the DR.  I'd estimate 10% reduction based on current numbers.  (which are after the 25% reduction from my previous optimizations)

::  PlaneShift Team Programmer  ::

Kerol

  • Hydlaa Notable
  • *
  • Posts: 574
  • Assets > Asshats
    • View Profile
Re: Network improvement
« Reply #7 on: July 31, 2006, 11:07:03 am »
Quote
1)  You need to make smaller posts...  You could've said all that in half the text.
I tried that before and noone got what I said :P
Quote
2)  Yes, you suggested switching to Jobber and other stuff.  That'd be someone else's system for our chat.
From my point of view "switching" and "including foreign code" are two different things.
Quote
3)  What I think you're suggesting is only sending DR for actors in sectors you can see, which we already do.  I'm honestly not sure what you're talking about there.
If you already do that, why do I get horrible lag by looking in directions where I can't actually see anyone (for instance to the plaza while standing behind a house or looking in the direction of the tavern) but I know that there are a lot of people? And that only happens when there actually are a lot of people, so it must be the movements by those, for me in that moment, invisible people.
Quote
4)  I mean no offense, but you simply don't know enough of what's going on here to actually come up with something that would make any meaningfull difference.  Theoretically, based on concept alone, some of these sorts of things might lower network traffic, but in no real amount that would affect anything, and would simply not be worth the effort.
That's why I'm proposing stuff like that ;)
Take the ideas and get inspired or leave it and just ignore it :)
Quote
6)  Artificially giving players lag is possibly the most ridiculous idea I've ever heard...  One of the many examples I could give:  Player with a bad connection interacts with a group, all others match the weakest link, everyone has a crappy connection, people start disconnecting because they can't stand the thing.
Erm.. I think you missed the point.
a) Not the worst connection but the average is the thing to make everything adapt to.
b) I only suggest "artificial lag" for movement and combat, nothing else.
c) In a perfect world, there is no lag, everything gets transmitted instantly. But in RL we have lag and the disturbing thing about lag is not the latency itself but the varying amount of lag, the unpredictability.
That's one problem. The other is that the server knows everything and because npcclient and psserver virtually have no lag in communication, mobs can react on movement of the player while the player has no idea what the mob already did. This is an unfair advantage for the mobs, but also for players with good ping times.
Well. The best solution from all is to remove lag completely, of course. But even with better pings the asynchron movement of entities stay a problem, probably not as much as now, but still.
I'm not sure, but there might also be other applications in PS for synchronised timing and timestamps. I think it's important to emphasize the "having everything in sync" part. How sync'ing is done is another question then; "artificial lag" might be an answer for that.


retired GM leader

DaveG

  • Forum Addict
  • *
  • Posts: 2058
    • View Profile
Re: Network improvement
« Reply #8 on: July 31, 2006, 12:41:21 pm »
WRT #3, sectors are big and you can see far.  Example: If I'm standing in the plaza, I can see into the rest of that whole area, but there's no way to tell when a building obscures my view, or even if I can see into the tavern sector.  Sure, you're not getting sent DR for people in the East or Arena, but yeah, you'll get ones from people around the West half of the city.  No real way around that, and it's not really a big deal most of the time.  DR is more efficient now, and again I say, I plan to make it much more so.

You also have to remember that there are 2 completely seperate sources of lag:
1) Delays in getting the data.  (this could be from network stuffs as you're talking about, OR from delays in the server handling messages)
2) Inefficiencies in rendering on your client.  CS' culling stuff (basically exactly what you're talking about, but with respect to rendering stuff it can see in the engine) is not exactly perfect, and they are currently working on a new system which we expect to give some notable performance inproovements.

WRT artificial lag:  Not the slightest way you could say it's good.  I've never really had too much of an issue, myself.  Are there really that many people who can't keep up with mobs due to lag?

::  PlaneShift Team Programmer  ::

Induane

  • Veteran
  • *
  • Posts: 1287
  • What should I put here?
    • View Profile
    • Vaalnor Inc.
Re: Network improvement
« Reply #9 on: August 01, 2006, 12:11:49 am »
If there are plans on making the npcclient and main server seperated already then why not simply do the same thing with chat.  Here is my though:

Use irc essentially.  Irc allows for creating rooms on the spot, etc...  it also allows you to create rooms where there is heirarchy - op's, etc.  My thought is to have the chat portion of the client connect to an irc server, either laanx or another one on another fragnetics server (not outsourced). Creating a new guild would automatically create a (in irc terms) room in the name of the guild with underscores between spaces in the name. i.e. the room  community_of_vaalnor.  Groups would also be created dynamically as rooms.  So we group together, the group code stays the same minus the chat portion that is there now.  The act of creating a group also creates a new channel.  The client could send keys to the irc server (or built in password hashes or whatever) so that only the planeshift client can use this irc server.  The chat window would simply be a portion of the gui that acts as an enhanced irc client.  As for interface, there should still be the same buttons along the top as normal, except there should be one for "alliance" as well.  In the case that you are in more than one group or more than one alliance clicking on the alliance or group button pops a few more buttons up allowing you to choose what alliance or group your chat goes to.  You could also specify a command like     /alliance alliancename  blahblahblah   or /group groupname blahblah blah

Is this feasible, stupid etc??  We really need alliance chat.


Kerol

  • Hydlaa Notable
  • *
  • Posts: 574
  • Assets > Asshats
    • View Profile
Re: Network improvement
« Reply #10 on: August 01, 2006, 06:06:52 am »
Quote
If there are plans on making the npcclient and main server seperated already then why not simply do the same thing with chat.
I know off first hand (Talad) that the npcclient and psserver _are_ seperated already. NPC's are handled on the server just as players.
In future, npcclient will be run on multiple servers independantly, each instance taking over a couple (or maybe only one, for an extra complex NPC) to distribute the load of calculating the AI.

Quote
Use irc essentially.
IRC is one of the most bugged and abused chat platforms out there. Just look at how simple it is to make a chatbot and bring down the server by spaming. There are also problems with security on that matter. I don't know how secure psserver is in respect of attacks from outside (a hacker trying to get in the chat system without login) but that needs to be taken into account.
If we debate how the chatsystem should work and whether to take code from another project (with which I don't see any problems with, appearantly not like DaveG, for whatever reasons), then I propose, again, the jabber protocoll, or SILC. With the jabber core protocol XMPP one has (amongst others) UTF-8 support, IRC-chatroom like conferences, very long messages are possible without the need to cut into pieces, messages sent to an offline person will be received on next log on, SSL encryption for client-to-server (also end-to-end is possible, but thats not relevant here), Gateways might be interesting in the future as well.
It has many more possibilities like VoIP compatibility, mulitple servers, P2P like filetransfer and so forth, but there DaveG will hit me with the hammer again :P
I think the possibilities given above, together with a very active development and nearly bug-free code as an international standard are a few good arguments for the jabber protocol.

Quote
Is this feasible, stupid etc??  We really need alliance chat.
Well.. I love the idea.. but I'm just a crazy person giving weird ideas to chew on..  :sorcerer:

Quote
there's no way to tell when a building obscures my view, or even if I can see into the tavern sector.
Why is it impossible to check if an object is displayed or not?

Quote
CS' culling stuff (basically exactly what you're talking about, but with respect to rendering stuff it can see in the engine) is not exactly perfect, and they are currently working on a new system which we expect to give some notable performance inproovements.
I was talking about culling for whole objects , yes. But not as much as culling graphically but also eliminating the unnecessary network traffic resulting from movement of hidding objects.

Quote
I've never really had too much of an issue, myself.  Are there really that many people who can't keep up with mobs due to lag?
Everyone who fought mobs in group with me had troubles with that at some point, yes. The lonely spawn campers wont have big troubles with that as they generally only take on foes that can be killed in a few hits and don't move at all. They don't fight the big mobs that actually move around. I think that's one of the major reasons why group-hunting isn't very popular atm.
The other big problem is lag in PvP fights. One says that tournament fights are based 50% on skill and 50% on luck.. and lag.
Not the latency itself is the major problem there, but having one person (or mob respectivly) being able to react faster than the opponent.
I don't think people will complain about a little higher latency if everything runs smooth and there are no disadvantages in fights just by lag/chance anymore.

PS: Sorry Dave for the long post again :)
PPS: What does WRT mean?


retired GM leader

DaveG

  • Forum Addict
  • *
  • Posts: 2058
    • View Profile
Re: Network improvement
« Reply #11 on: August 01, 2006, 04:24:20 pm »
Sorry,
WRT == With Respect To

Simply put, we have our own networking methods and we will always be sending chat over that.  You might think that there even is such a thing as simply "switching" to something else, but there is not.  What you really want is to put the chat on some other server to avoid lag, and not only do we not have another server but it can't be just seperated like that.  You also have to realize that no matter what, the chat needs to go through the main server for all sorts of thing ranging from simple flood prevention to NPC communication.  Also, we will not have chat available to out-off-game methods.  That would be an OOC nightmare, and be quite polluting to the game.  You are thinking there is something you can just "do" and there isn't.

To try to explain the problem of knowing if you can see something:  First of all, it would have to all be done on the server.  To know if it can see something, it first has to know where it is, thus defeating the purpose if it's already sent.  So, the server has to be doing all this, which means sending out millions of hitbeams.  It'd need a few from each actor to each other actor in the sector.  (not to mention that the server would then need all the models loaded up)  So, maybe 1 or 2 per actor and 6 or something actors per viewing actor, and then we've got to be doing these a few times a second, so we're talking a good 5000 of these going.  (vague crappy estimation)  Not only is the server CPU use shooting up, but we're delaying things, and the bandwidth drop from not relaying some DR messages would be fairly meaningless.  Frankly, it would add lag because now the server has to analyze if it wants to relay DR to people instead of just doing it.

Please realize, we have no bandwidth problem here.  Lag and bandwidth usage can be related but are not the same thing.  There are a few stupid bandwidth things that need to be fixed, and eventually when we get multi-hundred users at once the next DR stuff will be nice.  However, to remove delays there are probably some inefficiencies in the server in how it processes messages that need looking into.  That, and not having the server in a god awful place.  Seriously, a 333 ms ping is horrible.  Just the fact that you can play PS at all under these kind of network conditions is a miracle.  PS' network layer is one of the few that can actually handle this, and still work rather well.

::  PlaneShift Team Programmer  ::

Kerol

  • Hydlaa Notable
  • *
  • Posts: 574
  • Assets > Asshats
    • View Profile
Re: Network improvement
« Reply #12 on: August 01, 2006, 05:52:54 pm »
Quote
What you really want is to put the chat on some other server to avoid lag, and not only do we not have another server but it can't be just seperated like that.
Nope. I don't know about Induane, but I haven't suggested (in this thread) to use a seperate chat server. I just proposed to take jabber code and build it in with the hope to get UTF-8 support, offline messages and all that.
I see the problems with seperated chat servers from the main server, although there'd be solutions for those. I too think now that seperating the chat from the main server probably is more effort than use but nevertheless the chat system as it is now needs a whole lot of work and I simply think that taking code from another open project, an international standard, about bug-free and thoroughly tested, is just the easiest way to get the desired functionality without coding everything from scratch.

Quote
To try to explain the problem of knowing if you can see something:  First of all, it would have to all be done on the server.
Again I think you misunderstood what I tried to explain. Last try:
You have server S and client C that player P is using.
C has the maps stored locally, like they are now. C only knows only the own position in the world and the 3D stuff.
S knows the positions of all entities in the world. S needs to update C about the changing positions of the entities.
That is how it is now.
Now assume that in the 3D maps not only the graphical stuff is stored but also one sort of entity: simple polygons, stationary, not affecting anything else, invisible for P.
Those polygons have a pretty big expansion (could be a room, but also be a lot bigger).
As those polygons are built in the maps, C and S know where they are, the positions and dimensions.
On runtime, C calculates whether it can see polygon X (an arbitrarily picked one).
[now it's easier to explain the situation for the case that C can't see X, but in a possible implementation it should be for the case that C can see X as that minimises the package number required]
If not, the client sends a "i can't see polygon X" to the server.
The server checks [everyone of those polygons could be represented by an array and entities that enter such a polygon get added into the array, so its a simple inclusion check] whether the space of polygon X includes other entities. If there are, the server refuses to send DR regarding those entities to C.

The reason why this has not to be calculated on the server, but can be done on the client is simple: if someone would hack that system and would try to circumvent it, they only would hurt themself. There's no use in preventing the client from sending those "i cant see bla.." packages or giving false information regarding non-visible entities to the server.
The information coming from the server only gets more or less, but won't be altered by itself. If one would be so "intelligent" to disable the system, one would only get the lots of unnecessary load, nothing else. It can't prevent from getting damaged or change positions of the client, as the info on the server never gets changed.

The system also won't increase the load. In the worst case it just will be useless, nothing more.
I think one can call the idea "dynamically culled mini-regions for reducing unnecessary DR".


retired GM leader

DaveG

  • Forum Addict
  • *
  • Posts: 2058
    • View Profile
Re: Network improvement
« Reply #13 on: August 02, 2006, 12:38:43 am »
I think what you're suggesting is the client calculate what it can and cannot see, and send that to the server to decide what DR to send out for what it can and can not see.  This would be a big waste, as sending that data would be more than just donig the DR.

If I missunderstood you post (and I probably did), you don't need to explain it further.  Please understand, this has nothing to do with something usefull here.  Sure, if we could hook into some culling thing on the server, it might be nice to not send unneeded DR in case of a large group of actors moving around in an area.  However, as I have already said, it's not a big deal by any means, at this point.  There are other things that do much more than this theoretically could, and even then, we're not exactly using loads of bandwidth.

::  PlaneShift Team Programmer  ::

Wired_Crawler

  • Hydlaa Citizen
  • *
  • Posts: 429
    • View Profile
Re: Network improvement
« Reply #14 on: August 02, 2006, 02:26:25 am »
I think the discussion reaches its end, so let me add something.

Quote from: Kerol
1. In the past I already suggested to rebuild the chatsystem by making it a client-to-client system (for instance with a modified jabber client built into PS), thus halving the latency for chat communication and also reducing lost chat packages caused by too heavy traffic on the line to the server.

Jabber is NOT client-to-client system, it uses normal client-server communication and additionally server-to-server transport. The difference from other IM systems consists in fact, that everybody can install his own server (check, for example, Wikipedia if You don't trust me). So basically You want build jabber server into PS client. To have simulated client-to-client communication every PS client would have to provide jabber SERVER services, then every player would need to login to everyone else's jabber server... It's hopeless. Will You explain to every new player, how to configure firewall to enable in-game communication ;D ? Not to mention, that it would add security risk on player's computers, PS devs would need to write p2p code VERY thoroughly.

Quote from: Kerol
I just proposed to take jabber code and build it in with the hope to get UTF-8 support, offline messages and all that.

There isn't anything like "jabber code". Jabber is just popular name of implementation of XMPP protocol. There are many implementations of XMPP. I don't think there is available OpenSource version written in C++, easy to include in PlaneShift code. So, PS devs would need to implement XMPP in PS themselves, which would be a lot of (unneeded) work. It is better to just improve existing chat system.

About "visibility" culling - I thought I didn't understand You Kerol, but it looks like I did :P. What a strange idea of reducing network traffic by adding some more network traffic... (reducing lag by adding lag is also... erm... interesting ;)). Basically You want to divide maps to more, smaller sectors, nothing more. Handling of those sectors would be additional load to server, DaveG already explained it. Example of existing solution: stand near windowless tower. You can't see through its walls, and interior of tower is in other sector - you don't receive information about players inside tower. I mean, it is like it SHOULD work, but I'm not entirely sure if it does  :-\ - do you remember "hearing" players killing monsters in sewer while You were on plaza? Was it fixed?

Edit: Ah, and about Your idea of synchronizing messages between all clients and server - You would like to implement TCP+, I mean TCP, which assures that all packets come to application in order + additional features. Look, there IS a reason, for which all network games use unreliable UDP. It is exactly for the purpose You want to eliminate - lost and delayed packets from one client do not affect server and other players.
« Last Edit: August 02, 2006, 02:34:22 am by Wired_Crawler »
"Close the world, txEn eht nepO."