Server Lag : what it is and isn't....

[Lambchops]

So the server is lagging a bit at the moment. Sounds like a good time to talk about general lag issues.

Here's what I know about the way WC2 and battle.net worked 15 years ago... we're now using a PvPGN server and an AH server also, so things could be different now. If I'm wrong about any of this stuff, please reply and add your info.... but AFAIK it's like this:

First there's 2 types of network traffic you need to know about:
(don't worry it's really not that technical...)


   UDP: User Datagram Protocol
                  and
   TCP: Transmission Control Protocol


The difference is this:
- TCP makes a connection between 2 computers, in this case a player's computer and the server. Once a TCP connection has been made, the TCP protocol guarantees that every message either of them sends to the other is successful. It checks, and if there's a problem, the message is sent again, until it goes through, or eventually the connection is abandoned, whereupon both ends will be notified that the connection has failed.

On the other hand..
- UDP does not do this at all. If one computer sends a UDP packet to another, it has no idea if the other computer receives it or not. Some program running on the other computer can send back a message confirming the packet arrived, if it wants, but that is not part of the UDP protocol (whereas it is built into TCP).

There's also another 2 terms I'm going to use a bit which are:


   IPX: Internetwork Packet Exchange
               and
   IP: Internet Protocol


- IPX is just an old network protocol that is no longer used. Although internally different, as far as the descriptions above go, IPX can be considered the same as UDP. It was popular in the late 80's and early 90's but by '96 it was on the way out and was not widely supported after that.

- IP is the basis of the internet. It's that base layer that is sending packets from one "IP Address" to another. You have probably heard the term "TCP/IP" before, well this is the "IP" part. By this terminology the UDP traffic could be called "UDP/IP" because the "IP" protocol is operating beneath both of them. Often these days people say "IP" when referring to an IP Address, however here I'm talking about the protocol itself. When I'm talking about an IP address I say "address".


So... going back to 1996 and Warcraft II:Tides of Darkness when the internet was still in it's infancy, the guys at Blizzard used a network protocol they were familiar with to implement the pre-battle.net multiplayer games. That was IPX. For those of you old enough to remember Kali this is where that fits in. Kali is an IPX emulator, so it allowed people to play multiplayer games based on the old IPX protocol on that new wonder of the modern age "The Internet!".

But the point is this: the networking side of the game engine has remained pretty much unchanged from 1996. When Blizzard released Warcraft II:Battle.net Edition in 1999, they included all sorts of cool new stuff related to battle.net: server channels, messaging, friend lists etc.., and of course the most important thing - the ability to make a game or join a game. All these battle.net features worked over a TCP connection. This connection is established when you logon to battle.net and (being TCP) it maintains itself, and guarantees the integrity of all messages passed through it.... for example unless the connection fails and one of you is actually dis-connected from the server, a /m to another player NEVER fails, even with bad lag it will still go through eventually.

If you make a game, your client (wc2 program) will talk to the server over this TCP connection and tell it the name of your game and other details. If click the JOIN button the server will talk to your client over this TCP connection and give it a list of games. This list includes the game name, the pud, the settings, and most importantly the IP address of the host.

It is at THIS point in the process that battle.net ceases to be part of the equation. When you join a game, the internals, for the most part are still exactly the same as they were in 1996, the only difference is that instead of using the old IPX protocol, which was not supported by the internet, BNE substituted the UDP protocol, which was supported. This, however is just wrapping your sandwich in cling-wrap instead of tin-foil, its still a BLT inside. When you join a game (or try to!) you start sending peer-to-peer UPD packets the host's client, which then sends you a list of addresses for the other clients already in the game, and sends your address to them. To successfully join your client needs to make UPD "connections" to all of the other clients in the game.

I say "connections" in quotes, because of course UDP (or IPX) does not make connections it just blindly sends packets and hopes they arrive. Your client (the wc2 program) IS, of course keeping track of what packets arrive, and if one connection is losing packets you will see it on the lag screen, or as we know if YOU are the person with the bad connection you will often see several or all of the other players listed on the lag screen, because from your client's point of view packets are missing from lots of other clients.

So this is how WC2 games work over a network, with all of the clients all firing UDP packets at each other. So what about the server? Well you trusty TCP connection to the server remains in place, but this connection does not have anything to do with the game. It can't cause game lag. The only thing that can cause game lag is problems with the UDP traffic between the clients.

The BNE client knows to send any chat that starts with a "/" to the server over this TCP connection and the server will then process that message or reply with a WTF? and your client will say "Unknown Command". If the server is lagging it can take 5 or even 10 seconds to get a response. Can you imagine if it took 5 or 10 seconds for every update in the game? There's just no way.... that would be beyond bad lag. Generally speaking for a game not to have annoying lag, the clients needs to update each other every 1 second or so, ideally much less. That means every client needs to be able to send and receive a UPD packet to/from every other client no later than once a second.

BUT, the clients do not, as part of the game send any UPD packets to or receive any UDP packets from the server. The only time this happens is when you first login to the server, it does a check to see if you can send/receive UDP packets over port 6112. The reason this will usually succeed even though you may still be unable to host is to do with the way your networking infrastructure (modem/router/network card/drivers etc.) handles traffic. In short, once the TCP connection is established with the server, the underlying IP "Internet Protocol" automatically sends any traffic from the server to the client at the other end of the connection, regardless of if it is TCP, UDP or whatever.

This is how the "lat trick" works if you haven’t got your game port properly routed. You have a TCP connection to the server, which gives you the IP address of the host. So your client sends a UDP packet to the host saying "I want to join", but at this point you there hasn't been any communication whatsoever between the two clients. Any channel chat or /m messaging is relayed through the server, so if the game host hasn't got the game port "opened" (properly routed to the client) then when a random UDP packet turns up from some random client, the IP has no idea what to do with it. But if you alternately make games, then try to join each others' games, you can train the IP running in your hardware to associate traffic over that port with a certain IP address.

But if you can't get UDP packets through to the server when you log in it will give you a warning message saying you won't be able to join games. It will, however still let you log in and chat/message/see the game list etc... everything that happens over the TCP connection. If you join like this you will have the grey "bot" icon which has an unplugged power cord on it.

So back to joining a game. Why can you join a game even if you haven’t got your game port properly routed? Why is it just when you host? This is all IP layer stuff, like the "lat trick" above. The problem is with that very first "I want to join" message that your client sends to the game host. If the host's game port isn't routed to the correct end-point then this message turns up from an unknown address on an un-defined port, and the IP routing (quite rightly - or you would be vulnerable to all sorts of attack) discards the packet. However, if the host has already said "send me all the packets on port 6112 (or whatever port) then the packet is forwarded to the host's computer, where the wc2 program already has a socket open listening for UDP traffic on port 6112.

Once that first message has gone through, the host gives the client a list of addresses for the other clients in the game, and also gives all those other clients the address of the new client joining, and everyone starts sending UDP packets at everyone else.... and once you have sent an outbound packet over given port to a given address, the IP accepts that your client machine is exchanging data over that port with that external address, so it thereafter routes all incoming traffic to that client.

But, friends and fiends, no amount of server "lag" will make your game lag.... nope, sorry... and here's the proof: what happens when the server crashes during a game? You get a message saying "connection to battle.net lost". This is the TCP connection timing out. As we know TCP connections are "guaranteed" by the TCP protocol. It is internally sending back and forth packets to keep the connection alive and verify that any communication has reached its destination, until your client fails to receive any response from the server for a pre-defined amount of time, when your TCP drivers give up and notify the client that the connection has been broken.

But when this happens does the game end? No, because the server isn't part of the game, once a game has started you could blow up the server with an RPG and it wouldn't stop the game. It stops any messaging etc., any command starting with a "/" but the game continues. When the game ends, even if the server is back up and running you don't go back to the channel, you are back in the pre-connection part of WC2, because the wc2 client wasn't written to automatically attempt to re-establish a TCP connection with the server (this is inherently unsafe as it would have to be storing and re-transmitting your password without your knowledge).

So if there is server lag, it can be slowing down/lagging all the stuff that involves the server: logging in, chatting, messaging, your friend list, getting the game list etc., but it doesn't lag games. Perhaps in theory if there is server lag the TCP service is generating a little bit more network traffic saying "hello? can you hear me?" a bit more often, but if your network is operating on such a tiny amount of bandwidth that it would even notice this then I would expect that you would already be having all sorts of problems.

So what causes lag? Lag,(or lack of it) is a function of the peer-to-peer UDP times... or more to the point the worst peer-to-peer route between any two players in the game. There are 2 factors in "worst". Obviously speed is one, but even worse is actual packet loss, where a packet is sent from one place and and just never turns up at its destination. A crackly phone line can do this - or a recalcitrant server (not the game server - just some random internet server). This sort of thing is what causes "conflicts" between two players, that otherwise tend not to have problems. This is because *somewhere* in-between those two end-points is a server that is not doing the right thing with UDP traffic... or perhaps with all traffic to/from certain locations ... or maybe it's just plain overloaded.

So while we're on the subject, what's with "/ping"?
The "/ping" or "/p" command is not a battle.net command, it's one of the extra features built into PvPGN (that's the software that runs the war2.ru server - it's a battle.net emulator). "Ping" is a basic networking feature which just sends a packet from one computer to another which asks the remote computer to send it back. The "ping" time is the time it takes for the message to go to the remote location, get processed, and return to where it came from. Usually this is repeated a number of times and an average time is reported.

As we know, when you first login to the server, your /ping will be reported as "0". It appears that checking client ping times is either averaged out over number of checks over a reasonably long period of time, or (more likely) it is a task that the server deems to be low priority and schedules it to be done some time when it is twiddling it's thumbs with nothing better to do. So you wait a few minutes and the server reports a time in milliseconds. Try this for fun: on windows go start-->run then type in "cmd"[ENTER]. This will bring up a DOS style command prompt. Then type "ping google.com"[ENTER]. You can do the same thing from a UNIX machine.

The ping reported by the "/ping" command is the round-trip time from server.war2.ru to your client and back again. It does not necessarily have anything at all to do with game lag, although an excessively high number here can be indicative of a network issue. But as we know, lag is related to peer-to-peer UDP times, connection time to the server has nothing to do with it... although if you have a really slow connection you will also have a high server ping time, but a high server ping time does not necessarily mean you have a slow connection.

Another interesting thing here is that right now, when the server is having some genuine lag issues it is reporting my /ping as 247. This is extremely low for me (being in AUS). Normally my /ping is reported at 500+. It's just conjecture, but it could be the result of the server's host provider throttling TCP traffic. As pinging is a "control message", operating on the ICMP protocol - which is usually prioritised by servers, if the connection is being throttled at the TCP layer, it could leave other IP layer traffic with extra bandwidth, hence faster ping times... or maybe they just switched to a host in AUS?...nah lol.

(((EDIT: more likely by the time I wrote this much, iL had already fixed the server issues so it was all nice and speedy, but at the time I assumed it was still having issues, as I was writing not playing)))

So what's with the green/yellow/red bars on the game list and next to each player when you join a game? These are ping times, but are very different ping times. The bars in the game list is the ping time between the host and the server. When you join a game and are waiting - before it starts, as we know, at this stage it's all about peer-to-peer UDP traffic, so those bars are the ping time between you and that other player. If you see one player go red and the rest stay green, its probably their problem. If you see all players suddenly go red, kick your GF off facebook  


IS ANY OF THIS WRONG?....or does some part of it just seem wrong?.... post a reply and tell me what you think!

....and for the extra bonus points... WHO WAS PAYING ATTENTION? - What small change could have been made to the battle.net protocol to eliminate the whole "opening ports" to host a game issue? ? ? I'll post the answer to that later when everyone has just said "FFS, doesn't that nerd ever shut up?" and completely ignored this post.

[Sentinel1]

What is the logic behind "conflicts?"  There's a game up and either Player A or Player B can join the game and it doesn't lag, but if both are in, it does.

What is the logic behind the lat trick?  Two people who do not have port forwarding set up can still host a 1v1 with each other.  Player A hosts, Player B tries to join but cant.  Then Player B hosts and as long as Player B tried to join A's game, A will be able to join B's game.

Hopefully you know, always was curious about these two things.

[Lambchops]

What is the logic behind the lat trick?  Two people who do not have port forwarding set up can still host a 1v1 with each other.

The info is all there .. but it's a bit scattered (like me;) so...

With regard to lat trick:

once you have sent an outbound packet over given port to a given address, the IP accepts that your client machine is exchanging data over that port with that external address, so it thereafter routes all incoming traffic to that client.

When you try to join a game, the server supplies the host's IP Address and port, then your client tries to send a UDP datagram to the host but...

The problem is with that very first "I want to join" message that your client sends to the game host. If the host's game port isn't routed to the correct end-point then this message turns up from an unknown address on an un-defined port, and the IP routing (quite rightly - or you would be vulnerable to all sorts of attack) discards the packet. However, if the host has already said "send me all the packets on port 6112 (or whatever port) then the packet is forwarded to the host's computer

So player A is hosting and player B tries to join.  At this point you have player B has sent an outgoing UDP datagram to A, but A's routing just says "WTF is a 6112??" bugger that.

Then the positions are reversed so then player A is sending outgoing UDP on port 6112 to player B, and B has already sent one the same to A so at this point player B's IP routing could say "ok then i'll send that to (player B's computer's local IP address) because we have already sent one of those out to that other address".  It may take a few goes back and forth depending on the actual logic in your router's firmware.

The thing is that routers are not designed to just blindly forward everything that the outside world throws at them to your computer... you can tell them to if you want, this is called DMZ ("de-millitirized zone"), but you still have to tell the router which computer is the DMZ, because, and this is the other point, they are also not designed to have only one computer attached to them... but really you just want to tell it to send you port 6112 traffic, not ALL trafic. The underpinning logic of the internet, and therefore the internet protocol is that any router can get a packet intended for any address and send it in the right direction to get there... and the next router (server... whatever) will do the same, until the packet eventually gets where its going.

When you use your browser to pull a web address, your browser initiates a TCP connection with the remote server (over port 80 for plain http). Because this connection started at your computer and went out, and the connection is maintained by the TCP protocol, your router automatically sends back the server's reply over this already established connection that your computer started.

When somebody tries to join a game you have hosted, they send a UDP packet to your external IP Address, there could be any number of devices connected to your router, or being something that originated externally, it could be malicious, it is an unknown, and without specific instructions, like "send all udp traffic on port 6112 to local IP address 192.168.0.4" (this is forwarding the port) it will just discard the packet.

With the "lat trick" you are manually forcing each client to send UDP traffic to each other over a certian port and simulating a "transaction" like a TCP connection, which "tricks" or "trains" (depending on how you look at it) the IP layer into forwarding the traffic to the desired endpoint.

Hope this was a bit more understandable.... I'll do a seperate post on conflict

[iL]

BTW: about current problem with lagging: if i understood properly, the problem was related to long /commands handling. When you type /w or /con or /p or w/e.
That was not related to any network problems. The problem happened with database connection. That is about that php extension for pvpgn. I just removed that database connections from handlers now and that delay is gone.

P.S: Lambchops, nice to see such detailed and interesting posts about network connections, i think that will be useful to community. Would be interesting to read your post about conflict also.

[Player]

Good post.  Actually read the whole thing because it was interesting.

[Player]

BTW: about current problem with lagging: if i understood properly, the problem was related to long /commands handling. When you type /w or /con or /p or w/e.
That was not related to any network problems. The problem happened with database connection. That is about that php extension for pvpgn. I just removed that database connections from handlers now and that delay is gone.

FYI - I see this type of server lag all the time.  For example, /f l, right now it's instant.  Most days, it's like a 2-3 second delay before the list shows up.

[Lambchops]

What is the logic behind "conflicts?"  There's a game up and either Player A or Player B can join the game and it doesn't lag, but if both are in, it does.

Lag,(or lack of it) is a function of the peer-to-peer UDP times... or more to the point the worst peer-to-peer route between any two players in the game

The server is not part of the game. All clients talk directly to all other clients. If you have a 2v2 game:  A and B vs. C and D.  then A and B talk to each other.... but A and C also talk to each other .... and A<-->D and B<-->C and B<-->D and C<--->D ..... (did I leave anyone out?) .... so they ALL talk directly to each other.

OK so then we factor in what I was saying about routing in the lat trick post above:

The underpinning logic of the internet, and therefore the internet protocol is that any router can get a packet intended for any address and send it in the right direction to get there... and the next router (server... whatever) will do the same, until the packet eventually gets where its going.

The routing tables in most home routers are pretty simple, they mainly just say "anything that isnt a local address send HERE" with "local address" being defined as whatever your local network is (often 192.168.X.Y or 10.0.0.X combined with a sub-net mask ) and "HERE" being the gateway address supplied by your ISP, which is whatever the IP address is of their server that they are selling you access to....

When you get to your ISP's server it has its own routing table which may or may not just forward everything to a central node, but at some point there will be a bunch of rules like "everything from 0.x.x.x to 127.x.x.x send to server A" and "everything from 128.x.x.x to 255.x.x.x send to server B" ... this is a simple example, there will most likely be a heap more routing rules than that. And those servers will have their own routing tables which will define, for them, the quickest way to send all packets based on their IP addresses... these can also change dynamically depending on what time of day it is and how much load various other servers are under. And of course these servers will have rules telling them to send packets with YOUR ip address back to your ISP's server, which then forwards them to you.

Everybody forwards everybody's traffic... and everybody sends it via whatever route is defined in their routing tables .... in theory this is the "fastest route from A to B" but there may be a variety of other factors at play, i.e. social/political/monetary... who knows? All you know is that you send the info to your ISP, it picks another server to forward it to, which in turn forwards it to another ... etc., until it reaches the destination.... You can check this out on one of the "Ip tools" type websites that does a "traceroute".... this will show you the list of servers that form the link between you and another given server.

SO... back to our 2x2 game... so say, for example, player A is in AUS, player B is in GBR, player C is in USA and player D is in UKR.... perhaps player A can talk to B,C and D fine, same for player B, in fact everyone can talk to everyone else, EXCEPT... somewhere between player C and player D, one of the servers chooses to route the traffic through some shit-box old crappy server in Iceland because they don't pay as much tax that way (or WTF ever). It may even be in one direction only i.e. D can send traffic to C very fast, but when C sends traffic to D there's a problem... (you remember that "ping" times are round trip times so a problem in either direction will show up) ... maybe the guy programming the routers in Iceland was married to a Ukrainian and she ran off with the guy next door, so he said,"screw you and yours" and downgraded all traffic to UKR to the lowest possible priority.... who knows....

Of course these things aren't as simple as that example... it's more likely to do with rival companies downgrading each others' traffic... or a combination of all sorts of things... but the point remains that what causes lag is the speed and reliability of round trip UDP data between all players, and the actual route that the data goes through from player A->B is different from A->C, in fact it may even be different from B->A and all it takes is one crappy or overloaded server to be the "weakest link in the chain" and cause a crappy connection. When it happens that by virtue of whatever happens to be written in the routing tables of a bunch of servers wherever in the world, that one or more of these weak links ends up in the route that data is sent from one player to another, those two players will "conflict".

It may be only between two specific players, but from what I have seen, certian players seem to have problematic connections and will tend to conflict with a few people. One way to fix this *COULD* be to change the route by using a fast proxy or VPN. A VPN establishes a connection over HTTPS between the client and the VPN provider, from where the data would most likely be routed through a different set of servers to the other players. Many servers give priority to this type of traffic so it could even end up being a faster connection, ..... or bloody aweful lol, its a matter of experimenting to find what works

[tk[as]]

good post. im stoned right now but i'll re-read again and i'm sure it'll be even better

[Lambchops]

BTW: about current problem with lagging: if i understood properly, the problem was related to long /commands handling. When you type /w or /con or /p or w/e.
That was not related to any network problems. The problem happened with database connection. That is about that php extension for pvpgn. I just removed that database connections from handlers now and that delay is gone.

Nice one
....and thanks for donating your time and expetise to maintaining the server for us. I for one appreciate it.... I'm sure you have a job and a family and pleanty of other demands on your time.

Good post.  Actually read the whole thing because it was interesting.

P.S: Lambchops, nice to see such detailed and interesting posts about network connections, i think that will be useful to community. Would be interesting to read your post about conflict also.

good post. im stoned right now but i'll re-read again and i'm sure it'll be even better

Thanks, for the feedback guys (and your questions Sentinel1), I'm glad you found it useful.

 I've been meaning to write something about this for a while...  just to clear up a few misconceptions about a bloody awesome server that is provided for free by people who do it for no more than the love of the game.

Love your work beautiful people 

[Lambchops]

You can check this out on one of the "Ip tools" type websites that does a "traceroute".... this will show you the list of servers that form the link between you and another given server.

One thing that should be mentioned here is that this will show you the route that the servers process "traceroute" packets through, but this is not necessarily the route that they send TCP port:80 traffic through, let alone UDP port:6112 traffic. Maybe it is.... maybe it isn't. But I can give you a very good example of this kind of "selective" routing, being that if I log on to an "ISP speed testing" type website I will get absolutely phenominal speeds and its will tell me that I have a fully awesome connection, when I know very well that in reality my connection is hard limited at about 20% of that speed downstream, and a whole lot less again upstream.

I can see only two possible explainations for this:

1) The people who designed all the major speed testing sites can't do simple arithmetic.
2) My ISP knows excatly what the addresses of all the major speed testing sites are, and gives maximum priority and bandwidth to all speed testing packets being sent to and from these addresses....

I'll let you decide which is more likely

[Lambchops]

....and for the extra bonus points... WHO WAS PAYING ATTENTION? - What small change could have been made to the battle.net protocol to eliminate the whole "opening ports" to host a game issue?

         Awww c'mon….  Anybody? Pls tell me somebody is picking up what I'm putting down….

             ( peeps with a pre-existing obligation not to discuss the internals are excused )

[tk[as]]

I'm at work now.. gonna re-read when home and sober and take a stable at it

[Lambchops]

I'm at work now.. gonna re-read when home and sober and take a stable at it

go hard mr khan

[tk[as]]

as im reading ... would it be possible to use the lat trick for multiple players? i've only ever tried in 1v1 .. But theoretically if you have 4 players, each one of them unable to host/join eachother's game, they could all do the lat trick individually to eachother, and as long as all of the players attempted to send/receive UDP packets individually to each player, you should be able to eventually get all 4 of them in the same game, no?

Also when you're talking about /ping .. you're talking about our ping in relation to the server.war2.ru.. what about /ping (specific player)   .. the number shot back at me is the time it takes in milliseconds for our computers to communicate with eachother, correct?  so, a player with a ping of 3500 vs 350 is going to make a noticable difference in game.


so for the bonus question .. small change to protocol to eliminate need to open ports. and im just going to take a huge shot in the dark because even after reading all of it, im by no means an expert in any of it

the only things i can come up with is somehow form a TCP connection (even if only temporarily) with all of the players on the server so you auto-send/recieve UDP traffic to those players after that   .. or somehow, when you first log onto the server and successfully send/receive UDP packets to the server, have the server forward your UDP packets to all other players on the server so there is a "connection" made between you and all of the other players as soon as you log on... but im not sure if the 2nd would be possible since the forwarded UDP packets would be coming from the server IP address, and not your own.

[tk[as]]

there definitely needs to be a connection made some how between the two players as soon as you log on.. either TCP or if you could have the server forward UDP packets from one player to another and visa versa so both players recognized eachother. those are my guesses.