16 Oct, 2008, Lobotomy wrote in the 1st comment:
Votes: 0
I don't recall who said it before (DavidHaley, perhaps?), but I recall being advised previously that it would be preferrable for a codebase if the socket code were largely detached from the core game logic. As I'm working on recoding the network code in my codebase, that suggestion has been on my mind. I've recently redone the DNS lookup code that comes with SocketMUD, so I feel that I have a pretty fair grasp of how to use threads now as well as their purpose (i.e, having the DNS run in a separate thread keeps the main process from being locked up while the address is resolved). The thing is, though, I'm not certain what aspects of the socket code that whoever it was that brought it up were being referred to. I suspect that it is the read()/write() calls which could either be done in a single thread for both or one thread for each, but I also suspect that there may be more areas which could benefit from it.

So, I'd basicly like to hear some insight or clarification on the matter, as well as any other thoughts or ideas for ways that the socket code could be improved; along with some sort of confirmation as to whether I'm on the right track or not.

Also as a quick side note, I do have another question. I notice that at current the code uses write() to spit out text in chunks that are at most 4k in size. Is there any particular reason for it being done at that rate that anyone is aware of? Should it be decreased or increased at all; maybe even set to just send everything there is to send for that socket all at once? I'd expect it to be fine to just have it send everything all at once anyways (since it will do that regardless?), but I'm guessing there may be reasons otherwise that I am not aware of (protocol limit(s), maybe?).

:thinking:
16 Oct, 2008, quixadhal wrote in the 2nd comment:
Votes: 0
The 4K chunk size comes from the most common disk/memory page size, and is also a convenient packet size to send out. Even though you're using tcp (a stream protocol), it's still more efficient if you can avoid crossing packet boundaries due to the unpredictable nature of lag.

As for splitting all socket I/O off to threads, that's a perfectly good thing to do. Just be sure you use semaphore locking when you're moving data in or out of buffers. You don't want someone's strcat() to "lose" data because when it did the initial scan for the NULL terminator, it was at byte 117, then your output thread sent the data and stamped a null at byte 0, then your strcat got control again and wrote the data… which is now hidden because anyone looking at the buffer will see it as an empty string.
16 Oct, 2008, Lobotomy wrote in the 3rd comment:
Votes: 0
quixadhal said:
The 4K chunk size comes from the most common disk/memory page size, and is also a convenient packet size to send out. Even though you're using tcp (a stream protocol), it's still more efficient if you can avoid crossing packet boundaries due to the unpredictable nature of lag.

I see.

quixadhal said:
Just be sure you use semaphore locking when you're moving data in or out of buffers. You don't want someone's strcat() to "lose" data because when it did the initial scan for the NULL terminator, it was at byte 117, then your output thread sent the data and stamped a null at byte 0, then your strcat got control again and wrote the data… which is now hidden because anyone looking at the buffer will see it as an empty string.

Yeah, I pretty much figured out the jist of that from observing how the thread interaction works while fixing the DNS lookup code. The primary flaw in trying to use the existing design that I can see (i.e, buffers on the socket itself) with threads is that it may cause messages to be sent and received out of order with what is going on in the main process. Such as if a socket in one part of the list is buffered a message to be sent, then another socket further on down the list is buffered a message to be sent, it may be possible for the later socket to have their message sent before the first one is; depending on where the sending thread is at in polling the socket list.

To that end, I'm thinking that I'd change things to use a single send queue of sorts so that each message will be sent in chronological order (by being added to the queue in that order). Ultimately it might still not be received in chronological order, due to lag or othersuch, but I'd like to try and keep the order as much as possible.
16 Oct, 2008, Vassi wrote in the 4th comment:
Votes: 0
quixadhal said:
The 4K chunk size comes from the most common disk/memory page size, and is also a convenient packet size to send out. Even though you're using tcp (a stream protocol), it's still more efficient if you can avoid crossing packet boundaries due to the unpredictable nature of lag.

As for splitting all socket I/O off to threads, that's a perfectly good thing to do. Just be sure you use semaphore locking when you're moving data in or out of buffers. You don't want someone's strcat() to "lose" data because when it did the initial scan for the NULL terminator, it was at byte 117, then your output thread sent the data and stamped a null at byte 0, then your strcat got control again and wrote the data… which is now hidden because anyone looking at the buffer will see it as an empty string.



Splitting I\O into threads scares the crap out of me for a MUD. It's something I've done on a regular basis with other, small, apps and it works extremely well when each connection is its own client that is serviced completely independently of any others. In the case of a MUD, though, where everything is so interdependent it has never seemed like a good idea. You could do sockets asynchronously and then 'synchronize' them through a single message pump but that's probably way more work than it's worth.

That said, having a locking\thread-safe event queue is an excellent way to allow "some" things to be run concurrently, so you can still synchronize them with the game via the queue. For instance, saving a player or looking up a flatfile can be done in a separate thread and then the actual player feedback is 'synchronized' by logging it as an event for the main thread to consume at its leisure.

Isolating your I\O is an excellent idea, what you're looking to separate is your read and write calls. Ideally you'd want your socket code to only care about pumping out bytes and reading in bytes. I think David said he had more or less the same approach as I have, which is bytes come in -> bytes are filtered through a 'protocol' layer and converted from a byte array into actual text, telnet codes are filtered out, etc. Then the actual command is sent to a third layer which actually interfaces with the game.

Likewise, and possibly even more importantly, you want to absolutely get away from having ANY game code call directly on your socket write. It'd be better to filter that through a sanitizing\formatting function or object - even if it's just to send a dinky little line. This lets you handle colors and other out-bound protocols more elegantly. I.E. instead of displaying your room like this in a look command:

player.send("[ZoneName]\r\n");
player.socket.write("This is a very cool room description\r\n");
player.socket.write("also here is a squirrel.");

//you'd want something like this to format your output.
player.report.room(roomStruct);


Needless to say, this kind of abstraction works best in an OO model (i.e. C++ et all.) - but is not impossible via procedural languages (C)

Edit: I also don't want to imply that this is the "best" way, but I consider it a "better" way, considering how snippet happy C\C++ codebases are. Such a modular approach makes it easier for you to change your mind (cuz that never happens) about features and implementations, it also makes prototyping new ideas and concepts very easy.
17 Oct, 2008, quixadhal wrote in the 5th comment:
Votes: 0
Actually, that's how I usually do string handling in "real" languages where strings aren't just obnoxious arrays of characters. :)

$foo = "Silly Room\n";
$foo .= " This is a " . $room.light_level > 50 ? "bright " : $room.light_level < 10 ? "dark " : "" . " room.\n";
$foo .= "You wish this room looked more interesting.\n";
send($ch, $foo);


Yeah, that's pseudo-perl.
17 Oct, 2008, Lobotomy wrote in the 6th comment:
Votes: 0
quixadhal said:
As for splitting all socket I/O off to threads, that's a perfectly good thing to do.

So, are there any particular performance and/or responsiveness benefits for a mud to handle socket I/O in seperate thread(s) as opposed to within the main process, or is it something that would be overkill and/or is it just otherwise roughly the same either way? I'm curious if there's a tangible benefit to that route, or if I'd just be wasting my time with it. Unlike DNS lookups which can hang for up to seconds or more at a time, I'm not aware if read/write calls can cause particular performance problems for a mud process that necessitate separate threads.

:thinking:
17 Oct, 2008, Caius wrote in the 7th comment:
Votes: 0
I think the benefits of threading would probably be offset by the added complexity. If the codebase had been designed with threading in mind from the beginning it would be one thing. But adding threading as an afterthought is generally not a good idea.

If you want to go with multi-threading anyway, I recommend getting a good book on the subject. I haven't seen many online tutorials that cuts it. Multi-threading has many subtleties that may blow up in your face much later if you're not exercising caution and programming discipline. And if you don't have time to learn threading thoroughly, then don't use it. Really, I'm serious.
17 Oct, 2008, Lobotomy wrote in the 8th comment:
Votes: 0
Caius said:
I think the benefits of threading would probably be offset by the added complexity. If the codebase had been designed with threading in mind from the beginning it would be one thing. But adding threading as an afterthought is generally not a good idea.

If you want to go with multi-threading anyway, I recommend getting a good book on the subject. I haven't seen many online tutorials that cuts it. Multi-threading has many subtleties that may blow up in your face much later if you're not exercising caution and programming discipline. And if you don't have time to learn threading thoroughly, then don't use it. Really, I'm serious.

I appreciate the information, however that is not at all what I'm asking about here. My only concern at this point is whether or not the performance increase, if any, justifies the use of one or more separate threads for the socket I/O. Should I need help understanding how to use or implement threads I'll ask for advice on the matter at that time.
17 Oct, 2008, Caius wrote in the 9th comment:
Votes: 0
Lobotomy said:
I appreciate the information, however that is not at all what I'm asking about here. My only concern at this point is whether or not the performance increase, if any, justifies the use of one or more separate threads for the socket I/O. Should I need help understanding how to use or implement threads I'll ask for advice on the matter at that time.

There's no simple yes/no answer to that question. Say you use two threads. One main tread that runs the game logic and one thread that maintains the network. If the network thread is slowed down by a read or write operation, then yes, the main thread will happily churn on. However in this scenario you'll still have the issue that you only read/write one socket at a time, so the players may not necessarily notice better responsiveness. Then you have the option of the network thread spawning a handful of worker threads. In this scenario player A on a speedy connection may have instant responsiveness even though player B is on a super-slow connection, as long as they're serviced by two different threads. This scenario adds complexity because you need to write code that manages and synchronizes the worker threads. Furthermore it will mean that player B may be disadvantaged during PvP with player A. When using threads this way you have no control over in which order events happens.

I agree with Vassi that threading makes perfect sense when you need to service multiple connections, but they don't interact with eachother. Webservers, ftp servers and the like are typical examples. But for a mud you may very well trade one headache for another.
17 Oct, 2008, Lobotomy wrote in the 10th comment:
Votes: 0
Caius said:
There's no simple yes/no answer to that question. Say you use two threads. One main tread that runs the game logic and one thread that maintains the network. If the network thread is slowed down by a read or write operation, then yes, the main thread will happily churn on. However in this scenario you'll still have the issue that you only read/write one socket at a time, so the players may not necessarily notice better responsiveness. Then you have the option of the network thread spawning a handful of worker threads. In this scenario player A on a speedy connection may have instant responsiveness even though player B is on a super-slow connection, as long as they're serviced by two different threads. This scenario adds complexity because you need to write code that manages and synchronizes the worker threads. Furthermore it will mean that player B may be disadvantaged during PvP with player A. When using threads this way you have no control over in which order events happens.

I agree with Vassi that threading makes perfect sense when you need to service multiple connections, but they don't interact with eachother. Webservers, ftp servers and the like are typical examples. But for a mud you may very well trade one headache for another.

I see.
17 Oct, 2008, quixadhal wrote in the 11th comment:
Votes: 0
That's because read/write calls are done on sockets that have been set to non-blocking mode. Right now, most MUD's are coded as a big loop that performs all the I/O at either the top or bottom of that loop. If your game logic code spins into a loop because someone decided to use bubble sort instead of quick/heap sort, you don't get around to handling I/O right away. Because sockets have internal buffers, as long as that isn't too long, it's not a huge deal…. but people will notice the lag as a brief pause and then a burst of output.

If you spawn a single thread for I/O, the only real advantage you'll see is that users whose output got queued before your lag spike won't notice the lag, since their output will go out "on time". This assumes you have more than enough CPU to catch up before the next pass through the loop.

However, if you spawn a new thread for every connection, you might see one other benefit. With seperate threads, you can stop using non-blocking I/O, which means you don't have to run a poll loop for I/O. Thus, the system can let the blocked processes sleep, which might give you a noticeable drop in CPU use. It's been ages since I've done blocking network I/O, but I believe you just spawn two threads, one for reading, one for writing, and each one gets their own buffer. So, one thread does a recv and dumps it into the "input" buffer for the game to grab. Another does a write on anything in the "output" buffer. Normally, write/send won't block unless the other end of the socket is busy, however you can have that thread sleep and have your main thread send it a signal when something is deposited in the output buffer.

Would it be enough of a gain to be worth the effort? Hmmmm…. probably not. Not unless you're designing your codebase with that in mind. It would simplify the parts of the loop that handle I/O, new connections, etc… but that's about it.
17 Oct, 2008, Vassi wrote in the 12th comment:
Votes: 0
Lobotomy said:
Caius said:
There's no simple yes/no answer to that question. Say you use two threads. One main tread that runs the game logic and one thread that maintains the network. If the network thread is slowed down by a read or write operation, then yes, the main thread will happily churn on. However in this scenario you'll still have the issue that you only read/write one socket at a time, so the players may not necessarily notice better responsiveness. Then you have the option of the network thread spawning a handful of worker threads. In this scenario player A on a speedy connection may have instant responsiveness even though player B is on a super-slow connection, as long as they're serviced by two different threads. This scenario adds complexity because you need to write code that manages and synchronizes the worker threads. Furthermore it will mean that player B may be disadvantaged during PvP with player A. When using threads this way you have no control over in which order events happens.

I agree with Vassi that threading makes perfect sense when you need to service multiple connections, but they don't interact with eachother. Webservers, ftp servers and the like are typical examples. But for a mud you may very well trade one headache for another.

I see.


In short, if you're not already familiar with threading the answer you're looking for is no when it comes to threading the I\O. Abstracting it is a solid yes, but threading it is something you need to be very good with. Either you go whole hog - which means locking, everything, as you handle it (and at that point you're pretty much overriding the performance benefits, in my opinion, unless you have a lot of connections) or, as was stated earlier, you end up still having to synchronize everything into a single queue and it's pretty much…why did I bother?

It makes more sense to spin off threads for tasks that might take a while if handled synchronously but that don't really impact the player such as saving their character (make a copy, send the copy off to a file dump thread) or when loading a help file, or when doing OLC, have the extra threads communicate the results of the task through a central messaging\event queue.

I don't blame you if you don't want to believe us, I didn't believe anyone when they told me - and i wasted a whole year fixing problems that didn't have to be problems if I had simply stuck with my single threaded model, though to be fair I did learn a lot of useful things about threading in the process.

Still, this time 'round (3rd time is a charm) I'm taking my own advice by including a thread-safe priority queue for events, and from the sounds of it, the base you're starting with already has one. Well, a priority queue, I'm not sure if it's thread safe yet ;)
17 Oct, 2008, Vassi wrote in the 13th comment:
Votes: 0
quixadhal said:
However, if you spawn a new thread for every connection, you might see one other benefit. With seperate threads, you can stop using non-blocking I/O, which means you don't have to run a poll loop for I/O. Thus, the system can let the blocked processes sleep, which might give you a noticeable drop in CPU use. It's been ages since I've done blocking network I/O, but I believe you just spawn two threads, one for reading, one for writing, and each one gets their own buffer. So, one thread does a recv and dumps it into the "input" buffer for the game to grab. Another does a write on anything in the "output" buffer. Normally, write/send won't block unless the other end of the socket is busy, however you can have that thread sleep and have your main thread send it a signal when something is deposited in the output buffer.


Two threads per connection each with their own buffer will get you in trouble right quick with a semi-popular MUD (let's say 20+? connections) if you're not extremely careful, having too many threads is worse than not having enough. At that point you can, and should, be implementing some kind of buffer manager to re-use buffers only when needed - which is not trivial if you've never done it before (mostly, again, making it thread-safe) A single threaded approach lets you share the same input buffer for all reads, since it's guaranteed to only be used by one connection at a time.

You can 'mask' some of the unresponsiveness by scheduling your reads and writes on a single threaded process and not processing more than x bytes at a time, particularly for the writes. I also always recommend queuing the inputs and only processing one command per player on any loop. That way if a player is 'lagging' and suddenly hammers the server with 10 commands they don't all get processed at the same time, delaying the next person in the queue.
17 Oct, 2008, quixadhal wrote in the 14th comment:
Votes: 0
It also makes a BIG difference what OS and language you're using. For example, under Solaris, threads are implemented (or were) within the same process space, which meant that the entire process was managed the same, regardless of how many threads might be in it. OTOH, linux treats threads a first class processes that just share the same process group. Thus, they are treated by the scheduler as individual processes.

So, under solaris, you could spawn a dozen threads and you'd still get one time slice, so it has no real impact on the CPU load of the server. In linux, spawning a dozen threads can suck up a dozen time slices, which can make your sys-admin very unhappy.

The LPMUD world has handled DNS resolution by running a seperate server that just accepts requests from the main server and responds whenever it gets around to it.
17 Oct, 2008, David Haley wrote in the 15th comment:
Votes: 0
(Warning, very long post, see end for summary :wink:)

quixadhal said:
The 4K chunk size comes from the most common disk/memory page size, and is also a convenient packet size to send out. Even though you're using tcp (a stream protocol), it's still more efficient if you can avoid crossing packet boundaries due to the unpredictable nature of lag.

This is not something that anybody should ever care about when writing user-level programs with modern operating systems. When you call these functions, you are not actually sending data on the socket. You are informing the OS that you want that data to be sent. The OS TCP library will then do whatever it does, and ship it to the OS IP library, which will do whatever it does, and so on until it finally reaches the network hardware driver. (Note that the OS libraries might combine steps, especially TCP/IP, for efficiency.) Therefore, if you write something too big, one of the components in the stack will cheerfully split it for you. It might combine packets for you, split packets, or even buffer things and send several at once even though you made separate requests.

In brief, the network API should be treated as a black box into which you shove data and from which data appears. Trying to outsmart the networking layer by playing games with packet sizes is really not something that's useful to do when you are talking to the TCP/IP layer. It's better to just let it do its job.

quixadhal said:
With seperate threads, you can stop using non-blocking I/O, which means you don't have to run a poll loop for I/O. Thus, the system can let the blocked processes sleep, which might give you a noticeable drop in CPU use.

The polling call (select) sleeps while waiting for something to happen, and the OS wakes up the process when something happens. So I don't think that this will really be a factor here.

It was suggested to use a thread per connection. FWIW, that sets off all kinds of warning bells in my head. The added complexity is considerable, and the gains are very unclear if all you're doing is IO. Programs that do this, like Apache, aren't doing it to handle IO in separate threads; they're doing it to handle the entire request logic in the separate thread.

Note also that there is a limit on the number of threads that you can spawn.

Caius said:
In this scenario player A on a speedy connection may have instant responsiveness even though player B is on a super-slow connection, as long as they're serviced by two different threads.

I'm not sure this is a factor. If you use non-blocking IO, which everybody should be doing, then if you try to write to a socket that's not ready the OS will just tell you to come back later. So even with a completely single-threaded model, one person's connection speed is basically irrelevant w.r.t. overall responsiveness.

Vassi said:
You can 'mask' some of the unresponsiveness by scheduling your reads and writes on a single threaded process and not processing more than x bytes at a time, particularly for the writes. I also always recommend queuing the inputs and only processing one command per player on any loop.

This is an excellent reason for the networking and game logic to be decoupled. The MUD server should treat the networking module as a "black box" (sound familiar? :wink:) that dumps output to player connections and generates player input (usually in the form of whole lines).

The game logic should use this black box, conceptually at least, by pinging it every tick to see if it has input for the player. The game logic can then choose to handle all of that player's inputs, or more reasonably as Vassi suggested only handle one command and then move on to other players.

I'm not sure there's any reason to worry about the writes, though, again assuming that you are using non-blocking IO. Just dump as much as you can, and the OS will eventually tell you when its buffers are full and you need to wait. At that point you just wait until the next tick and try to send more, again waiting if the OS tells you to. This way, if somebody can handle a lot of data, they'll get it quickly, and nobody has to suffer because one person has a slow connection.


—————————


Let me make some general comments on this topic…

Threading is complicated.
Networking is complicated, albeit in different ways.
Threads should only ever be used when you're actually gaining something from them.
When combining two complicated things, care should be taken to understand what is going on, and where you are really getting benefits.

The general reason to use threads is when you want the whole program to continue during some expensive operation. That could mean you have a multi-user system and you don't want everybody to have to wait if one person is doing something expensive. That could mean that you have a GUI application and you want to be able to process the cancel button while some operation is underway.

The theme is that threads help you continue computation during a blocking operation. When you have long, blocking operations such as DNS lookups, the need for threading (or separate processes or whatever) is pretty clear. If you have no blocking operations – if everything is more or less instantaneous – the need for threading is much less obvious.

Most OS socket libraries provide some form of non-blocking input. As I mentioned above, that means that it'll keep taking data "instantly" and will eventually tell you "no, come back later".

In other words, with this way of handling sockets, you no longer have long, blocking operations. So why are threads useful? I would argue that they're not necessarily appropriate here, because there's no blocking operation to help you with. And of course, you now have to deal with synchronization.

Even if you split networking into its own thread(s), having more than one thread really does not seem obviously helpful to me. Now you have a whole bunch of threads – that you need to synchronize – whose sole purpose in life is to sit around making non-blocking IO calls. This means that the majority of time will be spent waiting on locks or other signals that it's time to write something. It's unclear why this is helpful.

To summarize (sorry for the very lengthy post) –
- Don't try to second-guess the OS's networking library. Chances are that it will be better than you anyhow, and besides, it is free to do whatever it wants with your packets and is not guaranteed to write packets in the size you give it.
- Don't use threads unless you have a very clear reason for them.
- Separating game logic and networking into different modules can be extremely useful, but you don't necessarily want multiple threads to process them.
17 Oct, 2008, elanthis wrote in the 16th comment:
Votes: 0
Listen to David. He will make you strong. ;)
17 Oct, 2008, Vassi wrote in the 17th comment:
Votes: 0
elanthis said:
Listen to David. He will make you strong. ;)


Isn't that what I said? :wink:
17 Oct, 2008, David Haley wrote in the 18th comment:
Votes: 0
I wanted to clear up some misconceptions I noticed, but it's true that I did say many things that others had said or suggested. :wink: I figure that when it comes to threading, it really can't hurt to put out some extra words of caution…
0.0/18