As the title suggests the current protocol specification is currently leaning towards maintaining human readability. One of the reasons is the ease of conceptualization for new client developers. Your thoughts on the subject would be appreciated. In the short future this will be solidified by Davion as a lot of the current development hinges on it.

Human readable packets make things easier to debug. You can get a pretty quick picture of what's going over the wire.

Though on the server end, once debugged, switching to numeric codes or whatever might be somewhat helpful for the internal stuff nobody would ever see.

I agree that packets should be as readable as possible. The best way to further adoption of this network is to support new client design, and this is a step in the right direction. Fortunately the string utilities in Ruby make this easily accomplished on the server.

Human readability is most important when the packet format is obscure to begin with. It gives you a relatively short-lasting benefit, at the cost of increasing packet bandwidth usage forever.

In the scale of things, the problem with implementing clients isn't parsing packets, it's figuring out what the contents actually mean. If the packet type names and values are obscure to begin with, having their names in English won't make any difference.

In short, the problems in practice are not due to not understanding what packet type 23 is – that is trivial to figure it. It's figuring out why the server is sending you that packet in the first place.

Key=value pairs are important when packets can actually be dynamic. HTTP headers are an example of this: a bunch of stuff can go into those, so it makes sense to have a variable-length list of arguments with each argument type being specified. If this were the case with IMC, that would be a reason to have key=value pairs.

Also, no matter what you do, the delimiters will have to be escapable anyhow. For example, if the packet format uses quotation marks to enclose arguments (assuming spaces are the delimiter between arguments), you will need to escape quotation marks inside arguments. You will never be able to escape the need for escaping things one way or another – choosing such common strings/characters as protocol-significant values simply makes this more obvious.

Please keep the big picture in mind: clients are developed very rarely, and bandwidth is used all the time. Even with compression, there's no point sending over huge quantities of data just in the same of saving somebody from having to look uip 23 to "chat message".

Along the lines of the points DH made, I'd like to see readability kept at a series of #defines, allowing you to form packets as "SOME_PACKET <> SOME_TYPE <> SOME_CHAN <> blah blah text <> EOF", which would obviously just break down to a minimal amount of numbers and the pertinent text, code wise. Any unnecessary strings should be kept out for ease of dealing with things.

Two more notes:

- human readability is no substitute for good documentation. Either way, you need excellent and precise technical documentation that leaves no ambiguity; when you have this level of documentation, the fact that the packet says "imc_login" instead of 2 is kind of irrelevant.

- do not allow lax packets. There should be exactly one format, and it should be stuck to religiously. No fuzziness. I care about this far more than human readability.

If the stated goal is ease of parsing, then point 2 seems kind of incontrovertible.

2 posts in a row David?

Making up for lost time I see ;D

Those weren't posts in a row, they were made 1.5 hours apart after I had more time to think and after somebody else had replied. :wink:

We should totally make the packets Binary!

Have the Server translate the packet to binary, and then have the client translate it back!

hehehe.

(Yes, I am still alive, I just read, never post.)

Could we please try to keep the discussion on-topic… :thinking:

I'll avoid overstating my case or minimizing legitimate points made by others by just saying I can see legitimate reasons for both ways.

It would be helpful to have examples of what the human readable format would actually look like, i.e. what the specification of a "human readable packet" is.

Human readability is a good thing, as long as it isn't taken to extremes (I'm looking at you, XML!). YAML is slightly better, but it's still pretty verbose for something that is likely not all that dynamic in format to begin with.

There's also the question of making everything delimited or having known header fields be fixed size. Fixed size records might seem like dinosaur bones from the later COBOL era, but they are quite handy sometimes.

For example:

You want to send a "tell" to the character Bob at FooMUD. You are Joe at BarMUD. For the sake of the example, we'll say the packet type for single-target tells is 42. The packet could look like:

Tell : { From : { User : Joe , Mud : BarMUD } , To : { User : Bob , Mud : FooMUD } , Content : { Bugger off\, fool! } }

or

42\tJoe@BarMUD\tBob@FooMUD\tBugger off, fool!\n

or

042Joe\tBarMUD\tBob\tFooMUD\tBugger off, fool!

or even

042Joe BarMUD Bob FooMUD QnVnZ2VyIG9mZiwgZm9vbCE=

The top one is sorta-like YAML (I didn't look it up to see), and is self-documenting… easy to read and debug… but a bit verbose as a result. The last couple use some fixed field widths, and some fixed field ordering, which would need to be well documented, but it also easy to read once you've looked it up.

So, what *kind* of readability do we want? What level of self-documentation vs. looking things up? Do we want to have to escape characters in the data, or encode it (the last one was base64, for example)?

EDIT: I totally agree with David… whatever format is developed, it needs to be adopted and adhered to rigidly. The server should flat out refuse to accept malformed packets (reporting such), rather than trying to kludge them into an acceptable format. That format needs to be well documented, so when someone wants to write a client in Turbo-Pascal, or write a twitter2IMC3 bridge for the iPhone, they at least know for sure what they're trying to do. :)

See, personally, of the ones above, I think the first is excessively verbose, the last is too strict and like the third is too hard to parse (a straight string split doesn't suffice, and fixed widths are dangerous anyhow!), and the second is approaching something that I find reasonable.

By the way, as soon as you have delimiters, you will need escaping one way or the other.

Only if they can appear in the data.

For this: 42\tJoe\tBarMUD\tBob\tFooMUD\tQnVnZ2VyIG9mZiwgZm9vbCE=

You're not allowed to have \t in a username, a mud name, or the packet type code, and the data won't have it because it's been encoded (wrapped).

In the above, you'd split on TAB, and the first entry is the packet type. Knowing that, you expect entries 2 and 3 to be the sender-name and sender-mud, and entries 4 and 5 are the target-name and target-mud, and then entry 6 is the payload.

I don't agree that fixed widths are dangerous (we ARE following the spec to the letter, right???), but they also might not save us that much unless we're talking bulk data.

Well, I suppose you could forbid a tab ever appearing in absolutely any field, but that seems dangerous and unnecessary to boot.

It's a little interesting that you tell me you don't need escaping, and then go on to say that it can't be in the data because "it's been encoded" – I'm not entirely sure how "encoding" a tab is any different than "escaping" it. :rolleyes:

Fixed widths are dangerous because they're somewhat harder to parse (and so error-prone), because they're very limiting in terms of future expansion, and because they're harder to generate (you have to remember to clip or pad as appropriate). So, they are wasteful when the data is smaller than the fixed width, and limiting (read: bad) when the data is larger than the fixed width. In other words, they don't really seem to be accomplishing anything here… fixed width messages are appropriate when you're sending fixed-size binary data, not packets containing variable length data (character names, MUD names, messages, …)

Well if a delimiter must be used, it should be one the users can't type into their terminals to send to the game. Otherwise you run into the existing problem with quotation marks.

And fixed width = bad bad bad. Way too prone to mistakes with trying to stuff data in that's too large.

No need for escape characters if you include length fields for strings.
Why would you choose to annoy the user over the implementer?

My post contains 2 points. The first is that I feel the more important problem is one of consistency (and the lack thereof in some parts of IMC2). The second point is that we need to focus on who the intended audience for the protocol will be, and think about whether that audience will benefit from human readable packets.

As someone who has been implementing an IMC2 client lately, I would say I have a much larger interest in seeing a consistent packet format than anything else. Human readability does not matter that much to me, but throwing everything into key/value pairs would be very easy to parse. I don't really even mind the IMC2 practice of having a fixed format of args at the front of the packet, and then an open ended number of key=value pairs at the end of the packet.

However, there are some inconsistent quirks that make parsing much less straightforward. While white-space is supposed to be quoted when included as part of a value, there are some params that are lists of white-space delimited values. Excluded/Invited param on an ice-update packet, I'm looking at you. So you end up with this:

ICE@Server01 […] ice-update […] excluded=jim@MyMud bob@yourMud sally@herMud level=Imm localname=ichat

IMHO the proper way to handle this would be specify a separate delimiter for the list of excluded users, or to simply escape the whole thing in quotes. The current behavior is inconsistent and makes it very difficult to separate parsing from interpreting.

I can't name any specific problems to solve by creating a new format. Can someone please spell out any problems with the current format (besides the one I am complaining about ;). I understand the impetus to make it easier on developers by making the packets human readable. However, if you were to look at the IMC2 deployments out there, they all seem to stem from a handful of client libraries. So I think it is fair to assume that the same will be true of IMC3 – namely, a select group of (mostly) competent developers will build libraries that they will then distribute. If the lower-level packet handling is taken care of deep in the bowels of the client libraries, will its readability really give that much of a benefit?

Which user are you talking about? I thought that the IMC client would be the one escaping things before shipping them off to the IMC server; the MUD player would be no more aware that something happened than of the packet format in the first place.


Frankly, I think that most complaints stem from the fact that the format and its semantics are somewhat obscure, and inconsistent. What people really want is clarity, I believe, not necessarily complete human readability.