14 Jul, 2010, Scandum wrote in the 1st comment:
Votes: 0
I've been on the fence regarding nesting support for MSDP for a while, but I've come to the conclusion that it's for the better to add it as most modern clients have associative array support, or a hack job resembling it.

As usual, all feedback and comments are appreciated, and when it comes to the specification I've got the final word after careful consideration as the BV (benevolent dictator). If I had the time I'd setup a representative democracy where people get a number of votes based on their relative importance in the Mud Community, for example, Scandum 1432, KaVir 128, David Haley 1, etc. [/humor]

Here's what I got so far:

Associative Arrays

Sometimes it's useful to send data as a bundle. Associative arrays can be created in MSDP using MSDP_OPEN and MSDP_CLOSE after MSDP_VAL, and nest a variable and value inside the MSDP_OPEN and MSDP_CLOSE arguments. There is no limit on the number of nests, or the number of variables and values within a nest. It is up to the client to decide how to exactly process associative arrays.

An associative array containing room data in MSDP would look like:

IAC SB MSDP MSDP_VAR "ROOM" MSDP_VAL MSDP_OPEN MSDP_VAR "VNUM" MSDP_VAL "1234" MSDP_VAR "AREA" MSDP_VAL "MIDGAARD" MSDP_VAR "EXITS" MSDP_VAL MSDP_OPEN MSDP_VAR "N" MSDP_VAL "1235" MSDP_VAR "S" MSDP_VAL "4321" MSDP_CLOSE MSDP_CLOSE

If you write MSDP_VAL preceded by a variable as " = ", MSDP_OPEN as "{ ", MSDP_CLOSE as "} ", and MSDP_VAR preceded by a value as ", " an associative array in MSDP would look like:

IAC SB MSDP "ROOM" = { "VNUM" = "1234", "AREA" = "MIDGAARD", "EXITS" = { "N" = "1235", "S" = "4321" } } IAC SE

MSDP_OPEN would have value 3, MSDP_CLOSE value 4.

Creating a parser that converts MSDP to JSON, Lua, etc, should be fairly easy as I don't think there are any inconsistencies.
14 Jul, 2010, David Haley wrote in the 2nd comment:
Votes: 0
Why use special numbers for open and close when you can use { and } in the first place, because you won't have ambiguity w.r.t. the quotation mark?

In fact, why do you have the var/val tags in the first place, when it's unambiguous that you have a sequence of name/val pairs? Your other notation is far clearer, not to mention more compact, than the first.

Quote
It is up to the client to decide how to exactly process associative arrays.

What does this mean? How can there be many ways to process them?

Quote
Associative arrays can be created in MSDP using MSDP_OPEN and MSDP_CLOSE after MSDP_VAL

You are slightly misusing the term associative array here; you already had an associative array of key/value pairs. This is just a nested version of something you already have. So you could always create associative arrays with depth 1.
14 Jul, 2010, Scandum wrote in the 3rd comment:
Votes: 0
David Haley said:
Why use special numbers for open and close when you can use { and } in the first place, because you won't have ambiguity w.r.t. the quotation mark?

I'm not sure if I understand you. { and } would be plain text meaning you'd have to start escaping. The quote characters in the examples mean that the encased word is a string, the quotes themselves are not send.

David Haley said:
In fact, why do you have the var/val tags in the first place, when it's unambiguous that you have a sequence of name/val pairs? Your other notation is far clearer, not to mention more compact, than the first.

Because MSDP is based on NEW-ENVIRON which uses var/val tags. There's no logical difference between marking the start of a variable, or instead marking the end of a value. In hind sight it probably would have been better to mark the end of a value instead because that's the norm nowadays, then again it's possible to write it down as:

$ROOM = { $VNUM = 1234 $AREA = MIDGAARD $EXITS = { $N = 1235 $S = 4321 } }

which doesn't look all that bad though there's no clear separation between strings and control characters in that notation.

David Haley said:
What does this mean? How can there be many ways to process them?

A client with no nesting support can create several events, for example: ROOM_VNUM = 1234 + ROOM_AREA = MIDGAARD + ROOM_EXITS_N = 1235

It's possible for a client to store it all nested but still generate an event for each value, or store it all nested and generate one event. A client might also convert a MSDP table into a Lua or JSON table, and provide that to the scripting interface. There are a decent amount of sane options.


David Haley said:
Quote
Associative arrays can be created in MSDP using MSDP_OPEN and MSDP_CLOSE after MSDP_VAL

You are slightly misusing the term associative array here; you already had an associative array of key/value pairs. This is just a nested version of something you already have. So you could always create associative arrays with depth 1.

Good point, guess I'll call them nested variables.
14 Jul, 2010, David Haley wrote in the 4th comment:
Votes: 0
Quote
I'm not sure if I understand you. { and } would be plain text meaning you'd have to start escaping. The quote characters in the examples mean that the encased word is a string, the quotes themselves are not send.

Ah, I forgot that. In that case the delimiters serve the role of quotation marks, and you need the special numbers for open/close.

Quote
which doesn't look all that bad though there's no clear separation between strings and control characters in that notation.

One way or the other it must be unambiguous, and that means that if your delimiter can be contained in values, some sort of escaping is necessary. I'm not sure why any compatibility with new-environ is really needed, though. I have a general preference for a more human-readable format so that you can debug with the raw stream, but that's not really a big deal.

Quote
A client with no nesting support can create several events, for example: ROOM_VNUM = 1234 + ROOM_AREA = MIDGAARD + ROOM_EXITS_N = 1235

It's possible for a client to store it all nested but still generate an event for each value, or store it all nested and generate one event. A client might also convert a MSDP table into a Lua or JSON table, and provide that to the scripting interface. There are a decent amount of sane options.

I might not be following but it seems to me that it would be kind of nonsensical to treat nested data without nesting. I agree that the internal representation (JSON, Lua, Python dict, whatever) is irrelevant, but this is only as long as the structure is preserved. What it would mean to flatten everything out? You'd have variable conflicts, for starters.
14 Jul, 2010, Scandum wrote in the 5th comment:
Votes: 0
David Haley said:
I might not be following but it seems to me that it would be kind of nonsensical to treat nested data without nesting. I agree that the internal representation (JSON, Lua, Python dict, whatever) is irrelevant, but this is only as long as the structure is preserved. What it would mean to flatten everything out? You'd have variable conflicts, for starters.

A client could use a dash to avoid conflicts. So if the mud sends ROOM_AREA besides ROOM[AREA] this would end up as ROOM_AREA and ROOM-AREA.

I'm not overly concerned with clients that lack nesting support as it's something that can be added if the client has a capable developer, and many servers might see no need for nesting because it's bulkier than an array.
14 Jul, 2010, David Haley wrote in the 6th comment:
Votes: 0
I guess what I'm trying to say is that behavior should not be left underspecified. It seems that the intention, given your most recent post, is pretty clearly that clients should support nesting. Therefore the spec should say that a client should either handle the nesting structure correctly, or report some sort of error (or discard it).
14 Jul, 2010, KaVir wrote in the 7th comment:
Votes: 0
Scandum said:
I've been on the fence regarding nesting support for MSDP for a while, but I've come to the conclusion that it's for the better to add it as most modern clients have associative array support, or a hack job resembling it.

I guess I'm still on the fence. One of the things I like about MSDP is its simplicity, and this seems to move away from that.

MSDP already allows you to create whatever variables you like, with whatever values you like. For example, I use spaces to separate multiple values for some of my variables, as I find it easier to parse than using lots of MSDP_VALs. If a particular mud owner decides to add a "ROOM" variable that sends a JSON representation of the room, then that's already within the scope of the specification.

I suppose you could provide an MSDP_JSON indicator for ease of parsing, eg:

IAC SB MSDP MSDP_VAR "ROOM" MSDP_JSON … IAC SE

And possibly extend it to others formats such as MSDP_LUA, MSDP_PLAINTEXT, MSDP_BINARY, etc, with MSDP_VAL representing an unspecified format. But once again this seems to be moving beyond the original scope of MSDP.
15 Jul, 2010, Scandum wrote in the 8th comment:
Votes: 0
KaVir said:
I guess I'm still on the fence. One of the things I like about MSDP is its simplicity, and this seems to move away from that.

As a server developer you don't have to use or support nesting. Clients are pretty much forced to support nesting, but that shouldn't be an issue.

KaVir said:
MSDP already allows you to create whatever variables you like, with whatever values you like. For example, I use spaces to separate multiple values for some of my variables, as I find it easier to parse than using lots of MSDP_VALs. If a particular mud owner decides to add a "ROOM" variable that sends a JSON representation of the room, then that's already within the scope of the specification.

That is my biggest worry, each mud creating its own standard to do the exact same thing. With nesting standardized the clients can translate MSDP tables into native data tables, making plugin development a lot easier. Imagine MUD A) using JSON over MSDP, MUD B) using Lua over MSDP, and MUD C) using LPC over MSDP - it would be a nightmare for cross-server plugin development.
15 Jul, 2010, KaVir wrote in the 9th comment:
Votes: 0
Scandum said:
That is my biggest worry, each mud creating its own standard to do the exact same thing. With nesting standardized the clients can translate MSDP tables into native data tables, making plugin development a lot easier. Imagine MUD A) using JSON over MSDP, MUD B) using Lua over MSDP, and MUD C) using LPC over MSDP - it would be a nightmare for cross-server plugin development.

Well my point was that those three muds would most likely support different MSDP variables anyway, so a complex plugin would probably only ever work out-of-the-box for one mud - you could certainly adapt it for another, but you'd need to do some tweaking. If you're doing that anyway, it's not much extra effort to add a "json.decode (params)" or whatever when you read in the variables.

IMO: ATCP/2 is the works-for-all-muds protocol, while MSDP is the tailored-to-each-mud protocol. Each approach has its pros and cons.
15 Jul, 2010, Scandum wrote in the 10th comment:
Votes: 0
KaVir said:
If you're doing that anyway, it's not much extra effort to add a "json.decode (params)" or whatever when you read in the variables.

That's the whole problem though, adoption of MSDP will be slowed down quite a bit if every client needs a special decoders for each MUD.

It's fairly easy to change a couple of variables that are different. Writing a JSON parser in tintin++ requires that extra bit of skill and intelligence most users do not possess.
15 Jul, 2010, KaVir wrote in the 11th comment:
Votes: 0
Okay, let's see if we can work this out with a real example. Last night I was having a play with spell affects - I wanted to send these as a single "AFFECTS" variable which contained all of the spells as its value. I started out with my usual approach of spaces as separators:

IAC SB MSDP MSDP_VAR "AFFECTS" MSDP_VAL "icon,type,name,duration icon,type,name,duration icon,type,name,duration" IAC SE

I ran into an immediate problem - the spell names can contain spaces.

Now I know the original idea was to handle each spell as a separate value, like this:

IAC SB MSDP MSDP_VAR "AFFECTS" MSDP_VAL "icon,type,name,duration" MSDP_VAL "icon,type,name,duration" MSDP_VAL "icon,type,name,duration" IAC SE

However I found this difficult to parse in MUSHclient - it was much easier to just pair up a generic variable/value pair using MUSHclient's SetVariable() and GetVariable() functions.

So I tried using the value '3' as a separator, and that actually works really well. I can pair up the variables through my generic subnegotiation-reading function without caring what they are, and then break them down further when it's time to use them.

MSDP_VAR is 1 and MSDP_VAL is 2. If we were to say MSDP_ELEMENT is 3, the sequence could be written as:

IAC SB MSDP MSDP_VAR "AFFECTS" MSDP_VAL "icon,type,name,duration" MSDP_ELEMENT "icon,type,name,duration" MSDP_ELEMENT "icon,type,name,duration" IAC SE

I guess it could even be broken down further, perhaps with MSDP_FIELD as 4?

IAC SB MSDP MSDP_VAR "AFFECTS" MSDP_VAL "icon" MSDP_FIELD "type" MSDP_FIELD "name" MSDP_FIELD "duration" MSDP_ELEMENT "icon" MSDP_FIELD "type" MSDP_FIELD "name" MSDP_FIELD "duration" MSDP_ELEMENT "icon" MSDP_FIELD "type" MSDP_FIELD "name" MSDP_FIELD "duration" IAC SE
15 Jul, 2010, Scandum wrote in the 12th comment:
Votes: 0
Maybe some of the guys at the MC forums can help you out writing an MSDP parser that directly loads a nested MSDP table into a Lua table?

Using $ { } : notation the most efficient thing to do would be:

$AFFECTS : { $name : icon : type : duration } : { $name : icon : type : duration } : etc. Then in Lua (if you got a routine to load nested msdp variables into a lua table) it should be possible to get all the keys (affect names), and use those to access the arrays.

I'm not sure why you don't send it as a full associative array, it's a little more data, but you only need to send the data once, and it'll be much more accessible once it's inside a lua table as key value pairs.

MSDP is currently 100% Lua and TinTin++ compatible, adding 2 dimensional array support would add no additional functionality at the cost of increased complexity.
16 Jul, 2010, KaVir wrote in the 13th comment:
Votes: 0
Scandum said:
Maybe some of the guys at the MC forums can help you out writing an MSDP parser that directly loads a nested MSDP table into a Lua table?

If I'd wanted to do that I could have just used the Aardwolf 102 protocol. Interestly enough, Lasher explicitly mentioned the way I use MUSHclient's GetVariable/SetVariable as the reason why he likes the idea of MSDP, and said he had pretty much exactly the same thing in mind.

Scandum said:
MSDP is currently 100% Lua and TinTin++ compatible, adding 2 dimensional array support would add no additional functionality at the cost of increased complexity.

It allows me to deal with the few cases of nested data I've encountered, without losing use of the GetVariable/SetVariable functions provided by MUSHclient. It's less complex and less data than an associative array.
16 Jul, 2010, Scandum wrote in the 14th comment:
Votes: 0
Your plugin is in Lua however, wouldn't it make more sense to use Lua tables for storing variables?

Regardless, as long as you don't use value 0, 1, 2, 3, 4, and 255 you can use anything you like. Ideally this being:

IAC SB MSDP MSDP_VAR "AFFECTS" MSDP_VAL "icon,type,name,duration" MSDP_VAL "icon,type,name,duration" MSDP_VAL "icon,type,name,duration" IAC SE

as this should work across clients. I'm not exactly sure why this is hard to parse, though I guess you won't find build-in functions for parsing the MSDP format.

At this stage of the protocol (1 mud 1 client) it's not too late to make some fundamental changes and adopt a standard nesting format where the start of a variable is assumed by it's position, and instead the end of a key value pair is marked, typically with a comma in string notation. So that would mean switching to using = , { and } instead of $ = { and } and you'd send:

IAC SB MSDP "X" MSDP_VAL "SOMETHING" MSDP_COMMA "Y" MSDP_VAL "NOTHING" IAC SE which in string notation would look like "X" = "SOMETHING", "Y" = "NOTHING"

To keep things simple there'd be no array support, though one could use: "var" = { "1" : "bli", "2" : "bla", "3" : "blo" } which is Lua compatible.

So this would make MSDP a barebone typeless subset of Lua, and people who don't like nesting can use "var" = "value" and use : and , (or something of that nature) for 2 dimensional arrays.
16 Jul, 2010, KaVir wrote in the 15th comment:
Votes: 0
Scandum said:
I'm not exactly sure why this is hard to parse, though I guess you won't find build-in functions for parsing the MSDP format.

It was the built-in functions I was explicitly interested in. However while writing up a more lengthy response I realised that I could rewrite my parser to read front-to-back (instead of back-to-front like I do now) and simply add all the values together, keeping MSDP_VAL as the separator, so that:

IAC SB MSDP MSDP_VAR "AFFECTS" MSDP_VAL "icon,type,name,duration" MSDP_VAL "icon,type,name,duration" MSDP_VAL "icon,type,name,duration" IAC SE

Would be stored as:

"AFFECTS" = "icon,type,name,duration\2icon,type,name,duration\2icon,type,name,duration"

That could then be stored with the MUSHclient SetVariable() function, and later retrieved with GetVariable() when it's time to use the data.

By the way, is there a limit to how many MDSP_VALs I can send to TinTin++? I thought one of my players said something about a limit of 100, but I'm not sure if he was talking about something else. If I change my map data to be sent this way, it'll include 121 MSDP_VALs.
16 Jul, 2010, Scandum wrote in the 16th comment:
Votes: 0
There's a limit of 100 variables for triggers and regular expressions, but an MSDP array can be any length. It's easier to parse MSDP than standard markup with regular expressions because the data is more uniform.

I'm still seriously considering switching to a standard markup, and I guess array support could be added, so that would mean you could send: "AFFECTS" = { "x", "y", "z" } which if loaded into Lua would automatically be converted to "AFFECTS" = {"1" = "x", "2" = "y", "3" = "z"}, in tintin it'd become "AFFECTS" = {x;y;z} and from what you've mentioned it'd be easier for you as well since your parser currently converts MSDP to standard notation?
16 Jul, 2010, KaVir wrote in the 17th comment:
Votes: 0
Scandum said:
I'm still seriously considering switching to a standard markup, and I guess array support could be added, so that would mean you could send: "AFFECTS" = { "x", "y", "z" } which if loaded into Lua would automatically be converted to "AFFECTS" = {"1" = "x", "2" = "y", "3" = "z"}, in tintin it'd become "AFFECTS" = {x;y;z} and from what you've mentioned it'd be easier for you as well since your parser currently converts MSDP to standard notation?

To be honest it'd be easier if it stayed the way it is now. Having said that, I notice the specification states that "It is up to the client to decide how to exactly process multiple values. It's also possible to attach an array to a single variable by using MSDP_VAL more than once, with the default value (if any) reported last."

Supposing a client decides to implement MSDP as a built-in protocol, does that mean it could choose to discard all but the last value before passing the data on to the client/script writer?
16 Jul, 2010, Scandum wrote in the 18th comment:
Votes: 0
As it is I think the widest adoption will be with a Lua compatible format. So unless you have very strong objections I'll make the changes and hope for the best.

Regarding arrays, especially with MSSP there were cases, like multiple ports, where it was important to indicate that the most significant value should be reported last.
17 Jul, 2010, Scandum wrote in the 19th comment:
Votes: 0
Regarding 2 dimensional arrays the following format would work: affects = {{"a", "b", "c"}, {"a", "b", "c"}, etc} which should still be fairly easy to convert to a 2 dimensional array, while compatible with nested formats.

Regarding a switch, currently every MSDP packet starts with: IAC SB MSDP VAR, when moving to an IS (1), COMMA (2), OPEN (3), CLOSE (4) model, data will start either with OPEN or plain text, so for temporary backward compatibility it's possible to check if the data starts with VAR to read it as the old format, otherwise it's treated as the new format.
17 Jul, 2010, KaVir wrote in the 20th comment:
Votes: 0
It's one thing extending the standard to include new things, but I'd really rather not see the old approach dropped - it works well, it does everything I need, it's already being adopted by one client and considered by two others…and other developers have started showing an interest.
0.0/77