There may come a time in the future when crawling is the only behavior a listing site does and all the MUD admin does is register connection info for the crawler to get the rest from. That day is not now, so some kind of method to indicate that your game supports MSSP seems reasonable enough. That can go into the listing registration form, and I think having it defaulted to enabled is fine since people who don't want it can uncheck the box. Existing listings could then come back and uncheck it if they don't want it.

Setting it up as opt-in would likely stifle adoption as the general tendency would be to simply not click out of general laziness on the part of whoever is submitting the listing.

This is what I'm getting at. I think that some folks are liable to construct comprehensive
list-scraping crawlers…whose administrative process for opt-out may be difficult to find
or just flat out unavailable.

I don't think this is a far out possibility, I think it's an eventuality. Indeed I'm inclined to think
of it as a common scenario, eventually, assuming MSSP catches on.

It seems as though in order to ensure such crawlers receive your "don't crawl me" message,
the mud would have to actually implement MSSP to some extent.

Anything else is an out-of-band hail mary, hoping that joe schmoe's crawler will have one or more of:

1) A known website with an opt-out feature.
2) A trackable origin whose email can be figured out and the owner of which is reachable and reasonable.
3) The courtesy to check a mud's website (if extant) for some standard opt-out url and obey it.

I think this is hoping for a lot. I am just not certain that out of band opt out makes
enough practical sense to bother formalizing.

-Crat
http://lpmuds.net

That seems entirely reasonable and I think would be a good solution to solve any issues with the MUD Bytes crawler.




That may very well be the way listing sites go. I think what we're ultimately discussing here is making MUD Bytes as user friendly and respectful of Admins' wishes as possible, and who knows, maybe we're in a position to set some sort of precedence in that way.

The people who don't want MSSP, I think, then have the issue of dealing with each site and each crawler on their own, as I don't really see that being much of our concern, anyway. If MSSP does become the standard across the board, then refusing MSSP will likely entail having to implement it to check a "yes" in an opt-out field, but again, I think if the Administration here is willing to make it possible to opt-out without implementing MSSP or making changes to your game, then that's great.

I think overall MSSP is a good idea (as long as it isn't spamming you, and you have some control over the frequency of the crawler checking in) and I'll likely be on board when there seems to be a community consensus on the values and when Mud Bytes adopts it as its standard. I hope TMC and other sites will jump on board as well (of course with the hope they will be respectful of the wishes of Admins that don't want any part of it).

Ok, so whomever wrote the part of the crawler in question screwed up and made it loop on an error condition when it should have just marked that entry as "down" and moved on until the next schedule run.

So, since we all know there's no reason to EVER check a mud every couple of minutes, it was obviously a mistake in coding. I'm sure it will be fixed soon.

I do agree that an opt-out idea is good.

This is part of the MB MCCP protocol:

PLAYERS                                           Current number of logged in players.

Isn't that a reason? I mean what's the point of viewing a MUD listing if this data is outdated by hours?

Are we seriously trying to chart player trends down to the minute? I guess I can see why some people might want that but the interval at which its checked really should be up to the admin. So maybe a crawl-delay value makes sense after all. With crawlers using a reasonable default like 30 minutes for those games who don't offer the value up as part of their data.

So am I going to get banned from any of you guys muds if I have a poor connection and reconnect every few minutes?

Well, if that's the case… don't every disconnect. Just force every mud to let you hang around running the "who" command every 30 seconds so your graphs will be accurate.

There are better tools for that kind of data collection, and they're typically best run locally. MRTG is good for this. It would be nice if MSSP had an average players per unit-of-time field.



If you reconnect at more-or-less random intervals between 5 and 30 minutes? Doubtful… we might feel sorry for you. If you reconnect every 2 minutes like clockwork? Yeah, you're a bot.

The point isn't bot or not. Or clockwork or not. The point is people thinking getting a connection attempt (Even if blocked) every 2 minutes is an outrage. It's a possible scenario with actual players. (Not to menton just other crawlers.) It doesn't cause a denial of service and it should be handled gracefully even if something is trying to spam you into submission. It should be a non-issue for anyone with enough coding ability to actually run a successful mud.

This whole situation makes me want to make my own crawler that isn't documented. And mine is going to attempt to connect to your muds not once every 2 minutes, but *every* minute. Until the police come to my house for spamming your logs.

Er I meant MSSP not MCCP. But are we still talking about MSSP? "force every mud"? Isn't that what we were just arguing about how we don't want to force MUDs to do anything to handle how crawlers work? Besides, this would also require every MUD to code in "who" to work before login, and a good amount of MUDs don't have that.

I'm pretty sure nobody want to maintain a sub-negotiations stream that's going to be mostly idle when they can just log the connection attempts as a crawler instead of an actual player login attempt. (Since the whole debate is over the way we're choosing to log stuff, not actually if this effects performance.)

Ok, well, for the record, I'm in a little town hijacking an wireless network labled "Grandpa's Internet"… sorry grandpa :S. The crawler isn't supposed to go crazy! It must be a issue with the cron job governing the script (I'll take a look at it in a sec if Samson hasn't already). It's -supposed- to run every 3 hours.

For those who want to opt out of the crawling, currently the only way to do that is to remove your MUD from the mudlist. I appologize for the excessive spamming (btw, if someone could tell me if it's still going on, that'd be great). It really shouldn't do that.

All better. The cron job is now set to go off every 30 minutes. Don't know why, but according to the WebCP it was set to go off every 60 seconds. Which is simply not right :P

It hasn't happened against since I've been home, but looking over my logs, it's happened twice over the last couple of days it seems.

I'm still in support of letting the MUD Admin determine how often the crawler is allowed to connect, if possible.

For an in development game such as mine with occasional extended periods of little to no player activity, I think 30 minutes would still be a bit excessive. Once an hour (or every 3 as Davion said) would be more reasonable, but I honestly don't see the point in it running more than once a day. At least, the way I see it for my game, I wouldn't see any point in it being more frequent than that.

The business about checking who lists and such… we don't have it implemented to check the who list prior to login (and I don't think I've even ever played one that did) so I don't think that's a terribly important thing to be concerned with, myself.

Well in looking at the settings options again, it appears as though the system somehow believed it was being told to connect in every minute of each 3rd hour. Meaning every 3 hours, it would spend an entire hour connecting to stuff every minute, then go dark until the 3 hours were up and start over again. I hate cron notation though, so I could be reading it wrong.

Anyway. Since part of the purpose is to allow for it to track player trends, it doesn't make sense to only connect once per day. It makes sense to connect several times per day. And at every 30 minutes I can't see how anyone could argue it's any form of spam or DoS or whatever. There would have to be serious structural problems in the game for that to be the case.

If the crawler only runs once a day, say, at midnight. Most games will report a very low player connections. Probably give people the wrong impression. If you really can't take a simple ping every half hour, you'll have to remove yourself from the mudlist for immediate gratification, or wait till we get around to adding an opt-out option to our MUD Listings. But again, I appologize for the initial spam. Wont happen again!

ninja's are afoot

The only reason people have, or should have, for caring is that they get jittery about seeing it in their connection logs. It's not as if a lone connection once an hour that shuffles a few field/value pairs around is going to kill bandwidth. One way or another people will eventually realize that, like public web servers, MUD servers will get "spurious" connections and that's just how things work.

I also think that some people are underestimating the relevance of crawlers that aren't associated with their own listing website. That is, I could easily write a crawler that borrowed MB's listing and poked at those MUDs. Or, a crawler that scraped any other MUD list for that matter. The only way to say anything to these crawlers would be to either implement MSSP and issue it as an instruction, or have webspace on which to store the "robots.txt" file. Neither of these options are foolproof.

If it actually becomes a standard, not in the sense of one lone dude writing a document that has controversy but in the sense that a lot of people officially adopt it, I would say that eventually people will deal with it just like people deal with search engines.

Speaking of which, I still urge caution on moving too far with MSSP except as a beta test given how little consensus there is as to what things actually are and what they mean.

QFT

-Crat
http://lpmuds.net

100!