Friday Five 0131 | Main | Blogging news and more

Friday, January 31, 2003

I startle

Who did I startle, you ask? Why, none other than the folks at the National Center for Supercomputing Applications at the University of Illinois. Yeah, you've heard of them.

I'm the only person I know who actually corresponds with operators of web crawlers or robots. Maybe others do, and I just haven't met them yet. Or maybe I've met them, but they just haven't told me. Who knows?

Anyway, I don't recommend corresponding with just any crawler operator. That could lead to big trouble. But sometimes, if the situation is right, it's worth it. Like today. The VIAS bot visited my web site. Instead of me trying to explain what happened, following is the e-mail I sent them:

Subject: VIAS...

Today the VIAS bot visited my web site, hit the robots.txt file, but neglected to obey it: One would think that the highly esteemed University of Illinois would operate its robot in conformity with standards set for such robots. Quite possibly there is some little glitch in your program that causes this fault that you are unaware of. If that is the case, perhaps you can take the opportunity to correct the glitch. For your reference, following are the lines from my log files where the VIAS bot visited. Thank you and good luck. [log lines omitted here]

Am I the only person that still says "glitch?" I never see that any more. I need to get my terminology straight. Shortly thereafter, I received the following reply:

Thanks for your note.

We have always intended to honor the robots.txt convention, thus I was startled to receive your note.

However, you were right. We have discovered that there was a small set of circumstances under which our crawlers did not behave correctly. We have fixed that bug, and we are quite convinced it is now resolved. But - to be fully safe, we are leaving our crawlers inactive until we have done some further testing. I anticipate we will be restarting the crawlers within the next day or two. Please let me know if you see any other odd behaviour from our system.

Thank you for taking the time to let us know about this. We appreciate it. Thanks also for including your log information.

Wow! Very impressive. I wrote him back thanking him and apologing for my tone.

Posted by Marie at January 31, 2003 11:46 PM


I don't think your tone was too bad. The note was short, businesslike, and to the point. You may have wanted to apologize for the irritation you were feeling, but I don't think it carried over much into the note. After all, it's understandable that you would be irritated when a robot doesn't obey your directions.

Posted by: Laurabelle at February 3, 2003 10:14 AM