Huhu you say? | Main | When nothing else will do

Sunday, February 23, 2003

User agents and stuff

Someday I'm going to put this all together so I can not only understand it, but actually do something valuable with it. But for now, here are some links re various user agents, banning IPs, and such. Most of these, but not all, were found at WebMasterWorld on the Search Engine Spider Identification category. The links, in no particular order:

Search Engine IP Addresses
SpiderHunter.com
Search Engine Hunter
List of User Agents
Database of Web Robots
BotSpot
Robots, Spiders and Other UA's
Fleiner on how to keep out bad bots
Robot Cop
Open Directory: .htaccess
Comprehensive guide to .htaccess
Ralf D. Kloth & List of Bad Bots & How to build a Bot Trap

That should be enough for now.

Posted by Marie at February 23, 2003 2:02 PM