IT Directory, Information Technology Web Directory, IT Info and Service - Communications : Internet : Crawlers, Robots, & Spiders

Categories

•	Business

•	Communications

•	Configuration Management

•	Consultants

•	Databases

•	Desktop Publishing & Presentation

•	Directories

•	Education and Training

•	Games and Emulators

•	General Software

•	News and Information

•	Operating Systems

•	Retailers

•	Shareware & Freeware

•	Software Engineering

•	System Management

•	Translation & Multilingual Support

•	Word Processors & Office Suites

Crawlers, Robots, & Spiders
Home : Communications : Internet : Crawlers, Robots, & Spiders

BotKnowledge.com
Directory of intelligent software agents, knowbots, and bots. Includes FAQ and newsletter.
http://www.botknowledge.com/

Collegebot
Dedicated to indexing and searching education and academic related pages.
http://www.collegebot.com/

Grub
Open source, distributed Internet crawler.
http://www.grub.org/

MegaSpider
Searches search engines.
http://www.MegaSpider.com/

Searchbots
Offers bots with different search programs, or allows users to customize their own.
http://www.searchbots.net/

Spider Cloak
Offers search engine cloaking script that hides the source code to prevent theft and generates optimized marketing pages for submission.
http://www.spidercloak.com/

SpiderHunter.com
Demonstrates how to write cloaking scripts and track spiders.
http://www.spiderhunter.com/

Webglimpse
Search engine software that consists of Glimpse, an engine for indexing and searching text files on a Unix system and Webglimpse, the spider and manager for the files to be searched.
http://www.webglimpse.net/