Web crawler
Overview
World Wide Web
The World Wide Web is a system of interlinked hypertext documents accessed via the Internet...
in a methodical, automated manner or in an orderly fashion. Other terms for Web crawlers are ants, automatic indexers, bots, Web spiders, Web robots, or—especially in the FOAF
FOAF (software)
FOAF is a machine-readable ontology describing persons, their activities and their relations to other people and objects. Anyone can use FOAF to describe him or herself...
community—Web scutters.
This process is called Web crawling or spidering. Many sites, in particular search engines
Web search engine
A web search engine is designed to search for information on the World Wide Web and FTP servers. The search results are generally presented in a list of results often referred to as SERPS, or "search engine results pages". The information may consist of web pages, images, information and other...
, use spidering as a means of providing up-to-date data.
