Source: 13_WebCraw_Part-I.pdf
A Web crawler (spider, robot, bot) is software that downloads pages from the Web. The basic cycle: start from seed pages, parse them for new links, add undownloaded links to a central queue, pick the next page, and repeat until a stop criterion is met. Crawling began with Matthew Gray’s World Wide Web Wanderer (1993) and Brian Pinkerton’s WebCrawler (1994), and now underpins every major search engine. Crawlers serve many applications: general web search (balancing coverage and quality), vertical search (limited by topic/country/language/format — shopbots, news crawlers, feed crawlers), focused/topical crawling (driven by a query or example docs, exploiting topical locality), web characterization (statistics), mirroring (keeping copies), web archiving (keeping history, e.g., the Internet Archive), and site analysis (link/code validation, vulnerability detection).
Crawlers are classified along three axes — freshness, quality, volume — and must observe politeness: never overload a site, waiting a delay between requests. The architecture has three modules: the scheduler (maintains the URL queue; split into long-term scheduling = which pages to visit, and short-term = re-arranging for politeness via per-site queues), the downloader (fetches pages), and storage (indexes pages and provides metadata; subdivided into text, metadata, links).
Implementation faces many practical issues: keeping traffic uniform despite variable DNS/server response times; the taxonomy of web pages (public/private, static/dynamic — infinitely many dynamic pages, so crawlers cap depth); wrongly-coded HTML (parsers must tolerate errors; soft-404 error pages account for ~29% of dead links); duplicates (intentional mirrors and unintentional ones from embedded session-ids that must be normalized); and granularity (blogs/forums — sometimes only aggregating pages should be indexed). For scalability and fault tolerance, crawling is done in parallel and distributed, where the key challenge is avoiding downloading the same page twice; an assignment function decides which process handles each URL and must satisfy balancing, contra-variance (more processes → fewer hosts each), and dynamic add/remove of processes.
| Term | Meaning | Use Case |
|---|---|---|
| Web crawler (spider/bot) | Software downloading web pages | Building a search corpus |
| Seed pages | Starting URLs for a crawl | Bootstrapping crawling |
| Coverage vs. quality | Scan many pages vs. high-value pages | General web search balance |
| Vertical crawler | Crawls a topic/country/language/format subset | Shopbots, news, feeds |
| Focused/topical crawler | Crawls pages about a given topic | Efficient niche crawling |
| Topical locality | Linked pages tend to share a topic | Guiding focused crawls |
| Web characterization | Deriving statistics about the Web | Research, sampling |
| Mirroring / archiving | Copying / keeping history of sites | Internet Archive |
| Politeness | Not overloading servers; delay between requests | Responsible crawling |
| Scheduler / downloader / storage | The three crawler modules | Crawler architecture |
| Long/short-term scheduling | What to visit / politeness re-arranging | Queue management |
| Soft-404 | Error page returned with a 200 status | Detecting dead links |
| Session-id duplicates | Same page via URLs with tracking ids | Deduplication |
| Assignment function | Maps URLs to crawling processes | Distributed crawling |
| Balancing / contra-variance | Equal load / shrink with more processes | Assignment properties |
flowchart TD
A[Seed Pages] --> B[Downloader
fetch pages]
B --> C[Parse & extract links]
C --> D{New URL?}
D -- Yes --> E[Add to Scheduler queue]
D -- No --> F[Discard duplicate]
E --> G[Long-term scheduling
which page next]
G --> H[Short-term scheduling
per-site politeness queues]
H --> B
B --> I[Storage
text + metadata + links]
I --> J[Index / characterization / archive]
B --> K{Stop criterion met?}
K -- No --> G
K -- Yes --> L[End crawl]