> For the complete documentation index, see [llms.txt](https://liuzhenglaichn.gitbook.io/system-design/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://liuzhenglaichn.gitbook.io/system-design/web-crawler/design-a-web-crawler.md).

# Design a web crawler

## Reference

* <https://github.com/donnemartin/system-design-primer/blob/master/solutions/system_design/web_crawler/README.md>

## Usecases

* Search engine
* Copyright violation detection
* keyword based finding
* web walware detection
* web analytics
* datascience data

## Things to consider

* Politeness/crawl rate
* DNS query
* Distributed crawling
* Priority crawling
* Duplicate detection

## Questions

* How to generate content signature?
* what to do with similar content?