> For the complete documentation index, see [llms.txt](https://liuzhenglaichn.gitbook.io/system-design/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://liuzhenglaichn.gitbook.io/system-design/web-crawler.md).

# Web Crawler

- [Design a web crawler](https://liuzhenglaichn.gitbook.io/system-design/web-crawler/design-a-web-crawler.md)
- [Design a decentralized web crawler](https://liuzhenglaichn.gitbook.io/system-design/web-crawler/design-a-decentralized-web-crawler.md)