Design Web Crawler
Overview– A web crawler is a program that automatically traverses the web by downloading web pages and following links from one page to another. It is used to index the web for search engines, collect data for research, or monitor websites for changes. Problem Definition– Design a web crawler to extract text data from the web and store it to train an LLM. Your crawler can run for only 5 days. Functional Requirements– Crawl the full web starting from seed urls Extract text data and store Scale– 10B web pages 2MB per page avg. 5 days to scrape Unlimited resources (withing reason) Non-Functional Requirements– Fault tolerant Politeness Scale to 10B pages Efficiently crawl in under 5 days Core Entities— Text Data URL metadata Domain metadata Interface– input – set of seed urls Output – text data Data Flow– Take seed urls from a frontier and the IP from DNS Fetch HTML Extract text from HTML Store that text in database Extract the urls in the text and add to our frontier Repeat steps 1-5 until all urls have been crawled High-Level Design– Efficiency– Don’t crawl a URL that has already been added. Don’t parse a webpage with duplicate content that has already been parsed.
