What is web scraping?
Web scraping is the automated extraction of data from websites by software that downloads pages and pulls out the wanted content, such as prices, product specifications, or public filings. A web crawler is the related program that discovers pages by following links. Search engines run crawlers at internet scale, and the same technique assembles training data for many large language models. A scraper usually targets known pages with a known layout, while a crawler explores.
Scraped pages arrive as HTML that must be parsed into structured data. Language models have made that step easier, since a model can extract the fields from a page without a hand-written rule for each layout.
Web scraping in practice
An electrical distributor monitors manufacturer and competitor sites for list price changes and stock status on a few thousand SKUs and loads the results into its pricing warehouse nightly. A collections firm pulls public court docket updates for accounts in litigation. A local government crawls its own website to find broken links and outdated forms before residents do.
What to watch for
- Read the site’s terms of service and robots.txt file; violating them creates legal and reputational exposure.
- Expect breakage, because a site redesign silently changes the layout and the scraper starts returning wrong fields.
- Validate scraped content, which is unstructured data of unknown quality, before it feeds a pricing decision or a training set.
- Do not collect personal data from the web without a lawful basis; GDPR and similar laws require one even for public data.