What Is Web Scraping
What web scraping actually is, why it exists, and the legal and ethical lines you need to know before writing a single line of code.
The problem scraping solves
The web has more useful data than any API will ever expose. Prices, listings, reviews, schedules, public filings — most of it sits inside HTML pages meant for humans to read in a browser, not for programs to consume. Web scraping is the practice of writing a program that fetches those pages and pulls structured data out of them automatically.
The core idea is simple: a webpage is just text (HTML) sent over HTTP. If you can fetch that text and understand its structure, you can extract anything a human could read off the screen — at a scale no human could match.
Website (HTML) → Fetch → Parse → Extract → Structured data (CSV, JSON, database)
Everything in this course is a variation on that one pipeline. The difference between a five-line script and a production crawler is how much each of those four steps has to handle — pages that need a browser to render, sites that actively try to block you, or a target that returns ten million rows instead of ten.
Why "traditional" scraping still matters
Modern tools like Firecrawl or LLM-based extractors can point at a URL and hand back clean JSON with almost no setup. It's tempting to skip straight to those and never learn what's underneath. That's a mistake for three reasons:
- Cost. Managed scraping APIs charge per page. A script using plain HTTP requests and a parser costs you a few milliseconds of CPU time. At scale, that difference is the entire budget.
- Control. When a managed tool gets something wrong — misses a field, times out, gets blocked — you need to understand what it's doing underneath to debug it, or replace it.
- It's most of the job. The vast majority of scraping targets are plain server-rendered HTML with no anti-bot protection. Reaching for a browser or an AI extractor for a page that a two-line HTTP request could handle is solving an easy problem with an expensive tool.
This course spends most of its time on the traditional approach — HTTP requests, HTML parsing, sessions, pagination — because that foundation is what tells you when you actually need something heavier, and it's what everything heavier is built on top of. Modern AI-driven tools get exactly one module, at the end, once you know what problem they're actually solving.
What scraping is used for
- Price monitoring — tracking competitor prices or your own listings across marketplaces.
- Aggregation — job boards, real estate listings, flight search engines that pull from many sources into one place.
- Research — academics and journalists collecting public data at a scale manual collection can't reach.
- Lead generation — pulling public contact or company information from directories.
- Monitoring — watching a page for changes (stock availability, news updates, status pages).
- Training data — collecting text or structured data to train or fine-tune models.
Notice what all of these have in common: they operate on data the site already makes publicly visible. That distinction matters for the next section.
The legal and ethical line
Scraping is a technique, not a crime, but how you use it can cross into legal and ethical trouble. Three separate things are worth telling apart, because people conflate them constantly:
robots.txt is a file athttps://example.com/robots.txt that tells automated crawlers which paths the site owner would prefer aren't crawled. It is a convention, not a technical barrier — nothing stops your code from ignoring it, but doing so is generally considered bad practice and can be used as evidence of bad faith if there's ever a legal dispute.
User-agent: *
Disallow: /admin/
Disallow: /search
Crawl-delay: 2
Terms of Service are a legal agreement between the site and its users. Many sites explicitly prohibit automated access in their ToS. Violating ToS is a contract issue, not automatically a criminal one, but it can get your account or IP banned and, in some jurisdictions, has been the basis for civil suits (the hiQ v. LinkedIn case in the US is the standard reference — it went back and forth for years and the exact legal boundary is still debated).
Data protection law (GDPR in the EU, CCPA in California, similar laws elsewhere) governs personal data specifically. Scraping publicly visible names, emails or profile data can still violate these laws if you store, process or resell that data — the fact that it was "public" on the page does not automatically make bulk collection and reuse of it legal.
A practical checklist before scraping anything:
- Check
robots.txtfor the path you intend to hit, and respectDisallowandCrawl-delay. - Read the ToS if the data is commercially sensitive or you plan to publish/resell what you collect.
- Prefer an official API if one exists — it's faster, more stable, and removes the legal ambiguity entirely.
- Never scrape personal data (emails, phone numbers, private profiles) without a clear, lawful reason.
- Rate-limit yourself. Hammering a site with requests can degrade it for real users — this is the one rule that's both a legal and an engineering concern.
robots.txt), and don't take more than you need, faster than you need it.
How this course is organised
- Fundamentals (this module) — how the web actually works, so the rest of the course has something to stand on.
- Traditional scraping — HTTP requests, HTML parsing, selectors, sessions and pagination. This is the bulk of the course, and the skillset that solves most real scraping problems.
- Dynamic sites and hidden APIs — recognizing when a page needs JavaScript to render, finding the JSON API behind it, and how sites detect and block scrapers.
- Browser automation frameworks — Selenium, Playwright and Puppeteer in depth, for the cases that genuinely need a real browser.
- CAPTCHAs and boundaries — how CAPTCHAs actually work, a concrete line between what's reasonable to scrape and what isn't, and what remains genuinely hard even with modern tools.
- Scaling and production — concurrency, the Scrapy framework, and running a scraper as a long-lived system rather than a one-off script.
- Build your first scraper — the same practice project built four ways (requests, Selenium, Playwright, Scrapy), measured and compared side by side.
- Modern AI scraping — Firecrawl,
crawl4ai, ScrapeGraphAI and LLM-based extraction: what problem they solve, and when reaching for them beats writing a parser by hand.
Before you start, if you want some good ai scraping tutorials, subscribe to Remi Taffin's youtube channel. One of the best guy in the internet who explains scraping. (not a sponsored message)
By the end you should be able to look at almost any webpage and know exactly which layer of this course you need — and, just as importantly, which layers you can skip.