How the Web Actually Works
HTTP requests and responses, HTML and the DOM, and the difference between server-rendered and client-rendered pages — the mental model every scraper depends on.
Everything is a request and a response
When you type a URL into a browser, one thing happens: your browser sends an HTTP request to a server, and the server sends back an HTTP response. That's the entire protocol. A scraper does exactly the same thing, minus the browser drawing pixels on screen.
Your program → HTTP request → example.com's server
Your program ← HTTP response ← example.com's server
A request has three parts worth knowing cold:
- Method — usually
GET(fetch a page) orPOST(submit data, like a login form). - Headers — metadata about the request: what browser claims to be sending it (
User-Agent), what content types it accepts, cookies, authentication tokens. - Body — data sent with the request, present on
POST/PUTbut empty on a plainGET.
A response mirrors that shape:
- Status code — a three-digit number summarising what happened.
- Headers — metadata about the response: content type, cookies to set, caching rules.
- Body — the actual content, usually HTML, JSON, or an image.
Status codes worth memorising
| Range | Meaning | Example |
|---|---|---|
| 2xx | Success | 200 OK — request succeeded, 201 Created — resource successfully created |
| 3xx | Redirect | 301 Moved Permanently, 302 Found |
| 4xx | Client error | 403 Forbidden, 404 Not Found, 429 Too Many Requests |
| 5xx | Server error | 500 Internal Server Error, 503 Service Unavailable |
403 and 429 are the two you'll see most as a scraper — they mean the server identified your request as automated and refused it or asked you to slow down. Handling those gracefully is most of what "not getting blocked" actually means, and it's covered properly in the anti-bot module later in this course.
HTML is just structured text
The response body for a normal webpage is HTML — plain text with a specific structure. Right-click any page and choose "View Page Source" to see exactly what your scraper will receive, before any JavaScript runs.
<!DOCTYPE html>
<html>
<head>
<title>Example Product Page</title>
</head>
<body>
<div class="product">
<h1 class="product-title">Wireless Headphones</h1>
<span class="price" data-currency="USD">49.99</span>
<ul class="reviews">
<li>Great sound quality</li>
<li>Battery lasts all day</li>
</ul>
</div>
</body>
</html>
<div>, <span>, <h1>), and tags can carry attributes (class, id, data-currency) that give scrapers something reliable to target. The entire job of the parsing module later in this course is: given this text, get 49.99 out of it programmatically, and don't break the next time the page's styling changes.
The DOM: HTML after the browser is done with it
The DOM (Document Object Model) is the tree structure a browser builds after parsing HTML — every tag becomes a node with parent/child/sibling relationships.
html
└── body
└── div.product
├── h1.product-title
├── span.price
└── ul.reviews
├── li
└── li
.product .price reads as "find an element with class price that's inside an element with class product" — a direct description of a path through this tree. Understanding this structure is 80% of what makes HTML parsing feel intuitive instead of arbitrary.
Browser developer tools (F12, or right-click → Inspect) show you the live DOM, which can differ from the raw HTML source when JavaScript has modified the page after load. That distinction is the single most common source of confusion for people learning to scrape, and it deserves its own section.
Server-rendered vs. client-rendered pages
This is the most important concept in this entire module, because it determines which half of this course you actually need for a given site.
Server-rendered (SSR) / static HTML — the server builds the complete HTML, including all visible content, and sends it in the initial response. "View Source" shows you everything you'd see in the browser. A plain HTTP request gets you the full page. This is how the traditional web worked for two decades, and it's still how a huge share of the web works today — blogs, documentation, most e-commerce category pages, government sites, most content that doesn't feel like an "app."
Client-rendered (CSR) — the server sends a nearly empty HTML shell plus a bundle of JavaScript. The browser runs that JavaScript, which then fetches data (usually from an API, as JSON) and builds the visible page dynamically. "View Source" shows you a mostly empty<div id="root"></div> — none of the actual content is there. A plain HTTP request to this kind of page gets you nothing useful.
<!-- What a CSR page's raw HTML source actually looks like -->
<!DOCTYPE html>
<html>
<body>
<div id="root"></div>
<script src="/static/bundle.js"></script>
</body>
</html>
You can tell them apart in seconds: view the page's source (not the DevTools inspector — the raw source), and search for the text you see rendered in the browser. If it's there, it's server-rendered and a simple HTTP request will work. If it's missing, the page is built by JavaScript, and you need either a headless browser (module 3) or to find the underlying API the JavaScript itself is calling (also module 3, and usually the better answer).
| Server-rendered | Client-rendered | |
|---|---|---|
| Content in raw HTML | Yes | No (built by JS after load) |
| A plain HTTP request works | Yes | No |
| Typical stack | PHP, Django, Rails, static site generators | React, Vue, Angular SPAs |
| Scraping approach | HTTP request + HTML parser | Headless browser, or find the underlying API |
| Relative cost | Cheap | Expensive |
Most real-world scraping targets you'll encounter are still server-rendered, or are a hybrid — server-rendered for the first page load, with JavaScript only handling interactive bits like "load more" buttons. Checking which category a site falls into, before writing any code, is the single highest-leverage five minutes you'll spend on any scraping task.
What's actually different about a scraper vs. a browser
A browser does far more than fetch HTML: it runs JavaScript, applies CSS, loads images, manages a cookie jar, and renders pixels. A basic HTTP-based scraper does exactly one of those things — fetch the raw response — which is why it's so much faster and cheaper, and also why it fails completely on client-rendered pages. Every tool in this course is really just deciding how much of "being a browser" your scraper needs to fake in order to get the data.