4 lessons
Making HTTP Requests
Fetch a page with plain HTTP calls, understand what headers actually matter, and handle the failures that happen constantly in real scraping.
Parsing HTML with BeautifulSoup
Turn raw HTML text into structured data using CSS selectors, XPath and tree navigation — the step that actually extracts what you came for.
Sessions, Cookies and Logging In
How cookies keep you logged in, how to submit forms programmatically, and how to scrape pages that sit behind authentication.
Pagination and Crawling Multiple Pages
Move from scraping one page to scraping an entire site — following pagination, discovering links, and not scraping the same page twice.