The Scrapy Framework

When a scraper grows past a single script, Scrapy provides the structure — pipelines, middleware, built-in concurrency and retries — that hand-rolled code eventually reinvents anyway.