Rate Limiting
Rate limiting restricts how many requests a client can make in a given time, and it is one of the most common defenses scrapers must work around.
Definition
Rate limiting is a server-side control that caps how many requests a single client — usually identified by IP or API key — may make within a time window. Exceeding the limit returns errors like HTTP 429 (Too Many Requests) or triggers temporary blocks.
Working with rate limits
Scrapers respect or circumvent rate limits by pacing requests, adding delays, and distributing traffic across many IPs via rotating proxies. Respecting reasonable limits is also good etiquette and reduces the chance of a hard ban.
Examples
An API returning HTTP 429 after 100 requests per minute
Adding randomized delays between scraper requests to stay under a limit
Common Use Cases
Frequently Asked Questions
Keep Learning
All termsRotating Proxy
A rotating proxy automatically assigns a different IP address from a pool for each request or on a set interval, spreading traffic across many IPs to avoid blocks.
Read definitionWeb Scraping
Web scraping is the automated extraction of data from websites — fetching pages programmatically and parsing their content into structured data.
Read definitionCAPTCHA
A CAPTCHA is a challenge–response test used to tell humans and bots apart, such as identifying images or checking a box, to block automated access.
Read definition