ProxyHorizon Glossary
The Proxy & Networking Glossary
Plain-language definitions of the proxy, VPN, scraping and networking terms you'll run into — explained simply, with examples and use cases.
88 terms and counting
Loading…
Plain-language definitions of the proxy, VPN, scraping and networking terms you'll run into — explained simply, with examples and use cases.
88 terms and counting
Cookie management is the practice of storing, sending, and rotating browser cookies so a scraper can maintain logins and sessions across requests. Done well, it keeps automated traffic looking consistent and human.
Crawl budget is the number of pages a search engine or crawler will fetch from a site within a given time. It limits how much of a large website actually gets crawled and indexed.
Crawl rate is how fast a crawler sends requests to a website, usually measured in requests per second. Keeping it reasonable avoids overloading servers and triggering blocks.
Rate limiting restricts how many requests a client can make in a given time, and it is one of the most common defenses scrapers must work around.
Request throttling is deliberately slowing down how often a scraper sends requests so it stays under a site's limits. It helps avoid bans, server overload, and detection.
Robots.txt is a simple text file at the root of a website that tells crawlers and bots which pages they may or may not access. It is a voluntary set of rules, not a technical lock.
A rotating proxy automatically assigns a different IP address from a pool for each request or on a set interval, spreading traffic across many IPs to avoid blocks.
A session cookie is a temporary cookie a website uses to remember you during a single visit, such as keeping you logged in. It is deleted when you close the browser or the session ends.
A sticky session keeps the same proxy IP for a set period, so multi-step workflows like logging in and checking out stay on one consistent address.