What Is an API? How Web APIs Work

Start here

What is an API?

An API — application programming interface — is a defined way for one program to ask another for something. A website is an interface for a person to read. An API is an interface for software to call. The same data sits behind both; only the packaging differs.

The short answer

An API is a contract: call this URL in this way, get this shape of data back.
Almost every web API is HTTP requests carrying JSON — a method, a path, some headers, sometimes a body.
The status code tells you whose problem it is: 4xx is yours, 5xx is theirs, 429 means slow down.
Rate limits are the constraint that shapes real integrations more than anything else.
The page you are scraping usually calls an endpoint of its own. Calling that endpoint directly is faster and far less fragile.

The restaurant analogy is overused but it is overused because it works. You do not walk into the kitchen and start opening pans. You read a menu of things you are allowed to order, ask in a format the waiter understands, and food comes back. The menu is the documentation, the waiter is the API, and the kitchen stays private — which is the entire point. The service can rewrite its database on Tuesday and, provided the menu is unchanged, your code never notices.

That stability is the real product. Anyone can return data. What an API sells is a promise that the shape of the response will not change without warning — which is precisely the promise a website makes to nobody, and why scraping HTML breaks on a Thursday afternoon when somebody ships a redesign.

Take it apart

Anatomy of a request

Underneath every SDK, every client library and every integration platform is this: a short block of text sent over HTTP. Learn to read it and nothing else about APIs is mysterious.

Every API call is this shape

Choose a method, then tap any coloured part of the request to see what it is for.

GET /v1/proxies/residential?country=de&limit=25 HTTP/1.1
Host: api.example.com
Authorization: Bearer sk_live_9f2c…
Accept: application/json
User-Agent: my-app/1.4.0

Method: The verb. What you want done to the resource.

GET: Read a resource. Changes nothing on the server, so it is safe to retry and safe to cache.

Learn to read a cURL command

Every browser can export a request as curl, and every API documentation page shows one. It is the lingua franca of HTTP debugging — if you can read a cURL command you can reproduce any request in any language. Ours is at the cURL builder.

Reading the reply

Methods and status codes

The method says what you want. The status code says what happened. Most integration bugs are somebody treating a status code as a generic failure instead of reading what it actually says.

Reading the response code

The first digit tells you whose problem it is. The rest tells you what to do about it.

429Too Many Requests

You have exceeded the rate limit.

What to do

Honour Retry-After, then back off exponentially. This is an instruction, not a failure.

401 versus 403

401 means the server does not know who you are — the credential is missing, malformed or expired. 403 means it knows exactly who you are and this identity is not permitted. One is fixed by refreshing a token, the other by changing a permission. Retrying either without changing something is wasted effort.

Which errors are worth retrying

Retry 429, 500, 502, 503 and 504 — all transient. Never retry 400, 401, 403 or 404 unchanged; nothing about the second attempt will differ from the first.

A 200 is not proof of success

Plenty of APIs return 200 with an error object in the body, particularly older ones and almost every GraphQL endpoint. Check the payload, not just the code, or you will happily store a page of error messages as if it were data.

The landscape

REST, GraphQL and the rest

Four architectural styles cover almost everything you will meet. They are not competitors so much as answers to different questions.

REST

Resources at predictable URLs, HTTP methods as verbs, JSON in and out.

Strong when: Everywhere. Simple to cache, simple to debug with nothing but curl, and every language has a client.

Costs you: Fixed response shapes mean you often fetch far more than you need, or make three calls to assemble one screen.

GraphQL

One endpoint. You send a query naming exactly the fields you want.

Strong when: No over-fetching and no waterfall of round trips. Excellent when a client needs deeply nested, varied data.

Costs you: HTTP caching largely stops working, and an unbounded query can be an accidental denial of service against your own server.

gRPC

Binary protocol over HTTP/2, with a schema compiled into real client code.

Strong when: Fast and strongly typed. The default for internal service-to-service traffic at scale.

Costs you: Not readable by a human, awkward from a browser, and you cannot poke at it with curl.

WebSocket

A connection that stays open, with messages flowing both ways.

Strong when: The right answer for live data: prices, chat, collaborative editing, progress streams.

Costs you: Stateful, so scaling and reconnection logic become your problem rather than HTTP’s.

Versioning is the part everyone regrets skipping

A public API is a promise you cannot quietly break. Most put a version in the path — /v1/ — so a breaking change ships as /v2/ and existing integrations keep working. If you are consuming an API with no version in it at all, assume it can change under you at any time.

Proving who you are

API authentication

Almost every API needs to know who is calling — to bill you, to rate limit you, and to decide what you may see. There are five common schemes and they differ mainly in how much damage a leak does.

Five ways to prove who you are

Roughly in order of how much they limit the damage when something leaks.

Authorization: Bearer eyJhbGciOi…

Sent inAuthorization header

LifetimeMinutes to hours

Scoped permissionsUsually

Revoked byExpires on its own

A short-lived credential issued after you authenticate, often a JWT carrying its own claims. Expiry is the point: a stolen token is useless within the hour. Pair it with a refresh token so the long-lived secret is used rarely and travels less.

Whatever the scheme, the rule is the same: secrets belong in headers and environment variables, never in a URL, a front-end bundle or a git history. See proxy authentication for how the same choices play out with proxies.

The single most common security mistake

Putting a key in a front-end bundle. Anything shipped to a browser is public — minification is not obfuscation and a devtools network tab reveals it in seconds. If a browser needs to reach a third-party API, proxy the call through your own backend and keep the secret on the server.

The real constraint

Rate limits and pagination

Nothing shapes a real integration more than these two. One caps how fast you may ask; the other caps how much arrives per answer.

Twelve requests, a limit of five

Same workload, two clients. One of them finishes; the other spends its afternoon collecting 429s.

···#1

···#2

···#3

···#4

···#5

···#6

···#7

···#8

···#9

···#10

···#11

···#12

Sent

0 / 12

Succeeded

Rate limited

Firing everything at once wastes more than half the requests. Worse, many APIs count rejected calls against your quota, and repeated offences escalate from a temporary limit to a suspended key.

Real limiters usually work as a token bucket: tokens refill at a fixed rate and a burst is allowed until the bucket empties. Check the X-RateLimit-Remaining header rather than guessing.

Offset pagination

?limit=50&offset=100

Simple, and lets you jump to any page. It also gets slower the deeper you go, and if records are inserted while you paginate you will see duplicates or miss rows entirely.

Cursor pagination

?limit=50&after=eyJpZCI6MTAwfQ

The response hands you an opaque pointer to resume from. Consistent under concurrent writes and fast at any depth — at the cost of only ever moving forwards. This is what every large API converges on.

When a proxy genuinely helps — and when it does not

If a limit is per API key, more IP addresses change nothing and you are simply adding latency. If it is enforced per source IP, or the data varies by country, or the provider blocks datacenter ranges outright, then routing through proxies is the right tool. Work out which limit you are actually hitting before you pay for one.

The useful trick

The API behind a website

Most modern pages load an empty shell and then fetch their content as JSON. That fetch is an API call — one you can usually make yourself, skipping the rendering entirely.

Two routes to the same data

The page you are looking at usually fetched its content from an endpoint. You can often call that endpoint directly.

Render then parse — slow, heavy, fragile
Call the endpoint directly — fast and structured

Requests per record

HTML: 1 page + assets

API: 1 call

Breaks when

HTML: The design changes

API: The endpoint changes

Typical cost

HTML: Browser CPU + bandwidth

API: A few KB of JSON

Covered in more depth in the web scraping lesson and API scraping.

Open the network tab

In your browser devtools, filter to Fetch/XHR and reload the page. You are looking at the exact requests the page makes to populate itself.

Find the one carrying the data

Sort by size and look for a JSON response containing the values you can see on screen. That is the endpoint doing the real work.

Read what it needs

Note the method, the query parameters, and which headers matter — usually an accept header, sometimes a token, occasionally a signature the page computed in JavaScript.

Reproduce it

Right-click, copy as cURL, and run it. If it returns the same JSON outside the browser, you have replaced a rendering pipeline with a single request.

Be honest about the trade-off

An undocumented endpoint carries no promise of any kind. It can change shape, require a new signature, or disappear in a routine deploy, and nobody will tell you. It is genuinely the better technical route when it works — one request instead of a rendered page, structured data instead of parsed markup — but treat it as something that will break, and build the monitoring to notice when it does.

Whether you may use it at all is a separate question, governed by the site’s terms and your jurisdiction rather than by what is technically possible. Where a site publishes an official API, use it: it is stable, it is supported, and it removes the question entirely.

The call in reverse

Webhooks

Everything so far has you asking. A webhook flips it: you hand over a URL, and the service calls you when something happens.

Polling: asking again and again

Same information, wildly different cost.

Checking once a minute means 1,440 calls to discover three changes. It burns your rate limit, adds up to a minute of delay, and the API provider pays for almost entirely wasted traffic.

Always verify the signature header on an incoming webhook. An unauthenticated public URL that mutates your data is an open invitation — see webhook.

Verify every delivery

Your endpoint is public, so anyone can post to it. Providers sign each payload with a shared secret — check that signature before you trust a single field.

Respond fast, process later

Return 200 immediately and push the work onto a queue. Providers time out in seconds, and a slow handler turns into a flood of retries.

Expect duplicates

Delivery is at-least-once, not exactly-once. Key your handler on the event id so processing the same event twice changes nothing.

Practical advice

Building against an API without regretting it

Six habits that separate an integration that runs for years from one you rewrite every quarter.

Read the docs before the first call

Ten minutes with the reference usually reveals a bulk endpoint, a filter parameter, or a sandbox environment that saves hours of hammering the wrong route.

Retry with backoff and jitter

Exponential backoff stops you making an outage worse. Jitter — a small random offset — stops every one of your workers retrying in perfect unison.

Handle partial failure

At any scale, some calls fail. Decide in advance whether a batch is all-or-nothing or best-effort, and make failures visible rather than silently skipped.

Cache what does not change

Reference data, currency lists, category trees. An ETag and a conditional request turn most of these into a free 304.

Set a timeout on every request

A client with no timeout does not fail — it hangs, holds a connection, and takes your worker pool down with it.

Log the request id

Most APIs return one in a header. It is the only thing that makes a support conversation about a specific failed call productive.

Start in a terminal, not in a framework

Before you install a client library, make the call by hand with curl. You will understand the auth, see the real error bodies, and know exactly what the SDK is doing on your behalf. Every hour spent there saves several later.

Check your understanding

Test yourself

Five questions. Nothing is recorded — this is just for you.

Quick knowledge check0 / 5

1Which HTTP status means you have hit a rate limit?

2Which of these methods is NOT expected to be idempotent?

3Where should an API key normally be sent?

4A 401 response tells you…

5Compared with polling, a webhook…

Common questions

API FAQ

1What is an API in simple terms?

An API is a defined way for one program to ask another program for something. A website is an interface built for a human to read; an API is an interface built for software to call. Same data underneath, delivered in a form a machine can parse without guessing.

2What is the difference between an API and a website?

A website returns HTML — layout, styling and content tangled together, designed to be looked at. An API returns structured data, usually JSON, designed to be read by code. The website can be redesigned tomorrow and break every scraper; a versioned API is a promise that the shape of the response will stay stable.

3What does REST actually mean?

REST is a set of conventions rather than a standard: resources live at predictable URLs, HTTP methods say what you want done to them, and each request carries everything needed to serve it. In practice most APIs described as RESTful only follow some of the original constraints, and that is fine — the useful part is the shared vocabulary.

4What is the difference between REST and GraphQL?

With REST you call several fixed endpoints and take whatever fields each returns. With GraphQL you send one query describing exactly the fields you want and get precisely those back. GraphQL removes over-fetching and multiple round trips; it costs you cacheability and a good deal of server complexity.

5What does HTTP 429 mean?

Too Many Requests — you have exceeded the rate limit. It is not an error in your code and retrying immediately makes it worse. Read the Retry-After header if one is present, wait, and back off exponentially. A well-behaved client treats 429 as instruction, not failure.

6What is an API key and how is it different from a token?

An API key is a long-lived secret that identifies your application, usually sent in a header. A bearer token is typically short-lived, issued after authenticating, and scoped to specific permissions. Keys are simpler and more dangerous if leaked; tokens expire, which limits the damage.

7Is using a website’s hidden API legal?

It depends on the site’s terms, your jurisdiction and what you do with the data — and it is a question for a lawyer, not a guide. Technically these endpoints are the same ones the page itself calls. Practically, undocumented endpoints carry no stability promise and can change without notice, so anything built on one is fragile by nature.

8What is a webhook?

A webhook inverts the direction of the call. Instead of you asking every minute whether anything changed, the service sends an HTTP request to a URL you provide the moment something happens. It is dramatically more efficient than polling, at the cost of needing a publicly reachable endpoint you can secure.

9Why would an API call need a proxy?

Three common reasons: the endpoint rate-limits by source IP and you need more throughput than one address allows; the data is geographically different and you need to appear in a specific country; or the provider blocks datacenter ranges outright and your server sits in one. If none of these apply, a proxy adds latency and cost for nothing.

10What is idempotency and why does it matter?

An idempotent request produces the same result whether it runs once or five times. GET, PUT and DELETE are meant to be idempotent; POST is not. It matters because networks fail mid-request — if you cannot tell whether a payment went through, you need either an idempotent operation or an idempotency key to retry safely.

What Is an API? HTTP Requests, Auth and Rate Limits Explained