API & automation

Bot registry

The public, versioned list of AI crawlers Ranksify recognises. It is the same list the product classifies your server logs against, projected straight out of the code that does the matching — so the published registry cannot drift from what the app actually detects.

The live registry

The table below is fetched from /api/v1/bots.json in your browser when this page loads. It is not a copy kept in sync by hand; it is the endpoint’s own response.

Loading the live registry…

UA token is the lowercase token we look for inside a request’s User-Agent header. Match it case-insensitively with token boundaries (letters, digits, underscores and hyphens are token characters) — never as a bare substring or against the full UA string.

Recognised crawlers, at the time this page was built: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Claude-Web, Claude-User, anthropic-ai, PerplexityBot, Perplexity-User, Google-Agent, Bytespider, CCBot, Applebot, cohere-ai, Meta-ExternalAgent, Claude-Code, Cursor, Devin, opencode. The table above is the endpoint’s live answer and wins if the two ever differ.

Fetching it

Public, unauthenticated, GET only, and CORS-open (access-control-allow-origin: *), so you can call it from a browser, a build step, or an edge function. There are no request parameters.

fetch("https://app.ranksify.ai/api/v1/bots.json")
  .then((res) => res.json())
  .then(({ bots }) => bots.map((bot) => bot.token));

The response is a plain document, not the {"data":…} envelope the authed REST API uses — it is a public asset, like the OpenAPI document. Example response, abridged to one bot and one platform:

{
  "schemaVersion": 1,
  "source": "https://ranksify.ai/docs/bot-registry",
  "count": 16,
  "bots": [
    { "name": "GPTBot", "platform": "openai", "platformLabel": "OpenAI / ChatGPT", "token": "gptbot" }
  ],
  "platforms": { "openai": "OpenAI / ChatGPT" }
}

• bots[] — one entry per crawler: name as the vendor writes it, platform as a stable machine key, platformLabel as the human-readable name, token as the bounded UA token to match.
• count — the number of entries in bots. Read it rather than hardcoding a length.
• platforms — every platform key mapped to its label. This map is deliberately larger than the set of platforms in bots: it also carries referrer-only platforms, so labels resolve for every value the API can hand you.

Versioning

• The major version lives in the path (/api/v1/). schemaVersion names the document shape.
• Additive changes — new bots, new platforms, new fields — are non-breaking and do not bump schemaVersion. Renaming or removing a field bumps it.
• Consumers must match on token case-insensitively and ignore unknown fields.

Caching

The endpoint is served with cache-control: public, max-age=3600. Honour it — an hour is the refresh cadence the registry is designed around, and polling it faster tells you nothing new.

The middleware package follows exactly that pattern: it classifies synchronously from a bundled copy of this list, refreshes in the background once an hour, and keeps the last good copy if a refresh fails. Copy the shape if you build your own consumer — never block a request on fetching the registry.