Sitemap parser API

From $0.002 per successful call · failed calls are never charged · no account needed · Markdown version: /guides/sitemap-parser-api.md

Sitemap parser: reads one public XML sitemap or sitemap index as structured URLs and source-reported dates. Supports gzip, substring filtering and pagination. Large sitemaps are read from the start (512 KiB, 1000 entries at most, with a warning). Does not crawl pages or follow child sitemaps. Invalid XML and execution errors are not charged; a valid empty match is charged. Use to find which pages of a docs site to read before calling /extract.

When an agent wants this

Sitemap parser, URL extractor: documentation URLs from a bounded XML sitemap $0.002

Sitemap parser: reads one public XML sitemap or sitemap index as structured URLs and source-reported dates. Supports gzip, substring filtering and pagination. Large sitemaps are read from the start (512 KiB, 1000 entries at most, with a warning). Does not crawl pages or follow child sitemaps. Invalid XML and execution errors are not charged; a valid empty match is charged. Use to find which pages of a docs site to read before calling /extract.

Inputs: url (required): Absolute public HTTP(S) XML sitemap URL; child sitemaps are not fetched automatically; contains: Optional case-insensitive URL substring, at most 200 characters; maxUrls: Reported matching locations examined in one page (1-100, default 25); invalid/nonpublic locations may reduce returned entries; offset: Offset into distinct reported matching locations (0-1000, default 0); use nextOffset to continue.

Free first (25 successful calls per caller, no key):

curl "https://tools.funwords.eu/sitemap?url=https%3A%2F%2Fwww.sitemaps.org%2Fsitemap.xml&trial=1"

With a card key from /credits, in a header or, when your client can only fetch a URL, in the URL:

curl -H "Authorization: Bearer adt_…" "https://tools.funwords.eu/sitemap?url=https%3A%2F%2Fwww.sitemaps.org%2Fsitemap.xml"
curl "https://tools.funwords.eu/sitemap?url=https%3A%2F%2Fwww.sitemaps.org%2Fsitemap.xml&api_key=adt_…"

Without a key the same URL answers 402 Payment Required with x402 and MPP terms, so a wallet client pays USDC per call and retries. Over MCP (https://tools.funwords.eu/mcp) this tool is sitemap.

Example result
{
  "url": "https://agent-data-tools.revmesh2074.workers.dev/examples/sitemap/source.xml",
  "finalUrl": "https://agent-data-tools.revmesh2074.workers.dev/examples/sitemap/source.xml",
  "kind": "urlset",
  "entries": [
    {
      "url": "https://agent-data-tools.revmesh2074.workers.dev/examples/extract/sources/technical-guide.html",
      "lastModified": "2026-10-15"
    },
    {
      "url": "https://agent-data-tools.revmesh2074.workers.dev/examples/extract/sources/short-article.html",
      "lastModified": "2026-10-15"
    }
  ],
  "reportedEntries": 3,
  "uniqueReportedLocations": 3,
  "missingOrInvalidLocations": 0,
  "duplicateReportedLocations": 0,
  "rejectedLocationsInPage": 0,
  "matchingReportedLocations": 3,
  "contains": "",
  "offset": 0,
  "nextOffset": 2,
  "hasMore": true,
  "warnings": [
    "counts_describe_reported_locations_before_public_url_validation",
    "locations_and_dates_are_source_reported_not_probed"
  ],
  "text": "https://agent-data-tools.revmesh2074.workers.dev/examples/extract/sources/technical-guide.html\nhttps://agent-data-tools.revmesh2074.workers.dev/examples/extract/sources/short-article.html",
  "format": "text",
  "hashFormat": "text",
  "contentHash": "sha256:8055b8b5ced205885a181b4fa7145008ac8d5dc3fb3286007c4cf679f886b750",
  "sourceHash": "sha256:c24148c4774690472856c116012988bb52e992c6a3bbba156947be1dac6eba03",
  "fetchedAt": "2026-10-15T21:48:20.943Z"
}

Tool page, inputs and limits

Robots.txt check: may this crawler fetch this URL? $0.01

Read a site's robots.txt and say whether a crawler may fetch a URL, by RFC 9309 rules as Google applies them (most specific user-agent group, longest matching Allow or Disallow, Allow on ties, * and $ wildcards). Returns the deciding rule, the group used, the rule count and the sitemaps listed. A missing robots.txt (4xx) allows everything. Use before scraping or crawling a site politely; for the sitemap URLs themselves use sitemap.

Inputs: url (required): The page URL to check (its site's /robots.txt is read); agent: Crawler user agent or product token (default *).

Free first (25 successful calls per caller, no key):

curl "https://tools.funwords.eu/robots?url=https%3A%2F%2Fwww.google.com%2Fsearch&trial=1"

With a card key from /credits, in a header or, when your client can only fetch a URL, in the URL:

curl -H "Authorization: Bearer adt_…" "https://tools.funwords.eu/robots?url=https%3A%2F%2Fwww.google.com%2Fsearch"
curl "https://tools.funwords.eu/robots?url=https%3A%2F%2Fwww.google.com%2Fsearch&api_key=adt_…"

Without a key the same URL answers 402 Payment Required with x402 and MPP terms, so a wallet client pays USDC per call and retries. Over MCP (https://tools.funwords.eu/mcp) this tool is robots.

Example result
{
  "url": "https://www.google.com/search",
  "robotsUrl": "https://www.google.com/robots.txt",
  "status": 200,
  "agent": "Googlebot",
  "allowed": false,
  "matchedRule": "Disallow: /search",
  "group": "*",
  "rulesInGroup": 280,
  "sitemaps": [
    "https://www.google.com/sitemap.xml"
  ]
}

Tool page, inputs and limits

Questions

Does it crawl the pages?

No. It reads one XML sitemap or sitemap index (gzip supported); child sitemaps are not followed and URLs are not probed.

What are the limits?

Larger sitemaps are read from the start: the first 512 KiB and 1,000 entries, with a warning. Invalid XML is not charged; a valid empty match is.

How do I pay?

Each successful call costs the tool's price; failed calls are not charged. The first 25 calls are free. Then pay with card credits from $1 (the key goes in an Authorization header or as api_key= in the URL) or per call in USDC over x402 or MPP. No account is needed.