# Sitemap parser API

Sitemap parser: reads one public XML sitemap or sitemap index as structured URLs and source-reported dates. Supports gzip, substring filtering and pagination. Large sitemaps are read from the start (512 KiB, 1000 entries at most, with a warning). Does not crawl pages or follow child sitemaps. Invalid XML and execution errors are not charged; a valid empty match is charged. Use to find which pages of a docs site to read before calling /extract.

## When an agent wants this

- Find which pages of a documentation site to read before calling /extract
- Filter URLs by a substring and page through large sitemaps
- Read source-reported last-modified dates

## Sitemap parser, URL extractor: documentation URLs from a bounded XML sitemap

Sitemap parser, URL extractor: documentation URLs from a bounded XML sitemap. $0.002 per successful call; failed calls are never charged.

Sitemap parser: reads one public XML sitemap or sitemap index as structured URLs and source-reported dates. Supports gzip, substring filtering and pagination. Large sitemaps are read from the start (512 KiB, 1000 entries at most, with a warning). Does not crawl pages or follow child sitemaps. Invalid XML and execution errors are not charged; a valid empty match is charged. Use to find which pages of a docs site to read before calling /extract.

Inputs: url (required): Absolute public HTTP(S) XML sitemap URL; child sitemaps are not fetched automatically; contains: Optional case-insensitive URL substring, at most 200 characters; maxUrls: Reported matching locations examined in one page (1-100, default 25); invalid/nonpublic locations may reduce returned entries; offset: Offset into distinct reported matching locations (0-1000, default 0); use nextOffset to continue.

Free first (25 successful calls per caller, no key):

    curl "https://tools.funwords.eu/sitemap?url=https%3A%2F%2Fwww.sitemaps.org%2Fsitemap.xml&trial=1"

With a card key from https://tools.funwords.eu/credits, in a header or, when your client can only fetch a URL, in the URL:

    curl -H "Authorization: Bearer adt_…" "https://tools.funwords.eu/sitemap?url=https%3A%2F%2Fwww.sitemaps.org%2Fsitemap.xml"
    curl "https://tools.funwords.eu/sitemap?url=https%3A%2F%2Fwww.sitemaps.org%2Fsitemap.xml&api_key=adt_…"

Without a key the same URL answers 402 with x402 and MPP terms, so a wallet client pays USDC per call and retries. Over MCP (https://tools.funwords.eu/mcp) this tool is sitemap.

Example result:

    {
      "url": "https://agent-data-tools.revmesh2074.workers.dev/examples/sitemap/source.xml",
      "finalUrl": "https://agent-data-tools.revmesh2074.workers.dev/examples/sitemap/source.xml",
      "kind": "urlset",
      "entries": [
        {
          "url": "https://agent-data-tools.revmesh2074.workers.dev/examples/extract/sources/technical-guide.html",
          "lastModified": "2026-10-15"
        },
        {
          "url": "https://agent-data-tools.revmesh2074.workers.dev/examples/extract/sources/short-article.html",
          "lastModified": "2026-10-15"
        }
      ],
      "reportedEntries": 3,
      "uniqueReportedLocations": 3,
      "missingOrInvalidLocations": 0,
      "duplicateReportedLocations": 0,
      "rejectedLocationsInPage": 0,
      "matchingReportedLocations": 3,
      "contains": "",
      "offset": 0,
      "nextOffset": 2,
      "hasMore": true,
      "warnings": [
        "counts_describe_reported_locations_before_public_url_validation",
        "locations_and_dates_are_source_reported_not_probed"
      ],
      "text": "https://agent-data-tools.revmesh2074.workers.dev/examples/extract/sources/technical-guide.html\nhttps://agent-data-tools.revmesh2074.workers.dev/examples/extract/sources/short-article.html",
      "format": "text",
      "hashFormat": "text",
      "contentHash": "sha256:8055b8b5ced205885a181b4fa7145008ac8d5dc3fb3286007c4cf679f886b750",
      "sourceHash": "sha256:c24148c4774690472856c116012988bb52e992c6a3bbba156947be1dac6eba03",
      "fetchedAt": "2026-10-15T21:48:20.943Z"
    }

Tool page: https://tools.funwords.eu/tools/sitemap

## Robots.txt check: may this crawler fetch this URL?

Robots.txt check: may this crawler fetch this URL?. $0.01 per successful call; failed calls are never charged.

Read a site's robots.txt and say whether a crawler may fetch a URL, by RFC 9309 rules as Google applies them (most specific user-agent group, longest matching Allow or Disallow, Allow on ties, * and $ wildcards). Returns the deciding rule, the group used, the rule count and the sitemaps listed. A missing robots.txt (4xx) allows everything. Use before scraping or crawling a site politely; for the sitemap URLs themselves use sitemap.

Inputs: url (required): The page URL to check (its site's /robots.txt is read); agent: Crawler user agent or product token (default *).

Free first (25 successful calls per caller, no key):

    curl "https://tools.funwords.eu/robots?url=https%3A%2F%2Fwww.google.com%2Fsearch&trial=1"

With a card key from https://tools.funwords.eu/credits, in a header or, when your client can only fetch a URL, in the URL:

    curl -H "Authorization: Bearer adt_…" "https://tools.funwords.eu/robots?url=https%3A%2F%2Fwww.google.com%2Fsearch"
    curl "https://tools.funwords.eu/robots?url=https%3A%2F%2Fwww.google.com%2Fsearch&api_key=adt_…"

Without a key the same URL answers 402 with x402 and MPP terms, so a wallet client pays USDC per call and retries. Over MCP (https://tools.funwords.eu/mcp) this tool is robots.

Example result:

    {
      "url": "https://www.google.com/search",
      "robotsUrl": "https://www.google.com/robots.txt",
      "status": 200,
      "agent": "Googlebot",
      "allowed": false,
      "matchedRule": "Disallow: /search",
      "group": "*",
      "rulesInGroup": 280,
      "sitemaps": [
        "https://www.google.com/sitemap.xml"
      ]
    }

Tool page: https://tools.funwords.eu/tools/robots

## Questions

Docs for agents: https://tools.funwords.eu/llms.txt · OpenAPI: https://tools.funwords.eu/openapi.json · All guides: https://tools.funwords.eu/guides