--- title: Can search engines find my sitemap? How to check with the Sitemap Checker description: New pages take weeks to show up in search, and you're not sure crawlers even know your sitemap exists. To check a sitemap, look where crawlers look: /sitemap.xml, the common index paths, and the Sitemap: line in robots.txt. Here's how to do it in seconds with the free Sitemap Checker, a real case where a large site's sitemap was invisible at every standard location, and the one-line fix. canonical: https://watchfor.io/blog/how-to-check-a-sitemap --- # Can search engines find my sitemap? How to check with the Sitemap Checker 2026-10-04 · WatchFor Team · Guides · guides, seo, http New pages take weeks to show up in search, and you're not sure crawlers even know your sitemap exists. To check a sitemap, look where crawlers look: /sitemap.xml, the common index paths, and the Sitemap: line in robots.txt. Here's how to do it in seconds with the free Sitemap Checker, a real case where a large site's sitemap was invisible at every standard location, and the one-line fix. To check whether search engines can find your sitemap, request it where crawlers look for it: `https://yourdomain/sitemap.xml`, the common index paths, and any `Sitemap:` line in `robots.txt`. If none of those returns a valid XML sitemap with a 200 status, a crawler that hasn't been told the URL some other way (Search Console, Bing Webmaster Tools) won't find it. The free [Sitemap Checker](/sitemap-checker) does those lookups from a WatchFor probe location, counts the URLs in the sitemap it finds and samples a few of them for broken links. No signup. ## What this tool answers *Is there a sitemap where crawlers expect it, does it parse, and do the first URLs in it work?* Reach for it when new pages are slow to appear in search, after a CMS or framework migration, after adding locale routing or a CDN rule, or when Search Console says it "couldn't fetch" a sitemap. It checks what an outside crawler sees, not what your build output says. ## A real example ### Checking a sitemap that search engines can't discover A redesign added language routing, and since then new pages seem to be discovered more slowly. Does the sitemap still sit where crawlers look? We ran the [Sitemap Checker](/sitemap-checker) on `mozilla.org`, a large, well-run site that happens to show this exact pattern: ![The Sitemap Checker after checking mozilla.org from Vilnius: a red No sitemap badge, the URL count shown as a dash, domain mozilla.org](/blog/how-to-check-a-sitemap-result.webp) **No sitemap.** The check took 2.7 seconds. The checker asked for `/sitemap.xml`, `/sitemap_index.xml` and `/sitemap/sitemap.xml`, then read `robots.txt` for a `Sitemap:` line. None of those led to a file containing `` or ``. Following the first request by hand shows why: ```text https://mozilla.org/sitemap.xml → 301 to https://www.mozilla.org/sitemap.xml https://www.mozilla.org/sitemap.xml → 302 to /en-US/sitemap.xml https://www.mozilla.org/en-US/sitemap.xml → 404 ``` The locale redirect treats `/sitemap.xml` like a page and sends it to a language folder where no such file exists. And `robots.txt` contains only two `disallow` lines, with no `Sitemap:` line. The sitemap does exist. It is a sitemap index at `https://www.mozilla.org/all-urls.xml` that lists 100 child sitemaps, one per locale plus a global one. Nothing at the standard locations points to it. **The answer:** Mozilla has a sitemap, but a crawler can only use it if someone gave it the URL directly. Mozilla may well have submitted it in Search Console; any crawler that relies on discovery won't find it. On your own site, the fix is one line in `robots.txt`: ```text Sitemap: https://www.mozilla.org/all-urls.xml ``` Or exclude `/sitemap.xml` from the locale redirect so the standard URL answers directly. Either way, run the checker again afterwards: it should show **Sitemap found** and a URL count. ## How to read the results | What you see | What it means | | --- | --- | | **Sitemap found** (green) | A file at one of the standard locations returned 200 and contains `` or ``. | | **No sitemap** (red) | None did. Either there is no sitemap, it lives at a non-standard path that `robots.txt` doesn't mention, or it returns something other than 200. | | **Sitemap found** (amber) | A sitemap was found, but at least one sampled URL is broken. | | **URLs** | How many `` entries the sitemap holds. For a sitemap index, this is the number of *child sitemaps*, not pages. | | **Sampled** | How many entries were requested: the first 5 in the file, with HEAD requests, following redirects. | | **Broken** | Sampled entries that ended in a 4xx or 5xx status, or failed to connect. A redirect that ends in 200 counts as fine. | | **Broken links list** | The failing URLs with their status code, so you know exactly which entries to fix. | Only the domain is used: you can't point the checker at a custom sitemap path. That's deliberate. It shows what a crawler without your help finds. ## More ways to use it - **After a framework or CMS migration.** Generators often move the sitemap (`/sitemap-index.xml`, `/wp-sitemap.xml`, `/sitemap_index.xml`); make sure `robots.txt` names the new one. - **After adding locales or a CDN rule.** Redirects meant for pages often catch `/sitemap.xml` and `/robots.txt` too, as in the example. - **When Search Console says "Couldn't fetch".** Check whether the URL answers 200 from outside your network, not only from your office. - **Before launch.** Confirm the staging host doesn't leak into the production sitemap and that the first URLs aren't 404s. ## Tips and gotchas - **Only the first `Sitemap:` line in `robots.txt` is followed.** List your main index first. Search engines read every line; this checker reads one. - **A sitemap index isn't opened.** For an index, the five sampled entries are child sitemaps, checked only to respond. To validate the pages inside, see [how to validate an XML sitemap](/blog/how-to-validate-an-xml-sitemap). - **HEAD requests can mislead.** A few servers answer HEAD with 405 or 403 while GET works. If a URL is flagged broken, open it in a browser before rewriting anything. - **Five URLs is a sample.** "0 broken" means the first five work, not all of them. Put new or risky URLs near the top while you test. - **Size limits.** One sitemap file may hold at most 50,000 URLs and 50 MB uncompressed ([sitemaps.org](https://www.sitemaps.org/protocol.html)); above that, split into an index. - **The checker gives up after 12 seconds.** A sitemap generated on every request that takes longer reads as a failure, and slow sitemaps hurt crawlers too. ## Check it from the command line too Follow the standard location and print every hop: ```bash curl -sIL https://mozilla.org/sitemap.xml | grep -iE '^(HTTP|location)' ``` ```text HTTP/2 301 location: https://www.mozilla.org/sitemap.xml HTTP/2 302 location: /en-US/sitemap.xml HTTP/2 404 ``` Look for a `Sitemap:` line in `robots.txt`: ```bash curl -s https://www.mozilla.org/robots.txt ``` ```text user-agent: * disallow: /*/newsletter/existing/ disallow: /*/etc/ ``` And confirm the real file is valid and count its entries: ```bash curl -s -o /dev/null -w '%{http_code} %{content_type}\n' https://www.mozilla.org/all-urls.xml curl -s https://www.mozilla.org/all-urls.xml | grep -o '' | wc -l ``` ```text 200 text/xml 100 ``` ## Keep watching it Sitemaps break silently: a deploy empties one, a redirect rule swallows it, or a CMS plugin update moves it. A [sitemap monitor](/sitemap-monitoring) re-checks on a schedule, samples up to 50 listed URLs, and alerts when the sitemap disappears (**Sitemap Available**), when too many sampled URLs are broken, or, with **Minimum URL Count**, when a deploy shrinks it. ## FAQ **How do I find a website's sitemap?** Try `/sitemap.xml` and `/sitemap_index.xml`, then read `/robots.txt` for a `Sitemap:` line. If none works, the site may only have submitted it in Search Console, or it uses a custom path like Mozilla's `/all-urls.xml`. **Do I need to put my sitemap in robots.txt?** It's optional but useful. Search engines accept it from Search Console, Bing Webmaster Tools or a `Sitemap:` line in robots.txt; the robots.txt line is the only one every crawler can find without an account. **Why does Google say my sitemap couldn't be fetched?** Usually the URL returns a redirect to an error, a 404, a 403 from a bot rule, or HTML instead of XML. Run `curl -sIL` on the exact URL you submitted and fix whatever the last hop returns. [301 vs 302 redirects](/blog/301-vs-302-redirects) explains the redirect codes you'll see. --- Canonical page: https://watchfor.io/blog/how-to-check-a-sitemap · All articles: https://watchfor.io/blog/all.md · Site guide: https://watchfor.io/llms.txt