What sitemap formats does AskVault support for crawling?

Standard XML sitemaps, sitemap index files (including nested indexes), and gzipped sitemaps. AskVault follows sitemap index entries to collect the page URLs they list.

What if my website has no sitemap?

During onboarding, AskVault follows links from your homepage instead, up to two clicks deep. In the Knowledge page, a Domain Crawl without a sitemap indexes only the starting URL, so paste the pages you need with Exact Links.

Is there a limit to how many pages AskVault can find from a sitemap?

Yes. AskVault reads up to 5,000 page URLs from your sitemaps per crawl, and on the Free plan a crawl keeps at most 1,000 of them. Separately, how much it indexes is limited by your plan's content storage cap.

How to crawl a sitemap into AskVault

Written by Joseph Russo, Principal Architect, Networks and Tech, AskVault · Reviewed by Allison Chan

Last updated: September 26, 2026 · 3 min read

Why the sitemap matters

A sitemap is a list of your pages that your site publishes for search engines. AskVault uses it as the list of pages to read. Most WordPress, Shopify, Webflow, Wix and Squarespace sites publish one automatically.

Where AskVault looks

Given any URL on your site, AskVault looks for sitemaps on that site's domain, in this order:

Sitemap: lines in your robots.txt.
/sitemap.xml.
/sitemap_index.xml.

It doesn't matter which page you paste; AskVault always looks at these locations on your domain. If your sitemap lives somewhere else, add a Sitemap: line pointing to it in your robots.txt. Pasting the sitemap's own URL does not by itself make AskVault use that file.

Only pages on your site's own domain are kept (www. and the bare domain count as the same site). Pages on other domains or subdomains listed in your sitemap are skipped.

How to crawl

In the Knowledge page, open Add Source > Web Crawler.
Choose Domain Crawl and enter your site's URL.
Leave Use Sitemap on.
Optionally add URL Rules to keep or skip parts of the site (see URL crawling).
Click Preview crawl to see how many pages were found, then Start Crawl.

Progress shows in the Knowledge page as pages are indexed. Your agent can answer from each page as soon as it's ready.

Formats AskVault reads

Single sitemaps. One file listing your pages.
Sitemap indexes. A file that lists other sitemaps (posts, pages, products). AskVault opens each one, including indexes nested inside indexes.
Gzipped sitemaps (.xml.gz). Decompressed automatically.

Sites in several languages

If your sitemap is split by language (for example /en-us/ and /de-de/ sections), AskVault reads pages in the language of the URL you paste first. If that URL has no language in its path, pages without a language prefix go first. Other languages are not excluded: they're read afterwards, as the 5,000-URL limit and your storage cap allow.

To index only one language, paste a URL from that language's section (such as https://acme.com/de-de/) and add an include URL rule such as /de-de/.

Limits

5,000 URLs per crawl. AskVault reads up to 5,000 page URLs from your sitemaps. On a larger site, the rest are not found in that crawl, and URL rules don't change which 5,000 are read (rules are applied afterwards). On Free, a crawl keeps at most 1,000 of those pages: in the Knowledge page the rest aren't added, and onboarding sets them aside. Add important pages beyond the limit with Exact Links, or contact [email protected].
Content storage cap. 5 MB on Free, 15 MB on Starter, 40 MB on Growth, 100 MB on Business. Pages found beyond the cap are set aside until you upgrade.
robots.txt. Pages your robots.txt path rules disallow for AskVault (or *) are skipped. End-of-URL anchors ($) aren't supported yet; see full-site crawl.

Common pitfalls

Only the homepage was indexed. AskVault found no sitemap at the locations above. Check that https://yoursite.com/sitemap.xml opens in your browser, or add a Sitemap: line to robots.txt.

The sitemap is empty. Some CMS plugins publish an empty sitemap. Open it in your browser; if it lists no pages, fix it in your CMS first.

Some pages are missing. They aren't in your sitemap, robots.txt disallows them, they're past the 5,000-URL limit (1,000 pages on Free), or they were set aside by your storage cap.

FAQ

Does AskVault use `changefreq`, `priority` or `lastmod` from my sitemap?

changefreq, priority and lastmod don't schedule anything; a re-crawl schedule is set on the source instead. Paid plans can set an automatic re-crawl schedule, but until automatic re-crawl is on for your account a scheduled check may not re-read a changed page, so keep using Resync to be sure (details). See keeping website knowledge up to date.

Can my sitemap list pages on another domain?

Those pages are skipped. Crawl each domain separately.

What about image or news sitemaps?

Only page URLs are indexed. Images are not indexed.

Sitemaps are fetched without your cookies. For login-protected pages, paste them with Exact Links and add your session cookie under Advanced options; see cookies for login crawls.

Was this page helpful?