Full-site crawl
When to use it
- Setting up an agent. Onboarding crawls your site for you.
- Adding another site or a large section of one, from Knowledge > Add Source > Web Crawler > Domain Crawl.
For a known list of pages, use Exact Links instead (see batch URL ingest).
How pages are found
- From your sitemap. Both onboarding and the Knowledge page read your sitemap (see sitemap crawl), up to 5,000 page URLs per crawl (1,000 on Free).
- Without a sitemap, during onboarding. AskVault follows links from your homepage, up to two clicks deep.
- Without a sitemap, in the Knowledge page. Links are not followed; only the starting URL is indexed. Paste the pages you need with Exact Links.
There is no depth setting and no option to include other subdomains; crawl each subdomain separately.
URL rules
In a Domain Crawl you can add up to 10 include or exclude rules. Each rule is plain text matched anywhere in the page's path:
- Include
/docs/keeps only pages whose path contains/docs/. - Exclude
/tag/and exclude/category/skip taxonomy listing pages.
Wildcards like * are not supported, and the query string is not checked. See URL crawling for details and a known Preview limitation.
robots.txt
When AskVault finds pages for you from your sitemap, it reads your robots.txt first. There is no setting to ignore it:
- Rules under
User-agent: AskVaultapply to AskVault. Write the name without a version number: a line readingUser-agent: AskVault/1.0isn't matched. - Otherwise, rules under
User-agent: *apply. - Pages under path rules, such as
Disallow: /private/orDisallow: /*?sort=, are skipped. The*wildcard works.
What robots.txt doesn't cover today:
- End-of-URL anchors (
$, as inDisallow: /*.pdf$) aren't supported yet, so such a rule doesn't stop those pages being indexed. Use an exclude URL rule instead. Crawl-delayisn't read.- Pages you paste yourself with Exact Links or Website URL are fetched as you asked.
- Following links during onboarding. When a site has no sitemap, AskVault opens pages to find links on them before the robots.txt check is applied to the list of pages to index.
To keep AskVault out of part of your site, add a Disallow path rule under User-agent: AskVault. To let it into a path you disallow for other bots, add an Allow rule there.
Crawler identity
Page requests are sent with standard browser user-agent strings, not a distinct AskVault name, and AskVault does not publish a fixed list of crawler IP addresses. That means you can't allowlist the crawler in a firewall or bot-protection service, or filter it out of analytics, by user agent or IP today. robots.txt (above) is the reliable control.
Load on your server
- AskVault limits how many pages it fetches from one site at the same time.
- If requests to your site keep timing out, AskVault can slow down or pause requests to it for a short while.
- Pages that fail for temporary reasons (timeouts, server errors, rate limits) are usually retried automatically later.
Size and storage
AskVault reads up to 5,000 page URLs from your sitemaps per crawl, in the sitemap's own order. On a larger site the rest are not found in that crawl; add important pages beyond the limit with Exact Links. On Free, a crawl keeps at most 1,000 pages: in the Knowledge page the rest aren't added, and onboarding sets them aside. Your plan's content storage cap then sets how much of that is indexed (5 MB on Free, 15 MB on Starter, 40 MB on Growth, 100 MB on Business). Pages beyond the cap are set aside and shown as available after an upgrade, not silently dropped.
Crawls run in the background. Your agent can answer from each page as soon as it's indexed.
Keeping it fresh
Paid plans can set an automatic re-crawl schedule, but until automatic re-crawl is on for your account a scheduled check may not re-read a changed page, so keep using Resync to be sure (details). Refresh changed pages with Resync; see keeping website knowledge up to date.
Audience after a crawl
Crawled pages are created public. Audience has exactly two values,
public and internal, and is set per knowledge source after the crawl under Knowledge >
[select source] > Settings. There are no per-path rules that assign an audience during the
crawl. To keep a URL prefix out of the index entirely, exclude it with a URL rule instead.
Seeing which pages were skipped
Rolling out; not switched on for every account yet.
When a crawl leaves pages out, the Knowledge page shows a collapsed N pages skipped: see which line. Open it to see each skipped page and the reason in plain words:
- Blocked by the site's robots.txt: see robots.txt.
- Left out by your include/exclude rules: see URL rules.
- Skipped as low-value: tag, search, login or archive pages.
- Over your plan's content limit: these resume automatically if you upgrade.
- You removed this page: pages you deleted, which later crawls won't add back.
The list shows up to 200 pages for each reason, from your most recent website crawl, alongside the full count. Only the workspace owner can see it.
Common pitfalls
Only one page was indexed. No sitemap was found (in the Knowledge page), or your include rules matched nothing. See sitemap crawl.
Unwanted pages were indexed. Add exclude rules and crawl again, then delete the unwanted pages in the Knowledge page.
Many pages stay on Retrying soon, end up as Could not index, or are skipped for too little text. Your site's bot protection may be blocking the crawler. See BYOK Scraper.
JavaScript-rendered content. AskVault detects pages that need JavaScript and renders them in a headless browser automatically.
FAQ
Can I crawl a competitor's site?
Only if their robots.txt allows it, and within their Terms of Service.
Does the crawl show up in my analytics?
It can. Requests look like ordinary browser visits, so analytics tools that run on page load may count them, and they can't be filtered by a crawler name.
Will my server be hit hard?
AskVault limits how many pages it fetches from one site at the same time. If requests keep timing out, it can slow down or pause for a short while, and pages that failed for temporary reasons are usually retried later.