Skip to content
AskVault Docs
Try Free →

How to crawl a website into your AskVault knowledge base

Written by Joseph Russo, Principal Architect, Networks and Tech, AskVault · Reviewed by Allison Chan
Last updated: · 5 min read

What URL crawling does

You give AskVault a URL; it fetches the pages, extracts the readable text, and indexes it so your agent can answer from it with links back to the source page.

There are two places to add a website:

  • Onboarding. When you create an agent, you paste your homepage and AskVault finds your pages for you.
  • Knowledge. Add more sites or pages at any time from Add Source.

Adding a site from the Knowledge page

Add Source > Web Crawler opens the Crawl Website window. It has two modes:

  • Domain Crawl. Enter a starting URL. AskVault finds pages from your site's sitemap (see sitemap crawl) and queues them for indexing.
  • Exact Links. Paste specific page URLs, one per line (up to 50 at a time). Only those pages are indexed.

To index one page, you can also use Add Source > Website URL.

In Domain Crawl mode you can also set:

  • Use Sitemap. On by default. Keep it on: the Knowledge page finds pages through your sitemap.
  • Page budget. All pages (default) indexes every page found, up to your storage cap. Set a limit stops after the number of pages you enter.
  • URL Rules. Up to 10 include or exclude rules (see below).
  • Preview crawl. Shows how many pages were found and how many fit in your storage cap, without indexing anything.

Important: in the Knowledge page, Domain Crawl does not follow links from page to page. If AskVault can't find a sitemap for your site, or you turn Use Sitemap off, only the starting URL is indexed. For a site without a sitemap, paste the pages you need with Exact Links, or add the site during onboarding (see below).

How onboarding finds your pages

During onboarding, AskVault reads your sitemap. If your site has no sitemap, it follows links from your homepage instead, up to two clicks deep.

URL rules

Rules narrow a Domain Crawl to part of your site:

  • Include. Only pages whose path contains the text are kept. Example: /docs/ keeps https://acme.com/docs/setup and skips https://acme.com/pricing.
  • Exclude. Pages whose path contains the text are skipped. Example: /tag/ skips tag archive pages.

Write each rule as a plain piece of the path, one per rule. Wildcards such as * are not supported: /docs/* matches nothing. Rules check the path only, not the query string, so a rule like ?page= has no effect. If your include rules match no pages, only the starting URL is indexed.

Known limitation: Preview crawl currently evaluates rules as wildcard patterns, so with rules set its page count can differ from what the crawl indexes. After you click Start Crawl, the confirmation message shows how many pages were actually queued.

CSS selector and cookies

Under Advanced options:

  • CSS Selector. By default the crawler keeps the main content and strips navigation, headers and footers. If your layout confuses it, enter a selector for your main content area, such as main, #content or .article-body. It applies to every page in that crawl. See CSS selector targeting.
  • Cookies. For pages behind a login, paste your session cookie. See cookies for login crawls.

Limits

AskVault reads up to 5,000 page URLs from your sitemaps per crawl, in the sitemap's own order. On a larger site the rest are not found in that crawl; add important pages beyond the limit with Exact Links. On Free, a Domain Crawl keeps at most 1,000 pages and doesn't add the rest. Your plan's content storage cap then sets how much of that is indexed: 5 MB on Free, 15 MB on Starter, 40 MB on Growth and 100 MB on Business. When a crawl finds more than fits, the extra pages are set aside rather than dropped, and the confirmation message tells you how many. Upgrading lets them be indexed.

How the crawler handles JavaScript

Sites built with React, Vue, Angular or Next.js often send an almost empty page and draw the content with JavaScript. AskVault detects this and switches to a headless browser automatically. You don't need to configure anything. See the scraper architecture.

Sites that block crawlers

Some sites (often behind Cloudflare or similar bot protection) block automated visits. In the Knowledge page, those pages may show as Retrying soon for several days before being marked Could not index, or be listed as skipped for having too little text. On Growth and above, you can add your own key from a scraping provider under Settings > Advanced Scraping; Domain Crawls then use it when AskVault's own attempts fail. See BYOK Scraper. Growth+

PDFs on your site

PDF files that are listed in your sitemap, or that you paste as a URL, are downloaded and their text is indexed as their own page, with a citation back to the file. PDFs up to 15 MB are supported, and text is read from the first 300 pages. For Word and other office files, use file upload.

Keeping content fresh

Paid plans can set an automatic re-crawl schedule, but until automatic re-crawl is on for your account a scheduled check may not re-read a changed page, so keep using Resync to be sure (details). See keeping website knowledge up to date for how to refresh changed pages.

Common pitfalls

Only one page was indexed. AskVault couldn't find a sitemap, Use Sitemap was off, or your include rules matched nothing. Check that https://yoursite.com/sitemap.xml loads, remove wildcard characters from your rules, or paste the pages with Exact Links.

Some pages are missing. They aren't in your sitemap, your robots.txt disallows them, or they were set aside by your storage cap.

The agent quotes navigation or footer text. Paste those pages again with Exact Links, with a CSS selector for your main content area under Advanced options. Running a Domain Crawl again doesn't re-read pages that are already indexed.

FAQ

Can I crawl multiple sites into one workspace?

Yes. Run Add Source > Web Crawler once per site. All pages count toward the same workspace storage cap.

Does the crawler respect robots.txt?

Partly. When AskVault finds pages for you from your sitemap (onboarding and Domain Crawl), it skips pages under robots.txt path rules such as Disallow: /private/, and there is no override. The * wildcard works; end-of-URL anchors ($) and Crawl-delay aren't supported yet. For a site you control, allow or disallow AskVault under User-agent: AskVault (see full-site crawl). Pages you paste yourself with Exact Links or Website URL are fetched as you asked.

Does it follow "Load more" buttons or infinite scroll?

No. Content that only appears after clicking or scrolling may be missed. If that content has its own URL, add it with Exact Links.

How do I delete crawled content?

Select pages in the Knowledge page and click Delete, or use Delete in a page's options menu. Deleted pages stop being used in answers right away.

Was this page helpful?