Can I set a custom user-agent for the AskVault crawler?

No. There is no user-agent setting. For robots.txt, the crawler follows rules under 'User-agent: AskVault'.

What is the AskVault BYOK scraper option?

Bring Your Own Key: add a Firecrawl, ZenRows or ScrapFly API key, and AskVault uses it as a last resort for pages it can't read itself. Your provider bills you directly.

Can I throttle AskVault's crawl rate for a specific domain?

No. There is no crawl-rate setting. AskVault limits how many pages it fetches from one site at the same time. If requests to a site keep timing out, AskVault can slow down or pause requests to it for a short while. Pages that fail for temporary reasons (timeouts, server errors, rate limits) are usually retried automatically later.

Advanced scraping settings

Written by Joseph Russo, Principal Architect, Networks and Tech, AskVault · Reviewed by Allison Chan

Last updated: September 26, 2026 · 2 min read

What's on this page

Anti-bot fallback (BYOK). Pick a provider, paste your API key and click Save key. When a background crawl can't read a page with its own plain request or headless browser, it tries your provider. Your provider bills you for those requests. Full details: BYOK Scraper.

Once a key is saved, the page shows the provider and the first characters of the key, with Replace key and Remove key.

On Free and Starter, the page explains that Advanced Scraping needs the Growth plan.

Where the other crawl options are

Per-crawl options live in the Crawl Website window (Knowledge > Add Source > Web Crawler), not here:

Use Sitemap, Page budget and URL Rules for a Domain Crawl. See URL crawling.
CSS Selector under Advanced options. See CSS selector targeting.
Cookies for login-protected pages under Advanced options. See cookies for login crawls.

What AskVault handles automatically

These aren't settings; the crawler does them for every crawl:

JavaScript rendering. Pages that need JavaScript are detected and rendered in a headless browser.
Politeness. AskVault limits how many pages it fetches from one site at the same time. If requests to a site keep timing out, AskVault can slow down or pause requests to it for a short while.
Retries. Pages that fail for temporary reasons (timeouts, server errors, rate limits) are usually retried automatically several times, with growing gaps between attempts. Pages a site blocks may show as Retrying soon for several days before being marked Could not index, or be listed as skipped for having too little text.
robots.txt. When AskVault finds pages for you from your sitemap, it reads your robots.txt (rules under User-agent: AskVault, otherwise *) and skips pages under path rules such as Disallow: /private/. The * wildcard works; end-of-URL anchors ($) and Crawl-delay aren't supported yet. Pages you paste with Exact Links or Website URL are fetched as you asked.

There is no setting for the user agent, the crawl rate, custom request headers, retry counts or forcing JavaScript rendering.

FAQ

Can I use my provider's free tier?

Yes, as long as it's Firecrawl, ZenRows or ScrapFly. Other providers aren't supported.

Does JavaScript rendering work for React, Vue and Angular?

Yes. The headless browser runs the page's JavaScript before reading it.

Was this page helpful?