Skip to content
AskVault Docs
Try Free →

BYOK Scraper, bring your own scraping API key

Written by Joseph Russo, Principal Architect, Networks and Tech, AskVault · Reviewed by Allison Chan
Last updated: · 3 min read

When you need it

AskVault reads most sites with a plain web request, and switches to a headless browser for pages that need JavaScript. Some sites still refuse automated visitors, usually because of bot protection (Cloudflare, Akamai and similar) or rules that block data-center traffic. In the Knowledge page, pages like that may show as Retrying soon for several days before being marked Could not index, or be listed as skipped for having too little text.

If that's happening on a site you're allowed to index, BYOK Scraper gives AskVault another way in: your own account with a scraping provider that specialises in these sites.

Available on Growth and above. Growth+

How it works

For each page, AskVault tries in order:

  1. A plain web request.
  2. A headless browser, when the page needs JavaScript.
  3. Your BYOK provider, if the steps above didn't return readable content.

The decision is made page by page; AskVault does not remember which sites needed your key. There are no per-site allow or block lists.

Your key is used by background crawls: Add Source > Web Crawler > Domain Crawl, and the part of onboarding that keeps indexing after your agent is set up. It is not used for pages added with Exact Links or Website URL, or when you Force Resync a single page.

Supported providers

  • Firecrawl
  • ZenRows
  • ScrapFly

Setup

  1. Sign up with a provider and create an API key in their dashboard.
  2. In AskVault, open Settings > Advanced Scraping.
  3. Pick the provider and paste the API key, then click Save key.

That's it. Crawls that run after you save use your key when needed. The page shows which provider is configured and the first characters of the key; Remove key turns the fallback off.

Cost

Requests made with your key are billed by your provider under your plan with them, not by AskVault. Because the key is only used for pages AskVault couldn't read itself, it's usually a small share of your pages. On a site where every page is blocked, every page goes through your provider, so watch your usage in the provider's dashboard.

When BYOK isn't enough

  • Upload the content. Save the page as a PDF from your browser and upload it.
  • Paste it as a snippet. For a few pages, add the text as a snippet.

Your responsibilities

  • robots.txt still applies. When AskVault finds pages for you from a sitemap, pages your robots.txt path rules disallow are skipped, whether or not you use BYOK.
  • Only index content you're allowed to. A scraping provider doesn't change a site's terms of service.

Limits

  • Plan. Growth and above.
  • One provider per workspace.
  • Key storage. Encrypted at rest, and never shown in full again after you save it.

Common pitfalls

The provider rejects every request. The key is wrong or expired. Test it with the provider directly, then paste a new one.

Provider costs are higher than expected. Many pages on that site need the fallback. Check whether the content is available another way (a sitemap on a different host, a PDF, or an export you can upload).

FAQ

Does BYOK change what I pay AskVault?

No. Your provider bills you separately.

Can I use cookies with BYOK?

Cookies you enter under Advanced options are used by AskVault's own requests. They are not passed to your BYOK provider.

Can I rotate keys?

Yes. Paste the new key under Settings > Advanced Scraping and click Replace key.

Was this page helpful?