How AskVault reads JavaScript-heavy and protected websites
The core insight
A large share of modern websites work fine with a plain HTTP GET request. Server-rendered marketing pages, traditional CMS sites, blog posts, documentation hosted on most static-site generators. A fast HTTP fetch gets the full content in a fraction of a second.
The rest are JavaScript-rendered single-page apps, behind anti-bot protection, or both. For those you need a headless browser that executes JavaScript and returns the rendered DOM. Browser fetches are several times slower and far more memory-hungry than a plain fetch.
A naive scraper escalates everything to a browser. Slow and expensive. A better scraper tries HTTP first and only escalates when needed. That's the AskVault approach.
When a page needs more than a fast fetch
Two kinds of pages need more than a plain fetch.
JavaScript-rendered pages. The server answers, but the real content is built in the visitor's browser, so a plain fetch returns little more than an empty app shell. AskVault recognizes a response like that and re-reads the page with a real browser.
Bot-protected pages. Some sites answer automated requests with a block or a challenge page instead of the content. AskVault recognizes those responses too and moves to the next step instead of indexing a challenge page as if it were your content.
The escalation path
AskVault's scraper works through these steps and only moves on when the current one can't read the page:
- Fast fetch. Quick and light. Works on a large share of pages.
- Real browser. Runs the page's JavaScript and reads the finished page. Works on most JavaScript-rendered sites.
- Your own scraping key (BYOK). On Growth and above, if you've added a scraping-provider key, background crawls try it for pages the first two steps couldn't read. See BYOK Scraper.
AskVault does not operate its own anti-bot bypass service. When a site's bot protection blocks both the fetch and the browser, those pages aren't indexed unless you add a BYOK key. In the Knowledge page they may show as Retrying soon for several days before being marked Could not index, or be listed as skipped for having too little text.
Slow-loading single-page apps
Some single-page apps take a moment to paint their real content. When a page still looks nearly empty after the first browser read, AskVault gives it another, more patient pass before giving up, so late-loading content is captured. Compressed sitemaps (sitemap.xml.gz) are handled too, so a site that publishes only a compressed sitemap still has every URL found.
Remembering which sites need a browser
AskVault learns as it crawls. When a site keeps turning away the fast fetch or keeps returning empty pages, later pages from that site go straight to the browser instead of failing the cheap step first. It checks again now and then, so a site that recovers goes back to the fast path. On a large crawl this saves a lot of waiting.
Polite, resilient crawling
AskVault limits how many pages it fetches from one site at the same time, and it slows down or pauses for a short while when a site keeps timing out. Pages that fail for temporary reasons (timeouts, server errors, rate limits) are retried automatically later, so one slow site doesn't hold up the rest of the crawl.
How content is prepared after the fetch
Getting the page is half the work. Turning it into useful passages is the other half. AskVault:
- Drops page furniture. Navigation, headers, footers and link-only menus are left out so they don't crowd out your real content. Image alt text is kept, so descriptive captions stay searchable.
- Keeps headings with their text. Each passage carries the heading it sits under, so it still makes sense on its own.
- Keeps tables together. A table stays in one piece where it fits. A very large table is split by rows, and the header row is repeated on every part so no passage is a headerless fragment.
- Keeps code blocks intact. Code stays formatted as code, which matters for technical docs.
- Splits at natural breaks. Long sections are divided between paragraphs, never mid-sentence.
The result is a set of clean passages ready for search.
Cookie and authentication handling
Some content is gated behind a login. AskVault supports a Cookies field in Crawl config where you paste a session cookie. The scraper sends the cookie on every request and gets the authenticated version of the page.
Two warnings:
- Cookies expire. To update pages that are already indexed, paste them again with Exact Links and a fresh cookie under Advanced options. Resync doesn't reuse your cookie.
- Don't share live session cookies. Log out of the source account after indexing so the cookie can't be re-used by AskVault or anyone with access.
There is no custom request-header setting; cookies are the only way to crawl content behind a login.
Robots.txt and politeness
When AskVault finds pages for you from a sitemap, it reads the site's robots.txt (rules under User-agent: AskVault, otherwise *) and skips pages under path rules such as Disallow: /private/. The * wildcard works; end-of-URL anchors ($) and Crawl-delay aren't supported yet. Pages a customer pastes with Exact Links or Website URL are fetched as asked.
Page requests themselves use standard browser user-agent strings, and AskVault doesn't publish a fixed list of crawler IP addresses, so the crawler can't be allowlisted or blocked by user agent or IP today. Site owners who want us to back off can use robots.txt or email [email protected].
AskVault limits how many pages it fetches from one site at the same time. If requests to the host keep timing out, it can slow down or pause for a short while.
What you get in the dashboard
Operationally, all of this surfaces in Knowledge > Crawl Status:
- Per-document indexing state. QUEUED, INDEXING, READY, or one of several error states.
- Last-attempted timestamp. When the scraper last tried this URL.
- Failure breakdown. If indexing failed, the dashboard shows the specific failure (timeout, 403, empty body, content too thin) so you know whether to retry or change settings.
- Automatic retries. Temporary failures are retried automatically over time. Permanent failures (such as 404) don't retry and are surfaced in the KPI tile "X sources need attention".
- Pages that ran out of retries. A page that still can't be fetched after its automatic retries are used up is marked Could not index. The Knowledge page shows how many pages are in that state ("N pages couldn't be indexed after repeated attempts.") with a Retry these pages button that sends exactly those pages for another try. You can use it once every 10 minutes, and pages that would not fit your plan's content limit stay where they are until there is room. This is rolling out and may not be on your workspace yet.
Why this matters for your AI agent
If your AI agent doesn't index a page, it can't answer questions from that page. A scraper that fails silently on JavaScript-rendered content means the bot gives wrong-or-empty answers on the modern half of your site.
A common reason a bot answers wrong is that part of the site is rendered by JavaScript and a basic scraper skipped it. Indexing those pages with a real browser gives the bot the content it was missing.
Crawl coverage isn't just about completeness. It's about whether the bot can answer the question at all.
FAQ
Can I disable browser escalation if I'm sure my site is HTTP-only?
No, and you don't need to: a page that returns its content over plain HTTP never reaches the browser step.
What if the bot keeps failing on a specific page?
Check the Knowledge crawl-status page. If the page consistently fails, you can manually upload its content as a document or paste it as a Snippet. The bot will retrieve from those sources instead.
Can I bring my own scraping API key for hostile sites?
Yes, on the Growth plan and above. Add your own scraping-provider key under Settings > Advanced Scraping; background crawls use it for pages AskVault can't read itself. See BYOK Scraper.
Does the scraper render dynamic content like infinite scroll?
No. The browser renders the page as it first loads and doesn't scroll, so content that only appears after scrolling may be missed.
How does it handle pagination?
Pages are found from your sitemap (links are followed only during onboarding, when a site has no sitemap), so paginated pages are indexed only if they're listed there. "Load more" buttons are not clicked.
Related guides
- How to scrape a JavaScript-rendered website for LLM training
- How vector databases work
- What is RAG?
- How to restrict the AI bot to specific URLs only