Skip to content
AskVault Docs
Try Free →

Cookies for authenticated crawls

Written by Joseph Russo, Principal Architect, Networks and Tech, AskVault · Reviewed by Allison Chan
Last updated: · 4 min read

When to use

Three cases:

  1. Customer portal docs that require login.
  2. Paid-content pages for subscriber sites.
  3. Internal wiki accessible only to authenticated users.

For OAuth-supporting sources, prefer OAuth where AskVault has it. Cookies are a fallback for sites without an API or OAuth path.

Check who can see these pages. Content behind a login is usually not meant for anonymous website visitors. Today, a page you add with cookies is Public unless you change it, so your website chatbot can answer from it. We're rolling out a change that keeps content crawled with login cookies Internal Team Only by default. Until it reaches your workspace, set the Audience yourself (see Who can see the pages you add).

Setup

Under 5 minutes:

Step 1: log in to the target site

In your browser, log in normally.

Open DevTools (F12) > Application (or Storage) > Cookies, and copy the relevant cookie names and values for that site: or use a browser extension like "Cookie Editor" to export them as a name=value; name2=value2 string, or as JSON.

Step 3: paste into AskVault

  1. Knowledge > Add Source > Web Crawler > Exact Links, and paste the login-protected page URLs (or use Add a single URL for one page).
  2. Open Advanced options and paste the cookie string or JSON into the Cookies field.
  3. Start the crawl.

AskVault sends the pasted cookies as a request header for those pages. Behind-login pages become reachable for the pages you listed.

Cookies work with Exact Links and single-URL adds only. A whole-site crawl (the Domain Crawl tab) doesn't send cookies, it would only see the logged-out site, so list the login-protected pages with Exact Links instead.

What happens after the crawl

This is the part that trips people up:

  • The cookie string isn't stored. It's used for the one crawl run you triggered, then discarded: there's no persistent cookie storage to inspect or manage later.
  • Resync doesn't reuse it. Resync fetches pages without the cookie you pasted, since nothing was saved to replay, so a login-protected page can come back as your login screen. To refresh login-protected pages, paste those pages again with Exact Links and a fresh cookie under Advanced options instead of using Resync. That updates pages whose content changed.
  • AskVault doesn't capture cookie rotation. Even within a crawl, if the target site issues a new session cookie mid-crawl (Set-Cookie on a response), AskVault does not adopt it for subsequent requests: this is a deliberate safety choice, since the underlying HTTP client is shared across other customers' crawl jobs and auto-adopting a Set-Cookie could leak one tenant's session into another's requests.

Who can see the pages you add

Pages crawled with login cookies are becoming internal by default (being rolled out). Once that's on for your workspace:

  • New pages you add with cookies are Internal Team Only. They stay in your dashboard, but the website chatbot, the hosted chat page and other public channels don't answer from them.
  • Pages you already had keep their setting. Re-adding a page with fresh cookies doesn't change the Audience you chose for it, so if you made it Public on purpose it stays Public. Check pages you added with cookies before the change: they may still be Public.
  • Adding without cookies is unchanged. Pages you add without a cookie keep the usual Public default.

To let visitors see a page: open Knowledge, open that source's settings, and set Who can ask questions about this? to Public (Website Chatbot & Internal Team). To hide a page that's Public, pick Internal Team Only. You can also do this over the API: see Audience tags.

Cookies typically expire:

  • Session cookies. Expire on browser close.
  • Persistent cookies. Last days to months.

Either way, once the cookie you pasted expires, subsequent pages return unauthenticated (usually a login-wall page), so those pages won't have useful content indexed. Re-extract fresh cookies and paste those pages again with Exact Links and a fresh cookie under Advanced options to pick up the real content again.

Security considerations

Cookies are sensitive:

  • Treat like passwords. Don't share or commit them anywhere.
  • They're not persisted by AskVault: there's nothing sitting in a database to leak later, but the cookie string does pass through AskVault's servers for that one crawl, same as any other request data.
  • Use a dedicated service-account login rather than your personal account, so a stale or over-shared cookie doesn't expose your own session.
  • Check the Audience of what you crawled. Behind-login content answers anonymous website visitors if its source is Public. See Who can see the pages you add.

Limits

  • Cookie field length. 4,000 characters total (the whole string or JSON blob you paste), not per individual cookie.
  • Persistence. None. Paste the cookie again every time you need authenticated access.
  • Auto-rotation. Not supported. A site that rotates its session cookie mid-crawl will start failing partway through.

Alternatives

If cookies are too brittle:

  • OAuth, where AskVault supports it for the source type. Preferred when available.
  • API tokens, for sources with a token-based API (e.g. Confluence, Jira).
  • Direct file upload. Manually upload PDFs or exports of the behind-login content: see file uploads.
  • Public mirror. Many docs sites maintain a public mirror for SEO that doesn't need auth at all.

Common pitfalls

Cookies expire partway through a large crawl. Initial pages succeed; later ones fail once the session lapses. Re-extract fresh cookies and paste those pages again with Exact Links and a fresh cookie under Advanced options for the pages that failed.

Cookie doesn't match the target domain. The cookie's domain must match the site you're crawling (or be a parent domain).

Using Resync on login-protected pages. Cookies aren't stored, so Resync fetches those pages unauthenticated. When you need an updated authenticated pass, paste those pages again with Exact Links and a fresh cookie under Advanced options.

MFA-protected accounts. A session cookie captured after MFA works until it expires, same as any session cookie: but AskVault doesn't help you get past MFA itself. Use a service account without MFA if you need this to work repeatedly.

FAQ

Is this safe?

If you use a dedicated service account (not your personal admin login), yes. And since AskVault doesn't store the cookie afterward, there's no persisted copy to worry about beyond that one crawl request.

Can I use this for a site I don't own?

Only if you have explicit authorization. Respect the site's Terms of Service.

Will a later refresh keep authenticating automatically?

No. Cookies aren't retained between runs, so no later refresh can send them. Whenever you want an updated authenticated pass, paste those pages again with Exact Links and a fresh cookie under Advanced options. Running the crawl again only adds pages that aren't indexed yet.

What if the site rotates cookies aggressively during a single crawl?

AskVault doesn't adopt a rotated session mid-crawl, so a very aggressive rotation policy will make later pages in that same crawl fail. Consider an alternative (OAuth, API token, or file upload) for that source.

Was this page helpful?