Keeping website knowledge up to date
What's available today
A crawled website is a snapshot of your pages at the time they were read. When you change a page, your agent keeps answering from the old version until you refresh it. Today you refresh manually, in three ways:
- Resync selected / Resync all. In the Knowledge page, select one or more sources. The selection bar offers Resync selected (only the pages you picked) and Resync all (every source with a web address in the workspace, including imported ones; see the caution below).
- Force Resync. In a single page's Source Options menu. Checks that one page straight away.
- Run the Web Crawler again. Re-run Add Source > Web Crawler with the same starting URL to add pages that are new on your site since the last crawl. Pages already in your knowledge base are left as they are.
Resyncing does not use message credits.
How resync decides what changed
Resync asks your web server whether each page changed, using the page's standard cache headers (ETag, Last-Modified and content length). Pages that report no change are skipped. Pages that changed are queued and read again, and your agent answers from the new version once they finish.
Two things to know:
- Some servers always look "changed". If your server sends no cache headers, or sends one that changes on every request (common on Shopify), every page is re-read on each resync. That is safe, just slower.
- Small edits can be missed. If your server sends no ETag, resync treats a page as unchanged when its Last-Modified date is the same as last time, or when its size changed by about 5% or less. An edit such as a new price, a date or one sentence usually changes a page's size by less than that, so resync can report "already up to date". To pick up the edit on an ordinary page, paste it again with Add Source > Web Crawler > Exact Links, which compares the page's text and so catches any change. For a store product page, delete the page in the Knowledge page, then run Add Source > Web Crawler > Domain Crawl again for your store: that re-adds the page with its product details (price, stock), which Exact Links doesn't save.
What resync does not do
- It doesn't find new pages. Only pages already in your knowledge base are checked. Run the Web Crawler again for new pages.
- It doesn't remove deleted pages. If a page is gone from your site, delete it in the Knowledge page so your agent stops using it.
- Don't run it during a crawl. Starting Resync all while a crawl is still running interrupts that crawl. Wait until indexing finishes.
Automatic refresh
The dashboard has three automatic refresh settings:
- Auto-Sync Schedule, in the Web Crawler window for a Domain Crawl and in a source's details.
- Auto-Remove Dead Links, in the Web Crawler window for a Domain Crawl.
- Daily auto-resync of seed URLs, in Global AI Settings.
Auto-Sync Schedule can be set on paid plans (under Advanced options in the Web Crawler window, and in a source's details). Until the rollout below is on for your account, a scheduled check may find that a page changed without re-reading it, so keep using Resync to be sure. Auto-Remove Dead Links and Daily auto-resync of seed URLs still show as locked on every plan because of a known issue.
Rolling out: automatic re-crawl on by default
We're rolling out a change that turns automatic re-crawl on for paid plans. It is not on for every account yet. Until it's on for yours, the section above still applies.
Once it's on for your account:
| Plan | Default for a new website | You can choose |
|---|---|---|
| Free | Manual only | – |
| Starter | Weekly | Weekly or Off |
| Growth | Weekly | Daily, Weekly or Off |
| Business | Daily | Daily, Weekly or Off |
| Enterprise | Daily | Daily, Weekly or Off |
- New websites you add get your plan's default.
- Websites you already have. In a workspace where you never set a schedule, your websites get your plan's default. From now on, a schedule you set yourself, including Off, is kept.
- Monthly. Websites already set to Monthly keep re-crawling monthly. Monthly just isn't offered for new choices.
- You stay in control. Change the schedule, or turn it off, in a source's Auto-Sync Schedule.
- You can see it working. The Knowledge page shows a line such as "Auto re-crawl: Weekly · Last checked 3h ago · Next Oct 4, 2026". It also shows a status (Not checked yet, Synced, Re-crawling..., Partly checked or Needs attention) and your last five runs, each with how many pages changed.
- Only changed pages are re-read. Each run uses the same change check as Resync, so the notes in How resync decides what changed apply.
- If you change plans, the schedule you saved is kept. While your plan doesn't include it, it runs at the nearest schedule your plan does include (for example, Daily runs weekly on Starter), and moving back to a plan that includes it restores it. On the Free plan re-crawl is manual.
- Upgrading from Free. In a workspace where you never set a schedule, your websites get the new plan's default when you upgrade.
Other sources
- Uploaded files don't change on their own. Upload the new version and delete the old one.
- One-time imports (Notion, Zendesk, Confluence, GitHub and similar) don't sync. Run the import again to refresh; see each integration's page. Don't resync imported sources (why).
- Shopify stores connected through the Shopify app receive product changes (create, update, delete) from Shopify as they happen. That covers your product catalog only, not your store's other pages.
FAQ
Does AskVault re-crawl my website automatically?
It's rolling out to paid plans and isn't on for every account yet; see Rolling out. Until it's on for yours, use Resync as described above.
Does resync reset my audience settings?
No. A page's audience (public or internal) stays as you set it.
Can I see when a page was last refreshed?
Yes. Open the page in the Knowledge page; its Sync History section shows when it was last synced.