Skip to content
AskVault Docs
Try Free →

How to add your WordPress content to AskVault

Written by Joseph Russo, Principal Architect, Networks and Tech, AskVault · Reviewed by Allison Chan
Last updated: 3 min read

How WordPress content gets into AskVault

AskVault crawls your WordPress site from its sitemap, the same way it crawls any website:

  • WordPress 5.5 and later publishes a sitemap at /wp-sitemap.xml and lists it in your robots.txt.
  • SEO plugins such as Yoast or Rank Math publish their own (often /sitemap_index.xml) and list it in robots.txt too.

AskVault finds either automatically. It reads the published pages the sitemap lists (posts, pages and any public custom post types your sitemap includes) and indexes their visible text.

Setup

  1. New agent: paste your site's URL in the onboarding wizard. Existing agent: open Knowledge > Add Source > Web Crawler, choose Domain Crawl and enter your site's URL.
  2. Optionally add URL rules, such as exclude /tag/ and /category/ to skip archive listing pages (see URL crawling).
  3. Once pages are indexed, add the chat widget: see widget setup. You can paste the snippet into your theme's footer or with any header-and-footer code plugin.

What's included and what isn't

  • Included: published pages listed in your sitemap, with their visible text.
  • Not included: drafts, private or password-protected posts, comments, and pages your robots.txt disallows.
  • Members-only pages can be crawled with a session cookie; see cookies for login crawls.

Keeping it current

Paid plans can set an automatic re-crawl schedule, but until automatic re-crawl is on for your account a scheduled check may not re-read a changed page, so keep using Resync to be sure (details). After you publish or edit posts:

  • Resync in the Knowledge page re-reads pages that changed.
  • Running the Web Crawler again for the same site adds new posts.

See keeping website knowledge up to date.

Limits

  • Content storage cap. Your plan's MB allowance (5 MB on Free, up to 100 MB on Business).
  • Discovery. Up to 5,000 URLs from your sitemap per crawl (1,000 pages on Free).

Common pitfalls

Only the homepage was indexed. AskVault couldn't find your sitemap. Check that https://yoursite.com/wp-sitemap.xml (or your SEO plugin's sitemap) opens, and that search-engine visibility isn't turned off under Settings > Reading in WordPress, which also disables the sitemap.

Answers include menu or footer text. Paste the affected pages again with Add Source > Web Crawler > Exact Links, with a CSS selector for your theme's content area (for example .entry-content) under Advanced options. Running the crawl again doesn't re-read pages that are already indexed.

FAQ

Does this work on WordPress.com?

Yes. WordPress.com sites publish a sitemap too, so they can be crawled like any other site.

Can I exclude specific posts?

Add an exclude URL rule for their path, or delete them in the Knowledge page after the crawl.

Was this page helpful?