Batch URL ingestion
When to use it
- Migrating from another platform. You have a list of the pages you already trained on.
- No sitemap. In the Knowledge page, a Domain Crawl without a sitemap indexes only the starting URL; Exact Links lets you choose the pages.
- Specific pages only. You want your agent to answer from a handful of pages, not the whole site.
How to add a list of URLs
- In the Knowledge page, open Add Source > Web Crawler.
- Switch to Exact Links.
- Paste up to 50 URLs, one per line. The window shows how many it detected.
- Optionally open Advanced options to set a CSS selector or cookies for these pages.
- Click Scrape N URLs.
AskVault reads the pages one after another and adds each one to the Knowledge page as it's indexed. A page that can't be read (not found, blocked, or with almost no text) is not added, so check the Knowledge page afterwards to confirm every page you pasted is there.
For a longer list, submit it in batches of 50.
Good to know
- You choose the pages. Exact Links fetches exactly the URLs you paste, one at a time. Only add pages you have the right to index.
- Storage cap. Every indexed page counts toward your plan's content storage cap.
- Pasting a page again updates it if its content changed, instead of adding a second copy.
- Your BYOK scraper key is not used for Exact Links; see below.
- Audience. Pages are added as
public. Change a page's audience in its settings afterwards. - No API or CSV upload. Lists are pasted in the dashboard; there is no bulk-URL API or CSV import today.
Common pitfalls
Some pages didn't appear. They couldn't be read: the address is wrong, the page needs a login (add your cookie under Advanced options), or the site's bot protection blocked the request. For blocked sites on Growth and above, a Domain Crawl can use your BYOK Scraper key; Exact Links can't.
"invalid URL detected" message. A line isn't a valid web address. Each line needs one address, such as https://yoursite.com/pricing.