# How to control what goes into the knowledge base

You decide what coreAI can answer from. The knowledge base contains only what you have pointed the crawler at or sent in through the API, and you can keep content out or remove it at any time. This is what determines the quality of the answers: the AI cannot answer wrongly from content that isn't there.

## Two ways in: crawling and the API

coreAI fills the knowledge base in two main ways, and you can use one or both in the same assistant:

- **Crawling** fetches content from your website. It suits content that is already published, when you want to get started quickly.
- **The API** lets your system send content directly. It suits content that lives in a business system, a CMS or a curated database, when you want to control every single entity.

You can also upload files and paste in text directly.

## Crawling: you choose which pages are fetched

The crawler only fetches what you ask it to. You control the scope in three ways:

- A single base URL makes coreAI read the sitemap and fetch the URLs listed there.
- A list of URLs fetches exactly those pages and nothing more.
- A URL ending in `*`, for example `https://example.com/regulations/*`, fetches the page and everything under the same path.

The crawler respects `robots.txt`. You choose whether it should follow links to external PDFs and Google Drive documents, and whether the source should update automatically on a fixed schedule or only manually.

## Keep content out by path, age and file type

The exclusion list keeps out what should not be included. You write one pattern per line and can add filters in square brackets:

- `/archive/*` keeps the whole archive out.
- `/news/* [age > 3 months]` keeps out news older than three months.
- `/documents/* [type = pdf]` keeps out PDFs under `/documents/`.

Filters can be combined. When you save the exclusions, coreAI also deletes content that has already been imported and matches them. An old PDF that is still on the server will therefore not end up in the answers.

## Only the main content is included

coreAI removes navigation and other repeated page structure before the content becomes searchable. Menus, footers, breadcrumbs, cookie banners and carousels are skipped, and so are headers outside the main content. Elements with very little text and duplicate lines are filtered out.

If your website has its own blocks that add noise, such as "Related articles", add the CSS class or ID to the source. Then they are kept out too.

## The API: you send exactly what goes in

With the API, the knowledge base contains only what your system sends. Each entity has a stable ID from your system and a type, such as `documents`, `contents` or `products`, so the assistant knows whether it is answering from a rule, an article or a product.

- `POST` creates or replaces the whole entity.
- `PATCH` updates only the fields that have changed.
- `DELETE` removes the entity and the relationships around it.
- `GET` fetches entities by your IDs, so you can check what is stored.

When a rule changes on your side, you send the new version with the same ID. The old version is replaced, not stored alongside it, so there are never two versions of the same rule. You can also link related entities together, for example a main rule and its exceptions. The relationships are passed to the AI when it answers.

## Outdated content should be removed, not just ranked lower

coreAI does not rank results by date. An older document that is closer to the wording of the question can therefore rank above a newer one. The safe solution is to make sure outdated content is not in the knowledge base:

- With the API, your system deletes or replaces the entity when it no longer applies.
- With crawling, you keep old content out with age and file type filters, or by removing it from the path that is crawled.

Data that changes often, such as prices, opening hours or order status, does not need to be in the knowledge base at all. If you connect an MCP server, the assistant fetches the data directly from your system when the question requires it, and nothing is copied in. [Read how the MCP import works](https://coreai.no/en/articles/mcp-import).

API access is included in the Standard and Large plans. Read [the technical API article](https://coreai.no/en/articles/api-data-integration) to get started.