AI

Inside an AI Chatbot: Where the Data Comes From and Which Model Powers It

Ask a chatbot a question and an answer appears in a second or two. It feels like talking to something that knows things. It doesn’t, at least not in the way most people assume. A chatbot is two separate parts working together. One is a language model that writes. The other is a set of sources that tell it what to write about. Knowing which part does what explains most of what chatbots get right, and most of what they get wrong.

The model writes, it doesn’t know

A large language model (LLM) learns from huge amounts of public text before anyone ever talks to it. That training teaches it grammar, tone and a broad picture of the world, frozen at a cut-off date. It doesn’t teach it your service catalogue, your prices or last week’s news. Ask a bare model about those and it will still answer, fluently and often wrongly. The industry calls this hallucination. I think “confident guessing” is more honest.

Where the data actually comes from

Most useful chatbots don’t rely on what the model memorised. They look things up first. The technique is called retrieval-augmented generation, or RAG, and it was described in a 2020 paper by Lewis et al. When a question arrives, the system searches a set of trusted sources, picks the relevant passages and hands them to the model together with the question. The model’s job shrinks to reading those passages and writing a clear answer.

Newer systems go a step further with tool calling. Instead of running one search, the model chooses from several tools, each a small function that fetches one kind of data. A tool might search the documentation, read a live status page or list the user’s own orders. The model picks the tool that fits the question, reads the result and answers from it.

Choosing a model, and a provider

There are dozens of capable models now, and the gap between them narrows every few months. For a chatbot that answers from retrieved text, the biggest model is rarely the right one. Speed and cost per question matter more, because the hard part, finding the facts, is done before the model even sees the question.

The provider matters as much as the model. Every question a user types travels to whoever runs that model, and it may contain personal data. For a European research infrastructure that rules out a lot of options, so we route requests through EUrouter, a gateway that processes traffic inside the EU and keeps us on firmer ground under the GDPR. It speaks the same API as OpenAI, so switching model is a one-line configuration change. An example of a current good model is GLM-5.3 Flash, a small and fast model from Z.ai, picked because it calls tools reliably and answers quickly at a low cost. Next year a different one might be better, and that’s fine.

Our advice if you’re planning one

  • Start from your content. A chatbot is only as good as the pages it reads. If your documentation is out of date, fix that before choosing any model.
  • Know where the data goes. Check where the provider processes requests and whether it stores them. For European organisations this is a legal question first.
  • Let it say “I don’t know”. A bot that admits a gap and points to a human beats one that invents an answer.
  • Keep the model swappable. Models improve every few months. Don’t build your project around one of them.

If we had to keep only one of these, it would be the third. In our experience users forgive a bot that admits it doesn’t know. They stop trusting one the first time they catch it making something up.

Thinking about an AI assistant?

At COMMpla we design and build digital platforms for research and innovation projects, AI assistants included.

If you’re planning a chatbot for your website or portal and want to understand what it involves, from preparing your content to protecting your users’ data, get in touch with our team.

Maicol Battistini | Full-stack Developer