Aller au contenu
Web&Leads

Cette page n'est pas encore disponible en français. Vous lisez la version anglaise.

AI software

How we stopped our AI product from hallucinating: lessons from Listrino

Par Abdo Rakhma9 min de lecture

Prompts alone did not stop our AI product from inventing problems. What worked was treating the model as one step in a pipeline: collect evidence first, map it into a fixed set of themes, rank actions in code, let the model only write within pre-selected slots, then verify every claim against the source and drop anything unsupported. The result is output users can trust, at a predictable cost.

The problem: confident nonsense

Listrino analyzes Airbnb and Booking.com listings and tells hosts what to fix. Our first version was simple: send the listing and its reviews to a language model and ask for recommendations.

The output looked great. It was also often wrong. The model would say guests complained about a missing towel when no review mentioned towels. It would read "great value" as a complaint about price. It would suggest adding more photos to a listing that already had 40.

For a paid product, one invented problem destroys trust in the whole report. We had to fix it properly.

Lesson 1: Prompts are not guardrails

We started where everyone starts: stricter prompts. "Only use information from the reviews." "Do not invent problems." "Quote the review that supports each point."

It helped a little. It did not solve it. Models follow instructions most of the time, and most of the time is not enough when every report goes to a paying customer.

What we learned

Treat the prompt as a suggestion to the model, not as a guarantee to your users. Anything that must be true has to be checked outside the model.

Lesson 2: Collect evidence before you generate

We rebuilt the engine around an evidence bundle. Before any writing happens, the pipeline collects:

  • The listing itself: title, description, amenities, photo count, ratings.
  • All reviews, with category scores.
  • A pack of comparable top listings in the same city and property type, cached for 31 days.
  • Platform best practices we encoded as rules.

The model never sees "the internet". It sees a structured bundle of facts we can later check against.

Lesson 3: Use the model to classify, not to decide

Instead of asking "what should this host fix?", we ask a small model to map review text into a fixed set of themes, such as cleanliness, check-in, noise or Wi-Fi, with a regex fallback when the model is unsure.

Classification into a closed list is something models do reliably. Open-ended judgment is not.

Lesson 4: Rank in code

Once we have themes and evidence, ranking happens in TypeScript. Each possible action gets a severity score from how many reviews mention it and how strongly, and an effort score from the type of fix. Competitor gaps are measured from real data, never assumed.

This makes results reproducible. The same input gives the same priorities, which is impossible to guarantee with a model deciding the order.

Lesson 5: Let the model write inside slots

The writer model receives pre-selected actions with their evidence and fills short slots: a title, a one-line impact and a concrete task. We route these steps to smaller, cheaper models and keep the output schema strict with Zod.

The model is now doing what it is great at, clear writing, on facts it did not choose.

Lesson 6: Verify, then verify again

After writing, a deterministic guard runs on every item:

  • Does the quoted review text actually appear in the reviews?
  • Does the quote actually name the problem it claims to support?
  • Is a category flagged as a problem even though its score is high and no review complains?
  • Is "add more photos" suggested when the listing already has plenty?
  • Does a suggested title invent a city name or unit count that is not in the listing?

Anything that fails is dropped. We would rather show four solid recommendations than six with one invented.

Lesson 7: Make evidence visible

Every recommendation in a Listrino report shows how many reviews mention it. Users do not have to trust the AI. They can see why.

The result

  • Reports in under 2 minutes, compared to more than 15 hours of manual analysis.
  • No unsupported complaint passes the guard.
  • Predictable costs, because expensive models only run where they add value.

Applying this to your AI product

The same pattern works for support assistants, document analysis, sales research and compliance checks:

  1. Collect and structure evidence first.
  2. Use models for classification and writing, not for final decisions.
  3. Rank and decide in code where you can.
  4. Verify every claim against the source before output.
  5. Show users the evidence.

You can read the full technical story in the Listrino report engine case study. If you are building an AI product and want it to be right, not just impressive, see our AI software development service.

Questions fréquentes

What is an AI hallucination?

A hallucination is output that sounds plausible but is not supported by the input or by facts, such as an invented quote, statistic or problem. Language models produce them because they generate likely text, not verified facts.

Can better prompts stop hallucinations?

Better prompts reduce them but do not eliminate them. For products where accuracy matters, you need checks outside the model: grounding in retrieved data, constrained outputs and verification in code.

What is a grounded AI pipeline?

A pipeline where the model works only from evidence you collected and passed in, and where its output is checked against that evidence before anyone sees it.

Does verification make AI products slower or more expensive?

Code-level checks are fast and cheap. In our case, routing simple steps to smaller models and caching data kept reports under 2 minutes and costs predictable, even with verification added.

Un projet ou une idée d'IA ?

Dites-nous ce que vous voulez améliorer ou créer. Nous répondons sous un jour ouvré avec des recommandations franches pour la suite.