Prompts alone did not stop our AI product from inventing problems. What worked was treating the model as one step in a pipeline: collect evidence first, map it into a fixed set of themes, rank actions in code, let the model only write within pre-selected slots, then verify every claim against the source and drop anything unsupported. The result is output users can trust, at a predictable cost.
The problem: confident nonsense
Listrino analyzes Airbnb and Booking.com listings and tells hosts what to fix. Our first version was simple: send the listing and its reviews to a language model and ask for recommendations.
The output looked great. It was also often wrong. The model would say guests complained about a missing towel when no review mentioned towels. It would read "great value" as a complaint about price. It would suggest adding more photos to a listing that already had 40.
For a paid product, one invented problem destroys trust in the whole report. We had to fix it properly.
Lesson 1: Prompts are not guardrails
We started where everyone starts: stricter prompts. "Only use information from the reviews." "Do not invent problems." "Quote the review that supports each point."
It helped a little. It did not solve it. Models follow instructions most of the time, and most of the time is not enough when every report goes to a paying customer.
What we learned
Treat the prompt as a suggestion to the model, not as a guarantee to your users. Anything that must be true has to be checked outside the model.
Lesson 2: Collect evidence before you generate
We rebuilt the engine around an evidence bundle. Before any writing happens, the pipeline collects:
- The listing itself: title, description, amenities, photo count, ratings.
- All reviews, with category scores.
- A pack of comparable top listings in the same city and property type, cached for 31 days.
- Platform best practices we encoded as rules.
The model never sees "the internet". It sees a structured bundle of facts we can later check against.
Lesson 3: Use the model to classify, not to decide
Instead of asking "what should this host fix?", we ask a small model to map review text into a fixed set of themes, such as cleanliness, check-in, noise or Wi-Fi, with a regex fallback when the model is unsure.
Classification into a closed list is something models do reliably. Open-ended judgment is not.
Lesson 4: Rank in code
Once we have themes and evidence, ranking happens in TypeScript. Each possible action gets a severity score from how many reviews mention it and how strongly, and an effort score from the type of fix. Competitor gaps are measured from real data, never assumed.
This makes results reproducible. The same input gives the same priorities, which is impossible to guarantee with a model deciding the order.
Lesson 5: Let the model write inside slots
The writer model receives pre-selected actions with their evidence and fills short slots: a title, a one-line impact and a concrete task. We route these steps to smaller, cheaper models and keep the output schema strict with Zod.
The model is now doing what it is great at, clear writing, on facts it did not choose.
Lesson 6: Verify, then verify again
After writing, a deterministic guard runs on every item:
- Does the quoted review text actually appear in the reviews?
- Does the quote actually name the problem it claims to support?
- Is a category flagged as a problem even though its score is high and no review complains?
- Is "add more photos" suggested when the listing already has plenty?
- Does a suggested title invent a city name or unit count that is not in the listing?
Anything that fails is dropped. We would rather show four solid recommendations than six with one invented.
Lesson 7: Make evidence visible
Every recommendation in a Listrino report shows how many reviews mention it. Users do not have to trust the AI. They can see why.
The result
- Reports in under 2 minutes, compared to more than 15 hours of manual analysis.
- No unsupported complaint passes the guard.
- Predictable costs, because expensive models only run where they add value.
Applying this to your AI product
The same pattern works for support assistants, document analysis, sales research and compliance checks:
- Collect and structure evidence first.
- Use models for classification and writing, not for final decisions.
- Rank and decide in code where you can.
- Verify every claim against the source before output.
- Show users the evidence.
You can read the full technical story in the Listrino report engine case study. If you are building an AI product and want it to be right, not just impressive, see our AI software development service.