AI chatbot regression monitoring
Your AI feature can break with no error in the logs — a model swap, a prompt edit, a broken key, a truncated reply. Relvato sends the same prompt every run and tells you the moment the answer regresses.
Same prompt, every run — so drift becomes visible
Relvato turns an AI feature's unpredictable output into a deterministic check, without flaky assertions.
Pin a fixed prompt
Point Relvato at your AI endpoint — a chatbot, a RAG answer, an agent, any JSON endpoint — and give it one question to ask on every run. Keeping the prompt constant is what makes a regression visible: the app changed, the answer didn't have to.
Assert the essentials
Deterministic rules decide pass or fail: the endpoint responds (2xx), the reply isn't empty, it still contains every required phrase, contains no forbidden phrase, and stays within a length and latency budget.
Score against a golden answer
Optionally paste an approved “golden” reply. Relvato AI grades how closely each run's answer preserves its meaning (0–100), catching quality drift the keyword checks miss. The score only advises — it never decides pass or fail.
Alert and correlate
A regression fires an alert and lines up on the timeline with the deploy, model change, or prompt edit around it — so you know where to look first, not just that something broke.
The silent ways an AI feature regresses
A 200-OK healthcheck sees none of these. A fixed-prompt output check sees all of them.
Endpoint down or 4xx/5xx
A broken API key, a rate limit, an expired token, or a moved route — the feature returns an error or nothing at all.
Empty or truncated answer
The model returns blank, a stub, or a cut-off reply — often after a max-tokens or streaming change.
A required fact disappeared
The answer stops mentioning your return window, hours, pricing, or policy — a RAG or prompt change quietly dropped it.
Started saying the wrong thing
A forbidden phrase creeps in — “I'm sorry, I can't help”, a hallucinated competitor, a leaked system prompt, an apology loop.
Quality drift vs the golden answer
The reply still passes the keyword rules but has drifted in meaning or usefulness — the semantic judge flags the slide before customers do.
Slow or degraded
Latency blows past your budget or the reply shrinks — a model downgrade or a provider hiccup a status page won't show you.
One place for every kind of regression
The AI output check is part of the same monitoring that watches your pages, checkout, and uptime.
AI chatbot regression monitoring — FAQ
What is AI chatbot regression?
It's when an AI feature that used to work starts giving worse answers — errors, empty or truncated replies, a dropped fact, a forbidden phrase, or a drop in quality — usually after a model swap, a prompt or RAG change, a config edit, or a provider issue. Because there's no exception in the logs, it's easy to ship and never notice until customers complain.
How does Relvato detect it without flaky tests?
It sends the same fixed prompt to your endpoint every run and checks the reply against deterministic rules: reachable, non-empty, contains the required phrases, excludes the forbidden ones, and within a length and latency budget. Those rules — not the AI — decide pass or fail, so results are stable and repeatable.
Does the AI decide whether my feature passed?
No. The deterministic assertions decide pass/fail. The optional semantic judge (Relvato AI) only scores how close the answer stays to your approved golden reference and advises on quality drift — it never adjudicates the result.
Does it work with any AI backend?
Yes. Point it at any HTTP endpoint that returns text or JSON — an OpenAI or Anthropic call behind your own API, a Supabase Edge Function, a custom RAG or agent, a chatbot widget's backend. Set a response path to pull the answer out of a JSON reply when needed.
Will it cost me tokens?
It calls your own AI feature, so a run uses whatever that feature costs you — the same as one real user asking one question. You control how often it runs (on a schedule, on every deploy, or on change events), and it's opt-in by design.
Can I monitor a RAG pipeline or agent, not just a chatbot?
Yes. Anything that takes a prompt and returns text works — a support chatbot, a RAG answer endpoint, a summarizer, a classifier, an agent's final message. The fixed-prompt-plus-assertions approach is the same regardless of what's behind the endpoint.
Know the moment your AI answer changes
Add the AI feature output check in minutes — a fixed prompt, a few assertions, and an optional golden answer.