1,952 Responses, 8 Languages: A Three-Stage Audit of COSS's Agri Chatbot
A chatbot that advises farmers has to be right in Assamese and Tamil, not just in Hindi and English. We had native speakers score every answer, then had a second team check the first one's work.
An agricultural chatbot had to serve farmers and rural users in eight Indian languages. Automated checks could not say whether answers traced back to real source data or only sounded right.
A three-stage Human-in-the-Loop evaluation: full-coverage scoring, an independent blind audit and a QC reconciliation, with one native-speaker reviewer per language at each stage and nine rubric dimensions.
A verified 91.4% quality score, confirmed at 89.0% by a blind audit, and a precise fix list that points to citation accuracy and source grounding instead of a broad quality problem.
COSS Agricultural AI Chatbot
COSS set out to build an AI chatbot that gives agricultural guidance to farmers and rural users, and to do it in eight Indian languages instead of one region's.
In this setting an answer is something a farmer acts on. A confident reply with no real source behind it can mislead, which makes grounding and language quality a trust question as much as a technical one.
Does an Assamese Answer Trace Back to Real Source Data, or Just Sound Plausible?
A chatbot that works well in Hindi and English only serves part of India. COSS wanted farmers across eight languages to get the same quality of guidance, and that bar needs more than automated evaluation to confirm.
Fluency checks can tell you a sentence reads well. They cannot tell you whether a claim is supported by the data the chatbot was supposed to use, or whether quality holds evenly across languages.
- Is it grounded? Does each response trace back to real source and tool data?
- Can a farmer act on it? Does a Tamil answer help, or could it quietly mislead?
- Is quality consistent? Does it hold across all eight languages, or vary in ways one aggregate score would hide?
- Farmers acting on guidance that sounds sourced but is not
- One underperforming language hidden inside a healthy-looking average
- A reassuring score that no one had independently checked
Three-Stage Human-in-the-Loop Evaluation Across 8 Indian Languages
Nine dimensions across Response Usefulness, Linguistic Quality and Factual Grounding were finalised before scoring began, with 1 to 4 scales for nuanced qualities and 0/1 for pass/fail checks, so all eight language teams applied the same bar.
Eight evaluators, one per language, independently scored 100% of the dataset: 244 responses per language and 1,952 in total, against all nine dimensions.
A second set of eight evaluators scored about 12% of the data (240 of 1,952 responses) without seeing any Stage 1 result, to test whether those scores reflected real quality or evaluator leniency.
Eight reviewers audited both earlier stages, looked for scoring discrepancies, confirmed guideline adherence, and re-evaluated wherever a real defect turned up.
No Fabrication and Citation Accuracy were checked directly against the underlying source and tool data. Translation accuracy, grammar and language purity were judged by native speakers, so responses had to read naturally as well as be correct.
Findings became targeted recommendations that put source-grounded response generation first and flagged the rubric dimensions needing the most attention, in place of one blanket fix.
Together these span both pillars, AI Training and Validation & Reliability, with the outcome of continuously trusted AI that performs accurately, safely and reliably in the real world.
A 91.4% Quality Score, Confirmed at 89.0% by a Blind Audit and 94.4% QC-Validated
Weighted score across 100% of 1,952 responses.
A fresh team, no view of Stage 1, found Stage 1 was not inflated.
Accepted outright or with minor corrections after reconciliation.
Citation Accuracy and Source Data Comprehensiveness, instead of a broad quality problem.
The dimensions that mattered most
Citation Accuracy
Claims need to trace back to real source or tool data, not just sound sourced. This was the biggest area of cross-language variation and the clearest lever for trust.
Source Data Comprehensiveness
Some answers drew on only part of the available data, which left them correct but incomplete for a farmer trying to act.
Factual Grounding
A small share of responses stated something plausible that the source data did not support, the failure automated fluency checks miss.
Translation Consistency
A handful of responses read as stiff or literal. A small gap, but one that compounds at scale if left alone.
Usefulness and conversational quality held up
Actionability, conversation closure and accuracy and completeness were consistently strong in all eight languages. The core response quality was not the issue. Grounding and citation were.
| Before the evaluation | After the evaluation |
|---|---|
| A single pass with no independent check on evaluator leniency | A verified 91.4% overall quality score across all 8 languages |
| Aggregate scores that could hide one weak language | Cross-language comparison showing exactly where quality varied |
| No proof that citations traced back to real source data | Fabrication and Citation Accuracy checked against source and tool data for every response |
| An open-ended question: is the model good? | A specific list pointing at citation accuracy and source grounding |
COSS started with a reassuring number and no way to know whether to trust it. Three independent stages turned it into a verified score and a short, specific list of what to fix. The engineering team now knows the work is in source-grounded generation and citation accuracy, and that the underlying response quality is sound across all eight languages.
A 91.4% Score Meant Little Until a Blind Audit Agreed With It
A single pass can look good and still be lenient. A blind second team turns "our evaluators think this is good" into an independent confirmation, a far stronger claim for AI used where answers carry real consequences.
The useful finding was not "improve quality". It was two named dimensions, Citation Accuracy and Source Data Comprehensiveness. Score against rubrics detailed enough to hand engineering a precise target.
A blended 91.4% across eight languages can mask one that lags. Reporting at language level, with native speakers doing the judging, is what surfaces variation an average hides.
Want Your Multilingual Chatbot Checked Language by Language by Native Speakers?
For COSS, three independent stages across eight languages turned a reassuring score into a verified one, plus a precise fix list. Tell us your languages and use case and we will design the same for you.
Insights from the Oprimes team
The Role of AI in Enhancing User Testing
In the fast-evolving landscape of app development, ensuring a seamless user experience is paramount. Traditional user testing methods, while effective,...
Read more →
The AI Revolution: A Testing Framework for the Future of Software
What is AI? Artificial intelligence (AI) is a broad field that includes a variety of techniques and approaches for creating...
Read more →
Improve AI-ML-based facial recognition application accuracy by validation through diverse real data sets using a user testing model.
Conducting multiple face recognition trials in different environments and backgrounds to train the AI-based app and validate how it determines...
Read more →Your AI was built by humans.
Let the right humans validate it.
Book a 30-minute consultation with an Oprimes AI Trust Specialist. We will map your use case, recommend the right service pillar, and give you a delivery timeline before you commit to anything.
Trusted by 80+ enterprise AI teams across 6 industries. No obligation on first consultation.