Traditional QA is built on a comforting idea: given input X, you should get output Y. Write the case, run it, get a green check. Run it again tomorrow and it should still be green.
Language models do not work like that. The same prompt, sent twice, can come back as two different readings. One of them is exactly what a learner needed. The other has a grammar mistake that would teach the wrong thing, or a polite refusal because the topic tripped a safety filter, or a perfectly fluent paragraph in a language the app does not even support.
That is not an edge case. That is the product.
The product is the prompt box
Contexta lets people generate reading passages and vocabulary lessons from an AI prompt. You pick a topic, the model writes a text in your target language, you tap the words you do not know, and Contexta turns those words into fill-in-the-blank exercises.
That loop is the whole point of the app. It is also a QA surface that traditional test plans are not designed to cover. A user can type almost anything into that box. They will. You have to assume they will, and you have to decide what the app does when the model goes somewhere you did not intend.
The same prompt is not the same test
In a conventional app, "the user taps Generate" is one case. With an LLM it is a distribution of cases.
The model might produce a clean Japanese paragraph at the right level, with furigana on the right characters. It might also:
- Invent a word that looks Japanese and is not
- Write at the wrong difficulty, so a beginner gets newspaper prose
- Put the answer to a cloze exercise in the sentence, or write a sentence whose blank does not match the word the user flagged
- Switch language halfway through
- Refuse, hedge, or lecture, because the topic looked unsafe to the model even though it was harmless
None of those are crashes. They will not show up in a snapshot test. They will show up as a learner memorising the wrong sentence, which is worse.
String-matching the output is useless. "Semantically acceptable, in the right language, at the right level, and safe" is the actual assertion — and that is a judgement, not an equality check.
Users will try to move the model
A prompt field is still a prompt field, even if you branded it as "what do you want to read about?". People will test it. Some of that is curiosity. Some of it is an attempt to get the model to ignore the system instructions, dump its prompt, or produce something the app is not supposed to generate.
You cannot treat that as an abuse-team problem that sits outside QA. If the topic field is the input, then jailbreaks, instruction-overrides and "write this in a different format" are test cases. The app has to fail closed: keep the user inside a language lesson, or refuse cleanly, never silently cooperate.
The refusal path needs QA of its own. A wall of model-speak is a bad user experience. A quiet "we can't make a lesson from that — try a different topic" is a product decision, and it has to be tested like one.
Content policy is not theoretical
Contexta is a reading app, not a chatbot. That does not matter to the model. The input is still free text.
Users can ask for a lesson about something that does not belong in a vocabulary product — adult content, hate, violence, medical advice presented as fact. They can also ask for something that looks innocent in English and is not, once the model writes it in another language.
If you only test happy-path topics ("a day at the market", "preparing for a date"), you have not tested the product. The policy layer has to sit on both sides of the model: what you are willing to send, and what you are willing to show. Shipping a wrong sentence is a quality failure. Shipping a sentence that violates the store's content rules is a review failure, and possibly an account failure.
The language the user asked for is not always the language you support
Contexta currently covers a specific set of languages. The model, left to its own devices, covers far more. A user studying Japanese can still type "write this in French". They can mix languages. They can ask for a dialect you do not teach. They can pick Spanish in settings and then describe a topic that causes the model to answer in English, because English is the model's default comfort zone.
That is a hard bug in a language-learning app, and a silent one. The UI will still look like a lesson. The audio might still play. The user might not even notice they are studying the wrong language until they are several cards in.
The fix is not "hope the prompt is clear". Pin the language from the user's profile, validate it on the way out, and fail the generation if the output does not match. The prompt is a topic. It is not the spec.
You cannot brute-force this, and you cannot afford to
Traditional coverage thinking says: list the scenarios, write a case for each, run them all. With an LLM the scenario list does not end. Topics × languages × skill levels × adversarial phrasing × "what if the model is having an off day" is not a test suite. It is an infinite matrix, and every extra run costs tokens.
That cost is not abstract. Contexta meters AI usage with tokens for a reason — generation, lookups and exercise writing all hit a paid API. QA that calls the live model on every permutation will either be shallow or expensive, and usually both.
So the work is to choose the slices that actually matter:
- A golden set of prompts that includes the ugly cases, not just the demo cases
- Structured output (a schema, not a blob of prose) so the first line of defence is "this is not a lesson object" rather than a human reading 200 paragraphs
- Checks that are cheap and deterministic — language ID, schema validity, policy flags — before you spend a reviewer on pedagogical quality
- Sampling of real generations in production, because the prompts you invented in QA are not the prompts you will get
- A budget. If the eval harness can bankrupt the project, it will not get run, and an eval that does not get run is not QA
What "up to date" actually looks like
Using an LLM in a product is easy. The APIs are good now. Wrapping a prompt in a button and calling it a feature takes a weekend.
Shipping it as something a learner can trust takes a different discipline. You still need the traditional things — onboarding, payments, empty states, restore-purchase. You also need a QA process that assumes the model will be wrong, that the user will try to move it, that some requests should never become a lesson, and that you will never have enough cases to be sure.
The pass condition is not "the model returned 200". It is "across the ways this can go, the app stays a language app."
That is the work behind Contexta. The screens make it look like a reading product. The QA makes sure it stays one.
