Journal · 4 November 2026
How we test Dutch and English AI
A training run that finishes without an error does not prove that a model has improved. The measurement must be repeatable and continue to answer the same question.
Dutch and English separately
Sabine’s first product languages are Dutch and English. We measure each language separately, so an improved average cannot hide a regression in one language. Results are tied to the relevant model version and test settings.
Keep the test text consistent
Training text and evaluation text are kept separate. A fingerprint of the evaluation tokens shows whether two runs used the same test text. If that text or the evaluation settings change, the results must not simply be presented as one progress curve.
Perplexity is not the whole product
Perplexity describes how well a model predicts the next tokens in the test text; lower is better on that measure. It does not by itself prove factual accuracy, safe tool use or usefulness for real tasks. Those need additional content and user testing.
No winner without repetition
Experimental variants are evaluated alongside a baseline and appropriate controls. Data streams, settings, compute time and randomness can explain differences. A short smoke test only demonstrates that the pipeline runs. A quality claim requires multiple independent repetitions and verifiable results.