Don't trust the model card: three decision models on real data

The problem

A hundred real issues from the Rails repository, one question: is this about Active Record? This is how we all do it today. A prompt, a JSON schema so the model does not ramble, and a parse call that trusts the answer is really JSON. And look at what comes back: true or false, and nothing else. No idea how sure the model was. That matters, because a verdict without a confidence is useless for automation. Ten issues, ten verdicts, and every one of them arrives looking exactly as sure as the next. You cannot say: label this one for me, but let a human look at that one. There is nothing in the answer to sort by. Everything looks equally certain, and some of it is wrong.

The solution at a glance

There is a different kind of model for exactly this. They do not write text at all. You send a state and typed questions, and you get values with probabilities. Two of them launched three days apart in September: Jev, hosted by TypeSafe, and Laya, open weights from Convai, Apache two point zero, runs on your own machine. Both model cards promise a lot. So let us measure them.

Setup

The measurement stands or falls with the data. We took a hundred issues that the Rails maintainers themselves labelled by component. Their label is the truth, not my opinion. That is the whole trick, and you can do it with your own tickets, your own commits, your own support mail.

Building it

Laya first, because it costs nothing and needs no account. An environment, one install, four hundred and twenty one million parameters, and the weights live on your disk. No key, no gateway, no rate limit. Twenty lines. Load the model once, then one call per issue: the state is the issue text, the question is typed, and the answer is a probability. Note the type: noul, not boolean. That is their name for a yes-no proposition, and it is the kind of detail a model card buries. Return details gives us the probability instead of just the value, and that number is the whole reason we are here. We keep the milliseconds too, because the model card makes a claim about those. The count is an argument, so we can run fifteen now and a hundred later. And do not call the file laya dot py, or it will import itself. Fifteen issues, so you can watch it finish. On a laptop, with the network off if you like. The model card says thirty three milliseconds per decision. On a graphics card, yes. On my processor it is around six hundred, and that is still fast enough for a queue that nobody is watching. The full hundred is in the repository, and that is the run every number in this episode comes from.

Refactoring

Now the comparison. The report reads all three runs, counts the hits against the maintainer labels, and then does the part that actually decides whether you can use a model: it walks the confidence. How much runs without a human, and how much of that is right. The first line is the honest part: our live runs were short, so these are the recorded hundred, the same file you get from the repository. And there is the answer the model cards do not give you. On this task the hosted models are close: ninety one and ninety two out of a hundred. Laya gets seventy six. But look at the lower half. Jev is confident on ninety of the hundred and right on eighty six of those. Laya only claims certainty eighteen times, and it is right every single time. A weaker model that knows when to stay quiet is not a bad deal.

Gotchas

Three things before you copy this. The numbers on a model card come from their benchmark, not from your data: Laya claims eighty four percent, we measured seventy six on our task. The hosted one has queues: getting a hundred answers out of Jev took nineteen hours of retrying, because everybody is trying it right now. And the local one costs nothing per call, but it costs you a couple of gigabytes on disk.

Things to remember

And the method matters more than the model. A hundred labelled examples tell you more than any benchmark. Measure on your data, judge the confident subset instead of the average, and pick the threshold that decides what a human still sees. After that the model is a part you can swap.

Wrap-up

Everything is in the repository: the hundred issues, the four scripts, and both recorded runs, so you can check my numbers instead of believing them. Next episode we stay with Laya and go through a few use cases for a decision model that runs on your own machine.