I set my AI fraud threshold to 0.98. It paid out nothing

The problem

This is the payout queue of our shop. Sellers ask for their money, and a house rule decides: an account older than thirty days gets paid. Look at the list. Everything is approved. Everything. That rule has never once said no. And this is the seller it just paid. The account is well over a year old with a hundred and ninety two orders behind it, so the rule was happy. But the bank account was changed eight hours ago, and the payout is requested from Vietnam through an anonymising proxy. That is not the seller. That is somebody else holding the seller's password.

The solution at a glance

So we ask a model instead. The same local model as last episode, running next to the app, no key and no third party. One typed question, and back comes a number: just under zero point six. Not a yes, not a no. A probability that this request is legitimate, in a quarter of a second.

Setup

One number tells us nothing yet. We need labelled requests, and here is the uncomfortable part: nobody may publish real payout data, so this repository generates a hundred of them. The rules are in the generator, and the model never sees them. That makes this a fair test of the model and a poor test of your business. Measure on your own before you trust any threshold.

Building it

Now the decision. Three outcomes, not two: pay out, block, or put it in front of a human. The probability picks one. And for the two thresholds I will do what everybody does the first time, which is to guess them. Point nine eight feels safe for paying money out, and below point seven we stop. Let us run those thresholds against all hundred requests. And there it is. Zero paid out. Not one. Twenty six blocked, seventy four on a human desk. My point nine eight did not make the system careful, it switched the automation off completely, and nothing in the code would have told me.

Refactoring

The reason is that these probabilities are not calibrated. Nothing in our hundred ever reaches point eight nine, so point nine eight can never happen. The threshold is not a feeling, it is arithmetic: a wrong approval costs the payout, a review costs twelve euros of somebody's time. Here is every threshold with its price. Point eight five: twenty two paid out, none of them wrong, nine hundred and thirty six euros. Below it a fraud slips through and it costs four thousand. Above it we pay for reviews the model had already decided. So we read both numbers off that table instead of inventing them. Same code, same model, same hundred requests. Twenty two paid out automatically, none of them wrong. Seventeen blocked, none of them a real seller. Sixty one go to a human. Thirty nine percent without a person, and zero wrong decisions at both ends. That is a much smaller number than any vendor would print, and it is one I can defend. And live it looks like this. Three requests arrive, each one goes to the job queue from episode three, the model answers, and the compliance board updates itself through Turbo Streams from episode four. Green, amber, red. Nobody reloaded anything.

Gotchas

Three things that will bite you. The first one is in this file. If the model is down, the request has no probability, falls past both thresholds and gets blocked. A silent model is not a fraud signal, so it goes to a human. The second one is the wording. Same seller, a real one who simply changed bank. Ask whether the request shows signs of fraud, and he lands on a human desk. Ask whether the request is legitimate, and the same model blocks him and alerts security. One sentence rewritten, a customer suspended. The question is part of the system, so measure the question, not just the model.

Things to remember

And the third: the answer from this model carries a field called confidence. It is the distance from one half, not the probability of legitimate. Check that against an approval threshold and you will pay out exactly the requests the model was most sure were fraud.

Wrap-up

The generator, the hundred requests, the recorded run and the cost table are all in the repository, so you can move the thresholds and see what it would have cost. Next episode we stop asking one model one question, and let a team of bots work together on something every repository needs: a changelog, and the announcement that goes with it.