Decision models promised instant, cheap document classification. We tested them against real claims documents. Here's what held up, and what actually worked.
Every claims operation runs on documents. Before anything useful can happen with a document, someone or something has to answer a simple question: what is this? A demand letter, a vendor invoice, a police report, a medical bill, a summons. Get that wrong and everything downstream is wrong with it. The letter goes to the wrong queue, the deadline doesn’t get tracked, and the signal never fires.
At Isomer, classification is the first step in reading a claims organization’s communications. A carrier can have hundreds of document types, many of them overlapping, and every one of those documents has to land in the right place. So when a new class of AI model showed up promising instant, cheap classification, we paid attention.
This fall, “decision models” got a lot of attention. The pitch, popularized by TypeSafe’s Jev, is that instead of asking a large language model to write an answer, you ask it to pick from a fixed set of options and read its choice straight from the model’s internals. That makes it fast, cheap and deterministic. For a problem like ours, it sounded close to perfect.
We didn’t want to take that on faith, so we ran a spike. We took a random sample of real claims documents, built a careful set of reference labels, and tested every approach we could find against the same documents. That included Jev itself, the open-source projects inspired by it, small models running locally, fine-tuned classifiers, and the large hosted models we already use.
Jev delivered on the speed and cost. It was the cheapest and fastest option we tested. It just wasn’t more accurate than the classifier we already had in production.
That wasn’t the end of it. A few other things ruled it out for us:
The open-source alternatives didn’t fare better. The small local models were too small to tell hundreds of overlapping types apart. The most polished open-source project ships only models we can’t use, which brings me to the part of this that matters most to our customers.
In insurance, accuracy is only half the question. The other half is where the model runs, who operates it, and where it came from. Carriers are accountable for the AI their vendors run on their behalf, and regulators increasingly expect them to answer those questions.
That’s why we hold ourselves to a sovereign AI standard: customer data stays in environments our customers can trust, with vendors under agreements that cover it, using models whose origin we can stand behind. Several of the tools we tested failed that bar before we ever looked at their accuracy. A model that’s fast and cheap but can’t pass a carrier’s security review isn’t a shortcut. It’s a dead end.
That standard is a big part of why we deploy where our customers already are, including in their own Azure environment.
The approach that won was less exotic than the hype, and it borrowed the best idea from it.
Every time a language model picks an answer, it also knows how sure it is. Most applications throw that information away. We kept it. We have a fast, inexpensive model classify every document and report its confidence. When it’s confident, which is most of the time, we take its answer. When it isn’t, we send just that document to a more capable model to think it through.
The results:
The confidence score turned out to be the real prize. On the documents where the fast model was most confident, it was right almost every time. That gives us something better than an answer: it tells us which answers to trust, and where a person or a stronger model should take a second look. In claims, knowing when you’re unsure is worth as much as being right.
The method matters as much as the result. A few principles we hold to, well beyond document classification:
We’re rolling the new approach into production now, with confidence logged on every document so we can tune it for each customer over time.
The broader takeaway is the one we keep coming back to at Isomer. New models will keep arriving, and some of them will be genuinely useful. Our job is to test them honestly against real claims work, keep the ideas that hold up, and only ship what we can explain, audit and stand behind.
Get new posts by email.
New posts on claims risk and AI in claims. Unsubscribe any time.
Every claim has a clock running before it's entered into the claims system. From the moment a demand letter, medical bill or attorney email lands, its value to the carrier starts to decay, and that decay is measured in dollars.
Core-system agnostic deployment runs inside the insurer's own Azure tenant, so claim data never has to leave the environment the insurer already governs.