Jev was 0.89 sure, and wrong: 4 lessons from running it in production

Jev is all over AI Twitter right now.

It's a model that can't chat and can't write. It only answers multiple-choice questions. TypeSafe, the company behind it, announced a $40M seed round on September 15. Daniel Agrici built an open-source SEO audit tool on it, Jev SEO, and it passed 100 GitHub stars within days.

I build Loomaly, a website SEO audit tool. Its page-by-page judgments, internal link suggestions and conversion checks all run on Jev.

You've probably read the beginner guides by now. This post covers what they leave out: what running Jev on real websites is actually like.

The short version: Jev is worth using, mostly because it forces you to decide who gets the final say. Almost every problem we hit ended up fixed with a line of code.

Before Jev, it was bad

The first version of Loomaly had a large language model read each page and write its own audit.

Slow. A page took 53 to 170 seconds. One day we ran 190 audits and 163 of them timed out.

Unstable. Audit the same page three times, get three different lists of problems. It also made things up, like reporting low colour contrast when we had never given it any colours.

Expensive. So expensive that each scan only looked at 25 pages and guessed the rest.

A user would see a problem, go and check it, find it wasn't there, and stop trusting the rest.

After Jev

A judgment takes one to two seconds and costs about $0.00003 per page. That's cheap enough to judge every page instead of sampling. A full audit of popmart.com costs about one cent in Jev calls.

The results held still, too. We ran the same 153 judgments twice and got 150 identical answers.

Answers Jev isn't sure about are dropped: not shown, not counted in the score.

Internal link suggestions went from 14% accepted in the first test round to 80% in production.

Here's what we tripped over on the way.

Lesson 1: high confidence doesn't mean right

The thing people like most about Jev is that it tells you how sure it is.

That's true. Our data says the number is less useful than you'd think.

We asked Jev whether a sentence should get an internal link, then checked its answers against the suggestions people actually accepted and rejected:

Accepted: average confidence 0.619.

Rejected: average confidence 0.623.

Basically the same.

The most confident suggestion in that round was at 0.89. The sentence was "What are the alternatives to Google Analytics?", to be linked to the site's own article on Google Analytics alternatives. Right target, but the sentence was an H2 heading. No editor puts a link in a heading.

Jev also answers a second question about every suggestion: "how good is this anchor text?", scored 0 to 2. In a later round of checking, that score did pull apart: accepted averaged 1.63, rejected 1.39. So our free check now only shows suggestions scoring 1.4 or higher.

Don't go by confidence alone. Check it against real outcomes and keep the number that actually tells right from wrong.

Lesson 2: it can only judge what you show it

The first time we ran it on popmart.com, Jev said 168 pages were "too thin". On another run it said 56 live product pages gave shoppers "no next step".

Sounds professional. All wrong.

Those product pages load their content with JavaScript, so we had fetched empty pages. And we never passed it the page's buttons, like "Add to cart". It was judging a blank sheet.

The fix was two lines of code:

Under 150 characters of text, we don't ask whether the page is thin. We report that the page is empty to crawlers that don't run JavaScript.

If we didn't capture the page's buttons, we don't ask about a next step.

Daniel hit the same thing in Jev SEO: Jev twice suggested deleting perfectly good articles. His fix was also one line of code: no page over 600 words gets removed, whatever the model says.

Let the model score and rank. Keep the final say in code.

Lesson 3: how you ask decides what you get

Jev only answers multiple choice, but how you write the question matters a lot.

Don't ask in the negative. "Is this page missing author information?" is the most expensive kind of mistake in our records. Our code now blocks negative questions before the call and asks "Does this page show author information?" instead.

Don't hand it raw numbers. Jev reads numbers unreliably, so we work out things like page length in code and describe them in words.

One page per question. An outside benchmark found that packing many rows into one judgment drops ranking accuracy from 0.97 to 0.51.

Lesson 4: our first production run, 40 of 50 suggestions were links the site already had

This one was embarrassing.

In testing, the acceptance rate for internal link suggestions climbed from 14% to 78%. Then we went live, and 40 of the first 50 suggestions were links the site already had.

The test script and the production code read existing links differently, and testing never showed it.

We threw that round away. After the fix, acceptance was 80%. Now the code guarantees that a link that already exists, or a suggestion someone dismissed, never comes back. Suggesting a link that's already there is the fastest way to lose someone's trust.

Most of the climb from 14% to 80% came from code ruling out things Jev shouldn't be asked about. Image captions aren't body text (14 of 26 rejections in one round), neither are table-of-contents entries or headings, and an anchor can't cut a phrase in half.

If you want to use Jev

Decide who has the final say. Counting goes to code; Jev only answers the questions about meaning that code can't.

Calibrate against real outcomes. Don't trust confidence on its own; keep the score that predicts what happens.

Give code the veto. Anything irreversible, like deleting a page, shouldn't rest on a probability.

Look at what it actually read. If the page was empty, even the best model is guessing.

To see this running on your own site: loomaly.com/check/yourdomain.com. Free, no sign-up.

Credit: Jev SEO is Daniel Agrici's open-source project (github.com/AgriciDaniel/jev-seo, MIT). Jev is made by TypeSafe. Numbers come from Loomaly's design notes, calibration notes and production runs in September 2026; the 0.97 → 0.51 figure is from an outside benchmark.