The most expensive if
About two months ago, Jev was released, a “system one AI model” (as in Thinking, Fast and Slow: fast, intuitive, no thinking involved). It is basically a classifier that you can give a question (in the form of a written prompt) and a set of options and it will tell you what it thinks is the right answer – and it even gives you a nice little confidence score11 The paper “LLM-as-a-Jev” shows very nicely how a standard LLM can be turned into a decision model like Jev!. However, the real selling point is its price: output tokens (basically the decision) are “too cheap to meter” according to its developer, TypeSafe AI, and input tokens only cost $42 for a billion tokens (yes, billion, not million).
So the vibe coding crowd did what they do best and vibe coded some incredible applications that use Jev at their core. The best of the best of these amazing use cases are forever enshrined in a GitHub repository called “Awesome Jev use cases”, which is a gold mine for inspiration22 I need to warn you though: all these projects just lead you to X posts. If you want to keep away from that website, don’t click the links!:
- A Jev calculator that can calculate different numberrr… Wait, no, that doesn’t make sense. So, you basically input “citizens in Macedonia” and just divide that by, ehm, “number of limbs on a dog”…? And then multiply that by the “number of planets”? There are tons of really awesome use cases, you just have to select the right one.
- Just give Jev $10,000 and let it trade! You can see that… well okay, maybe that is also not the best use case. Don’t do that. Don’t let Jev trade autonomously.
- Well, but the really bestest use case is of course: let Jev doomscroll your feed for you. It decides which posts you should read and filters out “spam” posts, like, for example… the post where the official TypeSafe account announces Jev…? But it only costs around a cent! So, yeah, you could do that. By the way, that is one of the “most popular demos” from the GitHub repository.
Okay, I think I did enough market research and get the vibe: it doesn’t really need to make sense. Whenever there is a decision to be made, let Jev just handle it and don’t think about it too much. That mindset also fits the model, because the name Jev comes from the Jevons paradox, which basically says:
If AI gets cheaper, the AI slop bros are gonna use it even more, regardless of whether it makes sense.
I wanted to finally get my feet wet with this brand new technology, so I followed that mindset to its logical conclusion and asked myself: what is the most ubiquitous decision that is made quintillions of times every second all over the world? The answer is clear: Branch Prediction!
What is a branch predictor?
In a nutshell, the CPU (the small rectangular hot part in your vibe coding machine) is like a pipeline: each instruction has to be prepared ahead of time so it can load data, shuffle around some registers and make it all cozy, and when the time comes and the instruction has to be executed, everything is warm and ready. And because all that warming up takes a while, the CPU doesn’t wait for one instruction to finish before it starts on the next one: there’s always a whole queue of instructions in the pipeline at the same time, each one a little cozier than the one behind it.
Put yourself in the shoes of the CPU for a moment: whenever there is a conditional jump, a.k.a. a branch (every if, for and while has one), your pipeline runs dry, because you would need to know which way the condition resolves in order to continue filling it up. But now you are very special and have the brilliant idea to just guess whether the branch gets taken. So you start loading up the pipeline for that prediction. If the guess is right, your psychic abilities have already paid off. If it’s wrong, your precious work is discarded and you load the right instructions instead. That’s just what you would have had to do anyway, so a wrong guess costs you (almost33 If you ignore that one tiny little oopsie called Spectre.) nothing!
Given that neither the Daily Karmic Number nor the most recent horoscope is available to the average CPU, there are actual strategies to guesstimate whether a branch gets taken. In practice, the branch predictor gets to see the location of the branch and it knows the outcomes of previous branches. With that, the classic textbook predictor, the 2-bit counter, keeps a counter for every branch and just counts it up whenever that branch is taken and down whenever it isn’t (capped between 0 and 3). If the counter is currently at 0 or 1, it predicts “not taken” – if it is at 2 or 3, it predicts “taken”. The strategy is basically “I’m just doing what was correct last time”, but with a small buffer, so the winning streak is not interrupted by an occasional single untaken branch.
In reality, modern CPUs use far more advanced branch predictors that try to squeeze out every possible bit of information from these very limited inputs. In fact, there are even branch predictors that use small neural networks in order to find every hidden branch pattern inside your badly written while loops. AMD advertised its first Ryzen CPUs in 2017 with a “Neural Net Prediction”. So, the idea to use AI to predict branches has basically already been tested in production. Speaking of neural networks, how about we use an unartificial unintelligence (yes, I mean you) to do it? You get to see exactly what a branch predictor sees: which loop it is and what it did before44 Well, I’m giving you a little head start: an actual branch predictor doesn’t know how long the list is.. Will you have what it takes to be a real branch predictor?
Does the loop go around again?
The inner loop of a bubble sort on 12 numbers
You 0 / 02-bit counter 0 / 0
You got of 78. The 2-bit counter got 66.
Jev, you’re hired
Alright, I like your effort, but we really need something faster, and compared to a human or any LLM, Jev is much faster.
In order to benchmark how well my advanced AI branch predictor is doing, I wrote two reference algorithms: a bubble sort that sorts 12 numbers, and a binary search that looks up 20 random numbers in a sorted list of 64. This sums up to 417 branches in total that I recorded by replacing every conditional with my own branch recording function:
def bubble_sort(a: list[int]) -> list[int]:
n = len(a)
i = 0
while br(site="bubble:outer", cond=i < n):
j = 0
while br(site="bubble:inner", cond=j < n - 1 - i):
if br(site="bubble:swap", cond=a[j] > a[j + 1]):
a[j], a[j + 1] = a[j + 1], a[j]
j += 1
i += 1
return a
def br(site: str, cond: bool) -> bool:
# The real br-function also saves local variables, more on that later
TRACE.append({"i": len(TRACE), "site": site, "taken": cond})
return cond
So after the runs, I can inspect every decision and which path was taken. Now Jev comes into play: I put it into noul-mode (whatever that means) and prompted it with “A CPU is about to run this branch. Will the branch be taken this time?”. For every recorded step of the algorithm, I built up some context containing the last 16 results of that branch and the last 16 results of all branches together55 That’s what real branch predictors look at too: the history of the branch itself (its local history) and the history of all branches (the global history), because branches often depend on each other. and let Jev handle the prediction.
The result was very disappointing – maybe I overestimated my and Jev’s system one intelligence: the results are within spitting distance of just using the 2-bit counter. In fact, it guessed exactly what the 2-bit counter guessed on 393 of 417 branches and didn’t see a single exit coming either. At least we are burning almost 17 thousand US dollars on one single second of CPU time66 The cost per second assumes a billion branches per second for one CPU core.. That’s quite a feat given how cheap these predictions are.
I thought maybe, just maybe, the prediction is so bad because Jev only sees the 16 previous results and not the whole history. Turns out, giving it too much context makes it worse: prediction accuracy dropped to 62.1%, but at least the price per second increased to $48,800.
To be honest, I was very disappointed with Jev, but couldn’t let the thought of a really accurate branch predictor go, so I switched gears and went for something more system-two-ish.
Maybe a bigger model?
I pulled out my credit card, loaded up my Claude vault with precious token money and started implementing an LLM-based version of the branch predictor. I went with Claude Haiku 5.5 (because my wallet was crying already) and also reverted to the 16 global and 16 local results as the given context. The results were just as disappointing as Jev’s, only more expensive.
Again, I invented a branch predictor with access to all the knowledge of the world inside a high-dimensional complex manifold that in the end just copies the homework of the 2-bit counter. The slight increase in accuracy by 1.4 percentage points can be confidently attributed to pure luck. At least it’s almost three times as expensive as Jev, so we’re making progress. But I was not at the end of my rope yet.
Let it read the code
Okay, what else can we try? Sure, let the LLM read the code that is being executed, so it can understand and reason about what must happen when. Well, I added the program itself, how its input is made, and every earlier branch – basically everything except the contents of the variables themselves, because that would be cheating, right? Right?
Sadly, even that doesn’t really make a difference: Haiku gets slightly worse at 69.3% and Jev gets a little bit better at 73.6%. With blind determination, I went to the bank and got a loan that I directly invested into more Claude API credits. I activated thinking and watched the tokens fly across the screen.
Wait. With thinking, at almost $550,000 a second, Haiku was still no better than Jev (and only marginally better than a 2-bit counter)? That doesn’t make sense, right (foreshadowing intensifies)? So I investigated a bit, played around as a branch predictor myself and pretty soon came to the conclusion that there is a theoretical limit. With only the history, about 78% is the most anyone can get (I did the math with my brand new Jev calculator), because every loop exit comes as a surprise. With the code, you can at least count the loops, which raises the limit to about 84%. But even with all the time and money in the world, without the contents of the variables nobody can know whether two numbers are about to be swapped, or whether the number we’re looking for is in the left or the right half of the list. Everything above that is only possible if you have the data.
So the verdict is: even with thinking and code, Haiku still can’t reason its way to the theoretical limit and falls around 10 percentage points short. Sure, we are still not out of options, but I really have to start raising some real venture capital in order to keep the business going.
To the MAX
The next step is max effort, and because the venture capital hasn’t landed in my account yet, I only ran Haiku on the 66 swaps of the bubble sort. Of course, without the actual contents of the list it still can’t predict every swap, but it can use the results of the earlier swaps and a bit of pure logic to deduce the outcomes. The result?
It ran out of tokens 28 times. So I doubled the token limit, but it still didn’t produce an answer 7 times. Alright then. Keep your secrets. But whenever it did answer, it made the same call as a perfect predictor (58 out of 59 times). It reasoned its way exactly to the theoretical limit, it just needed 2.6 minutes and a few cents per branch to get there. Note that in the score card I only compare the swaps, so the 2-bit counter baseline sits at only 53%. And we also most definitely broke our money-spending personal best! Now we are talking real AI innovation money. Everything below a couple million dollars is just a small experiment. One thing I left for future work is running it with Fable 5.1 at max effort – I still need to set up a GoFundMe for that.
Just give it the answer
I don’t know if you noticed, but 77.3% is not quite 100%. The hunt for the best branch predictor is not over. How can a branch predictor know exactly which outcome to predict? Well, a perfectly intelligent system that knows the code, the contents of the variables and all of the history of the world could easily say what the outcome of an if statement is. So let’s build that!
I gave Claude Haiku the code and the variables. It can compare the numbers itself and “predict” perfectly. Simple and foolproof, right?
I see my error: I disabled thinking again, which obviously hurt Claude’s feelings. It again performs worse than the 2-bit counter, with the answer basically screaming in the prompt. Oh well, hardware development is hard, but AI-accelerated hardware development is the hardest. At least when I let Jev see everything, it got 76.3%. Good on you, Jev, but that number still isn’t perfect. We must do better.
Activating thinking it is – but not at max, it can’t be that hard.
The results are in the right ballpark at 2.5 seconds per branch and 137 thousand dollars per second. We finally did it! 🎉
Well, there is this one miss: 9 > 2 was right there in the prompt, and Haiku said “no swap” with 97% confidence. But no worries, we need room for improvement to get more venture capital in the future, so that is actually the better result.
You might say, “Just use a CPU to run the parts before the branch, then you can predict it no problem!” But your argument has a fatal flaw: that CPU would also need a branch predictor to be fast and efficient. In contrast: an LLM is pure matrix multiplication and can be implemented completely branchless77 Except for the loop that generates one token after the other until it’s done. But that’s just one branch, how hard can it be?. Also, it runs in the cloud, so you don’t have to worry about all the computing that’s going on there. Just go with the vibes!
Offline mode
But again I hear you say: “One fatal flaw still exists! You cannot predict branches when you are not connected to the internet! What then, branchy boy?” First of all, no need to get offensive. You probably want me to say that the best branch predictor just runs the program. That all I need is a real CPU.
I say BLASPHEMY! Of course I thought of that and just downloaded Qwen 3.5 35B, hooked it up to my branch prediction code and there you go: AI branch prediction – even on an airplane!
Qwen even gets all of the branches right. Yes, that’s a full 100%! At NO COST (don’t tell the VCs)! And it just takes… a full minute per branch! Okay, it might not be the fastest CPU, but it will definitely be the one with the most advanced branch predictor to date! The next step would be to bring the whole CPU inside an LLM, but that would be kind of a nonsense project, wouldn’t it?