Benchmark - Harumi vs a raw model
How Harumi AI Chat compares with the same model called directly on operations research problems, and how each number is measured.
What is compared
Each problem is given to two contestants that use the same model:
- Harumi is Harumi AI Chat as a customer gets it. It runs inside a real project, can write and run code in the project's sandbox, read and write the project's files, and fix its own mistakes before it answers. The knowledge base is on, with the problem under test removed from what it can retrieve, so it cannot look the answer up.
- Raw model is one call to the same model with the same problem text. It has no tools, no sandbox and no second attempt: it writes the code once and that is the answer.
Both are asked for the same thing, a Python function using Gurobi that returns the optimal objective value, and both are graded the same way. The only difference between the two is the platform around the model.
Results
Numbers below come from the most recent published run. The headline counts tiers 1 to 4; the sanity tier is shown in the per-tier table but kept out of the headline, because those word problems are old, small and very likely seen by every model during training.
Headline
No benchmark run has been published yet. This page fills in from the first full run; until then it shows no numbers.
"Harumi only" and "Raw model only" count problems, on the same problems for both, where exactly one of them was right. The p-value is an exact two-sided sign test on those problems: how likely a gap that large would be if neither contestant were better.
By tier
No benchmark run has been published yet. This page fills in from the first full run; until then it shows no numbers.
A tier marked with an asterisk is reported but not part of the headline.
Every problem
No benchmark run has been published yet. This page fills in from the first full run; until then it shows no numbers.
Problems are listed by path only. The text of held-out and licensed problems is not published.
How a solution is graded
A solution is graded by running its code, not by reading it:
- The code runs in a Harumi sandbox with Gurobi, the same environment for both contestants, on the problem's input.
- The objective value it returns must match the known answer within a relative tolerance of 0.01% (or an absolute 0.000001).
- If the code ran a solver, it must have ended at an optimal solution.
- For problems built to have no solution (infeasible or unbounded), the solver's final status must be that one. A solution that quietly relaxes the problem to get a number is wrong.
- For conversations with follow-up requests, every step must be right.
The time is the wall-clock time from the first message to a graded answer. The cost is estimated in US dollars from the model's token prices, counting every model call the run made (including prompt-cache reads and writes, and any call Harumi made to a stronger model for advice). It is not the same number as credits, which are what your plan is billed in.
The problems
| Tier | What it contains | Why it is there |
|---|---|---|
| Sanity | Small linear and integer word problems | A regression check. Models do well here with or without help, so it is not in the headline. |
| Established hard problems | Longer problems from classic operations research collections, in scheduling, lot sizing, routing and network design | Where a first attempt often needs a correction. |
| Public benchmarks | IndustryOR, MAMO ComplexLP and OptMATH-Bench | Recognised outside Harumi. Only problems whose answer was independently reproduced are used (see below). |
| Scaled instances | The same kinds of problems regenerated 10 to 100 times larger | The answer depends on reading a lot of data correctly. It does not stress the solver: these solve in milliseconds. |
| Held-out | Problems written by Harumi and never published: data in CSV or JSON files, problems with no solution, and conversations that change the problem | Cannot be in any model's training data, and closest to how customers use Harumi. |
Public benchmark labels
Public benchmarks are known to contain wrong answers. A model marked wrong for disagreeing with a wrong label would make the comparison meaningless, so each public problem was solved independently by a strong model, with a few attempts (the table shows how many), and kept only if one of them reproduced the published answer.
No benchmark run has been published yet. This page fills in from the first full run; until then it shows no numbers.
This keeps the problems that could be verified, so the public part of the benchmark leans easier than the original collections. The dropped count is shown so you can see how much was removed.
What this does not show
- One model at a time. Every comparison holds the model fixed. It says what the platform adds to a model, not which model is best.
- No retry loop for the raw model. The raw contestant gets one attempt on purpose, which is what calling a model directly means. A homemade loop that feeds errors back would close part of the gap; that variant is not measured here.
- A snapshot. Models and the platform change. The run date and the build that produced the numbers are recorded with the data.
- Problems with a known answer. Open-ended modelling work, where no single number defines success, is not covered.
- Estimated cost. It follows list token prices and does not include sandbox compute time.