Compare versions

What a comparison shows

A comparison puts 2 to 4 runs of the same project side by side, usually runs of different versions (branches) of your model. It compares the code, not the data: the idea is that the inputs stay the same and only the algorithm changes.

Each run gets a column with:

• its status, version (branch), and short commit;
• the numbers you declared as KPIs, with the change against the baseline;
• its runtime, and how it was run: when it started, who or what triggered it, the environment, and the command;
• its dashboard, drawn next to the others.

Declare the results to compare

Tell Harumi which values in your run's output/output.json are worth comparing by adding [[output.kpi]] entries to harumi.toml. Without them the page still opens, but it only shows the dashboards and says no KPIs are declared. The starter template already declares three:

[[output.kpi]]
key       = "objective"
label     = "Total cost"
direction = "min"
unit      = "USD"
primary   = true

[[output.kpi]]
key       = "solver_status"
label     = "Solver status"
direction = "none"

[[output.kpi]]
key       = "gap"
label     = "Optimality gap"
direction = "min"

objective, solver_status, and gap are the recommended keys for optimization models: solvers with a time limit or a heuristic don't always return the same answer, so the status and the gap tell you how far to trust the objective.

Each run reads the KPIs from the harumi.toml at its own commit, not the branch's latest one. Across versions they are combined by key:

• A version that doesn't have a KPI shows — for it.
• If two versions disagree on a KPI's direction, or on which one is primary, the page shows a warning and uses the baseline's declaration.

A harumi.toml that can't be read (for example, a syntax error) doesn't block the comparison. The page warns and shows that column without KPIs.

Open a comparison

From a project's dashboard, click Compare with… next to the run picker. It opens Compare versions with the latest successful run on the live branch as the baseline and the run you were looking at next to it.

The page is a row of full-height columns, one per run. A column with no run in it says Add run: click anywhere on it to pick a run from the list (you can search it by version, commit, or run id), and that run becomes the next column. Add in the top bar does the same. You can compare up to 4 runs, and the x on a column takes that run out.

Nothing is saved: the runs you picked are in the page's URL (/projects/<id>/compare?runs=…). Use Copy link to this view to send the comparison to other members of the project.

Read the result

• Baseline. The first column. Every change is measured against it, in green when it is better and red when it is worse. Use Move left and Move right to reorder; whichever run ends up first becomes the baseline.
• Winner. The completed run with the best value of the primary KPI. On a tie, the earlier column wins. Failed runs, and runs with no value for the primary KPI, keep their column but can't win. If nothing qualifies, the page says why: for example, that no primary KPI is declared.
• Differences. A dot next to Triggered by, Environment, or Command means it differs from the baseline, which is often the reason two results differ. Turn on Differences only (it needs two runs) to hide the dashboard rows that are the same in every run.

Did the runs use the same inputs?

A comparison is only fair if every run read the same data. Each run records a fingerprint of its inputs when it starts: the files in inputs/ and the CSVs fetched from your data sources. The page compares them and tells you one of three things:

• Every run used the same inputs.
• The runs didn't use the same inputs, followed by the files that differ. The results may differ because of the data, not just the code.
• Can't tell whether the runs used the same inputs. At least one run has no record of its inputs. Runs from before this was added don't have one, and they are not filled in afterwards.

When the results may be out of date

The page shows a warning, with a Refresh comparison button, when what you are looking at may not be what those versions give today:

• a run's version has newer changes than the run, or no longer exists;
• the runs didn't all use the same inputs, or at least one has no record of its inputs.

Refresh a comparison

To compare versions that haven't been run yet, to bring an out-of-date comparison up to date, or to be sure they read the same data, click Refresh comparison (in the top bar, or in the warning above). The versions of the runs you are comparing are already ticked, in column order; change them if you like. You can tick 2 to 4 versions, branches or tags. The first one ticked is the baseline. A tag is looked up once, when the comparison starts, so every run uses a fixed commit even if the tag moves meanwhile.

The versions run one after another, not at the same time:

1. The baseline fetches its inputs as usual and keeps a copy of them.
2. Each next version reads that copy instead of fetching its own, so every run gets exactly the same data.

The page shows Running 2/4 while it goes, with a Cancel button. Cancelling stops the run in progress and the rest never start; the page shows the runs that finished. A version that fails still gets its column, and the rest carry on.

Things to know:

• Because the runs go one at a time, a fresh comparison uses your one active run until it ends. Starting another run in the meantime is refused, and if you manage to start one in the moment between two of its runs, the comparison stops there.
• When a compared run came from a tag, the page flags it if the tag has since moved or been deleted, and the dialog opens with that tag already ticked.
• The comparison stops, and says why, if the uploaded files are too large to share between the runs, if a run isn't allowed to start (for example, not enough credits), or if you lose access to the project.
• A version that queries a data source the baseline doesn't declare fails with a message saying so, since there is no shared copy for it to read.

Ask Harumi AI Chat

Ask the chat to compare runs. It answers with a card that lists each version with its primary KPI and the winner, and Open comparison takes you to this page. If the project has no KPIs declared, the chat can propose them for you to review.

Share a view

A view is a run, a dashboard, or a comparison. Its address holds everything needed to open it again, so you can send it:

• Copy link to this view (the link icon in the top bar) copies the link to what you are looking at. Send it to other members of the project, who need access to open it.
• In Share, turn on Link to this view and copy a public link: it opens a visitor straight on that run, dashboard, or comparison. A link whose permissions can't open the view says so, and its copy button is off.
• A visitor on a public link has the same button. It is the link they came in by, pointed at the view they are on.

If a public link names a run it can't show (the link doesn't allow run history, or the run is gone), the visitor sees the latest run, with a note. A comparison leaves out the runs the link can't show.

Let a public link compare runs

A public share link can compare runs when you switch on Comparisons in the link's settings (harumi share update <LINK_ID> --comparisons in the CLI). It is off by default and needs Run history, because without past runs there is nothing to compare. Visitors can pick, drop, and reorder the runs the link shows (the live version's runs only), but can't start a fresh comparison.