How to Compare Two Simulations

TL;DR

Run two simulations, open one as the base, click Compare, and pick the second. Every tab (Job Details, Job Result, each Report) renders side by side with synced scrolling. Use it for champion vs. challenger, version vs. version, or sample vs. sample.

What is it?

Run two regular simulations and compare them side by side from one job’s results page.

Common workflows this supports:

• Champion vs. challenger: candidate model against the model in production.
• Version vs. version: a refactored policy against its previous version.
• Sample vs. sample: the same model on Train vs. Test, or on this quarter vs. last quarter.

Before you start

• Two completed simulations exist. Both must have status Completed (or Partially Completed, though the two sides may then cover different records).
• You have read access to both simulations.
• For an apples-to-apples comparison, both simulations should run on the same population. The simplest way is Prespecified Ids (see Run a Simulation).

How to Compare

You compare two completed simulations (see Before you start). The one you open first is the base, on the left; the second is the compared job, on the right.

Open the comparison view

1. Open the simulation you want as the base (the left side) from the Jobs tab.
2. Click Compare at the top of the page.
3. The job picker opens. By default it lists completed jobs from the same object and same version. To pick from a different object, version, or job of the same type, click the switch arrow () next to the picker.
4. Select the simulation to compare. The page reloads in side-by-side mode.

Work in the comparison view

The base job is on the left; the compared job is on the right.

• Every tab (Job Details, Job Result, and each Report tab) renders both jobs side by side.
• Scrolling is synced: scrolling one pane scrolls the other, so equivalent sections stay aligned.
• From the comparison header you can:
  • Change the compared job on the right by picking a different object, version, or job. The base job on the left is fixed - to compare against a different base, exit and reopen from that job.
  • Swap the two sides with the swap icon () so the compared job becomes the base.
  • Exit the comparison from the kebab menu (⋮) at the top right to return to the single-job view.

Reading the Comparison

The comparison view lines up each job’s existing outputs so you can read them together, side by side.

• Job Details side by side: shows how the two jobs were configured. Useful for spotting that the difference you’re seeing is really a sample/filter/version difference rather than a real model difference.
• Job Result side by side: row-level outputs for each. Download either pane as CSV/Parquet from its own toolbar.
• Report tabs side by side: the same report renders for each job, so you can compare the AUC, KS, decile table, confusion matrix (for Models) or approval rate / segment distribution (for Policies) directly.

A side-by-side shows the two jobs next to each other but doesn’t compute anything across them. For true diff metrics, PSI, CSI, a swapset, build reports that read both jobs at once. See Enable custom comparison in Register Reports. Once set up, those comparison reports populate automatically when you open the view.

What’s next

• Register Reports with Enable custom comparison turned on for figures that need to read both jobs at once.