Skip to content

How to Compare Two Simulations

TL;DR
Run two simulations, open one as the base, click Compare with other Simulation, and pick the second. Every tab (Job Details, Job Result, each Report) renders side by side with synced scrolling. Use it for champion vs. challenger, version vs. version, or train vs. holdout.

What is it?

Comparison is how you evaluate two simulation jobs against each other. You run two regular simulations and compare them side by side from one job's results page.

Common workflows this supports:

  • Champion vs. challenger: candidate model against the model in production.
  • Version vs. version: a refactored policy against its previous version.
  • Sample vs. sample: the same model on Train vs. Test, or on this quarter vs. last quarter.

Before you start

  • Two completed simulations exist. Both must have status COMPLETED (or PARTIALLY COMPLETED if you accept partial output).
  • You have read access to both simulations.
  • By default, you can compare jobs from the same object and same version. To compare across versions or across objects of the same type, use the switch arrow in the comparison picker.
  • For an apples-to-apples comparison, both simulations should run on the same population. The simplest way is Prespecified Ids (see Run a Simulation).

How to Compare

Step 1: Run the base simulation

Run a simulation on the first object. This is your base job: it will appear on the left side of the comparison view.

Step 2: Run the simulation you want to compare

Run a second simulation on whichever object/version you want to compare against. It can be:

  • The same object, different version (champion vs. challenger).
  • A different object of the same type (Model A vs. Model B).
  • The same object, different sample (Train vs. Test).

For a meaningful comparison, select the same reports on both simulations so the same reports render on each side.

Step 3: Open the comparison view

  1. Open the base simulation from the Jobs tab.
  2. Click Compare with other Simulation at the top of the page.
  3. The job picker opens. By default it lists completed jobs from the same object and same version. To pick a job from a different version or a different object of the same type, click the small switch arrow next to the picker.
  4. Select the simulation to compare. The page reloads in side-by-side mode.

Step 4: Work in the comparison view

The base job is on the left; the chosen job is on the right.

  • Every tab (Job Details, Job Result, and each Report tab) renders both jobs side by side.
  • Scrolling is synced: scrolling one pane scrolls the other, so equivalent sections stay aligned.
  • From the comparison header you can:
    • Switch the job on either side by changing its dropdown selection.
    • Swap the left and right jobs.
    • Exit the comparison from the kebab menu (⋮) at the top right to return to the single-job view.

Reading the Comparison

The comparison view doesn't add new analytics on its own. It puts the two jobs' existing outputs next to each other.

  • Job Details side by side: shows how the two jobs were configured. Useful for spotting that the difference you're seeing is really a sample/filter/version difference rather than a real model difference.
  • Job Result side by side: row-level outputs for each. Download either pane as CSV/Parquet from its own toolbar.
  • Report tabs side by side: the same report renders for each job, so you can compare the AUC, KS, decile table, confusion matrix (for Models) or approval rate / segment distribution (for Policies) directly.

To make the comparison meaningful, build reports that explicitly compare two jobs (PSI, CSI, swapset). See Enable custom comparison in Build a Custom Report — comparison-enabled figures get access to base_raw_output, compared_raw_output, base_entity, and compared_entity so you can compute true diff metrics.

Tips

  • Lock the population first. If both simulations are sampled differently, you can't tell whether a metric change is real or sampling noise. Run both on the same Prespecified Ids list when possible.
  • Use Custom Comparison reports for proper diffs. A side-by-side AUC of 0.82 vs. 0.79 tells you which is better; a PSI report tells you which features moved to cause the difference.
  • Save the comparison link. The URL captures both job IDs, so you can share it with reviewers or attach it to an approval request.

What's next

  • Run a Simulation on Prespecified Ids if you need to set up a fair comparison.
  • Build a Custom Report with Enable custom comparison turned on for figures that need to read both jobs at once.