Evaluate
eval measures tracking performance from perception builds or saved detections.
For existing KITTI 2D predictions, use the saved-box workflow.
For perception-build evaluation, it streams an immutable materialized build
through the live tracker API. Select an authored experiment with --experiment or use
dataset and component flags as shorthand for a matching catalog experiment.
In either case, --build is optional: when omitted, BoxMOT first materializes
(or reuses) a canonical build compatible with the selected tracker and then
evaluates it. Detector, segmentor, and appearance-encoder inference happens only
during that preparation step, never during replay. Automatic preparation
publishes image references, embeddings when the resolved tracker configuration
uses appearance, and masks when the selected tracker or scoring requires them.
Setting use_embeddings: false in --tracker-config skips the embedding stage,
as do motion-only trackers such as SFSORT, unless GTA postprocessing requires
cached embeddings. Appearance-enabled replay requires
cached embeddings; the live API's image-to-ReID fallback does not run during replay.
The nine Python box trackers can additionally run EdgeTAM mask propagation
during replay with --edgetam. Guidance is off by default; the flag resolves
the default edgetam.pt checkpoint into ./models. Select another checkpoint
with --mask-guidance-weights. Weights alone do not enable guidance, and
--no-edgetam disables it even when a path is supplied. The
temporal mask guide
lists supported trackers and their association behavior. Change --tracker
while retaining --tracker-backend python --asso-func iou, AABB geometry,
and per_class=False. Install the official package with
uv sync --extra cpu --extra yolo --group mask-guidance (select --extra cu130
for CUDA); the full checkpoint downloads into ./models on first use.
The tracker YAML's edgetam.max_objects defaults to 96;
--mask-guidance-max-objects N explicitly overrides that cap. The mask
thresholds and prompt overlap gate also come from the tracker configuration.
Replay a tuned best.yaml with --tracker-config and --edgetam, using
--mask-guidance-weights again if a custom checkpoint was selected; see
guided tuning.
Replay loads source frames even with a cached build and defaults to one
sequence worker. See the
MOT17 ablation evaluation example
for installation, device selection, and source requirements.
Matching complete builds are reused across BoxMOT releases and CPU, MPS, or CUDA devices. Source data, weights, precision, preprocessing, class mapping, stage settings, and requested outputs must still match. Reuse validates the saved artifacts and keeps the original build ID without rerunning perception.
Preparation caches detector output separately from ReID embeddings. If a
compatible dataset, detector, geometry, and class mapping have already been
materialized, changing the ReID model skips detector inference and runs the
remaining derived stages. The cache lives below
<selected-build-root>/.cache/detect; with the default root, BoxMOT can also
import an identical v1 build or seed detections from another compatible v1
build in the former platform-cache location.
Detector-native masks or embeddings are not represented by this geometry cache,
so builds that require either output run their detector stage normally.
To select a catalog experiment through its components, provide --dataset and
--detector, plus the split and ReID profile when applicable:
boxmot eval \
--dataset mot17 \
--split ablation \
--detector yolox-x-mot17 \
--reid lmbn-n-duke \
--tracker botsort
These flags resolve to the authored catalog experiment
mot17/ablation-yolox-lmbn.yaml. Its detector checkpoint, class map, and other
semantic settings remain authoritative. This is equivalent to:
Both forms materialize or reuse the exact same build. If the selectors match
no catalog experiment or more than one, evaluation reports an error. Matching
uses the exact dataset, detector, and ReID profiles plus the selected split.
The unique authored experiment supplies its detector checkpoint; append
/CHECKPOINT to --detector when the profile otherwise matches more than one
experiment. Omitting --reid selects only experiments without a ReID profile.
Use --experiment to select the intended configuration explicitly, or author
an experiment YAML for a combination absent from the catalog.
Pass exactly one of --experiment or --dataset. Direct component selectors
cannot be combined with --experiment.
--device selects the detector, segmentor, and ReID execution device when
automatic preparation needs new inference. It does not force regeneration of
matching saved outputs. With --edgetam, it also selects the
EdgeTAM replay device and can accompany an explicit --build. Other
perception-build workflows reject --device with an explicit --build.
After materialization completes, evaluation consumes the exact path returned
by that build operation. It does not scan --build-root, select a latest
directory, or risk replaying another configuration's build.
Pass --build to reuse a specific build. Image-dataset evaluation requires
--build when no detector is selected, because the dataset config alone does
not select the perception components needed for materialization:
boxmot eval \
--dataset mot17 \
--split ablation \
--build /srv/boxmot/materializations/BUILD_ID \
--tracker bytetrack
An experiment selected by filename or component shorthand additionally fixes semantic component fingerprints. Dataset mode uses the selected dataset adapter for ground truth. Before tracking, evaluation verifies the build's source catalog digest, split, class taxonomy, geometry, published requirements, and—when applicable—component fingerprints.
If an explicitly selected build is missing or incompatible, evaluation fails without modifying it or creating a replacement.
To use temporal masks with a published build, enable --edgetam:
boxmot eval --experiment mot17/ablation-yolox-edgetam-lmbn.yaml \
--build BUILD_ID --tracker occluboost --tracker-backend python \
--asso-func iou --edgetam --device mps --sequence-workers 1
This reuses the build's detections and embeddings. EdgeTAM propagates masks from track identities and previous-frame prompts during replay, using the source images. Published detection masks do not replace this inference. See temporal mask guidance for requirements and settings.
Cache replay inputs for repeated runs
Add --cache-inputs to reuse the inputs consumed by evaluations or tuning runs:
boxmot eval \
--dataset mot17 \
--split ablation \
--detector yolox-x-mot17 \
--reid lmbn-n-duke \
--tracker botsort \
--cache-inputs
The first run validates the sources and prepares mapped arrays. Image workflows cache AABB or OBB detections, requested embeddings and masks, and decoded image pixels when the tracker or visualization needs them. Image references can point to image files, NumPy arrays, or video frames. Ground-truth box annotations and instance PNG labels also have reusable parsed caches.
The cache is tied to the selected source content, sequence, split, geometry, and requested modalities. It is independent of the execution device and tracker thresholds. Later runs can reuse it when switching between CPU, MPS, and CUDA.
With the default build layout, these files live under runs/replay_cache/.
For a custom build location, the cache directory sits beside the build-root
directory. They are derived data: the Parquet build remains authoritative and
is never rewritten. Incomplete or invalid derived entries are rebuilt from it.
You can remove the derived cache when no runs are using it to reclaim disk
space; --cache-inputs prepares it again when needed.
Saved sensor datasets support the same flag:
Sensor caches contain the frame timeline, declared 2D detections and packed masks,
3D detections, calibration, optional ego poses, and the ground truth selected for scoring. With
--calibrate-kf, calibration reuses the cached 3D annotations and observations.
RGB pixels are cached when visualization requests them; ordinary EagerMOT replay
only needs image dimensions. Masks remain packed on disk and are unpacked one
frame at a time. Sensor caches live under <dataset-root>/.boxmot/replay_cache/;
separate image-workflow annotation caches use .boxmot/replay_cache/annotations/
near their annotation sources.
The flag defaults to off because preparation takes time and extra disk space.
Use --no-cache-inputs to read the source formats directly. Source changes are
checked during preparation; an unchanged sensor study uses its prepared input
snapshot. Changed or incomplete caches are rebuilt. Tracker state, threshold
decisions, fusion, predictions, and metrics remain fresh for every run.
Saved 2D detections
For OC-SORT on the saved KITTI boxes:
boxmot eval --experiment kitti-mots/2d \
--data-root ./kitti-mots --tracker ocsort --cache-inputs --project runs/kitti-2d
The kitti-mots/2d and kitti-mots/2d-lmbn-n-duke experiments select
existing TrackR-CNN boxes, images, and
native KITTI tracking ground truth from a multimodal sequence folder.
Use the shipped experiment to evaluate its validation split with LMBN:
boxmot eval \
--experiment kitti-mots/2d-lmbn-n-duke \
--data-root ./kitti-mots \
--tracker occluboost \
--cache-inputs \
--project runs/kitti-2d
The preset reads sequences/{partition}/{sequence}/images,
predictions/trackrcnn/{partition}/{sequence}.txt, and
{partition}/label_02/{sequence}.txt below --data-root.
Add --split train to evaluate the training split, or use
kitti-mots/2d with --tracker bytetrack for motion-only tracking.
All KITTI experiments reference the same kitti-mots dataset inventory.
The 2D presets select dataset.modalities to disable mask loading and exclude spatial
inputs for 2D tracking. Other physical layouts can use a local dataset YAML as
shown in the dataset guide.
This workflow generates ReID features from the images when appearance is enabled;
--cache-inputs caches them together with parsed inputs and required image pixels.
If decoded images would exceed available disk space, saved-box evaluation reads
the original images while keeping the smaller caches. If an annotation or ReID
cache write runs out of space, it uses the computed values without saving that
entry. All selected inputs still reach the tracker; only cache reuse changes.
BoxMOT's built-in HOTA, CLEAR, and Identity evaluators report 2D HOTA, MOTA,
and IDF1 with KITTI visibility, distractor, and DontCare preprocessing.
No TrackEval installation is required.
Use --sequence to restrict the split and --show or --save for visualization.
No detector or build is required. This workflow uses CPU count minus two workers,
capped by the selected sequence count and with a minimum of one. Set
--sequence-workers N to override the worker count. Each worker owns its tracker
and optional ReID encoder. Live preview (--show) forces one sequence worker
and warns whenever it reduces the worker count. This workflow
does not provide materialization, tuning, or KF calibration.
Saved TrackR-CNN masks
For downloaded KITTI TrackR-CNN text predictions, use
boxmot track --tracker maf_hda with --detections, --images, and --instances
directories. This command directly replays the saved boxes and
masks without materializing a perception build. See the
MAF-HDA evaluation example
for the full command.
EagerMOT with saved sensor inputs
Use the multimodal experiment for the same local KITTI folder:
boxmot eval --experiment kitti-mots/full \
--data-root ./kitti-mots --tracker eagermot --cache-inputs --project runs/kitti-multimodal
This selects saved Track R-CNN masks and boxes, PointGNN 3D boxes, calibration, camera poses, and images. See KITTI experiment selection for the matching 2D-only experiment and train split.
Evaluate a multimodal sequence dataset through the same command:
The dataset supplies sequence images, annotations, calibration, saved spatial
detections, and optional ego poses through modalities in its dataset.yaml.
Default mask evaluation also requires saved image detections with instance masks.
Each modality selects its encoding and relative paths; split overrides can
select different prediction sets.
You can pass the folder or its dataset.yaml file. Evaluation runs on CPU,
scores KITTI MOTS masks by default, and writes a new split directory under runs/eagermot.
Use --project to change that root or repeat --sequence to select sequences.
Sequences replay in parallel using the automatic worker count.
Set --sequence-workers 4 to allow at most four sequence workers.
To evaluate spatial tracks, declare ground_truth_3d with the existing native
KITTI tracking labels in training/label_02, then run:
boxmot eval --dataset ./kitti-mots --tracker eagermot \
--split val --eval-3d --project runs/kitti-3d
The terminal shows 3D tracking HOTA/MOTA/IDF1 using volumetric box IoU.
Tracking scores are saved in metrics.json/csv, and predictions in
kitti_3d/<sequence>.txt. This custom tracking protocol uses all supplied target
GT; it does not apply official KITTI difficulty or DontCare filtering.
For additional official 2D/3D AP40 (Easy / Moderate / Hard), declare
ground_truth_objects,
install the official evaluators,
and add --eval-ap to the command. This writes detection_metrics.json/csv
and separately scores projected 2D tracking into tracking_2d_metrics.json/csv.
The main tracking scores remain volumetric 3D metrics.
Ground-truth masks are neither required nor loaded in this mode. detections_2d
can be omitted for tracking from 3D observations alone. Ego poses are optional;
without them, the tracker models motion in camera coordinates. Images still
provide the frame timeline and dimensions, and calibration remains required.
Declared tracking inputs, including any image predictions and ego poses, are consumed.
--eval-3d and --eval-masks are mutually exclusive; --eval-3d requires an
EagerMOT sensor dataset and is available only on eval. --eval-ap requires
--eval-3d and exact per-image object labels, including fractional truncation.
Only that additional option requires object annotations and the official evaluators.
For image and box trackers, select the KITTI 2D dataset:
boxmot eval --experiment kitti-mots/2d --data-root ./kitti-mots --tracker bytetrack \
--split val --cache-inputs
It uses native label_02 image boxes and BoxMOT's built-in 2D HOTA/MOTA/IDF1
metrics, including KITTI preprocessing, without an external TrackEval dependency.
Use kitti-mots/2d-lmbn-n-duke when using BoT-SORT's appearance features.
For your own recordings, copy the sensor dataset template, then supply synchronized images, calibration, 3D predictions, and ground truth for the selected metric. Add ego poses when available and 2D mask predictions for default mask evaluation. Custom sequence names and splits are supported; detector outputs must follow the documented file formats. The default evaluates car and pedestrian masks:
Load tuning's class profiles with --class-config path/to/best.yaml. Use
--show or --save to preview or record tracks, and add --show-3d to overlay
their estimated 3D cuboids. Saved videos go under the result directory's
videos/ folder. --class-config and --show-3d apply only to this sensor workflow.
--show keeps sensor replay on the main thread, processing one sequence at a
time. With --save alone, each worker writes its sequence's video.
The shared Rich panel shows frame progress for each sequence and the selected
metrics. Add --show-timing to include replay timing in the result summary, or
--verbose to display tracker diagnostics alongside the panel.
With 3D annotations,
add --calibrate-kf to fit EagerMOT's five covariance scales per class before
evaluation. Reuse <run>/kf-tuning/calibrated.yaml with --class-config.
Calibration uses world coordinates when ego poses are declared, or camera
coordinates without them, matching tracking. Prediction advances one step per image;
see 3D Kalman calibration.
Saved sensor evaluation reads predictions directly. Perception/build options
and TrackEval comparison are unavailable for these datasets;
an explicit --device must be cpu.
An incompatible selection reports a short reason and next step. The check uses
the selected split and registered tracker inputs:
declared tracking inputs marked Unused cause rejection. Ground truth is used
separately for scoring.
The direct spatial saved-sensor workflow requires
--tracker eagermot --tracker-backend python. For existing image detections,
select the saved 2D dataset. To intentionally evaluate an
image-only experiment with a new detector, select only images and ground_truth in a separate
dataset config or explicit split override, then select a perception build or
detector through the ordinary evaluation workflow. Split modality overrides
set to null remove those inputs from the experiment.
Python callers use boxmot.engine.eval.evaluator.run_eval(args) and receive
the shared ValidationResult, including class-average metrics and exp_dir.
Set args.eval_3d = True to select spatial scoring.
See the EagerMOT evaluation example.
View tracking results
Add --show to preview annotated tracks, --save to write one MP4 per sequence,
or both. Replay uses cached detections and embeddings with an explicit --build;
optional EdgeTAM guidance still runs temporal inference on source frames:
boxmot eval \
--dataset mot17 \
--split ablation \
--build BUILD_ID \
--tracker botsort \
--tracker-config path/to/kf-tuning/calibrated.yaml \
--show --save
Use the calibrated.yaml printed by a previous calibration run; omit
--tracker-config to use the tracker defaults. The saved configuration retains
its timing mode. --show and --save also work with --calibrate-kf: the tracker
replay is displayed or recorded after noise calibration finishes.
When --edgetam is enabled, --show and --save automatically
include propagated masks tinted by track ID. For example, preview and record
one MOT17 ablation sequence on MPS:
boxmot eval --experiment mot17/ablation-yolox-lmbn.yaml \
--sequence MOT17-02-FRCNN --tracker bytetrack --tracker-backend python \
--asso-func iou --edgetam --mask-guidance-weights models/edgetam.pt \
--mask-guidance-max-objects 96 --device mps --sequence-workers 1 \
--show --save
This example uses the checkpoint installed through the EdgeTAM guidance setup. Overlays appear after confirmation and next-frame propagation. Objects without guidance have no propagated overlay; retained lost masks may remain visible during recovery. These masks aid box association and do not enable mask metrics. Detection and standalone masks keep their existing rendering when temporal guidance is absent.
Preview follows source timestamps when available. Press q or Esc to
close the preview while evaluation continues. Annotated videos are written to
<run>/videos/<sequence>.mp4 at 30 FPS. Frames are held across capture gaps,
quantized to a 30 FPS output grid, with one final 1/30-second frame. Sources
without timestamps use one output frame per input frame. eval --fps retains
its dataset-sampling meaning; it does not change the output video rate.
For image builds, visualization decodes source images and replays sequences
serially on the main thread, regardless of --sequence-workers. Reported replay
timing includes rendering, video writing, and preview pacing; omit these flags
for speed benchmarks.
Dataset FPS
Use --fps to select a lower dataset frame rate:
boxmot eval \
--dataset mot17 \
--split ablation \
--detector yolox-x-mot17 \
--reid lmbn-n-duke \
--tracker botsort --device mps --fps 5
For a 30 FPS sequence, --fps 5 selects every sixth frame. Materialization,
replayed image loading, and ground truth share this selection. Tracker results
and ground truth use matching contiguous frame numbers, while capture
timestamps retain the original elapsed time. Fractional rates such as
--fps 2.5 are supported. Selection uses the sequence's capture timeline,
including timestamps.csv when present; it never creates extra frames.
Automatic preparation first reuses an existing build at the requested rate. Otherwise, it looks for a compatible published full-rate materialization and copies the selected detections, embeddings, and masks into a smaller build without loading the models. The parent must match the source images, capture timestamps, ground-truth provenance, perception components, geometry, and class mapping, and contain every requested payload. If no compatible parent exists, materialization runs perception on the selected frames.
The target FPS enters the catalog and build identity, so each rate retains
its own immutable build. Omitting --fps during automatic preparation keeps
all original frames. With an explicit --build, omission uses the rate
recorded in that build. An explicit rate must match the build; omit --build
to prepare a different rate using automatic reuse, or run
materialize --experiment YAML --fps RATE first.
--variable-dt independently enables capture timestamps for Kalman prediction.
track --fps controls saved video playback speed; dataset frame selection
applies to materialize, eval, and tune.
Combine --fps 2 --calibrate-kf to fit Kalman noise using the selected 2 FPS
detections and ground truth before evaluating once. The
calibration example also enables timestamp-based prediction.
Legacy builds
An unbound build created before canonical experiment materialization can be
evaluated only with the explicit --allow-noncanonical-build escape hatch.
This relaxes the missing build-binding check; it does not turn incompatible
artifacts into compatible ones. Use it only after independently verifying the
build's source, split, detector, ReID model, geometry, and class taxonomy. For
example, to diagnose the known MMOT build on one sequence:
boxmot eval \
--experiment mmot-obb/test-yolo11l-lmbn.yaml \
--build faf16842d9d15fc048a50247df8ae927e7299d5760c9f461af4581cabfa279e6 \
--data-root datasets/mot \
--tracker botsort \
--sequence data23-1 \
--allow-noncanonical-build
Omit --sequence data23-1 to evaluate every sequence. Canonical builds remain
the default and do not need this flag.
CMC by sequence
Evaluate a joint KF/CMC selection result using its saved tracker profile:
boxmot eval --experiment mot17/ablation-yolox-lmbn.yaml \
--tracker botsort --tracker-config path/to/best.yaml
Replace the profile path with the output from
tune --select-kf --select-cmc.
A saved automatic camera strategy analyzes the current sequences during setup
and applies SOF only where camera motion is detected. The selected KF and
tracker thresholds stay fixed. Source images must remain available.
For a profile without a saved camera strategy, --cmc-policy auto requests
sequence analysis, --cmc-policy off disables CMC, and --cmc-policy PATH
loads explicit SOF decisions:
These policies require materialized image builds and Python 2D Kalman
trackers. --cmc-policy cannot be combined with --calibrate-kf; use joint
selection to fit noise under the candidate camera strategies.
Kalman calibration
EagerMOT supports 3D calibration from saved sensor detections and 3D ground truth, using fixed frame steps. The image-tracker workflow below supports AABB and OBB observations.
Use --calibrate-kf to estimate Kalman noise directly from cached detector
predictions and ground truth on the selected split, then evaluate the calibrated
tracker once:
boxmot eval \
--experiment mot17/ablation-yolox-lmbn.yaml \
--tracker botsort \
--fps 2 \
--variable-dt \
--calibrate-kf
This example selects 2 FPS data and uses its capture timestamps for prediction.
Omit --variable-dt to calibrate in fixed-step mode, or omit --fps to retain
the original dataset rate during automatic preparation.
Perception is materialized or reused once. Calibration matches detections to
ground truth by class and IoU (at least 0.5), then fits coordinate-wise
covariance multipliers for five noise groups from the observed errors:
| Parameter | What provides the calibration residuals |
|---|---|
kalman.noise.process_position_diagonal |
Position-noise contribution to ground-truth box prediction errors |
kalman.noise.process_velocity_diagonal |
Velocity-noise contribution to the same prediction errors |
kalman.noise.measurement_noise_diagonal |
Matched detector boxes minus ground-truth boxes |
kalman.noise.initial_position_diagonal |
Detection errors at each GT object's first matched observation |
kalman.noise.initial_velocity_diagonal |
Zero-initialized velocity errors relative to local ground-truth motion |
Measurement and initialization vectors use residual second moments. Both process groups are fitted together from constant-velocity prediction errors across three consecutive annotated frames. The fit accounts for process noise from both intervening intervals and the covariance between adjacent prediction errors. Adjacent error pairs require four consecutive annotated frames and help separate position from velocity noise, including at regular frame rates. Ground-truth motion contributes even when a detection is missed; missing annotations break the motion samples. The first matched detection for each object supplies an initialization proxy. Calibration requires at least one valid detection/ground-truth match. If a coordinate lacks sufficient evidence, its configured prior is retained and the report records why.
With --per-class, calibration also fits a profile for each selected class and
saves it under kalman.noise.by_class. A class with insufficient evidence uses
the pooled estimate for that parameter; if the pooled evidence is also sparse,
its configured prior is retained. Saved profiles preserve per-class tracking
when reused with --tracker-config.
The estimates scale the selected filter's reference covariance priors. Values
multiply covariance, not standard deviation. Calibration keeps all other
tracker settings fixed, including the timing mode. The filter's internal
base priors remain fixed. The saved *_diagonal vectors preserve the selected
filter's coordinate layout, and the five global *_scale multipliers are
saved as 1.0. Explicit scale refinement changes whole groups while keeping
the fitted coordinate ratios fixed. StrongSORT also fits its
measurement-confidence floor.
eval --calibrate-kf does not launch Ray Tune or Optuna, replay HOTA trials, or
require the evolve extra. Use boxmot tune
for metric-based search over tracker parameters. Kalman settings remain fixed
by default; select individual scales with --tune-kf to refine them around
their configured or calibrated values using a logarithmic search range.
Use tune --calibrate-kf to
calibrate once before searching the other tracker settings, keeping the KF
calibration fixed throughout the search.
A positive numerical floor keeps calibrated covariances valid. See
OC-SORT base process noise for the
relationship between model priors and calibrated multipliers.
Calibration supports AABB and OBB ground truth with Python ByteTrack, BotSort, StrongSort, OcSort, DeepOcSort, HybridSort, BoostTrack, and OccluBoost. Native backends and trackers without a Kalman filter are rejected before automatic materialization. HybridSORT's confidence state has no ground-truth confidence target, so confidence residuals are excluded from fitting; the shared scales still apply to that state during tracking.
Timing and saved settings
Fixed-step prediction remains the default. Add --variable-dt to fit and use
noise in seconds, with elapsed intervals from capture timestamps. Neither
mode searches dt, and measurement noise is not scaled by elapsed time.
--fps 2 selects dataset frames at 2 FPS; it does not enable --variable-dt.
kalman.noise.reference_dt_s fixes the interval used to convert historic per-frame priors
into seconds: its default 0.03333333333333333 represents a 30 FPS reference.
It is not the source clock or the prediction interval. This reference and
kalman.noise.time_unit stay fixed during calibration and tracker tuning. See
time-unit conversion for the covariance
scaling rules.
Each run writes:
<run>/kf-tuning/calibrated.yaml: resolved tracker settings, tracker name, and a nestedkalmanconfiguration containing timing and filter policies. Itsnoisegroup holds the five scales, explicit units, reference interval, and any class profiles.<run>/kf-tuning/calibration.json: calibration evidence, timing settings, and the final evaluation result.
Calibration fits noise statistics, so it does not guarantee a higher HOTA. The final replay uses the fitting split. Evaluate the saved settings on separate sequences before judging how well they generalize. Initial velocity uses a local ground-truth finite difference as a proxy. Annotation noise and camera motion contribute to motion errors measured in image coordinates. Measurement estimates describe detections that pass the IoU match threshold; they do not model false positives.
Use --tracker-config to evaluate the saved configuration on a separate split
with ground truth and a compatible build. For example, after preparing a
validation build whose sequences are held out from calibration:
boxmot eval \
--dataset mot17 \
--split val \
--build /srv/boxmot/materializations/HELD_OUT_BUILD \
--tracker botsort \
--tracker-config path/to/kf-tuning/calibrated.yaml
--tracker-config also accepts a partial runtime YAML or a built-in preset such
as botsort-mot17-ablation. It overlays tracker defaults. Explicit runtime
flags such as --asso-func can override scalar parameters, but a calibrated
config's time units must match the selected mode. For example,
--fixed-dt --tracker-config seconds-config.yaml is rejected when that file
declares kalman.noise.time_unit: seconds. Recalibrate in the intended mode instead of
reinterpreting the saved values. Keep the detector, geometry, backend, class
selection, and per-class tracking behavior consistent with calibration.
Live streams
Direct calibration needs ground truth. Run it on representative timestamped recordings, including dropped frames and missed detections, then load the saved profile for live tracking. Keep the same detector, geometry, timing mode, and reference interval:
boxmot track \
--source <stream> \
--tracker botsort \
--tracker-config path/to/kf-tuning/calibrated.yaml
BoostTrack and OccluBoost also support experimental online process-noise
adaptation with kalman.adaptive_kf: true. This learns from each track's prediction
errors; it does not learn measurement noise or initial uncertainty and does
not measure tracking accuracy. Incorrect associations can distort its estimates.
Calibration estimates the starting priors and preserves this setting; it does
not fit the online adaptation behavior. Keep it consistent during evaluation
and deployment.
Other trackers support the calibrated static profile and interval-aware
prediction; --variable-dt itself does not enable online noise adaptation.
Live intervals must come from trustworthy source timestamps. A nominal-FPS
fallback cannot reveal unseen capture dropouts or reconnect duration; use
capture/PTS metadata or provide Frame.timestamp_s through the Python API.
Elapsed inference time is not a capture timestamp.
Sequence parallelism
Parallel evaluation gives each sequence its own tracker in an isolated
process. Tracker state and native handles never cross sequence or process
boundaries.
By default, the worker count is the smaller of the selected sequence count and
the logical CPU count minus two, with at least one worker when sequences are
selected: min(sequences, max(1, logical_cpus - 2)). For example, nine selected
sequences on a machine with eight logical CPUs use six workers.
--sequence-workers N overrides automatic sizing with a positive integer cap;
the active worker count never exceeds the number of selected sequences.
Use --sequence-workers 1 to process sequences one at a time. The Rich panel
reports frame progress separately for every sequence while those jobs run.
Spawned sequence workers use one compute thread per BLAS, OpenMP, PyTorch, and OpenCV pool, overriding inherited thread counts to avoid nested CPU parallelism. These limits apply inside the workers; serial replay in the parent process keeps the caller's thread settings.
Image-build previews and saved videos use serial replay. Saved 2D preview forces
one sequence worker with a warning when it reduces the count. EagerMOT sensor replay runs serially
with --show. Saved 2D and EagerMOT both support parallel video writing with --save alone.
EdgeTAM mask guidance defaults to one sequence worker. An explicit higher
--sequence-workers value loads a separate model and temporal state in each
worker, increasing memory use.
Use --sequence to replay only one sequence while diagnosing a run. Repeat
the option to select more than one sequence:
boxmot eval \
--experiment mmot-obb/test-yolo11l-lmbn.yaml \
--build BUILD_ID \
--sequence data23-1 \
--tracker botsort
boxmot eval reuses the sequence frame counts already validated during dataset
setup. A standalone replay call falls back to reading only the samples table's
sequence IDs. Neither path eagerly loads masks or embeddings in the
coordinator. It marks each sequence as queued before starting the process pool.
A worker then opens only its assigned sequence and streams optional masks and
embeddings in bounded Arrow batches as frame iteration advances. Images are
decoded when the resolved tracker or --show/--save requires source pixels. A
dimensions-only tracker such as SFSORT receives width and height from cached
sample metadata without opening the source image unless visualization is enabled.
Current materializations also use bounded Parquet row groups so workers can
prune unrelated sample_id ranges. Older boxmot.dataset/v1 builds remain
readable, but builds created with large row groups can require more I/O during
the initial worker-loading phase.
Build resolution
- For detector-based evaluation, when
--buildis omitted with--experimentor--datasetplus--detector, BoxMOT resolves the authored experiment and materializes its deterministic build below--build-root; an identical complete build is validated and reused. - An existing
--buildpath is used directly. - A build ID is looked up only below
--build-root. --build-rootdefaults toBOXMOT_BUILDS_DIR, then./runs/materializations.- Dataset-only evaluation requires
--buildwhen no detector or saved detections are selected. - Saved 2D experiments replay their dataset inputs directly without a build.
- There is no latest-build selection.
The selected raw data root is used to verify ground-truth provenance. It
defaults to ./datasets/mot; pass --data-root explicitly to use another
location. When the selected split is absent, its configured Hugging Face
per_split resource is downloaded before materialization or ground-truth
validation. Existing populated splits are reused without downloading;
archive-backed datasets still require explicit setup.
Repeated evaluations reuse hashes and image dimensions only when a source or model file's path, device, inode, mode, size, modification time, and change time are unchanged. This shared metadata lives in the platform cache; the source tree is still traversed on every run so added and removed files are detected. This metadata cache is separate from the content-addressed detector output cache; materialization still establishes the source and model content digests before selecting either a complete build or reusable detections.
Postprocessing
Repeat --postprocessing to apply ordered steps after replay and before metrics:
boxmot eval \
--experiment mot17/ablation-yolox-lmbn.yaml \
--build BUILD_ID \
--tracker botsort \
--postprocessing gta \
--postprocessing gsi
Supported methods are gta (tracklet association), gsi (Gaussian-smoothed
interpolation), and gbrc (gradient-boosting smoothing and interpolation).
The example associates fragmented tracklets first, then fills and smooths their
trajectories. GTA requires cached detection embeddings and must run before
GSI or GBRC. Automatic preparation publishes embeddings when GTA is selected,
including for motion-only trackers; an explicit build must already contain
them. Interpolated rows preserve the track's class and carry detection index
-1.
Postprocessing uses the same --sequence-workers limit as tracking. Each
worker applies the selected methods in order to one sequence. The Postprocess
stage shows a Rich progress row for every sequence, including the current
method, phase, and completed work. Counts restart for each phase; phases with
unknown totals show an indeterminate bar until their work is complete.
Postprocessing supports canonical AABB box replay from a materialized build.
It is unavailable for OBB output, mask scoring, saved-detection replay, and
sensor/3D replay. Original tracker results are retained under the result
folder's raw/ directory. Metrics use the processed results, and
postprocessing.json records the applied steps and provenance. --show
previews and --save videos depict the online tracks produced during replay.
Output
Tracker output is serialized to MOT text at the evaluation boundary. Selected
postprocessing steps run before scoring and retain the raw result files. The
perception build remains immutable. --compare-trackeval is available for
supported AABB MOTChallenge datasets.
Perception-build evaluation writes class-keyed scores to
<project>/<dataset_id>/<name>/metrics.json after scoring succeeds, including
per-sequence scores and any class aggregates. The defaults are --project runs
and --name exp; existing run names increment unless --exist-ok is set.
The JSON is written for ordinary evaluation and CI runs alike. For MOT17, read
the combined pedestrian scores from pedestrian.HOTA, pedestrian.MOTA, and
pedestrian.IDF1, for example:
boxmot eval --experiment mot17/ablation-yolox-lmbn.yaml \
--tracker bytetrack --project runs --name bytetrack --exist-ok
jq '.pedestrian | {HOTA, MOTA, IDF1}' runs/mot17/bytetrack/metrics.json
Guided box trackers report box tracking metrics. Their output directory includes a
guidance fingerprint, and mask-guidance.json records the official EdgeTAM
revision, checkpoint SHA-256, precision, postprocessing, identity cap, effective
tracker thresholds, resolved tracker options, and named association behavior.
Changing these settings produces
a distinct output identity. Detection masks and --eval-masks are separate
from this auxiliary association cue.
Arguments
boxmot eval
Evaluate tracking performance from a perception build or saved dataset predictions.
Usage:
Options:
--experiment TEXT experiment YAML filename or path, e.g.
mot17/ablation-yolox-lmbn.yaml or
boxmot/configs/experiments/mot17/ablation-
yolox-lmbn.yaml
--dataset TEXT Dataset id, YAML file, or sensor dataset
folder containing dataset.yaml (eval and
tune).
--detector TEXT Detector profile ID or YAML config used to
resolve an authored experiment with
--dataset; append /CHECKPOINT to
disambiguate.
--reid TEXT ReID profile for an authored experiment or
for encoding saved 2D detections during
eval. Omit when appearance is disabled.
--build TEXT Materialized build ID or explicit build
directory. Omit with --experiment, or with
--dataset and --detector, to materialize and
reuse a compatible canonical build
automatically.
--build-root PATH Root for build IDs, automatic
materialization, and shared detector cache;
overrides BOXMOT_BUILDS_DIR and defaults to
./runs/materializations.
--device TEXT One device for uncached perception and
EdgeTAM mask guidance: cpu, mps, cuda:N, or
N (e.g. 0).
--data-root PATH Base folder for dataset storage.root;
defaults to ./datasets/mot for built-in
datasets or the local YAML folder.
--split TEXT Dataset split to use (e.g. train, val, test,
ablation). Overrides auto-detection from
source path.
--fps FLOAT Target dataset FPS: sample frames before
perception and align ground truth. Default:
original frames, or the selected build's FPS
when replaying.
--eval-masks Evaluate KITTI-MOTS with mask IoU. Image
workflows default to boxes; sensor workflows
default to masks.
--postprocessing [gsi|gbrc|gta]
Postprocess AABB tracking results before
scoring. Repeat in application order; GTA
must precede GSI/GBRC and requires cached
embeddings. Uses --sequence-workers with
per-sequence progress.
--eval-3d EagerMOT: 3D tracking metrics from KITTI
tracking ground truth.
--eval-ap Add official KITTI 2D/3D AP40; requires
--eval-3d and aligned per-image object
ground truth.
--tracker-backend [python|cpp] Tracker implementation backend. Native 'cpp'
is available for botsort, bytetrack,
occluboost, ocsort, and sfsort. [default:
python]
--tracker-config TEXT Tracker runtime YAML path or built-in
preset; overrides must preserve calibrated
time units.
--cmc-policy TEXT CMC policy: auto analyzes sequences during
setup (Python 2D Kalman trackers), off
disables CMC, or YAML.
--class-config FILE EagerMOT: car and pedestrian profiles from
tuning's best.yaml or KF calibration's
calibrated.yaml.
--asso-func [iou|giou|diou|ciou|hmiou|centroid]
Association geometry override for AABB and
OBB tracking. OBB ciou is a custom
experimental adaptation; OBB hmiou is an
experimental global-y height cue for scenes
where image vertical is meaningful.
--mask-guidance-weights PATH Checkpoint or exported TFLite bundle
directory used with --edgetam. Selecting
weights alone does not enable guidance;
edgetam.pt downloads into ./models. TFLite
guidance runs on CPU. Python AABB box
trackers with IoU only.
--edgetam / --no-edgetam Enable EdgeTAM mask guidance with tracker
YAML settings; uses edgetam.pt unless
weights are supplied. [default: no-edgetam]
--mask-guidance-max-objects INTEGER RANGE
Override the tracker config's
edgetam.max_objects (default 96) when
--edgetam is enabled. Maximum identities
with temporal mask memory; fixed when
tuning. [x>=1]
--sequence-workers INTEGER RANGE
Maximum sequence worker processes. Default:
1 for EdgeTAM evaluation/tuning; otherwise
min(sequences, CPU cores - 2), at least 1.
During tuning, this limit applies per trial.
[x>=1]
--cache-inputs / --no-cache-inputs
Build and reuse mapped inputs for repeated
eval/tune runs, including images,
detections, embeddings, masks, sensors, and
annotations. Uses extra disk space.
[default: no-cache-inputs]
--project PATH save results to project/name
--name TEXT save results to project/name
--exist-ok existing project/name ok, do not increment
--ci reuse existing runs in CI (no UI)
--tracker [boosttrack|botsort|bytetrack|deepocsort|eagermot|hybridsort|maf_hda|occluboost|ocsort|sfsort|strongsort]
one of: boosttrack, botsort, bytetrack,
deepocsort, eagermot, hybridsort, maf_hda,
occluboost, ocsort, sfsort, strongsort
[default: bytetrack]
--variable-dt / --fixed-dt Use capture timestamps for Kalman
prediction; otherwise keep the tracker YAML
setting (default: fixed dt).
--verbose print detailed logs
--show-timing / --hide-timing print runtime timing summary after
evaluation [default: hide-timing]
--per-class track each class separately
--show Preview tracking at source timing, after
calibration when --calibrate-kf is enabled.
Sensor replay displays tracked masks, IDs,
and classes.
--save Save annotated tracking videos from replay,
after calibration when --calibrate-kf is
enabled. EagerMOT writes one MP4 per
sequence under results/videos.
--show-3d EagerMOT sensor evaluation: overlay
estimated tracked 3D cuboids; requires
--show or --save.
--calibrate-kf Calibrate Kalman noise from detections and
ground truth, then evaluate once; Python
Kalman trackers only. EagerMOT requires
ground_truth_3d with track IDs.
--sequence NAME Limit the selected split to one sequence.
Repeat to select multiple sequences.
--allow-noncanonical-build Allow an unbound legacy build after
validating it against the selected dataset.
WARNING: use only when you have
independently verified the build provenance.
--compare-trackeval / --no-compare-trackeval
Compare BoxMOT metrics against TrackEval for
an AABB MOTChallenge benchmark.
--help Show this message and exit.