Skip to content

Component API Reference

Canonical structures

Import canonical values from boxmot.structures. Their implementation modules group 2D and 3D values by role:

Module Structures
Geometry Boxes, OrientedBoxes, Boxes3D
Detections Detections, Detections3D
Tracks Tracks, Tracks3D, MultimodalTracks
Camera CameraModel

Detection

Bases: Protocol

Detect objects in a batch of canonical frames.

capabilities property 

capabilities: DetectorCapabilities

Declare detector geometry and enrichment support.

predict 

predict(frames: Sequence[Frame]) -> list[Detections]

Return one detection collection for every input frame, in order.

Geometry and enrichment outputs provided directly by a detector.

Everything needed to construct and fingerprint a detector runtime.

option_values 

option_values() -> dict[str, JSONValue]

Return backend options as a new mutable mapping.

Construct a detector from a spec, profile name, YAML, or model artifact.

Runtime keywords override the resolved settings; options merges backend settings over authored options. References resolve and hash their artifacts before construction, downloading missing models when allow_download is true. An existing DetectorSpec keeps its resolved artifact identity.

Segmentation

Bases: Protocol

Produce one full-frame mask for every supplied detection.

segment 

segment(
    frames: Sequence[Frame],
    detections: Sequence[Detections],
) -> list[MaskBatch]

Return one detection-aligned mask collection per frame.

Everything needed to construct and fingerprint a segmentor runtime.

option_values 

option_values() -> dict[str, JSONValue]

Return backend options as a new mutable mapping.

Construct a segmentor, optionally sharing an official EdgeTAM model.

Appearance encoding

Bases: Protocol

Encode detection-aligned appearances for one or more frames.

embedding_dim property 

embedding_dim: int

Width of every descriptor returned by :meth:encode.

requirements property 

requirements: EncoderRequirements

Declare the payloads required to extract appearance crops.

encode 

encode(
    frames: Sequence[Frame],
    detections: Sequence[Detections],
) -> list[Tensor]

Return one float32 [N, D] CPU tensor per input frame.

Detection enrichments needed before appearance encoding.

Reusable appearance-model selection and inference settings.

Omitted overrides inherit the selected profile's settings. Construction is lazy when this configuration is passed to a tracker; creating this object does not resolve artifacts or download weights.

Parameters:

  • model (str | Path, default: 'osnet-x0-25-msmt17' ) –

    Built-in model/profile name, custom weights, or encoder YAML path. Defaults to OSNet x0.25 trained on MSMT17.

  • device (str | None, default: None ) –

    Inference device override, such as "cpu" or "cuda:0".

  • precision (Literal['fp16', 'fp32'] | None, default: None ) –

    FP16 or FP32 inference override; the selected backend validates support during encoder construction.

  • preprocessing (str | None, default: None ) –

    Crop preprocessing override, such as "resize" or "resize_pad". Crop geometry follows the input detections.

  • batch_size (int | None, default: None ) –

    Maximum number of crops per inference batch; the encoder default is 64 when neither the profile nor this config specifies it.

  • image_size (tuple[int, int] | None, default: None ) –

    Crop height and width override for the selected encoder.

  • embedding_dim (int | None, default: None ) –

    Descriptor width when the backend cannot declare it.

  • allow_download (bool, default: True ) –

    Allow downloading missing model artifacts on resolution.

from_mapping classmethod 

from_mapping(values: Mapping[str, Any]) -> ReIDConfig

Validate a friendly YAML mapping without resolving its model.

to_dict 

to_dict() -> dict[str, Any]

Return YAML-ready settings, retaining omitted profile overrides.

Everything needed to construct and fingerprint an appearance encoder.

option_values 

option_values() -> dict[str, JSONValue]

Return backend options as a new mutable mapping.

Construct an encoder from a resolved spec, profile, YAML, or model artifact.

Explicit runtime keywords override the selected configuration. An omitted allow_download inherits ReIDConfig.allow_download; other references allow downloads by default. options merges backend settings by key, preserving authored settings not overridden. References resolve and verify model artifacts, downloading missing weights when allowed; an explicit ReIDEncoderSpec must already be resolved.

Tracking

Bases: Protocol

Stateful, single-sequence tracker contract.

capabilities property 

capabilities: TrackerCapabilities

Return static capabilities of the tracker algorithm.

generates_embeddings property 

generates_embeddings: bool

Return whether missing embeddings can be generated from a frame.

requirements property 

requirements: TrackerRequirements

Return the inputs required by this resolved configuration.

reset 

reset() -> None

Clear sequence-local state while preserving configuration.

update 

update(
    detections: Detections,
    frame: Frame | ndarray | None = None,
    *,
    detections_3d: Detections3D,
    camera: CameraModel,
    timestamp_s: float | None = None,
) -> MultimodalTracks

update(
    detections: Detections,
    frame: Frame | ndarray | None = None,
    *,
    timestamp_s: float | None = None,
) -> Tracks

update(
    detections: ndarray,
    frame: Frame | ndarray | None = None,
    *,
    timestamp_s: float | None = None,
) -> ndarray

update(
    detections: Detections | ndarray,
    frame: Frame | ndarray | None = None,
    *,
    timestamp_s: float | None = None,
    detections_3d: Detections3D | None = None,
    camera: CameraModel | None = None,
) -> Tracks | ndarray | MultimodalTracks

Advance with a Frame or uint8 HWC BGR image; preserve the detection representation.

validate_timing 

validate_timing(
    frame: Frame | ndarray | None = None,
    *,
    timestamp_s: float | None = None,
) -> float | None

Validate capture timestamps without advancing sequence state.

Bases: Tracker, Protocol

Optional tracker interface for lazy, tracker-owned ReID inference.

configure_reid 

configure_reid(spec: ReIDEncoderSpec) -> None

Configure tracker-owned ReID before the first update of a sequence.

Inputs required by one resolved tracker configuration.

frame_pixels property 

frame_pixels: bool

Return whether the tracker needs decoded source-image pixels.

Normalized tracker selection.

The spec contains only algorithm selection and immutable configuration. Model construction belongs to component-specific factories and is deliberately absent.

option_dict property 

option_dict: dict[str, JSONValue]

Return a fresh mutable mapping for constructor dispatch.

Create a tracker from its registered name or an immutable specification.

config supplies the selected tracker's typed algorithm settings. Specification options override that configuration; an options mapping and then direct keyword arguments take precedence over specification options. Keyword arguments override specification fields or tracker-algorithm options. An optional options mapping overlays spec options; direct keywords win. Omitted values retain the spec selection and the tracker's configured defaults. reid supplies an encoder configuration or prebuilt AppearanceEncoder separately from the serializable tracking algorithm specification. mask_guidance enables the McByte++ mask cue for Python box trackers with AABB IoU matching. Supply a config for lazy construction or a MaskGuidance runtime dedicated to this tracker to share an already loaded model.

The factory does not construct models eagerly. Trackers consume required masks, embeddings, or frames through their structured update boundary; every ReID-enabled high-level tracker can additionally build its own ReID backend lazily when called without precomputed embeddings.

Trackers accept canonical Detections or, for standalone box-only calls, exact NumPy AABB N x 6 / OBB N x 7 rows. An optional canonical Frame may be passed separately. NumPy input returns packed float64 AABB M x 8 / OBB M x 9 rows; canonical input returns Tracks. Appearance and mask inference normally belong upstream in a PerceptionPipeline. Every high-level ReID-enabled tracker adapter can also lazily invoke ReID when its input has no embeddings and a Frame is supplied. Attached embeddings bypass that internal backend. Native adapters then pass the resolved features to their model-free C++ tracker libraries.

Batched Kalman filters

Python trackers automatically batch compatible Kalman predictions and matched corrections within each association stage. No CLI option is needed. Each track retains its own covariance, calibrated noise, observation history, and adaptive noise state.

For direct filter use, XYAH and XYWH provide multi_predict, multi_project, and multi_update. Pass means shaped (N, state_dimensions) and covariances shaped (N, state_dimensions, state_dimensions); corrections also accept measurements shaped (N, measurement_dimensions) and one confidence per row. These methods return arrays without modifying the filter's stored state.

XYSR, XYSCR, and XYHR provide predict_many(filters, ...) and update_many(filters, measurements, ...) for independent stateful filter instances. These methods modify each instance while batching the matrix work. Use these methods when a filter owns its state, noise, or observation history. EagerMOT's Kalman3D similarly provides multi_predict(filters) and multi_update(filters, boxes).

Batching preserves elapsed-time and oriented-box handling. Missing observations and observation-centric recovery retain each track's sequential history replay; per-track bookkeeping and adaptive-noise updates still run independently.

Batched camera-motion compensation

BoT-SORT, StrongSORT, BoostTrack, OccluBoost, DeepOCSORT, and HybridSORT batch camera-motion updates automatically when CMC is enabled. No additional CLI option is needed. The shared kernels transform box geometry, numerical Jacobians, velocities, and covariances in arrays. Observation-centric trackers also batch their retained observations and recovery states, preserving each tracker's existing compensation policy.

Camera motion is estimated once per frame. Choose sof for translation, rotation, and scale changes, or ecc for translation. These are the recommended fast methods. ORB and SIFT remain available as slower opt-in methods in the Python backend: select cmc_method="orb" or cmc_method="sift" in a Python tracker configuration that supports CMC. Their useful settings can exceed the one-millisecond latency target. Native backends support ECC and SOF only.

ECC and SOF resize before grayscale conversion and cap their default working width regardless of source resolution. SOF uses a 192-pixel width, up to 48 background corners, and an 11-by-11 optical-flow window with at most 10 iterations. It retains RANSAC inliers between feature refreshes (at most five frames), checking the current detection mask on every frame. ECC uses a 96-pixel width and at most 10 iterations. Small inputs are not enlarged. Native BoT-SORT and OccluBoost use the same settings. Explicit estimator scale/width overrides can increase cost.

The ECC and SOF defaults target one millisecond per frame; latency depends on hardware, scene content, OpenCV threads, and detection count. Measure warmed calls including preprocessing and state updates, alongside known-motion accuracy:

uv run --no-sync python -m tests.performance.trackers.motion.benchmark_cmc_latency \
  --widths 1280 1920 3840 --iterations 100 --json /tmp/cmc-latency.json

The benchmark defaults to ECC and SOF. Compare the slower opt-in methods with:

uv run --no-sync python -m tests.performance.trackers.motion.benchmark_cmc_latency \
  --methods orb sift --widths 1920 --iterations 100

This benchmark uses one OpenCV thread and preloaded BGR frames without detection masks. It reports mean, median, p95 and maximum latency plus original-image pixel error. Add --check-budget to fail when any measured p95 exceeds one millisecond. ECC's translation model cannot recover the benchmark's rotation/scale component. Re-run joint camera/Kalman selection for older fitted profiles: the estimator change advances the analysis rule to sof-motion-v4.

Frame pairs remain sequential. General affine and projective OBB fitting still uses one OpenCV rectangle fit per box, and mask rasterization retains individual polygon calls to preserve overlaps.

Compare cached replay against a saved package snapshot without rerunning detection or ReID:

uv run --no-sync python -m tests.performance.trackers.benchmark_cached_cmc \
  --baseline /path/to/snapshot-containing-boxmot \
  --build /path/to/existing/materialization \
  --experiment mot17/ablation-yolox-lmbn.yaml \
  --tracker botsort --workers 7 --repeat 3 \
  --output /tmp/cmc-comparison

Use --mode preloaded --sequence MOT17-04-FRCNN --frames 100 to exclude image decoding from the measured tracking loop. Both modes check track IDs and geometry; complete replay also checks the evaluation metrics.