HybridSort

Paper: Hybrid-SORT: Weak Cues Matter for Online Multi-Object Tracking

Hybrid-SORT argues that MOT pipelines lean too heavily on strong cues such as appearance and overlap, even though those cues often fail together during heavy occlusion. The paper supplements them with weaker but cheap signals such as velocity direction, confidence, and height state, then combines those cues in a training-free online tracker. That makes the method attractive when association needs extra structure without turning into a much heavier offline system.

What BoxMOT Needs For HybridSort

• A detector and, for the intended setup, a ReID model.
• AABB detections only in BoxMOT.
• A good fit when you want richer association than OC-SORT or BoT-SORT-style matching, especially on crowded MOT benchmarks.

Bases: BaseTracker

Initialize the HybridSort tracker.

Parameters:

Name	Type	Description	Default
reid_weights	Path | str | None	Path to the ReID model weights.	required
device	device	Device used for ReID inference.	required
half	bool	Whether to use half precision for ReID inference.	required
cmc_method	str	Camera-motion compensation method.	'ecc'
with_reid	bool	Whether to enable appearance features.	True
low_thresh	float	Low-confidence threshold for second-pass matching.	0.1
delta_t	int	Time window used for motion estimation.	3
inertia	float	Motion-consistency weight.	0.05
use_byte	bool	Whether to enable ByteTrack-style second association.	True
use_custom_kf	bool	Whether to use the HybridSort custom Kalman filter.	True
longterm_bank_length	int	Number of appearance features to keep in the long-term bank.	30
alpha	float	Feature update coefficient.	0.9
adapfs	bool	Whether to enable adaptive feature smoothing.	False
track_thresh	float	High-confidence threshold for the first association pass.	0.5
EG_weight_high_score	float	Embedding-guided association weight for high-score detections.	4.6
EG_weight_low_score	float	Embedding-guided association weight for low-score detections.	1.3
TCM_first_step	bool	Whether to enable TCM in the first step.	True
TCM_byte_step	bool	Whether to enable TCM in the Byte step.	True
TCM_byte_step_weight	float	TCM weight in the Byte step.	1.0
high_score_matching_thresh	float	Threshold for high-score matching.	0.7
with_longterm_reid	bool	Whether to enable long-term ReID features.	True
longterm_reid_weight	float	Weight applied to long-term ReID scores.	0.0
with_longterm_reid_correction	bool	Whether to enable long-term ReID correction.	True
longterm_reid_correction_thresh	float	Correction threshold for regular detections.	0.4
longterm_reid_correction_thresh_low	float	Correction threshold for low-score detections.	0.4
dataset	str	Dataset hint used by the association logic.	''
**kwargs	Any	Base tracker settings forwarded to :class:BaseTracker.	{}

Attributes:

Name	Type	Description
with_reid	bool	Whether appearance features are enabled.
model		ReID model used for appearance extraction when enabled.
cmc		Camera-motion compensation method.
active_tracks	list[KalmanBoxTracker]	Currently active tracks.

Source code in boxmot/trackers/hybridsort/hybridsort.py

class HybridSort(BaseTracker):
    """Initialize the HybridSort tracker.

    Args:
        reid_weights (Path | str | None): Path to the ReID model weights.
        device (torch.device): Device used for ReID inference.
        half (bool): Whether to use half precision for ReID inference.
        cmc_method (str): Camera-motion compensation method.
        with_reid (bool): Whether to enable appearance features.
        low_thresh (float): Low-confidence threshold for second-pass matching.
        delta_t (int): Time window used for motion estimation.
        inertia (float): Motion-consistency weight.
        use_byte (bool): Whether to enable ByteTrack-style second association.
        use_custom_kf (bool): Whether to use the HybridSort custom Kalman
            filter.
        longterm_bank_length (int): Number of appearance features to keep in
            the long-term bank.
        alpha (float): Feature update coefficient.
        adapfs (bool): Whether to enable adaptive feature smoothing.
        track_thresh (float): High-confidence threshold for the first
            association pass.
        EG_weight_high_score (float): Embedding-guided association weight for
            high-score detections.
        EG_weight_low_score (float): Embedding-guided association weight for
            low-score detections.
        TCM_first_step (bool): Whether to enable TCM in the first step.
        TCM_byte_step (bool): Whether to enable TCM in the Byte step.
        TCM_byte_step_weight (float): TCM weight in the Byte step.
        high_score_matching_thresh (float): Threshold for high-score matching.
        with_longterm_reid (bool): Whether to enable long-term ReID features.
        longterm_reid_weight (float): Weight applied to long-term ReID scores.
        with_longterm_reid_correction (bool): Whether to enable long-term ReID
            correction.
        longterm_reid_correction_thresh (float): Correction threshold for
            regular detections.
        longterm_reid_correction_thresh_low (float): Correction threshold for
            low-score detections.
        dataset (str): Dataset hint used by the association logic.
        **kwargs: Base tracker settings forwarded to :class:`BaseTracker`.

    Attributes:
        with_reid (bool): Whether appearance features are enabled.
        model: ReID model used for appearance extraction when enabled.
        cmc: Camera-motion compensation method.
        active_tracks (list[KalmanBoxTracker]): Currently active tracks.
    """

    def __init__(
        self,
        # ReID & CMC
        reid_weights: Optional[Union[Path, str]],
        device: torch.device,
        half: bool,
        cmc_method: str = "ecc",
        with_reid: bool = True,

        # Hybrid-SORT specific
        low_thresh: float = 0.1,
        delta_t: int = 3,
        inertia: float = 0.05,
        use_byte: bool = True,

        # KF / ReID
        use_custom_kf: bool = True,
        longterm_bank_length: int = 30,
        alpha: float = 0.9,
        adapfs: bool = False,
        track_thresh: float = 0.5,

        # Embedding-guided association
        EG_weight_high_score: float = 4.6,
        EG_weight_low_score: float = 1.3,

        # Two-step toggles / thresholds
        TCM_first_step: bool = True,
        TCM_byte_step: bool = True,
        TCM_byte_step_weight: float = 1.0,
        high_score_matching_thresh: float = 0.7,

        # Long-term reid
        with_longterm_reid: bool = True,
        longterm_reid_weight: float = 0.0,
        with_longterm_reid_correction: bool = True,
        longterm_reid_correction_thresh: float = 0.4,
        longterm_reid_correction_thresh_low: float = 0.4,

        # misc
        dataset: str = "",
        **kwargs: Any,  # BaseTracker parameters
    ):
        # Capture all init params for logging
        init_args = {k: v for k, v in locals().items() if k not in ('self', 'kwargs')}
        super().__init__(**init_args, _tracker_name='HybridSort', **kwargs)

        # store core knobs
        self.low_thresh = float(low_thresh)
        self.delta_t = int(delta_t)
        self.inertia = float(inertia)
        self.use_byte = bool(use_byte)

        self.use_custom_kf = bool(use_custom_kf)
        self.longterm_bank_length = int(longterm_bank_length)
        self.alpha = float(alpha)
        self.adapfs = bool(adapfs)
        self.track_thresh = float(track_thresh)

        self.EG_weight_high_score = float(EG_weight_high_score)
        self.EG_weight_low_score = float(EG_weight_low_score)
        self.TCM_first_step = bool(TCM_first_step)
        self.TCM_byte_step = bool(TCM_byte_step)
        self.TCM_byte_step_weight = float(TCM_byte_step_weight)
        self.high_score_matching_thresh = float(high_score_matching_thresh)

        self.with_longterm_reid = bool(with_longterm_reid)
        self.longterm_reid_weight = float(longterm_reid_weight)
        self.with_longterm_reid_correction = bool(with_longterm_reid_correction)
        self.longterm_reid_correction_thresh = float(longterm_reid_correction_thresh)
        self.longterm_reid_correction_thresh_low = float(longterm_reid_correction_thresh_low)
        self.dataset = str(dataset)

        # ReID module (BotSort-style)
        self.with_reid = bool(with_reid)
        self.model = None
        if self.with_reid and reid_weights is not None:
            self.model = ReID(weights=reid_weights, device=device, half=half).model

        # ECC CMC (BotSort-style)
        self.cmc = get_cmc_method(cmc_method)()

        # container
        self.active_tracks: List[KalmanBoxTracker] = []
        KalmanBoxTracker.count = 0

    @BaseTracker.setup_decorator
    @BaseTracker.per_class_decorator
    def update(self, dets: np.ndarray, img: np.ndarray, embs: np.ndarray = None) -> np.ndarray:
        """
        dets: ndarray [N,6] -> [x1,y1,x2,y2,conf,cls]
        img: HxWxC image
        embs: optional [N,D] appearance features. If None and with_reid=True, we extract features for provided dets.
        Returns: ndarray [M,8]: [x1,y1,x2,y2,track_id,conf,cls,det_ind]
        """
        self.check_inputs(dets, img, embs)
        self.frame_count += 1

        # --- parse inputs: detections are [x1,y1,x2,y2,conf,cls] ---
        n_dets_full = int(dets.shape[0])

        # core boxes + conf
        dets_5 = dets[:, :5].copy() if n_dets_full else dets.reshape(0, 5)
        confs = dets_5[:, 4] if n_dets_full else np.array([])

        # classes (int)
        dets_cls = dets[:, 5].astype(int) if n_dets_full else np.array([], dtype=int)

        # stable det_ind column for downstream logic and output
        dets_ind = np.arange(n_dets_full, dtype=int)

        # helper guards
        def _safe_detind(x: int, n: int) -> int:
            xi = int(x)
            return xi if 0 <= xi < n else -1

        def _safe_cls(x: int, n: int) -> int:
            xi = int(x)
            return xi if 0 <= xi < n else xi  # change to -1 if you prefer strictness

        # convenience view carrying det_ind in col 6th position
        dets_idx = np.hstack([dets_5, dets_ind.reshape(-1, 1)]) if n_dets_full else dets.reshape(0, 6)

        # ECC: compute warp using all current detections (BotSort pattern)
        warp = self.cmc.apply(img, dets_idx) if len(dets_idx) else np.eye(3)

        # Apply camera motion compensation to all active tracks
        for tr in self.active_tracks:
            tr.camera_update(warp)

        # ReID: get features if not provided
        if self.with_reid:
            if embs is None and len(dets_idx):
                # ReID features extracted on [x1,y1,x2,y2] boxes
                embs = self.model.get_features(dets_idx[:, 0:4], img)
            elif embs is None:
                embs = np.zeros((0, 128), dtype=np.float32)  # safe shape

        # FIRST/SECOND stage split (Hybrid semantics)
        inds_low = confs > self.low_thresh
        inds_high = confs < self.det_thresh
        inds_second = np.logical_and(inds_low, inds_high)

        dets_second = dets_idx[inds_second]         # low-conf for BYTE
        remain_inds = confs > self.det_thresh
        dets_keep = dets_idx[remain_inds]

        # NEW: classes aligned to the two sets
        cls_keep = dets_cls[remain_inds]
        cls_second = dets_cls[inds_second]

        # slice embeddings accordingly (if present)
        if embs is None or len(embs) == 0:
            id_feature_keep = np.zeros((len(dets_keep), 1), dtype=np.float32)
            id_feature_second = np.zeros((len(dets_second), 1), dtype=np.float32)
        else:
            id_feature_keep = embs[remain_inds]
            id_feature_second = embs[inds_second]

        # Build dets arrays used by original hybrid code (no det_ind in the math)
        dets_first = dets_keep[:, :5]
        dets_low = dets_second[:, :5]

        # carry det_ind arrays aligned with above
        det_inds_keep = dets_keep[:, 5].astype(int) if len(dets_keep) else np.array([], dtype=int)
        det_inds_second = dets_second[:, 5].astype(int) if len(dets_second) else np.array([], dtype=int)

        # ---- Predict step for existing tracks
        trks = np.zeros((len(self.active_tracks), 6))
        to_del = []
        for t in range(len(trks)):
            pos, kal_score, simple_score = self.active_tracks[t].predict()
            x1, y1, x2, y2 = pos[0].tolist()
            trks[t] = [x1, y1, x2, y2, kal_score, simple_score]
            if np.any(np.isnan(pos)):
                to_del.append(t)
        trks = np.ma.compress_rows(np.ma.masked_invalid(trks))
        for t in reversed(to_del):
            self.active_tracks.pop(t)

        # Prepare motion cues
        velocities_lt = np.array([t.velocity_lt if t.velocity_lt is not None else np.array((0, 0)) for t in self.active_tracks])
        velocities_rt = np.array([t.velocity_rt if t.velocity_rt is not None else np.array((0, 0)) for t in self.active_tracks])
        velocities_lb = np.array([t.velocity_lb if t.velocity_lb is not None else np.array((0, 0)) for t in self.active_tracks])
        velocities_rb = np.array([t.velocity_rb if t.velocity_rb is not None else np.array((0, 0)) for t in self.active_tracks])
        last_boxes = np.array([t.last_observation for t in self.active_tracks])
        k_observations = np.array([k_previous_obs(t.observations, t.age, self.delta_t) for t in self.active_tracks])

        # ===== First association (optionally embedding-guided)
        if self.EG_weight_high_score > 0 and self.TCM_first_step and len(dets_first) and len(trks):
            track_features = np.asarray([t.smooth_feat for t in self.active_tracks], dtype=float)
            emb_dists = embedding_distance(track_features, id_feature_keep).T

            long_emb_dists = None
            if self.with_longterm_reid or self.with_longterm_reid_correction:
                long_track_features = np.asarray(
                    [np.vstack(list(t.features)).mean(0) if len(t.features) else t.smooth_feat for t in self.active_tracks],
                    dtype=float,
                )
                long_emb_dists = embedding_distance(long_track_features, id_feature_keep).T

            matched, unmatched_dets, unmatched_trks = associate_4_points_with_score_with_reid(
                dets_first,
                trks,
                self.iou_threshold,
                velocities_lt,
                velocities_rt,
                velocities_lb,
                velocities_rb,
                k_observations,
                self.inertia,
                ASSO_FUNCS[self.asso_func_name],  # from BaseTracker
                emb_cost=emb_dists,
                weights=(1.0, self.EG_weight_high_score),
                thresh=self.high_score_matching_thresh,
                long_emb_dists=long_emb_dists,
                with_longterm_reid=self.with_longterm_reid,
                longterm_reid_weight=self.longterm_reid_weight,
                with_longterm_reid_correction=self.with_longterm_reid_correction,
                longterm_reid_correction_thresh=self.longterm_reid_correction_thresh,
                dataset=self.per_class and "perclass" or self.dataset,
            )
        elif self.TCM_first_step and len(dets_first) and len(trks):
            matched, unmatched_dets, unmatched_trks = associate_4_points_with_score(
                dets_first,
                trks,
                self.iou_threshold,
                velocities_lt,
                velocities_rt,
                velocities_lb,
                velocities_rb,
                k_observations,
                self.inertia,
                ASSO_FUNCS[self.asso_func_name],
            )
        else:
            matched = np.empty((0, 2), dtype=int)
            unmatched_dets = np.arange(len(dets_first))
            unmatched_trks = np.arange(len(trks))

        # Update matched (update features here)  —— pass cls & det_ind (safe)
        for m in matched:
            det_i = m[0]
            self.active_tracks[m[1]].update(
                dets_first[det_i, :],
                id_feature_keep[det_i, :],
                cls=_safe_cls(cls_keep[det_i], self.nr_classes),
                det_ind=_safe_detind(det_inds_keep[det_i], n_dets_full),
            )

        # ===== BYTE / low-score association (optional)
        if self.use_byte and len(dets_low) > 0 and unmatched_trks.shape[0] > 0:
            u_trks = trks[unmatched_trks]
            iou_left = np.array(ASSO_FUNCS[self.asso_func_name](dets_low, u_trks))
            iou_left_thre = iou_left.copy()
            if self.TCM_byte_step:
                iou_left -= np.array(cal_score_dif_batch_two_score(dets_low, u_trks) * self.TCM_byte_step_weight)

            if iou_left.max() > self.iou_threshold:
                if self.EG_weight_low_score > 0 and self.with_reid:
                    u_tracklets = [self.active_tracks[idx] for idx in unmatched_trks]
                    u_track_features = np.asarray([t.smooth_feat for t in u_tracklets], dtype=float)
                    emb_dists_low = embedding_distance(u_track_features, id_feature_second).T
                    matched_indices = linear_assignment(-iou_left + self.EG_weight_low_score * emb_dists_low)
                else:
                    matched_indices = linear_assignment(-iou_left)
                to_remove_trk_indices = []
                for mm in matched_indices:
                    det_rel, trk_rel = mm[0], mm[1]
                    trk_ind = unmatched_trks[trk_rel]
                    if self.with_longterm_reid_correction and self.EG_weight_low_score > 0 and self.with_reid:
                        if iou_left_thre[det_rel, trk_rel] < self.iou_threshold or emb_dists_low[det_rel, trk_rel] > self.longterm_reid_correction_thresh_low:
                            continue
                    else:
                        if iou_left_thre[det_rel, trk_rel] < self.iou_threshold:
                            continue
                    # do not update features in BYTE pass
                    self.active_tracks[trk_ind].update(
                        dets_low[det_rel, :],
                        id_feature_second[det_rel, :],
                        update_feature=False,
                        cls=_safe_cls(cls_second[det_rel], self.nr_classes),
                        det_ind=_safe_detind(det_inds_second[det_rel], n_dets_full),
                    )
                    to_remove_trk_indices.append(trk_ind)
                unmatched_trks = np.setdiff1d(unmatched_trks, np.array(to_remove_trk_indices))

        # ===== Final chance: IoU vs last boxes
        if unmatched_dets.shape[0] > 0 and unmatched_trks.shape[0] > 0:
            left_dets = dets_first[unmatched_dets]
            left_trks = last_boxes[unmatched_trks]
            iou_left = np.array(ASSO_FUNCS[self.asso_func_name](left_dets, left_trks))
            if iou_left.max() > self.iou_threshold:
                rematched = linear_assignment(-iou_left)
                to_remove_det_indices = []
                to_remove_trk_indices = []
                for mm in rematched:
                    det_rel, trk_rel = mm[0], mm[1]
                    if iou_left[det_rel, trk_rel] < self.iou_threshold:
                        continue
                    det_abs = unmatched_dets[det_rel]
                    trk_abs = unmatched_trks[trk_rel]
                    self.active_tracks[trk_abs].update(
                        dets_first[det_abs, :],
                        id_feature_keep[det_abs, :],
                        update_feature=False,
                        cls=_safe_cls(cls_keep[det_abs], self.nr_classes),
                        det_ind=_safe_detind(det_inds_keep[det_abs], n_dets_full),
                    )
                    to_remove_det_indices.append(det_abs)
                    to_remove_trk_indices.append(trk_abs)
                unmatched_dets = np.setdiff1d(unmatched_dets, np.array(to_remove_det_indices))
                unmatched_trks = np.setdiff1d(unmatched_trks, np.array(to_remove_trk_indices))

        # Mark remaining unmatched tracks
        for m in unmatched_trks:
            self.active_tracks[m].update(None, None)

        # Create new trackers for unmatched high-score detections
        for i in unmatched_dets:
            trk = KalmanBoxTracker(
                dets_first[i, :],
                id_feature_keep[i, :],
                delta_t=self.delta_t,
                use_custom_kf=self.use_custom_kf,
                longterm_bank_length=self.longterm_bank_length,
                alpha=self.alpha,
                adapfs=self.adapfs,
                track_thresh=self.track_thresh,
                cls=_safe_cls(cls_keep[i], self.nr_classes),
                det_ind=_safe_detind(det_inds_keep[i], n_dets_full) if len(det_inds_keep) else -1,
            )
            self.active_tracks.append(trk)

        # Collect outputs (match BotSort/OcSort style)
        outputs = []
        for trk in self.active_tracks[::-1]:
            if trk.last_observation.sum() < 0:
                d = convert_x_to_bbox(trk.kf.x)[0][:4]
            else:
                d = trk.last_observation[:4]

            # Only output fresh tracks and valid det_ind for this frame
            if (trk.time_since_update < 1) and (trk.hit_streak >= self.min_hits or self.frame_count <= self.min_hits):
                outputs.append([
                    *d.tolist(),
                    trk.id + 1,                 # track id
                    float(trk.conf),      # conf
                    int(trk.cls),               # cls (from detection)
                    int(trk.det_ind),           # det index (frame-local)
                ])

        # Remove dead tracks
        i = len(self.active_tracks)
        for trk in self.active_tracks[::-1]:
            i -= 1
            if trk.time_since_update > self.max_age:
                self.active_tracks.pop(i)

        return np.asarray(outputs) if len(outputs) else np.zeros((0, 8), dtype=float)