Validation & analytics¶

Score a trained detector with the standard COCO metrics, and — optionally — render the diagnostic plots you look at after the numbers to understand where the model fails.

DFINE.val — the one-call entry point (COCO metrics, optional plots).
Analytics artifacts — confusion matrix, P/R/F1-vs-confidence curves, per-class AP, and a worst-predictions gallery.
API reference — the underlying evaluate / metric classes.

Running validation¶

from dfine import DFINE

model = DFINE.from_pretrained("dfine-l")

# A COCO dataset root: val/ + annotations/instances_val.json (a stock MS-COCO
# val2017/ layout is auto-detected too).
metrics = model.val(data="coco/")
print(metrics["AP"])  # primary mAP@[.50:.95]
print(metrics["AP50"])  # mAP@0.50

val returns the 12 standard COCO metrics keyed by name (COCO_STAT_NAMES):

Key	Meaning
`AP`	mAP averaged over IoU 0.50–0.95 (the headline number)
`AP50`, `AP75`	mAP at IoU 0.50 / 0.75
`AP_small`, `AP_medium`, `AP_large`	mAP by GT object area
`AR_1`, `AR_10`, `AR_100`	average recall given 1 / 10 / 100 detections per image
`AR_small`, `AR_medium`, `AR_large`	average recall by object area

Label spaces must line up¶

The single most common validation footgun: predicted class ids must match the category_id values in your ground-truth annotations.

Stock MS-COCO uses sparse ids (1..90)

pydfine predicts contiguous 0..N-1 labels by default. To score against stock MS-COCO ground truth, build the model with remap_mscoco_category=True so the postprocessor emits the sparse ids the annotations use:

model = DFINE.from_pretrained("dfine-l", remap_mscoco_category=True)
metrics = model.val(data="coco/", remap_mscoco_category=True)

Datasets produced by yolo_to_coco are already 0-indexed contiguous, so the default remap_mscoco_category=False is correct for them.

Bring your own loader¶

from dfine.train.dataset import build_coco_val_dataloader

loader = build_coco_val_dataloader("coco/", cfg=model.config, batch_size=8)
metrics = model.val(val_loader=loader)  # loader.dataset must carry the GT `.coco`

From the CLI¶

dfine val dfine-l --data coco/
dfine val dfine-l --data coco/ --plots --output-dir runs/val   # + analytics plots

Analytics artifacts¶

Pass plots=True to also write a diagnostic bundle under output_dir (default runs/val). This needs matplotlib (the [train] extra) and assumes contiguous labels (remap_mscoco_category=False).

metrics = model.val(data="coco/", plots=True, output_dir="runs/val")

runs/val/
├── confusion_matrix.png    # predicted-vs-true grid (which classes get confused)
├── pr_curve.png            # per-class precision–recall at IoU 0.50
├── f1_curve.png            # F1 vs confidence  (its peak → best operating point)
├── p_curve.png             # precision vs confidence
├── r_curve.png             # recall vs confidence
└── worst/                  # highest-error frames, GT green / pred red
    ├── 00_err7_000042.jpg
    └── ...

The per-class AP table and the recommended confidence are also logged:

per-class AP  person 0.72  car 0.65  dog 0.51  ...
Best confidence: 0.31  (mean F1 0.68)
Analytics saved to runs/val

Consistent IoU matching (0.5)

All three custom accumulators — ConfusionMatrix, PRCurveMetrics, and WorstPredictions — match a detection to a ground-truth box at IoU ≥ 0.5 (a common single-threshold TP definition). This is a coarser, more interpretable view than COCO's IoU 0.50–0.95 sweep, and it is deliberately the same across every plot so the artifacts agree with each other. The numeric COCO metrics above are independent — they come from the COCO evaluator, not these classes.

Reading each plot¶

Plot	What it answers	How to read it
Confusion matrix	Which classes does the model mix up?	Column-normalized (per true class). Bright off-diagonal cells = systematic confusion; the last row/column is background (false negatives / false positives).
F1–confidence	What `conf` should I deploy at?	The black mean curve peaks at the best trade-off — that x-value is `Best confidence`.
P–confidence	How clean are detections as I raise the bar?	Precision rises with confidence; find where it saturates.
R–confidence	How much do I miss as I raise the bar?	Recall falls with confidence; the crossover with precision is the F1 peak.
PR curve	Per-class quality at IoU 0.50	Area under each curve ≈ that class's AP50; the mean line is mAP@.5.
`worst/` gallery	Where are the failures / label errors?	Frames with the most FP+FN, GT in green, predictions in red — spot mislabeled data and hard cases fast.

Pick a deployment confidence¶

The F1-vs-confidence sweep gives a data-driven answer to "what conf do I pass to predict?" — no eyeballing:

from dfine.train.dataset import build_coco_val_dataloader
from dfine.train.evaluator import evaluate
from dfine.train.metrics import PRCurveMetrics

loader = build_coco_val_dataloader("coco/", cfg=model.config)

# evaluate(plots=True) logs "Best confidence: X" for you; to compute it directly:
prm = PRCurveMetrics(num_classes=model.config.num_classes)  # matches at IoU 0.5
# ... feed prm.process_batch(det_boxes, det_scores, det_classes, gt_boxes, gt_classes) ...
best_conf, mean_f1 = prm.best_confidence()
results = model.predict("street.jpg", conf=best_conf)

Validate every epoch during training¶

DFINE.train auto-wires COCO validation whenever a val loader is present (built from data=, or passed explicitly) — the AP curve is logged and streamed to TensorBoard, and best.pth is saved on each improvement:

model.train(data="coco/", epochs=72)  # val/ split validated each epoch

To also render the full analytics bundle every epoch, pass val_plots=True — each epoch's confusion matrix, P/R/F1 curves, per-class AP, and worst-predictions gallery are written to output_dir/val/epoch{N}/ (kept side by side, not overwritten):

model.train(data="coco/", epochs=72, val_plots=True)
# runs/train/val/epoch0/…, runs/train/val/epoch1/…  (same artifacts as val(plots=True))

Note

val_plots needs matplotlib (the [train] extra) and contiguous labels — it is skipped with a warning when remap_mscoco_category=True or for segment/sem_seg tasks. It is off by default since it adds per-epoch compute and disk.

To customize, pass your own hook — see coco_val_fn (which itself takes plots=, plots_dir=, names=):

from dfine.train.evaluator import coco_val_fn

val_fn = coco_val_fn(model.postprocessor, model.device)
# trainer.fit(train_loader, val_loader=loader, val_fn=val_fn)

API reference¶

DFINE.val¶

val ¶

val(data: str | PathLike | None = None, *, val_loader=None, batch_size: int = 4, num_workers: int = 4, remap_mscoco_category: bool | None = None, plots: bool = False, output_dir: str = 'runs/val') -> dict[str, float]

Evaluate the model on a COCO val set and return the metrics dict.

Provide the data one of two ways (mutually exclusive):

data="path/to/coco" — a COCO root; the val loader is built from val/ + annotations/instances_val.json for you (a stock MS-COCO val2017/ layout is auto-detected too).
val_loader=... — a ready loader from build_coco_dataloader (its dataset must carry the ground-truth .coco).

Returns the 12 standard COCO metrics keyed by name (AP is the primary mAP@[.50:.95]); see :data:~dfine.train.evaluator.COCO_STAT_NAMES. For stock MS-COCO ground truth (sparse category ids), build the model with remap_mscoco_category=True so predicted labels match the annotations.

With plots=True also writes a confusion matrix + precision–recall curves and logs the per-class AP table under output_dir (default runs/val); needs matplotlib (the [train] extra). The confusion matrix assumes contiguous labels, so plots is ignored (with a warning) when remap_mscoco_category=True.

Source code in dfine/model.py

def val(
    self,
    data: str | os.PathLike | None = None,
    *,
    val_loader=None,
    batch_size: int = 4,
    num_workers: int = 4,
    remap_mscoco_category: bool | None = None,
    plots: bool = False,
    output_dir: str = "runs/val",
) -> dict[str, float]:
    """Evaluate the model on a COCO val set and return the metrics dict.

    Provide the data one of two ways (mutually exclusive):

    * ``data="path/to/coco"`` — a COCO root; the val loader is built from
      ``val/`` + ``annotations/instances_val.json`` for you (a stock MS-COCO
      ``val2017/`` layout is auto-detected too).
    * ``val_loader=...`` — a ready loader from ``build_coco_dataloader`` (its
      dataset must carry the ground-truth ``.coco``).

    Returns the 12 standard COCO metrics keyed by name (``AP`` is the primary
    mAP@[.50:.95]); see :data:`~dfine.train.evaluator.COCO_STAT_NAMES`. For stock
    MS-COCO ground truth (sparse category ids), build the model with
    ``remap_mscoco_category=True`` so predicted labels match the annotations.

    With ``plots=True`` also writes a confusion matrix + precision–recall curves and
    logs the per-class AP table under ``output_dir`` (default ``runs/val``); needs
    matplotlib (the ``[train]`` extra). The confusion matrix assumes contiguous labels,
    so ``plots`` is ignored (with a warning) when ``remap_mscoco_category=True``.
    """
    if data is None and val_loader is None:
        raise ValueError("Provide validation data via `data=` or `val_loader=`.")
    if data is not None and val_loader is not None:
        raise ValueError("Pass either `data=` or `val_loader=`, not both.")
    remap_mscoco_category = self._resolve_coco_remap(remap_mscoco_category)
    if data is not None:
        from .train.dataset import build_coco_val_dataloader

        val_loader = build_coco_val_dataloader(
            data,
            cfg=self.config,
            batch_size=batch_size,
            num_workers=num_workers,
            remap_mscoco_category=remap_mscoco_category,
        )

    if plots and remap_mscoco_category:
        # The analytics (confusion matrix / per-class AP) assume contiguous 0..N-1 labels;
        # with the sparse MS-COCO remap the GT label space disagrees, so the plots would be
        # silently wrong. Mirror DFINE.train's guard and skip them (COCO metrics unaffected).
        LOGGER.warning(
            "val(plots=True) needs contiguous labels — plots disabled because "
            "remap_mscoco_category=True (COCO metrics are unaffected)."
        )
        plots = False

    from .train.evaluator import evaluate

    return evaluate(
        self.model,
        self.postprocessor,
        val_loader,
        self.device,
        plots=plots,
        output_dir=output_dir,
        names=self.names,
    )

evaluate¶

dfine.train.evaluator.evaluate ¶

evaluate(model: Module, postprocessor: Module, data_loader: Iterable, device: device, *, iou_type: str = 'bbox', plots: bool = False, output_dir: str | None = None, names: dict[int, str] | None = None) -> dict[str, float]

Evaluate model over data_loader and return COCO metrics.

model runs in eval mode (restored afterwards); each batch is decoded by postprocessor to original-scale xyxy boxes, then scored against the loader's ground truth. Returns a dict keyed by :data:COCO_STAT_NAMES (AP is the primary mAP@[.50:.95]).

With plots=True also writes confusion_matrix.png + pr_curve.png under output_dir (default runs/val) and logs the per-class AP table — see :mod:dfine.train.metrics. names maps class id → label for the plots. The confusion matrix assumes contiguous labels (remap_mscoco_category=False).

Source code in dfine/train/evaluator.py

@torch.no_grad()
def evaluate(
    model: nn.Module,
    postprocessor: nn.Module,
    data_loader: Iterable,
    device: torch.device,
    *,
    iou_type: str = "bbox",
    plots: bool = False,
    output_dir: str | None = None,
    names: dict[int, str] | None = None,
) -> dict[str, float]:
    """Evaluate ``model`` over ``data_loader`` and return COCO metrics.

    ``model`` runs in eval mode (restored afterwards); each batch is decoded by
    ``postprocessor`` to original-scale ``xyxy`` boxes, then scored against the loader's
    ground truth. Returns a dict keyed by :data:`COCO_STAT_NAMES` (``AP`` is the primary
    mAP@[.50:.95]).

    With ``plots=True`` also writes ``confusion_matrix.png`` + ``pr_curve.png`` under
    ``output_dir`` (default ``runs/val``) and logs the per-class AP table — see
    :mod:`dfine.train.metrics`. ``names`` maps class id → label for the plots. The
    confusion matrix assumes contiguous labels (``remap_mscoco_category=False``).
    """
    from faster_coco_eval.utils.pytorch import FasterCocoEvaluator

    evaluator = FasterCocoEvaluator(_coco_gt(data_loader), [iou_type])

    # Analytics plots (confusion matrix / PR curves / worst frames) accumulate locally per
    # process, so under multi-GPU each rank sees only its data shard AND every rank would race
    # writing the same PNGs. Disable them there; the COCO metrics below are still gathered
    # across ranks by FasterCocoEvaluator and remain correct.
    from .distributed import get_world_size

    if plots and get_world_size() > 1:
        LOGGER.warning(
            "val analytics plots are disabled under multi-GPU (per-rank shards would be partial "
            "and ranks would race writing the same files); COCO metrics are unaffected."
        )
        plots = False

    cm = prm = worst = None
    if plots:
        from .metrics import ConfusionMatrix, PRCurveMetrics, WorstPredictions

        cat_ids = [int(c) for c in evaluator.coco_eval[iou_type].params.catIds]
        nc = (max(names) + 1) if names else (max(cat_ids, default=-1) + 1)
        cm, prm, worst = ConfusionMatrix(nc), PRCurveMetrics(nc), WorstPredictions()

    was_training = model.training
    model.eval()
    try:
        for samples, targets in data_loader:
            samples = samples.to(device)
            orig_sizes = torch.stack([t["orig_size"].to(device) for t in targets], dim=0)
            outputs = model(samples)
            results = postprocessor(outputs, orig_sizes)
            evaluator.update({int(t["image_id"].item()): r for t, r in zip(targets, results)})
            if cm is not None:
                _update_analytics(cm, prm, worst, results, targets)
    finally:
        if was_training:
            model.train()

    evaluator.synchronize_between_processes()
    evaluator.accumulate()
    evaluator.summarize()

    stats = evaluator.coco_eval[iou_type].stats.tolist()
    metrics = {name: float(v) for name, v in zip(COCO_STAT_NAMES, stats)}
    LOGGER.info(f"{rule(f'eval · {iou_type}', 'cyan')}  {metrics_line(metrics)}")

    if cm is not None:
        from .metrics import save_val_analytics

        save_val_analytics(
            evaluator.coco_eval[iou_type],
            cm,
            output_dir or "runs/val",
            names,
            prm=prm,
            worst=worst,
        )
    return metrics

dfine.train.evaluator.coco_val_fn ¶

coco_val_fn(postprocessor: Module, device: device, *, iou_type: str = 'bbox', plots: bool = False, plots_dir: str | Path | None = None, names: dict[int, str] | None = None) -> Callable[[nn.Module, Iterable], dict[str, float]]

Build a (module, loader) -> metrics closure for Trainer.fit(val_fn=…).

Captures the postprocessor/device so the trainer can score the (EMA) module each epoch: trainer.fit(train_loader, val_loader=…, val_fn=coco_val_fn(pp, dev)).

With plots=True the analytics bundle (confusion matrix, P/R/F1 curves, per-class AP, worst-predictions gallery) is also written every epoch under plots_dir/epoch{N}/ (plots_dir defaults to runs/train/val), where N counts validation calls from 0 — so each epoch's plots are kept side by side rather than overwritten. names maps class id → label for the plots. The analytics assume contiguous labels (remap_mscoco_category=False).

Source code in dfine/train/evaluator.py

def coco_val_fn(
    postprocessor: nn.Module,
    device: torch.device,
    *,
    iou_type: str = "bbox",
    plots: bool = False,
    plots_dir: str | Path | None = None,
    names: dict[int, str] | None = None,
) -> Callable[[nn.Module, Iterable], dict[str, float]]:
    """Build a ``(module, loader) -> metrics`` closure for ``Trainer.fit(val_fn=…)``.

    Captures the ``postprocessor``/``device`` so the trainer can score the (EMA) module
    each epoch: ``trainer.fit(train_loader, val_loader=…, val_fn=coco_val_fn(pp, dev))``.

    With ``plots=True`` the analytics bundle (confusion matrix, P/R/F1 curves, per-class AP,
    worst-predictions gallery) is also written **every epoch** under
    ``plots_dir/epoch{N}/`` (``plots_dir`` defaults to ``runs/train/val``), where ``N``
    counts validation calls from 0 — so each epoch's plots are kept side by side rather than
    overwritten. ``names`` maps class id → label for the plots. The analytics assume
    contiguous labels (``remap_mscoco_category=False``).
    """
    base = Path(plots_dir) if plots_dir is not None else Path("runs/train") / "val"
    state = {"call": 0}

    def _val_fn(module: nn.Module, loader: Iterable) -> dict[str, float]:
        out_dir = str(base / f"epoch{state['call']}") if plots else None
        metrics = evaluate(
            module,
            postprocessor,
            loader,
            device,
            iou_type=iou_type,
            plots=plots,
            output_dir=out_dir,
            names=names,
        )
        state["call"] += 1
        return metrics

    return _val_fn

Metrics¶

dfine.train.metrics.ConfusionMatrix ¶

ConfusionMatrix(num_classes: int, conf: float = 0.25, iou_thresh: float = 0.5)

Predicted-vs-true detection counts, matched by IoU (ultralytics-style).

matrix[p, t] counts predictions of class p matched to a true box of class t; index nc (the extra last row/col) is background — matrix[nc, t] are missed GT (false negatives), matrix[p, nc] are spurious detections (false positives). Feed one image at a time with :meth:process_batch.