Knee MRI Abnormality Detection

A weakly supervised, multi-series deep learning pipeline that predicts twelve knee abnormalities from raw MRI DICOM studies, built for the 2026 RSNA Knee Abnormality Detection challenge on Kaggle. Every module, test, and experiment in this project was designed and written by me.

819,078DICOM slices audited
4,407studies, only 58 expert-labeled
0.686 → 0.784pooled out-of-fold macro AUC
12,000+lines of library code

The problem

Each study is a full knee MRI session: several series (sagittal, coronal, axial; fluid-sensitive or not; fat-suppressed or not), each made of dozens of DICOM slices from scanners all over the world. The task is to output a probability for each of twelve findings: ACL and MCL injury, medial and lateral meniscal tears, medial, lateral, and patellofemoral osteoarthritis, effusion, synovitis, Baker's cyst, bone contusion, and fracture.

The hard part is supervision. Only 58 of 4,407 training studies have expert labels. Every study has its original radiology report, often not in English, but reports are not available at test time, so the final model must work from images alone.

Validation has few positives per finding Positive studies among the 58 expert-labeled studies. Rarer findings give each AUC less evidence.

Source: reports/data_audit/label_summary.csv

Approach

  1. Audit before modelling. A read-only, resumable inventory of every DICOM header and an exhaustive pixel-decode pass produced explicit exclusion policies for corrupt series, plus geometry audits of slice orientation, pixel spacing, and physical field of view. (dicom_audit.py, physical_geometry_audit.py)
  2. Turn reports into labels. A local Qwen2.5-3B-Instruct extractor read multilingual reports and emitted per-finding probabilities for the 4,349 unlabeled studies. Benchmarked against the 58 gold studies, it reached macro AUC 0.753. An equal-mean ensemble of three public CC0 label sets reached 0.896 and became the weak target. (report_extractor.py, weak_label_benchmark.py)
  3. Leakage-safe validation. Deterministic multilabel-stratified five-fold splits over the gold studies. Gold studies are removed from weak pretraining entirely, so no validation image is ever seen in training. (gold_folds.py)
  4. Model. A DINOv2 ViT-S/14 encoder embeds each slice. Target-specific attention pools slices into a series vector, series metadata is appended, and a second target-specific attention pools series into a study vector. Weak pretraining on 4,349 studies is followed by gold fine-tuning per fold. (imaging_training.py)
  5. Offline inference. Each test study is decoded once, every fold checkpoint scores it, and scores are rank-averaged per target to cancel calibration drift between checkpoints. The submission is validated for exact columns, identifier order, and finite in-range probabilities before it is written. (imaging_inference.py, submission.py)

Preprocessing: why the full field of view stays primary

Scanners differ widely. Across 24,386 series, the shorter in-plane field of view spans 70 to 310 mm, with the middle 90% between 140 and 182 mm, and pixel spacing runs from 0.17 to 0.56 mm over the same percentiles. A fixed crop would cut anatomy from some series and pad others, so every series is fit into 224 × 224 by physical size with symmetric padding. A square crop is only an optional detail view, and the audit picked its size: the largest candidate that needs no padding for at least 90% of series, overall and in every plane.

140 mm is the largest crop that clears the 90% bar Share of series covered without padding, at each of five candidate crop sizes.

Source: reports/dicom_audit/physical_geometry_summary.json

Weak labels: which source to trust

Scoring each label source against the expert labels came before any imaging change. The ensemble beat the Qwen extractor on all twelve findings, with macro AUC 0.896 against 0.753. The ensemble covers 57 of the 58 gold studies; Qwen covers all 58. This comparison chose a label source. It does not estimate leaderboard performance.

The label ensemble tracks expert labels more closely on every finding AUC of each weak-label source against gold labels, ordered by ensemble AUC. Axis starts at 0.5, chance level.

Sources: gold_benchmark_summary.json, public_weak_label_benchmark.json

Results

Out-of-fold AUC pooled across all 58 gold studies, where every prediction comes from a model that never saw that study. Between v1 and v2, only the weak-label source changed; architecture, cache, folds, and fine-tuning settings were held fixed.

Better labels lifted nine of twelve findings; three dropped Pooled out-of-fold AUC per finding, ordered by v2 AUC. Axis starts at 0.5, chance level. The table below lists every value.
Findingv1: Qwen labelsv2: label ensembleChange
ACL0.6350.755+0.120
MCL0.6440.603−0.041
Medial meniscus0.5990.780+0.181
Lateral meniscus0.7540.733−0.021
Medial OA0.8110.888+0.078
Lateral OA0.6960.652−0.044
Patellofemoral OA0.7350.766+0.031
Effusion0.8150.919+0.104
Synovitis0.7490.792+0.043
Baker's cyst0.7410.889+0.149
Bone contusion0.5210.822+0.301
Fracture0.5360.803+0.267
Macro AUC0.6860.784+0.097

Five-fold mean macro AUC rose from 0.724 to 0.812, and every individual fold improved. Brier score fell from 0.197 to 0.160. Source: reports/training.

Every fold improved, by 0.049 to 0.167 Best held-out macro AUC per fold. Each fold holds only 11 or 12 studies, so single-fold values are noisy.

With 58 validation studies, these numbers carry wide uncertainty. They guided engineering decisions; they are not a leaderboard score, and this is not a clinical tool.

Code walkthrough

Two excerpts from the repository, unmodified apart from trimmed input checks. Everything else is one click away on GitHub.

Target-specific attention over slices and series

Each finding gets its own attention weights, so the ACL head can focus on sagittal slices through the notch while the effusion head looks at the suprapatellar recess. Series metadata (plane, fluid sensitivity, fat suppression) is fused before series-level pooling.

src/rsna/imaging_training.py · forward pass View full source
features = self.slice_projection(self._encode(images, encoder_chunk_size))
series_target_features: list[Any] = []
offset = 0
for length in series_lengths:
    part = features[offset : offset + length]
    scores = self.slice_attention(part)
    weights = torch.softmax(scores, dim=0)
    series_target_features.append(torch.einsum("st,sd->td", weights, part))
    offset += length
stacked = torch.stack(series_target_features, dim=0)
expanded_metadata = metadata[:, None, :].expand(-1, len(TARGET_COLUMNS), -1)
enriched = self.series_projection(
    torch.cat([stacked, expanded_metadata], dim=-1)
)
series_scores = torch.einsum(
    "rtd,td->rt", enriched, self.series_queries
) / math.sqrt(feature_dimension)
series_weights = torch.softmax(series_scores, dim=0)
study_features = torch.einsum("rt,rtd->td", series_weights, enriched)
return torch.sum(study_features * self.head_weight, dim=-1) + self.head_bias

Rank-averaged fold ensemble

Pooled out-of-fold analysis showed that fold checkpoints rank studies well but disagree on score scale. Averaging percentile ranks per target instead of raw probabilities removes that drift at test time without using labels.

src/rsna/imaging_inference.py View full source
def _percentile_ranks(values: np.ndarray) -> np.ndarray:
    """Return average percentile ranks with deterministic tie handling."""
    order = np.argsort(values, kind="stable")
    sorted_values = values[order]
    ranks = np.empty(len(values), dtype=np.float64)
    start = 0
    while start < len(values):
        stop = start + 1
        while stop < len(values) and sorted_values[stop] == sorted_values[start]:
            stop += 1
        ranks[order[start:stop]] = (start + stop - 1) / 2.0
        start = stop
    return (ranks + 0.5) / len(values)


def rank_average_probabilities(model_probabilities: Any) -> np.ndarray:
    """Normalize each model/target score scale before ensemble averaging."""
    values = np.asarray(model_probabilities, dtype=np.float64)
    ranked = np.empty_like(values)
    for model_index in range(values.shape[0]):
        for target_index in range(values.shape[2]):
            ranked[model_index, :, target_index] = _percentile_ranks(
                values[model_index, :, target_index]
            )
    return np.mean(ranked, axis=0)

Full source map

ModulePurpose
dicom_audit.pyRead-only, resumable DICOM inventory and decoder audit
dicom_loader.pyGeometry-aware, bounded-memory DICOM series loading
orientation_audit.pyPer-slice orientation variation analysis
physical_geometry_audit.pyPixel spacing and physical field-of-view audit
pixel_integrity.pySeries exclusion policy from exhaustive pixel decoding
preprocessing_cache.pyResumable full-cohort preprocessing and tensor caching
report_extractor.pyMultilingual LLM extraction of findings from radiology reports
weak_label_benchmark.pyBenchmark and ensemble report-derived labels against gold
gold_folds.pyDeterministic, leakage-safe multilabel folds
imaging_training.pyMulti-series model, weak pretraining, and fold fine-tuning
oof_evaluation.pyStrict pooled out-of-fold evaluation
evaluation.pyCompetition metrics implemented from scratch
imaging_inference.pyOffline multi-checkpoint inference and rank ensemble
submission.pyStrict submission generation and validation
kaggle_package.pySelf-contained offline Kaggle model package

Also in the repository: 18 pipeline scripts, 7 notebooks, and 22 pytest test modules.

Tools

Python, PyTorch, DINOv2, Hugging Face Transformers (Qwen2.5-3B-Instruct), pydicom, NumPy, pandas, pytest, Jupytext, and Google Cloud Spot L4 GPUs with checkpointed, resumable training runs.