56 open-source projects,
every number verifiable

Unit timed: one csvstat run over a 550k-row CSV (end to end)

$ python csvkit/utilities/csvstat.py big.csv

Bit-identical: every reported statistic equals stock csvkit

Unit timed: one random graph through six analyses (betweenness, PageRank, clustering, …)
A library with no entry point: timing covers a benchmark script that exercises its public API.

$ python ao_bench.py

Every reported field identical across 10 checked graphs

Unit timed: one McMaster image: read → add noise → SwinIR-M colour denoising → PNG written → PSNR/SSIM

$ python main_test_swinir.py --task color_dn --noise 15 --model_path model_zoo/swinir/005_colorDN_DFWB_s128w8_SwinIR-M_noise15.pth --folder_gt testsets/McMaster

Every output PNG within 2 levels of stock; PSNR against ground truth moves ≤0.01 dB

Unit timed: one week of a climate summary chain (coarsen → rolling → groupby_bins → anomaly → daily max → NetCDF write)
A library with no entry point: timing covers a benchmark script that exercises its public API.

$ python ao_bench.py

Statistics within 1.1e-5 absolute of the stock chain

Unit timed: one batch of 32 images through ViT-B/32 zero-shot classification (decode → preprocess → image tower → probabilities)

$ python main_4cf6e0.py

Zero-shot accuracy 301/1034, identical to stock

Unit timed: one test image: JPEG decode → transform → generator → PNG written to disk

$ python test.py --dataroot datasets/horse2zebra/testA --name horse2zebra_pretrained --model test --no_dropout

Worst pixel differs by 2 uint8 levels (PSNR 64.9 dB), less than the stock program differs from itself

Unit timed: one sample (200 generated tokens, pythia-410m, bf16)

$ python litgpt/__main__.py generate checkpoints/EleutherAI/pythia-410m --max_new_tokens 200 --num_samples 8

Bit-identical against the frozen bf16 reference, pinned and held-out prompts

Unit timed: one batch of 64 ESC-50 clips through CLAP zero-shot classification (audio → log-mel → audio tower → scores)

$ python -m open_clip_train.main --model CLAP-HTSAT-tiny-Roberta-base-fused --pretrained laion --audio-zeroshot-dataset ashraq/esc50 …

ESC-50 zero-shot top-1 87.85% / top-5 98.85%, identical to stock

Unit timed: one batch of series fitted and forecast by StatsForecast
A library with no entry point: timing covers a benchmark script that exercises its public API.

$ python ao_bench.py

Bit-identical forecasts

Unit timed: one DINOv2 ViT-L/14 forward over a 16×3×518×518 batch (random weights, built from the repo's own hubconf)

$ python bench_dino.py

Output features within cosine 0.99998 / relative L2 0.0065 of the stock fp32 program

Unit timed: one 100-frame chunk of a REDS clip: frames read, BasicVSR++ 4x video super-resolution, 100 PNGs written

$ python inference/inference_basicvsrpp.py

Written PNGs within 1 grey level of the stock program's on 99.9% of pixels (PSNR 61.9 dB against it).

Unit timed: one sentence synthesised end to end (phonemiser → Tacotron2-DDC → vocoder → waveform)

$ python TTS/bin/synthesize.py --text "The quick brown fox jumps over the lazy dog, again and again." --model_name tts_models/en/ljspeech/tacotron2-DDC --out_path out.wav --use_cuda

Bit-identical waveform (PSNR 124 dB) on pinned and held-out sentences

Unit timed: one prompt: 96 new tokens greedy-generated by the 1.7B gated-attention Qwen3 checkpoint (batch 1)

$ python demo.py --generate --prompts prompts.txt

Next-token probabilities within 0.0078 of the stock fp32 program (relative L2 0.0028)

Unit timed: one frame (8 in-the-wild demo frames, batches of 4): person crop → Sapiens 0.3B 308-keypoint pose → keypoints written

$ python lite/demo/vis_pose.py sapiens_0.3b_goliath_best_goliath_AP_573_torchscript.pt2 --input <frames> --output-root <out> …

Pose heatmaps within 0.011 of the stock fp32 model (PSNR 82.8 dB), within 0.014 on held-out batch sizes.

Unit timed: one 33-frame clip encoded by the Wan 2.1 VAE (diffusers)

$ python wan21_encode.py

Latents within PSNR 73.6 dB of the fp32 encode

Unit timed: one ResNet-50 training step at the reference recipe's batch size

$ python train.py --model resnet50

Verified against the frozen stock reference, including a held-out set

Unit timed: one training iteration on the Tanks&Temples train scene (render → loss → backward → densify → step)

$ python train.py -s tandt_db/tandt/train -m output/train --iterations 7000

Per-iteration loss within 2.5e-6 relative of stock

Unit timed: one TSN R50 training iteration on Kinetics-400 at the config's batch size

$ python tools/train.py configs/recognition/tsn/tsn_imagenet-pretrained-r50_8xb32-1x1x3-100e_kinetics400-rgb.py

Verified against the frozen stock reference, including a held-out set

Unit timed: one demo frame: person detection → Sapiens2-1B 308-keypoint pose → skeleton overlay written

$ python tools/vis/vis_pose.py <detector> configs/keypoints308/…/sapiens2_1b_keypoints308_shutterstock_goliath_3po-1024x768.py <pose.safetensors> …

Pose heatmaps within 0.045 of the stock fp32 model (PSNR 85.0 dB), within 0.033 on a held-out frame; all 100 frames written.

Unit timed: one input image: decode → 4x RRDBNet (fp16) → post-process → encode + write

$ python inference_realesrgan.py -n RealESRGAN_x4plus -i inputs

PSNR 59.8 dB against the stock output, under one 8-bit code

Unit timed: one KITTI LiDAR scan through the PV-RCNN demo: read, voxelise, detect

$ python demo.py --cfg_file cfgs/kitti_models/pv_rcnn.yaml --ckpt pv_rcnn_8369.pth --data_path ${POINT_CLOUD_DATA}

Detections within 6e-4 of the stock program's, 170x below the model's 0.1 score threshold, also on held-out scans.

Unit timed: one image through PP-OCRv6 medium: decode → text detection → recognition → CTC decode

$ python run_ocr.py

Text and boxes identical on held-out images; the most sensitive pinned page differs by one or two characters (relative L2 0.054)

Unit timed: one image sequence: decode → recurrent CUT3R reconstruction → per-frame depth, confidence, colour and camera files written

$ python demo.py --model_path src/cut3r_512_dpt_4_64.pth --seq_path examples/001 --size 512 --vis_threshold 1.5 --output_dir tmp

Point clouds within relative L2 1.9e-4 of stock; written PNGs decode to the same pixels

Unit timed: one encode() request (a list of texts → unit-norm embeddings), as the example script issues it

$ python examples/sentence_transformer/applications/computing-embeddings/computing_embeddings.py

Cosine 0.99999 against the fp32 output

Unit timed: one output frame of the 10-minute-tutorial trailer rendered end to end (decode → effects → composite → ffmpeg encode)

$ python docs/_static/code/getting_started/moviepy_10_minutes/trailer.py

At most 1 code of pixel difference (PSNR 51.7 dB)

Unit timed: one ResNet-50 training step (end to end)

$ python train.py <data> --model resnet50

Verified against the frozen stock reference, including a held-out set

Unit timed: one image through detect.py: read → letterbox → yolov5s forward → NMS → annotated write

$ python detect.py --source data/images --weights yolov5s.pt

Bit-identical on pinned images and holdout

Unit timed: one optimizer step of supervised fine-tuning of TinyLlama-1.1B-Chat on GSM8K (batch 16, bf16)

$ python finetune.py --model_name_or_path=ckpts/tinyllama-1.1b-chat --dataset_name=gsm8k --model_family=llama2 …

Per-step training loss within 0.11% of the stock program's (its own repeat runs differ by 0.15%); gradient cosine 0.9999.

Unit timed: one batch-1 autoregressive decode run through DeepSpeed's inference engine

$ python benchmark/deepspeed/run.py

Verified against the frozen stock reference, including a held-out set

Unit timed: one nerfacto training iteration on a 4096-ray batch

$ ns-train nerfacto --data data/blender/lego

Training loss within 0.011% of the stock program's per step, about 0.0005 dB of rendered PSNR.

Unit timed: the whole training loop: 71 optimizer steps of GPT-2 (124M) on wikitext-2, warm-up included

$ python run_clm.py --model_name_or_path openai-community/gpt2 --dataset_name wikitext --dataset_config_name wikitext-2-raw-v1 …

Train loss 3.387 vs 3.376 stock, eval perplexity 19.74 vs 19.55

Unit timed: one DiT-XL/2 training step, batch 16 on one GPU

$ torchrun --nnodes=1 --nproc_per_node=1 train.py --model DiT-XL/2 --data-path <train dir> --global-batch-size 16 …

Per-step loss within 1.6e-4 of stock fp32; gradients match (cosine 1.000, relative L2 0.004), on held-out batches too.

Unit timed: one COCO image, from decode to rescaled boxes

$ python ultralytics/cfg/__init__.py predict model=yolov12n.pt source=data/val2017 …

Detections bit-identical to the stock program (max abs diff 0), also on 24 held-out images.

Unit timed: one clip encoded by the Wan 2.2 VAE (diffusers)

$ python wan22_encode.py

Latents within PSNR 78.3 dB of the fp32 encode

Unit timed: one generated 256×256 image (250 sampling steps + VAE decode)

$ python inference.py --config configs/reproductions/lightningdit_xl_vavae_f16d32_64ep_cfg.yaml --demo

Images match the stock output at 41.6 dB PSNR (40.0 dB on held-out batch sizes).

Unit timed: one clip: decode → upload to the GPU → CoTracker3 offline tracks a 10×10 grid → tracks drawn on every frame → mp4 written

$ python demo.py --grid_size 10

Tracked points, visibility flags and rendered marker colours bit-identical to the stock program's, also on held-out clips and grid sizes

Unit timed: one clip scored with detect-adaptive end to end (decode → per-frame score → cut decision → scene list)

$ python scenedetect/__main__.py -i demo.mp4 detect-adaptive list-scenes -n

Bit-identical scores and an identical cut list

Unit timed: one image + prompt through demo/inference_on_a_image.py (SwinT backbone, BERT, deformable decoder → boxes)

$ python demo/inference_on_a_image.py -c groundingdino/config/GroundingDINO_SwinT_OGC.py -p weights/groundingdino_swint_ogc.pth -i .asset/cat_dog.jpeg -o out -t "cat ear."

Boxes move by less than 0.1 px

Unit timed: one DAVIS 2017 val video tracked end to end with SAM 2.1 hiera-b+: frames decoded, frame-0 masks added, every frame propagated, label PNGs written

$ python ./tools/vos_inference.py --sam2_cfg configs/sam2.1/sam2.1_hiera_b+.yaml --sam2_checkpoint ./checkpoints/sam2.1_hiera_base_plus.pt --base_video_dir <DAVIS>/JPEGImages/480p … --output_mask_dir ./outputs/davis_2017_pred_pngs

Written label PNGs identical to the stock program's on at least 99.9% of pixels (only a few object-boundary pixels change label), also on held-out videos

Unit timed: one txt2img request (SD 1.5, 20 steps, 512-768 px) to PNG response

$ python launch.py --api --nowebui

Decoded images at PSNR 48.2 dB against the stock server's; 51.6 dB on held-out requests.

Unit timed: one REPA-E training step at 256×256 (SiT-B/2 + SD-VAE + DINOv2-B), batch 4

$ python train_repae.py --model=SiT-B/2 --vae=f8d4 --enc-type=dinov2-vit-b --batch-size=4 --mixed-precision=fp16 …

Per-step loss within 0.9% of stock, inside what the command's own fp16 already moves

Unit timed: one 33-frame clip encoded by the HunyuanVideo VAE (diffusers)

$ python hunyuan_encode.py

Latents within PSNR 50.9 dB of the fp32 encode

Unit timed: one short clip interpolated 2x end to end: decode → SSIM check → IFNet → encode

$ python inference_video.py --video=demo.mp4 --exp=1

Worst pixel 1 code off (PSNR 57.7 dB)

Unit timed: one 1280×720 JPEG frame: decode → SAM 3 text-prompted detection + segmentation ("person") → 0.5 threshold, masks upsampled to full size

$ python perf/confirm_e2e.py --arm baseline --frames 64

Detection verdicts identical on every query; scores and masks within cosine 0.99999 of stock

Unit timed: one scene reconstruction (decode, VGGT-1B, unprojection, COLMAP)

$ python demo_colmap.py --scene_dir=examples/kitchen

World point cloud matches the stock program's at PSNR 97.2 dB (relative L2 1.8e-4).

Unit timed: one clip streamed to the faster_whisper server over a fresh websocket session (first packet → transcript covering the clip → end of audio), Whisper small
A library with no entry point: timing covers a benchmark script that exercises its public API.

$ python ao_bench.py

Transcripts identical to the stock program's, word for word, on pinned and held-out clips

Unit timed: one multi-view Fast3R ViT-L/512 reconstruction of 20 demo-video frames

$ python bench_infer.py

Passed the output check against the stock program, frozen before optimization began

Unit timed: one optimizer step (8 x 64x128-token micro-batches, 65,536 tokens)

$ python main.py loader.global_batch_size=512 … model=small algo=bd3lm data=lm1b-wrap model.length=128 block_size=16 mode=train …

Training loss within 1.1% of the stock program's on the same batches; 0.77% on held-out batches.

Unit timed: one LJSpeech training step (generator → MPD + MSD discriminators → generator update), batch 16

$ python train.py --config config_v1.json

Mel L1 after two epochs 0.544 / 0.539 vs stock 0.545 / 0.544

Unit timed: one training iteration of train.py (shakespeare_char)

$ python train.py config/train_shakespeare_char.py --compile=False --max_iters=400 --eval_interval=200 --log_interval=10

Loss trajectory within 8e-5 relative, gradient cosine ≥ 0.99999

Unit timed: one clip through the VAE round trip (encode → decode → frames, diffusers)

$ python decode.py

PSNR 62.9 dB against the fp32 reference

Unit timed: one photo: face detection, restoration, 2x background upsampling, write-out

$ python inference_gfpgan.py -i inputs/whole_imgs -o results -v 1.3 -s 2

Restored images at PSNR 48.4 dB against the stock program's; 47.4 dB on held-out photos.

Unit timed: one LoRA fine-tuning step of VibeVoice-ASR on the toy dataset (decode → tokenise → collate → forward/backward), measured on an RTX 5090

$ python finetuning-asr/lora_finetune.py --model_path microsoft/VibeVoice-ASR --data_dir finetuning-asr/toy_dataset --output_dir out --num_train_epochs 3 --per_device_train_batch_size 1 --learning_rate 1e-4 --bf16 --report_to none

Loss within 6.5e-2 relative, gradient cosine 0.885; a switch restores the stock path, the user decides

Unit timed: one pre-training step of tabicl.train (stage-1 recipe, batch 32 synthetic tables from the mix-SCM prior, bf16)

$ python bench_train.py

Per-step loss within 2e-5 relative of the stock program's (its own repeat-run spread); gradient cosine 1.0

Unit timed: one training step of train.py on a batch of 16 coco128 images

$ python train.py --data coco128.yaml --weights yolov5s.pt --img 640 --epochs 3 --batch-size 16

Loss within 2e-3 relative, gradient cosine ≥ 0.996

Unit timed: one Faster R-CNN R50-FPN training iteration on COCO at the config's batch size

$ python tools/train.py configs/faster_rcnn/faster-rcnn_r50_fpn_1x_coco.py

Verified against the frozen stock reference, including a held-out set

Unit timed: one CDiT-XL/2 bf16 training step, batch 4 at 224 px on synthetic trajectories

$ python train.py --config config/ao_synth.yaml --epochs 2 --bfloat16 1 --torch-compile 0 …

Loss within 1.2e-7 of stock; gradient cosine 1.0, inside the stock program's own run-to-run spread

Unit timed: one audio file transcribed (--model small, float16, no alignment)

$ python whisperx/__main__.py audio.wav --model small --compute_type float16 --no_align --output_dir out

Features within 0.016 of the stock implementation

Unit timed: one series: TimeSeries → Scaler → NBEATSModel → fit (3 epochs) → forecast
A library with no entry point: timing covers a benchmark script that exercises its public API.

$ python ao_bench.py

Loss within 2e-3 relative, MAPE unchanged