MIC fleet candidates
Twelve uncovered models are solid candidates. bridgetower, grounding_dino and perceiver give the clearest value; pvt and lw_detr are the cheapest.
A good candidate has many failing tests on main and failures a MIC catches: load, config and processor errors, forward crashes, dtype or device bugs, numeric mismatches in logits or loss, and training or gradient bugs.
Most failures are long-standing: every top candidate except perceiver was already failing 30 to 90 days ago. A MIC recorded today would not have caught them as they appeared; it blocks at creation until the bug is fixed, or guards against the next regression once someone reviews which outputs are correct. Perceiver is the exception: a real regression that started 30 to 60 days ago and cleared on 10-20.
Strong candidates
"Failing tests" have at least one failure in the last 28 days; "Failures" sums failing runs per test over those days (the GPU model job runs daily, so 28 means every day).
| # | Model type | Failing tests | Failures (28 d) | What breaks | MIC catches it? | Suggested checkpoint |
|---|---|---|---|---|---|---|
| 1 | bridgetower | 5 | 119 | Processor won't load: "Unrecognized image processor" (4 integration tests) | Yes — loading is the first MIC step, so the MIC can't be built until it's fixed | BridgeTower/bridgetower-base |
| 2 | grounding_dino | 4 | 104 | Detection logits and loss mismatch (expected loss 1.1147, got 1.0834) | Yes — forward logits and loss; mm_grounding_dino (3 tests, 78 failures) could be a sibling | IDEA-Research/grounding-dino-tiny |
| 3 | perceiver | 8 | 168 | "Expected all tensors to be on the same device" in cat; real regression, now cleared | Yes — on the GPU checkpoint run (not the CPU fixture) | deepmind/language-perceiver |
| 4 | timesfm | 9 | 252 | fp16 overflow in 8 eager-vs-SDPA tests, one root cause | Partly — only if the contract runs an fp16 forward | google/timesfm-2.0-500m-pytorch |
| 5 | minicpm3 | 7 | 124 | KeyError: 'factor' in the rope config (2 tests); the rest are FA2 or compile | Yes for the config bug — pin the native class, not the repo's remote code | openbmb/MiniCPM3-4B |
| 6 | janus | 2 | 56 | Processor and model disagree: 0 image tokens vs 1,179,648 image features | Yes | deepseek-community/Janus-Pro-1B |
| 7 | blt | 2 | 56 | BltConfig has no num_hidden_layers | Yes — checkpoint is in a personal namespace | itazap/blt-1b-hf |
| 8 | luke | 3 | 64 | Base and large logits mismatch; tokenizer ids | Yes | studio-ousia/luke-base |
| 9 | pvt | 2 | 56 | Classification logits off by 0.074 | Yes — cheap (13M parameters) | Zetatech/pvt-tiny-224 |
| 10 | lw_detr | 2 | 40 | Detection head mismatch (tiny and xlarge) | Yes — cheap (13M); personal namespace | stevenbucaille/lwdetr_tiny_30e_objects365 |
| 11 | oneformer | 2 | 44 | Segmentation logits mismatch | Yes | shi-labs/oneformer_ade20k_swin_tiny |
| 12 | qwen3_vl_moe | 10 | 256 | Weight conversion fails at load (8 tests, one root cause) | Partly — real checkpoint is 31B: fixture only, saved in the Hub layout to reach the bug | Qwen/Qwen3-VL-30B-A3B-Instruct |
Weaker candidates
MIC-catchable failures, but each has a catch: a family the fleet already covers, mostly generation-text drift, or a real checkpoint too big for a 24 GB GPU.
| Model type | Failing tests | Failures (28 d) | What breaks | Catch |
|---|---|---|---|---|
| smollm3 | 2 | 56 | 3B logits mismatch; export static-cache text | Llama-family models are already well covered |
| kosmos2 | 3 | 84 | interpolate_pos_encoding and captioning logits | Partly generation text |
| mimi | 2 | 56 | Audio codec outputs differ | Only if the probe covers audio codes |
| diffusion_gemma | 6 | 166 | Minified-model forward mismatch (118 of 120 values) | Real checkpoint is 26B: fixture-only MIC |
| recurrent_gemma | — | — | 2B generate/sample text; a non-contiguous bias crash | Gated checkpoint; mostly generation text |
| seamless_m4t | 3 | — | Processor and generation errors | Mostly generation |
| mixtral | — | — | Only the tiny test model's logits fail | Real model is 47B |
| qwen2_moe | — | — | Logits mismatch | Real model is 14.3B, about 29 GB in bf16 |
| univnet | 2 | — | Two scalar mismatches | Checkpoint is in a personal namespace |
| blip_2, rag, bloom, plbart, encoder_decoder | — | — | Mostly greedy or beam-search output diffs | Generation text, which a MIC doesn't probe |
Excluded
- Already covered despite high counts: mistral3 (67 tests; 65 are an
OSError [Errno 5]on the runner's cache lock, i.e. infrastructure), whisper (40), smolvlm (20, network), gemma3n (14), and the other fleet models. ministral3 checkpoints reportmodel_type=mistral3, so it is partly covered. - Infrastructure, Hub or network: moshi (81 tests; 68 are a 401 on a tokenizer repo), minicpmv4_7 (70; 64 are the cache-lock I/O error), embedding_gemma2 (404), aria (remote protocol error), cohere2_moe (corrupt cached safetensors), cohere2_vision (OOM and a local path).
- Cascade: idefics3 (46 tests, 752 failures) is one multi-GPU beam-search test triggering a CUDA device-side assert that spreads through the rest.
- Multi-GPU, offload, FA2, compile, quantization or kernels only: sam, sam_hq, depth_pro, timm_wrapper, pe_video, pe_audio_video, glm4v_moe, lfm2_moe, higgs_audio_v2_tokenizer, gpt_bigcode, kimi_k25, qwen3_moe, deepseek_v32, deepseek_v3, flex_olmo, paligemma, paligemma2, colpali, cohere_compass, pegasus_x, bark, falcon, bamba, mllama, granitemoehybrid, granite_swa, granitemoe_swa, mimo_v2_flash, nemotron_h, exaone4, mamba2, cwm.
- Too new or too mixed: pocket_tts (15) and gradium_phonon_tts (9) only started failing on 10-21, with mixed causes (missing Hub repos, torchcodec, ONNX, config validation). Worth another look in a couple of weeks.
Method
- Failing tests:
GET https://transformers-ci.huggingface.cool/project-health/rows?days=30, the data behind the dashboard's Fix failures tab (PromQLmax by (test_nodeid,test_job,run_id,trace_id,exception_type)(last_over_time(pytest_test_last_failure_info{pr="main"}[30d]))). Four-week figures sum the last 28 daily buckets: 1,401 failing rows, 270 model directories, 244 not covered. - Failure messages: Tempo searches through the Grafana datasource proxy, one per model and day of last failure (Tempo caps a query at 26 hours), bucketed per error signature.
- Chronic or new: the same per-model counts shifted 30 and 60 days back.
- Coverage: the live
prototype-fleet.lock(33 contracts) andfleet-lock.json(6) inhuggingface-mic/transformers-fleet. Checkpoints: Transformers test files plus Hub metadata for size, gating and remote code.
Caveats
- "MIC catches it?" is a keyword heuristic on test names and messages (offload, parallel, FA2, compile, export, quantization, OOM, Hub and network count as no; generation-text diffs as partly), checked by hand for the top entries.
- The data can't tell a numeric mismatch that is a regression from stale test expectations.
- Each test shows only its latest failure message, so a test whose failure mode changed within the window shows the newest one.