Introduction
Mixture-of-Experts (MoE) architectures promise efficient scaling by activating only a subset of parameters per input. The core assumption is that the learned router selects experts that are best suited for the current input. But does it?
We compare natural routing probabilities in Phi-mini-MoE (reported as a 16-expert, 32-layer model) with scores obtained by forcing one expert at a time. The measured correlations are small in the reported versions. This probes alignment with an intervention-based score, rather than directly measuring whether the router optimizes the model’s full task objective.
Method
For each of 32 layers, 16 experts, and 4 domains (Math, Science, General Knowledge, Reasoning), we:
- Force-route all tokens through a single expert at a single layer
- Measure log-probability delta vs. uniform routing baseline
- Compare forced performance with the router’s natural activation patterns
Statistical framework
- Pairwise comparisons: Wilcoxon signed-rank test for non-parametric paired differences
- Multiple comparison correction: Benjamini-Hochberg step-up FDR at α = 0.05
- Router calibration: Spearman rank correlation with Fisher z-transform
Findings
Finding 1: Significant specialization in 207 of 896 tested combinations
207 out of 896 expert-layer-domain combinations (23.1%) show statistically significant specialization after BH-FDR correction. Best experts differ across domains:
- Math: Expert 7 at Layer 30
- Science: Expert 1 at Layer 3
- General Knowledge: Expert 11 at Layer 4
- Reasoning: Expert 7 at Layer 30
Finding 2: Small correlations with forced-expert scores
Fisher z-averaged Spearman correlation between routing probability and expert quality:
| Domain | Spearman ρ | Interpretation |
|---|---|---|
| Math | +0.043 | Effectively zero |
| Science | −0.027 | Effectively zero |
| General Knowledge | −0.146 | Weakly negative |
| Reasoning | +0.063 | Effectively zero |
| Average | −0.017 | Small observed correlation |
These point estimates show weak alignment with forced-expert scores in v3.2. A small or negative rank correlation does not by itself prove that the router avoids the best expert or that its joint decisions harm end-to-end performance.
Finding 3: One expert is the most activated in all four domains
Expert E2 is the most-activated expert across all four domains (~20% above uniform activation of 0.0625), despite never being the best expert for any domain tested. This pattern motivates checking load balancing and expert interactions; it does not establish the cause of the activation preference.
Finding 4: Expert impact varies by tested layer
A two-regime model emerges:
- Early layers (L3–L5) and near-output layer (L30): Highest expert impact (avg best delta +0.647 to +0.779)
- Middle layers (L8, L13): Minimal to no effect
Finding 5: Step-up versus step-down correction changes which tests pass
The incorrect step-down implementation yields 0/896 surviving tests (no specialization found), while the correct step-up yields 207/896. This correction changed which tests passed the reported threshold. It is a methodological correction to this analysis, not evidence of a universal router failure.
v4.0 update (n=50)
The existing web report supplies the following update from n=10 to n=50. Its run-level artifacts and relation to the 5,120 count remain unresolved:
| Metric | v3.2 (n=10) | v4.0 (n=50) |
|---|---|---|
| BH FDR significant | 207/896 (23%) | 482/896 (54%) |
| Mean Spearman ρ | −0.017 | +0.069 |
| Math ρ | +0.043 | +0.098 |
| Science ρ | −0.027 | −0.005 |
| General ρ | −0.146 | +0.051 |
| Reasoning ρ | +0.063 | +0.132 |
The v4.0 table reports more significant combinations and a small positive mean correlation. Increased sample size can change statistical power; these counts alone do not establish a stronger causal account of routing behavior.
Discussion and interpretation limits
MoE training optimizes the router and experts jointly. Load balancing and interactions among selected experts are possible explanations for disagreement with isolated-expert rankings. This study does not isolate those explanations. Forcing all tokens through one expert changes the computation, so its score is a proxy rather than the same objective as normal multi-expert routing.
Proposed follow-up experiments
- Semantic routing replacement: Replace learned routers with domain-aware selection
- Cross-model expert grafting: Transfer specialized experts between models (connects to the Blades framework)
- Architecture redesign: Rethink training objectives to align routing with expert quality
Conclusion
The reported Phi-mini-MoE experiments find small correlations between natural routing and forced-expert scores. Reconciling the run counts, selected comparisons and v4.0 provenance is necessary before stronger conclusions. Testing replacement routers on held-out end-to-end tasks would determine whether this diagnostic translates into useful performance gains. No such gain is established by the tables here.