Introduction
The demand for AI systems with diverse capabilities continuously grows, yet training new models from scratch for each task is computationally prohibitive. Current approaches to capability enhancement each have significant limitations:
- Fine-tuning: Risk of catastrophic forgetting, requires substantial computational budget
- Model merging: Parameter averaging approaches often degrade performance (Task Arithmetic, DARE)
- Ensembles: Multiple forward passes increase inference cost and latency
- Adapters (LoRA, Prefix Tuning): Additional parameters and potential incompatibilities between modules
We propose Blades, a framework for injecting source-model hidden states into a target model during inference. A learned gate selects features for transfer. This mechanism requires source hidden states as well as the target computation; the report does not establish a single-model inference cost.
The original paper describes the approach as requiring no additional training while also specifying a learned gate. It does not give a reproducible account of which parameters were frozen, fitted or calibrated in each experiment, or whether source states were cached. We therefore do not treat zero training or zero added inference cost as established results.
The term “Blades” draws an analogy from mechanical engineering: just as a mechanic swaps engine components to enhance performance, we can inject computational modules (blades) between model layers to dynamically enhance capabilities at runtime.
The Blades framework
Blades consists of three components:
- Source Model: A specialized model containing the capability to transfer (e.g., strong reasoning abilities)
- Target Model: A base model that lacks or performs poorly on the capability
- Injection Mechanism: A gated aggregation layer that inserts source hidden states into target computation
Injection mechanism
During inference on the target model, at selected layer ℓ:
h_target(ℓ) = h_target(ℓ) + α · g(w) ⊙ h_source(ℓ)Where:
h_target(ℓ): hidden state from target model at layer ℓh_source(ℓ): hidden state from source model at the same layerg(w): learned gating function (e.g., sigmoid gate) with parameters wα: scalar weight parameter (range 0.1–0.3)⊙: element-wise multiplication
Injection depth hypothesis
The three tested layers suggest layer 28 as a candidate injection point for the reported 32-layer setting. Here, N − 4 equals 87.5% of depth; for other layer counts the fraction is (N − 4)/N. This is a heuristic from a limited sweep, not an established optimum across architectures.
Hypothesis: Later injection may retain useful high-level features while reducing interference with earlier processing. The layer comparison below does not by itself establish distinct causal roles for each layer range.
Experiments
Phase 1: Transfer across four model pairs
We tested hidden state injection across four model pairs to identify conditions for success:
| Exp | Source → Target | Dim Change | Result | Outcome |
|---|---|---|---|---|
| T01 | CLIP → GPT-2 | 512 → 768 (+50%) | No effect | ✗ |
| T02 | CLIP → Gemma-270M | 768 → 640 (−17%) | No effect | ✗ |
| T03 | MediPhi → Gemma-270M | 3072 → 640 (−79%) | Degradation | ✗ |
| T04 | Phi-4 reasoning → MediPhi | 3072 → 3072 (0%) | +14.2 percentage points | ✓ |
Only the same-dimension, same-family transfer (T04) improved performance in this set. Cross-modal (T01, T02) transfers had no reported effect, and T03 degraded. These observations do not isolate dimension matching from model family, modality or other differences.
The original PDF, section 4.1, reports the following results on the same direct medical task:
| Configuration | Reported accuracy |
|---|---|
| MediPhi target | 55.4% |
| Phi reasoning source alone | 62.5% |
| Combined system | 69.6% |
The increase over the target is 14.2 percentage points. The PDF calls the source Phi-4-mini-reasoning in the setup and shortens it elsewhere; an exact checkpoint identifier is not supplied here. Its limitations section states these are direct task evaluations, not standardized MedQA or MMLU results. Sample size, split and uncertainty are not reported in this article.
Phase 2: Comparison of three injection layers
| Layer | Position (%) | Accuracy | Original delta (unreconciled) | Original interpretation |
|---|---|---|---|---|
| 24 | 75% | 48.1% | −7.3% | Degradation (early interference) |
| 28 | 87.5% | 67.8% | −1.6% | Near-baseline, stable |
| 30 | 93.75% | 60.5% | −4.9% | Output preparation interference |
Layer 28 has the highest listed accuracy among these three settings. The inconsistent baselines prevent a reliable interpretation of the original stability claim; a full layer sweep was not reported.
Phase 3: Reported multi-blade synergy scores
| Blades | Target | Synergy Score | Domain |
|---|---|---|---|
| medical + medical_pubmed | MediPhi | +27.8% | Same |
| medical + medical_pubmed | Clinical | +22.2% | Same |
| medical_clinical + medical_pubmed | MediPhi | +16.7% | Same |
| reasoning + medical | MediPhi | −27.8% | Cross |
The reported scores favor same-domain combinations in this limited medical-domain experiment. The score definition and denominator are not supplied here, so these percentages are retained as original synergy scores, not relabeled as accuracy gains.
Observations and open hypotheses
- Injection depth: Layer N − 4 is a candidate for the tested 32-layer setting.
- Compatibility: The successful pair shared hidden dimensions and model family; necessity of either condition is unproven.
- Capability gap: A relationship between source-target capability differences and transfer benefit remains a hypothesis.
- Gating: The original report gives +8.9% over identity injection; the metric denominator and fitting procedure need documentation.
- Domain combinations: Original synergy scores include +27.8% for a same-domain pair and −27.8% for a cross-domain pair; generality is untested.
- Router control: The report gives 1.67× domain selectivity; this is not itself a downstream accuracy result.
- FFN projection: Projection is a proposed path for dimension bridging, with transfer validation incomplete.
Connection to the Model Garage toolkit
The original project links Model Garage, a toolkit intended for extracting, composing and managing model components. Its stated capabilities include:
- Hidden State Extraction: Efficient extraction at selected layers
- Gating Mechanisms: Pre-implemented learned gating (sigmoid, linear, softmax)
- Injection Automation: Automated injection at specified layers with configurable parameters
- Validation Pipelines: Benchmarking against standard tasks (MMLU, MedQA, etc.)
Conclusion
The reported Phi-to-MediPhi experiment motivates further study of hidden state transfer: combined accuracy exceeded both individual-model scores on one direct task. The failed transfers and limited layer sweep are equally important boundaries. Establishing reproducible gains requires exact checkpoints, task data, gate-fitting details, consistent baselines and measured inference cost.
Version and reproduction notes
This web revision corrects claim scope and percentage-point units without rerunning experiments. The linked PDF remains the original preprint version. No run-level dataset or exact code revision is identified in this record; artifact availability alone does not establish reproducibility.