Introduction
Dense transformer feed-forward layers compute many neuron activations per token. This report asks whether domain-dependent activation profiles can identify computations worth skipping, and how those profiles vary across the tested models. Preserving task quality is a separate question requiring downstream evaluation.
The reported profiles suggest opportunities for selective computation. Model family, architecture, size and profiling inputs vary together, so this comparison cannot isolate a scaling law.
Method
Neuron activation profiling
We profile neuron activations across diverse inputs to measure domain-specific activation strengths. For each model layer, we compute mean absolute activation per neuron across representative inputs.
A neuron is specialized if its activation for one domain far exceeds its average activation — the same neuron may be critical for one domain and negligible for another.
Negative selection strategy
For domain d in layer ℓ, the proposed rule marks a neuron as a candidate for masking when its profiled activation is below a percentile threshold. Low activation in a profile does not guarantee that the neuron is unnecessary on a new input. The threshold therefore needs validation on held-out task data.
Findings
Finding 1: Specialization in the reported scale comparison
| Model | Params | Avg Specialized | Theoretical compute ratio |
|---|---|---|---|
| phi-2 | 2.7B | 30.9% | 1.45x |
| Phi-3.5-mini | 3.8B | 30.2% | 1.43x |
| K2-Think | 32B | 68.2% | 3.14x |
The compute ratios are estimates associated with skipping operations. The report does not show how they account for kernel efficiency, memory traffic, profiling and routing overhead, or unchanged model operations. They are not measured wall-clock speedups.
Finding 2: Layer-wise activation profiles
The original 32B profile lists the following values and proposed interpretations. The PDF describes 61 layers in its setup but includes L62 here; the checkpoint and layer-index convention need reconciliation. These values are preserved from the report:
| Depth | Layer | Specialization | Proposed role |
|---|---|---|---|
| 3% | L2 | 14.4% | Syntax recognition |
| 28% | L18 | 79.6% | Domain knowledge |
| 53% | L34 | 72.9% | Domain computation |
| 88% | L56 | 86.6% | Peak specialization |
| 97% | L62 | 76.9% | Output preparation |
These profiles motivate testing different masking budgets per layer. They do not establish that 70–90% masking preserves task quality, or that the proposed semantic roles are causal findings.
Finding 3: Differences between the tested architectures
Domain specialization varies significantly across architectures:
| Architecture | Model | Avg Specialized |
|---|---|---|
| Gemma | gemma-2b | 51.8% |
| Qwen | Qwen1.5-1.8B | 47.6% |
| Llama | TinyLlama-1.1B | 7.4% |
The tested Gemma and Qwen examples have higher reported specialization than TinyLlama. Different model sizes, training and architectures prevent extending this ranking to entire families.
Finding 4: Candidate masking in layer 0
Analysis of Layer 0 across domains (Phi-3.5-mini):
| Domain | Candidate neurons to skip | Theoretical layer compute ratio |
|---|---|---|
| Math | 90.4% | 10.47x |
| Code | 72.9% | 3.70x |
| Factual | 52.1% | 2.08x |
| Language | 48.3% | 1.88x |
These percentages identify candidates from Layer 0 activation profiles. They do not establish safe masking rates on unseen queries, and the layer-only ratios are not whole-model speedups.
Connection to expert-level sparsity
The original report discusses Kimi K2.5 as an expert-level sparsity example, listing 1T parameters and 8 active experts out of 384 (2.1%). This architectural comparison is separate from the dense-neuron measurements above. It does not validate the neuron-masking method or its task quality at 1T scale.
Reported output similarity
At 5–15% sparsity with domain-aware selection, the report gives cosine similarity of 0.999+ between dense and masked outputs. This indicates close vector directions in that measurement. It is not exact equality, does not bound output magnitude differences, and does not establish unchanged predictions or downstream task accuracy. The article does not identify the evaluated tensor, sample count or tolerance needed to reproduce this comparison.
Connection to MoE
MoE and sparse pathways are complementary:
- MoE: Coarse-grained sparsity (choose 8 of 384 experts)
- Sparse Pathways: Fine-grained sparsity (choose neurons within layers)
- Combined: Experts + intra-expert neuron sparsity = even more efficient inference
Conclusion
The reported activation profiles motivate domain-aware neuron masking experiments. Establishing a scaling relationship requires a defined analysis population and controls for architecture; establishing practical savings requires measured runtime and held-out task accuracy. The linked PDF remains the original preprint version, including stronger claims corrected in this web revision. No new measurements were added.