Skip to main content

Research paper

Context ExtensionMemory SystemsApproximate Nearest NeighborHierarchical Indexing

Hierarchical Attention Tree: Extending LLM Context Through Structural Memory

HAT uses session and document structure to retrieve stored vectors. On a synthetic 1,000-point benchmark it reported higher recall and faster construction than the tested HNSW configuration, while HNSW queries were faster.

Andrew Young Preprint · not peer reviewed 6 min read

Updated September 22, 2026 · Editorial correction dated 2026-09-22; no new experiment or independent reproduction. The linked PDF is the original version and does not include these web corrections.

Paper Code

Scope Synthetic hierarchical retrieval: a 1,000-point, 128-dimensional comparison; a 500-5,000-point series; and a separate 60K-token retrieval example. Limitations and artifacts

Abstract

We present the Hierarchical Attention Tree (HAT), an experimental vector index that uses a known session-document-chunk hierarchy. On the reported synthetic benchmark of 1,000 points in 128 dimensions, HAT reached 100% recall@10 and built in 30 ms; the tested HNSW configuration reached 70.6% and built in 2.1 s. HNSW had lower query latency: 0.49 ms versus 1.42 ms. A separate 60K-token synthetic retrieval experiment reports 100% retrieval recall and 3.1 ms latency; it does not establish downstream LLM answer accuracy. Mapping these experiments to larger-scale summary graphics remains unresolved, and these results do not establish general-purpose retrieval superiority.

On this page
  1. The context window problem
  2. The HAT approach
  3. Comparing index structures
  4. The hippocampus analogy
  5. Algorithm
  6. Beam search query
  7. Experimental results
  8. Experiment A: HAT vs HNSW on synthetic hierarchical data
  9. Experiment B: Synthetic scale series
  10. Experiment C: Retrieval for a 60K-token corpus
  11. Reconciling the experiments with the original graphics
  12. Reported routing variants
  13. Consolidation phases
  14. Implementation
  15. Contributions
  16. Conclusion
  17. Get started

The context window problem

Large language models have a fundamental limitation: finite context windows. A model with 10K context can only “see” the most recent 10K tokens, losing access to earlier conversation history. Current solutions include:

  • Longer context models: Expensive to train and run (128K+ context)
  • Summarization: Lossy compression that discards detail
  • Retrieval-augmented generation: Retrieves selected context for a query; embedding and caching choices determine which computations repeat

The HAT approach

HAT takes a different approach: exploit known structure.

AI conversation data can be organized into a hierarchy that retrieval methods may exploit:

  1. Session conversation boundary
  2. Document topic or turn
  3. Chunk individual message
The structure HAT assumes is already known for conversation data.

HAT uses centroid-based beam search through that structure. Recall depends on the data and search settings; the reported 100% values below are observations on synthetic hierarchical data, not a guarantee.

Research question. Can known conversation structure make retrieval beyond a model’s context window useful and efficient?

The 60K-token example reports retrieval over a corpus about six times the stated 10K native context allocation. It does not increase the model’s native window. The HAT repository clarifies that the reported recall is synthetic-data retrieval recall, with downstream QA evaluation still unestablished in this record.

Comparing index structures

AspectHNSWHAT
OrganizationProximity graph constructed from vectorsSupplied session-document-chunk hierarchy
RoutingGraph traversalCentroid-based beam search
Additional structureGraph linksContainer links, centroids and timestamps
Main comparison limitRecall depends on graph and search settingsRecall depends on hierarchy quality and beam settings

The difference in assumptions:

HNSW: “Here are points, I’ll learn to navigate them.”

HAT: “I know these are containers. I’ll use what I know.”

These are different structural assumptions, not a ranking of all retrieval workloads.

The hippocampus analogy

The following biological analogy motivates the design; it is not a validated model of human memory:

Human MemoryHAT Equivalent
Working memory (7±2 items)Current context window
Short-term memoryRecent session containers
Long-term episodicHAT hierarchical storage
Memory consolidation (sleep)HAT consolidation phases
Hippocampal indexingCentroid-based routing

Algorithm

HAT organizes points into a tree with four levels:

  1. Global root
  2. Session conversation boundaries
  3. Document topic groupings
  4. Chunk leaf nodes with points
The four levels of the HAT tree. Each non-leaf level keeps a centroid of what sits below it.

Each non-leaf container maintains:

  • Centroid: Mean of descendant embeddings
  • Children: Pointers to child containers
  • Timestamp: For temporal locality

Beam search query

def hat_query(query, k, beam_width):
    beam = {root}

    for level in [Session, Document, Chunk]:
        candidates = []
        for container in beam:
            for child in container.children:
                score = cosine(query, child.centroid)
                candidates.append((child, score))
        beam = top_b(candidates)

    return top_k(beam)

Traversal cost: O(b × d × c) container comparisons for beam width b, tree depth d and examined branching factor c, excluding the vector-comparison cost. This becomes logarithmic in n only when depth grows as log n and beam width and branching remain bounded; it is not a general recall or runtime guarantee.

Experimental results

Experiment A: HAT vs HNSW on synthetic hierarchical data

Setup: 1000 points = 20 sessions × 5 documents × 10 chunks, 128 dimensions.

MetricHATHNSWΔ
Recall@1100.0%76.0%+24.0 percentage points
Recall@5100.0%72.0%+28.0 percentage points
Recall@10100.0%70.6%+29.4 percentage points
Build Time30ms2.1s70× faster
Query Latency1.42ms0.49msHNSW 3× faster

Recall in Experiment A

1,000 synthetic points with a known session-document-chunk hierarchy

Recall@1
100.0%
76.0%
Recall@5
100.0%
72.0%
Recall@10
100.0%
70.6%
Figure 1. Reported recall on synthetic data whose hierarchy is supplied to HAT. HNSW tuning, the query set and run count are not recorded, and this is not a comparison at matched settings.

Index build time in Experiment A

milliseconds, lower is faster

HAT
30 ms
HNSW
2,100 ms
Figure 2. HAT built its index about 70 times faster in this experiment, from the rounded table values.

Query latency in Experiment A

milliseconds, lower is faster

HAT
1.42 ms
HNSW
0.49 ms
Figure 3. The trade-off runs the other way for queries: HNSW answered about three times faster.

Tradeoff: In this experiment HAT had higher reported recall and faster construction; HNSW had lower query latency. The appropriate tradeoff depends on recall requirements and baseline tuning. The known hierarchy favors HAT, and the table does not establish how the methods compare on unstructured data or at matched recall.

Recall@k measures retrieval of the relevant top-k items under the benchmark’s ground truth. The exact query set, ground-truth construction, HNSW tuning, hardware, run count and uncertainty are not recorded in this article. The 70× construction ratio is from the rounded table values.

Experiment B: Synthetic scale series

PointsHAT BuildHNSW BuildHAT R@10HNSW R@10
50016ms1.0s100%55%
100025ms2.0s100%44.5%
200050ms4.3s100%67.5%
5000127ms11.9s100%55%

HAT reports 100% recall in these four synthetic settings. The 1,000-point row differs from Experiment A in both timing and HNSW recall. It is retained as a separate reported series; the source record does not identify the settings or seeds that explain that difference.

Experiment C: Retrieval for a 60K-token corpus

Reported setup: 2,000 synthetic messages (~60K tokens), sentence-transformers embeddings, and a proposed gemma3:1b integration. The recorded recall concerns retrieved messages, not a scored LLM answer benchmark.

MetricValue
Total tokens60,000
Native context sees10,000 (16.7%)
Reported retrieval recall100%
Retrieval latency3.1ms
Memory usage3.3 MB

Retrieving a matching message is a prerequisite for useful question answering, but does not guarantee a correct answer. The repository explicitly separates these retrieval figures from downstream LLM task accuracy.

Reconciling the experiments with the original graphics

The original overview graphics mention a 500K-point, 1,536-dimensional study and 15M tokens while reusing values also shown in a 1,000-point, 128-dimensional build comparison. A run-level mapping connecting those scales is not supplied here. Do not apply the small-benchmark build times or recall to the larger-scale claim.

The original build graphic gives 31 ms versus 2,094 ms (about 67.5×). Its phrase “over two orders of magnitude” is incorrect: that would require more than 100×. The rounded 30 ms/2.1 s journal table yields 70×. These are separately presented values; this revision does not assert they are the same run.

Reported routing variants

Subspace Routing (Grassmann geometry):

  • Recall: −8.7% vs centroids
  • Latency: +11.8%

Learnable Routing Weights:

  • Recall: −2% to +4%
  • Latency: ~0%

Interpretation: The reported routing variants did not consistently improve on centroids in these settings. The original recall changes above do not specify relative-percent versus percentage-point units; raw comparisons are needed to settle that distinction. These results do not establish that centroid routing is sufficient for every hierarchy or dataset.

Consolidation phases

Inspired by sleep-staged memory consolidation:

PhaseOperationsTime
Light (α)Recompute centroids9ms/1K
Medium (β)+ Merge/split containers9ms/1K
Deep (δ)+ Prune empty, optimize layout9ms/1K
Full (θ)Complete rebuild10ms/1K

Implementation

HAT is implemented in Rust with Python bindings via PyO3:

from arms_hat import HatIndex

# Create index
index = HatIndex.cosine(1536)

# Add messages with session/document structure
index.new_session()
id = index.add(embedding)

# Query
results = index.near(query_embedding, k=10)

# Persistence
index.save("memory.hat")

Contributions

  1. A centroid-based index that uses supplied session-document-chunk structure
  2. Synthetic retrieval comparisons with explicit construction/query tradeoffs
  3. A small-scale retrieval pipeline for a corpus beyond a stated context allocation
  4. Reported negative results for more elaborate routing variants

Conclusion

HAT explores whether supplied conversation structure can help retrieval. The synthetic results motivate testing on realistic workloads, matched recall/latency settings and downstream tasks. This web revision adds scope and accounting corrections without rerunning experiments; the linked PDF is the original paper version.

Get started

HAT is available as the arms-hat crate:

Part of the ARMS (Attention Reasoning Memory Store) project at Automate Capture Research.

Paper record

Scope

Synthetic hierarchical retrieval: a 1,000-point, 128-dimensional comparison; a 500-5,000-point series; and a separate 60K-token retrieval example.

Author-reported results

Limitation

Known hierarchy favors HAT. HNSW queries were faster, downstream QA accuracy was not established, and larger-scale figure claims are not reconciled here.

Artifacts

HAT repository and original paper PDF linked. The article separates reported experiments; exact run revisions, baseline tuning and uncertainty are not recorded here.

Citation

Use this record when citing the current public version.

Andrew Young (2026). Hierarchical Attention Tree: Extending LLM Context Through Structural Memory. Automate Capture Research. https://research.automate-capture.co/journals/hat-hierarchical-attention-tree/