In LSH-based attention, the probability of two tokens i and j attending to each other is proportional to their hash collision probability. For queries q_i and keys k_j, the LSH function h satisfies:

where d_k is the key dimension. This allows approximating softmax attention with sub-quadratic cost.

Efficient Implementations

Practical optimizations include:

• Memory-efficient kernels: FlashAttention exploits GPU memory hierarchy to reduce I/O operations.
• Block-sparse GPU operations: Frameworks like Triton enable efficient sparse matrix multiplication.
• Gradient checkpointing: Recomputes attention activations during backward passes to save memory.

For a sequence of length 16K, dense attention requires ~200GB of memory, while block-sparse attention (b=64) reduces this to ~3GB, enabling training on consumer GPUs.

Hierarchical attention mechanisms address the quadratic complexity of standard self-attention by decomposing long sequences into multi-level representations. At each level l, the input sequence is partitioned into non-overlapping segments of length k_l, where higher levels operate on increasingly coarse-grained representations. The attention scores between segments at level l are computed as:

where h_i^(l) represents the aggregated representation of segment i at level l, typically computed via mean pooling or a learned compression function. The hierarchical attention weights are then used to route information between segments, reducing the computational complexity from O(n²) to O(n log n) for a sequence of length n.

Memory-Augmented Hierarchical Attention

Persistent memory components further enhance hierarchical attention by maintaining:

• Segment-level memory: Fixed-size vectors storing compressed representations of historical segments
• Topic memory: Dynamically allocated slots capturing cross-segment thematic patterns
• Latent variables: Stochastic memory addressing for probabilistic retrieval

The memory update rule for segment i at time t follows:

where γ controls the memory retention rate and 𝒩(i) denotes the neighborhood of segments attending to i.

Implementation Considerations

Practical implementations must handle:

• Gradient flow through multiple hierarchy levels
• Synchronization between different attention granularities
• Memory compression techniques to prevent capacity overload

The segment aggregation function often employs a convolutional bottleneck:

where ∗ denotes 1D convolution and k is the pooling stride between levels. This architecture enables efficient processing of documents exceeding 100k tokens while maintaining coherent attention patterns across the full context.

Recurrent Neural Networks (RNNs) and Transformers each have distinct advantages in handling sequential data. RNNs, particularly Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) variants, excel at processing sequences step-by-step with inherent memory mechanisms. Transformers, however, leverage self-attention to capture long-range dependencies efficiently but suffer from quadratic memory complexity with sequence length. Hybrid architectures aim to combine these strengths, mitigating their individual weaknesses.

Architectural Fusion Strategies

Three primary approaches dominate hybrid model design:

• Recurrent Attention: Replaces standard RNN attention mechanisms with multi-head self-attention, allowing the model to focus on relevant historical states while maintaining sequential processing.
• Transformer with Recurrent Memory: Augments Transformer layers with recurrent connections, either between layers or within the attention mechanism itself, to enhance temporal modeling.
• Hierarchical Hybridization: Uses RNNs for local sequence modeling and Transformers for global context, often through a multi-scale architecture.

Mathematical Formulation

The recurrent attention mechanism can be formalized by modifying the standard attention computation. For a hidden state h_t at time step t, the attention over previous states incorporates both content-based and recurrent positional information:

where f(h_t, h_i) is a learned function that encodes the temporal distance between steps t and i, typically implemented as:

Here, φ is a positional embedding function and W_r, v are learnable parameters.

Memory-Augmented Transformers

The Transformer-XL architecture demonstrates an effective hybridization by introducing segment-level recurrence. For a segment of length L, the model caches hidden states from the previous segment, allowing information to propagate beyond the fixed context window. The extended context is computed as:

where SG denotes stop-gradient, ∘ is concatenation, and τ indexes the segment. The attention mechanism then operates over this extended context.

Practical Implementations

Several production systems leverage hybrid architectures:

• Google's Meena chatbot uses a Transformer with lightweight recurrent connections to maintain conversation state.
• DeepMind's AlphaFold 2 employs axial attention with LSTM components for protein structure prediction.
• OpenAI's GPT-3 variants experiment with memory tokens that exhibit recurrent update patterns.

Performance Trade-offs

Benchmarks on the PG-19 long-document dataset reveal characteristic trade-offs:

The optimal architecture depends on the specific constraints—pure Transformers maintain higher throughput, while hybrids achieve better perplexity at the cost of increased architectural complexity.

Transformer-based language models face inherent limitations in processing long sequences due to the quadratic computational complexity of self-attention. For sequences exceeding the model's maximum context window, two primary strategies emerge: chunking and sliding window attention. These approaches enable efficient processing while mitigating memory and computational bottlenecks.

Chunking Strategy

Chunking decomposes a long sequence into fixed-length, non-overlapping segments processed independently. Given an input sequence X of length N and a chunk size C, the sequence is divided into K = ⌈N/C⌉ chunks:

Each chunk X_i is processed separately by the transformer, and outputs are aggregated. While computationally efficient, this approach risks losing cross-chunk dependencies. To mitigate this, hierarchical aggregation or memory-augmented architectures (e.g., Transformer-XL) can be employed.

Sliding Window Attention

Sliding window attention restricts the self-attention mechanism to a local neighborhood around each token, reducing the effective context size. For a window size W, token x_i attends only to tokens in the range [i-W/2, i+W/2]. The attention matrix A becomes sparse:

This reduces complexity from O(N²) to O(NW). Overlapping windows or dilated attention patterns can improve information flow across distant tokens.

Hybrid Approaches

Recent work combines chunking with sliding windows for improved efficiency. The Longformer architecture, for instance, uses a mix of local sliding windows and task-specific global attention. Another variant, BlockBERT, processes chunks in parallel while maintaining cross-block attention through a sparse pattern:

where M is a block-sparse mask allowing intra-chunk and limited inter-chunk attention.

Practical Implementation

Implementing these strategies requires careful memory management. For chunking, gradient checkpointing can reduce memory usage during training. Sliding window attention benefits from optimized sparse matrix operations in frameworks like PyTorch or JAX. Below is a PyTorch snippet for sliding window attention:

import torch
import torch.nn.functional as F

def sliding_window_attention(Q, K, V, window_size):
    batch_size, num_heads, seq_len, d_k = Q.shape
    mask = torch.ones(seq_len, seq_len, dtype=torch.bool, device=Q.device)
    for i in range(seq_len):
        left = max(0, i - window_size // 2)
        right = min(seq_len, i + window_size // 2 + 1)
        mask[i, left:right] = False
    attn_weights = torch.matmul(Q, K.transpose(-2, -1)) / (d_k ** 0.5)
    attn_weights = attn_weights.masked_fill(mask, float('-inf'))
    attn_weights = F.softmax(attn_weights, dim=-1)
    return torch.matmul(attn_weights, V)

Key trade-offs include balancing window size (larger windows improve accuracy but increase computation) and chunk overlap (reduces information loss at boundaries). Empirical studies suggest optimal performance when the window size matches the model's receptive field for the task.

Standard transformer models rely on sinusoidal positional encodings to inject sequence order information, defined for position pos and dimension i as:

While effective for moderate sequence lengths, this fixed encoding scheme exhibits two critical limitations for long-context processing:

• High-frequency components decay too rapidly - The wavelength of higher dimensions (large i) becomes shorter than the sequence length, causing positional signals to degenerate into near-random noise.
• Lack of extrapolation capability - The sinusoidal basis provides no meaningful way to represent positions beyond those seen during training.

Rotary Positional Embeddings (RoPE)

RoPE reformulates positional encoding as a rotation operation in complex space. For a query or key vector x at position m, the transformation applies:

where θ_j = 10000^-2j/d. This formulation preserves relative position information through the product of rotation matrices:

demonstrating that attention scores depend only on the relative position n-m. RoPE's rotational structure enables stable extrapolation beyond trained sequence lengths while maintaining theoretical relative position awareness.

XPos: Generalized Relative Positional Encoding

XPos introduces a learnable decay factor γ ∈ (0,1] to explicitly control how positional information attenuates with distance:

The γ parameter is trained jointly with the model, allowing adaptive adjustment of position sensitivity based on the task's locality requirements. For language modeling, typical learned values fall in the range γ ∈ [0.95, 0.99], indicating moderately strong position decay.

Randomized Positional Encoding

To address the extrapolation problem, randomized methods like ALiBi (Attention with Linear Biases) replace additive positional encodings with a decaying attention bias:

where mask is an upper triangular matrix with linearly decreasing negative values (e.g., -1, -2, -3,...). The slope m is typically set to 1/2^k for head k, creating a geometric progression of attention decay rates across heads. This approach requires no positional embeddings at all, instead relying on the inductive bias of the attention mechanism itself to learn position-aware patterns.

Implementation Considerations

When implementing these enhanced positional encoding schemes, several practical factors must be considered:

• Memory overhead - RoPE and XPos require storing rotation matrices, increasing memory usage by ~15% compared to standard sinusoidal encodings.
• Computational cost - The matrix multiplications in RoPE add approximately 10-20% to the attention computation time.
• Training stability - Randomized methods like ALiBi often require lower learning rates (typically 0.5-1.0x baseline) during initial training phases.

Empirical studies on the PG19 long-document benchmark show relative performance improvements for various encoding schemes compared to standard positional encodings in a 1B parameter model:

Transformer-based language models suffer from quadratic memory and computational complexity with respect to sequence length, making long-context processing prohibitively expensive. Dynamic context pruning and caching address this by selectively retaining only the most relevant tokens while discarding or compressing less important information. This approach enables efficient processing of sequences exceeding the model's nominal context window.

Attention Score-Based Pruning

The core mechanism relies on analyzing attention scores to identify and prune low-contribution tokens. For each attention head h at layer l, we compute the importance score I_i for token i as:

where A_ij^h represents the attention score from token i to j in head h, and H is the total number of attention heads. Tokens with scores below a dynamic threshold τ are candidates for pruning:

where μ and σ are the mean and standard deviation of importance scores in the current context window, and α is a tunable aggressiveness parameter.

Hierarchical Caching Strategies

Pruned tokens are not simply discarded but compressed into hierarchical cache representations. The caching mechanism operates at three levels:

• Local phrase-level caching: Short sequences (2-5 tokens) are compressed using learned phrase embeddings
• Segment-level caching: Longer coherent segments are stored as key-value pairs with reduced dimensionality
• Document-level caching: Global themes and entities are maintained in a persistent memory bank

The cache retrieval process uses a content-based addressing mechanism:

where q is the current query, k_i are cache keys, and β controls the sharpness of retrieval.

Dynamic Cache Management

Cache maintenance follows an importance-and-recency policy. Each cache entry e has a dynamic priority score:

where I_e is the historical importance, R_e is a recency factor, and λ balances the two components. The system continuously evicts low-priority items when the cache reaches capacity.

Implementation Considerations

Practical implementations must address several challenges:

• Gradient propagation: Straight-through estimators enable training through pruning decisions
• Hardware efficiency: Block-sparse attention patterns match modern accelerator architectures
• Consistency: Cached representations must remain coherent with the active context

Recent architectures like Memorizing Transformers and Compressive Transformers demonstrate these techniques can achieve 4-8× context length expansion with minimal accuracy degradation on tasks like book summarization and long-document QA.

Challenges in Long-Context Summarization

Traditional summarization techniques struggle with long documents due to the quadratic computational complexity of attention mechanisms in transformer-based models. For a document with N tokens, the memory requirement scales as O(N²), making processing of lengthy texts computationally prohibitive. Additionally, information density varies non-uniformly across long documents, with critical content often distributed sparsely.

Hierarchical Attention Mechanisms

Hierarchical approaches decompose the summarization task into multiple levels of abstraction. First, the document is segmented into chunks or paragraphs, each processed independently. Then, a second-level attention mechanism operates on the compressed representations of these segments. This reduces the effective sequence length from N to N/k, where k is the chunk size.

Two-Stage Attention Architecture

The first stage computes intra-segment attention:

The second stage performs inter-segment attention:

Memory-Efficient Attention Variants

Sparse attention patterns like Longformer's dilated sliding window attention or BigBird's block-sparse attention reduce the quadratic dependency. For a sequence length N and window size w, memory complexity becomes O(N×w) instead of O(N²).

Content Selection Strategies

Advanced methods combine extractive and abstractive approaches:

• Salience Scoring: Uses learned importance scores to select key sentences
• Graph-Based Methods: Constructs document graphs where nodes represent text units and edges represent semantic relationships
• Reinforcement Learning: Optimizes for summary quality metrics like ROUGE through policy gradient methods

Practical Implementation Considerations

When implementing long-context summarizers:

• Segment documents using semantic boundaries (paragraphs/sections) rather than fixed-length chunks
• Employ gradient checkpointing to reduce memory usage during training
• Use mixed-precision training to handle larger batch sizes
• Implement caching mechanisms for intermediate representations

Evaluation Metrics for Long Summaries

Beyond standard ROUGE scores, consider:

where S is the generated summary and r is the reference. Also evaluate:

• Factual consistency using question-answering metrics
• Information coverage through latent semantic analysis
• Compression ratio while maintaining information density

Case Study: Scientific Paper Summarization

For scientific papers (typically 5,000-15,000 tokens), a hybrid approach proves effective:

1. Extract key sentences from sections using section-aware attention
2. Generate abstractive summaries of methodology and results
3. Combine outputs with template-based structuring

Long-context question answering (QA) systems in large language models (LLMs) require specialized architectures and attention mechanisms to process and retrieve information from extended sequences efficiently. Traditional transformer-based models struggle with quadratic memory complexity in self-attention, making them impractical for contexts exceeding a few thousand tokens. Recent advances address this through sparse attention, memory-efficient mechanisms, and retrieval-augmented generation.

Architectural Modifications for Long-Context QA

Sparse attention mechanisms, such as those in Longformer or BigBird, reduce computational overhead by limiting the attention span to a fixed window or employing a combination of local and global attention patterns. For a sequence of length N, standard self-attention computes O(N²) interactions, while sparse attention reduces this to O(N) or O(N log N).

In long-context QA, the model must also handle hierarchical representations. Techniques like hierarchical chunking divide the input into segments, process them independently, and then aggregate results. For example, given a document split into k chunks {C₁, C₂, ..., Cₖ}, the model computes:

Retrieval-Augmented Generation (RAG)

Retrieval-augmented models, such as RAG or REALM, decouple memory storage from processing by using an external retrieval system to fetch relevant passages before generating answers. Given a query q, the system retrieves top-k documents D = {d₁, d₂, ..., dₖ} from a corpus and conditions the answer generation on both q and D:

This approach scales to arbitrarily long contexts by leveraging efficient nearest-neighbor search algorithms (e.g., FAISS) over dense vector embeddings.

Memory-Efficient Attention

Memory constraints in long-context QA are mitigated through techniques like gradient checkpointing, mixed-precision training, and flash attention. Flash attention, for instance, optimizes GPU memory bandwidth by fusing attention computations into a single kernel, reducing HBM accesses. The I/O complexity drops from O(N²) to O(N):

where M is the SRAM size. This enables training on sequences exceeding 100K tokens.

Evaluation Metrics for Long-Context QA

Standard QA metrics (e.g., F1, EM) are insufficient for long-context settings. Additional measures include:

• Positional bias: Whether the model favors earlier or later segments.
• Context utilization: The fraction of retrieved passages actually used in answering.
• Latency-throughput tradeoff: Inference speed vs. accuracy at scale.

Benchmarks like SCROLLS or LongEval provide standardized datasets for assessing these dimensions.

Case Study: GPT-4 with 32K Context

GPT-4’s extended context window employs a hybrid of sparse attention and dynamic retrieval. When processing a query, the model first identifies salient spans via a lightweight retrieval step, then applies full attention only to those regions. This balances accuracy and computational cost, achieving a 40% improvement in QA accuracy over fixed-window baselines on the NarrativeQA dataset.

Challenges in Long-Context Code Generation

Large Language Models (LLMs) exhibit degraded performance when generating or analyzing code beyond their typical context window, often leading to:

• Variable scope errors due to lost references in distant context
• API call inconsistencies when documentation exceeds context limits
• Function duplication from failure to recall existing implementations
• Attention collapse in transformer architectures for long sequences

Architectural Solutions

Modified transformer architectures address these limitations through:

Where M is a sparse attention mask enabling:

• Block-local attention for code structure preservation
• Strided patterns to maintain cross-file references
• Memory tokens that compress critical context

Practical Implementation

For Python code generation with extended context, consider this architecture:

class LongContextCodeGenerator(nn.Module):
    def __init__(self, base_model, max_ctx=32k):
        super().__init__()
        self.base = base_model
        self.compression = nn.Linear(
            base_model.config.hidden_size,
            base_model.config.hidden_size//4
        )
        self.memory_slots = nn.Parameter(
            torch.randn(16, base_model.config.hidden_size)
        )

    def forward(self, input_ids):
        # Segment input into chunks with overlap
        chunks = self._segment_with_overlap(input_ids)
        
        # Process each chunk with memory carryover
        outputs = []
        memory = self.memory_slots
        for chunk in chunks:
            out, memory = self._process_chunk(chunk, memory)
            outputs.append(out)
        
        return self._reconstruct(outputs)

Analysis Techniques

For code analysis across long contexts, employ:

• Hierarchical summarization to maintain architectural understanding
• Cross-file dependency graphs using graph attention networks
• Dynamic context retrieval based on static analysis patterns

Dependency Graph Construction

The adjacency matrix A for code dependencies follows:

Optimization Strategies

Key optimizations for production systems include:

• Selective token processing prioritizing code over comments
• Incremental parsing with syntax-tree checkpointing
• Hotspot caching for frequently accessed code segments

Evaluating how well large language models (LLMs) retain and utilize long-context information requires rigorous quantitative and qualitative metrics. Two primary dimensions must be assessed: context retention (the model's ability to recall and reference earlier content) and coherence (the logical consistency and flow of generated text).

Quantifying Context Retention

Context retention is typically measured using needle-in-a-haystack tests, where a critical piece of information (the "needle") is embedded within a long document (the "haystack"). The model must retrieve or utilize this information accurately when prompted. The retention score R can be formalized as:

where N is the number of test cases, f is the model's response function, x_i is the query, c_i is the context containing the needle, y_i is the correct answer, and 𝕀 is the indicator function.

More sophisticated variants include:

• Positional sensitivity tests: Measuring how retention varies with the needle's position in the context window.
• Multi-hop reasoning tests: Requiring the model to combine information from different parts of the long context.

Assessing Coherence

Coherence evaluation requires both automated metrics and human judgment. Key automated metrics include:

Computational Complexity of Attention Mechanisms

The standard self-attention mechanism in transformers exhibits quadratic complexity O(n²) with respect to sequence length n, due to the pairwise computation of attention scores. For long sequences, this becomes prohibitive in both time and memory. The memory consumption scales as:

where b is batch size, s is sequence length, and h is hidden dimension size. The factor of 4 accounts for key, query, value matrices and attention weights.

Key Metrics for Evaluation

When benchmarking long-context LLMs, researchers track several core metrics:

• Throughput (tokens/second): Measures processing speed during generation
• Memory footprint (GB): Peak GPU memory consumption
• Latency (ms/token): Time per generated token
• FLOPs utilization (%): Hardware efficiency

Efficient Attention Variants

Sparse attention methods reduce complexity by computing only a subset of attention scores. For local windowed attention with window size w, complexity becomes:

Memory-efficient attention implementations exploit the associative property of matrix multiplication to reduce intermediate memory allocation. The memory savings can be derived by recomputing attention scores during the backward pass rather than storing them:

Kernel Optimization Techniques

Modern implementations use fused CUDA kernels to minimize memory transfers. The speedup factor S from kernel fusion can be modeled as:

where t_mem represents memory transfer time and t_compute is raw computation time. Typical values range from 1.5-3× for attention operations.

Memory Bandwidth Considerations

The roofline model predicts maximum attainable performance based on operational intensity I:

For attention layers, operational intensity remains low (<0.1 FLOP/byte) due to large memory footprints, making memory bandwidth the limiting factor rather than compute throughput.

Practical Measurement Approaches

Profiling tools like NVIDIA Nsight Systems measure:

• Kernel execution timelines
• Memory copy operations
• GPU utilization metrics

Key measurements include memory bandwidth utilization (percentage of theoretical peak) and L2 cache hit rates, which significantly impact performance for long sequences.

Evaluating the performance of large language models (LLMs) on long-context tasks requires standardized benchmarks that measure both accuracy and computational efficiency. Key metrics include retrieval accuracy, memory retention, and inference speed as context length scales. Below, we analyze recent comparative studies across leading models.

Benchmarking Methodologies

Modern long-context evaluations employ synthetic and real-world datasets designed to stress-test models. Common approaches include:

• Needle-in-a-Haystack (NIAH): Tests retrieval accuracy by embedding a target fact (the "needle") in a long document (the "haystack"). Performance is measured via recall@k.
• Long-range dependency tasks: Evaluates memory retention through narrative QA, mathematical reasoning, or multi-hop question answering over extended contexts.
• Inference latency profiling: Measures throughput (tokens/second) and memory usage across context lengths from 4k to 128k tokens.

Model-Specific Performance

Recent studies reveal stark differences in architectures' ability to handle long contexts:

Transformer-Based Models

Standard transformers exhibit quadratic memory growth with sequence length. At 32k tokens:

• GPT-4 achieves 78% NIAH accuracy but suffers 4× latency increase versus 8k contexts.
• Llama 2-70B shows 62% accuracy with 16GB memory overhead.

Efficient Variants

Specialized architectures demonstrate improved scaling:

• FlashAttention-2: Reduces memory to O(N) through optimized attention computation, enabling 64k contexts on single GPUs.
• Hyena: Replaces attention with implicit convolutions, maintaining 91% accuracy at 128k tokens with subquadratic runtime.

Hardware Considerations

Performance varies significantly by hardware configuration:

Memory bandwidth emerges as the critical bottleneck, with HBM3 architectures showing 2.3× speedups over GDDR6 for 64k+ contexts.

• Reasoning Degradation in LLMs with Long Context Windows: New Benchmarks ... — GPT-4 has a context window of 128,000 tokens, while Gemini boasts a staggering 2 million. Although these numbers are exciting, the reality is somewhat different. You may have observed, as I have, that the quality of reasoning by LLMs tends to falter with lengthy inputs—a phenomenon that current evaluations fail to adequately capture. The prevalent benchmark for assessing LLMs' handling of ...
• PDF Lost in the Middle: How Language Models Use Long Contexts — long input contexts. In particular, we observe that performance is often highest when rele-vant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models. Our analysis provides a better ...
• LongLLMLingua: ACCELERATING AND ENHANCING LLM L CONTEXT ... - OpenReview — In long context scenarios, large language models (LLMs) face three main chal-lenges: higher computational/financial cost, longer latency, and inferior perfor-mance. Some studies reveal that the performance of LLMs depends on both the density and the position of the key information (question relevant) in the input prompt.
• natanaelwf/Reasoning-Degradation_Paper - GitHub — Research: Papers are written based on other papers. Nuances and insights need to be extracted from reading lengthy content. ... These findings reveal that minor structural modifications in a simple test can significantly alter the performance of LLMs in long context windows, which is not always noticeable in short context windows ...
• Noteworthy LLM Research Papers of 2024 - sebastianraschka.com — This article covers 12 influential AI research papers of 2024, ranging from mixture-of-experts models to new LLM scaling laws for precision. ... consisting of multiple stages, including short- and long-context pretraining. So, for optimal results, the recipes suggested in this paper may need to be tweaked under certain circumstances ...
• Lost in the Middle: How Language Models Use Long Contexts — Abstract. While recent language models have the ability to take long contexts as input, relatively little is known about how well they use longer context. We analyze the performance of language models on two tasks that require identifying relevant information in their input contexts: multi-document question answering and key-value retrieval. We find that performance can degrade significantly ...
• Long Context vs. RAG for LLMs: An Evaluation and Revisits - arXiv.org — hances retrieval accuracy for tasks requiring long-range or multi-step reasoning. 2.2 Long-Context LLMs Many research efforts focus on extending input and output windows to accommodate more context (see Figure1b), enabling applications such as extended dialogues, large document processing, and complex multimodal tasks. Thus, our analysis focuses on
• LIGHTTRANSFER OUR L -C LLM IS SECRETLY A HYBRID MODEL WITH ... - OpenReview — Recent advancements in large language models (LLMs) have extended their capacity for handling long context inputs and generating long-form reasoning. For example, LLaMA-3.1 supports context lengths up to 128K (Dubey et al., 2024), while OpenAI's o1 can produce sequences of up to 100K tokens (OpenAI, 2024).
• (PDF) \textsc{Long$$^2$$RAG}: Evaluating Long-Context \& Long-Form ... — However, current benchmarks for evaluating RAG systems suffer from two key deficiencies: (1) they fail to adequately measure LLMs' capability in handling \emph{long-context retrieval} due to a ...
• How Long-Context LLMs are Challenging Traditional RAG Pipelines — Key Takeaway: Hybrid architectures inherit the low-latency benefits of long-context LLMs while retaining the scalability and adaptability of RAG systems. 🌍 8.6 Real-World Applications of Hybrid ...

• LeanContext: Cost-efficient domain-specific question answering using LLMs — LLMs can learn domain-specific information in two ways, (a) via fine-tuning the model weights for the specific domain, (b) via prompting means users can share the contents with the LLMs as input context. Fine-tuning these large models containing billions of parameters is expensive and considered impractical if there is a rapid change of context over time (Schlag et al., 2023) e.g. a domain ...
• arXiv:2411.02886v2 [cs.CL] 3 Mar 2025 — TokenSelect on three representative long-context benchmarks using three open-source LLMs. The experimental results demonstrate that our TokenS-elect can achieve up to 23.84×speedup in atten-tion computation compared to FlashInfer (flashin-fer ai), and up to 2.28×acceleration in end-to-end inference latency compared to state-of-the-art long-
• InfLLM: Training-Free Long-Context Extrapolation for LLMs with an ... — LLMs to efficiently process long sequences with a limited context window and well capture long-distance dependencies. Without any training, InfLLM enables LLMs that are pre-trained on sequences consisting of a few thousand tokens to achieve comparable performance with competitive baselines that continually train these LLMs on long sequences.
• Long-Context Windows in Large Language Models: Applications in ... - Medium — Findings: The authors reported that GPT-4 and Claude 2 can handle extended context with only a slight drop in reasoning quality, whereas open-source long models (e.g. a 32k fine-tuned Llama 2 ...
• Lost in the Middle: How Language Models Use Long Contexts — Abstract. While recent language models have the ability to take long contexts as input, relatively little is known about how well they use longer context. We analyze the performance of language models on two tasks that require identifying relevant information in their input contexts: multi-document question answering and key-value retrieval. We find that performance can degrade significantly ...
• LongLLMLingua: ACCELERATING AND ENHANCING LLM L CONTEXT ... - OpenReview — code completion, and document summarization also necessitate the processing of long contexts. There are three main challenges when LLMs are used in long context scenarios: (1) The higher com-putational and financial cost required to run these models or to call APIs from companies providing LLM services.
• (PDF) LongRecipe: Recipe for Efficient Long Context ... - ResearchGate — Ultimately, *we can extend the effective context window of open-source LLMs from 8k to 128k, achieving performance close to GPT-4 with just one day of dedicated training using a single GPU with ...
• How Long-Context LLMs are Challenging Traditional RAG Pipelines — With the advent of long-context models — including state-of-the-art architectures like GPT-4, Claude 2, and LLaMA 3.2 — LLMs are now equipped with extended context windows capable of ...
• Marathon: A Race Through the Realm of Long Context with - arXiv.org — Presently, the open-source community is lack of benchmarks specifically designed to evaluate the proficiency of models in handling extended, multimodal contexts. Therefore, establishing a comprehensive benchmark for multimodal, long-context capabilities is of significant importance.
• natanaelwf/Reasoning-Degradation_Paper - GitHub — Full content of the paper 'Challenging Large Language Models (LLMs) Beyond Information Retrieval: Reasoning Degradation with Long Context Windows.' - natanaelwf/Reasoning-Degradation_Paper

• Understanding LLMs: A Comprehensive Overview from Training to Inference — Training and deploying LLMs demand expertise in handling large-scale data and substantial practical experience in distributed parallel training [27; 28; 29]. This requirement emphasizes the need for researchers developing LLMs to possess significant engineering capabilities in addressing the challenges encountered during LLM development.
• New LLM Pre-training and Post-training Paradigms - Sebastian Raschka, PhD — Build a Large Language Model (from Scratch) is a highly focused book dedicated to coding LLMs from the ground up in PyTorch, covering everything from pre-training to post-training—arguably the best way to truly understand LLMs. Machine Learning Q and AI is a great book for those who are already familiar with the basics; it dives into intermediate and advanced concepts covering deep neural ...
• The Ultimate Guide to Fine-Tuning LLMs from Basics to Breakthroughs: An ... — Large Language Models (LLMs) represent a significant leap in computational systems capable of understanding and generating human language. Building on traditional language models (LMs) like N-gram models [1], LLMs address limitations such as rare word handling, overfitting, and capturing complex linguistic patterns.Notable examples, such as GPT-3 and GPT-4 [2], leverage the self-attention ...
• Revolutionizing Long Context Processing in LLMs — The layers of attention in LLMs can vary significantly. Some layers are better at handling local dependencies while others work best with larger contexts. This variability means that injecting necessary pieces of information into the model improves performance, kind of like adding a pinch of salt to your soup to bring out the flavors.
• Long-Context Large Language Models - Lecture 15 - Class Central — Led by Professor Song Han, this 74-minute session examines advanced concepts in language model development, focusing specifically on handling extended context lengths in LLMs. Learn about the technical challenges, architectural considerations, and innovative solutions for processing and managing long-form text sequences in modern language models.
• Long-context LLMs Struggle with Long In-context Learning - arXiv.org — In this paper, we propose to adopt in-context learning (ICL) on extreme-label classification tasks (Anil et al., 2022; Milios et al., 2023) to evaluate long-context LLMs. Unlike the prior tasks, in-context learning requires LLMs to recognize the task by scanning over the entire input to understand the label space.
• Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG — Retrieval-augmented generation (RAG) empowers large language models (LLMs) to utilize external knowledge sources. The increasing capacity of LLMs to process longer input sequences opens up avenues for providing more retrieved information, to potentially enhance the quality of generated outputs. It is plausible to assume that a larger retrieval set would contain more relevant information ...
• Optimizing LLMs for Speed and Memory - Hugging Face — Memory requirements of LLMs can be best understood by seeing the LLM as a set of weight matrices and vectors and the text inputs as a sequence of vectors. In the following, the definition weights will be used to signify all model weight matrices and vectors. At the time of writing this guide, LLMs consist of at least a couple billion parameters.
• How Long-Context LLMs are Challenging Traditional RAG Pipelines — With the advent of long-context models — including state-of-the-art architectures like GPT-4, Claude 2, and LLaMA 3.2 — LLMs are now equipped with extended context windows capable of ...
• LoongTrain: Efficient Training of Long-Sequence LLMs with Head-Context ... — system to e ciently train LLMs with long sequences at scale. The core of LoongTrain is the 2D-Attention mechanism, which combines both head-parallel and context-parallel tech-