Benchmarking Inference Speed in LLMs

#llm #inference speed #benchmarking #optimization #quantization #pruning #hardware acceleration #latency #model deployment

1. Key Metrics for Measuring Inference Speed

Key Metrics for Measuring Inference Speed

Latency

Latency measures the time taken for a single inference request to complete, typically from input submission to output generation. For autoregressive models like GPT-3, latency is dominated by sequential token generation, making it highly sensitive to context length. The relationship between latency (L) and sequence length (n) can be modeled as:

$$ L(n) = t_{\text{prefill}} + n \cdot t_{\text{decode}} $$

where tprefill is the initial processing time for the prompt and tdecode is the per-token generation time. Modern transformer architectures exhibit tdecode values ranging from 10ms to 100ms per token on high-end GPUs, depending on model size and optimization techniques.

Throughput

Throughput quantifies the number of inferences completed per unit time (typically tokens/second) under maximum load. Unlike latency, throughput benefits from batch processing due to parallelizable matrix operations in transformer attention layers. The theoretical upper bound for throughput (T) with optimal batching is:

$$ T = \frac{B \cdot n}{t_{\text{prefill}} + n \cdot t_{\text{decode}}} $$

where B is batch size. In practice, memory bandwidth and KV cache management impose constraints, causing throughput to plateau at large B. For example, NVIDIA's benchmarks show Llama 2-70B achieves 2,300 tokens/sec on eight H100 GPUs with continuous batching.

Memory Bandwidth Utilization

Inference speed is fundamentally limited by memory bandwidth (β) and arithmetic intensity (I). The roofline model predicts maximum achievable performance as:

$$ \text{Performance} \leq \min(\pi, \beta \cdot I) $$

where π is peak compute throughput. Large language models typically operate in the memory-bound regime due to their low arithmetic intensity (0.1-1 FLOP/byte). Techniques like quantization reduce memory bandwidth pressure by decreasing model weights from FP16 (2 bytes) to INT8 (1 byte) or INT4 (0.5 bytes).

Hardware-Specific Metrics

Modern accelerators introduce specialized metrics for LLM inference:

NVIDIA's TensorRT-LLM reports these metrics through its benchmarking suite, enabling direct comparison across hardware platforms. For instance, the H100 GPU achieves 3× higher tokens/sec/Watt than A100 for Llama-2-13B through FP8 quantization and optimized attention kernels.

1.2 Factors Influencing Inference Latency

Inference latency in large language models (LLMs) is governed by a complex interplay of computational, architectural, and hardware-specific factors. Understanding these variables is critical for optimizing real-world deployment.

Model Architecture and Size

The transformer architecture introduces several latency-sensitive components:

$$ O(n^2 \cdot d) $$

where d represents the hidden dimension size. This becomes particularly problematic for long-context models.

Hardware Considerations

Modern accelerators exhibit different performance characteristics for key operations:

$$ \text{Latency} = \frac{\text{FLOPs}}{\text{Hardware Throughput}} + \text{Memory Access Penalty} $$

Key hardware factors include:

Quantization and Precision

Reducing numerical precision from FP32 to INT8 typically provides:

$$ \text{Speedup} \approx \frac{\text{Original Bitwidth}}{\text{Quantized Bitwidth}} \times \text{Hardware Efficiency Factor} $$

However, this introduces:

Software Optimizations

Modern inference runtimes employ several acceleration techniques:

Batch Processing Dynamics

Batch processing amortizes memory bandwidth costs but introduces new constraints:

$$ \text{Effective Throughput} = \frac{b \cdot n}{\max(t_{\text{serial}}, t_{\text{parallel}})} $$

where b is batch size and n is sequence length. The parallelizability depends on:

Context Window Effects

Variable sequence lengths create unique challenges:

$$ M_{\text{KV}} = 2 \cdot b \cdot n \cdot h \cdot l \cdot s $$

where h is heads, l is layers, and s is bytes per parameter. This directly impacts:

Factors Influencing Inference Latency – Benchmarking Inference Speed in LLMs – Tutorial Diagram
Diagram Description: The diagram would physically show the quadratic scaling relationship between sequence length and computational complexity in the attention mechanism, contrasting it with linear scaling of layer depth.

1.3 Hardware and Software Stack Considerations

The inference speed of large language models (LLMs) is heavily influenced by the underlying hardware and software stack. Optimizing these components requires a deep understanding of computational bottlenecks, memory hierarchies, and parallel processing capabilities.

Hardware Considerations

Modern LLM inference relies on three primary hardware configurations:

The theoretical peak throughput can be calculated for each device type. For GPUs with tensor cores:

$$ \text{TFLOPS} = \frac{\text{Cores} \times \text{Clock (GHz)} \times \text{Operations/cycle} \times \text{Precision Factor}}{10^3} $$

Where precision factors are 2 for FP16, 4 for INT8, and 8 for INT4 quantization. Memory bandwidth limitations create a roof for achievable performance:

$$ \text{Max Tokens/s} = \frac{\text{Bandwidth (GB/s)} \times 10^9}{\text{Model Size (bytes)} \times \text{Sequential Accesses}} $$

Software Optimization Techniques

The software stack introduces several optimization layers:

The total latency breakdown follows:

$$ T_{\text{total}} = T_{\text{compute}} + T_{\text{memory}} + T_{\text{communication}} + T_{\text{overhead}} $$

Framework-Specific Optimizations

Different inference frameworks employ distinct optimization strategies:

Framework Key Features Best Use Case
TensorRT-LLM Kernel auto-tuning, in-flight batching NVIDIA GPUs with dynamic workloads
vLLM PagedAttention, continuous batching High-throughput serving
ONNX Runtime Hardware-agnostic graph optimizations Cross-platform deployment

The choice of framework impacts achievable throughput through:

System-Level Bottlenecks

Real-world performance often deviates from theoretical peaks due to:

The effective memory bandwidth accounting for these factors becomes:

$$ BW_{\text{effective}} = BW_{\text{peak}} \times \prod_{i=1}^{n} (1 - \text{Penalty}_i) $$

Where penalties include contention (0.1-0.3), NUMA effects (0.05-0.2), and thermal throttling (0-0.15).

Hardware and Software Stack Considerations – Benchmarking Inference Speed in LLMs – Tutorial Diagram
Diagram Description: The diagram would show the relationship between hardware components (GPU/TPU/CPU) and their memory bandwidth/compute throughput tradeoffs, along with software stack optimizations as layered blocks.

2. Designing Effective Benchmarking Experiments

2.1 Designing Effective Benchmarking Experiments

Accurate benchmarking of inference speed in large language models (LLMs) requires careful experimental design to isolate variables, minimize noise, and ensure reproducibility. The following principles guide robust benchmarking setups:

Controlled Hardware Environment

Hardware consistency is critical for meaningful comparisons. Benchmarking must occur on identical or standardized hardware configurations, with attention to:

The computational throughput of matrix operations scales with memory bandwidth according to:

$$ \text{TFLOPS} = \frac{\text{Bandwidth (GB/s)} \times \text{Operation Intensity (FLOPs/byte)}}{10^3} $$

Input Sequence Design

Token sequence characteristics significantly impact inference latency. A well-designed benchmark suite should include:

Measurement Protocol

Precise timing requires:

The end-to-end latency for generating n tokens decomposes as:

$$ T_{\text{total}} = T_{\text{prompt}} + \sum_{i=1}^n T_{\text{token}_i} $$

Software Stack Control

Framework-specific optimizations can distort comparisons. Standardize:

Cross-Framework Validation

For architectural comparisons, implement identical models across frameworks (PyTorch, JAX, TensorFlow) while controlling for:

The computational intensity I of a transformer layer relates hardware utilization to model parameters:

$$ I = \frac{2 \times (\text{seq_len} \times d_{\text{model}}^2)}{\text{Memory Accesses}} $$

2.2 Standardized Benchmarking Frameworks

Standardized benchmarking frameworks provide reproducible methodologies for measuring inference speed across different hardware and software configurations. These frameworks eliminate variability introduced by ad-hoc testing procedures, ensuring fair comparisons between models. Key frameworks include MLPerf Inference, Hugging Face’s Transformers Benchmark, and NVIDIA’s TensorRT LLM Benchmark.

MLPerf Inference

MLPerf Inference is a widely adopted benchmark suite that evaluates latency and throughput under controlled conditions. It supports multiple scenarios:

The benchmark reports results in queries per second (QPS) and tail latency (P99). For transformer-based models, MLPerf uses fixed input sequences and enforces strict reproducibility rules, such as:

$$ \text{Latency} = t_{\text{end}} - t_{\text{start}} $$

Hugging Face Transformers Benchmark

Hugging Face’s benchmark focuses on real-world usability by testing models with dynamic input lengths and mixed precision (FP16/INT8). Key metrics include:

The framework automates warm-up iterations and statistical aggregation to reduce measurement noise. For example, the effective throughput is computed as:

$$ \text{Throughput} = \frac{N \times L}{\sum_{i=1}^{N} t_i} $$

where N is the batch size, L is sequence length, and ti is the latency for the i-th sample.

NVIDIA TensorRT LLM Benchmark

TensorRT LLM provides hardware-specific optimizations for NVIDIA GPUs, including kernel fusion and memory-efficient attention. Its benchmark measures:

The framework uses CUDA events for precise timing and supports quantized models. A critical optimization is the use of persistent thread blocks for attention layers, reducing overhead from:

$$ \text{Overhead} = t_{\text{kernel launch}} + t_{\text{synchronization}} $$

Cross-Framework Comparison

Results across frameworks are not directly comparable due to differing configurations. MLPerf uses fixed workloads, while Hugging Face and TensorRT LLM allow dynamic inputs. For research, MLPerf provides stricter controls, whereas TensorRT LLM reflects production-grade optimizations.

2.3 Handling Variable Input Lengths and Batch Sizes

Transformer-based language models process input sequences in parallel, but their computational efficiency is highly sensitive to input length and batch size variations. The self-attention mechanism's quadratic complexity with respect to sequence length (O(n²)) makes dynamic length handling particularly challenging for real-time applications.

Dynamic Batching Strategies

Static batching pads all sequences to the maximum length in a batch, wasting computation on padding tokens. Dynamic batching groups sequences of similar lengths to minimize padding while maintaining parallel processing:

$$ \text{Efficiency} = \frac{\sum_{i=1}^{B} L_i}{B \times \max(L_1,...,L_B)} $$

where B is batch size and L_i is sequence length. Advanced frameworks like NVIDIA's FasterTransformer implement:

Kernel Optimization for Ragged Tensors

Modern inference engines use specialized kernels for processing uneven sequences:

$$ \text{FLOPs} = 2 \times B \times \sum_{i=1}^{B} (L_i \times d_{model} \times d_{ff}) $$

Where dmodel is hidden dimension and dff is feed-forward dimension. Techniques include:

Memory Bandwidth Considerations

Variable-length processing exacerbates memory bandwidth bottlenecks. The effective bandwidth utilization follows:

$$ \beta_{eff} = \frac{\sum_{i=1}^{B} L_i \times d_{model}}{\max(L_i) \times B \times d_{model}} \times \beta_{peak} $$

Optimizations include:

Real-World Performance Tradeoffs

Benchmarks on A100 GPUs with Llama-2-70B show:

Strategy Throughput (tokens/sec) Latency (ms/token)
Static batching 1,240 38
Dynamic batching 2,810 17
Memory-optimized 3,450 14

The optimal strategy depends on the latency-throughput requirements of the deployment scenario. Streaming applications favor dynamic batching, while batch processing benefits from memory-aware approaches.

Handling Variable Input Lengths and Batch Sizes – Benchmarking Inference Speed in LLMs – Tutorial Diagram
Diagram Description: The diagram would show the comparison between static batching (uniform padding) and dynamic batching (grouped by similar lengths) with visual representation of sequence lengths and padding tokens.

3. Model Quantization and Pruning

Model Quantization and Pruning

Quantization: Reducing Precision for Efficiency

Quantization reduces the numerical precision of model parameters, typically from 32-bit floating-point (FP32) to 8-bit integers (INT8) or lower. The process involves mapping a continuous range of values to a discrete set, minimizing memory footprint and accelerating computation. For a weight tensor W with range [wmin, wmax], the quantized version Wq is computed as:

$$ W_q = \text{round}\left(\frac{W - w_{min}}{s}\right) $$ $$ s = \frac{w_{max} - w_{min}}{2^n - 1} $$

where s is the scaling factor and n is the target bit-width. Dequantization reconstructs the approximate original values:

$$ W' = W_q \times s + w_{min} $$

Post-training quantization (PTQ) applies this transformation after training, while quantization-aware training (QAT) simulates quantization during training to preserve accuracy. Mixed-precision quantization dynamically allocates bit-widths per layer based on sensitivity analysis.

Pruning: Removing Redundant Parameters

Pruning eliminates less important weights or neurons, creating sparse models. The magnitude-based pruning criterion removes weights below a threshold θ:

$$ \text{Prune if } |W_{ij}| < θ $$

Structured pruning removes entire channels or heads in transformer models, enabling hardware-friendly sparsity. The Lottery Ticket Hypothesis suggests that subnetworks capable of matching original performance exist within dense networks. Iterative pruning retrains the model after each sparsification step to recover accuracy.

Hardware Implications

Quantized models leverage integer arithmetic units (e.g., NVIDIA Tensor Cores) for 2-4× speedup over FP32. Sparse models require specialized kernels (e.g., CUDA Sparse Tensor Cores) to skip zero-valued computations. The FLOPs reduction ratio R for a pruned model with sparsity S is:

$$ R = \frac{1}{1 - S} $$

Modern compilers like TensorRT and TVM fuse quantization/dequantization ops and optimize kernel selection for target hardware.

Case Study: GPT-3 Optimization

Applying 8-bit quantization to GPT-3 (175B parameters) reduces memory usage from 700GB to 175GB. Combining 50% magnitude pruning with quantization achieves 10× inference speedup on A100 GPUs while retaining 98% of the original accuracy on benchmark tasks.

Trade-offs and Limitations

Model Quantization and Pruning – Benchmarking Inference Speed in LLMs – Tutorial Diagram
Diagram Description: The diagram would show the step-by-step transformation of a weight tensor through quantization and dequantization, with clear visual distinction between FP32 and INT8 representations.

3.2 Efficient Attention Mechanisms

Standard attention mechanisms in transformers exhibit quadratic complexity

$$ O(n^2) $$
with respect to sequence length n, creating bottlenecks for long-context inference. Recent advances optimize this through sparse, linear, or memory-efficient approaches while preserving model performance.

Sparse Attention Variants

Sparse attention reduces computation by limiting the attention field through predefined patterns. The Longformer introduces dilated sliding windows with global tokens, achieving

$$ O(n) $$
complexity. For a sequence of length n and window size w, the computation becomes:

$$ C_{sparse} = 4nwd^2 + 2n^2d $$

where d is the embedding dimension. The first term accounts for local attention within windows, while the second handles optional global attention positions.

Low-Rank Approximation Methods

Linformer's key insight projects the n×d key and value matrices to k×d dimensions (k ≪ n) via learned projections. The modified attention score calculation becomes:

$$ \text{Attention}(Q, K, V) = \text{softmax}\left(\frac{Q(E_iK)^T}{\sqrt{d}}\right)E_jV $$

where Ei, Ej are projection matrices. This reduces memory usage from O(n^2) to O(nk) while maintaining 95% of original accuracy on benchmark tasks.

Memory-Centric Optimizations

FlashAttention exploits hardware memory hierarchy through:

The algorithm achieves 2-4× speedup on modern GPUs by reducing memory reads/writes from

$$ O(n^2) $$
to
$$ O(n^{1.5}) $$
for sequences under 8k tokens.

Hybrid Approaches

Recent architectures like Sparse Sinkhorn Attention combine:

This achieves O(n log n) complexity with < 1% accuracy drop on GLUE benchmarks compared to full attention, while enabling processing of 64k-token sequences on consumer hardware.

Efficient Attention Mechanisms – Benchmarking Inference Speed in LLMs – Tutorial Diagram
Diagram Description: The section describes multiple attention mechanism variants with complex spatial relationships and computational tradeoffs that would be clearer visually.

3.3 Hardware-Specific Optimizations

Optimizing inference speed in large language models (LLMs) requires hardware-aware strategies that exploit the underlying architecture of modern accelerators. The primary bottlenecks—memory bandwidth, compute throughput, and parallelism—must be addressed through a combination of low-level optimizations and hardware-specific techniques.

GPU-Specific Optimizations

Modern GPUs, such as NVIDIA's A100 and H100, leverage tensor cores for mixed-precision matrix operations. To maximize throughput:

$$ \text{Effective Bandwidth} = \frac{\text{Total Bytes Transferred}}{\text{Execution Time}} $$

Theoretical peak bandwidth is rarely achieved due to memory access patterns. For instance, the A100's 1555 GB/s bandwidth can drop to 30-40% utilization without proper coalescing.

TPU-Specific Optimizations

Google's TPUs employ systolic arrays optimized for large matrix multiplications. Key considerations include:

Quantization and Sparsity

Reducing precision from FP32 to INT8 or INT4 via quantization can yield 2-4x speedups on supported hardware:

$$ W_{quant} = \text{round}\left(\frac{W}{s}\right) \cdot s, \quad s = \frac{\max(|W|)}{2^{b-1}-1} $$

where s is the scaling factor and b is the bit-width. NVIDIA's TensorRT and AMD's ROCm implement dynamic quantization with calibration to minimize accuracy loss.

Structured sparsity (e.g., 2:4 pattern) enables additional speedups by skipping zero-valued computations:

Memory Hierarchy Optimization

Modern accelerators feature complex memory hierarchies (HBM, L2 cache, shared memory). Effective strategies include:

The roofline model provides an analytical framework for identifying bottlenecks:

$$ \text{Attainable GFLOPs} = \min(\pi, \beta \cdot \text{Operational Intensity}) $$

where π is peak compute and β is memory bandwidth. For a 7B parameter model with 0.1 FLOP/byte operational intensity, memory bandwidth typically limits performance.

Hardware-Specific Optimizations – Benchmarking Inference Speed in LLMs – Tutorial Diagram
Diagram Description: The diagram would show the memory hierarchy of modern accelerators (HBM, L2 cache, shared memory) and how operator tiling interacts with these layers.

4. Benchmarking Popular LLMs (GPT, Llama, Mistral)

4.1 Benchmarking Popular LLMs (GPT, Llama, Mistral)

When benchmarking inference speed across large language models (LLMs), three key architectures dominate contemporary research and deployment: OpenAI's GPT family, Meta's Llama series, and Mistral AI's models. Each exhibits distinct computational characteristics that influence real-world performance.

Computational Complexity and Architectural Differences

The inference latency of transformer-based LLMs primarily depends on their attention mechanism scaling. For a model with n layers, h attention heads, and sequence length s, the time complexity of self-attention is:

$$ O(s^2 \cdot h \cdot d) $$

where d represents the hidden dimension size. GPT-4 employs a mixture-of-experts architecture that dynamically routes computations, while Llama 2's grouped-query attention reduces memory bandwidth requirements. Mistral 7B achieves efficiency through sliding window attention with:

$$ O(s \cdot w \cdot h \cdot d) $$

where w is the fixed window size (typically 4096 tokens).

Quantitative Benchmarking Methodology

Standardized benchmarking requires controlling for:

The inference time T for a forward pass can be modeled as:

$$ T = T_{prefill} + k \cdot T_{decode} $$

where k is the number of generated tokens, with prefill time dominated by matrix multiplications and decode time by memory bandwidth.

Empirical Performance Comparison

On an A100 GPU with FP16 precision and 2048-token sequences:

Model Params (B) Prefill (ms) Decode (ms/token) Mem (GB)
GPT-4 ~220 420 85 84
Llama 2 70B 70 380 62 48
Mistral 7B 7 110 18 14

The memory-bandwidth-bound nature of decoding becomes apparent when examining the relationship between model size and token generation speed. For the 70B parameter regime, the theoretical roofline for memory bandwidth (1555GB/s on A100) predicts:

$$ T_{decode} \approx \frac{2 \cdot \text{params} \cdot \text{bytes}}{\text{bandwidth}} $$

which aligns with observed measurements when accounting for kernel launch overheads.

Optimization Techniques

Recent advances in inference optimization demonstrate significant speedups:

When applied to Llama 2 70B, these techniques can achieve:

$$ \text{Speedup} = 1.8 \times \text{FlashAttention} \times 1.3 \times \text{Parallelism} $$

yielding 150ms/token latency at scale. The optimal configuration depends on the specific deployment constraints, with smaller models like Mistral 7B benefiting more from quantization than architectural optimizations.

4.2 Real-World Deployment Scenarios

In production environments, LLM inference speed is constrained by hardware limitations, batch processing requirements, and latency-sensitive applications. The interplay between model architecture, quantization techniques, and hardware acceleration determines practical throughput. For instance, a 175B-parameter model like GPT-3 requires approximately 350GB of GPU memory in FP32 precision, making naive deployment infeasible for real-time applications.

Latency-Critical Applications

Chatbots and voice assistants demand sub-200ms response times to maintain conversational flow. This imposes strict constraints on autoregressive generation length and batch size. The end-to-end latency L for generating n tokens can be modeled as:

$$ L = t_{prefill} + n \cdot t_{decode} $$

where tprefill is the initial prompt processing time and tdecode is the per-token generation time. Optimizing this tradeoff requires:

Throughput-Optimized Scenarios

Batch processing applications like document summarization prioritize tokens/second over individual request latency. Here, the key metric becomes hardware utilization efficiency, measured by the ratio of achieved throughput to theoretical maximum:

$$ \eta = \frac{T_{actual}}{T_{peak}} $$

Modern inference servers achieve 60-80% utilization through:

Edge Deployment Constraints

Mobile and embedded devices introduce additional challenges due to thermal limits and memory bandwidth. A quantized 7B-parameter model running on a smartphone GPU typically achieves 5-15 tokens/second, with performance bounded by:

$$ t_{decode} \propto \frac{M}{B \cdot f} $$

where M is model size, B is memory bandwidth, and f is clock frequency. Practical deployments use:

Case Study: Large-Scale Search Augmentation

A major search engine deployed a 137B-parameter LLM for query understanding, requiring <1ms latency per token at 50k queries/second. Their solution combined:

This achieved 0.8ms/token latency while maintaining 99.9% cache hit rate for frequent queries, demonstrating how architectural innovations can overcome theoretical hardware limits.

4.3 Trade-offs Between Speed and Accuracy

Quantifying the Pareto Frontier

The relationship between inference speed and model accuracy in LLMs is governed by a Pareto frontier, where improvements in one metric degrade the other. This trade-off arises from architectural choices, computational constraints, and statistical limits. For a transformer-based model with L layers, d attention heads, and hidden dimension h, the theoretical lower bound on latency for autoregressive generation can be expressed as:

$$ \tau_{\text{min}} = N \cdot \left( L \cdot t_{\text{layer}} + C \cdot \log_2 h \right) $$

where N is sequence length, tlayer is per-layer processing time, and C accounts for memory bandwidth constraints. Meanwhile, the perplexity (PP) degradation when applying quantization or pruning follows:

$$ \Delta \text{PP} \approx \alpha \cdot \left( \frac{\Delta W}{W_{\text{FP32}}} \right)^2 + \beta \cdot \left( \frac{\Delta A}{A_{\text{FP32}}} \right)^2 $$

where ΔW and ΔA represent weight and activation quantization errors, with coefficients α, β empirically determined through neural tangent kernel analysis.

Architectural Levers for Optimization

Three primary techniques alter the speed-accuracy curve:

Hardware-Aware Optimization

The optimal operating point depends on hardware characteristics. For NVIDIA A100 GPUs with 1,555 GB/s memory bandwidth, the compute-bound regime occurs when:

$$ \frac{4Lh^2}{N_{\text{batch}} \cdot \text{TFLOPS}^{-1} > \frac{12Lh}{B_{\text{mem}}} $$

This suggests different optimization strategies for memory-bound versus compute-bound scenarios. TensorRT-LLM demonstrates this by achieving 3.1× faster inference than vanilla FP16 with INT8 quantization on memory-bound workloads, but only 1.8× improvement on compute-bound tasks.

Case Study: Mixture of Experts

Sparse MoE models like Switch Transformer exemplify the trade-off's nonlinear nature. With expert utilization k/N (where k is active experts per token), the speedup follows:

$$ S = \frac{N}{k + c(N - k)} $$

where c ≈ 0.1 represents routing overhead. At 64 experts with k=2, this achieves 8.9× speedup over dense models while maintaining 98.7% of the accuracy on multilingual benchmarks.

Trade-offs Between Speed and Accuracy – Benchmarking Inference Speed in LLMs – Tutorial Diagram
Diagram Description: The diagram would show the Pareto frontier curve plotting inference speed versus model accuracy, with labeled points for different optimization techniques (distillation, quantization, early exit).

5. Key Research Papers on LLM Inference

5.1 Key Research Papers on LLM Inference

5.2 Open-Source Benchmarking Tools

5.3 Recommended Books and Articles