Overview of Text Generation in NLP

Text generation in natural language processing refers to the task of producing coherent and contextually relevant sequences of words from a given input or initial state. Modern approaches leverage probabilistic language models that estimate the conditional probability distribution over possible word sequences, typically factorized autoregressively:

Role of Decoding Strategies in Language Models

Decoding strategies govern how language models generate sequences by selecting tokens from a probability distribution at each step. The choice of strategy significantly impacts the quality, diversity, and computational efficiency of the generated text. Two primary approaches dominate modern implementations: greedy search and beam search, each with distinct trade-offs in performance and output characteristics.

Probability Distributions and Token Selection

At each step t, a language model outputs a probability distribution P(y_t | y_<t, x) over the vocabulary, where y_<t represents previously generated tokens and x is the input context. The decoding strategy determines how to select the next token y_t from this distribution. The simplest approach, greedy search, selects the token with the highest probability:

where V is the vocabulary. While computationally efficient, this approach often leads to suboptimal sequences due to its myopic nature—it cannot revise earlier choices even if a lower-probability token at step t would lead to a higher-probability sequence overall.

Beam Search: Balancing Quality and Efficiency

Beam search addresses this limitation by maintaining k candidate sequences (beams) at each step, where k is the beam width. At step t, it extends each partial sequence in the beam with the top k most probable next tokens, resulting in k² candidates. These are pruned back to the top k sequences based on their cumulative log probabilities:

The process repeats until sequences reach an end-of-sequence token or a maximum length. Beam search often produces higher-quality outputs than greedy search but at increased computational cost. Variations like length normalization adjust scores to avoid bias toward shorter sequences:

where α is a tunable parameter typically between 0.6 and 1.0.

Practical Considerations

In real-world applications, the choice between greedy and beam search depends on the task:

• Greedy search is preferred when latency is critical (e.g., real-time chatbots) or when the model's probability distributions are sharply peaked.
• Beam search excels in tasks requiring coherence and long-range dependencies (e.g., summarization, machine translation), though larger beams increase memory and compute requirements quadratically.

Advanced variants like diverse beam search introduce mechanisms to promote diversity among beams, mitigating the common issue of repetitive or generic outputs in standard beam search. Meanwhile, stochastic methods like top-k sampling and nucleus sampling offer alternative approaches for generating diverse and creative text.

Key Metrics for Evaluating Generated Text

Evaluating the quality of machine-generated text requires a combination of automated metrics and human judgment. While no single metric captures all aspects of text quality, several well-established measures provide quantitative insights into different dimensions of generated output.

Perplexity

Perplexity measures how well a language model predicts a given sequence of words. It is derived from the cross-entropy loss and represents the exponential of the average negative log-likelihood per token:

2.1 How Greedy Search Works Step-by-Step

Greedy search is a deterministic decoding strategy for autoregressive text generation where, at each timestep t, the model selects the token with the highest predicted probability from the vocabulary distribution P(w_t|w_1:t-1, x). Unlike beam search, it maintains only a single active sequence, making it computationally efficient but prone to locally optimal choices.

Mathematical Formulation

Given an input sequence x and partially generated output w_1:t-1, the greedy selection criterion is:

where V is the vocabulary and P is the model's output distribution. The search terminates when either:

• The end-of-sequence (EOS) token is selected
• A predefined maximum length is reached

Step-by-Step Execution

Consider generating text from a transformer-based language model with vocabulary V = {A, B, C, EOS}:

1. Initialization: Start with the input prompt "The" and hidden state h₀
2. Timestep 1:
   $$ P(w_1|\text{"The"}) = \{A:0.6, B:0.3, C:0.1\} $$
   Select w₁ = A (highest probability)
3. Timestep 2:
   $$ P(w_2|\text{"The A"}) = \{A:0.2, B:0.7, C:0.1\} $$
   Select w₂ = B
4. Termination:
   $$ P(w_3|\text{"The A B"}) = \{A:0.1, EOS:0.8, C:0.1\} $$
   Select w₃ = EOS, yielding final output "The A B"

Computational Complexity

The time complexity for generating n tokens is O(n|V|d), where d is the model's hidden dimension. This linear scaling makes greedy search attractive for real-time applications, though it requires n sequential forward passes.

Limitations and Failure Modes

Greedy search frequently produces degenerate outputs due to:

• Local maxima: Early high-probability choices may lead to low-probability overall sequences
• Repetition: The model may enter loops (e.g., "The the the...")
• Lack of exploration: Alternative high-probability paths are permanently discarded

For the sequence "The A B" above, the joint probability is 0.6 × 0.7 × 0.8 = 0.336, while a potentially better sequence "The B A EOS" with probabilities 0.3 × 0.6 × 0.9 = 0.162 would never be discovered.

Practical Considerations

Greedy search performs adequately when:

• The output distribution is sharply peaked (low entropy)
• Exact reproducibility is required (deterministic output)
• Latency constraints prohibit beam search's O(kn|V|d) complexity

Modern implementations often combine greedy search with:

• Top-k sampling (select from k highest-probability tokens)
• Temperature scaling (sharpening/flattening distributions)
• Repetition penalties

Advantages and Limitations of Greedy Search

Greedy search is a deterministic decoding strategy that selects the token with the highest probability at each step in the sequence generation process. Formally, given a sequence of previously generated tokens y_<t, the next token y_t is chosen as:

2.3 Practical Example: Implementing Greedy Search in Python

Greedy search operates by selecting the token with the highest probability at each decoding step without considering future consequences. This local optimization strategy is computationally efficient but may lead to suboptimal global sequences. Let's implement it step-by-step using PyTorch and Hugging Face's Transformers library.

Core Implementation Components

The greedy decoding process requires three key components:

• Token probability distribution generation from the language model
• Argmax operation to select the highest-probability token
• Recursive sequence construction until termination

Complete Python Implementation

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

def greedy_decode(model, tokenizer, input_text, max_length=50):
    # Tokenize input and convert to tensor
    input_ids = tokenizer.encode(input_text, return_tensors='pt')
    
    # Initialize output sequence with input_ids
    generated = input_ids
    
    # Disable gradient calculation for inference
    with torch.no_grad():
        for _ in range(max_length):
            # Forward pass through model
            outputs = model(generated)
            
            # Get logits of last token position
            next_token_logits = outputs.logits[:, -1, :]
            
            # Apply temperature scaling (optional)
            temperature = 1.0
            next_token_logits = next_token_logits / temperature
            
            # Greedy selection: argmax
            next_token = torch.argmax(next_token_logits, dim=-1, keepdim=True)
            
            # Append to generated sequence
            generated = torch.cat((generated, next_token), dim=-1)
            
            # Stop if EOS token is generated
            if next_token.item() == tokenizer.eos_token_id:
                break
    
    return tokenizer.decode(generated[0], skip_special_tokens=True)

# Example usage
model_name = "gpt2"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

input_text = "The future of AI is"
output_text = greedy_decode(model, tokenizer, input_text)
print(output_text)

Mathematical Foundation

The greedy search algorithm implements the following decision rule at each timestep t:

where V is the vocabulary, x is the input context, and y_1:t-1 represents previously generated tokens. The joint probability of the sequence decomposes as:

Performance Considerations

The implementation makes several optimizations:

• Batch Processing: The model processes the entire sequence in a single forward pass for each token
• Memory Efficiency: Only stores the generated token IDs rather than full probability distributions
• Early Stopping: Terminates when encountering the end-of-sequence token

Practical Limitations

While simple to implement, greedy search suffers from several weaknesses:

• Local Optima: May get stuck in repetitive loops or generic responses
• Lack of Exploration: Never considers high-probability alternatives that could lead to better global sequences
• Diversity Issues: Tends to produce deterministic, less creative outputs

Advanced Variants

Simple modifications can improve greedy search performance:

# Temperature-scaled greedy decoding
def temperature_scaled_greedy(..., temperature=0.7):
    ...
    next_token_logits = next_token_logits / temperature
    next_token = torch.argmax(next_token_logits, dim=-1)
    ...

Temperature scaling (T ∈ (0,1]) softens the probability distribution before argmax:

Core Algorithm of Beam Search

Beam search is a heuristic search algorithm that explores multiple candidate sequences in parallel while maintaining a fixed-size subset of the most promising hypotheses, known as the beam width (k). Unlike greedy search, which selects the single highest-probability token at each step, beam search retains k partial sequences, expanding them iteratively to balance exploration and exploitation.

Mathematical Formulation

Given a sequence of tokens y_<t generated up to step t, beam search aims to maximize the joint probability of the entire sequence:

3.2 Hyperparameters: Beam Width and Length Penalties

The effectiveness of beam search hinges on two critical hyperparameters: beam width and length normalization. These parameters directly influence the trade-off between computational efficiency and output quality.

Beam Width (k)

Beam width determines the number of candidate sequences retained at each decoding step. A larger k increases the likelihood of finding high-probability sequences but at the cost of higher computational overhead. The probability of a sequence y given input x is:

3.3 Trade-offs Between Diversity and Coherence

Text generation strategies like greedy search and beam search inherently face a fundamental tension between diversity and coherence. Greedy search, which selects the token with the highest probability at each step, tends to produce highly coherent but often repetitive and predictable outputs. Beam search mitigates this by maintaining multiple candidate sequences, but even then, the likelihood-focused objective can lead to generic or overly safe responses.

Quantifying the Diversity-Coherence Trade-off

The trade-off can be formalized using entropy-based metrics. For a sequence of tokens y_1:t generated by a language model with vocabulary V, the conditional entropy at step t+1 is:

Higher entropy indicates greater diversity in potential next tokens, while lower entropy suggests more deterministic, coherent continuations. Beam search with a narrow beam width k effectively truncates the probability distribution, reducing entropy and favoring high-probability (coherent) tokens.

Techniques for Balancing the Trade-off

Several methods have been proposed to explicitly control this trade-off:

• Temperature scaling: Modifying the softmax distribution via a temperature parameter τ:

where τ → 0 approaches greedy sampling (high coherence), and τ → ∞ yields uniform sampling (high diversity).

• Top-k and top-p sampling: Restricting sampling to the k most likely tokens or the smallest set whose cumulative probability exceeds p. This preserves coherence while allowing controlled diversity.
• Diversity-promoting objectives: Modifying beam search to penalize sequences with high n-gram overlap or incorporating mutual information maximization.

Empirical Observations

Recent studies on open-ended generation tasks reveal that human-like text requires navigating a "narrow pathway" between these extremes. For example:

• In dialogue systems, beam search with k=10 produces more engaging responses than greedy decoding, but may still lack originality.
• Creative writing applications often benefit from hybrid approaches like top-p sampling with p=0.9, maintaining narrative flow while avoiding repetition.

The optimal balance depends heavily on the application domain—technical documentation generation prioritizes coherence, while poetry generation may intentionally sacrifice some coherence for artistic diversity.

Emerging Approaches

Advanced methods like contrastive search explicitly optimize for both aspects by selecting tokens that:

where α controls the diversity penalty based on token embedding similarity. This achieves state-of-the-art results by dynamically adjusting the coherence-diversity balance during generation.

3.4 Case Study: Beam Search in Machine Translation

Beam search is a critical component in neural machine translation (NMT) systems, where generating fluent and accurate translations requires balancing exploration and exploitation. Unlike greedy search, which selects the highest-probability token at each step, beam search maintains k partial hypotheses (beams) and expands them iteratively, pruning low-scoring candidates.

Mathematical Formulation

Given a source sentence X and target sentence Y, the translation probability is modeled as:

4.1 Performance Comparison on Standard Benchmarks

When evaluating text generation strategies, empirical performance on standardized benchmarks provides critical insights into the trade-offs between greedy search and beam search. Key metrics include perplexity, BLEU score, ROUGE-L, and human evaluation scores, measured across datasets like WMT (Machine Translation), CNN/Daily Mail (Summarization), and WikiText (Language Modeling).

Quantitative Metrics

Greedy search, which selects the token with the highest probability at each step, often achieves lower computational overhead but suffers from local optima. Beam search (with beam width B) explores multiple hypotheses, improving sequence likelihood but at the cost of increased latency. The likelihood of a generated sequence y given input x can be formalized as:

4.2 Computational Efficiency and Memory Usage

Greedy search and beam search exhibit fundamentally different computational behaviors due to their contrasting exploration strategies. Greedy search operates with constant memory O(1) per time step, maintaining only a single candidate sequence. The time complexity scales linearly with sequence length L as O(L·V), where V is vocabulary size, as it performs a simple argmax operation over logits at each step.

where d_model represents the transformer's hidden dimension. Beam search with width k requires maintaining k active sequences, resulting in memory complexity O(kL). The time complexity becomes O(L·k·V) due to the top-k selection process:

Memory Bandwidth Bottlenecks

Modern GPUs face significant memory bandwidth constraints when executing beam search. Each candidate sequence requires separate attention key-value caches in autoregressive transformers, creating k-fold memory pressure compared to greedy decoding. For a model with n_layers layers and cache size d_head, the KV cache memory consumption is:

Practical implementations often hit memory limits before compute limits - a 175B parameter model with k=8 and L=2048 can require over 80GB just for KV caches at FP16 precision.

Parallelization Trade-offs

Beam search enables two parallelization dimensions: intra-sequence (across timesteps) and inter-sequence (across beams). However, the irregular computation patterns of active beam pruning create workload imbalance. Modern frameworks like TensorRT-LLM implement:

• Speculative execution for beams with common prefixes
• Dynamic batching of variable-length beam candidates
• Memory sharing for identical beam histories

These optimizations can reduce memory overhead by 30-50% while maintaining the same search quality.

Quantitative Comparison

Benchmarks on an A100 GPU with Llama-2-7B reveal stark differences:

The quadratic growth in memory and linear decrease in throughput demonstrate the fundamental trade-off between exploration quality and computational cost.

When to Choose Greedy or Beam Search

The choice between greedy search and beam search depends on the trade-offs between computational efficiency, output quality, and task-specific requirements. Each strategy has distinct advantages and limitations that make them suitable for different scenarios.

Computational Efficiency vs. Output Quality

Greedy search is computationally efficient, requiring only a single forward pass per time step. At each step, it selects the token with the highest probability:

where V is the vocabulary and x is the input context. This makes greedy search ideal for real-time applications where latency is critical, such as autocomplete systems or voice assistants. However, it often produces suboptimal sequences due to its myopic decision-making.

Beam search maintains k candidate sequences at each step, where k is the beam width. The probability of a partial sequence is computed as:

By exploring multiple hypotheses, beam search generally produces higher-quality outputs but requires O(k) more computation than greedy search. The choice of k significantly impacts performance—larger beams improve quality but increase latency and memory usage.

Task-Specific Considerations

Use greedy search when:

• Low-latency generation is required (e.g., conversational agents).
• The model is well-calibrated, and local maxima reliably lead to good global solutions.
• Repetition or generic outputs are acceptable (e.g., short responses in chatbots).

Use beam search when:

• Output quality is prioritized over speed (e.g., document summarization, machine translation).
• The task benefits from global sequence optimization (e.g., maintaining coherence in long-form text).
• Diverse outputs are needed (achievable by tuning beam diversity parameters).

Practical Trade-offs

In machine translation, beam search (with k=5 to 10) is standard because it reduces fluency errors. For open-ended generation (e.g., story writing), smaller beams (k=2 to 5) balance quality and creativity. Greedy decoding suffices for constrained tasks like named entity recognition, where correctness depends more on input context than sequential decisions.

Recent hybrid approaches, such as adaptive beam search, dynamically adjust k based on uncertainty metrics. For example:

where H(P_t) is the entropy of the token distribution at step t, and θ is a threshold. This conserves resources during low-uncertainty steps while maintaining quality for ambiguous predictions.

5.1 Stochastic Beam Search and Temperature Sampling

Stochastic beam search introduces randomness into the traditional beam search algorithm by probabilistically selecting candidates at each decoding step. Unlike deterministic beam search, which retains the top-k highest-scoring sequences, stochastic beam search samples sequences according to their probability distribution, enabling more diverse outputs while maintaining coherence.

Mathematical Formulation

Given a sequence probability distribution P(y_t | y_<t, x) at step t, stochastic beam search applies the following steps:

Here, τ (temperature) controls the sharpness of the distribution. Lower values (τ < 1) amplify high-probability tokens, while higher values (τ > 1) flatten the distribution.

Temperature Sampling

Temperature sampling modifies the softmax output to control the trade-off between diversity and likelihood:

Key effects of temperature:

• τ → 0: Approximates greedy search (deterministic).
• τ = 1: Standard softmax sampling.
• τ → ∞: Uniform sampling over vocabulary.

Implementation Considerations

Stochastic beam search requires careful handling of sequence probabilities during sampling. Unlike standard beam search, where scores are cumulative log-probabilities, stochastic variants often use:

where η is a noise term (e.g., Gumbel noise for differentiable sampling). Practical implementations often combine temperature sampling with top-k or top-p (nucleus) filtering to avoid low-probability tokens.

Comparative Analysis

Empirical studies show stochastic beam search with temperature tuning achieves:

• Higher diversity (measured by distinct-n scores) than deterministic beam search.
• Better perplexity-temperature trade-offs than pure random sampling.
• Improved performance in open-ended generation tasks (e.g., story generation).

The method is particularly effective when combined with techniques like length normalization and repetition penalty, as it allows controlled exploration of the solution space without collapsing to high-likelihood but generic outputs.

5.2 Combining Beam Search with Top-k or Top-p Sampling

Beam search, while effective for deterministic text generation, often suffers from lack of diversity and repetitive outputs. Integrating stochastic sampling methods like top-k or top-p (nucleus sampling) with beam search can mitigate these issues while retaining coherence. The hybrid approach leverages the exploratory nature of sampling while maintaining the structured search of beam decoding.

Mathematical Formulation

Given a beam width B, top-k restricts the sampling pool to the k most probable tokens at each step, while top-p dynamically truncates the distribution by selecting the smallest set of tokens whose cumulative probability exceeds p. The combined strategy modifies the beam search scoring function:

5.3 Recent Innovations in Decoding Strategies

Traditional decoding methods like greedy search and beam search have limitations in generating diverse and coherent text. Recent innovations address these shortcomings through probabilistic, constrained, and adaptive techniques.

Nucleus Sampling (Top-p Sampling)

Nucleus sampling dynamically truncates the probability distribution by selecting the smallest set of tokens whose cumulative probability exceeds a threshold p. This avoids both the determinism of greedy search and the repetition issues of beam search. The probability mass is redistributed among the selected tokens:

6.1 Key Research Papers on Decoding Strategies

• PDF Faithfulness-Aware Decoding Strategies for Abstractive Summarization — according to the decoding strategies. We explore three common decoding strategies: greedy, beam search, and nucleus sampling (Holtzman et al., 2020). Greedy search selects the next token by the most probable token yt = argmax y p(y jy1: t 1;x ). Beam search extends greedy search by keeping top- k hypothesis at each time step, where k is the ...
• Generating Human-level Text with Contrastive Search in Transformers — Natural language generation (i.e. text generation) is one of the core tasks in natural language processing (NLP). In this blog, we introduce the current state-of-the-art decoding method, Contrastive Search, for neural text generation.Contrastive search is originally proposed in "A Contrastive Framework for Neural Text Generation" ([Official Implementation]) at NeurIPS 2022.
• (PDF) Best-First Beam Search - Academia.edu — With certain values of these attributes, we recover many common search algorithms: greedy search, beam search, best-first search (Dijkstra, 1959), and A∗ search (Hart et al., 1968). We propose an alternate prioritization function for beam search that allows for faster decoding while still returning the same k -optimal set of hypotheses.
• PDF Best-k Search Algorithm for Neural Text Generation - ACL Anthology — et al.,2020;Fabbri et al.,2021). The decoding strategy is another crucial piece in this paradigm. If we form text generation as a search problem, de-coding strategies are essentially search algorithms over the space composed by vocabulary V. Beam search, a heuristic search algorithm, has been the go-to choice for many years. However, the gener-
• Sheet 6.3: Decoding strategies — Neural Pragmatic Natural Language ... — Sheet 6.3: Decoding strategies# Given a (blackbox) function that gives us a next-word probability, how do we use this to generate naturally sounding text? This tutorial explores a bunch of options, using the GPT-2 distribution provided by 🤗's transformer package. ... greedy sampling. beam search. top-\(k\) sampling.
• PDF Towards Fast Inference: Exploring and Improving Blockwise Parallel Drafts — ken distribution (Holtzman et al.,2019), or by a beam search through the space of possible sequences to return a probable sequence. Greedy decoding, a special case of beam search, generates each token as yˆ t+1 = argmaxp θ(y t+1|¯x,y ≤t). In this work, we consider greedy decoding exclusively, as this is the setting thatStern et al.(2018 ...
• 10.8. Beam Search — Dive into Deep Learning 1.0.3 documentation - D2L — In Section 10.7, we introduced the encoder-decoder architecture, and the standard techniques for training them end-to-end.However, when it came to test-time prediction, we mentioned only the greedy strategy, where we select at each time step the token given the highest predicted probability of coming next, until, at some time step, we find that we have predicted the special end-of-sequence ...
• Understanding Decoding Strategies In Large Language Models (LLMs ... — Darius Baruo Aug 22, 2024 04:58 Explore how Large Language Models (LLMs) choose the next word using decoding strategies. Learn about different methods like greedy search, beam search, and more. Large Language Models (LLMs) are trained to predict the next word in a text sequence. However, the […]
• PDF Text Encoding and Decoding from Global Perspectives — Ye Ma Text Encoding and Decoding from Global Perspectives ator is almost impossible to generate the whole sentence at once, the heuristic algorithm { beam search has been the natural choice for text decoding. Inevitably, beam search often gets stuck of local optimum as it decodes word-by-word. Although global optimum
• (PDF) Follow the Wisdom of the Crowd: Effective Text Generation via ... — Greedy and beam search are known to suffer from text degeneration and linguistic diversity issues, while temperature, top-k, and nucleus sampling often yield diverse but low-quality outputs.

6.2 Recommended Books and Online Courses

• arXiv:2303.03278v1 [cs.CL] 6 Mar 2023 — Beam search extends greedy search by keeping top-khypothesis at each time step, where kis the number of beams. Another approach to decoding is to use sampling, where we consider nucleus sam-pling.Holtzman et al.(2020) surprisingly find that methods that optimize probability, such as beam search, may lead to text degeneration, and thus pro-
• A Contrastive Framework for Neural Text Generation - arXiv.org — Deteriminstic Methods. Two widely used deterministic approaches are greedy and beam search which aim to select the text continuation with highest probability based on the model's probability distribution p . However, solely maximizing the output probability often leads to dullness [22] and degeneration [11,14] in the generated text ...
• PDF A Contrastive Framework for Neural Text Generation — Deteriminstic Methods. Two widely used deterministic approaches are greedy and beam search which aim to select the text continuation with highest probability based on the model's probability distribution p θ. However, solely maximizing the output probability often leads to dullness [13] and degeneration [7,10] in the generated text.
• Sheet 6.3: Decoding strategies — Neural Pragmatic Natural Language ... — Sheet 6.3: Decoding strategies# Given a (blackbox) function that gives us a next-word probability, how do we use this to generate naturally sounding text? This tutorial explores a bunch of options, using the GPT-2 distribution provided by 🤗's transformer package. ... greedy sampling. beam search. top-\(k\) sampling.
• Leveraging transferability and improved beam search in textual ... — In the NLP field, existing methods of black-box adversarial attacks can be roughly divided into two categories: 1) Query-based attacks find vulnerable tokens by querying the output decisions and scores of the target model, and then apply different strategies to these tokens to generate adversarial texts; 2) transfer-based attacks utilize a surrogate model to approximate the decision boundary ...
• ai开发 - 鹅厂专家讲透AI文本生成解码策略与代码实现 - 腾讯云技术社区 - SegmentFault 思否 — 腾小云导读. 本文以 huggingface-transformers 的文本生成解码代码为例，对文本生成常用的五种解码策略 greedy search、beam search、sample、sample and rank & beam sample、group beam search 进行逐行解读。
• PDF Lecture 5 Local Search - Department of Computer Science, University of ... — In this section, I will discuss Greedy Descent, our rst local search algorithm. This algorithm has many other names: hill-climbing, greedy ascent, and iterative best im-provement. I will refer to this algorithm as Greedy Descent since our goal is to minimize a cost function. Greedy Descent works as follows: Start with a random state.
• (PDF) A Contrastive Framework for Neural Text Generation - ResearchGate — Text generation is of great importance to many natural language processing applications. However, maximization-based decoding methods (e.g. beam search) of neural language models often lead to ...
• PDF NEURAL GENERATION A DISSERTATION - Stanford University — neural generation of open-ended text and dialogue a dissertation submitted to the department of computer science and the committee on graduate studies of stanford university in partial fulfillment of the requirements for the degree of doctor of philosophy abigail see august 2021
• 10.8. Beam Search — Dive into Deep Learning 1.0.3 documentation - D2L — In Section 10.7, we introduced the encoder-decoder architecture, and the standard techniques for training them end-to-end.However, when it came to test-time prediction, we mentioned only the greedy strategy, where we select at each time step the token given the highest predicted probability of coming next, until, at some time step, we find that we have predicted the special end-of-sequence ...

6.3 Open-source Implementations and Toolkits

• 9.7. Beam Search — Dive into Deep Learning 0.1.0 documentation - DJL — 9.7. Beam Search¶. In Section 9.6, we predicted the output sequence token by token until the special end-of-sequence "" token is predicted.In this section, we will begin with formalizing this greedy search strategy and exploring issues with it, then compare this strategy with other alternatives: exhaustive search and beam search.. Before a formal introduction to greedy search, let us ...
• PDF A Contrastive Framework for Neural Text Generation — Deteriminstic Methods. Two widely used deterministic approaches are greedy and beam search which aim to select the text continuation with highest probability based on the model's probability distribution p θ. However, solely maximizing the output probability often leads to dullness [13] and degeneration [7,10] in the generated text.
• PDF Best-k Search Algorithm for Neural Text Generation - ACL Anthology — strategy is another crucial piece in this paradigm. If we form text generation as a search problem, de-coding strategies are essentially search algorithms over the space composed by vocabulary V. Beam search, a heuristic search algorithm, has been the go-to choice for many years. However, the gener-ated sequences are usually repetitive because many
• Sheet 6.3: Decoding strategies — Neural Pragmatic Natural Language ... — Sheet 6.3: Decoding strategies# Given a (blackbox) function that gives us a next-word probability, how do we use this to generate naturally sounding text? This tutorial explores a bunch of options, using the GPT-2 distribution provided by 🤗's transformer package. ... greedy sampling. beam search. top-\(k\) sampling.
• PDF Comparison of Diverse Decoding Methods from Conditional Language Models — found that beam search is an effective strategy to heuristically sample sufficiently likely sequences from these probabilistic models (Sutskever et al., 2014). However, for more open-ended tasks, beam search is ill-suited to generating a set of diverse candidate sequences; this is because candidates Beam Search A bus is stopped at a bus stop.
• Leveraging transferability and improved beam search in textual ... — In the NLP field, existing methods of black-box adversarial attacks can be roughly divided into two categories: 1) Query-based attacks find vulnerable tokens by querying the output decisions and scores of the target model, and then apply different strategies to these tokens to generate adversarial texts; 2) transfer-based attacks utilize a surrogate model to approximate the decision boundary ...
• ai开发 - 鹅厂专家讲透AI文本生成解码策略与代码实现 - 腾讯云技术社区 - SegmentFault 思否 — 腾小云导读. 本文以 huggingface-transformers 的文本生成解码代码为例，对文本生成常用的五种解码策略 greedy search、beam search、sample、sample and rank & beam sample、group beam search 进行逐行解读。
• 鹅厂专家讲透ai文本生成解码策略与代码实现 - 腾讯云开发者社区-腾讯云 — 如上图所示，beam search 的候选路径有6条，group beam search 将这6条候选路径两两作为一组，分为三组。 每一步都在各组内的词表空间下去取 top-2 的结果作为当前预测的 token，对于当前组来说，通过 对之前组已生成的 token 进行惩罚 ，来保证当前组生成的 token 与 ...
• 10.8. Beam Search — Dive into Deep Learning 1.0.3 documentation - D2L — In Section 10.7, we introduced the encoder-decoder architecture, and the standard techniques for training them end-to-end.However, when it came to test-time prediction, we mentioned only the greedy strategy, where we select at each time step the token given the highest predicted probability of coming next, until, at some time step, we find that we have predicted the special end-of-sequence ...
• Decoding Methods in Neural Language Generation: A Survey - MDPI — Neural encoder-decoder models for language generation can be trained to predict words directly from linguistic or non-linguistic inputs. When generating with these so-called end-to-end models, however, the NLG system needs an additional decoding procedure that determines the output sequence, given the infinite search space over potential sequences that could be generated with the given vocabulary.