How text generation works?

How text generation works?

Text generation usually involves the following steps:

1. Importing Dependencies.
2. Loading of Data.
3. Creating Character/Word mappings.
4. Data Preprocessing.
5. Modelling.
6. Generating text.

Why is text generation important?

Text generation is a subfield of natural language processing. It leverages knowledge in computational linguistics and artificial intelligence to automatically generate natural language texts, which can satisfy certain communicative requirements.

What is neural text generation?

Neural text generation models conditioning on given input (e.g. machine translation and image captioning) are usually trained by maximum likelihood estimation of target text. However, the trained models suffer from various types of errors at inference time.

What do you need to know about text generation?

Text generation is the task of generating text with the goal of appearing indistinguishable to human-written text. Additionally, these models are typically trained via maxi- mum likelihood and teacher forcing.

How are neural models used for text generation?

Neural text generation models are often autoregressive language models or seq2seq models. These models generate text by sampling words sequentially, with each word conditioned on the previous word, and are state-of-the-art for several machine translation and summarization benchmarks.

Why are maximum likelihood trained models used for text generation?

The Google Brain team argues that maximum likelihood trained models that are widely used for generating text, can result in low quality of the resulting samples since text generation requires conditioning on sequences of words that were not necessarily present during training.

How to overcome sparse reward in text generation?

To overcome the issue of sparse reward in long text generation, Guo et al. propose a hierarchy design for a generator with a Manager and a Worker. A Manager module receives the information on the high-level feature extracted by the discriminator given the current generated word sequence.