Transducer models The Transducer elegantly solves both problems associated with CTC, while retaining some of its advantages over attention models.

Loren Lugosch Publications Teaching Blog Posts Some Favorites Loren Lugosch Follow Montreal, QC, Canada Twitter LinkedIn Google Scholar Sequence-to-sequence learning with Transducers Published: November 16, 2020 The Transducer (sometimes called the RNN Transducer or RNN-T, though it need not use RNNs) is a sequence-to-sequence model proposed by Alex Graves in Sequence Transduction with Recurrent Neural Networks.

The paper was published at the ICML 2012 Workshop on Representation Learning.

Graves showed that the Transducer was a sensible model to use for speech recognition, achieving good results on a small dataset (TIMIT).

Since then, the Transducer hasnt been used as much compared to CTC models (like Deep Speech 2 ) or attention models (like Listen, Attend, and Spell ).

Last year, however, the Transducer got some serious attention when Google researchers showed that it could enable entirely on-device low-latency speech recognition for Pixel phones.

And more recently, the Transducer was used to achieve a new state-of-the-art word error rate for the LibriSpeech benchmark.

So what is the Transducer, and when might you want to use it In this post, we will see where Transducer models fit in with other sequence-to-sequence models and a detailed explanation of how they work.

This post also includes a Colab notebook with a PyTorch implementation of the Transducer for a toy problemwhich you can skip straight to here.

Attention models The problems were interested in here are sequence transduction problems, where the goal is to map an input sequence mathbfx x1, x2, dots xT to an output sequence mathbfy y1, y2, dots, yU.

The go-to models for sequence transduction problems are attention-based sequence-to-sequence models, like RNN encoder-decoder models or Transformers.

Heres a diagram of an attention model.

In the diagrams below, Ill use red to indicate that a module has access to mathbfx, blue to indicate access to mathbfy, and purple to indicate access to both mathbfx and mathbfy.) -- The model encodes the input mathbfx into a sequence of feature vectors, then computes the probability of the next output yu as a function of the encoded input and previous outputs.

The attention mechanism allows the decoder to look at different parts of the input sequence when predicting each output.

Attention models can be applied to any problem, but they are not always the best choice for certain problems, like speech recognition, for a few reasons 2: The attention operation is expensive for long input sequences.

The complexity of attending to the entire input for every output is O(TU)and for audio, T and U are big.

Attention models cannot be run online (in real time), since the entire input sequence needs to be available before the decoder can attend to it.

Attention models also dont take advantage of the fact that, for speech recognition, the alignment between inputs and outputs is monotonic: that is, if word A comes after word B in the transcript, word A must come after word B in the audio signal (see image below, from Chan et al., for an example of a monotonic alignment).

The fact that attention models lack this inductive bias seems to make them harder to train for speech recognition; its common to add auxiliary loss terms to stabilize training.

This leads us to Connectionist Temporal Classification (CTC) models, which are more suitable for some problems than attention models.

CTC models CTC models assume that there is a monotonic input-output alignment 3.

This ends up making the model a lot simpler.

So simple We only need a single neural net to implement a CTC model, and no expensive global attention mechanism.

But CTC models have a couple problems of their own: Problem 1: The output sequence length U has to be smaller than the input sequence length T.

This might not seem like a problem for speech recognition, where T is much larger than Ubut it prevents us from using a model architecture that does a lot of pooling, which can make the model a lot faster.

Problem 2: The outputs are assumed to be independent of each other.

The result is that CTC models often produce outputs that are obviously wrong, like I eight food instead of I ate food.

Getting good results with CTC usually requires a search algorithm that incorporates a secondary language model.

Sequent Learning