Neural Network Components

Neural network architectures are constructed using different neural network building blocks. In this section, we discuss various such building blocks that are relevant to NLP and our tasks.

5After speech transcription, if necessary.

2.2 Neural networks for sequence processing Token representations

Natural language consists of discrete elements, such as words, which, as previously discussed, can be decomposed into sub-word units. In order to use such discrete elements in a model, we need to find vector representations for the tokens.

A simple vector representation for tokens that has long been in use is theone-hotvector representation.

This representation can be formalized as a function 𝑓

one−hot:V ↦→N₀^{| V |}that maps any token𝑤from the vocabularyVonto a vector𝑣of the size of the vocabulary, such that only the element in the vector corresponding to the token𝑤is equal to one and the others are equal to zero:

𝑣𝑖 = (

1, if𝑖=id(𝑤) 0, otherwise

, (2.2)

where the function id(𝑤):V ↦→N₀maps the word𝑤to its unique integer id.

Since the one-hot representation do not scale well in practice, and the distance between two words is equal for any two words, tokens are usually represented asdensevectors of relatively low dimensionality. The token embedding function is the a function 𝑓

emb:V ↦→R

𝑑

, where 𝑑is some chosen dimension for the token embedding space. Note that such a dense embedding allows the vector representations of different words to be arbitrarily close to each other, which is useful for handling synonyms. In addition, the dimensionality of the vectors can be kept low and does not directly depend on the vocabulary size. The function 𝑓

embcan be implemented as:

𝑓emb(𝑤) =𝐴·𝑣

𝑤, (2.3)

where𝑣

𝑤 ∈N₀^{| V |}is a one-hot vector for word𝑤and𝐴∈R

𝑑× | V |

is a trainable matrix containing the vectors for every word inV.

In the earlier days of deep learning for NLP, several fundamental works have investigated different ways of pre-training word vectors using large unsupervised corpora. Among the most commonly used ones are Word2Vec [39] and GloVe [40]. These methods provide initializations for word vector parameters that capture synonymity and other relations between words, induced from statistics on a large text corpus (see2.4.1). Essentially, similar words obtain similar embeddings, which improves generalization to unobserved words. While recent focus in NLP has shifted towards pretrained transformers (see Section2.4.2), pretrained word embeddings are still used in applications due to their simplicity and efficiency.

Recurrent Neural Networks

Natural languages, as well as formal languages, are sequential in nature, which means that we require models that can process and generalize for different sequence lengths. Recurrent neural networks (RNN) are a type of neural networks suitable for this case. RNNs typically process the sequence one token at a time, and use a “state” variable/vector that describes the sequence elements “consumed” so far. Thus, the typical RNN is a parameterized function 𝑓 :R

𝑑𝑖 ×R

𝑑ℎ ↦→R

𝑑ℎ

that computes a new stateℎ

𝑡 ∈R

𝑑

ℎ from the previous stateℎ

𝑡−1 ∈R

𝑑

ℎ and current input𝑥

𝑡 ∈R

𝑑 𝑖: ℎ𝑡 = 𝑓(ℎ_𝑡−

1, 𝑥

𝑡;𝜃) (2.4)

Chapter 2 Background and Preliminaries

x_t h_t-1

U

W U_r

U_z W_r W_z

h_t

Figure 2.1: Gated Recurrent Unit (GRU)

RNNs usually share their parameters across time steps to improve generalization.

The most basic RNN can be implemented as a single layer with an affine transformation and non-linearity:

ℎ𝑡 =tanh(𝑊 ℎ_𝑡−

1+𝑈 𝑥

𝑡+𝑏) , (2.5)

where𝑊 and𝑈are trainable linear transformations and𝑏is a trainable bias term.

However, it has been shown that such basic RNNs suffer from gradient problems [41–44] due to multiplicative updates and non-linearities in the backpropagation path to an early state. In practice, gated RNNs are used, such as the Long Short-Term Memory (LSTM [42]) unit or the Gated Recurrent Unit (GRU [45]). Both implementadditivegated state updates, which prevents the gradients from vanishing or exploding due to linear transformations and repeated non-linearities. The GRU is defined according to the following equations and is illustrated in Figure2.1:

z𝑡 =𝜎(𝑊

𝑧x𝑡+𝑈

𝑧h𝑡−1+b𝑧) (2.6)

r𝑡 =𝜎(𝑊

𝑟x𝑡+𝑈

𝑟h𝑡−1+b𝑟) (2.7)

hˆ𝑡 =tanh(𝑊x𝑡+𝑈(h𝑡−1⊙r𝑡) +bℎ) (2.8) h𝑡 =(1−z𝑡) ∗h𝑡−1+z𝑡 ∗hˆ𝑡 , (2.9) where the𝑊

×and𝑈

×are trainable matrices and𝑏

×are trainable bias terms.

The current hidden stateh𝑡 at time step𝑡of the RNN (which is also its output at time𝑡) is computed by interpolating between the stateh_𝑡−

1at previous time step and the candidate state ˆh𝑡(Equation2.9), withz𝑡the update vector and·the element-wise vector product. Theupdate gatez𝑡for the interpolation is computed using the current inputx𝑡 and the previous stateh𝑡−1 (Equation2.6), where𝑊

𝑧 and 𝑈𝑧 are parameter matrices to be learned during training and𝜎 is the sigmoid activation function 𝜎(𝑥) = 1/(1+𝑒⁻

𝑥) applied element-wise to the vector entries. The current candidate state ˆh𝑡 is computed based on the current inputx𝑡and previous stateh𝑡−1(Equation2.8), whereWandUare parameter matrices,tanhis the hyperbolic tangent activation function andr𝑡 is the value of thereset gate, computed as in Equation2.7, with parameter matricesW𝑟 andU𝑟. A schematic representation of the GRU can be found in Figure2.1.

The advantage of using gated units such as GRU or long short-term memory (LSTM) [42] is their ability to better process longer sequences, which arises from their additive manipulation of the state vector and explicit filtering using gates. In the case of the GRU, the reset gater𝑡 determines which parts of the previous stateh_𝑡−

1are “ignored” in the computation of the candidate state and the update

2.2 Neural networks for sequence processing gatez𝑡 determines how much of the previous state is “leaked” into the current stateh𝑡. The update gate could decide to forget the previous state altogether or to simply copy the previous state and ignore the current input. Both gates are parameterized (and thus trainable) and their values depend on both the inputx𝑡and the previous state.

The LSTM is similar but uses two “state” variablesc𝑡andy𝑡, wherec𝑡is the “cell state” andy𝑡 is the previous output state. The LSTM is defined according to the following equations:

i𝑡 =𝜎(𝑊

𝑖x𝑡+𝑈

𝑖y𝑡−1+b𝑖) (2.10)

f𝑡 =𝜎(𝑊

𝑓x𝑡 +𝑈

𝑓y𝑡−1+b𝑓) (2.11)

o𝑡 =𝜎(𝑊

𝑜x𝑡+𝑈

𝑜y_𝑡−1+b𝑜) (2.12)

cˆ𝑡 =tanh(𝑊

𝑐x𝑡 +𝑈

𝑐y𝑡−1+b𝑐) (2.13)

c𝑡 =f𝑡 ⊙c𝑡−1+i𝑡⊙cˆ𝑡 (2.14)

y𝑡 =o𝑡⊙tanh(c𝑡) , (2.15)

where the𝑊

×and𝑈

×are trainable matrices andb_×are trainable bias terms.

Note that in both, there is a mechanism for the network to “forget” parts of the previous state. Also, in both cases, state updates are additive, which allows the gradient to be backpropagated while being only element-wise multiplied with the forget gate vectors.

Convolutional Neural Networks

While typically, RNNs and Transformers are used, convolutional neural networks (CNNs) can also be applied on text. A 1D CNN is parameterized using a tensor𝑊 ∈R^{𝑙×𝑑×ℎ}, where𝑙is the window size or kernel size,𝑑is the input dimension andℎthe output dimension. The output is computed by sliding the weights𝑊over the length dimension of the input (the input is shaped asR

𝑠×𝑑

, multiplying every 𝑙×𝑑window from the input with everyℎ-slice of size𝑙×𝑑of𝑊using element-wise multiplication, and summing up. The result is inR^𝑠×ℎ. Additionally, CNNs use non-linearities, where a ReLU is a standard choice, as well as local (and global) pooling.

Attention

Originally, attention was proposed as a way to produce a latent alignment for neural machine translation [46,47]. Since then, it has been a crucial component in many NLP applications and is the primary mechanism of context integration in transformers.

Most generally, given some input sequence𝑋 of length𝐿of vectorsx𝑖 ∈R

𝑑

and aqueryvector q∈R^ℎ, attention is a function that comparesqto everyx𝑖and builds a weighted sum ofx𝑖that can be trained to contain most relevant information from𝑋 pertaining to the given queryq. We can describe

Chapter 2 Background and Preliminaries

the attention mechanism using the following equations:

𝑎𝑖=comp(x𝑖,q) (2.16)

𝛼𝑖= 𝑒

𝑎 𝑖

Í^𝐿

𝑖=0𝑒

𝑎𝑖 (2.17)

v=

𝐿

∑︁

𝑖=0

𝛼𝑖·x𝑖 , (2.18)

where comp(·):R

𝑑×R

ℎ↦→Ris a function comparing two vectors.

One variant of attention uses a feedforward network on the concatenation ofx𝑖 andq[46]:

comp(x,q) =w^Ttanh(𝑊x+𝑈q) (2.19) Another commonly used variant, usually calledmultiplicativeattention, simply uses the dot product as comp(·):

comp(x,q)=q^Tx . (2.20)

The query𝑞and key vectors𝑥can also be projected before the dot product:

comp(x,q) =(𝑊q)^T(𝑈x) , (2.21)

where𝑊 and𝑈are trainable matrices.

Im Dokument Question Answering over Knowledge Graphs (Seite 34-38)