**Neural Network Tutorials - Herong's Tutorial Examples** - 1.20, by Dr. Herong Yang

RNN Recursive Function

This section provides a quick introduction of RNN (Recurrent Neural Network). It starts from a generic neural network hidden layer, and slowly converts it into a RNN layer by combining the weighted average with the activation function into a recursive function to manage a feed from one sample to the next sample recursively.

From previous section, we learn that a RNN layer requires a recursive function R() that takes two inputs and generate two outputs. One output, called y, goes to the layer and the other output, called s, stays in the same layer and is used as an input the next sample.

In this section, we will learn how to construct the recursive function R().

A simple option to construct the recursive function R() is to set the state vector, s_{t},
to be the same as the output vector, y_{t}. And use an activation function
as the recursive function with input extended to take both inputs x_{t} and s_{t-1}:

Generic form: (y_{t}, s_{t}) = R(x_{t}, W_{t}, s_{t-1}, U_{t}) Simplified form: y_{t}= f(W_{t}·x_{t}+ U_{t}·s_{t-1}) s_{t}= y_{t}Or: y_{t}= f(W_{t}·x_{t}+ U_{t}·y_{t-1}) Or: y_{t}= f( |W_{t}, U_{t}| · |x_{t }| )_{ }(_{ }_{ }|y_{t-1}| ) f() represents the activation function, same as traditional neural networks. · represents the dot operation of a matrix and a vector.

The above recursive function become different variations with different activation functions:

f() = sigmoid(): y_{t}= sigmoid(W_{t}·x_{t}+ U_{t}·y_{t-1}) f() = tanh(): y_{t}= tanh(W_{t}·x_{t}+ U_{t}·y_{t-1}) f() = ReLU(): y_{t}= ReLU(W_{t}·x_{t}+ U_{t}·y_{t-1}) ...

If this simplified recursive function model is used, a RNN cell (actually a RNN layer) can be illustrated in different diagrams. Here are some examples I have collected from the Internet.

Table of Contents

Deep Playground for Classical Neural Networks

Building Neural Networks with Python

Simple Example of Neural Networks

TensorFlow - Machine Learning Platform

PyTorch - Machine Learning Platform

CNN (Convolutional Neural Network)

►RNN (Recurrent Neural Network)

What Is RNN (Recurrent Neural Network)

What Is LSTM (Long Short-Term Memory)