Residual neural network

Residual connection In a multilayer neural network model, consider a (non-residual) subnetwork with a certain number of stacked layers (e.g., 2 or 3). Let H(x; \alpha) denote the subnetwork. Suppose H^* is the desired optimal output of this subnetwork. Residual learning simply adds x directly to the output, such that the optimal learned output now becomes be H^* - x, which is interpreted as a "residual" with respect to x. The operation of "adding x" is implemented via a "skip connection" that performs an identity mapping to connect the input of the subnetwork with its output. This connection is referred to as a "residual connection" in later work. Let F(x; \alpha) = H(x; a) + x. The function F is often represented by matrix multiplication interlaced with activation functions and normalization operations (e.g., batch normalization or layer normalization). As a whole, one of these subnetworks is referred to as a "residual block". In an LSTM without a forget gate, an input x_t is processed by a function F and added to a memory cell c_t, resulting in c_{t+1} = c_t + F(x_t). An LSTM with a forget gate essentially functions as a highway network. To stabilize the variance of the layers' inputs, it is recommended to replace the residual connections x + f(x) with x/L + f(x), where L is the total number of residual layers. Projection connection If the function F is of type F: \R^n \to \R^m where n \neq m, then F(x) + x is undefined. To handle this special case, a projection connection is used: y = F(x) + P(x) where P is typically a linear projection, defined by P(x) = Mx where M is a m \times n matrix. The matrix is trained via backpropagation, as is any other parameter of the model. Signal propagation The introduction of identity mappings facilitates signal propagation in both forward and backward paths. Forward propagation If the output of the \ell-th residual block is the input to the (\ell+1)-th residual block (assuming no activation function between blocks), then the (\ell+1)-th input is: x_{\ell+1} = F(x_{\ell}) + x_{\ell} Applying this formulation recursively, e.g.: \begin{align} x_{\ell+2} & = F(x_{\ell+1}) + x_{\ell+1} \\ & = F(x_{\ell+1}) + F(x_{\ell}) + x_{\ell} \end{align} yields the general relationship: x_{L} = x_{\ell} + \sum_{i=\ell}^{L-1} F(x_{i}) where L is the index of a residual block and \ell is the index of some earlier block. This formulation suggests that there is always a signal that is directly sent from a shallower block \ell to a deeper block L. Backward propagation The residual learning formulation provides the added benefit of mitigating the vanishing gradient problem to some extent. However, it is crucial to acknowledge that the vanishing gradient issue is not the root cause of the degradation problem, which is tackled through the use of normalization. To observe the effect of residual blocks on backpropagation, consider the partial derivative of a loss function \mathcal{E} with respect to some residual block input x_{\ell}. Using the equation above from forward propagation for a later residual block L>\ell: \begin{align} \frac{\partial \mathcal{E} }{\partial x_{\ell} } & = \frac{\partial \mathcal{E} }{\partial x_{L} }\frac{\partial x_{L} }{\partial x_{\ell} } \\ & = \frac{\partial \mathcal{E} }{\partial x_{L} } \left( 1 + \frac{\partial }{\partial x_{\ell} } \sum_{i=\ell}^{L-1} F(x_{i}) \right) \\ & = \frac{\partial \mathcal{E} }{\partial x_{L} } + \frac{\partial \mathcal{E} }{\partial x_{L} } \frac{\partial }{\partial x_{\ell} } \sum_{i=\ell}^{L-1} F(x_{i}) \end{align} This formulation suggests that the gradient computation of a shallower layer, \frac{\partial \mathcal{E} }{\partial x_{\ell} }, always has a later term \frac{\partial \mathcal{E} }{\partial x_{L} } that is directly added. Even if the gradients of the F(x_{i}) terms are small, the total gradient \frac{\partial \mathcal{E} }{\partial x_{\ell} } resists vanishing due to the added term \frac{\partial \mathcal{E} }{\partial x_{L} }. == Variants of residual blocks ==