Neural Network Toolbox

Early Stopping

Another method for improving generalization is called early stopping. In this technique the available data is divided into three subsets. The first subset is the training set, which is used for computing the gradient and updating the network weights and biases. The second subset is the validation set. The error on the validation set is monitored during the training process. The validation error will normally decrease during the initial phase of training, as does the training set error. However, when the network begins to overfit the data, the error on the validation set will typically begin to rise. When the validation error increases for a specified number of iterations, the training is stopped, and the weights and biases at the minimum of the validation error are returned.

The test set error is not used during the training, but it is used to compare different models. It is also useful to plot the test set error during the training process. If the error in the test set reaches a minimum at a significantly different iteration number than the validation set error, this may indicate a poor division of the data set.

Early stopping can be used with any of the training functions that were described earlier in this chapter. You simply need to pass the validation data to the training function. The following sequence of commands demonstrates how to use the early stopping function.

First we create a simple test problem. For our training set we generate a noisy sine wave with input points ranging from -1 to 1 at steps of 0.05.

• p = [-1:0.05:1];
  t = sin(2*pi*p)+0.1*randn(size(p));
  

Next we generate the validation set. The inputs range from -1 to 1, as in the test set, but we offset them slightly. To make the problem more realistic, we also add a different noise sequence to the underlying sine wave. Notice that the validation set is contained in a structure that contains both the inputs and the targets.

• val.P = [-0.975:.05:0.975];
  val.T = sin(2*pi*v.P)+0.1*randn(size(v.P));
  

We now create a 1-20-1 network, as in our previous example with regularization, and train it. (Notice that the validation structure is passed to train after the initial input and layer conditions, which are null vectors in this case since the network contains no delays. Also, in this example we are not using a test set. The test set structure would be the next argument in the call to train.) For this example we use the training function traingdx, although early stopping can be used with any of the other training functions we have discussed in this chapter.

• net=newff([-1 1],[20,1],{'tansig','purelin'},'traingdx');
  net.trainParam.show = 25;
  net.trainParam.epochs = 300;
  net = init(net);
  [net,tr]=train(net,p,t,[],[],val);
  TRAINGDX, Epoch 0/300, MSE 9.39342/0, Gradient 17.7789/1e-06
  TRAINGDX, Epoch 25/300, MSE 0.312465/0, Gradient 0.873551/1e-06
  TRAINGDX, Epoch 50/300, MSE 0.102526/0, Gradient 0.206456/1e-06
  TRAINGDX, Epoch 75/300, MSE 0.0459503/0, Gradient 0.0954717/1e-06
  TRAINGDX, Epoch 100/300, MSE 0.015725/0, Gradient 0.0299898/1e-06
  TRAINGDX, Epoch 125/300, MSE 0.00628898/0, Gradient 
  0.042467/1e-06
  TRAINGDX, Epoch 131/300, MSE 0.00650734/0, Gradient 
  0.133314/1e-06
  TRAINGDX, Validation stop.
  

The following figure shows a graph of the network response. We can see that the network did not overfit the data, as in the earlier example, although the response is not extremely smooth, as when using regularization. This is characteristic of early stopping.

Regularization

Summary and Discussion

© 1994-2005 The MathWorks, Inc.