Network Definition :: Advanced Topics (Neural Network Toolbox)

Neural Network Toolbox

Network Definition

The first step is to create a new network. Type in the following code to create a network and view its many properties.

```
net = network
```

Architecture Properties

The first group of properties displayed are labeled architecture properties. These properties allow you to select the number of inputs and layers, and their connections.

Number of Inputs and Layers.. The first two properties displayed are numInputs and numLayers. These properties allow us to select how many inputs and layers we want our network to have.

net =
    Neural Network object:
    architecture:
         numInputs: 0
         numLayers: 0

Note that the network has no inputs or layers at this time.

Change that by setting these properties to the number of inputs and number of layers in our custom network diagram.

```
net.numInputs = 2;
net.numLayers = 3;
```

Note that net.numInputs is the number of input sources, not the number of elements in an input vector (net.inputs{i}.size).

Bias Connections. Type net and press Return to view its properties again. The network now has two inputs and three layers.

net =
    Neural Network object:
    architecture:
         numInputs: 2
         numLayers: 3

Now look at the next five properties.

       biasConnect: [0; 0; 0]
      inputConnect: [0 0; 0 0; 0 0]
      layerConnect: [0 0 0; 0 0 0; 0 0 0]
     outputConnect: [0 0 0]
     targetConnect: [0 0 0]

These matrices of 1's and 0's represent the presence or absence of bias, input weight, layer weight, output, and target connections. They are currently all zeros, indicating that the network does not have any such connections.

Note that the bias connection matrix is a 3-by-1 vector. To create a bias connection to the ith layer you can set net.biasConnect(i) to 1. Specify that the first and third layers are to have bias connections, as our diagram indicates, by typing in the following code.

net.biasConnect(1) = 1;
net.biasConnect(3) = 1;

Note that you could also define those connections with a single line of code.

```
net.biasConnect = [1; 0; 1];
```

Input and Layer Weight Connections. The input connection matrix is 3-by-2, representing the presence of connections from two sources (the two inputs) to three destinations (the three layers). Thus, net.inputConnect(i,j) represents the presence of an input weight connection going to the ith layer from the jth input.

To connect the first input to the first and second layers, and the second input to the second layer (as is indicated by the custom network diagram), type

net.inputConnect(1,1) = 1;
net.inputConnect(2,1) = 1;
net.inputConnect(2,2) = 1;

or this single line of code:

```
net.inputConnect = [1 0; 1 1; 0 0];
```

Similarly, net.layerConnect(i.j) represents the presence of a layer-weight connection going to the ith layer from the jth layer. Connect layers 1, 2, and 3 to layer 3 as follows.

net.layerConnect = [0 0 0; 0 0 0; 1 1 1];

Output and Target Connections. Both the output and target connection matrices are 1-by-3 matrices, indicating that they connect to one destination (the external world) from three sources (the three layers).

To connect layers 2 and 3 to network outputs, type

```
net.outputConnect = [0 1 1];
```

To give layer 3 a target connection, type

```
net.targetConnect = [0 0 1];
```

The layer 3 target is compared to the output of layer 3 to generate an error for use when measuring the performance of the network, or when updating the network during training or adaption.

Number of Outputs and Targets

Type net and press Enter to view the updated properties. The final four architecture properties are read-only values, which means their values are determined by the choices we make for other properties. The first two read-only properties have the following values.

numOutputs: 2  (read-only)
numTargets: 1  (read-only)

By defining output connections from layers 2 and 3, and a target connection from layer 3, you specify that the network has two outputs and one target.

Subobject Properties

The next group of properties is

subobject structures:
            inputs: {2x1 cell} of inputs
            layers: {3x1 cell} of layers
           outputs: {1x3 cell} containing 2 outputs
           targets: {1x3 cell} containing 1 target
            biases: {3x1 cell} containing 2 biases
      inputWeights: {3x2 cell} containing 3 input weights
      layerWeights: {3x3 cell} containing 3 layer weights

Inputs

When you set the number of inputs (net.numInputs) to 2, the inputs property becomes a cell array of two input structures. Each ith input structure (net.inputs{i}) contains addition properties associated with the ith input.

To see how the input structures are arranged, type

net.inputs
ans = 
    [1x1 struct]
    [1x1 struct]

To see the properties associated with the first input, type

```
net.inputs{1}
```

The properties appear as follows.

ans = 
       range: [0 1]
        size: 1
    userdata: [1x1 struct]

Note that the range property only has one row. This indicates that the input has only one element, which varies from 0 to 1. The size property also indicates that this input has just one element.

The first input vector of the custom network is to have two elements ranging from 0 to 10. Specify this by altering the range property of the first input as follows.

```
net.inputs{1}.range = [0 10; 0 10];
```

If we examine the first input's structure again, we see that it now has the correct size, which was inferred from the new range values.

ans = 
       range: [2x2 double]
        size: 2
    userdata: [1x1 struct]

Set the second input vector ranges to be from -2 to 2 for five elements as follows.

net.inputs{2}.range = [-2 2; -2 2; -2 2; -2 2; -2 2];

Layers. When we set the number of layers (net.numLayers) to 3, the layers property becomes a cell array of three-layer structures. Type the following line of code to see the properties associated with the first layer.

net.layers{1}
ans = 
     dimensions: 1
    distanceFcn: 'dist'
      distances: 0
        initFcn: 'initwb'
    netInputFcn: 'netsum'
      positions: 0
           size: 1
    topologyFcn: 'hextop'
    transferFcn: 'purelin'
       userdata: [1x1 struct]

Type the following three lines of code to change the first layer's size to 4 neurons, its transfer function to tansig, and its initialization function to the Nguyen-Widrow function as required for the custom network diagram.

net.layers{1}.size = 4;
net.layers{1}.transferFcn = 'tansig';
net.layers{1}.initFcn = 'initnw';

The second layer is to have three neurons, the logsig transfer function, and be initialized with initnw. Thus, set the second layer's properties to the desired values as follows.

net.layers{2}.size = 3;
net.layers{2}.transferFcn = 'logsig';
net.layers{2}.initFcn = 'initnw';

The third layer's size and transfer function properties don't need to be changed since the defaults match those shown in the network diagram. You only need to set its initialization function as follows.

```
net.layers{3}.initFcn = 'initnw';
```

Output and Targets. Take a look at how the outputs property is arranged with this line of code.

net.outputs
ans = 
     []    [1x1 struct]    [1x1 struct]

Note that outputs contains two output structures, one for layer 2 and one for layer 3. This arrangement occurs automatically when net.outputConnect was set to [0 1 1].

View the second layer's output structure with the following expression.

net.outputs{2}
ans = 
        size: 3
    userdata: [1x1 struct]

The size is automatically set to 3 when the second layer's size (net.layers{2}.size) is set to that value. Take a look at the third layer's output structure if you want to verify that it also has the correct size.

Similarly, targets contains one structure representing the third layer's target. Type these two lines of code to see how targets is arranged and to view the third layer's target properties.

net.targets
ans = 
     []    []    [1x1 struct]
net.targets{3}
ans = 
        size: 1
    userdata: [1x1 struct]

Biases, Input Weights, and Layer Weights.. Enter the following lines of code to see how bias and weight structures are arranged.

net.biases
net.inputWeights
net.layerWeights

Here are the results for typing net.biases.

ans = 
    [1x1 struct]
              []
    [1x1 struct]

If you examine the results you will note that each contains a structure where the corresponding connections (net.biasConnect, net.inputConnect, and net.layerConnect) contain a 1.

Take a look at their structures with these lines of code.

net.biases{1}
net.biases{3}
net.inputWeights{1,1}
net.inputWeights{2,1}
net.inputWeights{2,2}
net.layerWeights{3,1}
net.layerWeights{3,2}
net.layerWeights{3,3}

For example, typing net.biases{1} results in the following output.

ans = 
       initFcn: ''
         learn: 1
      learnFcn: ''
    learnParam: ''
          size: 4
      userdata: [1x1 struct]

Specify the weights tap delay lines in accordance with the network diagram, by setting each weight's delays property.

net.inputWeights{2,1}.delays = [0 1];
net.inputWeights{2,2}.delays = 1;
net.layerWeights{3,3}.delays = 1;

Network Functions

Type net and press Return again to see the next set of properties.

  functions:
          adaptFcn: (none)
           initFcn: (none)
        performFcn: (none)
          trainFcn: (none)

Each of these properties defines a function for a basic network operation.

Set the initialization function to initlay so the network initializes itself according to the layer initialization functions that we have already set to initnw the Nguyen-Widrow initialization function.

```
net.initFcn = 'initlay';
```

This meets the initialization requirement of our network.

Set the performance function to mse (mean squared error) and the training function to trainlm (Levenberg-Marquardt backpropagation) to meet the final requirement of the custom network.

net.performFcn = 'mse';
net.trainFcn = 'trainlm';

Weight and Bias Values

Before initializing and training the network, take a look at the final group of network properties (aside from the userdata property).

weight and bias values:
                IW: {3x2 cell} containing 3 input weight matrices
                LW: {3x3 cell} containing 3 layer weight matrices
                 b: {3x1 cell} containing 2 bias vectors

These cell arrays contain weight matrices and bias vectors in the same positions that the connection properties (net.inputConnect, net.layerConnect, net.biasConnect) contain 1's and the subobject properties (net.inputWeights, net.layerWeights, net.biases) contain structures.

Evaluating each of the following lines of code reveals that all the bias vectors and weight matrices are set to zeros.

net.IW{1,1}, net.IW{2,1}, net.IW{2,2}
net.IW{3,1}, net.LW{3,2}, net.LW{3,3}
net.b{1}, net.b{3}

Each input weight net.IW{i,j}, layer weight net.LW{i,j}, and bias vector net.b{i} has as many rows as the size of the ith layer (net.layers{i}.size).

Each input weight net.IW{i,j} has as many columns as the size of the jth input (net.inputs{j}.size) multiplied by the number of its delay values (length(net.inputWeights{i,j}.delays)).

Likewise, each layer weight has as many columns as the size of the jth layer (net.layers{j}.size) multiplied by the number of its delay values (length(net.layerWeights{i,j}.delays)).

Custom Network Network Behavior