The simplest approximation formulation is one based on the linear least-squares-error
criterion. Indeed, one of the pioneering neural models is a linear network,
named ADALINE (ADAptive LINear Element), proposed by Widrow. As tou can
see in this figure, ADALINE is a linear
single layer network with a net value
where is the number of inputs nodes.
Without los of generality, it is more convenient to regard the threshold
just as an extra weight, that is, .
The net value can be rewritten as
where and .
Note that z is the augmented pattern x.
The weight vector w can be obtained by minimizing the least-squares-erros
The delta learning rule adopted in ADALINE is a data-adaptive technique
for deriving a least-squares-error solution. It is based on an iterative
gradient-descent algorithm. The gradient is defined as the first partial
derivative of with respect to :
In a vector form:
In order to minimize E, the weights should be updated in the direction
opposite to that of the gradient, that is,
Nonlinear Multilayer Backpropagation Networks
About this Tutorial
Artificial Neural Networks