Approximation
and Optimization Neural Networks
An Approximation based formulas can be viewed as an approximation regression
for the trained data set. The training data are given in input/teacher
pairs, denoted as 
, where M
is the number of training pairs. The desired values at the output nodes
corresponding to the input patterns 
are assigned as a teacher's values. The objective of the network training
is to find the optimal weights to minimize the error between the teacher
value and the actual response. A popular criterion is the minimum-squares
error between the teacher and the actual response. To acquire a more versatile
nonlinear approximation capability, multilayer networks (together with
the back-propagation learning rule) are
usually adopted.
The model function is a function of inputs and weights: 
,
assuming there is a single output. In the basic approximation-based formulation,
the training procedure involves finding the weights to minimized the least-squares-error
energy function: 
. The weight vector
w can be trained by minimizing the energy function along the gradient
descent direction:
-
In the retrieving phase, the "winner" that wins the recognition is the
output node that yields the maximum response to the input pattern.
In the optimization Based Formulation, the energy function 
must be specifically chosen for the application. Again the weight vector
w can be trained by a gradient-descent updating rule:
-
For example, the 9maximum) likelihood function 
is a popular criteria function for stochastic neural networks. Therefore,
-
where 
denotes the training set for the i-th
class. In the retrieveing phase, the class with 
corresponding to the test pattern
x will be declared winner.
ADALINE
Contents
Artificial Neural Networks
About this Tutorial