The back-error propagation module

In this paragraph the back-error propagation panel is described. The panel will open when the item BACK PROPAGATION in the MODEL sub-menu is chosen.

Back-error propagation: an overview

The back-error propagation is one of the most popular neural network paradigm, and it is used successfully in many applications in different areas, that require pattern matching: given a pattern in input, the network produces a pattern in response. It is also one of the easiest to understand, because its learning and update procedure is intuitive: if the network gives a wrong answer the weight are corrected so that the error is lessened so that future responses are more likely to be correct. Back propagation network show different layers of nodes; each layer is fully connected to the layer above. When an input is fed into the net, the updating rule force the propagating of activation values forward from the input layer to the output nodes, which provide the network response. When the network corrects its internal internal parameters, the correction starts from the output nodes backward through each layer to the input one.

It was first introduced by Webos in 1974, and independently by Parker in 1982. It became well known only after Rumelhart and McClelland famous book Parallel Distributed Processing in 1986. It was successfully used by Sejnowski and Rosenberg in their program, in which a network was trained to learn the phonetics rules. The back-error propagation has been a big improvement from the perceptron, or other early adaptive networks, such as the adaline (adaptive linear neuron) or the madaline architecture (multilayer adaline), which was proved by Minsky and Papert to be very limited: the perceptron could only classify patterns that are linearly separable. This limitation is inherently tied to the fact that it could only have two layers of units and only one could change weights. On the opposite the back-propagation can have two or more layers of units, with addaptive weights. The network is trained, that is it is presented with a series of patterns pairs. Each pair consists of an input patterns and an output pattern, which is the desired response of the network to the input and is used to determine the error values in the network when the weights are adjusted. When the network is trained with success, it produces correct answers more and more often as the training session goes on. The root-mean-squared (RMS) error is computed to quantify the learning rate. As the learning process goes on, its RMS error decreases - less than 0.1 would mean that the network has learned its training set.

The network response to an input is not a yes/no answer. At each step of the learning phase the network goes closer and closer to the desired response. This process is called the convergence process of the network, where the RMS value gets closer to zero. But this does not happen always, sometimes the network does not seem to be learning any more. If one think of RMS values as functions of the weights, looking for RMS values getting closer to zero means looking for minimum points. At times, instead of reaching a global point of minimum in the RMS value, a local minimum is found and the learning phase is stuck. The standard back-error propagation does not provide means to get around this problem, or better to get out of local minima. Nevertheless it may try not to enter into a local minimum with a number of techniques. After the network has learnt, it is usually testes to assess its performance.

The back-error propagation panel

In the panel there are some general buttons: DISCONNECT, CONNECT, RESET, EXIT, STEP and LEARN. Moreover there are three sliders to tune up some parameters, asociated to the simulation (the first two) and to the visualization (the third one): eta, the number of learning steps and the video refresh frequency. A message advise users that no training set is loaded, and two buttons are dedicated to loading a training file or to create a new one. Finally some labels display the error, and the steps in the learning process already done, and a small graphical window displays the error trend. Now we will describe the actions associated to every button in detail.

CONNECT: Causes the association between the network presently displayed in the main window with the back-propagation model. During this action the network is tested to see if its structure is compatible with the back-propagation achitecture. You can not load another network, not even change the framework of the current one, while there's one connected. You have to use the DISCONNECT button first.
DISCONNECT: Disconnects the network from the model. Only after this action, it is possible to load another network file in the main window, change the framework of the current net adding/deleting arcs and neurons, or quitting the panel to leave the model.
RESET: Resets the network neuron states and arc weights to the default values; it is a shortcut for the EDIT menu item RESET NET.
EXIT: Exits from this model, deleting the panel.
LEARN: Performs the specified number of steps of the learning phase on the training set currently loaded. The number of steps to be performed are set by moving the slide, labelled Steps, with the mouse pointer, holding the left button pressed. During the learning phase, the edit area will be refreshesd with a frequency regulated by the slider, labelled Refresh freq..
STEP: Executes one complete step of computation, showing the net response to an input submission.
LOAD: Loads a training set file, the format of which will be described below.
EDIT: Runs a text editor to create a training set file without exiting inne application. However the file created will not be loaded automatically. Only after the new training file is saved (by default in the sub-directory TS), you can load it, as any other training file, with the LOAD button.

Back to index