pnd |
Application for training and testing Parallel Non linear Dichotomizers (PND) ensembles.
The application pnd implements Parallel Non linear Dichotomizers (PND) ensembles of learning machines. PND solve multiclass classification problems using a set of two-class classifiers (dichotomizers).
They are based on Output Coding (OC) decomposition methods, that decompose a K-class (K > 2) classification in a set of two-class more simpler subproblems and then recompose the outputs of the dichotomizers coding a class by suitable codewords. For a brief overview of the OC decomposition methods, see, for instance:
F. Masulli and G. Valentini, Comparing Decomposition Methods for Classification, KES'2000, Fourth International Conference on Knowledge-Based Intelligent Engineering Systems & Allied Technologies, IEEE, Piscataway, NJ, 2000.
The implemented methods includes One-Per-Class (OPC), Pairwise Coupling (PWC), Pairwise Coupling Correcting Classifiers (CC) and Error Correcting Output Coding (ECOC) decomposition methods. Each dichotomizer can be a Multi-Layer-Perceptron (MLP) or a simple linear perceptron.
You can select a decomposition method, different learning algorithms for each dichotomizers, you can tune different parameters and you can store onto a file the outputs and the parameters of the trained ensemble. There are many possible options, but if any are not set by the user, default ones are used. The only mandatory parameter is the name of the training file. If you do not specify a separated test file, the testing is performed on the training file. Be aware that the dimension of the input of the dichotomizers must agree with the dimension of the samples of the traininig and test files.
Usage: pnd trainfile [options]
Options:
-test string test file name -decomp string Decomposirtion schemes: OPC (one per class) PWC (pairwise coupling) CC (correcting classifiers) ECOC_ex (Error Correcting Ouput Code - exaustive algorithm) ECOC_BCH (Error Correcting Ouput Code - BCH algorithm) -ndico unsigned codeword length -nc unsigned number of the classes -nl unsigned number of the layers (considering also the input layer): 1: one layer (no hidden units) 2: two layers (1 hidden layer) 3: three layers (2 hidden layers) -h1 unsigned number of hidden units of the first hidden layer of the three layers MLP -h2 unsigned number of hidden units of the second hidden layer of the three layers MLP or number of hidden units of the two layers MLP (default 4) -d unsigned sample dimension/number of input units -maxit unsigned maximal number of iterations of the backpro algorithm (default 500) -maxerr double maximal normalized root mean square error (RMS) (default 0.1) -alg string Backpropagation learning algorithm type: gd (gradient descent) gd_dl (gradient descent linear decrement) gd_de (gradient descent exponential decrement) md (gradient descent with momentum) md_dl (gradient descent with momentum and linear decrement) md_de (gradient descent with momentum and exponential decrement) bold (bold driver) boldmom (bold driver with momentum) -rate double learning rate (default 0.07) -mom double momentum rate (default 0.5) -incbold double bold driver increment rate -decbold double bold driver decrement rate -decr double gradient decrement (for gradient descent with momentum and linear or exponential decrement) -seed seed for the random initialization of the weights. If 0 the initialization is performed using current computer time. -s string save pnd parameters and weights onto a file -out string save pnd ouptuts onto a file -serr string save pnd history RMS errors onto a file -p integer 1 = printing a point at each iteration; 0 = noprint Examples:
For all the examples we refer to a training set named foo.train and a testing set named foo.test with 5-dimensional examples and 7 classes. These are only few examples. See above Usage for a complete reference of the available parameters. Note that you can place the parameters in any order, and most of the parameters are not mandatory. More precisely, you can also supply no parameters, but remember that the number of inputs of each dichotomizer must agree with the dimension of the input examples and you must also specify the correct number of classes.
- Training and testing a PND using MLP with one hidden layer as dichotomizers:
pnd foo.train -test foo.test -d 5 -nc 7
Note that the OPC decomposition is the default one. Note that the order of the the supplied parameters is not important.
Note also that the number of inputs of the dichotomizer must agree with the dimension of the input examples and you must also specify the correct number of classes.- Training and testing a PND selecting a specific decomposition scheme:
Selecting a Pairwise Correcting Classifiers (CC) decomposition scheme:
pnd foo.train -test foo.test -d 5 -nc 7 -decomp CC
Selecting an Error Correcting Output Coding (ECOC) decomposition scheme using a BCH algorithm for generating the codewords:
pnd foo.train -test foo.test -d 5 -nc 7 -decomp ECOC_BCH -ndico 15
Selecting an Error Correcting Output Coding (ECOC) decomposition scheme using an exhaustive algorithm for generating the codewords:
pnd foo.train -test foo.test -d 5 -nc 7 -decomp ECOC_ex
- Training and testing a PND varying number of hidden units and/or learning parameters:
If you want to select a learning rate equal, for instance, to 0.1:
pnd foo.train -test foo.test -d 5 -nc 7 -rate 0.1
If you want to select 20 neurons for the hidden layer:
pnd foo.train -test foo.test -d 5 -nc 7 -h2 20
- Training and testing an ECOC-PND with dichotomic MLP with two hidden layer:
pnd foo.train -test foo.test -d 5 -nc 7 -nl 3 -decomp ECOC_BCH -ndico 15
If you want to select a learning rate equal, for instance, to 0.04:
pnd foo.train -test foo.test -d 5 -nc 7 -rate 0.04 -nl 3 -decomp ECOC_BCH -ndico 15
If you want to select 30 neurons for the fist hidden layer and 22 for the second hidden layer:
pnd foo.train -test foo.test -d 5 -nc 7 -decomp ECOC_BCH -ndico 15 -nl 3 -h1 30 -h2 22
- Training and testing an OPC-PND with dichotomic simple perceptrons:
As the dichotomizers are linear we name these ensembles of learning machines PLD, that is Parallel Linear Dichotomizers: pnd foo.train -test foo.test -d 5 -nc 7 -nl 1 -decomp OPC
- Training a CC-PND changing learning algorithms:
Using a backpropagation with a momentun term:
pnd foo.train -test foo.test -d 5 -nc 7 -alg md -decomp CC
If you want to select a learning rate equal, for instance, to 0.1 and a momentum rate equal to 0.4:
pnd foo.train -test foo.test -d 5 -nc 7 -alg md -rate 0.1 -mom 0.4 -decomp CC
If you want to select a bold driver learning algorithm with increment rate equal to 1.04 and decremet rate equal to 0.5:
pnd foo.train -test foo.test -d 5 -nc 7 -alg bold -incbold 1.04 -decbold 0.4 -decomp CC
- Training and testing a PND and saving it onto a file:
pnd foo.train -test foo.test -d 5 -nc 7 -s mypnd
The file mypnd.poli stores the parameters and weights of the PND ensembles. Actually the methods to reload the stored pnd have to be written. The next release of NEURObjects will fix this problem.- Saving the outputs or the history error:
For saving the computed outputs of the net onto a file named output:
pnd foo.train -test foo.test -d 5 -nc 7 -out output
For saving the history of the RMS errors during the learning epochs onto a set of files whose base name is history:
pnd foo.train -test foo.test -d 5 -nc 7 -serr history
It generates history1, history2, ... , history7 , one for each dichotomizer.
- A more complicated example:
We want to train and test a CC-PND with dichotomic MLP with one hidden layer and 22 hidden neurons using a backpropagation algorithm with an exponential decrement of the learning rate during the epochs, with an initial rate of 0.3. We want also that the learning of each dichotomizer ends when the normalized RMS errors goes below 0.04 or we have reached 1000 iterations; then we want that the resulting pnd is stored in the file finishweights and the outputs relative to the test set is stored in the file output, while the history errors during the epochs stored in the file whose base name is history.
In order to obtain this result you must type:
pnd foo.train -test foo.test -decomp CC -d 5 -nc 7 -h2 22 -alg gd_de -rate 0.3 -maxerr 0.04 -maxit 1000 -s finishweights -out output -serr history
Alphabetic index Hierarchy of classes