Neural network analysis and selection

Last time I spoke about exploring some algorithms that could be able to analyze a set of neural networks with different structures and then select the best one.
These kind of algorithms are very interesting in the fact that they make the best choice of neural network in term of architecture (i.e. the number of hidden neurons or of hidden layers…). It fits the neural network structure to the problem they have to solve, keeping the number of parameters (i.e. the complexity) of the problem as low as possible.

I found these algorithms very interesting to me because they could allow me to automate the process of trying manually different values of neural networks parameters such as the number of hidden neurons. It also extract some figures/scores that are comparable even if the networks have different structures. These scores would allow me to numerically compare many neural networks and see clearly which one performs best to finally use this one and throw away the others.

I then decided to give this big part a try. This is a big challenge because it includes lots of calculations that are not straight forward to me.
Fortunately, I bought a very complete book that’s called Apprentissage Statistique – G. Dreyfus. This book even if I don’t find well structured gives a lot of good tips and tricks and describes in quite good details each steps one should follow to make these algorithms come true.

So I followed the guidelines from that book and went through these steps :

  • Understand enough of the whole algorithms to determine the class architecture I should use. This step is something I always do to start with as good conditions as I can and not be trapped later on because I didn’t think about a particular case or didn’t understand how some stuff was actually working. That’s why I think it is really important to put things into perspective before rushing headlong (because you can’t see the wall in front of you once you run) !
  • Code some calculation routines such as calculating the number of parameter of a given model, calculating its Jacobian matrix, the mean squared errors, the residuals, the leverages, the determination coefficients, the Predicted Residual Sum of Squares (PRESS) score, etc.
  • Put all these figures together to make a decision on which model is the best among a batch.

While going through all these steps, I hit a complex problem I already faced some time ago : how can I know when I should stop my model learning loop ?
This is a recurrent problem when dealing with neural network. The learning phase is an iterative weight modification of very small amounts until the model finally converges or reaches an acceptable threshold of error for example 0.01%.
The problem in neural network is that if you have you model learn for too few iterations it does not perform well, but in the opposite, if you have it learn for too long, it becomes overfitted. This means it has become too specific to the problem you want to solve and thus, it does not perform well neither. So you have to find a middle point that is hard to find because it depends on many parameters such as the number of parameters of the model, the structure of the model (number of neurons, number of hidden layers and so on).
Until that day, I always used the basic workaround which consists of setting an empiric number of iterations which seems to give good results. But now I couldn’t use that one since the selection algorithm will have to perform on multiple different model structures so that I couldn’t consider using the basic approach…

I searched on the web to find how I can deal with that recurrent problem. I finally found that marvelous algorithm from Lutz Prechelt. « Early Stopping – but when? » In Genevieve B. Orr and Klaus-Robert Müller: Neural Networks: Tricks of the Trade, volume 1524 of LNCS, Springer, 1997.

This paper looked so incredible to me because I found it quite easy to code and to integrate in my current workflow. Once again, I tried to put it into perspective and decided how I should structure it in term of code.

After some tests and some hours of intense reflexion I finally came with a working early stop algorithm that I could integrate in the model analyzer algorithm and have it decide for me when it should stop learning.

In the end I have a working analyze and selction algorithm that is able to benchmark a given set of models and extract the best one to be used further on ! This was really exciting since I thought it would be quite out of my reach because it meant going through a lot of steps with each their own issues and difficulties…

I made a quick snapshot of the final results it gives me.

Model analyze and selection - resultsOn the upper left corner there’s the GraphViz code of the architecture of the whole system (see my previous post). On the bottom, there’s graphs that shows the final results :

  • The blue plot on the bottom represents the input data of the models
  • The blue plot on the top represents the desired output data (the data I measured directly on the system)
  • The orange plot is the model output of a linear model (ARX) that’s been taught at the same time with the neural networks. This is to compare this linear model with non linear models (i.e. neural networks)
  • The red one is the neural network that has been selected by the selection algorithm. So this one is the one that performs best
  • Finally, the orange bar plot represents the final scores of all the benchmarked models. On the X-Axis the number represents the number of neurons in the hidden layer.
  • The graph with Frequency written on it was for test purposes so there’s nothing on it…

Ok, this is it ! I’ll have to test this deeper to make sure everything works correctly but the results are pretty encouraging !

Publicités

Some news on my DeMIMOI library

First of all, I wish everyone coming to this blog a happy new year ! Should this year be healthy and make your projects come true !

It’s been a while now since I published the code of my DeMIMOI core library. I had some time to make some improvements and add some more features to make it even more attractive to me, so possibly for anyone willing to give it a try too.

I thought well when I hoped this platform would help me easily build systems and make a bigger one by connecting smaller ones. I successfully managed to use the library with neural networks (special thanks to Accord.Net and AForge.Net). For example I made DeMIMOI models of a NARX (Nonlinear AutoRegressive eXogenous) neural network and a ARX regression model (AutoRegressive eXogenous).
These models can mimic almost any dynamic system mostly because they have a backward link that make their outputs at time t-1 one of their inputs.
Using the DeMIMOI library this backward link is easy to model and to code since it’s just a connection between an input and an output that, in code, is translated by a myModel.Outputs[0][0].ConnectTo(myModel.Inputs[0][0]) for example. The data flow is then automatically managed by the DeMIMOI core.

I also started to put a database manager in a DeMIMOI model which can currently connect to a Mongo database. So I can save and retrieve data to build models or to act as some kind of a memory… Well it’s still in development now but the main features are already up ! I mean at least reading data from a pre-existing database. One step at a time right ?

To give you a better idea of what I’m talking about and what I’m doing, I’ll show you a picture of the system I’m currently working on.

First of all, let me just explain the background.

While coding the DeMIMOI ARX and NARX models, I wanted to build models that can learn the behavior of the servomotors of my AL5C robotic arm.
On a previous attempt last year, I had the arm moving freely and randomly while an application was recording at each time step the angle values from the potentiometers of each servo (thank you so much Phidgets !).

The results have been stored in a database that I can read using my DeMIMOI memory manager. Those data can then be sent to the ARX and NARX models, and also be used by the learner models to fit them to the servo behavior.

For this purpose, I coded the system which is described by the following diagram :

DeMIMOI NARX and ARX learning Lynxmotion AL5C servoBy the way, this image has been created using the GraphViz code that is automatically generated by the DeMIMOI collection that holds all the blocks of the system.
That’s a feature I’m quite proud of since it allows me to quickly check if what I coded is what I expected the system to be !

On the left of the diagram is the memory manager that reads the servo values from a Mongo database. The data produced is then fed to the ARX model and its teacher. They don’t need extra data processing, as opposed to the NARX model.

Indeed, the NARX model is made of a neural network that needs normalized data. That’s why you can see that the data coming from the memory block is also fed to a normalizer block that converts the raw data to a [-1, 1] range before being sent to the NARX and its teacher. Then the data coming out of the NARX model is denormalized to revert it back to its original data space.

On the ARX and NARX models you can clearly see the backward link that connects the output to the input. This link makes the network seamlessly recurrent. And again, in term of coding, it’s nothing harder than creating the link !

You also may have noticed the three probes that display some data values. This was for me to quickly check what are the model ouputs (simulated) compared to the real output (measured on the real system). On that run, the ARX is better than the NARX, but I currently didn’t push any analyze further to explain this…

My next work will now be focused on analyzing the results deeper, maybe working on the so called BPTT neural network (ouch ! it hurts !) or maybe even trying to make some kind of automated learning shell that would be able to test multiple models and parameters and then select the best one…
I know ! It seams like there’s going to be a huuuuge mountain to climb… I fight this feeling by telling me that I already climb quite a big part that it would be even worse to stop now !

I’ll let you know of my progress in a while. And do not hesitate to say hi or comment on this ! I’d be curious to know what it feels like for someone external !