Neural network analysis and selection

Last time I spoke about exploring some algorithms that could be able to analyze a set of neural networks with different structures and then select the best one.
These kind of algorithms are very interesting in the fact that they make the best choice of neural network in term of architecture (i.e. the number of hidden neurons or of hidden layers…). It fits the neural network structure to the problem they have to solve, keeping the number of parameters (i.e. the complexity) of the problem as low as possible.

I found these algorithms very interesting to me because they could allow me to automate the process of trying manually different values of neural networks parameters such as the number of hidden neurons. It also extract some figures/scores that are comparable even if the networks have different structures. These scores would allow me to numerically compare many neural networks and see clearly which one performs best to finally use this one and throw away the others.

I then decided to give this big part a try. This is a big challenge because it includes lots of calculations that are not straight forward to me.
Fortunately, I bought a very complete book that’s called Apprentissage Statistique – G. Dreyfus. This book even if I don’t find well structured gives a lot of good tips and tricks and describes in quite good details each steps one should follow to make these algorithms come true.

So I followed the guidelines from that book and went through these steps :

  • Understand enough of the whole algorithms to determine the class architecture I should use. This step is something I always do to start with as good conditions as I can and not be trapped later on because I didn’t think about a particular case or didn’t understand how some stuff was actually working. That’s why I think it is really important to put things into perspective before rushing headlong (because you can’t see the wall in front of you once you run) !
  • Code some calculation routines such as calculating the number of parameter of a given model, calculating its Jacobian matrix, the mean squared errors, the residuals, the leverages, the determination coefficients, the Predicted Residual Sum of Squares (PRESS) score, etc.
  • Put all these figures together to make a decision on which model is the best among a batch.

While going through all these steps, I hit a complex problem I already faced some time ago : how can I know when I should stop my model learning loop ?
This is a recurrent problem when dealing with neural network. The learning phase is an iterative weight modification of very small amounts until the model finally converges or reaches an acceptable threshold of error for example 0.01%.
The problem in neural network is that if you have you model learn for too few iterations it does not perform well, but in the opposite, if you have it learn for too long, it becomes overfitted. This means it has become too specific to the problem you want to solve and thus, it does not perform well neither. So you have to find a middle point that is hard to find because it depends on many parameters such as the number of parameters of the model, the structure of the model (number of neurons, number of hidden layers and so on).
Until that day, I always used the basic workaround which consists of setting an empiric number of iterations which seems to give good results. But now I couldn’t use that one since the selection algorithm will have to perform on multiple different model structures so that I couldn’t consider using the basic approach…

I searched on the web to find how I can deal with that recurrent problem. I finally found that marvelous algorithm from Lutz Prechelt. « Early Stopping – but when? » In Genevieve B. Orr and Klaus-Robert Müller: Neural Networks: Tricks of the Trade, volume 1524 of LNCS, Springer, 1997.

This paper looked so incredible to me because I found it quite easy to code and to integrate in my current workflow. Once again, I tried to put it into perspective and decided how I should structure it in term of code.

After some tests and some hours of intense reflexion I finally came with a working early stop algorithm that I could integrate in the model analyzer algorithm and have it decide for me when it should stop learning.

In the end I have a working analyze and selction algorithm that is able to benchmark a given set of models and extract the best one to be used further on ! This was really exciting since I thought it would be quite out of my reach because it meant going through a lot of steps with each their own issues and difficulties…

I made a quick snapshot of the final results it gives me.

Model analyze and selection - resultsOn the upper left corner there’s the GraphViz code of the architecture of the whole system (see my previous post). On the bottom, there’s graphs that shows the final results :

  • The blue plot on the bottom represents the input data of the models
  • The blue plot on the top represents the desired output data (the data I measured directly on the system)
  • The orange plot is the model output of a linear model (ARX) that’s been taught at the same time with the neural networks. This is to compare this linear model with non linear models (i.e. neural networks)
  • The red one is the neural network that has been selected by the selection algorithm. So this one is the one that performs best
  • Finally, the orange bar plot represents the final scores of all the benchmarked models. On the X-Axis the number represents the number of neurons in the hidden layer.
  • The graph with Frequency written on it was for test purposes so there’s nothing on it…

Ok, this is it ! I’ll have to test this deeper to make sure everything works correctly but the results are pretty encouraging !

Some news on my DeMIMOI library

First of all, I wish everyone coming to this blog a happy new year ! Should this year be healthy and make your projects come true !

It’s been a while now since I published the code of my DeMIMOI core library. I had some time to make some improvements and add some more features to make it even more attractive to me, so possibly for anyone willing to give it a try too.

I thought well when I hoped this platform would help me easily build systems and make a bigger one by connecting smaller ones. I successfully managed to use the library with neural networks (special thanks to Accord.Net and AForge.Net). For example I made DeMIMOI models of a NARX (Nonlinear AutoRegressive eXogenous) neural network and a ARX regression model (AutoRegressive eXogenous).
These models can mimic almost any dynamic system mostly because they have a backward link that make their outputs at time t-1 one of their inputs.
Using the DeMIMOI library this backward link is easy to model and to code since it’s just a connection between an input and an output that, in code, is translated by a myModel.Outputs[0][0].ConnectTo(myModel.Inputs[0][0]) for example. The data flow is then automatically managed by the DeMIMOI core.

I also started to put a database manager in a DeMIMOI model which can currently connect to a Mongo database. So I can save and retrieve data to build models or to act as some kind of a memory… Well it’s still in development now but the main features are already up ! I mean at least reading data from a pre-existing database. One step at a time right ?

To give you a better idea of what I’m talking about and what I’m doing, I’ll show you a picture of the system I’m currently working on.

First of all, let me just explain the background.

While coding the DeMIMOI ARX and NARX models, I wanted to build models that can learn the behavior of the servomotors of my AL5C robotic arm.
On a previous attempt last year, I had the arm moving freely and randomly while an application was recording at each time step the angle values from the potentiometers of each servo (thank you so much Phidgets !).

The results have been stored in a database that I can read using my DeMIMOI memory manager. Those data can then be sent to the ARX and NARX models, and also be used by the learner models to fit them to the servo behavior.

For this purpose, I coded the system which is described by the following diagram :

DeMIMOI NARX and ARX learning Lynxmotion AL5C servoBy the way, this image has been created using the GraphViz code that is automatically generated by the DeMIMOI collection that holds all the blocks of the system.
That’s a feature I’m quite proud of since it allows me to quickly check if what I coded is what I expected the system to be !

On the left of the diagram is the memory manager that reads the servo values from a Mongo database. The data produced is then fed to the ARX model and its teacher. They don’t need extra data processing, as opposed to the NARX model.

Indeed, the NARX model is made of a neural network that needs normalized data. That’s why you can see that the data coming from the memory block is also fed to a normalizer block that converts the raw data to a [-1, 1] range before being sent to the NARX and its teacher. Then the data coming out of the NARX model is denormalized to revert it back to its original data space.

On the ARX and NARX models you can clearly see the backward link that connects the output to the input. This link makes the network seamlessly recurrent. And again, in term of coding, it’s nothing harder than creating the link !

You also may have noticed the three probes that display some data values. This was for me to quickly check what are the model ouputs (simulated) compared to the real output (measured on the real system). On that run, the ARX is better than the NARX, but I currently didn’t push any analyze further to explain this…

My next work will now be focused on analyzing the results deeper, maybe working on the so called BPTT neural network (ouch ! it hurts !) or maybe even trying to make some kind of automated learning shell that would be able to test multiple models and parameters and then select the best one…
I know ! It seams like there’s going to be a huuuuge mountain to climb… I fight this feeling by telling me that I already climb quite a big part that it would be even worse to stop now !

I’ll let you know of my progress in a while. And do not hesitate to say hi or comment on this ! I’d be curious to know what it feels like for someone external !

Getting the images from the webcams

Ok, so now that we have a robot stereo vision enabled, we’d like to see what it sees right ?

Choosing the framework

Let’s start ! Getting images from webcams is quite easy with our favorite libraries ! The best one in my opinion to do so is Aforge (you could of course use Accord.net which also refers to Aforge dlls).

The reason for this choice is that with the framework you can control some camera parameters such as exposure, zoom, pan, tilt, focus and some others.
Controlling these parameters depends on the capabilities of your webcam.
In my case, with the C525 from Logitech, I can control at least focus, zoom, pan and exposure.
That’s quite interesting since I may want to leave them in the auto mode or maybe to control them to set them to a value I found is good for my application.

Using the framework to get the images

Using Aforge to get image streams from webcam is quite easy and it’s almost already done for you !
As you can see on the following link, there’s a sample application that shows you how to get and display two webcam streams : http://www.aforgenet.com/framework/samples/video.html.

Improving and customizing the framework functions

I found it easy but not complete. It gives you the connected cameras in an indexed list which is not necessarily ordered the same way each time you run your program (maybe depending on which webcam you plugged first the very first time you plug it to a new computer…).
For example if you have a stereo webcam configuration, you don’t want your cameras to be inverted in any way since it may cause your application not to work anymore…
Moreover, you cannot save camera settings for the next time you run your app, to automatically recover them.
Finally sometimes, setting up the environnement (objects to use, events, setting up camera…) can be a bit tricky.

I then decided to code a class to avoid these issues.
I can share the code if somebody is interested.

Rotating the images

As a result from the webcam positions and orientations on the robot we have to rotate the images to have them the right way using the RotateFlip function from Aforge.
In my configuration, the left cam is rotated by 270° and the right one is rotated by 90°.

Final result

And here is the final result !

Left imageRight image

Perfect ! We now have the images from the webcams !
I had to properly turn the webcams to have a quite well « calibrated » image.
I mean, the images are visually aligned as the top of the light is on nearly the same height in both images and the webcams seems to be planar.

We are now ready for applying the stereo imaging algorithm !
I’ll make a special post on this topic.

Accord.Net contribution

Accord.Net

I contributed to add the Denavit Hartenberg model support to the Accord.Net library and I’m happy today because my contribution has been released !

In my project, I needed to manage the Kinematics of the robot arm, so I tryed it the hard way first (sinus multiplying cosinus all the way round with angles everywhere). Though, I managed to obtain something quite functional. Some time later I found an article talking about the Denavit Hartenberg (DH) transform which made me realize I spent a lot of hard time to calculate my Kinematics… The DH makes it much more easily !! It also offers a good frame for making it more generic to apply it to any robot configuration.

So I then decided to go for it and code it in C# to add it to my toolbox. I built it generic and flexible to allow easy manipulation of complex models and even push it further to allow animation and on-the-fly modification of the model with model combinations. As I’m more comfy when it’s visual, I also developped a visualization module to display what the model looks like and how it’s moving. Then I decided to share it with the Accord.Net community.

Now, enough words, as a drawing is always better than thousands of hours of explanations, give a look below !

DH model sample app

Some links :

Finally I would like to thank Cesar Souza and all the people involved in Accord.Net and AForge.Net for providing us with such great libraries ! Thank you so much !