Some news on my DeMIMOI library

First of all, I wish everyone coming to this blog a happy new year ! Should this year be healthy and make your projects come true !

It’s been a while now since I published the code of my DeMIMOI core library. I had some time to make some improvements and add some more features to make it even more attractive to me, so possibly for anyone willing to give it a try too.

I thought well when I hoped this platform would help me easily build systems and make a bigger one by connecting smaller ones. I successfully managed to use the library with neural networks (special thanks to Accord.Net and AForge.Net). For example I made DeMIMOI models of a NARX (Nonlinear AutoRegressive eXogenous) neural network and a ARX regression model (AutoRegressive eXogenous).
These models can mimic almost any dynamic system mostly because they have a backward link that make their outputs at time t-1 one of their inputs.
Using the DeMIMOI library this backward link is easy to model and to code since it’s just a connection between an input and an output that, in code, is translated by a myModel.Outputs[0][0].ConnectTo(myModel.Inputs[0][0]) for example. The data flow is then automatically managed by the DeMIMOI core.

I also started to put a database manager in a DeMIMOI model which can currently connect to a Mongo database. So I can save and retrieve data to build models or to act as some kind of a memory… Well it’s still in development now but the main features are already up ! I mean at least reading data from a pre-existing database. One step at a time right ?

To give you a better idea of what I’m talking about and what I’m doing, I’ll show you a picture of the system I’m currently working on.

First of all, let me just explain the background.

While coding the DeMIMOI ARX and NARX models, I wanted to build models that can learn the behavior of the servomotors of my AL5C robotic arm.
On a previous attempt last year, I had the arm moving freely and randomly while an application was recording at each time step the angle values from the potentiometers of each servo (thank you so much Phidgets !).

The results have been stored in a database that I can read using my DeMIMOI memory manager. Those data can then be sent to the ARX and NARX models, and also be used by the learner models to fit them to the servo behavior.

For this purpose, I coded the system which is described by the following diagram :

DeMIMOI NARX and ARX learning Lynxmotion AL5C servoBy the way, this image has been created using the GraphViz code that is automatically generated by the DeMIMOI collection that holds all the blocks of the system.
That’s a feature I’m quite proud of since it allows me to quickly check if what I coded is what I expected the system to be !

On the left of the diagram is the memory manager that reads the servo values from a Mongo database. The data produced is then fed to the ARX model and its teacher. They don’t need extra data processing, as opposed to the NARX model.

Indeed, the NARX model is made of a neural network that needs normalized data. That’s why you can see that the data coming from the memory block is also fed to a normalizer block that converts the raw data to a [-1, 1] range before being sent to the NARX and its teacher. Then the data coming out of the NARX model is denormalized to revert it back to its original data space.

On the ARX and NARX models you can clearly see the backward link that connects the output to the input. This link makes the network seamlessly recurrent. And again, in term of coding, it’s nothing harder than creating the link !

You also may have noticed the three probes that display some data values. This was for me to quickly check what are the model ouputs (simulated) compared to the real output (measured on the real system). On that run, the ARX is better than the NARX, but I currently didn’t push any analyze further to explain this…

My next work will now be focused on analyzing the results deeper, maybe working on the so called BPTT neural network (ouch ! it hurts !) or maybe even trying to make some kind of automated learning shell that would be able to test multiple models and parameters and then select the best one…
I know ! It seams like there’s going to be a huuuuge mountain to climb… I fight this feeling by telling me that I already climb quite a big part that it would be even worse to stop now !

I’ll let you know of my progress in a while. And do not hesitate to say hi or comment on this ! I’d be curious to know what it feels like for someone external !

Getting the images from the webcams

Ok, so now that we have a robot stereo vision enabled, we’d like to see what it sees right ?

Choosing the framework

Let’s start ! Getting images from webcams is quite easy with our favorite libraries ! The best one in my opinion to do so is Aforge (you could of course use Accord.net which also refers to Aforge dlls).

The reason for this choice is that with the framework you can control some camera parameters such as exposure, zoom, pan, tilt, focus and some others.
Controlling these parameters depends on the capabilities of your webcam.
In my case, with the C525 from Logitech, I can control at least focus, zoom, pan and exposure.
That’s quite interesting since I may want to leave them in the auto mode or maybe to control them to set them to a value I found is good for my application.

Using the framework to get the images

Using Aforge to get image streams from webcam is quite easy and it’s almost already done for you !
As you can see on the following link, there’s a sample application that shows you how to get and display two webcam streams : http://www.aforgenet.com/framework/samples/video.html.

Improving and customizing the framework functions

I found it easy but not complete. It gives you the connected cameras in an indexed list which is not necessarily ordered the same way each time you run your program (maybe depending on which webcam you plugged first the very first time you plug it to a new computer…).
For example if you have a stereo webcam configuration, you don’t want your cameras to be inverted in any way since it may cause your application not to work anymore…
Moreover, you cannot save camera settings for the next time you run your app, to automatically recover them.
Finally sometimes, setting up the environnement (objects to use, events, setting up camera…) can be a bit tricky.

I then decided to code a class to avoid these issues.
I can share the code if somebody is interested.

Rotating the images

As a result from the webcam positions and orientations on the robot we have to rotate the images to have them the right way using the RotateFlip function from Aforge.
In my configuration, the left cam is rotated by 270° and the right one is rotated by 90°.

Final result

And here is the final result !

Left imageRight image

Perfect ! We now have the images from the webcams !
I had to properly turn the webcams to have a quite well « calibrated » image.
I mean, the images are visually aligned as the top of the light is on nearly the same height in both images and the webcams seems to be planar.

We are now ready for applying the stereo imaging algorithm !
I’ll make a special post on this topic.