At the previous step I collected data using my robot arm and the webcams I installed on it. The data contains the detected object coordinates on the left and right image, the robot arm joint angles where it detected the object and the same variables but when the arm is placed manually on the object.
So, for each detected object (i.e. database record), I recorded these values we have this :
- Position of object on the left image (xL, yL)
- Position of object on the right image (xR, yR)
- Arm joint angles where the object has been seen : (alphaI, betaI, thetaI, gammaI)
- Arm joint angles at the object location : (alphaT, betaT, thetaT, gammaT)
Now the goal is to build a model that will predict the arm joint angles that will place the arm at the location of the object. To make this prediction, the model inputs will be the object position on each image and the current arm joint angles.
I started by trying to analyze the data I gathered to determine its properties. I quickly found that the data won’t be linear, so that I’ll need a non linear model to map the data. I’m used to neural networks, they are powerful non linear models, so I decided to implement a framework that will help me make the model learn the data.
The learning process
To ease the learning process using the DeMIMOI models, I decided to proceed in 4 steps :
- Prepare and preprocess the data to be learnt
The idea is to build a specific dataset out of the raw data. For example if I have to make a mathematical operation on the raw data, say a subtraction or even a sine.
The resulting data is stored in a temporary storage (the PC RAM in my case).
- Teach the model
The learning algorithm uses the temporary storage to feed the model with the data to learn.
- Plot the results of the obtained model
At this step, you want to see what the model learnt, so you just plot the output of the model and the desired output together to see the error.
- Save the model
Once the model fulfill the needs, it can be saved for a later use.
Learning process implementation
For the first three steps, I created three DeMIMOI_Collection that each holds the required elements of each step.
Now I’m going to show you each of them and describe what’s going on.
- Data preparation and preprocessing
This step requires raw data. The data comes from the Long Term Memory block which hides a MongoDB database. The raw data flowing out of this block is then directly fed to the STM (Short Term Memory) block. On this particular case, I could have sent the raw data directly to the model,but I needed to make some calculations before, which where then removed…
- Teaching the model
This step is the learning step. The data created at the previous step is fed to the Neural Network learning algorithm. Before sending the data to this block, we must make some data scaling since neural networks need [0, 1] or [-1, 1] normalized data depending on the activation function. This is why I put two normalization blocks, one on the inputs and the other one on the outputs.
On this picture, we clearly see what are the inputs and the desired outputs we want the model to learn.
- Plotting the results
Once the model is built and ready to operate, I want to see the results of the learning step. I reuse the STM data as input, the data normalizer to ensure data normalization, then the neural network gets normalized data, that is in turn normalized back to output units. Finally this data is plotted by the DeMIMOI_Chart0 that is mapped to the Windows Forms Chart control I placed on the form.
This results to this interface :
We can see that for each graph, the neural network outputs are really close to the desired outputs, which means that it converged to a « good » solution.
The left textbox contains Graphviz code to draw the system structure I showed all along this article. The right one contains the Graphviz neural network structure.
Now, it seems that we managed to get a model that is able to predict the end effector position to place it on the object that has been seen. So now, that’s the exciting part : let’s build the application that will demonstrate this ability in real time ! It will be the subject of my next post !