https://doi.org/10.37815/rt=
e.v33n2.852
Artículos originales
Artificial
intelligence-controlled pole balancing using an Arduino board
Balanceo de un poste, controlado por una inteligencia artificial usa=
ndo
una placa Arduino
José Revelo1 https://orcid.org/0000-0003-1159-1523, <=
/span>Oscar Chang1 https://orcid.org/0000-0003-1241-1782
1
jose.revelo@yachaytech.edu.ec, oscar.chang@yachaytech.edu.ec Sent: 2021/07/1=
1 Accepted: 2021/09/28 Published: 2021/11/30
Palabras clave: aprendizaje por refuerzo, agente inteligente,
microcontrolador, Industria 4.0. Abstract Automation Process (AP) is an important issue in the current digitiz=
ed
world and, in general, represents an increase in the quality of productivity
when compared with manual control. Balance is a natural human capacity as it
relates to complex operations and intelligence. Balance Control presents an
extra challenge in automation processes, due to the many variables that may=
be
involved. This work presents a phy=
sical
balancing pole where a Reinforcement Learning (RL) agent can explore the
environment, sense its position through accelerometers, and wirelessly
communicate and eventually learns by itself how to keep the pole balanced u=
nder
noise disturbance. The agent uses RL principles to explore and learn new po=
sitions
and corrections that lead toward more significant rewards in terms of pole
equilibrium. By using a Q-matrix, the agent explores future conditions and
acquires policy information that makes it possible to maintain stability. An
Arduino microcontroller processes all training and testing. With the help of
sensors, servo motors, wireless communications, and artificial intelligence,
components merge into a system that consistently recovers equilibrium under
random position changes. The obtained results prove that through RL, an age=
nt
can learn by itself to use generic sensors, actuators and solve balancing
problems even under the limitations that a microcontroller presents<=
span
lang=3DEN-AU style=3D'mso-ansi-language:EN-AU'>.Resumen
El proceso de automatización (=
AP)
actual es de gran importancia en el mundo digitalizado, en rasgos generales,
representa un aumento en la calidad de producción con el trabajo hecho a ma=
no.
El equilibrio es una capacidad natural del ser humano que está relacionada =
en
trabajos y conducta inteligente. Equilibrarse representa un desafío adicion=
al
en los procesos de automatización, debido a la presencia de múltiples varia=
bles
involucradas. Este artículo presenta el equilibrio físico y dinámico de un
poste en el que un agente, mediante el uso de aprendizaje por refuerzo (RL),
tiene la capacidad de explorar su entorno, detectar su posición a través de
sensores, aprendiendo por sí mismo cómo mantener un poste equilibrado bajo
perturbaciones en el mundo real. El agente usa los principios de RL para
explorar y aprender nuevas posiciones y correcciones que conducen a recompe=
nsas
más significativas en términos de equilibrio del poste. Mediante el uso de =
una
matriz Q, el agente explora las condiciones futuras y adquiere información =
de
política que hace posible mantener el equilibrio. Todo el proceso de
entrenamiento y pruebas se realizan y gestionan íntegramente en un
microcontrolador Arduino. Con la ayuda de sensores, servo motores,
comunicaciones inalámbricas e inteligencia artificial, todos estos componen=
tes
se fusionan en un sistema que recupera constantemente el equilibrio bajo
cambios aleatorios de posición. Los resultados obtenidos demuestran que a t=
ravés
de RL un agente puede aprender por sí mismo a utilizar sensores, actuadores
genéricos y resolver problemas de balanceo incluso bajo las limitaciones que
presenta un microcontrolador.
Introduction<=
/span>
F=
actories
continue to improve and introduce new technology to reduce their production
costs, such as the solution given to repetitive work in the delivery of goo=
ds
industry (Azadeh =
et al.,
2019). New technologies have to t=
arget
and solve two main aspects to keep progressing and improve tasks in multiple
areas: transparency of costs that makes replication easier and produce these
technologies; recreate a more human-like robot behavior.
<= o:p>
O=
ne primary
aspect and basic act that robots should replicate from humans, is to be
proficient at balancing. Whether we want a humanoid or a computer to replace
the workers, they have to be capable of performing on a similar level to hu=
mans
(Hyon et=
al.,
2007). Ships’ displacement is oft=
en
dependent on careful balancing; in sea navigation, equilibrium is critical.=
The
balancing mechanism is based on fin stabilizers, which rotate and change the
fin angle to resist ocean disturbance (Sun et =
al.,
2018). Balance is also present in
aviation. Unmanned aerial vehicles (UAVs) use stabilizers that play an
important role minimizing disruption effects and ensuring a smoother flight=
(Korkmaz=
et al.,
2013), by dealing with three diff=
erent
angles known as yaw, roll, and pitch. Balancing is a critical component that
affects the ability to perform a wide variety of tasks and is essential in =
the
operation require by loaders, robotic arms, ships, anti-seismic systems, or
robots themselves.
<= o:p>
M=
achine
activity does not present adaptation or reaction to new events that occur, =
and
are not predictable during normal function. This is critical if correct
balancing is a goal in changing environments. Unpredictable events and vary=
ing
conditions of the environment are why automation has limited uses in many
factories and moving vehicles. AI may be set to perform a task in varying
circumstances; it may give improved results, and be efficient in activities
currently executed only by humans. A contemporary and relevant tool in AI is
Reinforcement Learning (RL), a methodology by which agents independently le=
arn
efficient control policies. They then use this knowledge to solve logical or
mechanical problems (Foerste=
r et
al., 2017).
<= o:p>
T=
he first
reason to use Arduino is its flexibility to adapt to different projects. AI
requires data recollection and this may be a difficult task because data is
proprietary or cost prohibitive. In this sense, Arduino is handy and very
helpful for recollecting data from the world.
<= o:p>
A=
rduino is
reasonably priced for building several prototypes. Also, as shown in (López-R=
odríguez
& Cuesta, 2016; Taha & Marhoon, 2018), is how feasible it is to b=
uild
more than one specific model or prototype with customizable devices, depend=
ing
on the situation. The low cost advantage also decreases barriers to perform=
a
study in different areas of AI devices, or reducing problems to replicate a=
nd
produce multiple similar technologies that instantly interact with the real
world.
<= o:p>
W=
ith these
premises, this paper develops a dynamic balancing pole that AI manages runn=
ing
in an Arduino microcontroller. Here, an RL agent, with the capacity to read
sensors and control the angles of servo motors that drive an x, y balancing
pole, learns by itself how to keep the pole balanced under noise disturbanc=
e,
responding to natural changes of its environment. The prototype demonstrates
the capabilities of microcontrollers that have some limitations, compared w=
ith
regular desk computers, and how extensive is there potential.
Related work
T=
he
following section examines and evaluates works that are relevant to the pla=
nted
objectives and share common themes, strategies, and method elaboration.
<= o:p>
Arduino fully managed systems
D=
ue to
their low and medium cost, Arduino-based projects can be produced or recrea=
ted.
Due to easy access to a variety of hardware resources, it is simple to exec=
ute
multi-purpose projects. A mobile robotic platform is presented in ADDIN CSL_CITATION {"citationItems":[{=
"id":"ITEM-1","itemData":{"author":=
[{"dropping-particle":"","family":"Araúj=
o","given":"André","non-dropping-particle&quo=
t;:"","parse-names":false,"suffix":"&quo=
t;},{"dropping-particle":"","family":"Po=
rtugal","given":"David","non-dropping-particl=
e":"","parse-names":false,"suffix":"=
;"},{"dropping-particle":"","family":&qu=
ot;Couceiro","given":"Micael
S","non-dropping-particle":"","parse-names&qu=
ot;:false,"suffix":""},{"dropping-particle":&=
quot;","family":"Rocha","given":"Rui
P","non-dropping-particle":"","parse-names&qu=
ot;:false,"suffix":""}],"container-title":&qu=
ot;Journal
of Intelligent \\& Robotic
Systems","id":"ITEM-1","issue":"2&q=
uot;,"issued":{"date-parts":[["2015"]]},"=
;page":"281-298","publisher":"Springer",=
"title":"Integrating
Arduino-based educational mobile robots in ROS","type":"=
;article-journal","volume":"77"},"uris":=
["http://www.mendeley.com/documents/?uuid=3D13d04b91-18e9-4b20-94af-8a=
79eeca6626"]}],"mendeley":{"formattedCitation":&qu=
ot;(Araújo
et al., 2015)","plainTextFormattedCitation":"(Araújo et
al., 2015)","previouslyFormattedCitation":"(Araújo et a=
l.,
2015)"},"properties":{"noteIndex":0},"schema&=
quot;:"https://github.com/citation-style-language/schema/raw/master/cs=
l-citation.json"}(Araújo =
et al.,
2015). In the work, it is claimed=
that
robots must be as inexpensive as possible for students and researchers to
conduct real-world experiments. The work in (Ram et =
al.,
2017) demonstrates the versatilit=
y of
Arduino when used in an IoT project, a proper power supply, wireless
communication and sensor devices are good features of working with Arduino
boards. The Arduino microcontroller is used in (Gonzále=
z &
Calderón, 2019) to build a Supervisory Cont=
rol and
Data Acquisition (SCADA) system with high configuration possibilities, easy
access to libraries and a fast lea=
rning
rate for new users. In (Wu et a=
l.,
2017) it is described the Arduino=
board's
data collection, tracking, and scalability projection capabilities.
<=
span
style=3D'mso-spacerun:yes'>
T=
he
majority of the work discuss the low-cost advantages of Arduino projects, as
well as how to adapt or include more electronic devices to create more robu=
st
devices with no issues. Furthermore, the majority of the works incorporate =
two
elements: automation and real-time operations. Arduino can conduct real-time
operations without the use of simulations, concentrating on the problems th=
at
are important to the target, enabling it to detect planning mistakes or mis=
sed
steps in the early stages of development. This has been a big help for
researches, both technically and theoretically, showing that Arduino is a v=
ery
useful tool.
<= o:p>
Artificial intelligence with microcont= rollers
I=
n (Jain, 2=
018), a self-driving car is demo=
nstrated
using a Raspberry Pi and a Convolutional Neural Network. An Arduino board
guides the movements of the vehicle. It receives an order and drives the ca=
r in
a specific direction. The author concludes that the car's design and testing
were successful. In addition, the work (López-R=
odríguez
& Cuesta, 2016) shows a mobile robot for
educational purposes based on Android and Arduino. The robot has a light
sensor, GPS, camera, accelerometer, Bluetooth, and Wi-Fi, and can accept
commands from a smartphone via an Internet connection. The author claims th=
at
software alteration can be easily applied for future projects. In the third
study (Lengare=
&
Rane, 2015), image processing is used to
monitor the movements of a human arm and reproduce this action in a robotic=
arm
controlled by an Arduino. The author claims that an Arduino board can monit=
or
the robot's behavior; however, there are many ways to accomplish the same t=
ask.
<= o:p>
I=
t is clear
from the previous work that the Arduino will perform actions with good resu=
lts
if the instructions come from a logical process of an efficient AI or a hum=
an.
The argument is that with the use of a microcontroller, the majority of the
work present no hardware problems. As a consequence, with the right
instructions, Arduino will perform a variety of tasks. The intelligence of =
data
analysis is executed outside the microcontroller in all previous projects, =
but
as further projects illustrate, it is also possible to code the intelligenc=
e or
incorporate learning techniques in the Arduino itself, without having to ra=
ise
costs or resources.
<= o:p>
A=
rtificial
Intelligence has shown better results than conventional approaches in sever=
al
of projects; these works involve intelligent agents that perform several ta=
sks
in pursuit of optimum results. It was shown in (Jimenez=
et al.,
2020; Salazar et al., 2013), that an intelligent agent’s
capable of considering timing, amount of water, and properly implementing t=
hem
in what they detail as Spatio-temporal variations of the soil–plant–atmosph=
ere
system, gathering impressive results in terms of water efficiency and preci=
sion
irrigation. As agents work with microcontrollers like Arduino, they may cre=
ate
further applications. For example, (Mata-Ri=
vera et
al., 2019) shows an intelligent traffi=
c light
control device that uses Arduino sensors to detect the color of traffic lig=
hts,
as well as the tone, distance, and motion of vehicles in the streets where =
it
is installed. The entire process is executed in Arduino, demonstrating how =
this
platform can perform complex tasks using machine learning techniques.
<= o:p>
S=
ince the
recently mentioned works used an Arduino to carry out the whole operation,
these project reinforces the concept that Arduino hardware and software are
plenty capable, and assisting this study in realizing that AI can be
implemented in a microcontroller. Of course, Arduino has limitations, but t=
he
previous works only require a little extra effort to comprehend the obstacl=
es,
adapt the prototype, and include RL.
<= o:p>
Reinforcement learning in Real environ= ments
T=
he aim of
developing robots is to allow them to participate in a real-world environme=
nt
where unexpected events can occur. The goal is to make machines more effect=
ive
in performing various tasks in a manner that is equivalent to, if not super=
ior
to, a human. To accomplish this action in computers, a variety of techniques
are used.
<= o:p>
T=
hree
classifications of methods are given in (Kormush=
ev et
al., 2013), as well as mentioned how c=
omputers
can learn to deal with such tasks. Direct programming is defined as the low=
est
method. This is more akin to a programming method than it is to learning. T=
he
second method is imitation learning, which is more comparable to a learning
strategy. The problem with this approach is that it requires a specialist to
perform a perfect presentation of how to complete the job. The last method,=
RL
is defined as a trial-and-error method that explores the environment and the
robot's body. RL has three benefits over the other two approaches: learning=
new
tasks, optimizing efficiency, and adapting to new situations.
<=
span
style=3D'mso-spacerun:yes'>
R=
L has been
used in real-world settings to solve physical problems in various experimen=
ts,
but simulations can also be used in training. According to (Pan et =
al.,
2017), depending on the context, =
the training
will present undesirable driving behaviors that can cause harm to the
environment. As a result, depending on the case, a simulation or real-world
environment may be used to train a computer. The problem with transitioning
from a virtual to a real-world environment is that it takes extra time to a=
pply
the learned skills to a different scenario than the one in which they were
educated. The research of (Miglino=
et al.,
1995) demonstrates how simulation=
s can
solve problems that appear to be closer to the original target but are not;=
in
other words, the simulation environment lacks realistic behavior. However,
conducting training in the real world poses a challenge; as mentioned in (Nagaban=
di et
al., 2018), samples can be extremely c=
ostly.
The tools used can influence this complication about cost of collecting
samples. As mentioned, an Arduino’s main advantage is its low cost.
<=
span
style=3D'mso-spacerun:yes'>
T=
he work
from (Gu et a=
l.,
2017) creates and trains a robot =
to open
a door from scratch, which is a realistic application that requires complex
contact dynamics to imitate human physical abilities. In addition, it
demonstrates how real robotic platforms can perform physical tasks in 3D re=
al
environments. It is also claimed that RL is an effective training method for
these systems. The work of (Sharma =
et al.,
2020) uses a free-reward RL algor=
ithm to
teach a robot how to travel properly within its structure and navigate, 20
hours of preparation, the machine had mastered a variety of locomotion gait=
s.
After a short time, the RL methods yielded results. This research showed ho=
w RL
can be used to teach an AI how to control its movements in the real world,
without wasting too much time in the training process.
M
T=
his
section describes the proposed system, which learns to detect a properly st=
able
position. The construction of the balancing prototype has three main points=
to
be described, divided into the following sections: General description of t=
he
system. A detailed description of the transmitter device. A detailed
description of the receiver device. In the end, generating an algorithmic
description of the training process.
<= o:p>
Problem Solving Phases
The structure to balance consists of a pole with a platform at the t=
op,
where objects can be placed. The pole cannot stand up straight due to a
flexible component in the base, the pole is connected to this component whi=
ch
allows the pole to constantly bend in any direction. To control the pole
displacement, two servo motors are connected using a rigid wire. One servo
motor moves the pole up and down (north-south), the second mo=
ves the
pole from left to right (east-west). An upper view of the balancing structu=
re
shown in Figure 1 A, and a front=
view
in Figure 1 B.
Figure <=
/span>1<=
/span>
A View of the Structure to Balanc= e
![3D"Interfaz](3D"852-FINAL-GALERADA_archivos/image003.jpg")
Analysis of the Problem
The first issue is to translate the environment in terms that is
possible to convert into a programming platform, select the correct data ty=
pe,
and attempt not to exceed the memory capacity of the Arduino.
The speed execution of the code is enough to sustain the speed movem=
ent
of the servo motors. To train an agent and measure a reward, is necessary to
measure after a change or move is completed. The Arduino board can execute
multiple moves virtually before a servo motor actually has finished the fir=
st
move. In other words, the measurement has some imprecision issues, due to
movements that are supposed to happen but are retarded because of inertial
forces.
Implementation
The agent’s work is to randomly explore the environment through its =
own
actions. The agent will try different moves depending of its actual state,
meaning that the agent will increase and decrease the angle in the servo,
allowing the agent to explore which action will yield better or worse resul=
ts,
depending on the detected state. The Q-matrix, also known as a Q-table,
represent the conditions and possible behavior of the agent represented. The
dimensions of this Q-matrix are 9x2, represented in , where =
the number of columns repres=
ents the
number of possible actions from which the pole will pass. The number of fil=
es
represents the inclination states in which the platform will be.
Figure <=
/span>2<=
/span>
Q-Matrix, States(Rows) and Actions (Columns)
![3D"Diagrama
Descripción](3D"852-FINAL-GALERADA_archivos/image005.png")
T=
he se=
rvo
motors in use operate in degrees from 0⁰-180; one pushes the pole fo=
rward or
backward Figure 3 (a), while the=
other
moves the pole left Figure 3 (c), or right =
Figure 3 (d). However, =
if the
system detects that is in balance, both servo motors will not move pole Figure 3 (b) and Figure 3 (e). The first=
column depicts the action o=
f decreasing
the current angle at which the servo motor is located, while the second col=
umn
depicts the action of raising the current angle at which the servo motor is
located. The servo motor has 144⁰ of inclination since the eighth row=
is
8, which is the highest value that can be found in this matrix beginning at
state 0 from 0⁰, in the servo motor.
<= o:p>
T=
he error
to balance the pole is 18⁰ due to the Q-matrix design. Both servos
increase or decrease its angle 18⁰, the angle was chosen to save space
memory in Arduino, and also the servo motors have a better response with an=
gle
variations of a larger magnitude, one-degree variation will be a good
improvement, but better precision servo motors are required.
Figure <=
/span>3<=
/span>
Possible actions performed by the= servos
![3D"Gráfico,](3D"852-FINAL-GALERADA_archivos/image007.jpg")
<= o:p>
T=
he
prototype stabilizes the post in 18 steps, with each servo performing nine
steps simultaneously, in a worst-case scenario. Delays measured in the serv=
o's
movement speed, allowing the balancing time to increase or decrease. Servos
perform angle variations every 200 milliseconds; hence these nine moves can=
be
completed in less than 1800 milliseconds. In the worst-case scenario, the
prototype's balancing time is nearly two seconds.
<= o:p>
System description
The prototype design is structured by two devices
working together to balance the pole, as seen in Figure 4. This details which componen=
t is
connected and how it communicates and flows. It is known as a Q-table, from
upper components to which are below. the device A and B have wireless
communication, the implementation and a deeper description of the internal
process of each device will be described in the following sections. Device =
A,
due to its structure, could be reused in more areas to estimate more precise
position changes. The device B does not have the same reuse capacity; it ha=
s to
be modified depending on the balanced structure.
Figure <=
/span>4<=
/span>
Prototype Whole Process Flow Diag= ram
![3D"Diagrama
Descripción](3D"852-FINAL-GALERADA_archivos/image009.jpg")
Device =
A
Has the task to recollect and notify positi=
on
changes of the platform in which it is placed. Figure 5 shows the electronic elements used as the
Arduino Nano, the accelerometer and gyroscope that are in the MPU6050 senso=
r,
and the nRF24L01 RF transceiver. Device A performs the flow process in Figure
6.
Figure <=
/span>5<=
/span>
Two
Pictures From a Different Angle, of Each Device and its Elements![3D"Imagen](3D"852-FINAL-GALERADA_archivos/image011.jpg")
Device B
This device, managed by Arduino Uno, will execute the process from <=
/span>Figure
7. Afterwards, the Train process
finishes and will execute the steps from Figure
8. Device A in Fi=
gure 5 shows the two servo motors, that manage =
the
positions of the pole, also have the nRF24L01 RF transceiver for wireless
communication.
Figure <=
/span>6<=
/span>
Arduino NANO Process Flow Diagram
![3D"Diagrama
Descripción](3D"852-FINAL-GALERADA_archivos/image013.jpg")
Device A process
Setup
and Loop Functions
The setup and loop functions are the two main functions of most Ardu=
ino
programs. After uploading the program to an Arduino microcontroller, the se=
tup
function is the first to run. The loop function is a loop that repeats the
actions inside the microcontroller until the microcontroller resets or the
power supply is disconnected. Both functions are required in Arduino progra=
ms.
Sense
and Transmission Flow Process
The Arduino Nano carries out the entire process depicted in <=
!--[if supportFields]>
<= o:p>
T=
he loop,
as seen in Figure 6, is the second part that constantly reads
position changes. MPU6050 is constantly read during sensor reading, and the
data obtained is then sent to data processing, where all information is
processed in degrees and g (terms of standard gravity). After that, data
transmission begins, which reports the changes in the pole's extreme to the
Arduino UNO.
<= o:p>
Device B process
Train Flow Process
Setup in a=
nd a=
re very
similar, except for the variables initialization step with the R and Q matr=
ices
for the training process. As seen in Figure 6, use the same
parameters for wireless communication initialization.
In the loop section, the data reception is emitted from the accelerom=
eter
in the Arduino Nano. Action Selection consists of a random selection of the
available options from the servo motors. Now Perform action executes the
selected action from the previous step. Calculating a reward is the next st=
ep,
updating the Q-table. A fixed number of iterations is achieved all the steps
are repeated. The end section is saving the experience that is in the Q-tab=
le,
earned in the loop section for forthcoming testing.
<= o:p>
Arduino UNO Train Phase Process F= low Diagram
![3D"Diagrama
Descripción](3D"852-FINAL-GALERADA_archivos/image015.png")
Arduino UNO Already Trained Process Flow
Diagram![3D"Diagrama
Descripción](3D"852-FINAL-GALERADA_archivos/image017.jpg")
Testing Flow Process
Setup in a=
nd s=
how the
same steps. Then in the loop function, the data reception is received from =
the
Arduino Nano, then it checks the Q-table, selects the best option, and perf=
orms
the action selected in the servo motors establishing the balance and repeat=
ing
the process from the data reception.
Code structure
T=
he
pseudocode from Figure
9 is train agent and executed to =
train
the agent. The input of the algorithm are the variables that describe the
current state of the environment.
<= o:p>
F=
irst are
initialized the different variables the Qm
is Q-matrix and all the values are initialized from zero, which will present
changes in the values while learns the policy, K is the number of limited episodes, Moves is the variable that limits the number of moves an agent =
can
execute before finish an episode.
<= o:p>
A=
fter the
train starts, first are selected an x
and y position, a random number
selected between the 0 and the number of columns for x and the number of files for y
of the Qm matrix, the variable =
Agent is assigned a random starting
action from Qm[x][y] position.
<= o:p>
T=
he
following inner loop works as a limited sequence of steps where the agent c=
an
move around the environment until it reaches the goal or executes a limited
number of moves defined previously in the variable Moves. The function StepSelection looks at matrix Qm neighborhood values, selects an
action of the possible options. The possible options of the act allow the a=
gent
to move in four possible ways; up, down, left, right.
<= o:p>
U=
pdating
the variables Agent, x, y, depe=
nd on
the previous move. The variable State saves the new environment variables. =
The
matrix Qm updates the index x, =
y value calling the function rewar=
d,
which uses the variables State =
and action to calculate the new value
function Reward.
<= o:p>
I=
f the
agent reaches the goal state the episode finishes, otherwise this inner loo=
p is
repeated until the end of the allowed number of steps. The output is a Q-ta=
ble
matrix, the Qm which has the
experience obtained after the agent exploration.
<= o:p>
Agent Training Pseudocode =
![3D"Interfaz](3D"852-FINAL-GALERADA_archivos/image019.jpg")
Table =
1 shows the Q-table values for the servo mot=
or
that pushes the pole left and right, while
Table 2 shows the Q-table values fo=
r the
second servo motor's forward and backward. The table depicts the Q-table du=
ring
various training episodes, and it is possible to observe how the absolute v=
alue
rises. The 0 value does not change during training because the episode ends
when it enters this state; this state is considered the most stable because=
it
receives a reward of 0, while the other states receive a punishment dependi=
ng
on the platform's inclination. It is easy to see how values that are farther
from the stable state are punished more severely, resulting in extreme valu=
es
gaining more knowledge, or learning faster than states closer to the optimal
state. As compared to Q-tables with more iterations, it is possible to see =
that
certain values in one co=
lumn
that should be lower than the other column are not. In the first 50 episode=
s in
the table Q-table from Table =
1, it was determined that the agent does not
have the experience to choose the proper actions at this point in the train=
ing.
Table 1=
Agent 1 Q-table Evolution of Values at Different Training Episodes =
|
100 |
500<=
/b> |
1000 |
||||
|
State |
Actions |
|||||
|
0 |
-7.63 |
-7.63 |
-16.42 |
-16.42 |
-16.13 |
-16.13 |
|
1 |
-7.59 |
-5.66 |
-17.45 |
-13.32 |
-17.44 |
-13.32 |
|
2 |
-3.91 |
-3.06 |
-12.95 |
-9.45 |
-12.96 |
-8.38 |
|
3 |
-1.44 |
-1.76 |
-6.56 |
-4.94 |
-8.48 |
-3.93 |
|
4 |
0 |
0 |
0 |
0 |
0 |
0 |
|
5 |
-0.86 |
-0.54 |
-1.00 |
-2.33 |
-1.00 |
-2.75 |
|
6 |
-3.10 |
-4.99 |
-3.90 |
-7.04 |
-3.90 |
-6.77 |
|
7 |
-7.04 |
-9.41 |
-8.51 |
-13.99 |
-8.51 |
-12.37 |
|
8 |
-9.49 |
-9.49 |
-12.66 |
-12.66 |
-12.66 |
-12.66 |
Table
2
|
EPISODE |
50 |
100<=
/b> |
500 |
1000 |
||||
|
State |
Actions |
|||||||
|
0 |
-3.254 |
-3.54 |
-3.254 |
-3.54 |
-9.02 |
-9.02 |
-14.30 |
-14.30 |
|
1 |
-2.15 |
-1.73 |
-17.45 |
-1.73 |
-10.21 |
-8.22 |
-15.15 |
-13.49 |
|
2 |
-3.35 |
-2.24 |
-12.95 |
-2.24 |
-9.13 |
-7.74 |
-13.52 |
-10.74 |
|
3 |
-3.75 |
-2.77 |
-6.56 |
-2.77 |
-9.04 |
-3.99 |
-11.75 |
-6.32 |
|
4 |
0 |
0 |
0 |
0 |
0 |
0 |
0 |
0 |
|
5 |
-0.83 |
-0.47 |
-0.83 |
-0.47 |
-1.00 |
-1.46 |
-1.00 |
-1.67 |
|
6 |
-1.18 |
-1.73 |
-1.18 |
-1.73 |
-2.68 |
-3.38 |
-1.90 |
-3.40 |
|
7 |
-2.68 |
-4.69 |
-2.68 |
-4.69 |
-3.71 |
-7.44 |
-3.71 |
-6.50 |
|
8 |
-5.78 |
-5.78 |
-5.78 |
-5.78 |
-7.34 |
-7.34 |
-7.34 |
-7.34 |
In the cumulative reward Figure
10, the x-axis
represents the number of episodes trained, while the y-values represent the
absolute value total of each variable corresponding to the Q-table. The gra=
phic
demonstrates that the agent is learning, gaining experience after each
iteration from the first to the nth episode. Figure
10 plots a summar=
y of
all the learning work, both curves present a general growth doing decently.
Over the last two hundred iterations agent 1 does not have a noticeable increase in compari=
son
to earlier episodes. Agent 2 shows that it could improve its experience, th=
is
behavior comes from the random aspect of the training process, at some epis=
odes
would have a noticeable increase in comparison to earlier episodes, this
behavior comes from the random aspect of the training process, at some epis=
odes
will have a noticeable.
Figure 10
Cumulative Rew=
ard as a
Function of the Number of Episodes, from both Agents
![3D"Título:](3D"852-FINAL-GALERADA_archivos/image021.png")
In Figure 11 are the Q-matrix at episode 100 and 1000, =
of
both agents. The colors with a more negative value represent a higher point=
and
have a darker color, while values closer to zero are more clear and lower points. The agent's path will be the clearest option where it w=
ould
be located, at episode 1000, the disparity in colors between the different
states of the Q-matrix is more apparent compared at episode 100, where the =
path
to follow is somewhat diffuse.
Figure 11
Agent 1 and Agent 2 Heatmap Q-matrix representation
![3D"Gráfico,](3D"852-FINAL-GALERADA_archivos/image023.jpg")
=
The pole
is at the leftmost position; Figure
12 A depicts the direction in wh=
ich the
servo will move the pole; internally, the agent determines which alternativ=
e is
most appropriate. Figure
12 B is a basic example that con=
cludes
when the pole is balanced.
Figure 12
Prototype
balancing the pole simple case =
![3D"Pantalla](3D"852-FINAL-GALERADA_archivos/image025.jpg")
Figure
13 shows an interesting behavi=
or,
which an external disturbance is manually introduced, tilting the base, sim=
ilar
to what happens in air and sea ships. The agents must now solve a fresh and
unexpected problem.
In
Figure
13
Prototype behavior t=
ilting
the platform
![3D"Un](3D"852-FINAL-GALERADA_archivos/image027.jpg")
Figure
14 is an internal
representation of the actual behavior after finishing the training followin=
g a
path and choosing the clearest option, depicting the direction that agent 1
will take if it is in the rightmost state, a right inclination. Agent 1 will
gradually shift the pole to the left until it reaches the state in row 4. T=
he
agent 2 detects that is at a state 0, thanks to the sensometer, which means=
the
platform is facing backward, and it will shift to the front by increasing t=
he
signal sent to the servo motor by 18⁰, moving from state 0 to state 4=
.
Internal Behavior of Agents= =
![3D"Imagen](3D"852-FINAL-GALERADA_archivos/image029.jpg")
T=
his
project includes a study of Arduino microcontrollers, some of their
peripherals, and the implementation of a RL agent in a completely Arduino-b=
ased
prototype consisting of two devices that operate by wireless communication.
Within the experimental limitations, the automation process implementation =
with
Artificial Intelligence and RL agents running in Arduino boards, presented =
very
promising results that, in principle, can extend to more complex intelligent
machines in the real world. This will require increasing the number of
variables, the resolution of the sensors, the power of the actuators, etc.<=
span
style=3D'mso-spacerun:yes'> According to the obtained results durin=
g the
project’s execution, the following facts can be concluded:
· =
The restricted memory space =
of the
Arduino for storing variables is not a barrier to running learning RL
algorithms. In particular, Q-learning algorithms can be fully implemented in
current Arduino microcontrollers, as demonstrated.
· =
In the real world, most envi=
ronments
are analog; however, as demonstrated in this study, using Q-learning and a =
finite
Q-matrix it is possible to convert analog changes in discrete signals and u=
se
them to control a pole balancing situation.
· =
Wireless communication adds =
value to
Arduino projects by allowing them to collect data and/or send instructions =
in
complex environments with limited space and/or constant movement. Also, ope=
ns
possible valuable applications in the world of the Internet of things (IoT)=
.
· =
Agents and RL algorithms are
powerful tools because they allow robots to train themselves in many tasks,
such as maintaining dynamic balance using only their implemented components
such as sensors and actuators.
· =
Arduino platforms have acces=
s to a
variety of peripherals with methodical inclusion, and is a fertile ground to
work and develop self-learning macro sensors, robots, and control systems.<=
/span>
Referencias
Araúj=
o, A.,
Portugal, D., Couceiro, M. S., & Rocha, R. P. (2015). Integrating Arduino-based educational mobile robots=
in
ROS. Journal of Intelligent \& Robotic Systems, 77(2),
281–298.
Azade=
h, K.,
De Koster, R., & Roy, D. (2019). =
Robotized
and automated warehouse systems: Review and recent developments. Transpo=
rtation
Science, 53(4), 917–945.
Foerster, J., Nardelli, N., Farquhar, G., Afouras, =
T.,
Torr, P. H. S., Kohli, P., & Whiteson, S. (2017). Stabilising experience
replay for deep multi-agent reinforcement learning. International Confer=
ence
on Machine Learning, 1146–1155.
Garcia, J., & Shafie, D. (2020). Teaching a
humanoid robot to walk faster through Safe Reinforcement Learning. Engin=
eering
Applications of Artificial Intelligence, 88, 103360.
González, I., & Calderón, A. J. (2019). Integra=
tion
of open source hardware Arduino platform in automation systems applied to S=
mart
Grids/Micro-Grids. Sustainable Energy Technologies and Assessments, =
36,
100557.
Gu, S., Holly, E., Lillicrap, T., & Levine, S.
(2017). Deep reinforcement learning for robotic manipulation with asynchron=
ous
off-policy updates. 2017 IEEE International Conference on Robotics and
Automation (ICRA), 3389–3396.
Hyon, S.-H., Hale, J. G., & Cheng, G. (2007).
Full-body compliant human--humanoid interaction: balancing in the presence =
of
unknown external forces. IEEE Transactions on Robotics, 23(5),
884–898.
Jain, A. K. (2018). Working model of self-driving c=
ar
using convolutional neural network, Raspberry Pi and Arduino. 2018 Second
International Conference on Electronics, Communication and Aerospace Techno=
logy
(ICECA), 1630–1635.
Jimenez, A.-F., Cardenas, P.-F., Canales, A., Jimen=
ez,
F., & Portacio, A. (2020). A surve=
y on
intelligent agents and multi-agents for irrigation scheduling. Computers=
and
Electronics in Agriculture, 105474.
Korkmaz, H., Ertin, O. B., Kasnako\uglu, C., &
others. (2013). Design of a flight stabilizer system for a small fixed wing=
unmanned
aerial vehicle using system identification. IFAC Proceedings Volumes=
, 46(25),
145–149.
Kormushev, P., Calinon, S., & Caldwell, D. G.
(2013). Reinforcement learning in robotics: Applications and real-world
challenges. Robotics, 2(3), 122–148.
Lengare, P. S., & Rane, M. E. (2015). Human hand
tracking using MATLAB to control Arduino based robotic arm. 2015
International Conference on Pervasive Computing (ICPC), 1–4.
López=
-Rodríguez,
F. M., & Cuesta, F. (2016). Andruin=
o-A1:
Low-Cost Educational Mobile Robot Based on Android and Arduino. Journal =
of
Intelligent and Robotic Systems: Theory and Applications, 81(1),
63–76. https://doi.org/10.1007/s10846-015-0227-x
Mata-=
Rivera,
M. F., Zagal-Flores, R., & Barría-Huidobro, C. (2019). Telematics and Computing: 8th International Congres=
s,
WITCOM 2019, Merida, Mexico, November 4--8, 2019, Proceedings
Miglino, O., Lund, H. H., & Nolfi, S. (1995).
Evolving mobile robots in simulated and real environments. Artificial Li=
fe,
2(4), 417–434.
Nagabandi, A., Clavera, I., Liu, S., Fearing, R. S.,
Abbeel, P., Levine, S., & Finn, C. (2018). Learning to adapt in dynamic,
real-world environments through meta-reinforcement learning. ArXiv Prepr=
int
ArXiv:1803.11347.
Pan, X., You, Y., Wang, Z., & Lu, C. (2017).
Virtual to real reinforcement learning for autonomous driving. ArXiv
Preprint ArXiv:1704.03952.
Ram, S. A., Siddarth, N., Manjula, N., Rogan, K., &=
amp;
Srinivasan, K. (2017). Real-time automation system using Arduino. 2017
International Conference on Innovations in Information, Embedded and
Communication Systems (ICIIECS), 1–5.
Salaz=
ar,
R., Rangel, J. C., Pinzón, C., & Rodríguez, A. (2013). Irrigation system through intelligent agents
implemented with arduino technology.
Sharma, A., Ahn, M., Levine, S., Kumar, V., Hausman, K., & Gu, S. (2020). Emergent real-world robotic skills via unsupervised off-policy reinforcement learning. ArXiv Preprint ArXiv:2004.12974.<= o:p>
Sun, M., Luan, T., & Liang, L. (2018). RBF neur=
al
network compensation-based adaptive control for lift-feedback system of ship
fin stabilizers to improve anti-rolling effect. Ocean Engineering, <=
i>163,
307–321.
Taha, I. A., & Marhoon, H. M. (2018).
Implementation of controlled robot for fire detection and extinguish to clo=
sed
areas based on Arduino. Telkomnika, 16(2), 654–664.
Wu, D., Liu, S., Zhang, L., Terpenny, J., Gao, R. X=
.,
Kurfess, T., & Guzzo, J. A. (2017). A fog computing-based framework for
process monitoring and prognosis in cyber-manufacturing. Journ=
al of
Manufacturing Systems, 43, 25–34.