A low-cost wireless extension for object detection and data logging for educational robotics using the ESP-NOW protocol

ESP-NOW

The ESP32 (Espressif Systems, 2022) supports a proprietary communication protocol called ESP-NOW that utilizes the 2.4 GHz band and is optimized to send short messages of up to 250 bytes to other ESP32 nodes with very little overhead, a quick response time on the order of one millisecond, low power usage, and a range between nodes of over 200 m (Yukhimets, Sych & Sakhnenko, 2020). These characteristics make the ESP32 ideal for reading data from and sending instructions to real time control systems and data logging (Linggarjati, 2022). The onboard ESP32 does not require a separate battery and can be powered by the output voltage on any of the VEX robot three-wire ports which support an output of 5 volts at a maximum of 2 amps. The ESP-NOW link was operated at a bit rate of 1 Mbps.

The ESP32 also has Wi-Fi communication built into the board. Wi-Fi communication would work well for the transmission of data from the board to a computer for applications such as data logging. However, the overhead associated with this communication would make it ill-suited for usage in a real-time control system where the external computer is driving the robot and responding to sensor data transmitted by the robot.

OpenMV

The H7 OpenMV camera (Abdelkader, El-Sonbaty & El-Habrouk, 2017) module used in this study has an ARM 32-bit Cortex M7 CPU and consumes 140 mA when active. It can be powered using the five-volt three-wire port power on the VEX V5 microprocessor. The USB connection between the camera and the computer can transmit data at rates up to 12 Mb/s. The camera can take 16-bit RGB565 images at a resolution of 640 × 480 at 75 frames per second. The onboard processor is capable enough to run small tensorflow lite models, run the opencv image processing library, and perform basic image pre-processing if needed. Programs can be written for and run on OpenMV hardware using Python.

Connectivity

Three different hardware configurations were envisioned for this system, as shown in Fig. 1. Each of these configurations has strengths that would make it useful for different applications.

The first configuration, Fig. 1A, has the VEX V5 microprocessor connected to an ESP32 with the OpenMV camera, computer, and another ESP32 connected to each other. Direct connection of OpenMV to the computer via USB supports the rapid transmission of images. This is useful for real time computationally intensive image analysis. Commands are then sent to the robot via the ESP-NOW connection. This configuration is suitable for a case where the robot actuation is occurring in response to a visual que in a fixed region, such as machine automation tasks and quality control. This first configuration was selected for the sorting experiments detailed later in this article.

The second configuration, Fig. 1B, has the OpenMV and VEX V5 microprocessor connected to the same ESP32. The VEX V5 is connected via a RS485 bridge while the OpenMV camera is connected via a serial connection. The vision analysis would be limited to tensor flow lite models, which can be run directly on the OpenMV camera. This configuration has the advantage that the OpenMV camera, the VEX V5 microprocessor, and the ESP32 can be powered using the VEX battery.

The third configuration, Fig. 1C, has the VEX V5 microprocessor, the computer, and the OpenMV camera, each connected to one of the two independent ESP32s. If the camera and the robot were both remotely located and the camera was not traveling with the robot, this configuration might be considered. This would be suitable for a case where a tensor flow lite model can be used and run directly on the OpenMV camera, negating the need for heavy computation on the computer.

All three of these configurations are viable and benefit from the added computational power gained by having a computer in the loop. In addition, each ESP32 can be used to interface with additional sensors such as lidar, temperature, or GPS sensors via the I2C protocol. These types of sensors are unavailable through the official VEX manufacturer, and I2C is unavailable on the VEX V5 microprocessor. These additional sensor types would greatly increase the scope and type of experiments that could be conducted using the VEX V5 microprocessor.

Wiring

The VEX V5 microprocessor has 21 smart ports, of which 20 are located on the front of the device as shown in Fig. 2. These smart ports can communicate with external devices using the RS485 standard and provide power at 12 Volts. The VEX V5 microprocessor also has eight three-wire ports located on the side of the device, which can be used to supply power at 5 Volts. The ESP32 power input pin is connected to a 5 Volt power supply via the three-wire port on the VEX V5 microprocessor. The RS485 signal emitted through the smart port on the VEX V5 microprocessor is decoded by a MAX485 chipset. Power to the RS485 decoder is supplied via the 3.3 V power output pin on the ESP32. The RS485 decoder used in this study can be set to transmit or receive a signal based on the state of the DE and RE pins. Bidirectional communication is achieved by driving these pins using the ESP32. When a signal is received via the ESP-NOW protocol, the ESP32 sets the state of the RS485 to transmit mode. A message is then sent via RS485 to the VEX V5 microprocessor. The ESP then sets the RSS485 decoder back to receive mode so that the VEX V5 microprocessor may transmit messages through the ESP32 back to the computer system. The wiring diagram showing the connection between the ESP32, RS485 decoder, and VEX V5 microprocessor is shown in Fig. 2.

ESP32 communication code

The Arduino IDE was used to code a communication link between two ESP32 microcontrollers using the ESP-NOW communication protocol. The ESP32 connected to the VEX V5 microprocessor continuously transmits system status data via the ESP-NOW protocol at a fixed interval to another ESP32. The ESP32 connected via USB to the computer receives the data and repeats them to the computer via a serial link. Commands sent by the computer through the USB are echoed in the opposite direction back to the ESP32 connected to the VEX V5 microprocessor.

VEX code

A code template written in C++ is provided which runs the communication link on the VEX V5 Microprocessor. It provides a starting point for users interested in modifying and extending the capabilities of this system. The code automatically detects the presence of VEX peripherals and transmits data from each of these peripherals at a regular interval to the computer. The code also provides a mechanism for the VEX V5 microprocessor to recognize custom commands sent from a computer and to run predefined tests in response to computer instruction. The code can be edited using Visual Studio Code with the VEX VS Code Extension installed and then downloaded to the VEX V5 microprocessor.

Python GUI

A graphical user interface, Fig. 3, was written in python using the Tkinter graphical user interface (GUI) library. This library is included in the standard installation of Python. The data received from the VEX V5 microprocessor are plotted in real time using MatplotLib (Hunter, 2007). This data can be stored or retrieved from files on the hard drive, plotted, and analyzed using the python code. The python code can also capture and display images from an OpenMV camera connected via USB to the computer. It can be used to perform machine learning operations on the images and send commands back to the VEX V5 microprocessor. Users can modify the code to create hybrid control systems that are running some processing on the computer platform and some on the VEX V5 microprocessor while sharing data and commands between the two. The Python multiprocessor module is used to isolate GUI operations from data transmission, resulting in a smooth GUI and much improved responsiveness when compared to a single-processor algorithm.

Trajectory analysis

Repeatability and reliability are a recurrent problem in middle- and high-school robotics competitions and in the educational curriculum. Understanding how design choices and control algorithms influence the reproducibility of a particular motion is particularly useful. In this section, a simple six wheel drivetrain powered by two motors, as shown in Fig. 12A, is created and outfitted with a VEX GPS sensor. The VEX GPS sensor allows the drivetrain’s position to be determined within a competition field. Though the name implies this sensor relies on GPS, it is in fact vision based and uses images of an encoded strip placed along the field perimeter to determine the location.

The drivetrain was programmed to travel around the field in a prescribed square pattern, as shown in Fig. 12B with segment definitions described in Table 7. Tests are triggered by the GUI that runs on the computer and are parameterized, allowing students to adjust travel distances. Sensor data are sent back at a rate of 20 readings per second to the computer. This allows students to review real-time data that give the robot’s position and orientation, which can be graphed and analyzed in a variety of ways. A graph displaying the real-time position and orientation of the drivetrain is provided. Data from multiple trials can be saved and graphed simultaneously giving a clear graphical indication of the drivetrain repeatability, as shown in Fig. 12C, where the position of the drivetrain completing the square path shown in Fig. 12B during eight trial runs are graphed simultaneously.

Beyond the visualization of trajectories, a jupyter notebook is provided which can be used to evaluate the repeatability using saved data in more detail. The VEX code that runs the drivetrain is built so that each segment of a trajectory is labeled independently. For example, segment 1 involves moving the drivetrain along a straight path for a given distance, while segment 2 involves turning the robot to the right by 90 degrees. The position and orientation of the drivetrain at the endpoint of each segment is plotted in Fig. 13A for all eight trials. Figure 13B displays the position and orientation of the drivetrain at the end of each segment averaged over all eight trials, as well as an error bar that indicates the maximum and minimum position. These data can be used to assess the reliability and repeatability of the drivetrain and compare the performance of design changes. The provided jupyter notebook can also be used to tabulate all of the trajectory averages as shown in Table 8.

Tuning a PID controller

A proportional-integral-derivative controller, or PID controller, is a control feedback loop that corrects for deviations from the desired value. PID controllers are commonly used to precisely control parameters such as temperature, speed, or pressure. In robotics, PID controllers can be used to control the velocities of a drivetrain, for lifting objects, and for spinning flywheels. For example, the angular velocity of a flywheel might need to be precisely controlled to propel a projectile a particular distance. The controller would adjust the flywheel motor power as a function of the difference between the target flywheel angular velocity and the actual angular velocity. Here the angular velocity could be determined via a rotation sensor on the flywheel axle.

Another common example would be lifting an object to a desired height. The experimental setup shown in Fig. 14A illustrates the case where an arm has been fitted with a circular disc weight. The arm is driven directly through a motor at the base. A rotation sensor is attached to the axle passing through the arm and motor to monitor the arm configuration. The arm is to be lifted to a particular target angle. Motor power can be determined using a PID controller that responds to the difference, $e(t)$, between the target arm angle, ${\theta }_o$ and the actual arm angle measured by the rotation sensor, $\theta (t)$ where the difference is defined by:

(6) $$e(t)=\theta (t)-{\theta }_o$$

A PID controller determines the output response as a function of a term that is proportional to the difference between the target and the actual value, a term that is proportional to the integral of this difference, and a term that is proportional to the derivative of this difference.

The equation for the instantaneous motor output, $u(t)$, is then defined as follows:

(7) $$u(t)=K_{p}e(t)+K_i{\displaystyle \int e(t)dt+K_d\frac{de(t)}{dt}}$$where $K_p$ is the coefficient of the proportional term, $K_i$ is the coefficient of the integral term and $K_d$ is the coefficient of the derivative term.

Each of the three coefficients $K_p$, $K_i$, and $K_d$ must be found for each application independently, as the optimal values depend on the characteristics of the system being driven. This process is known as tuning, and is often done by entering approximate coefficient values into the equation, observing the system response, and then further refining the coefficients.

Tuning is greatly simplified when real time data logging is possible. Students can observe the response of the system as they modify the PID parameters to determine if a particular set of coefficients is effective or not. Figure 14B displays the system response with various PID parameters and a target angle of 45 degrees.

Real time data logging shows much more accurately whether a particular set of coefficients result in an effective controller. Figure 14B illustrates that a controller with only a proportional term never reaches the target angle for this system. The proportional component of the controller increases the response when error values are high and decreases the response when error values are low according to tuned coefficients. Although the proportional component alone is an effective controller in many cases, it is not suitable for lifting applications where the output power may never be large enough to hold the weight at the target angle. The addition of an integral term allows the system to settle near the target value, and the addition of a derivative term eliminates the oscillations by slowing down the response when the measured value changes too quickly. This allows the system to settle at the target value much more quickly. All of this information is captured visually as the student modifies parameters of the PID model.