Modular automatic design of collective behaviors for robots endowed with local communication capabilities

Chocolate

Chocolate (Francesca et al., 2015) generates probabilistic finite-state machines by assembling parametric modules—behaviors and conditions—that exploit the capabilities of the e-puck platform, as formally defined by the reference model RM1.1; further details are available in Hasselmann et al. (2018). It is important to notice that Chocolate’s modules are strictly part of the definition of Chocolate itself. They were defined once and for all in a mission-agnostic way (Francesca et al., 2014b). They are not supposed to be modified or integrated by other modules when Chocolate is applied to a specific mission. Yet, these modules have parameters that impact their functioning and that are automatically fine-tuned on a per-mission basis. In Chocolate, low-level behaviors are:

1. exploration: the robot performs a random walk, while avoiding obstacles;

2. stop: the robot stands still;

3. phototaxis: the robot goes towards the light source, if perceived;

4. anti-phototaxis: the robot goes in the opposite direction;

5. attraction-to-neighbors: the robot goes towards its neighboring peers;

6. repulsion-from-neighbors: the robot goes in the opposite direction.

Conditions are:

1. black-floor: change state if floor is black;

2. white-floor: change if it is white;

3. gray-floor: change if it is gray;

4. neighbor-count: change if sufficiently many neighboring peers are perceived;

5. inverted-neighbor-count: change if they are sufficiently few;

6. fixed-probability: change state with a fixed probability.

For details on these modules and their tunable parameters, we refer the reader to the original publication in which the modules were introduced (Francesca et al., 2014b).

The topology of the probabilistic finite-state machine, the modules to be included and their parameters are defined by an optimization process. The space of the probabilistic finite-state machines that Chocolate can possibly generate is constrained to those comprising at most four states having each at most four outgoing edges. As an optimization algorithm, Chocolate uses the implementation of iterated F-race provided by the R package irace (López-Ibáñez et al., 2016), with its default parameters. Iterated F-race is based on F-race (Birattari et al., 2002), a racing procedure (Maron & Moore, 1997) originally proposed for the automatic configuration of stochastic optimization algorithms and metaheuristics (Hoos & Sttzle, 2004; Gendreau & Potvin, 2019). In F-race, a set of candidate solutions are randomly sampled and then sequentially evaluated, over a set of test cases, to eventually select the most suitable one. Along the sequential evaluation of candidate solutions, a Friedman test is repeatedly performed to identify candidate solutions that perform significantly worse than at least another one. These solutions are discarded so that the evaluation can focus on the best ones. The algorithm terminates when only one candidate solution remains or when a predefined budget of evaluations is depleted. Iterated F-race consists of multiple iterations of F-race. After the first iteration, each subsequent one operates on a set of candidate solutions that are sampled around those that the previous iteration selected as the best ones. The algorithm terminates when a predefined budget of evaluations is depleted. Within the optimization process, simulations are performed using the ARGoS3 simulator (Pinciroli et al., 2012), version beta 48, together with the argos3-epuck library (Garattoni et al., 2015). ARGoS3 is a modular multi-physics robot simulator specifically conceived to simulate robot swarms. Chocolate uses ARGoS3’s 2D dynamic physics engine to simulate the robots and the environment. The argos3-epuck library provides low-level implementations of the sensors and actuators of the e-puck robot with fine control on noise levels for all actuators and sensors. ARGoS3 and the argos3-epuck library inject a realistic level of sensor and actuator noise in all simulations as suggested by Miglino, Lund & Nolfi (1995) as a good practice for reducing the impact of the reality gap.

As Chocolate is unable to produce behaviors that leverage communication, it will likely fail to be effective on the missions we consider in the following of the paper. We include it in the analysis to act as a baseline, so as to appraise the importance of communication.

EvoCom

EvoCom is an automatic design method that extends EvoStick (Francesca et al., 2014a) by adding communication capabilities. EvoStick is a standard neuro-evolutionary robotics method. It was used in Francesca et al. (2014a) as a yardstick against which the authors compared their newly proposed method, but had been previously analyzed in Francesca et al. (2012). EvoStick was then included in other empirical studies (Francesca et al., 2014a; Francesca et al., 2015; Birattari et al., 2010; Ligot & Birattari, 2019). We chose EvoStick as a starting point for our development because, to the best of our knowledge, it is the only neuro-evolutionary method for the design of robot swarms that has so far been studied following the tenets of off-line automatic design (Birattari et al., 2019), as defined in the Introduction. Namely, it is the only neuro-evolutionary design method that has been tested on multiple missions without any mission-specific modification so as to evaluate its ability to generate control software without any human intervention. The goal for EvoCom is to introduce a comparison point with Gianduja, the idea is not to find the best possible neural network topology for a given mission but rather to use a general purpose topology that could work with any given mission.

EvoCom creates a feed-forward fully-connected neural network with no hidden nodes, which comprises 30 input nodes and 3 output nodes. Inputs and outputs are based on the elements of the reference model RM2, see Table 1.

The inputs are: 8 proximity sensors, 8 light sensors, 3 ground sensors, 10 values computed based on data from the range-and-bearing module, and 1 bias. The outputs are: 2 for wheel speed and 1 for broadcasting the message. The proximity sensors, light sensors, and ground sensors directly feed their value, as defined in RM2, to the corresponding inputs of the neural network. The range-and-bearing module determines the values of 10 inputs: 5 inputs concern the detection of neighboring peers and are computed on the basis of n and V; the other 5 concern messages received and are computed on the basis of b and V_b. Also in the case of n, V, b, and V_b, the definition is given in RM2. One of the five inputs devoted to neighboring peers equals $z\left(n\right)=1-\frac{2}{1+e^n}$; the other four are the scalar projection of V onto four unit vectors that point at 45°, 135°, 225°, and 315° with respect to the front of the robot. One of the five inputs devoted to messages equals $z\left(b\right)=1-\frac{2}{1+e^b}$; the other four are the scalar projection of V_b onto the aforementioned four unit vectors. The two output nodes encoding wheel speed range in [−0.12,0.12] m/s . The one devoted to the message takes a binary value: 1 if the message should be broadcast, 0 otherwise. The activation of the output neurons is computed as the weighted sum of all input units plus a bias term, filtered through a logistic function. All synaptic weights are real values in the range [ − 5, 5] and are optimized with an evolutionary algorithm. The initial population comprises 100 randomly generated individuals. At each iteration, each individual is evaluated 10 times in simulation. A new population is obtained via elitism and mutation: the best 20 individuals are kept unmodified, while the other 80 are obtained by mutation of the 20 best ones. The evolutionary algorithm stops when a predefined budget of evaluations is depleted. All simulations are performed using ARGoS3 and the argos3-epuck library under the same conditions and with the same noise levels as in Chocolate.

Gianduja

As already mentioned, Gianduja is the main method that we propose in this paper. It addresses the limitation of Chocolate regarding local communication: in Chocolate, robots cannot explicitly communicate and are only capable of detecting the presence of peers in their neighborhood. In Gianduja, as this method is based on the same reference model as EvoCom, namely RM2, robots are able to locally broadcast one message (one bit of information) and react to it. Messages are sent via the range-and-bearing module by setting a bit of the ping’s payload. Robots can thus identify their neighbors and estimate their position.

Gianduja generates control software for the e-puck platform, as formally specified by RM2, which is the same reference model adopted in EvoCom. Gianduja operates on eight low-level behaviors. Six are the same of Chocolate, extended with a binary parameter m: if m = 1, the message is broadcast while the behavior is performed; otherwise, it is not. The other two are:

1. attraction-to-message: the robot goes towards messaging peers;

2. repulsion-from-message: the robot goes in the opposite direction.

The direction of the messaging peers is provided by V_b, as defined in RM2. Behavior transitions can be triggered by eight conditions. Six are the same of Chocolate, the other two are:

1. message-count: change state if sufficiently many peers are messaging;

2. inverted-message-count: change if they are sufficiently few.

Under the message-count condition, a state transition happens with probability $z\left(b\right)=\frac{1}{1+e^{\eta \left(\xi -b\right)}}$; under inverted-message-count, the probability is $\bar{z}\left(b\right)=1-z\left(b\right)$. Here, η and ξ are parameters of the behavior to be assigned by the design process.

In all other respects, Gianduja is identical to Chocolate. In particular, the two methods adopt the same optimization algorithm with the same parameters, the same simulator with the same noise levels, and the same constraints on how probabilistic finite-state machines are assembled.

GiandujaE

GiandujaE is derived from Gianduja and differs from it only in the way received messages are handled. GiandujaE does not use conditions message-count and inverted-message-count. The mechanism for handling the number of messages received is embedded in all remaining conditions, which are extended with two parameters: message ID ν and threshold τ. Message ID can take two values: if ν = 0, the condition behaves as in Gianduja; if ν = 1, the condition is enabled and behaves as in Gianduja only if the number of neighboring peers that are messaging is larger than the threshold τ, which can take an integer value between 0 and 20.

In the following of the paper, a letter E in the name of the method indicates that received messages are handled as described above—the letter E is the mnemonic for embedded. In principle, this way of handling messages allows more complex behaviors to be obtained. It should be expected that the conjunction of a condition along with the presence of a message allows probabilistic finite-state machine of simpler structure, without sacrificing capabilities.

Missions

In all missions, robots operate in a dodecagonal arena of 4.91 m². The arena is surrounded by walls. The floor is gray, although some areas may be black or white, depending on the specific mission. Details are provided in the following. All mission are intended to be performed by a swarm of N = 20 robots within T = 120 s.

The three missions are AGGREGATION, STOP, and DECISION. We select them because, a priori, one could imagine that communication would play a different role in each of them. Indeed, it is reasonable to expect that AGGREGATION can be solved without the use of communication but that communication can contribute to increasing performance. On the other hand, it is reasonable to expect that DECISION and STOP strictly require communication to be performed. It is also reasonable to expect that the semantics that the optimization would associate to the message is different for each mission.

AGGREGATION. Two circular spots of diameter 0.6 m mark the floor of the arena: one is white and the other black. They are positioned on the left-hand half of the arena, 0.25 m apart. At the beginning of each run, the robots are randomly positioned in the right-hand half of the arena so that no robot is on the spots—see Figs. 1A, 1D. The robots must aggregate on the white spot as quickly as possible. The black spot is included in the arena to acts as a possible disturbance to the automatic design process.

Performance is measured by the following objective function—the higher, the better: ${\displaystyle C_a=24000-\sum _{t=1}^T\sum _{i=1}^N{I}_i\left(t\right);I_i\left(t\right)=\left\{\begin{array}{cc}0,\hfill & \text{if robot}i\text{is on the white spot};\hfill \\ 1,\hfill & \text{otherwise.}\hfill \end{array}\right.}$ Performance is non-negative and its theoretical maximum is 24000.

STOP. One circular white spot with diameter 0.2 m is located near the walls of the arena, in the top-left quadrant. The spot is smaller than in the other missions so that it is more challenging for the robots to find it. At the beginning of each run, the robots are randomly positioned in the right-hand half of the arena so that none of them is on the spot—see Figs. 1B, 1E). The robots must find the spot as quickly as possible and stop right after. Performance is measured by the following objective function—the higher the better: ${\displaystyle C_s=48000-\left(\bar{t}N+\sum _{t=1}^{\bar{t}}\sum _{i=1}^N{\bar{I}}_i\left(t\right)+\sum _{t=\bar{t}+1}^T\sum _{i=1}^N{I}_i\left(t\right)\right);}$ ${\displaystyle I_i\left(t\right)=\left\{\begin{array}{cc}1,\hfill & \text{if robot}i\text{is moving};\hfill \\ 0,\hfill & \text{otherwise;}\hfill \end{array}\right.{\bar{I}}_i\left(t\right)=1-I_i\left(t\right).}$ Here, $\bar{t}$ is the time at which the first robot finds the white spot. In the definition of I_i (and ${\bar{I}}_i$), a robot is moving if it traveled more than 5 mm in the last 100 ms. We adopted this definition to avoid penalizing EvoCom which is unable to generate behaviors in which robots can stop still, due to the way the outputs of the neural network are encoded. Performance is non-negative and its theoretical maximum is 48000.

DECISION. One circular spot with diameter 0.6 m is located in the center of the arena. In each experimental run, the spot can be either black or white, with equal probability. A light source is positioned outside the arena, at its right. At the beginning of each run, the robots are randomly positioned in the arena—see Figs. 1C, 1F. The robots must gather in the right-hand half of the arena if the spot is white, or in the left-hand side if it is black. The performance measure—the higher, the better—is: ${\displaystyle C_D=24000-\sum _{t=1}^T\sum _{i=1}^N{I}_i\left(t\right);}$ ${\displaystyle I_i\left(t\right)=\left\{\begin{array}{cc}0,\hfill & \text{if robot}i\text{is in the correct half of the arena;}\hfill \\ 1,\hfill & \text{otherwise.}\hfill \end{array}\right.}$ Performance is non-negative and its theoretical maximum is 24000.

Protocol

We compare four design methods on three missions. All experiments involve a swarm of 20 e-puck robots. For each mission, each method is executed 14 times so as to obtain 14 instances of control software. Each execution is allowed a budget of 200000 simulation runs. We evaluate each method by running the 14 instances of control software it produced once in simulation and once in pseudo-reality—explanation follows¹. For Chocolate, Gianduja, and EvoCom, the control software produced is run once also on real robots. The order of the experimental runs is randomized to avoid any bias. The performance of the real robots is automatically computed by a tracking system based on an overhead camera (Stranieri et al., 2013). The simulation model used as a pseudo-reality for the evaluations of the control software produced is available as Supplemental Information (Hasselmann & Birattari, 2019).

Aggregate results

The aggregate results obtained in real-robot experiments are presented in Fig. 3; those obtained in pseudo-reality are presented in Fig. 4. Across all missions, the Friedman test indicates that, on the robots, the control software generated by Gianduja performs significantly better than the one of Chocolate and EvoCom. In pseudo-reality, Gianduja performs significantly better than EvoCom and Chocolate. All in all, on the aggregate data, pseudo-reality provides a correct prediction of the relative performance in robot experiments.

The aggregate results over the three missions confirm that communication improves performance and that Gianduja leverages it effectively yielding significantly better results than EvoCom. At least in the setting considered in our experiments, EvoCom was unable to evolve effective behaviors. The performance of the behavior produced by EvoCom is worse than the one produced by Chocolate, which by construction does not use communication. This can be considered as a major failure. The results on the three missions considered fail to provide any evidence that GiandujaE is better than Gianduja.

Gianduja2

Gianduja2 extends Gianduja by allowing the communication of two bits of information, which amounts to the ability to broadcast and react to 2² − 1 = 3 different messages. The term −1 accounts for the possibility of not sending any message. Gianduja2 operates on the same low-level behaviors of Gianduja, with only a minor modification. Here, the parameter m can take four integer values: {0, …, 3}. If m = 0, the robot does not broadcast any message—or, more precisely, it broadcasts via its range-and-bearing module the null message that only allows its neighboring peers to detect its presence, without conveying any extra signification. If m = {1, 2, 3}, the robot broadcasts the corresponding message, to convey the meaning that the optimization process will have automatically associated with the message itself, for the specific mission at hand. Moreover, in Gianduja2, the two behaviors attraction-to-message and repulsion-from-message take one additional parameter $\overline{m}\in \left\{1,2,3\right\}$, which specifies the message to whose senders the robot is attracted, or from whose senders it is repelled. Finally, the two conditions message-count and inverted-message-count take one additional parameter $\overline{m}\in \left\{1,2,3\right\}$, which specifies the message to which the condition should be sensitive. The transition probability is computed via z(⋅), as defined in Gianduja.

Gianduja2E

Gianduja 2E derives from Gianduja2, in the same sense in which GiandujaE derives from Gianduja. Like GiandujaE, Gianduja 2E does not use the conditions message-count, and inverted-message-count. Also in Gianduja 2E, the conditions black-floor, white-floor, gray-floor, neighbor-count, inverted-neighbor-count, and fixed-probability are extended with two parameters: message ID ν and threshold τ. Parameter ν can take four different values: if ν = 0, the condition behaves as in Gianduja2; if ν = {1, 2, 3}, the condition is enabled and behaves as in Gianduja2 if the number of neighbors sending message 1, 2, 3, respectively, is greater than τ.

EvoCom2

EvoCom2 extends EvoCom by allowing the communication of two bits of information. With respect to EvoCom, EvoCom2 has five additional inputs and one additional output: the network totalizes 35 inputs and 4 output nodes. The five additional inputs are used to encode information concerning the messages received. In EvoCom, five inputs are devoted to this; in EvoCom2, they are ten. Two are computed on the basis of the number of neighboring peers that broadcast the various messages available, and the remaining eight on the basis of the direction from which messages arrive. The additional output is used to encode the broadcast state. In EvoCom, one output is devoted to this; in EvoCom2, they are two.

Before we can formally describe how these inputs and outputs are computed, we need to define some notation. In EvoCom2, messages are identified via a binary representation. Table 3 provides a mapping between a message ID h and its binary representation. Let us define the function bit_p(h) that returns TRUE/1, if the p-th bit of the binary representation of h is set; and FALSE/0, otherwise. In EvoCom2, p ∈ {0, …, ℓ − 1}, with ℓ = 2. Bit p = 0 is the less significant one. For p ∈ {0, …, ℓ − 1}: let γ_p = ∑_h:phb_(h) be the number of neighboring peers broadcasting any message h whose binary representation has the p-th bit set; and let Γ_p = ∑_h:phV_(h), be the composition of vectors V_h concerning any message h whose binary representation has the p-th bit set.

With this notation, we can represent the ten inputs devoted to encoding information concerning the messages received: two are $z\left({\gamma }_p\right)=1-\frac{2}{1+e^{{\gamma }_p}}$, with p ∈ {0, …, ℓ − 1}; eight are the projections of Γ_p, with p ∈ {0, …, ℓ − 1}, onto the unit vectors pointing at 45°, 135°, 225°, and 315° with respect to the front of the robot. The two outputs determine which message should be broadcast according to the binary representation given in Table 3.

Gianduja3

Gianduja3 extends Gianduja2, and eventually Gianduja, by allowing the communication of three bits of information, which amounts to the ability to broadcast and react to 2³ − 1 = 7 different messages. The term −1 accounts for the possibility of not sending any message. Gianduja3 is defined as Gianduja2, with the difference that the parameter m can take eight integer values, {0, …, 7}; and the parameter $\overline{m}$ can take seven: {1, …, 7}.

Gianduja3E

Gianduja 3E derives from Gianduja3, in the same sense in which GiandujaE and Gianduja 2E derive from Gianduja and Gianduja2, respectively. Gianduja 3E is defined as Gianduja 2E, with the difference that the parameter ν can take eight integer values, {0, …, 7}.

EvoCom3

EvoCom3 extends EvoCom2, and eventually EvoCom, by allowing the communication of three bits of information. It is defined as EvoCom2, with ℓ = 3. Here, the total number of inputs is 40 and the total number of output is 5. The number of inputs used to encode the number of neighboring peers that broadcast the various messages available is ℓ = 3. The number of inputs used to encode the direction from which messages arrive is 4ℓ = 12. The number of outputs used to encode the broadcast state is ℓ = 3.

Missions

In all three missions, N = 20 robots operate in the same dodecagonal arena described in Study A. As in Study A, the amount of time available to the robots for performing each mission is T = 120 second. The three missions are AGGREGATION, STOP, and DECISION. They were selected because we thought that, to be successfully performed, they require the use of multiple messages with different semantics. In all three missions, there is a beacon that can broadcast a message to which robots must react. The role of the beacon is played by an e-puck robot that does not move throughout the experiment. It is not considered to be part of the swarm.

BEACON AGGREGATION. Two circular spots of diameter 0.6 m mark the floor of the arena: one is white and the other black. They are positioned on the left-hand half of the arena, separated by 0.25 m—see Fig. 5A. A beacon positioned between the two spots broadcasts either message 1 or 2 for the whole duration of a run. The message to be broadcast is selected randomly with equal probability at the beginning of each run. At the beginning of each run, the robots are randomly positioned in the right-hand half of the arena so that no robot is on either spot. Robots must aggregate on the black spot, if the beacon broadcasts 1; on the white one, if it broadcasts 2.

Performance is measured by the following objective function—the higher the better: ${\displaystyle C_{BA}=24000-\sum _{t=1}^T\sum _{i=1}^N{I}_i\left(t\right);}$ ${\displaystyle I_i\left(t\right)=\left\{\begin{array}{cc}0,\hfill & \text{if robot}i\text{is on the correct spot};\hfill \\ 1,\hfill & \text{otherwise.}\hfill \end{array}\right.}$ Performance is non-negative and its theoretical maximum is 24000.

BEACON STOP. The arena’s floor is gray and a beacon is positioned in the middle of the arena—see Fig. 5B. The beacon broadcasts message 1 or 2 after a time uniformly sampled in [40 s; 60 s]—the message to be broadcast is selected randomly with equal probability at every run. At the beginning of each run, the robots are randomly positioned in the arena. Robots must stop as soon as the beacon starts broadcasting either one of the two messages.

Performance is measured by the following objective function—the higher the better: ${\displaystyle C_{BS}=24000-\left(\sum _{t=1}^{\bar{t}}\sum _{i=1}^N{\bar{I}}_i\left(t\right)+\sum _{t=\bar{t}+1}^T\sum _{i=1}^N{I}_i\left(t\right)\right);}$ ${\displaystyle I_i\left(t\right)=\left\{\begin{array}{cc}1,\hfill & \text{if robot}i\text{is moving};\hfill \\ 0,\hfill & \text{otherwise;}\hfill \end{array}\right.{\bar{I}}_i\left(t\right)=1-I_i\left(t\right).}$ In this equation, $\bar{t}$ is the time at which the beacon starts broadcasting a message. In the definition of I_i (and ${\bar{I}}_i$), a robot is considered moving if it traveled more than 5 mm in the last 100 ms. This accounts for the inability of EvoCom to generate behaviors in which robots can stop still. Performance is non-negative and its theoretical maximum is 24000.

BEACON DECISION. The floor of the right-hand quarter of the arena is black; the rest is gray. A light source is positioned outside the arena on its right-hand side—see Fig. 5C. A beacon is positioned at the middle of the interface between the black and the gray parts and acts as a timed trigger. At the beginning of a run, the beacon starts broadcasting either message 1 or 2—randomly selected with equal probability. After 60 S, the beacon switches to the other message. At the beginning of each run, the robots are randomly positioned in the arena.

Robots must position themselves either on the black or on the gray part of the arena, depending on the message broadcast by the beacon: on the black, if the message is 1; on the gray, if it is 2. The performance of the robots is measured by the following objective function—the higher, the better: ${\displaystyle C_{BD}=24000-\sum _{t=1}^T\sum _{i=1}^N{I}_i\left(t\right);}$ ${\displaystyle I_i\left(t\right)=\left\{\begin{array}{cc}0,\hfill & \text{if robot}i\text{is in the correct side of the arena};\hfill \\ 1,\hfill & \text{otherwise.}\hfill \end{array}\right.}$ Performance is non-negative and its theoretical maximum is 24000.

Protocol

The protocol is similar to the one of Study A. We compare six design methods on three missions. All experiments involve a swarm of 20 e-puck robots. For each mission, each method is executed 14 times so as to obtain 14 instances of control software. Each execution is allowed a budget of 200000 simulation runs. We evaluate each method by running the 14 instances of control software it produced once in simulation and once in pseudo-reality (Ligot & Birattari, 2019).

Aggregate results

Figure 7 reports the aggregate results in pseudo-reality. They confirm that Gianduja {2,2E,3,3E} perform significantly better than EvoCom. The results also show that the addition of a third bit of data is not needed to perform the three missions considered: indeed, Gianduja {2,2E} perform better than Gianduja {3,3E}. Nonetheless, it is interesting to observe that Gianduja could handle nicely the increased search space resulting from the addition of the third bit. Although the performance of Gianduja3 (Gianduja 3E) is lower than the one of Gianduja2 (Gianduja 2E), the drop is relatively small: in Table 4, we report the relative median performance of Gianduja {3,3E} with respect to Gianduja {2,2E} and we compare it with the one of EvoCom2.