AlphaDDA: strategies for adjusting the playing strength of a fully trained AlphaZero system to a suitable human training partner

The value of the board state

AlphaDDA obtains the value of the board state from the DNN. The DNN estimates the value and the move probability from the board state. The input of the DNN comprises the T board states immediately before placing the disc at the nth turn. The value at the nth turn $v_n$ is a continuous variable ranging from 1 to $-1$. When $v_n$ is closer to the disc color of AlphaDDA $c_{\mathrm{A}\mathrm{l}\mathrm{p}\mathrm{h}\mathrm{a}\mathrm{D}\mathrm{D}\mathrm{A}}$, the DNN assumes that the winning probability of AlphaDDA is higher. $c_{\mathrm{A}\mathrm{l}\mathrm{p}\mathrm{h}\mathrm{a}\mathrm{D}\mathrm{D}\mathrm{A}}$ is 1 and $-1$ when AlphaDDA is the first player and the second player, respectively. In this study, AlphaDDA uses the mean of the value ${\overline{v}}_n$ defined as

(1) $${\overline{v}}_n={\displaystyle \frac{1}{N_h}\sum _{i=0}^{N_h-1}(v_{n-i}),}$$

where $N_h$ is the number of the values. If $n-N_h<0$, $N_h=n$. AlphaDDA adjusts its skill to that of an opponent by changing its skill according to ${\overline{v}}_n$.

AlphaDDA1

AlphaDDA1 balances its skill and that of the opponent by changing the number of simulations $N_{\mathrm{s}\mathrm{i}\mathrm{m}}({\overline{v}}_n)$ according to the value ${\overline{v}}_n$. AlphaDDA1 is AlphaZero with a variable number of simulations depending on ${\overline{v}}_n$. The quality of a move selected by the MCTS depends on $N_{\mathrm{s}\mathrm{i}\mathrm{m}}({\overline{v}}_n)$. Thus, AlphaDDA1 can change its skill according to ${\overline{v}}_n$. The number of simulations $N_{\mathrm{s}\mathrm{i}\mathrm{m}}({\overline{v}}_n)$ is defined as

(2) $$N_{\mathrm{s}\mathrm{i}\mathrm{m}}({\overline{v}}_n)=\lceil {10}^{-A_{\mathrm{s}\mathrm{i}\mathrm{m}}({\overline{v}}_n{c}_{\mathrm{A}\mathrm{l}\mathrm{p}\mathrm{h}\mathrm{a}\mathrm{D}\mathrm{D}\mathrm{A}}+B_{\mathrm{s}\mathrm{i}\mathrm{m}0})}\rceil ,$$

(3) $$N_{\mathrm{s}\mathrm{i}\mathrm{m}}({\overline{v}}_n)\leftarrow \{\begin{array}{cc}{N}_{\mathrm{m}\mathrm{a}\mathrm{x}}\hfill & \mathrm{i}\mathrm{f}{N}_{\mathrm{s}\mathrm{i}\mathrm{m}}({\overline{v}}_n)>N_{\mathrm{m}\mathrm{a}\mathrm{x}}\hfill \\ N_{\mathrm{s}\mathrm{i}\mathrm{m}}({\overline{v}}_n)\hfill & \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}\hfill \end{array},$$

where $A_{\mathrm{s}\mathrm{i}\mathrm{m}}$, $B_{\mathrm{s}\mathrm{i}\mathrm{m}0}$, and $N_{\mathrm{m}\mathrm{a}\mathrm{x}}$ are hyperparameters. $\lceil \cdot \rceil$ is a ceiling function. $N_{\mathrm{s}\mathrm{i}\mathrm{m}}({\overline{v}}_n)$ is always greater than or equal to 1 because $\lceil {10}^x\rceil \ge 1$, where $x\in R$. The base is not limited to 10 because if $N_{\mathrm{s}\mathrm{i}\mathrm{m}}({\overline{v}}_n)$ increases exponentially, that is sufficient.

$N_h$, $A_{\mathrm{s}\mathrm{i}\mathrm{m}}$, $B_{\mathrm{s}\mathrm{i}\mathrm{m}0}$, and $N_{\mathrm{m}\mathrm{a}\mathrm{x}}$ are set as the values shown in Table 1. These values are determined using a grid search such that the difference in the win and loss rates of AlphaDDA1 against the other AIs is minimal. The details of the grid search are shown in “Grid search”.

AlphaDDA2

AlphaDDA2 changes its skill by damaging the DNN using dropout. Dropout refers to ignoring some units selected at random and is generally used while training a DNN to avoid overfitting. In AlphaDDA2, dropout is used to damage the DNN. In other words, AlphaDDA2 is AlphaZero with the DNN damaged by dropout. The damaged DNN provides a more inaccurate output. Consequently, AlphaDDA2 becomes weaker owing to the inaccurate output. A larger probability of dropout causes more inaccurate outputs and makes AlphaDDA2 weaker.

AlphaDDA2 changes the probability of dropout $P_{\mathrm{d}\mathrm{r}\mathrm{o}\mathrm{p}}$ according to the value calculated by the DNN with $P_{\mathrm{d}\mathrm{r}\mathrm{o}\mathrm{p}}=0$. The value is calculated by the DNN not damaged by the dropout. The dropout is inserted into the fully connected layers in the value and policy heads, as shown in Fig. 2. $P_{\mathrm{d}\mathrm{r}\mathrm{o}\mathrm{p}}$ changes with the mean value ${\overline{v}}_n$ as follows:

(4) $$P_{\mathrm{d}\mathrm{r}\mathrm{o}\mathrm{p}}(\overline{v})=A_{\mathrm{d}\mathrm{r}\mathrm{o}\mathrm{p}}({\overline{v}}_n+P_{\mathrm{d}\mathrm{r}\mathrm{o}\mathrm{p}0}),$$

(5) $$P_{\mathrm{d}\mathrm{r}\mathrm{o}\mathrm{p}}(v)\leftarrow \{\begin{array}{cc}0\hfill & \mathrm{i}\mathrm{f}{P}_{\mathrm{d}\mathrm{r}\mathrm{o}\mathrm{p}}({\overline{v}}_n)<0\hfill \\ P_{\mathrm{m}\mathrm{a}\mathrm{x}}\hfill & \mathrm{i}\mathrm{f}{P}_{\mathrm{d}\mathrm{r}\mathrm{o}\mathrm{p}}({\overline{v}}_n)>P_{\mathrm{m}\mathrm{a}\mathrm{x}}\hfill \\ P_{\mathrm{d}\mathrm{r}\mathrm{o}\mathrm{p}}({\overline{v}}_n)\hfill & \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}\hfill \end{array},$$

where $A_{\mathrm{d}\mathrm{r}\mathrm{o}\mathrm{p}}$ and $P_{\mathrm{d}\mathrm{r}\mathrm{o}\mathrm{p}0}$ are hyperparameters.

$N_h$, $A_{\mathrm{d}\mathrm{r}\mathrm{o}\mathrm{p}}$, and $P_{\mathrm{d}\mathrm{r}\mathrm{o}\mathrm{p}0}$ are set as the values shown in Table 2. These values are determined using a grid search such that the difference in the win and loss rates of AlphaDDA2 against the other AIs is minimal. The details of the grid search are shown in “Grid search”. $P_{\mathrm{m}\mathrm{a}\mathrm{x}}$ is set as 0.95. This limit of $P_{\mathrm{d}\mathrm{r}\mathrm{o}\mathrm{p}}(v)$ is required to avoid the outputs of all the units in the DNN being 0.

AlphaDDA3

AlphaDDA3 adjusts its skill to that of an opponent using the new UCT score based on two ideas. The first is that AlphaDDA3 assumes that an opponent generates a board state with a mean value of ${\overline{v}}_n$ because the value depends on the player’s skill. For example, if an opponent is weaker, it continues to select a worse move, and the state remains bad for it. Thus, the DNN continues to predict that AlphaDDA3 beats the opponent. Second, AlphaDDA3 selects a better or a worse move if the value is further from or closer to $c_{\mathrm{A}\mathrm{l}\mathrm{p}\mathrm{h}\mathrm{a}\mathrm{D}\mathrm{D}\mathrm{A}}$, respectively. A value further from $c_{\mathrm{A}\mathrm{l}\mathrm{p}\mathrm{h}\mathrm{a}\mathrm{D}\mathrm{D}\mathrm{A}}$ means that the DNN predicts that the opponent beats AlphaDDA3 and vice versa. The UCT score based on these two ideas is denoted by

(6) $$U(s_t,a)=W(s_t,a)/N(s_t)+C\sqrt{2\ln (N(s_t)+1)/(n(s_t,a)+1)},$$

(7) $$W(s_t,a)\leftarrow W(s_t,a)-|v(s_t)+{\overline{v}}_{n}c(s_t,a)c_{\mathrm{A}\mathrm{l}\mathrm{p}\mathrm{h}\mathrm{a}\mathrm{D}\mathrm{D}\mathrm{A}}|,$$

where $W(s_t,a)$ is the cumulative value, $N(s_t)$ is the visit count of the node $s_t$, $n(s_t,a)$ is the visit count of the edge $(s_t,a)$, C is the exploration rate, $v(s_t)$ is the value of state $s_t$, $c_{\mathrm{A}\mathrm{l}\mathrm{p}\mathrm{h}\mathrm{a}\mathrm{D}\mathrm{D}\mathrm{A}}$ is the disc color of AlphaDDA3, and $c(s_t,a)$ is the disc color of the player for $(s_t,a)$. The absolute value of the difference between $v(s_t)$ and ${\overline{v}}_n$ is subtracted from $W(s_t,a)$ if the player for $(s_t,a)$ is the opponent. This subtraction means that the opponent selects a move such that the board state provides the same value as the mean value ${\overline{v}}_n$. The absolute value of the sum of $v(s_t)$ and ${\overline{v}}_n$ is subtracted from $W(s_t,a)$ if the player for $(s_t,a)$ is AlphaDDA3. The subtraction means that AlphaDDA3 prefers the move in which the sum of $v(s_t)$ and ${\overline{v}}_n$ is 0. In other words, AlphaDDA3 prefers the move in which $v(s_t)$ is $-{\overline{v}}_n$. This preference means that AlphaDDA3 selects a worse move if ${\overline{v}}_n$ is closer to $c_{\mathrm{A}\mathrm{l}\mathrm{p}\mathrm{h}\mathrm{a}\mathrm{D}\mathrm{D}\mathrm{A}}$ and vice versa.

The hyperparameters $N_h$ and C are set as the values shown in Table 3. The values of $N_h$ and C are determined using a grid search such that the difference in the win and loss rates of AlphaDDA3 against the other AIs is minimal. The details of the grid search are shown in “Grid search”.

Games

In this study, AlphaDDAs play Connect4, 6x6 Othello, and Othello. These games are two-player, deterministic, and zero-sum games of perfect information. Connect4 was published by Milton and is a two-player connection game. The players drop discs into a $7\times 6$ board. When a player forms a horizontal, vertical, or diagonal line of its four discs, the player wins. Othello (Reversi) is also a two-player strategy game that uses an $8\times 8$ board. In Othello, the disc is white on one side and black on the other. Players take turns placing a disc on the board with their assigned color facing up. During a play, any discs of the opponent’s color are turned over to the current player’s color if they are in a straight line and bounded by the disc just placed and another disc of the current player’s color. 6x6 Othello is Othello using a 6x6 board.

Opponents

In this study, eight AIs, AlphaZero, MCTS1, MCTS2, MCTS3, MCTS4, Minimax1, Minimax2, and Random, were used to evaluate the performance of AlphaDDA. AlphaZero was trained as described in “Alphazero”. MCTS1, MCTS2, MCTS3, and MCTS4 are MCTS methods with 300, 100, 200, and 50 simulations, respectively. In all the MCTSs, the child nodes of the node at five visits are opened. Minimax1 and Minimax2 are Minimax methods that select a move using the minimax algorithm based on an evaluation table. The details of MCTS and Minimax are described in “Monte carlo tree search” and “Minimax algorithm”, respectively. Random selects a move from valid moves uniformly at random. AlphaZero, MCTS1, MCTS2, Minimax1, and Random are used for the grid search (“Grid search”), calculation of Elo rating (“Elo rating of board game playing algorithms”), and investigation of the dependence of the strength of AlphaZero on the parameters (“Dependence of the strength of AlphaZero on its parameters” and “Dependence of Elo rating on the probability of dropout”). All the AIs are used to evaluate the performances of AlphaDDAs. In particular, MCTS3, MCTS4, and Minimax2 are only used to evaluate the performance of AlphaDDAs and not used for the grid search.

Software

The Python programming language and its libraries, NumPy for linear algebra computation and PyTorch for deep learning, were used. The program codes of AlphaDDA, AlphaZero, MCTS, Minimax, and the board games are available on GitHub (https://github.com/KazuhisaFujita/AlphaDDA).

Elo rating of board game playing algorithms

This subsection shows the relative strengths of five AI agents: AlphaZero, MCTS1, MCTS2, Minimax1, and Random. The Elo rating is used to evaluate the relative strength of each AI. The Elo rating is calculated through 50 round-robin tournaments between the agents. The Elo ratings of all the agents are initialized to 1,500. The details of the Elo rating calculation are provided in “Elo rating”.

Table 4 shows the Elo ratings of the agents for Connect4, 6x6 Othello, and Othello. AlphaZero is the strongest player. This result shows that AlphaZero masters all the games well and is stronger than the other agents. MCTS1 shows the second-best performance. Random is the worst player for all the games.

Dependence of the strength of AlphaZero on its parameters

The skill of AlphaZero depends on the values of its parameters. To effectively change AlphaZero’s skill by varying its parameters, the dependence of the skill on each parameter needs to be investigated. This subsection examines the relationship between the Elo rating of AlphaZero and the values of two parameters: the number of simulations $N_{\mathrm{s}\mathrm{i}\mathrm{m}}$ and $C_{\mathrm{p}\mathrm{u}\mathrm{c}\mathrm{t}}$. $N_{\mathrm{s}\mathrm{i}\mathrm{m}}$ is the number of times a game tree is searched in MCTS and directly affects the decision of a move. $C_{\mathrm{p}\mathrm{u}\mathrm{c}\mathrm{t}}$ is a parameter in the UCT score and affects the tree search. The DNN of AlphaZero with the changed parameter has the same weights as those of AlphaZero used in “Elo rating of board game playing algorithms”. In other words, AlphaZero with the changed parameter is already trained. The opponents are AlphaZero, MCTS1, MCTS2, Minimax1, and Random. The opponents’ Elo ratings were fixed to the values listed in Table 4. AlphaZero with the changed parameter plays 50 games with each opponent (25 games when AlphaZero with the changed parameter is the first player and 25 games when AlphaZero with the changed parameter is the second player).

Figure 3A shows that the Elo rating depends on $N_{\mathrm{s}\mathrm{i}\mathrm{m}}$ for all the games. The Elo ratings increase with $N_{\mathrm{s}\mathrm{i}\mathrm{m}}$ for all the games. For Connect4, the Elo rating quickly increases from 5 to 20 simulations. No significant change in the Elo rating is observed from 40 simulations for Connect4. For 6x6 Othello and Othello, the Elo ratings increase with $N_{\mathrm{s}\mathrm{i}\mathrm{m}}$ and do not change significantly from 200 simulations. These results show that the skill of AlphaZero depends on $N_{\mathrm{s}\mathrm{i}\mathrm{m}}$. However, AlphaZero is not weak enough even though $N_{\mathrm{s}\mathrm{i}\mathrm{m}}=1$ because the Elo rating of AlphaZero is still larger than that of Random when $N_{\mathrm{s}\mathrm{i}\mathrm{m}}=1$.

Figure 3B shows the dependence of the Elo rating on $C_{\mathrm{p}\mathrm{u}\mathrm{c}\mathrm{t}}$. The Elo rating does not change significantly with $C_{\mathrm{p}\mathrm{u}\mathrm{c}\mathrm{t}}$ apart from $C_{\mathrm{p}\mathrm{u}\mathrm{c}\mathrm{t}}=0$ for all the games. This result suggests that we cannot control the skill of AlphaZero by changing $C_{\mathrm{p}\mathrm{u}\mathrm{c}\mathrm{t}}$.

Dependence of Elo rating on the probability of dropout

This subsection examines the change in the skill of AlphaZero with the DNN damaged by dropout. Dropout refers to ignoring units selected at random and is used to render the value and policy inaccurate in this experiment. A dropout was inserted into the head part of the DNN, as shown in Fig. 2. The DNN weights of AlphaZero with dropout are the same as those of AlphaZero used in “Elo rating of board game playing algorithms”. AlphaZero with dropout plays games with AlphaZero, MCTS1, MCTS2, Minimax, and Random. The opponent’s Elo ratings were fixed to the values in Table 4. AlphaZero with dropout plays 50 games with each opponent (25 games when AlphaZero with dropout is the first player and 25 games when AlphaZero with dropout is the second player).

Figure 4 shows the dependence of the Elo rating of AlphaZero with dropout on the probability of dropout. The Elo ratings of AlphaZero with dropout decrease with the probability of dropout. This result shows that we can weaken AlphaZero using dropout. However, AlphaZero with dropout is still stronger than Random, even though the probability of dropout is 0.95.

Evaluation of AlphaDDAs

This subsection tests the ability of AlphaDDAs to adapt to an individual opponent in Connect4, 6x6 Othello, and Othello. AlphaDDAs compete with the AI agents: AlphaZero, MCTS1, MCTS2, MCTS3, MCTS4, Minimax1, Minimax2, and Random. The weights of AlphaDDA and AlphaZero are the same as those of AlphaZero used in “Elo rating of board game playing algorithms”. The win-loss-draw rate is used for an metric to evaluate AlphaDDA’s DDA performances. The win-loss-draw rate is a good metric for deciding whether AlphaDDA can adjust its skill to an opponent. AI’s skill that is appropriate for an opponent is one in which the player has the same chance to either win or lose the game (Demediuk et al., 2017) or the draw rate is 1.

Connect4

Table 5 shows the win, loss, and draw rates of AlphaDDAs against the AI agents for Connect4. The win-loss-draw rate is calculated through 100 games with each AI (50 games when AlphaDDA is the first player and 50 games when AlphaDDA is the second player).

AlphaDDA1 is on par with AlphaZero, MCTS2, and MCTS4. AlphaDDA1 is weaker than MCTS1 and MCTS3 but can adjust its skill against the opponents. AlphaDDA1 is stronger than Minimax1 and Minimax2 but can adjust its skill against the opponents. Thus, AlphaDDA1 achieves DDA against AlphaZero, MCTS1, MCTS2, MCTS3, MCTS4, Minimax1, and Minimax2. AlphaDDA1 can adjust its strength against the AI agents not used for grid search. However, AlphaDDA1 has an extremely high win rate against Random.

AlphaDDA2 is on par with AlphaZero and MCTS2. AlphaDDA2 is weaker than MCTS1 and MCTS3 but can adjust its skill to them. AlphaDDA2 is stronger than MCTS4, and Minimax1 but adjusts its skill to them. AlphaDDA2 defeats Minimax2 and Random. Thus, AlphaDDA2 can adjust its skill to AlphaZero, MCTS1, MCTS2, MCTS3, MCTS4, and Minimax1.

AlphaDDA3 is defeated by MCTS1, MCTS2, MCTS3, MCTS4, Minimax1, and Minimax2. AlphaDDA3 does not win against AlphaZero but has a chance to draw against AlphaZero. AlphaDDA3 is on par with Random. Thus, AlphaDDA3 can adjust its skill to Random.

Tables 6 and 7 show the win, loss, and draw rates of AlphaDDAs against the AI agents for Connect4 when AlphaDDAs are the first and the second players, respectively. These win-loss-draw rates are calculated through 50 games with each AI. AlphaDDA1 for the first player is stronger than for the second player. AlphaDDA1 for the first player defeats Minimax1. This result suggests that AlphaDDA1 for the first player does not adjust its skill to Minimax1. The win-loss-draw rates of AlphaDDA2 for the first player show a similar tendency to those for the second player. AlphaDDA3 for the first player draws against AlphaZero with a probability of about 0.5. However, AlphaDDA3 for the second player is defeated by AlphaZero. This result shows that AlphaDDA3 for the second player cannot adjust its strength to AlphaZero.

6x6 Othello

Table 8 shows the win, loss, and draw rates of AlphaDDAs and the AI agents for 6x6 Othello. The win-loss-draw rate is calculated through 100 games with each AI (50 games when AlphaDDA is the first player and 50 games when AlphaDDA is the second player).

AlphaDDA1 is on par with MCTS3. AlphaDDA1 is weaker than AlphaZero and MCTS1 but can adjust its skill against the opponent. AlphaDDA1 is stronger than MCTS2, MCTS4, and Minimax1 but can adjust its skill against the opponents. AlphaDDA1 defeats Minimax2 and Random. These results show that AlphaDDA1 can adjust its skill to AlphaZero, MCTS1, MCTS2, MCTS3, MCTS4, and Minimax1.

AlphaDDA2 is on par with MCTS3. AlphaDDA2 is weaker than MCTS1 but adjusts its skill to the opponent. AlphaDDA2 is stronger than MCTS2, MCTS4, and Minimax1 but can adjust its skill to the opponents. AlphaDDA2 defeats Minimax2 and Random. These results suggest that AlphaDDA2 can adjust its skill to MCTS1, MCTS2, MCTS3, MCTS4, and Minimax1.

AlphaDDA3 is defeated by AlphaZero, MCTS1, MCTS2, MCTS3, and MCTS4. AlphaDDA3 is on par with Random. AlphaDDA3 is weaker than Minimax1 but can adjust its skill against the opponent. AlphaDDA3 is stronger than Minimax2 but has a chance to draw against the opponent. Thus, AlphaDDA3 adjusts its skill to Minimax1 and Random.

Tables 9 and 10 show the win, loss, and draw rates of AlphaDDAs against the AI agents for 6x6 Othello when AlphaDDAs are the first and the second players, respectively. These win-loss-draw rates are calculated through 50 games with each AI. AlphaDDA1 for the first player is slightly stronger than MCTS2. AlphaDDA1 for the first player defeats Minimax1. AlphaDDA1 for the second player loses against AlphaZero but draws against AlphaZero with a probability of 0.3. These results show that AlphaDDA1 for the first player does not adjust its skill to Minimax1. AlphaDDA2 for the second player adjust its skill to AlphaZero. AlphaDDA3 for the first player defeats against Minimax2. This result shows that AlphaDDA3 for the first player does not adjust its skill against Minimax2.

Othello

Table 11 shows the win, loss, and draw rates of AlphaDDAs and the AI agents for Othello. The win-loss-draw rate is calculated through 100 games with each AI (50 games when AlphaDDA is the first player and 50 games when AlphaDDA is the second player).

AlphaDDA1 is on par with AlphaZero and MCTS2 because its win rates against them are 0.56 and 0.52, respectively. AlphaDDA1 is stronger than MCTS4, Minimax1, and Minimax2 but can adjust its skill to the opponents. AlphaDDA1 is weaker than MCTS1 and MCTS3 but can adjust its skill to the opponents. AlphaDDA1 defeats Random. These results suggest that AlphaDDA1 can adjust its skill to AlphaZero, MCTS1, MCTS2, MCTS3, MCTS4, Minimax1, and Minimax2.

AlphaDDA2 is on par with AlphaZero, and MCTS3. AlphaDDA2 is weaker than MCTS1 but can adjust its skill to MCTS1. AlphaDDA2 is stronger than MCTS2, MCTS4, and Minimax1 but can change its skill. AlphaDDA2 defeats Random. Thus, AlphaDDA2 can change its skill against AlphaZero, MCTS1, MCTS2, MCTS3, MCTS4, and Minimax1.

AlphaDDA3 is on par with Random. AlphaDDA3 is weaker than Minimax1 but has a chance of a draw. It is defeated by AlphaZero, MCTS1, MCTS2, MCTS3, and MCTS4. Thus, AlphaDDA3 can adjust its skill to Random.

Tables 12 and 13 show the win, loss, and draw rates of AlphaDDAs against the AI agents for Othello when AlphaDDAs are the first and the second players, respectively. The win-loss-draw rate is calculated through 50 games with each AI. AlphaDDA1 for the first player is stronger than MCTS4. AlphaDDA1 for the second player is stronger than Minimax1. AlphaDDA1 for the second player is slightly weaker than MCTS3. AlphaDDA1 for the first player defeats Minimax2. This result suggests that AlphaDDA1 for the first player does not adjust its skill to Minimax2. AlphaDDA2 for the second player defeats MCTS4. This result suggests that AlphaDDA2 for the second player does not adjust its skill to MCTS4. AlphaDDA3 for the first player is slightly stronger than Random.

Alphazero

AlphaDDA is the trained AlphaZero with an added DDA. AlphaZero consists of the DNN and MCTS, as shown in Fig. 1A. The DNN receives an input consisting of the board states and the disc color of the current player. The DNN provides the value of a state and the move probabilities that indicate the probabilities of selecting valid moves. The MCTS is used to decide the best move based on the value and move probabilities.

Deep neural network in AlphaZero

The DNN predicts the value $v(s)$ and the move probability p with components $p(a\mid s)$ for each a, where s is a state and a is an action. In a board game, s and a denote a board state and a move, respectively. The input of the DNN consists of the board state and the disc color of the current player. The architecture of the DNN is shown in Fig. A1. The DNN has a “Body” that consists of residual blocks and a “Head,” which consists of the value head and the policy head. The value head and the policy head output the value $v(s)$ and the move probabilities p, respectively. AlphaDDA’s DNN is the same as that of AlphaZero.

The input to the DNN is an $M\times N\times (2T+1)$ image stack that consists of $2T+1$ binary feature planes of size $M\times N$, where $M\times N$ is the board size, and T is the number of histories. The first T feature planes represent the occupation of the discs of one player. A feature value is 1 if the corresponding cell is occupied by the disc, and 0 otherwise. The next T feature planes similarly represent the occupation of the other player’s discs. The final feature plane represents the disc color of the current player. The store colors of the first and second players are represented by 1 and −1, respectively.

Grid search

In this study, grid search is used for hyperparameter tuning. Grid search is the most basic hyperparameter optimization method (Feurer & Hutter, 2019) and is simple to implement (Bergstra & Bengio, 2012).

In grid search, the sets of the parameters for AlphaDDA1, AlphaDDA2, and AlphaDDA3 shown in Tables A2–A4 are evaluated. AlphaDDAs play the games with AlphaZero, MCTS1, MCTS2, Minimax1, and Random. AlphaDDAs with each pair of parameters play forty games with each AI (20 games when AlphaDDA is the first player and 20 games when AlphaDDA is the second player). Grid search finds the set of the parameters that achieves the smallest performance metric.

The performance metric is the sum of the differences between the number of wins and the number of losses. The following equation denotes the mathematical description of the metric D.

(11) $$\begin{array}{l}D={\displaystyle \sum _X(|N_{X_{\text{first}}}^{\text{Win}}-N_{X_{\text{first}}}^{\text{Loss}}|+|N_{X_{\text{second}}}^{\text{Win}}-N_{X_{\text{second}}}^{\text{Loss}}|)},\\ \text{X=\hspace{0.17em}}\{\text{AlphaZero},\text{MCTS1},\text{MCTS2},\text{Minimax1},\text{Random}\}.\end{array}$$

$N_{X_{\mathrm{f}\mathrm{i}\mathrm{r}\mathrm{s}\mathrm{t}}}^{\mathrm{W}\mathrm{i}\mathrm{n}}$ and $N_{X_{\mathrm{f}\mathrm{i}\mathrm{r}\mathrm{s}\mathrm{t}}}^{\mathrm{L}\mathrm{o}\mathrm{s}\mathrm{s}}$ are the number of wins and the number of losses, respectively, when the opponent X is the first player. $N_{X_{\mathrm{s}\mathrm{e}\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{d}}}^{\mathrm{W}\mathrm{i}\mathrm{n}}$ and $N_{X_{\mathrm{s}\mathrm{e}\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{d}}}^{\mathrm{L}\mathrm{o}\mathrm{s}\mathrm{s}}$ are the number of wins and the number of losses, respectively, when the opponent X the second player.

Monte carlo tree search

The MCTS is a best-first search method that does not require an evaluation function (Winands, 2017). MCTS consists of four strategic steps: Selection step, Playout step, Expansion step, and Backpropagation step (Winands, 2017). In the Selection step, the tree is traversed from the root node to a leaf node. The child node c of the parent node p is selected to maximize the following score:

(12) $$\mathrm{U}\mathrm{C}\mathrm{T}=q_c/N_c+C\sqrt{{\displaystyle \frac{\ln (N_p+1)}{N_c+\epsilon }}},$$

where $q_c$ is the cumulative value of c, $N_p$ is the visit count of p, $N_c$ is the visit count of c, and $\epsilon$ is a constant value to avoid division by zero. In the Playout step, a valid move is selected at random until the end of the game is reached. In the Expansion step, child nodes are added to the leaf node when the visit count of the leaf node reaches $N_{\mathrm{o}\mathrm{p}\mathrm{e}\mathrm{n}}$. The child nodes correspond to the valid moves. In the Backpropagation step, the result of the playout is propagated along the path from the leaf node to the root node. If the MCTS player itself wins, loses, and draws, $q_i\leftarrow q_i+1$, $q_i\leftarrow q_i-1$, and $q_i\leftarrow q_i$, respectively. $q_i$ is the cumulative value of node i. The set of the four steps mentioned above is called simulation. The simulation is executed $N_{\mathrm{s}\mathrm{i}\mathrm{m}}$ times. MCTS selects the action corresponding to the most visited node from the root node. In this study, $C=0.5$, $\epsilon ={10}^{-7}$, and $N_{\mathrm{o}\mathrm{p}\mathrm{e}\mathrm{n}}=5$.

Minimax algorithm

The minimax algorithm is a basic game tree search that finds the action generating the best value. Each node in the tree has a state, player, action, and value. The minimax algorithm first creates a game tree with a depth of $N_{\mathrm{d}\mathrm{e}\mathrm{p}\mathrm{t}\mathrm{h}}$. In this study, $N_{\mathrm{d}\mathrm{e}\mathrm{p}\mathrm{t}\mathrm{h}}=3$. Here, the root node corresponds to the current state and the minimax player itself. Second, the states corresponding to the leaf nodes are evaluated. Third, all nodes are evaluated from the parents of the leaf nodes to the children of the root node. If the player corresponding to the node is the opponent, the value of the node is the minimum value of its child nodes. Otherwise, the value of the node is the maximum value of its child nodes. Finally, the minimax algorithm selects the action corresponding to the child node of the root with the maximum value.

The evaluation of a leaf node is specialized for each game. For Connect4, the values of the connections of two same-color discs and three same-color discs are, respectively, $R\times c_{\mathrm{d}\mathrm{i}\mathrm{s}\mathrm{c}}{c}_{\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{a}\mathrm{x}}$ and $R^2\times c_{\mathrm{d}\mathrm{i}\mathrm{s}\mathrm{c}}{c}_{\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{a}\mathrm{x}}$, where $c_{\mathrm{d}\mathrm{i}\mathrm{s}\mathrm{c}}$ and $c_{\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{a}\mathrm{x}}$ are the colors of the connecting discs and the minimax player’s disc, respectively. The value of the node is the sum of the values of all the connections on the board corresponding to the node. The value of the terminal node is $R^3{c}_{\mathrm{w}\mathrm{i}\mathrm{n}}{c}_{\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{a}\mathrm{x}}$, where $c_{\mathrm{w}\mathrm{i}\mathrm{n}}$ is the color of the winner. R is 100 and 2 for Minimax1 and Minimax2, respectively.

For 6x6 Othello and Othello, the value of a node is calculated using the following equation:

(13) $$E=\sum _x\sum _{y}v(x,y)o(x,y)c_{\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{a}\mathrm{x}},$$

where $v_{(x,y)}$ is the value of the cell $(x,y)$, and $o(x,y)$ is the occupation of the cell $(x,y)$. For 6x6 Othello and Othello, Minimax1 evaluates each cell using Eqs. (14) and (15), respectively. For 6x6 Othello and Othello, Minimax2 evaluates each cell to 1. $o(x,y)$ is 1, $-1$, and 0 if the cell $(x,y)$ is occupied by the first player’s disc, the second player’s disc, and empty, respectively. For 6x6 Othello and Othello, the minimax algorithm expands the tree to the terminal nodes after the last six turns. The value of the terminal node is $E_{\mathrm{e}\mathrm{n}\mathrm{d}}=1000c_{\mathrm{w}\mathrm{i}\mathrm{n}}{c}_{\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{a}\mathrm{x}}$, where $c_{\mathrm{w}\mathrm{i}\mathrm{n}}$ is the color of the winner.

(14) $$v_{6\times 6}=\left(\begin{array}{cccccc}30& -5& 2& 2& -5& 30\\ -5& -15& 3& 3& -15& -5\\ 2& 3& 0& 0& 3& 2\\ 2& 3& 0& 0& 3& 2\\ -5& -15& 3& 3& -15& -5\\ 30& -5& 2& 2& -5& 30\end{array}\right).$$

(15) $$v_{8\times 8}=\left(\begin{array}{cccccccc}120& -20& 20& 5& 5& 20& -20& 120\\ -20& -40& -5& -5& -5& -5& -40& -20\\ 20& -5& 15& 3& 3& 15& -5& 20\\ 5& -5& 3& 3& 3& 3& -5& 5\\ 5& -5& 3& 3& 3& 3& -5& 5\\ 20& -5& 15& 3& 3& 15& -5& 20\\ -20& -40& -5& -5& -5& -5& -40& -20\\ 120& -20& 20& 5& 5& 20& -20& 120\end{array}\right).$$

Elo rating

Elo rating is a popular metric to measure the relative strength of a player. We can estimate the probability that player A defeats player B $p(A\mathrm{d}\mathrm{e}\mathrm{f}\mathrm{e}\mathrm{a}\mathrm{t}\mathrm{s}\mathrm{B})$ using the Elo rating e. $p(A\mathrm{d}\mathrm{e}\mathrm{f}\mathrm{e}\mathrm{a}\mathrm{t}\mathrm{s}\mathrm{B})$ is denoted by

(16) $$p(A\mathrm{d}\mathrm{e}\mathrm{f}\mathrm{e}\mathrm{a}\mathrm{t}\mathrm{s}\mathrm{B})=1/(1+10^{(\mathrm{e}(\mathrm{B})-\mathrm{e}(\mathrm{A}))/400}),$$

where e(A) is the Elo rating of player A. After $N_G$ games, the new Elo rating of player A $e^{\mathrm{\prime }}(A)$ is calculated using the following equation:

(17) $$e^{\mathrm{\prime }}(A)=e(A)+K(N_{\mathrm{w}\mathrm{i}\mathrm{n}}-N_G\times p(A\mathrm{d}\mathrm{e}\mathrm{f}\mathrm{e}\mathrm{a}\mathrm{t}\mathrm{s}\mathrm{B})),$$

where $N_{\mathrm{w}\mathrm{i}\mathrm{n}}$ is the number of times that player A wins. In this study, $K=8$.

Examples of games

Figure A2 is an example of a game of MCTS2 vs AlphaDDA1 in Othello. When an estimated value is closer to $-1$, AlphaDDA1 is more confident of winning because it is the second player in this case. MCTS2 played a move to the B2 square at the 9th turn. This move is worse because the move increases the probability of AlphaDDA1 obtaining the A1 corner. Corner discs are crucial because they cannot be flipped and commonly form an anchor which can stabilize other discs (Rosenbloom, 1982). In order to obtain the A1 corner, AlphaDDA1 played the C2 square at the 10th turn. MCTS1 could not flip the disc on C3 at the 11th turn, and AlphaDDA1 put the disc on the A1 square. Furthermore, AlphaDDA1 maintained the mean estimated value close to its disc color. This example suggests that AlphaDDA1 chooses a better move with a high estimated value.

Figure A3 is an example of a game of MCTS1 vs AlphaDDA1 in Connect4. MCTS1 dropped its disc into the 7 column at the 11th turn. MCTS1 will win if it drops its disc into the 6 column at its next turn. However, AlphaDDA1 dropped its disc into 1. As a result, MCTS1 won. If AlphaDDA1 had searched the game tree to two depths, it could have avoided this loss. AlphaDDA1’s move was caused by MCTS with a few simulations. If the mean estimated value is about $-1$ (AlphaDDA1 is the second player), the number of simulations of AlphaDDA1’s MCTS is 7 in Connect4. AlphaDDA1 with a few simulations will choose a wrong move and not avoid losing not only because AlphaDDA1’s MCTS shallowly searches the game tree but also because its DNN evaluates a situation on the assumption that its MCTS with sufficient simulations can avoid losing after a few turns. This example is extreme, but it suggests that AlphaDDA1 may choose a wrong move when it is confident of winning.

Figure A4 is an example of a game of Random vs AlphaDDA2 in Othello. Random played a move to the G7 square at the 7th turn. This is a worse move because the G7 square is a neighbor to the H8 corner and this move allows AlphaDDA2 to obtain the corner. AlphaDDA2 played a move to the D2 square at the 8th turn to obtain the corner. However, this move is not best because Random can obstruct AlphaDDA2’s occupation of the corner square if Random puts its disc on the C4. The best move is F5. Random did not avoid the occupation of the corner at the 9th and the 11th turns. As a result, AlphaDDA2 could put its disc on the H8 square at the 10th and the 12th turns. However, AlphaDDA2 did not put its disc on the H8 square at the 10th and the 12th turns. These results show that AlphaDDA2 cannot choose the best move due to the deterioration of DNN by dropout.

Figure A5 is an example of a game of MCTS1 vs AlphaDDA3 in Othello. When an estimated value is closer to 1, AlphaDDA3 is more confident of winning because it is the first player in this game. MCTS1 played a move to the G7 square at the 14th turn. This move is worse because AlphaDDA3 can occupy the H8 corner square at the 17th turn if AlphaDDA3 puts a disc on F7 at the 15 turn. However, AlphaDDA3 played G8 at the 15th turn. This move worsens the situation for AlphaDDA3 because MCTS1 can put its disc on the H8 corner at the next turn. In fact, this move changed the mean estimated value from 0.83 to $-0.61$. AlphaDDA3 was confident of winning at the 15th turn and balanced the mean estimated value by choosing the bad move. On the other hand, AlphaDDA3 chose the move to improve the mean estimated value at the 17th turn, and the move changed the mean estimated value to 0.49 at the 19th turn. These results show that AlphaDDA3 can balance the estimated values, but the balancing accompanies a rapid change of the mean estimated value.

Figure A6 is an example of a game of AlphaDDA3 vs AlphaZero in Connect4. AlphaDDA3 is the first player in this game, and its disc color is 1 (white). AlphaZero dropped a disc into the 5 column at the 8th turn. This move resulted in three black discs in a column. AlphaDDA3 predicted its loss and avoided the loss by playing 5. This result shows that AlphaDDA3 can avoid its loss when AlphaDDA3 predicts its loss. In other words, AlphaDDA3 may not avoid its loss when it predicts its win.