Computational modelling reveals the influence of object similarity and proximity on visually guided movements

Implementation of the visual processing module

The original implementation involved modelling a simple choice-reaching task in which an odd colour target had to be identified from an experimental stimulus. Image processing were employed to detect the red or green colours (with more colours in the display allowed by SH-CoR); identify the odd target and create competitive colour feature maps. The processed information was further fed and computed in the other modules (Strauss & Heinke, 2012; Strauss et al., 2015).

But to account for the size IFs in the extension, the extraction of the size-related salience of the items along with the target (odd-colour) identification was required. Saliency levels were assigned to each item based on both colour and size. The saliency levels represent a saliency score given to an area of interest that reflects its visual prominence (Clarke, Dziemianko & Keller, 2014), thereby improves the interpretability of the model.

For a given experimental stimulus, the visual processing module pre-processes the stimulus, extracts the items in the display and compares them to find the configuration type. The algorithmic schema (Fig. 2) of the visual processing module is the following: The experimental stimulus (coloured image input) is resized and converted into a gray-scale image before being converted to black and white. This transformation aids in identifying the image’s connected components (Jeong, 2019). The connected components represent the various items present in the experimental stimulus. The colours of each item are scanned to identify the odd colour, which allows the items to be classified as a target or a distractor. Furthermore, the sizes of the items are compared to determine whether they are large or small. The module assigns a predefined saliency level based on the item configuration detected. The saliency levels are designed such that similar items have same saliency levels, while targets will have higher levels than the distractors, indicating their greater visual prominence. The predefined levels are arbitrary and need extensive tuning to obtain desired model performance. This tuning process involves testing several combinations to identify the most effective settings. These saliency levels are further used by motor modules for movement production.

Modelling target selection

Three competition nodes were created, one for each item in the display, instead of one for each colour as in previous models. The associated saliency levels are fed into these nodes. The dynamic neural field (DNF) theory governs the behaviour of these nodes (Erlhagen & Schoner, 2002). However, a new implementation of a recurrent on-centre, off-surround network (i.e., a RNN) replaced the original implementation of the single neuronal dynamics (Grossberg, 1973; Amari, 1977). It incorporates coefficients for local excitation and global inhibition between nodes and allows competing behaviour. The rate of change of the node activation, $\dot{u_i}$, was implemented as follows: (1)${\displaystyle \tau \cdot \dot{u_i}=-A\cdot u_i+B\cdot f\left(u_i\right)+C\cdot I_i+D\cdot \sum f\left(u_i\right)+h_i+ϵ,}$ (2)${\displaystyle f\left(u_i\right)=1/\left(1+e^{\beta }\left(u_i-u_0\right)\right)}$

Here for given node i, u_i represents the activation, h stands for the resting level, A is a constant equals to 1 in the current work, B is the self-excitation level of the node, the sigmoid function f(u_i) is the output of the current node, I_i is the input to the node (in this case, the saliency level), C is the coefficient for local active nodes, D is the coefficient for the global active nodes, τ is the time factor of how fast the node adapt to the change in the parameters and ϵ is the random gaussian noise for robustness. The sign and magnitude of the coefficients, C and D, determine the corresponding excitation/inhibition effects from the neighbouring nodes. The tuned coefficients (see Table 1) employ a winner-takes-all (WTA) mechanism where a node with the highest activation is selected while competing with other nodes. The activations of the nodes depend proportionately on the saliency level assigned. Implementing the modified node dynamics as above will imply a lower inhibition effect between the nodes when the difference of their activation is less and vice versa. The reduced inhibition effect will lead to a longer suppression time of the other nodes whilst the winner node is selected. As a result, the saliency levels are carefully selected after proper tuning. As in Woodgate’s experimental results, the chosen saliency levels should be such that the larger distractors must have a less inhibitory impact on the target than the smaller ones. As a result, the saliency levels are constructed in the descending order: small target > large target > small distractor > large distractor

Spatial location maps are created to represent the spatial features of the items in the display. These maps have Gaussian inputs centred at the location (centroid) of each item present in the stimulus. The output from each of the saliency competition nodes is multiplied with the corresponding spatial location maps, and these three maps are summed together to create a single map of the size of the display item (e.g., the black star in Fig. 3). This resulting summation is input to the target location map. A topological connection between the saliency competition and the target location maps exists. The representation of the items on the target location map appears even before a single target is selected. Once the competition in the saliency competition nodes is over, the units at the target location will only be activated in the target location map. These parallel representations cause the curved trajectories consistent with Song and Nakayama’s theory (2008), and the competition in the saliency competition nodes are consistent with grouping interactions, as in Woodgate (2015). The output from the target location map is relayed into the motor stage of the model.

Motor planning and movement production

The motor stage encodes the location information from target selection modules into motor parameters using activation blobs governed by the DNF framework (Erlhagen & Schoner, 2002; Schoner, Spencer & Group, 2016). The elements of the motor stage are retained the same as in SH-CoR, with only a few modifications to their DNF parameters (see Makwana et al. (2023) for more details). The motor stage comprises three maps: the hand map H_loc (a 2D Gaussian), the hand-target difference map (Displacement Representation D map), and the velocity map (V map). The displacement representation was created through a convolution of hand map with the target location, i.e., output of target selection (T_loc), i.e., via a CNN: (3)${\displaystyle D\left(\underset{¯}{x},t\right)=\int H_{loc}\left(\underset{¯}{y}-\underset{¯}{x},t\right)\cdot f\left(T_{loc}\left(\underset{¯}{x}\right)\right)d\underset{¯}{x}}$ where x and y represent the hand position as a function of time in each simulated step, we update the hand position based on the velocity in a close loop control. The D map forms the input of V map plus another 2D Gaussian to initiate activations: (4)${\displaystyle I^{Vel1}\left(\underset{¯}{x},t\right)=D\left(\underset{¯}{x},t\right)+Amp\cdot e^{{\underset{¯}{x}}^2/2{\sigma }^2}.}$ The velocity representation uses a two-layer DNF namely layer Vel1 and Vel2 to generate and stabilise the blob-shape activation: (5)${\displaystyle \tau \cdot {\dot{x}}_i^{Vel1}=-x_i^{Vel1}+I^{Vel1}+ex{c}_{loc}\left(x_i^{Vel1}\right)+inh\left(x_i^{Vel1}\right)+h_i+ϵ,}$ (6)${\displaystyle \tau \cdot {\dot{x}}_i^{Vel2}=-x_i^{Vel2}+ex{c}_{loc}\left(x_i^{Vel2}\right)+inh\left(x_i^{Vel1}\right)+h_i+ϵ,}$ (7)${\displaystyle ex{c}_{loc}\left(\underset{¯}{x}\right)=\int w_{exc}\left(\underset{¯}{x}-{\underset{¯}{x}}^{\prime }\right)\cdot f\left({\underset{¯}{x}}^{\prime }\right)d{\underset{¯}{x}}^{\prime }}$ (8)${\displaystyle inh\left(\underset{¯}{x}\right)=\int w_{inh}\left(\underset{¯}{x}-{\underset{¯}{x}}^{\prime }\right)\cdot f\left({\underset{¯}{x}}^{\prime }\right)d{\underset{¯}{x}}^{\prime }+g_{inh}\cdot \int f\left(x^{\prime }\right)d{x}^{\prime }}$ in which the kernel w with strength c and width parameter σ as defined with: (9)${\displaystyle w\left(\underset{¯}{x}\right)=\frac{c}{\sigma \sqrt{2\pi }}\cdot exp\left(-\frac{|\underset{¯}{x}{|}^2}{2{\sigma }^2}\right).}$ Here, the movement direction information is encoded into the velocity parameter. The velocity parameter is then converted into end effector joint velocities using inverse kinematics. Also, the information is fed back into the model to update the current hand position in the hand map. Random noise is added at different stages: the saliency competition nodes, the velocity map and the hand target difference map, to model the variations in the human trials.

Parameter tuning

The model is tuned in a trial and error fashion to adjust the parameters to get the desired activations at the different topological levels of the model (Table 2). As discussed in previous sections, the designed saliency levels are arbitrary; tuned appropriately along with the model’s DNF element parameters. The saliency levels are set up so that the target has the highest saliency level and gets chosen. The current saliency levels for various item types are shown in Table 3.

For example, the ddT configuration: a small distractor on the left, another small distractor in the middle, and a large target on the right will be assigned saliency levels, 4.6, 4.6 and 4.8, respectively. For the dissimilar distractor condition, both the possible positions of the distractors are considered (large distractor can be either left or right). The stimuli are permutated and given to the model in line with the literature to avoid top-down influence from irrelevant feature predictability (Bacon & Egeth, 1994; Woodgate, 2015).

After picking set of parameters, the model was validated for 25 sets, each with 24 stimuli images (30 × 30 pixels) corresponding to different possible item combinations (as shown in Fig. 1), totalling 600 trials. A noise of magnitude 0.005 at the saliency competition nodes, the hand-target difference map, and the velocity map was added to model the variations in the human trial. At the end of each trial, a break time was introduced to clear the residual activations in the maps so that they won’t influence the successive trials. Based on the saliency levels assigned to items, competition occurs at different model stages. The model moves the end effector to create a hand trajectory when there is sufficient activation in the velocity map. But a threshold of 0.5 is set in the velocity map to avoid random noise from moving the hand. At the end of each trial, the hand velocity and the hand position of the end effector are registered.

Data analysis and model evaluation

The IL and MD can be calculated using the hand velocity and the hand position of the end effector. IL is calculated as the time steps it takes the end effector in a trial to reach 10% of the model’s maximum velocity of the hand across all trials. The hand velocity of each trial is fitted to a polynomial of degree 10 to improve the resolution of the IL. This 10% is an arbitrary value representing the minimal activation required to initiate the hand movement. MD can be calculated as the largest perpendicular distance between data points on the trajectory and the straight line drawn from the start of the hand position to the target. Using IL and MD, a qualitative assessment can be made of whether the modelling results were qualitatively similar to human performances. For example, CoRLEGO results must demonstrate a significant decrease in IL when the distractor size increases, which indicates the facilitation of search. The model results can be normalised and compared by analysing how closely they fit the experimental results. In computational modelling experiments like CoRLEGO, qualitative analysis is more appropriate than quantitative analysis (see Heinke (2009) for further discussion).