Biologically-inspired emotional processing for adaptive decision-making in non-stationary environments

Environment a: sudden strategy reversal

Environment A simulates catastrophic strategy failure through four phases across 200 trials. The critical transition at trial 100 tests rapid adaptation when previously reliable strategies suddenly fail.

Phase structure:

• Initial stable period (trials 1–100). Action 0 dominates with a mean reward of approximately $0.8\pm 0.1$, establishing strong behavioral patterns in favor of this choice.

• Abrupt reversal (after trial 100). The optimal action shifts sharply from 0 to 4, creating a sudden mismatch between learned preferences and the new reward landscape.

• New regime (trials 101–200). Action 4 yields the highest reward ( $\approx 0.9\pm 0.1$), while the previously optimal Action 0 drops to a baseline level ( $\approx 0.2\pm 0.1$).

Minor variations in the later subphases introduce additional complexity without disrupting the main narrative of a single large reversal. Gaussian noise ( $\text{σ}\mathbf{=}\mathbf{0.15}$) prevents single-trial learning while still maintaining clearly distinguishable strategies.

Environment B: predictable alternation

Environment B implements perfectly regular oscillations between two optimal strategies, testing pattern recognition rather than change detection.

Pattern structure:

• Every 40 trials, the optimal action alternates between 0 and 4.

• Rewards are extreme: the optimal action yields a mean reward of 0.95, whereas the opposite action yields 0.01.

• Intermediate actions (1, 2, 3) consistently provide minimal reward (0.05).

• Low noise ( $\sigma =0.05$) maintains near-deterministic outcomes and sharply defined phases.

This deterministic design evaluates whether emotion-based processing helps or hinders performance when simple pattern learning is sufficient for near-optimal behavior.

Environment C: stochastic disruptions

Environment C simulates unpredictable market conditions through random structural changes and false signals. Three change points occur randomly between trials 30–180, testing robustness when change timing is unknowable.

Key features:

• - Random transitions: Change points vary each run, preventing pattern learning.

• - Unpredictable optimum: At each phase change, the best action is reassigned randomly (constrained to differ from the previous optimal arm), yielding a mean reward of approximately $0.8\pm 0.15$.

• - False signals: On 5% of trials, temporary reward spikes ( $\approx 0.7\pm 0.15$) occur and persist for 1–2 trials, mimicking noise traders or false breakouts in real markets.

• - High uncertainty: All non-optimal actions provide baseline reward of approximately $0.2\pm 0.15$.

This design tests whether agents can maintain performance when both the timing and direction of changes are fundamentally unpredictable, requiring continuous monitoring rather than reliance on simple periodic patterns.

External limbic system: emotion-like computational signals

The External Limbic System generates eight emotion-like signals informed by affective neuroscience. Drawing on dimensional models of emotion (Plutchik, 1980; Russell & Barrett, 1999), ECIA implements signals organized along four functional dimensions relevant to reinforcement learning:

• Joy/Fear: Valence dimension—reward approach vs. threat avoidance, implementing the fundamental “approach-withdraw” behavioral axis.

• Hope/Sadness: Temporal dimension—future expectation vs. sustained loss, capturing forward-looking optimism and retrospective discouragement.

• Curiosity/Anger: Exploration dimension—information-seeking vs. obstacle confrontation, balancing exploration with persistence.

• Pride/Shame: Self-evaluation dimension—success reinforcement vs. failure correction, implementing performance-based strategy adjustment.

This mapping ensures that ECIA captures the core functional dimensions of affect—valence, arousal, expectation, and self-monitoring—which collectively support adaptive decision-making (Carver & Scheier, 1990; Russell & Barrett, 1999). Alternative emotion taxonomies exist (e.g., Ekman’s six basic emotions; Panksepp’s seven affective systems; Plutchik’s eight primary emotions) (Ekman, 1972; Panksepp, 2004; Plutchik, 1980). Rather than directly replicating any single taxonomy, ECIA’s selection prioritizes functional relevance to reinforcement learning: valence-based approach/avoidance, temporal expectation, exploration-exploitation balance, and performance-based self-regulation. The eight-emotion structure balances computational tractability with functional completeness, avoiding both oversimplification (fewer emotions) and excessive complexity (more granular taxonomies).

Each signal’s triggering condition corresponds to known neural mechanisms:

• Fear: Activated by negative reward changes (reward < previous_reward − 0.15), echoing amygdala engagement in threat detection. The amygdala rapidly responds to potential threats and negative outcomes (LeDoux, 1996), with activity modulated by prediction error magnitude (Schultz, Dayan & Montague, 1997). In ECIA, fear intensity scales with the size of the recent reward drop, promoting withdrawal from actions associated with declining outcomes, analogous to Plutchik’s withdrawal dimension (Plutchik, 1980).

• Joy: Triggered by high absolute rewards (reward > 0.7), inspired by ventral tegmental area (VTA) dopamine responses to appetitive outcomes. Dopaminergic neurons show phasic increases to unexpected or large rewards (Berridge & Kringelbach, 2015; Schultz, 1998). ECIA’s joy signal grows with reward magnitude, reinforcing successful actions and implementing the “approach” pole of the valence axis.

• Hope: Responds to positive reward trends (linear regression slope > 0.03 over the recent four trials), motivated by dopamine “ramping” activity. During approach to anticipated rewards, dopamine firing can increase gradually, encoding proximity to expected positive outcomes (Howe & Dombeck, 2016). ECIA implements this as trend detection, providing anticipatory signaling that enables proactive rather than purely reactive adaptation.

• Sadness: Activated during sustained low performance (mean reward < 0.4 over six trials), reflecting prolonged negative states described in depression neuroscience. Extended negative prediction errors are associated with reduced striatal responses and altered prefrontal–limbic connectivity (Eshel & Roiser, 2010). ECIA’s sadness signal integrates negative outcomes over time and dampens exploratory drive during extended poor performance, representing the functional opposite of joy.

• Curiosity: Driven by insufficient coverage of the action space (action diversity < 0.8), implementing an information-seeking mechanism analogous to the locus coeruleus–norepinephrine (LC–NE) system. LC–NE activity is sensitive to uncertainty and information gaps, supporting exploratory behavior (Aston-Jones & Cohen, 2005; Gottlieb & Oudeyer, 2018). In ECIA, curiosity increases when options are under-sampled and boosts exploration of rarely chosen actions, corresponding to Panksepp’s “seeking” system.

• Anger: Activated when performance falls below an expected improvement trajectory (reward < expected − 0.2), modeling frustration-like responses. The anterior cingulate cortex signals conflicts between expected and actual outcomes and can trigger increased effort allocation (Amsel, 1992; Botvinick, Cohen & Carter, 2004). ECIA’s anger signal temporarily increases exploration away from the most recent choice when anticipated learning progress does not materialize, encouraging strategic switching rather than passive persistence.

• Pride: Triggered by sustained high performance (success rate > 0.6 over five trials), inspired by social reward processing in the ventromedial prefrontal cortex (VMPFC). VMPFC activity tracks positive self-relevant outcomes and goal achievement (Izuma, Saito & Sadato, 2008). ECIA’s pride signal strengthens exploitation of successful strategies, stabilizing effective policies.

• Shame: Activated by sustained failures (failure rate > 0.5 over four trials), motivated by aversive social feedback signals in anterior insula and dorsal anterior cingulate. These regions respond to negative social evaluation and self-conscious emotions (Takahashi et al., 2004). ECIA’s shame signal temporarily suppresses recently selected actions after repeated failures, discouraging perseveration on ineffective choices.

All emotional signals undergo temporal decay (γ = 0.96) with momentum effects, implementing the time-course dynamics observed in affective responses—emotions build gradually but persist beyond immediate triggers (Kuppens, Oravecz & Tuerlinckx, 2010). Signals are normalized to prevent runaway activation, analogous to homeostatic regulation in biological affect systems.

Importantly, labels such as ‘Fear’, ‘Joy’ and ‘Pride’ are used as convenient shorthand for these computational functions; they do not imply that ECIA literally implements human subjective emotions or detailed biological circuitry. The threshold and window parameters were selected based on preliminary experimentation and alignment with environmental characteristics. The Fear threshold (0.15) was set to exceed routine stochastic variation in Environments A and C (σ = 0.15). Temporal windows reflect differential timescales: shorter windows for reactive signals (Hope: four trials) and longer windows for sustained states (Sadness: six trials). The decay parameter (γ = 0.96) produces a half-life of approximately 17 trials.

Prefrontal decision unit: integrated action selection

The Prefrontal Decision Unit integrates multiple information sources for action selection, inspired by prefrontal cortex’s role in complex decision-making (Miller & Cohen, 2001). The selection rule combines learned values with emotional, memory-based, and uncertainty signals:

$A^{\ast }=arg\underset{a}{max}\left[Q\left(s,a\right)+\eta \cdot E\left(s,a\right)+M\left(s,a\right)+U\left(s,a\right)+\text{ξ}\right]$

where:

• -

  Q(s,a): Learned action values (analogous to striatal value encoding)

• -

  E(s,a): Emotional influence (η = 0.55), integrating limbic signals

• -

  M(s,a): Episodic memory bias (hippocampal-inspired retrieval)

• -

  U(s,a): Uncertainty bonus (LC-NE exploration signal)

• -

  ξ: Gaussian exploration noise (motor variability)

The linear additive form represents a simplifying assumption. Biological prefrontal integration likely involves nonlinear interactions and gating mechanisms (Miller & Cohen, 2001), and alternative integration schemes (e.g., multiplicative gating, attention-weighted combination) remain directions for future work. Nevertheless, linear formulation was chosen for interpretability: additive weights allow clear attribution of behavioral effects during ablation analysis. To mitigate dominance of individual terms, emotional signals are normalized (as described above), and η = 0.55 was selected to balance emotional modulation against learned values.

This architecture mirrors the known integration of striatal value signals, prefrontal cognitive control, limbic emotional input, and hippocampal memory in mammalian decision-making (Doya, 2008; O’Doherty, Cockburn & Pauli, 2017).

Episodic memory system: experience storage and retrieval

The Episodic Memory System stores significant experiences based on prediction error magnitude (|PE| > 0.015), implementing computational principles of hippocampal memory formation. The hippocampus preferentially encodes surprising events—those with high prediction errors—while routine experiences are less likely to form lasting memories (Duncan et al., 2012; Lisman & Grace, 2005).

Stored memories include:

• Action-outcome associations

• Contextual features (temporal phase, normalized time)

• Emotional state at encoding

• Prediction error magnitude (salience marker)

Memory retrieval uses similarity-based search (threshold = 0.035), analogous to pattern completion in hippocampal CA3 networks. When current context matches stored patterns, relevant memories bias action selection, implementing rapid reinstatement of successful strategies (Norman & O’Reilly, 2003).

The system maintains 15 high-quality memories, prioritizing recent and high-salience experiences. This capacity limitation mirrors working memory constraints and implements a computational analogue of memory consolidation—only significant experiences survive competitive interference.

Adaptive learning module: dopamine-inspired plasticity

The Adaptive Learning Module implements learning rate modulation based on prediction errors and emotional intensity, inspired by dopaminergic modulation of synaptic (Schultz, 2015; Sharpe et al., 2017).

Base learning rate α = 0.22 is adjusted by:

• 1.

  Prediction error magnitude: Larger surprises increase plasticity (surprise-driven learning (Pearce & Hall, 1980)).

• 2.

  Emotional intensity: High arousal states enhance memory encoding (amygdala-hippocampus interactions (McGaugh, 2004)).

• 3.

  Curiosity signals: Exploration states increase learning rates (LC-NE enhancement of plasticity (Bouret & Sara, 2004)).

This creates adaptive learning dynamics where the system learns faster during surprising, emotionally-salient, or exploratory episodes—precisely when environmental information is most valuable.

Computational abstraction and biological correspondence

ECIA’s components are computational abstractions capturing functional principles of biological affective-cognitive systems. This level of abstraction is standard in computational neuroscience:

• -

  Q-learning abstracts synaptic plasticity to simple value updates.

• -

  Actor-critic models abstract basal ganglia-cortical loops.

• -

  Temporal difference learning abstracts dopamine prediction errors.

Similarly, ECIA abstracts affective neuroscience findings into computationally tractable algorithms. The key distinction is that triggering conditions, interaction patterns, and temporal dynamics align with empirical neuroscience rather than being arbitrary design choices.

These four components interact through bidirectional connections:

• -

  Emotions modulate memory storage salience and retrieval priorities.

• -

  Memory retrieval triggers anticipatory emotional responses.

• -

  Learning rates are jointly determined by prediction errors and emotional states.

• -

  Decision-making integrates all information streams through prefrontal control.

This creates emergent adaptive behaviors that exceed the capabilities of individual components, analogous to how integrated brain systems produce flexible intelligence. Component contributions and synergistic effects are analyzed through systematic ablation studies reported in Results ‘Component Contributions’.

Improved baselines for non-stationary environments

Standard reinforcement learning algorithms assume stationary reward distributions, making them unsuitable for direct comparison in our time-varying environments. We therefore implemented three adaptive baseline algorithms specifically designed for non-stationary multi-armed bandit problems: Context-Aware ε-Greedy extends the standard ε-greedy approach with environmental change detection. The algorithm monitors reward distributions by comparing recent performance (most recent 10 trials) against older patterns (trials 20–30 steps back). When the mean reward difference exceeds a threshold (0.25) and surpasses combined noise variance, the algorithm interprets this as an environmental change. Upon detection, it performs a partial reset of Q-values (Q ← 0.5Q) to balance knowledge retention with adaptation, while temporarily increasing exploration. This mechanism allows the algorithm to maintain useful knowledge while remaining responsive to distributional shifts. Parameters: ε = 0.1, α = 0.1, window size = 50, change threshold = 0.25. Sliding Window Upper Confidence Bound (SW-UCB) addresses non-stationarity by maintaining only recent observations for each action (Garivier & Moulines, 2008). Rather than accumulating all historical data, SW-UCB uses a fixed window of the most recent W observations per action. The upper confidence bound is calculated using only these windowed statistics:

$$UCB(a)={\mu }_w(a)+c\sqrt{\frac{ln(t)}{n_w(a)}}.$$where μ_W(a) represents the mean reward within the window, n_W(a) the windowed action count, and c = 2.0 the exploration parameter. This windowing naturally discounts outdated information, enabling the algorithm to track changing reward distributions without explicit change detection. Parameters: W = 100, c = 2.0, minimum samples = 5. Adaptive Thompson Sampling extends Bayesian Thompson Sampling with exponential recency weighting (Raj & Kalyani, 2017). The algorithm maintains a Gaussian posterior distribution for each action’s expected reward but applies a discount factor γ = 0.99 to all sufficient statistics at each time step before incorporating new observations. Specifically, sum_rewards ← γ × sum_rewards + new_reward, with similar updates for sum_squared_rewards and effective_count.

This exponential weighting gives recent observations higher influence on the posterior distribution while gracefully degrading the impact of older data. The posterior mean and variance are computed from these discounted statistics, and action selection follows standard Thompson Sampling by drawing samples from each action’s posterior. Parameters: γ = 0.99, minimum std = 0.1, periodic reset threshold = 100 trials.

Naive baselines

For reference and to demonstrate the necessity of adaptation mechanisms, we also implemented standard versions without temporal adaptations:

• -

  Naive ε-Greedy: Standard ε-greedy (ε = 0.1) with sample-average Q-value updates (α = 1/n).

• -

  Naive UCB: Standard UCB1 algorithm with exploration parameter c = 0.5.

• -

  Naive Thompson Sampling: Standard Gaussian Thompson Sampling without discount factor.

These serve as reference points to isolate the contribution of adaptation mechanisms in non-stationary environments.

ECIA and ablation variants

As described in §2.2, ECIA comprises four integrated components. We tested seven systematic ablation variants to quantify individual component contributions:

• -

  ECIA-NoEmotion: Emotion vector set to zero.

• -

  ECIA-NoMemory: Memory bias removed.

• -

  ECIA-NoDopamine: Fixed learning rate (α = 0.1).

• -

  ECIA-NoDop-NoMem: Combined removal of dopamine and memory.

• -

  ECIA-NoDop-NoEmo: Combined removal of dopamine and emotion.

• -

  ECIA-NoMem-NoEmo: Combined removal of memory and emotion.

• -

  ECIA-NoAll: All advanced components removed.

These ablations isolate the contributions of emotional processing, episodic memory, and adaptive learning rates to ECIA’s overall performance.

Fair comparison protocol

To ensure fair comparison, all agents operate on the same observation stream: the sequence of chosen actions and received rewards. Additionally, normalized time (t/T) is available as contextual information; however, only ECIA explicitly incorporates this temporal context through episodic memory storage and retrieval. In Environments A and B, environmental phases are deterministic functions of time, meaning any algorithm could infer phase information from the time step alone. In Environment C, phase transitions occur at stochastic change points; here, contextual information provides additional environmental cues.

We acknowledge that this explicit use of normalized time may confer an advantage in Environments A and B, where phase is a deterministic function of time. However, in Environment C, phase transitions occur at stochastic change points unknown to all agents, yet ECIA maintains its performance advantage, suggesting benefits beyond time-based prediction. Ablation experiments isolating the contribution of temporal context were not conducted and represent a direction for future work.

The critical distinction lies in information utilization strategy. Improved baselines detect environmental changes through reward pattern analysis, inferring changes from statistical properties of observed rewards to adapt after distributional shifts occur. ECIA integrates context into its emotion-memory-dopamine system, enabling anticipatory behavior through emotional signals (hope, fear) and rapid retrieval of similar past situations via episodic memory.

This design evaluates different adaptive strategies—context-integrated processing vs. purely statistical adaptation—rather than comparing algorithms with unequal information access.

Practical implications

The environment-specific performance patterns provide actionable guidance for algorithm selection. ECIA is best suited for domains characterized by:

• 1.

  High environmental uncertainty with unpredictable change timing (analogous to Environments A and C): financial markets with regime shifts, clinical decision support where patient conditions change abruptly, adaptive control systems in dynamic operational contexts.

• 2.

  Multiple competing action strategies requiring rapid re-evaluation when conditions shift, rather than simple binary choices.

• 3.

  Availability of computational resources sufficient to support the multi-component architecture (approximately 10x overhead vs. simplest baselines).

Conversely, ECIA is less suitable for:

• 1.

  Highly predictable environments with deterministic patterns (analogous to Environment B): manufacturing process control, scheduled logistics, routine administrative tasks where simple rules suffice.

• 2.

  Hard real-time systems where microsecond-level latency requirements preclude the emotional processing overhead.

• 3.

  Extremely high-dimensional problems where the episodic memory system’s quadratic similarity comparisons become prohibitive (though this could be addressed with approximate nearest-neighbor methods in future work).

Limitations

Several limitations qualify these findings. First, while computationally tractable, ECIA introduces additional hyperparameters (emotional decay rates, memory similarity thresholds, dopamine modulation coefficients) requiring tuning. The current parameter values (η = 0.55, γ = 0.96, etc.) were selected through preliminary experimentation rather than systematic optimization, and performance is sensitive to these settings. Future work should develop principled methods for automatic parameter configuration and explore whether different values might better approximate human emotional decision-making or optimize artificial agent performance. Similarly, sensitivity analysis for recovery metric parameters (stability window, threshold ratio) was not conducted.

Second, the experiments employed synthetic multi-armed bandit environments. While these effectively isolate fundamental decision-making dynamics under non-stationarity, they lack the high-dimensional state spaces, partial observability, and long-horizon credit assignment challenges of real-world problems. Extending ECIA to Deep Reinforcement Learning frameworks—where emotional signals would modulate neural network learning rates and attention mechanisms—represents an important direction for validating these principles in complex domains.

Third, the selection of eight specific emotions, while informed by dimensional models of affect remains somewhat arbitrary (Plutchik, 1980; Russell & Barrett, 1999). Alternative emotion taxonomies might yield different performance characteristics (Ekman, 1972; Panksepp, 2004). Systematic ablation across emotion subsets could identify minimal sufficient sets for different problem classes.

Additionally, in Environment C, the provision of phase context following stochastic change points may have provided ECIA with an informational advantage beyond what would be available in purely observational settings. Future work should therefore evaluate ECIA under observational conditions without explicit phase cues to isolate the contribution of emotional processing from contextual awareness. Furthermore, evaluating ECIA on widely used benchmark suites for non-stationary bandits would facilitate direct comparison with alternative adaptive algorithms and strengthen the generalizability of these findings. Temporal visualizations such as cumulative regret curves and post-change recovery trajectories were not included; such diagnostics would provide additional empirical support for adaptation dynamics.

Broader perspectives: speculative computational insights

While our synthetic bandit experiments cannot directly validate evolutionary hypotheses—lacking population dynamics, generational turnover, or ecological realism—they may nonetheless offer computational insights into enduring puzzles about human affective behavior. The following discussion extends beyond our experimental findings and should be interpreted as hypothesis-generating rather than empirically validated claims.

Why do emotions persist in human cognition despite their measurable costs in stable contexts? Our results suggest a possible computational answer: emotions may be not uniformly irrational but selectively rational—adaptive for managing uncertainty, maladaptive for exploiting stability. In environments where change is constant (financial crises, medical emergencies, social conflicts), rapid affective responses may enable survival through quick strategy shifts. In stable niches (routine procedures, assembly line work), these same mechanisms may become costly noise that disrupts efficient pattern execution.

More intriguingly, why do individuals vary so dramatically in emotional reactivity? Some people remain calm and analytical under pressure; others respond intensely and rapidly to minor provocations. Traditional psychology frames this through personality dimensions (neuroticism, impulsivity, conscientiousness), but our computational model suggests an alternative speculative interpretation: these individual differences might represent different cognitive niche specializations.

If ancestral human populations faced diverse environmental regimes—some groups in relatively stable resource-rich valleys, others in volatile tundra or coastal regions subject to rapid climate shifts during glacial transitions—different emotional response profiles might have been computationally advantageous in different contexts. An individual with high emotional reactivity and rapid affect-driven decision-making (high “Fear” and “Curiosity” sensitivity, low emotional decay rates) might struggle in predictable environments but excel when conditions shift unpredictably, analogous to how ECIA outperforms in Environments A and C. Conversely, an individual with dampened affective responses and stable analytical processing (low emotional sensitivity, fixed learning rates) might show superior performance in predictable contexts but suffer during volatility.

This computational framework does not constitute evidence for specific evolutionary trajectories—establishing such connections would require extensive interdisciplinary research combining agent-based evolutionary modeling, paleoclimatic reconstructions, comparative neuroscience, and cross-cultural psychological studies. However, it offers mechanistic hypotheses that such research could rigorously test: (1) that affective variability may represent computational trade-offs optimized for different environmental regimes rather than simply reflecting “better” or “worse” cognitive control, and (2) that the persistence of emotion-driven decision-making in modern humans may reflect specialization for managing uncertainty rather than representing evolutionary lag or suboptimal design (Bach & Dayan, 2017; Eldar et al., 2016).

Future computational work could implement evolutionary simulations where agent populations face varying environmental volatility over generations, testing whether emotional response diversity emerges spontaneously through selection pressures and whether resulting population-level affective distributions resemble those observed in human behavioral studies.