Reinforcement Learning for Personalised Antidepressant Sequencing in Treatment-Resistant Bipolar Depression: A Simulation-Based Policy Optimisation Study

Treatment-resistant bipolar depression (TRD-BD), defined in this simulation as failure to achieve remission after at least two adequate pharmacological trials for the current bipolar depressive episode, including at least one antidepressant trial delivered with mood-stabilising treatment, is managed through a sequence of uncertain decisions: continue, switch antidepressant class, or augment with a mood stabiliser or atypical antipsychotic. Current practice relies on consensus bipolar-disorder guidelines and sequential-care principles; the STAR*D study is referenced only as a structural example of multistep treatment sequencing in unipolar major depression, not as a validated bipolar-depression protocol. The sequencing problem is therefore a natural candidate for reinforcement learning (RL), but no RL framework has yet been developed specifically for personalised antidepressant sequencing in bipolar depression. We formulated antidepressant sequencing in TRD-BD as a finite-horizon Markov Decision Process and developed an offline RL framework for policy optimisation. A clinically grounded simulator encoded a 15-dimensional policy state comprising MADRS, YMRS, prior treatment failures, weeks on current treatment, side-effect burden, current drug class, a three-dimensional responder-profile covariate, age, and BD subtype, together with an 8-action treatment space (continue; switch to SSRI, SNRI, bupropion, or MAOI; augment with lithium, quetiapine, or lamotrigine). Although the responder profile represents an underlying biological trait, it was supplied to the policy as an oracle covariate in this proof-of-concept simulation; in real practice, it would usually be unobserved, making the clinical problem partially observable. The reward integrated MADRS improvement, remission, side-effect burden, switching cost, and manic-switch risk. A Deep Q-Network (DQN) was trained on 6172 transitions from 800 simulated trajectories generated by a stochastic behavioural clinician policy. Trajectories terminated at remission, manic switch, intolerable adverse effects, or the eight-stage horizon, yielding a mean logged trajectory length of 7.72 stages. Exact behavioural-policy action probabilities were recorded by the simulator and used for importance-weighted off-policy estimators. DQN was compared with Fitted Q-Iteration, the behavioural clinician policy, a fixed sequential protocol, and a random policy. The proposed DQN policy achieved an estimated discounted return of 41.6 (95% CI: 39.0 - 44.0), substantially exceeding Fitted Q-Iteration (24.6), the clinician policy (12.8), and the fixed protocol (12.1). The DQN policy achieved a remission rate of 61.8% (95% CI: 57.0% - 66.0%), nearly quadrupling the clinician policy’s 16.5% and reaching remission in fewer treatment stages (mean 6.5 vs 7.7). Mean MADRS reduction was 24.3 points versus 15.6 for the clinician policy. Bayesian posterior analysis assigned the DQN policy a 100.0% probability of being optimal. Policy analysis revealed that the learned strategy strongly favoured lithium augmentation selected in 88% of states over the clinician’s switch-heavy, lower-augmentation pattern. Within this synthetic environment, reinforcement learning derived a personalised sequencing policy that outperformed the simulated behavioural and fixed-protocol comparators. The learned preference for early lithium augmentation reflects the simulator parameters and should be interpreted as a proof-of-concept result rather than a clinical treatment recommendation. Validation with real TRD-BD trajectories and conservative offline-RL methods is required before any clinical inference or deployment.