Bandits atop Reinforcement Learning: Tackling Online Inventory Models with Cyclic Demands.

This article addresses the challenge of online inventory management under unknown cyclic stochastic demand distributions, a setting where demand patterns repeat periodically but are not known in advance. It develops reinforcement learning (RL) algorithms—specifically, Elimination-based Half Q-Learning (HQL) for episodic lost-sales models, Full Q-Learning (FQL) for episodic multiproduct backlogging models, and Meta-HQL and Mimic-QL for nondiscarding variants—that achieve optimal or near-optimal regret bounds, measured against clairvoyant policies with full demand knowledge. The algorithms leverage structural properties of inventory problems, such as one-sided and full feedback, to remove regret dependence on the size of the state-action space, improving over prior methods that assume independent and identically distributed demands. Empirical evaluations on synthetic data and real sales data from Rossmann, a major European drugstore chain, demonstrate that Meta-HQL rapidly converges to near-optimal policies and significantly outperforms policies assuming i.i.d. demand. The paper also discusses extensions to multiproduct settings with lead times, fixed ordering costs, and order limits, and highlights applications of the proposed methods to other operations research problems exhibiting similar feedback structures.