The stochastic and dynamic inventory routing problem with fine time granularity: A reinforcement learning approach

Reinforcement learning (RL) is an approach in which machines solve learning problems effectively, evaluating designs through mathematical analysis or computational experiments. In this work, we apply and extend these RL methods to the inventory routing problem (IRP), a problem that integrates inventory management with vehicle routing and delivery scheduling decisions over several periods. We present a new problem, the stochastic and dynamic IRP with fine time granularity (SDIRPFG), in which the agent must make real-time plan revisions under uncertainty, with time periods representing a fine time discretization. Information on customer inventory levels is provided to the agent, who makes delivery and routing decisions, changing or adjusting the current plan in place to handle events that arise during the operation. We propose benchmark policies for the SDIRPFG, myopic, rollout and cost function approximation (CFA) policies, and we also present online Monte Carlo (MC) control methods with value function approximations (VFAs). For the last method, we propose new learning methodologies and approximations. In particular, we introduce function approximations using Fourier bases, an interpretable approximation barely studied in the literature. In an extensive computational study, including synthetic and real-world instances, we test our policies, methods, and VFAs. We study the convergence of objective functions, provide guidelines on how to use our methods, and report SDIRPFG insights and characteristics. Using cost functions that we can plot, we show that our VFA is interpretable.