Suppose an insurance company has surplus $X_t$ at time t, with where $\mu >0$, $\sigma >0$, and ${\left\{W_t\right\}}_{t\ge 0}$ is a standard Brownian motion defined on the filtered probability space $(\mathrm{\Omega },\mathcal{F},{\left\{{\mathcal{F}}_t\right\}}_{t\ge 0},\mathbb{P})$. As remarked by Asmussen and Taksar (1997), such a surplus process (1) can be viewed either as a direct modelling with drifted Brownian motion or as an approximation to the classical compound Poisson model.

A dividend strategy or policy is defined as $\mathbf{a}={\left\{a_t\right\}}_{t\ge 0}$, where $a_t$ is the dividend paying rate at time t, i.e., the cumulative amount of dividends paid from time $t_1$ to time $t_2$ is given by ${\int }_{t_1}^{t_2}{a}_{t}dt$. We consider herein the Markov feedback controls, i.e., $a_t=a\left(X_t\right)$, where $a(·)$ is a function of the surplus level $X_t$. Note that $a_t$ is nonnegative for any t. Further, we assume that $a_t$ is upper bounded by a positive constant M, which is consistent with the assumption made in the literature. We give the formal definition of the admissible dividend policy below.

Denote by $\mathcal{A}$ the set of admissible dividend policies. For an insurance company whose surplus process evolves according to (1) and pays the dividend according to policy $\mathbf{a}={\left\{a_t\right\}}_{t\ge 0}\in \mathcal{A}$, the controlled surplus process for this insurance company is Define the ruin time to be the first time that the surplus level hits zero, i.e., For an insurance company starting with initial surplus $x\in [0,\infty )$, the problem is to find the optimal dividend policy that maximizes the expected value of exponentially discounted dividends to be accumulated until the ruin time, that is, where $\rho >0$ is the discounting rate. Then the optimal dividend problem is

First, we briefly review the results of solving the dividend optimization problem (3) classically. Let $V_{cl}\left(x\right)$ be the value function of the dividend optimization problem: Assume that the value function $V_{cl}\left(x\right)$ is twice-continuously differentiable. The standard dynamic programming approach leads to the following Hamilton-Jacobi-Bellman equation, with boundary condition $V_{cl}\left(0\right)=0$. It can be easily seen that the optimal dividend paying rate at surplus level x is Assume that $V_{cl}\left(x\right)$ is a concave function. Then there exists a nonnegative constant $x_b$ such that $V_{cl}^{\prime }\left(x\right)\le 1$ when $x\ge x_b$ and $V_{cl}^{\prime }\left(x\right)>1$ when $0\le x<x_b$. Substitute (5) into (4), then it turns into the following ODEs: Combining with the boundary condition, one can derive that where $C_1$, $C_2$ and $x_b$ are determined by the smooth pasting conditions, i.e., If $\frac{M}{\rho }-\frac{1}{{\theta }_3}>0$, there exists unique solution to (8). In this case, $V_{cl}\left(x\right)$ is given by (7), where $C_1,C_2,x_b$ are determined uniquely through (8). Consequently, the optimal dividend policy is to pay the maximal rate M when surplus level x exceeds threshold $x_b$ and to pay nothing if else. If $\frac{M}{\rho }-\frac{1}{{\theta }_3}\le 0$, then $V_{cl}\left(x\right)=\frac{M}{\rho }(1-e^{-{\theta }_{3}x})$. In this case, the optimal dividend policy is always to pay the maximal rate M. Detailed proofs can be found in Asmussen and Taksar (1997). It is also straightforward to check that the optimal value function $V_{cl}\left(x\right)$ is concave on x and always smaller than $M/\rho$ which is the limit of $V_{cl}\left(x\right)$ as x going to infinity. Figure 1 below illustrates the value function and the corresponding optimal dividend policy under the following parameter values: $\mu =1$, $\sigma =1$, $\rho =0.3$, $M=0.6$ (left panels), $M=1.2$ (middle panels), and $M=1.8$ (right panels), respectively.

The above optimal dividend policy (5) is implemented based on the complete information, i.e., the model parameters $\mu$ and $\sigma$ are known. In reality, however, it is difficult to know exactly the values of $\mu$ and $\sigma$, due to the uncertainty in premium rate, claim arrival process and claim size. Therefore, we consider a technique named reinforcement learning to learn the optimal (or near-optimal) dividend paying strategy through trial-and-error interactions with the unknown territory.

Whereas the majority of the work in reinforcement learning consider Markov decision process in discrete time, we follow the pioneering work of Wang et al. (2020) who model the reinforcement learning in continuous time as a relaxed stochastic control problem. At time t with surplus level $X_t$ the dividend paying rate a is randomly sampled according to a distribution ${\pi }_t:=\pi (a;X_t)$, where $\pi (·;·):[0,M]\times [0,\infty )\mapsto [0,\infty )$, satisfying ${\int }_0^M\pi (a;x)da=1$ for any $x\in [0,\infty )$. We call $\pi :={\left\{{\pi }_t\right\}}_{t\ge 0}$ the distributional dividend policy. Following the same procedure as in Wang et al. (2020), we derive the exploratory dynamic of the surplus process under $\pi$ to be and the expected value of total discounted dividends under exploration to be where the ruin time is In addition to the expected value of total discounted dividends under exploration, Shannon’s differential entropy is introduced into the objective to encourage exploration. For a given distribution $\pi$, the entropy is defined as Thus, the objective of entropy-regularized exploratory dividend problem is where $\lambda >0$ is the so-called temperature parameter. Note that $\lambda$ controls the weight to be put on the exploration and is exogenously given. If $\lambda =0$, the distribution degenerates to the Dirac measure, which is the solution to classical optimal dividend problem without exploration. The entropy-regularized exploratory dividend problem is where $\Pi$ is the set of admissible exploratory dividend policies. We give the formal definition of admissible exploratory dividend policy $\pi$ below.

The following proposition will be used later.

To solve the supremum in (13) together with the constraint that ${\int }_0^M\pi (a;x)da=1$, we introduce the Lagrange multiplier $\eta$: Maximizing the integrand above pointwisely and using the first-order condition leads to the solution Because ${\int }_0^M{\pi }^{*}(a;x)da=1$, we solve that where

Recall that the classical optimal dividend policy given in (5) is two-threshold strategy, i.e., it pays nothing, $a^{*}\left(x\right)=0$, if $V_{cl}^{\prime }\left(x\right)>1$ or pays the maximal rate, $a^{*}\left(x\right)=M$, if $V_{cl}^{\prime }\left(x\right)\le 1$. In contrast, the exploratory dividend policy is not restricted to two extreme actions only but gives the probability to take certain actions. This result is very similar to [38] in which the authors study the temperature control problem for Langevin diffusions by incorporating randomization of the temperature control and regularizing its entropy. The classical optimal control of such a problem is of the bang-bang type, whereas the exploratory control is a state-dependent, truncated exponential distribution. Likewise, the optimal distribution ${\pi }^{*}(a;x)$ given in (15) is also a continuous version of Boltzmann distribution or Gibbs measure which is widely used in discrete reinforcement learning.

When $V^{\prime }\left(x\right)>1$, ${\pi }^{*}(a;x)$ is decreasing in a so it has large probability to take small dividend pay-out rate close to 0; when $V^{\prime }\left(x\right)<1$, ${\pi }^{*}(a;x)$ is increasing in a so it has large probability to take large dividend pay-out rate close to M ; when $V^{\prime }\left(x\right)=1$, it degenerates to a uniform distribution on $[0,M]$. In other words, the optimal exploratory dividend policy is an “exploration” of the classical dividend pay-out policy: it searches around the current optimal dividend rate given by the classical solution, 0 or M, with the probability to take a certain rate decreasing as it moving away from the classical solution.

The exploratory surplus process under the optimal policy is well-posed. Note that the optimal distributional policy is ${\pi }^{*}={\left\{{\pi }_t^{*}\right\}}_{t\ge 0}$, where Applying the optimal distributional policy (17) into the exploratory surplus process (9), we obtain that Since ${\int }_0^{M}a{\pi }^{*}(a;X_t^{{\pi }^{*}})da\in [0,M]$, the SDE (18) has bounded drift and constant volatility. As a result, there exists unique solution $\left\{X_t^{{\pi }^{*}}\right\}$ to (18).

Substituting the optimal distribution ${\pi }^{*}(a;x)$ as shown in (15) into the HJB equation (13), we have the following equation for $V\left(x\right)$: or equivalently, The following verification theorem shows that $V\left(x\right)$ that solves (19) is indeed the value function of the exploratory dividend problem (12).

Theorem 1 shows that solution to the exploratory HJB equation (19) could be the value function of exploratory dividend problem (12). On the other hand, a similar argument could show that the value function shall also satisfy (19), while the optimal exploratory dividend strategy is given by (17). To establish a rigorous statement, we need the following result. The next proposition shows that the value function $V\left(x\right)$ converges as x going to infinity.

Compared with the differential equation (6) which solves the classical value function, the exploratory HJB equation (19) has a nonlinear term $ln{Z}_M((1-V^{\prime }\left(x\right))/\lambda )$, which makes it difficult to be solved explicitly. The theorem below guarantees the existence and uniqueness of solution $V\left(x\right)$.

Theorem 2 follows from the results in Tang et al. (2022, Theorem 3.9, 3.10) and in Strulovici and Szydlowski (2015, Proposition 1). It is straightforward to check that the conditions to guarantee the existence and uniqueness of the solution to (19) and its twice-continuous differentiability are satisfied.

Theorem 2 also states that when $\lambda$ becomes smaller, the exploratory value function converges to the classical value function. Indeed, a stronger convergence is established by Tang et al. (2022) that V converges to $V_{cl}$ locally uniformly as $\lambda$ going to 0. Note that the parameter $\lambda$ is the weight to be put on the exploration in contrast to the exploitation. If it is more close to 0, the entropy term has smaller effect on the total objective value and the optimal exploratory distribution ${\pi }^{*}(a;x)$ in (15) are more concentrated and close to the Dirac distribution – the optimal solution to the classical dividend optimization problem. Then not surprisingly, the exploratory value function $V\left(x\right)$ also converges to the classical value function $V_{cl}\left(x\right)$ as $\lambda$ going to 0.

Now, thanks to Theorem 2, we have $V\left(x\right)$ that solves the exploratory HJB equation (19). On the other hand, it is straightforward to show that according to (20), if $M<\lambda ln\left(1/\lambda +1\right)$, the limit of $V\left(x\right)$ is negative; if $M>\lambda ln\left(1/\lambda +1\right)$, the limit of $V\left(x\right)$ is positive; if $M=\lambda ln\left(1/\lambda +1\right)$, the limit of $V\left(x\right)$ is zero. The next theorem shows that indeed, we classify $V\left(x\right)$ into three cases based on its monotonicity.

The following corollary is a direct result from above theorem.

Note that in Theorem 1 we need $\left|V\right|$ and $|V^{\prime }|$ to be bounded so that V – solution to (19) – is indeed the value function of exploratory optimal dividend problem. Corollary 1 verifies the boundedness conditions are satisfied. In other words, the solution to the exploratory HJB equation (19) is the value function of the exploratory dividend problem.

On the other hand, the above inequality becomes an equality if the supremum in (13) is achieved, that is, $\tilde{\pi }={\pi }^{*}$, where ${\pi }^{*}$ is given by (15). Thus, $V\left(x\right)$ is the value function. Q.E.D.

Define function $G_{\lambda ,M}$ to be where function $Z_M$ is given in (16).