1. Introduction

2. Related work

3. Overview

3.2. Architecture

After we tried running multiple types of agents on the various parts of the dataset and obtaining poor performance, we noticed that the performance was correlated with the training data; thus, we concluded that partitioning the data in some meaningful way might be fruitful. We describe this next.

The preprocessing for the high-level agent is done by the clustering mechanism, where we classify each running episode in one of the existing clusters ($c_i\in \{0,1,2,3,4\}$) and feed as state the last n such classifications. To select the low-level agent to act, we consider this representation more appropriate since the classes represent the expertise of each agent in some sense. Moreover, neighbouring episodes will have the same class, thus inducing smoothness in the representation and thus action selection, which is in line with our goals. We show the overall architecture of our learning system in Figure 1.

Figure 1. Depiction of the overall architecture.

Figure 1. Depiction of the overall architecture.

Figure 2. State representation for the high-level agent.

Figure 2. State representation for the high-level agent.

The low-level prediction did not improve much (or at all, the results were inconclusive) the performance of the low-level agents; thus, we do not include it in the final results. Looking at the low-level prediction in Figure 5b, a more accurate prediction model is needed. For the high-level state, we tried multiple approaches:

• using raw returns as in the low-level agent
• using the recent performance of each individual agent in its own simulated environment (similar to [7]) and concatenating these for all agents
• using concatenated samples from the prediction model instead of median or mean
• using the result of the classification of the recent past and the prediction, applying $\mathcal{C}$

The best version used both classification and prediction, see Figure 7. In general, larger state spaces induce more variability, in our experience, which is also what happened when using raw samples from the prediction model. We show the full algorithm in Algorithm 1.

3.2.1. Clustering as preprocessing for the high-level agent

We endow the high-level agent with a preprocessing module that can cluster the incoming time series into a specific cluster (one of 5 discovered beforehand when clustering the training set, see Figure 3).

Figure 3. The 5 clusters discovered.

Figure 3. The 5 clusters discovered.

We denote by $s_h$ the state of the high-level agent. The state at timestep t is given by $s_h=[{\mathbf{c}}_{t-n:t};{\tilde{\mathbf{c}}}_{t:t+m}]$ where ${\mathbf{c}}_{t-n:t}$ is a vector of the last n classifications, where each point $c_{t-i}$ with $0\le i\le n$ denotes the classification of the episode which ends at $t-i$. ${\tilde{\mathbf{c}}}_{t:t+m}$ is the prediction of the following episodes classes by the classification, which is fed with the prediction of the next timesteps. Thus, each point ${\tilde{c}}_{t+j}$ with $0\le j\le m$, is the classification of the (predicted) episode ending at $t+j$. Because the high-level takes decisions at a coarser data granularity, having such a multiple-step prediction model makes sense. We use a DeepAR model for our multiple-step prediction, which generally keeps the shape of the future trend but lacks minute precision, which is what we needed. We show example predictions in Figure 5a.

3.2.2. Pretraining the low-level agents

We pretrain the low-level agents on specific data clusters, so each one specializes in one type of trend. We show the clusters and their means in Figure 3. We see that the 5 clusters catch some of the main trends in the market.

Figure 4. Performance of the low-level agents, each trained on one cluster.

Figure 4. Performance of the low-level agents, each trained on one cluster.

We show Figure 4 the best performance of individual agents on the test set consisting of 15000 points (21 months). We select the best one (among 24 trials) for each class for comparison. We perform 30 repetitions for each and depict the mean and standard deviation.

3.3. Prediction

In the context of RL, prediction assumes the existence of a model of the environment with which the agent can interact without incurring the losses associated with the real environment. This complete model comprises state and reward models, predicting the following states and rewards. We are only interested in the state model, as the reward model is completely dependent on the state model. In the sense that if we know the next state (price) we can easily compute the rewards, given the action taken.

Figure 5. Sample predictions.

Figure 5. Sample predictions.

For the low-level, we tested with the true values instead of the predictions to see if the performance improved. We noticed a significant improvement. We repeated the value of the future step 10 times and then concatenated this to the state vector for the low level as a way of biasing the agent, such that this future value has a more considerable influence on the decision (having a single value did not seem to work too well). So it is clear that accurate predictions can significantly improve performance and training time. We thus add a 1-step prediction for each of the OHLCV time series, thus having five different prediction models for each agent. As we mentioned, this does not improve performance conclusively.

For the high-level agent, the representation is in terms of classes (0 to 4), and thus the prediction should be the same. We also note that the history window size has now doubled compared to the low-level since an episode now defines every point ending at that point. So we predict multiple steps into the future and look at more steps into the past. We show an example in Figure 2.

4. Experiments and results

4.1. Hierarchy with two simple strategies

We test two basic strategies, a mean reversion and a moving average crossover, which can be considered a momentum strategy. Figure 6 shows the performance of the two basic strategies. We then check to see if using a DRL agent to select between the two strategies is of any help. We also add the hold option for the agent to choose from three discrete actions. We show the performance of the DRL agent (we use a PPO agent as before) in Figure 6. We see that performance is indeed better than each of the two individual strategies. We use a minimal $[64,32]$ network with a $relu$ activation function. We try three decision intervals for the PPO: every step, once in 5 steps (PPO 5), and once in 10 steps (PPO 10).

The mean reversion strategy uses two indicators, a Relative Strength Index indicator with an exponential moving window working on a period of 6 hours and a moving average on a period of 20 hours. The strategy buys if the RSI value is less than some threshold (40 in our case) and the price value is less than the lower Bollinger band and sells if the RSI value is higher than some threshold (60 in our case) and the current price is higher than the higher Bollinger. The higher and lower Bollinger thresholds are at $2\times \sigma$, with $sigma$ the rolling standard deviation of the time series (on the same period as the mean, 20 hours). The strategy holds or does nothing if none of the above conditions are met.

The moving average crossover strategy is even more straightforward. We have two different moving averages (in our case, one works on a 100 hours window and the other on a 400 hours window), and when the shorter one surpasses the lower one, it is a buy signal, and when the reverse is the case, that is a sell signal.

Figure 6. Performance of a hierarchical agent (PPO) using two basic strategies as low-level agents.

Figure 6. Performance of a hierarchical agent (PPO) using two basic strategies as low-level agents.

We see that these models can be leveraged to perform better when combined through an additional layer of decision-making. Thus, we include these 2 simple models in our final set of low-level agents. The results in Figure 7 and Figure 8 are obtained with this setup

4.2. Risk-return optimization

The Markowitz model implies that one wants to minimize risk and maximize returns, and a hierarchical approach lends itself naturally to such a dual optimization. We thus test this approach as well, with the high-level agent optimizing for risk. The conditional value rewards the high-level at risk, or CVaR. To define the CVaR, we first need to define the Value at Risk (VaR).

VaR, Value at Risk is a measure describing how much value is at risk during a specific period and its probability of occurrence. Many critics have criticized this as being uninformative or giving a false measure of certainty and security where there should be none. Some of the main arguments against using VaR is that it is a measure of describing rare events that are hard to describe.

Formally, let X be profits (positive) and losses (negative) random variable (rv) associated with some distribution. The VaR at level $\alpha \in (0,1)$ gives the smallest number y such that the probability that $Y:=-X$ is not larger than y is at least $1-\alpha$. We denote this as $Va{R}_{\alpha }\left(X\right)$, and the most general definition is given by:

$$Va{R}_{\alpha }\left(X\right)=-inf\{x\in \mathbb{R}:F_X\left(x\right)>\alpha \}=F_Y^{-1}(1-\alpha )$$

(6)

The function $F_X$ is the cumulative distribution function (cdf) of the random variable X (for any rv, its cdf is well defined). There are multiple ways of computing the VaR:

• either by looking at past returns, assuming that future outcomes will be similar to past ones
• by assuming a parametric distribution of the returns, like a Gaussian
• by using Monte Carlo simulations of predicted returns

We use the first method as it is the simplest and quite efficient. We use $\alpha =0.05$.

CVaR, Conditional Value at Risk is a measure related to VaR but more comprehensive as it looks at the expected value beyond the VaR threshold:

$$CVa{R}_{\alpha }\left(X\right)=-\frac{1}{\alpha }{\int }_0^{\alpha }Va{R}_{\gamma }\left(X\right)d\gamma$$

(7)

If we assume that the distribution is continuous, than this is equivalent to the conditional tail expectation given by:

$$\mathbb{E}[-X|X\le -Va{R}_{\alpha }\left(X\right)]$$

(8)

CVaR is also known as the expected shortfall and can be seen as describing the average loss case when the losses are severe (they only occur $\alpha$ percent of the time).

When comparing the original version, using a cumulative return for the high-level, versus the risk-return version, using CVaR as a reward, we get a significant difference, with a larger Sharpe ratio for the CVaR agent (running a t-test between the two populations, gave a p-value of 0.02, so the null hypothesis can be rejected, i.e., the two populations means are not equal).

Figure 7. Final performance on BTCUSDT of major models tested.

Figure 7. Final performance on BTCUSDT of major models tested.

Figure 8. Cumulative rewards versus CVaR.

Figure 8. Cumulative rewards versus CVaR.

To evaluate the performance of the risk-return model, we compute the monthly Sharpe ratio, which will be independent of the frequency of trading or data used. Otherwise, we risk sampling bias, with the Sharpe ratio dependent on the period length used for calculation. The monthly Sharpe ratio is computed as follows:

$$\begin{array}{c}S=\frac{R_e}{{\sigma }_e}\mathrm{with}{R}_e=\frac{1}{n}\sum _{i=1}^n(R_i-R_f)\mathrm{and}\hfill \\ \hfill {\sigma }_e=\sqrt{\frac{1}{n-1}{\sum }_{i=1}^n{(R_i-R_f-R_e)}^2}\end{array}$$

(9)

where $R_e$ is the average monthly excess return, $R_i$ is the return of the portfolio in month i, and $R_f$ is the risk-free excess return (which can be taken to vary each month but which we keep constant for simplicity) and n is the number of months. The risk-free return is generally quite low (initially taken as the return of US bonds); we take it as $5\%$.

5. Conclusion

6. Future directions

References

1. Mnih, V.; Kavukcuoglu, K.; Silver, D.; Rusu, A.A.; Veness, J.; Bellemare, M.G.; Graves, A.; Riedmiller, M.; Fidjeland, A.K.; Ostrovski, G.; et al. Human-level control through deep reinforcement learning. nature 2015, 518, 529–533. [Google Scholar] [CrossRef] [PubMed]
2. Li, Y. Deep reinforcement learning: An overview. arXiv preprint arXiv:1701.07274 2017. arXiv:1701.07274 2017.
3. Millea, A. Deep reinforcement learning for trading—A critical survey. Data 2021, 6, 119. [Google Scholar] [CrossRef]
4. Pricope, T.V. Deep reinforcement learning in quantitative algorithmic trading: A review. arXiv preprint arXiv:2106.00123 2021. arXiv:2106.00123 2021.
5. Zhang, Z.; Zohren, S.; Roberts, S. Deep reinforcement learning for trading. The Journal of Financial Data Science 2020, 2, 25–40. [Google Scholar] [CrossRef]
6. Betancourt, C.; Chen, W.H. Deep reinforcement learning for portfolio management of markets with a dynamic number of assets. Expert Systems with Applications 2021, 164, 114002. [Google Scholar]
7. Millea, A.; Edalat, A. Using Deep Reinforcement Learning with Hierarchical Risk Parity for Portfolio Optimization. International Journal of Financial Studies 2022, 11, 10. [Google Scholar]
8. Ebens, H.; Kotecha, C.; Ypsilanti, A.; Reiss, A. Introducing the multi-asset strategy index. The Journal of Alternative Investments 2008, 11, 6–25. [Google Scholar] [CrossRef]
9. Cornalba, F.; Disselkamp, C.; Scassola, D.; Helf, C. Multi-Objective reward generalization: Improving performance of Deep Reinforcement Learning for selected applications in stock and cryptocurrency trading. arXiv preprint arXiv:2203.04579 2022. arXiv:2203.04579 2022.
10. Dai, M.; Zhang, Q.; Zhu, Q.J. Trend following trading under a regime switching model. SIAM Journal on Financial Mathematics 2010, 1, 780–810. [Google Scholar] [CrossRef]
11. Ang, A.; Timmermann, A. Regime changes and financial markets. Annu. Rev. Financ. Econ. 2012, 4, 313–337. [Google Scholar]
12. Sutton, R.S.; Barto, A.G. Reinforcement learning: An introduction; MIT press, 2018.
13. Guo, X.; Hernández-Lerma, O. Continuous-time Markov decision processes. In Continuous-Time Markov Decision Processes; Springer, 2009; pp. 9–18.
14. Yu, P.; Lee, J.S.; Kulyatin, I.; Shi, Z.; Dasgupta, S. Model-based deep reinforcement learning for dynamic portfolio optimization. arXiv preprint arXiv:1901.08740 2019. arXiv:1901.08740 2019.
15. Moerland, T.M.; Broekens, J.; Jonker, C.M. Model-based reinforcement learning: A survey. arXiv preprint arXiv:2006.16712 2020. arXiv:2006.16712 2020.
16. Deisenroth, M.; Rasmussen, C.E. PILCO: A model-based and data-efficient approach to policy search. In Proceedings of the Proceedings of the 28th International Conference on machine learning (ICML-11). Citeseer, 2011, pp. 465–472.
17. Ha, D.; Schmidhuber, J. World models. arXiv preprint arXiv:1803.10122 2018. arXiv:1803.10122 2018.
18. Wei, H.; Wang, Y.; Mangu, L.; Decker, K. Model-based reinforcement learning for predictions and control for limit order books. arXiv preprint arXiv:1910.03743 2019. arXiv:1910.03743 2019.
19. Bishop, C.M. Mixture density networks. Neural Computing Research Group Report: NCRG/94/0041 1994.
20. Levy, A.; Platt, R.; Saenko, K. Hierarchical actor-critic. arXiv preprint arXiv:1712.00948 2017, arXiv:1712.00948 2017, 1212. [Google Scholar]
21. Nachum, O.; Gu, S.S.; Lee, H.; Levine, S. Data-efficient hierarchical reinforcement learning. Advances in neural information processing systems 2018, 31. [Google Scholar]
22. Vezhnevets, A.S.; Osindero, S.; Schaul, T.; Heess, N.; Jaderberg, M.; Silver, D.; Kavukcuoglu, K. Feudal networks for hierarchical reinforcement learning. In Proceedings of the International Conference on Machine Learning. PMLR; 2017; pp. 3540–3549. [Google Scholar]
23. Dietterich, T.G.; et al. The MAXQ Method for Hierarchical Reinforcement Learning. In Proceedings of the ICML. Citeseer, Vol. 98; 1998; pp. 118–126. [Google Scholar]
24. Wiering, M.; Schmidhuber, J. HQ-learning. Adaptive Behavior 1997, 6, 219–246. [Google Scholar]
25. Pateria, S.; Subagdja, B.; Tan, A.h.; Quek, C. Hierarchical reinforcement learning: A comprehensive survey. ACM Computing Surveys (CSUR) 2021, 54, 1–35. [Google Scholar]
26. Suri, K.; Shi, X.Q.; Plataniotis, K.; Lawryshyn, Y. TradeR: Practical Deep Hierarchical Reinforcement Learning for Trade Execution. arXiv preprint arXiv:2104.00620 2021. arXiv:2104.00620 2021.
27. Wang, R.; Wei, H.; An, B.; Feng, Z.; Yao, J. Commission fee is not enough: A hierarchical reinforced framework for portfolio management. arXiv preprint arXiv:2012.12620 2020. arXiv:2012.12620 2020.
28. Botvinick, M.; Weinstein, A. Model-based hierarchical reinforcement learning and human action control. Philosophical Transactions of the Royal Society B: Biological Sciences 2014, 369, 20130480. [Google Scholar] [CrossRef] [PubMed]
29. Xu, D.; Fekri, F. Interpretable Model-based Hierarchical Reinforcement Learning using Inductive Logic Programming. arXiv preprint arXiv:2106.11417 2021. arXiv:2106.11417 2021.
30. Sutton, R.S.; Precup, D.; Singh, S. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial intelligence 1999, 112, 181–211. [Google Scholar] [CrossRef]
31. Kulkarni, T.D.; Saeedi, A.; Gautam, S.; Gershman, S.J. Deep successor reinforcement learning. arXiv preprint arXiv:1606.02396 2016. arXiv:1606.02396 2016.
32. Srinivas, A.; Krishnamurthy, R.; Kumar, P.; Ravindran, B. Option discovery in hierarchical reinforcement learning using spatio-temporal clustering. arXiv preprint arXiv:1605.05359 2016. arXiv:1605.05359 2016.
33. Lloyd, S. Least square quantization in PCM. Bell Telephone Laboratories Paper. Published in journal much later: Lloyd, SP: Least squares quantization in PCM. IEEE Trans. Inform. Theor.(1957/1982) 1957, 18, 11. [Google Scholar]
34. Salinas, D.; Flunkert, V.; Gasthaus, J.; Januschowski, T. DeepAR: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting 2020, 36, 1181–1191. [Google Scholar]