Article
Version 1
Preserved in Portico This version is not peer-reviewed
Hierarchical Model-Based Deep Reinforcement Learning For Single-Asset Trading
Version 1
: Received: 19 May 2023 / Approved: 22 May 2023 / Online: 22 May 2023 (16:24:21 CEST)
A peer-reviewed article of this Preprint also exists.
Millea, A. Hierarchical Model-Based Deep Reinforcement Learning for Single-Asset Trading. Analytics 2023, 2, 560-576. Millea, A. Hierarchical Model-Based Deep Reinforcement Learning for Single-Asset Trading. Analytics 2023, 2, 560-576.
Abstract
We present a hierarchical reinforcement learning (RL) architecture that employs various low-level agents to act in the trading environment, i.e. the market. The highest level agent selects among a group of specialised agents, and then the selected agent decides when to sell or buy a single asset for some period. This period can be variable according to a termination function. We hypothesized that due to different market regimes, more than one single agent is needed when trying to learn from such heterogeneous data, and instead, multiple agents will perform better, with each one specialising in a subset of the data. We use $k-means$ clustering to partition the data and train each agent with a different cluster. Partitioning the input data also helps model-based RL (MBRL), where models can be heterogeneous. We also add two simple decision-making models to the set of low-level agents, diversifying the pool of available agents and thus increasing overall behaviour flexibility. We perform multiple experiments showing the strengths of a hierarchical approach and test various prediction models at both levels. We also use a risk-based reward at the high level, which transforms the overall problem into a risk-return optimization. This type of reward shows a significant reduction in risk while minimally reducing profits. Overall, the hierarchical approach shows significant promise, especially when the pool of low-level agents is highly diverse.
Keywords
deep reinforcement learning; trading; clustering; hierarchy; prediction; risk; bitcoin
Subject
Computer Science and Mathematics, Artificial Intelligence and Machine Learning
Copyright: This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
Comments (0)
We encourage comments and feedback from a broad range of readers. See criteria for comments and our Diversity statement.
Leave a public commentSend a private comment to the author(s)
* All users must log in before leaving a comment