This paper reviews the multiple roles of machine learning in improving the resilience of power grids, especially in applying new energy storage technologies. Energy storage technologies, such as compressed air energy storage, flywheel energy storage, and superconducting coil energy storage, significantly improve the power grid's ability to respond to load fluctuations and emergencies through intelligent control and optimisation of machine learning algorithms. This not only helps to optimise energy dispatch and improve the operational efficiency and flexibility of the grid but also significantly improves the stability and reliability of the grid so that it is better able to meet future challenges and needs.

This paper systematically discusses the application of reinforcement learning in automatic driving system. Reinforcement learning frameworks show significant advantages in optimizing decision making, predictive perception, path planning, and controller design, exceeding the limitations of traditional supervised learning methods. The paper highlights the critical role of components such as scene understanding, positioning, and map making in autonomous driving systems, which provide reliable environmental awareness through deep learning and sensor fusion technologies to support intelligent decision-making in complex urban environments. In addition, the paper discusses innovative approaches to safety reinforcement learning to reduce risk in autonomous driving and ensure that systems adhere strictly to safety constraints while maximizing expected rewards. These findings provide an important theoretical and practical basis for further improving algorithm robustness, managing multi-agent interactions, and integrating ethical considerations in the future.

This study presents a groundbreaking approach in the field of gaming AI, focusing on the classic game” Pacman” through the lens of DRL integrated with advanced optimization techniques. The core of the research involves adapting DRL models for” Pacman”, utilizing the ESO for hyper- parameter tuning. These novel adaptations significantly enhance the AI agent’s performance, demonstrating a remarkable improvement in adaptability, responsiveness, and efficiency within the game environment. A pivotal aspect of this research is the innovative integration of metaheuristic optimization techniques into the DRL framework, a first in the domain of Atari gaming AI. This integration has proven essential in fine-tuning the DRL models, leading to a more effective and dynamic gaming experience. The study thoroughly evaluates and compares the performance of these algorithms, providing empirical support for their effectiveness and setting a new benchmark in AI-driven game development. The implications of this research extend beyond gaming AI, opening up possibilities for future exploration in several directions. Expanding DRL models to other complex gaming environments, continuous algorithmic enhancements, real-time learning adaptations, and applying these principles to robotics and autonomous vehicles are examples. The research also emphasizes ethical and responsible AI use in gaming to address fairness and addiction issues.

This paper presents MADS (Multi-Agents for Data Science), an innovative and simple open-source multi-agent framework designed for systemic pipeline execution in applied supervised machine learning. By leveraging the capabilities of multi-agent systems (MAS), we introduce a universal approach to optimize and streamline machine learning pipelines. Our framework highlights the differences between various types of agents, such as reinforcement learning (RL) agents and large language model (LLM) agents, and their distinct contributions to the process. While we currently employ LLM agents to automate and enhance machine learning tasks, we acknowledge the potential of incorporating RL agents in future iterations to further improve performance and adaptability. The primary objective is to enhance the efficiency, scalability, and adaptability of supervised learning applications across various domains. This integration addresses the complexity and manual effort typically associated with machine learning workflows, paving the way for more automated, robust, and scalable solutions. Our approach demonstrates significant improvements in task automation, reduced human intervention, and enhanced model performance. The MADS framework, which will soon be available as an open-source implementation, represents a pivotal contribution to the field of machine learning, promising to facilitate broader adoption and collaborative advancement.

While machine learning models are powering more and more everyday devices, there is a growing need for explaining them. This especially applies to the use of Deep Reinforcement Learning in solutions that require security, such as vehicle motion planning. In this paper, we propose a method of understanding what the RL agent’s decision is based on. The method relies on conducting statistical analysis on a massive set of state-decisions samples. It indicates which input features have an impact on the agent’s decision and the relationships between decisions, the significance of the input features, and their values. The method allows us for determining whether the process of making a decision by the agent is coherent with human intuition and what contradicts it. We applied the proposed method to the RL motion planning agent which is supposed to drive a vehicle safely and efficiently on a highway. We find out that making such analysis allows for a better understanding agent’s decisions, inspecting its behavior, debugging the ANN model, and verifying the correctness of input values, which increases its credibility.

Presence detection is essential in many applications, including smart homes, office automation, and public space management. Traditional technologies like infrared sensors and cameras frequently encounter obstacles due to privacy concerns, high costs, and complexity. The study provides a novel way to detect presence based on Wi-Fi signals and machine learning algorithms, utilizing the ESP8266 microcontroller. The suggested system attempts to reliably detect the presence of people in a given place by evaluating fluctuations in Wi-Fi signal strength and patterns. The theoretical basis of this strategy is investigated, including the ESP8266’s capabilities for Wi-Fi data gathering and the use of machine learning methods for data analysis. The methodology includes data collecting with the ESP8266, preprocessing, feature extraction, and training and evaluating machine learning models. This system’s potential uses include smart home automation, office occupancy tracking, and crowd management in public settings. The paper also discusses potential issues such as signal interference, data privacy concerns, and the limitations of Wi-Fi detection. Future work is also mentioned, including the suggested system’s installation and testing and enhancement proposals to improve accuracy and scalability. This research aims to provide a cost-effective, non-intrusive, and scalable presence detection system by harnessing the power of Wi-Fi signals and machine learning, enhancing intelligent environments

In this paper, we investigate the proftability of investing in movies for retail investors, focusing on the return on investment (ROI) metric, which is calculated as revenue divided by budget. Our analysis encompasses 5000 movies from the movie database TMDB, examining factors such as production scale, genres, key actors, directors, and more. Additionally, we employ textual analysis techniques like Latent Semantic Analysis (LSA) on movie overviews and tags to incorporate movie themes. We then evaluate various supervised classifcation models including Logistic Regression, Random Forest, and Light Gradient Boosting (LightGBM), comparing their performance. Our fndings highlight the signifcance of production scale, team structure, and movie themes in identifying potential high-return opportunities for investors.

This paper explores the application of machine learning in financial time series analysis, focusing on predicting trends in financial enterprise stocks and economic data. It begins by distinguishing stocks from stocks and elucidates risk management strategies in the stock market. Traditional statistical methods such as ARIMA and exponential smoothing are discussed in terms of their advantages and limitations in economic forecasting. Subsequently, the effectiveness of machine learning techniques, particularly LSTM and CNN-BiLSTM hybrid models, in financial market prediction is detailed, highlighting their capability to capture nonlinear patterns in dynamic markets. The study demonstrates the advancements in predictive accuracy and robustness achieved by deep learning methods through empirical analysis and model validation. The findings contribute significantly to academic discourse and offer practical insights for investors, financial analysts, and policymakers navigating market volatility and optimizing investment strategies. Finally, the paper outlines prospects for machine learning in financial forecasting, laying a theoretical foundation and methodological framework for achieving more precise and reliable economic predictions.

With the widespread application of artificial intelligence technology in various industries, users' attention to privacy and data security has increased significantly. Federated learning, as a new technology paradigm combining privacy-enhanced computing and artificial intelligence, resolves the contradiction between data security and open sharing. This paper presents the benefits of federated learning in terms of privacy, real-time processing, model robustness, compliance and cross-industry applications. At the same time, when combined with Edge AI technology, federated learning promotes the decentralisation of intelligent systems, improving data privacy protection and model accuracy. This paper also discusses the application cases of federated learning in the medical field, through local data processing and model training, effectively protecting user privacy, realizing medical data sharing and model optimization, and promoting the development of artificial intelligence.

This paper systematically discusses the application and development of federated learning in data privacy protection and data value sharing. With the rapid development of global information technology, especially the explosive growth of data from Internet of Things devices, data security and privacy protection are facing unprecedented challenges. This paper first analyzes the growth trend of global data volume and its importance to next generation technologies such as artificial intelligence technologies such as deep learning. Second, the paper provides an in-depth look at the impact of current data privacy regulations on data flows and value creation, particularly the EU's GDPR and China's Data Security and Personal Information Protection Law. Then, this paper introduces in detail federated learning, as a new distributed machine learning paradigm, which effectively solves the contradiction between existing data sharing and privacy protection by protecting individual data privacy and realizing global model collaborative construction. Finally, this paper discusses the combination of blockchain technology and federated learning, and proposes BeFL architecture as a new secure, decentralized and trusted federated learning system, which is expected to provide a comprehensive solution for large-scale data processing and value creation in multi-party scenarios. The research in this paper not only deepens the understanding of federation learning in theory, but also provides important reference and enlightenment for future research and application in related fields.

In this report we describe the case of a healthy, young, athletic woman who developed acute lymphoblastic leukaemia (ALL)/lymphoblastic lymphoma (LBL) after receiving the second dose of the Pfizer/BioNTech modified mRNA (modRNA) COVID-19 genetic vaccine (marketed as Comirnaty^®). The first dose of the genetic vaccine did not appear to illicit any noticeable side effects, but within 24 hours of the second dose the patient suffered widespread and intensifying bone pain, fever, vomiting, and general malaise. Due to the persistence of the symptoms, the patient underwent a series of tests and examinations including a full laboratory workup, a consult with a clinical immunologist and rheumatologist, a Positron Emission Tomography (PET) imaging, as well as an osteomedullary biopsy. These together led to a definitive diagnosis of ALL. A time interval of 16 weeks from the second vaccination to the diagnosis of cancer was noted. Several similar cases with identical pathology which developed after the modRNA COVID-19 vaccination, are described in case reports in the scientific literature. The massive and indiscriminate use of genetic vaccines to fight COVID-19 is raising serious concerns about their safety and about the technology platform as a whole for this purpose. Growing evidence is accumulating regarding the biodistribution and persistence of the modRNA which can reach, thanks to the lipid nanoparticles, a multitude of tissues and organs of the body, including the bone marrow and other blood-forming organs and tissues. Moreover, there is evidence that the modRNA vaccines display a particular tropism for the bone marrow, influencing the immune system at multiple levels and being able to trigger not only autoimmune-based pathologies, but also neoplastic mechanisms. The aim of this article is to assess, on the basis of the available scientific literature, the risk of developing haematopoietic cancers after modRNA vaccination, and to investigate the potential genetic mechanisms involved in the pathogenesis of disease.

With the continuous advancement of digital transformation, digital payments are playing an increasingly important role in the financial industry. This study aims to utilize machine learning models to predict and analyze digital payment behavior. Initially, the background and significance of digital payments in the financial sector are introduced. Subsequently, the current status and trends of traditional digital payment distribution are reviewed, alongside related work on digital payment behavior prediction. Methodologically, principles and applications of machine learning models such as logistic regression, decision trees, and random forests are elaborated, along with experimental design and data preprocessing methods. The experimental results and discussion section illustrates the performance of each model in digital payment prediction and explores their impact on credit decisions. This exploration equips financial institutions with more effective user behavior analysis and risk management tools, thereby fostering future development and application of digital payment technologies.

A new data processing method based on Machine Learning (ML) algorithms has been developed and tested in clinical environments during COVID-19 trials at the Medical University of South Carolina (MUSC). Breath samples have been collected in 2020–2021 from 100 participants using gas chromatography combined with ion mobility spectrometry (GC-IMS, G.A.S. mbH). Polymerase chain reaction (PCR) tests combined with clinical diagnostics were used as a reference method. Once the ML training-testing cycle is completed, the diagnostics of a single data set may be completed in as little as a few seconds. A convincing difference in the Volatile Organic Compounds (VOS) breath biomarker signature with COVID-19 positive and negative participants has been found. This approach is a generic method that can be applied to data sets collected with other types of hardware, e.g., different settings of GC, GC-MS, GC-GC, or other methods providing high information density in 2-dimensional data sets.

Reinforcement Learning (RL) has demonstrated considerable potential in solving challenges across various domains, notably in autonomous driving. Nevertheless, implementing RL in autonomous driving comes with its own set of difficulties, such as the overestimation phenomenon, extensive learning time, and sparse reward problems. Although solutions like Hindsight Experience Replay (HER) have been proposed to alleviate these issues, the direct utilization of RL in autonomous vehicles remains constrained due to the intricate fusion of information and the possibility of system failures during the learning process. In this paper, we present a novel RL-based autonomous driving system technology that combines Obstacle Dependent Gaussian (ODG) RL, Soft Actor-Critic (SAC), and meta-learning algorithms. Our approach addresses key issues in RL, including the overestimation phenomenon and sparse reward problems, by incorporating prior knowledge derived from the ODG algorithm. We evaluated our proposed algorithm on official F1 circuits, using high-fidelity racing simulations with complex dynamics. The results demonstrate exceptional performance, with our method achieving up to 89% faster learning speed compared to existing algorithms in these environments.

In the past two decades, metaheuristic optimization algorithms (MOAs) have been increasingly popular, particularly in logistic, science, and engineering problems. The fundamental characteristics of such algorithms are that they are dependent on a parameter or a strategy. Some online and offline strategies are employed in order to obtain optimal configurations of the algorithms. Adaptive operator selection is one of them, and it determines whether or not to update a strategy from the strategy pool during the search process. In the filed of machine learning, Reinforcement Learning (RL) refers to goal-oriented algorithms, which learn from the environment how to achieve a goal. On MOAs, reinforcement learning has been utilised to control the operator selection process. Existing research, however, fails to show that learned information may be transferred from one problem-solving procedure to another. The primary goal of the proposed research is to determine the impact of transfer learning on RL and MOAs. As a test problem, a set union knapsack problem with 30 separate benchmark problem instances is used. The results are statistically compared in depth. The learning process, according to the findings, improved the convergence speed while significantly reducing the CPU time.

Artificial intelligence provides a new research concept for digital image processing. However, at present, artificial intelligence is rarely introduced into the teaching of digital image processing in colleges and universities, and there are problems such as obsolete teaching content, single teaching methods, and simple course experiments, which affect the teaching effect and are not conducive to the cultivation of comprehensive and innovative talents. Digital image processing technology brings more possibilities to communication engineering and makes communication more convenient. For example, video calls and photo transmission make people's communication methods in daily life more and more diversified. Time and space limitations allow people to meet online, creating more communication possibilities. However, there are still many problems and methods worthy of in-depth exploration. Therefore, this paper has a comprehensive understanding and mastery of the traditional and deep learning methods of digital image processing to improve the relevant project practice and scientific research exploration ability and refer for similar research conclusions.

Domestic crime, conflict, and instability pose a significant threat to many contemporary governments. These challenges have proven to be particularly acute within modern-day Mexico. While there have been significant developments in predicting intrastate armed and electoral conflict in various contemporary settings, such efforts have thus far been limited in their use of spatial as well as temporal correlations, as well as in the features they have consid- ered. Machine learning, especially deep learning, has been proven to be highly effective in predicting future conflicts using word embeddings in Convolutional Neural Networks (CNN) but lacks the spatial structure and, due to the black box nature, cannot explain the importance of predictors. We develop a novel methodology using machine learning that can accurately classify future anti-government violence in Mexico. We further demonstrate that our approach can identify important leading predictors of such violence. This can help policymakers make informed decisions and can also help governments and NGOs better allocate security and humanitarian resources, which could prove beneficial in tackling this problem. Using a variety of political event aggregations from the ICEWS database alongside other textual and demographic features, we trained various classical machine learning algorithms, including but not limited to Logistic Regression, Random Forest, XGBoost, and a Voting classifier. We considered a mix of CNN + Long Short Term Memory (LSTM) networks to decode the spatial and temporal relationship in the data. The performance of all the black box deep learning models was not at par with the classical machine learning models. We also evaluated the temporal relationships in the data to identify important predictors that fuel anti-government violence along with their directions using Shapley additive values. The voting classifier, utilizing a subset of features derived from LASSO across 100 simulations, consistently surpasses alternative models in performance and demonstrates efficacy in accu- rately classifying future anti-government conflicts. Notably, Random Forest feature importance indicates that some features, including but not limited to homicides, accidents, material conflicts, and positive worded citizen information sentiments emerge as pivotal predictors in the classification of anti-government conflicts. Finally, we conclude the research by analysing the spatial structure of the data using Moran’s I index extended version for spatiotemporal data to identify the global spatial dependency and local clusters followed by modelling the data spatially and evaluating the same using Gaussian Process Boost(GPBoost). The global spatial autocorrelation is minimal, characterized by local- ized conflicts cluster within the region. Furthermore, in Phase 3, the Voting Classifier method demonstrates superior performance over GPBoost, leading to the inference that no substantial spatial dependency exists among the various locations.

Dynamic adaptive video streaming over HTTP (DASH) plays a crucial role in video transmission across networks. Traditional adaptive bitrate (ABR) algorithms adjust the quality of video segments based on network conditions and buffer occupancy. However, these algorithms rely on fixed rules within a complex environment, making it challenging to achieve optimal decisions considering the overall context. In this paper, we propose a novel Deep Reinforcement Learning-based approach for streaming DASH, focusing on maintaining consistent perceived video quality throughout the streaming session to enhance user experience. Our approach optimizes the Quality of Experience (QoE) by dynamically controlling the quality distance factor between consecutive video segments. We evaluate this approach through a simulation model that encompasses diverse wireless network environments and various video sequences. Additionally, we compare our proposed approach with state-of-the-art methods. The experimental results demonstrate significant improvements in QoE, ensuring users enjoy stable, high-quality video streaming sessions.

The adoption of algorithms based on Artificial Intelligence (AI) has been rapidly increasing during the last years. However, some aspects of AI techniques are under heavy scrutiny. For instance, in many use cases, it is not clear whether the decisions of an algorithm are well-informed and conforming to human understanding. Having ways to address these concerns is crucial in many domains, especially whenever humans and intelligent (physical or virtual) agents must cooperate in a shared environment. In this paper, we introduce an application of an explainability method based on the creation of a Policy Graph (PG) based on discrete predicates that represent and explain a trained agent’s behaviour in a multi-agent cooperative environment. We show that from these policy graphs, policies for surrogate interpretable agents can be automatically generated. These policies can be used to measure the reliability of the explanations enabled by the PGs, through a fair behavioural comparison between the original opaque agent and the surrogate one. The contributions of this paper represent the first application of policy graphs in the context of explaining agent behaviour in collaborative multi-agent scenarios and presents experimental results that sets this kind of scenario apart from previous application in single-agent scenarios: when requiring collaborative behaviour, predicates that allow representing observations about the other agents are crucial to replicate the opaque agent’s behaviour and increase the reliability of explanations.

Scholars and policymakers see sustainability as depending on the interrelationships between the three pillars of the environment, society and the economy. However, it remains less clear how key stakeholders with a responsibility for educating the next generation perceive the concept of sustainability and act accordingly. In order to gain new insights, this research scrutinises participant perceptions on sustainability and climate change in two eco-schools in England for primary children (aged 5-11 years). Our case study involved individual interviews with classroom teachers and headteachers, group interviews with fourth- and fifth-year students (ages 10-12 years), and in-class observations. We also analysed data from student exercise books and photographs of school grounds to understand participants’ self-reported knowledge and perceptions of sustainability and climate change. Within a framework drawing on theories of science communication and transformative learning (a learning approach based on having challenging experiences), the results show that the integration of sustainability into the curriculum was limited and problematic. That is despite the fact that all students and teachers were aware of the environmental dimensions of sustainability, like climate change and the overuse of natural sources. These findings suggest that schools are no different to other institutional settings when it comes to dealing with the challenges of integrating sustainability into daily practice. We conclude that there is a need for in-service teacher education programmes to enable and motivate teachers to provide richer teaching-learning environments so as to enable effective learning in schools about sustainability and climate change.

In recent years, the viability of employing multi-agent reinforcement learning technology for adaptive traffic signal control has been extensively validated. However, owing to restricted communication among agents and the partial observability of the traffic environment, the process of mapping road network states to actions encounters numerous challenges. To address this problem, this paper proposes a multi-agent deep reinforcement learning model with an emphasis on communication content (CMARL). The model decouples the complex relationships between multi-signal agents through centralized training and decentralized execution. Specifically, we first pass the traffic state through an improved deep neural network to achieve the extraction of high-dimensional semantic information and the learning of the communication matrix. Then the agents selectively interact with each other based on the learned communication matrix and generate the final state features. Finally, the features are inputted to the QMIX network to achieve the final action selection. We compare the CMARL model with 6 other baseline algorithms in real traffic networks. The results show that CMARL can significantly reduce vehicle congestion, and run stably in various scenarios.

In this study, a comprehensive review on production scheduling based on reinforcement learning (RL) techniques using bibliometric analysis has been carried out. The aim of this work is, among other things, to point out the growing interest in this domain and to outline the influence of RL as a type of machine learning on production scheduling. To achieve this, the paper explores production scheduling using RL by investigating the descriptive metadata of all pertinent publications contained in the Web of Science database. The study focuses on a wide spectrum of publications spanning the years between 1998 and 2023. The findings and recommendations of this study can serve as new insights for future research endeavors in the realm of production scheduling using RL techniques.

Abstract: As the policies and regulations currently in place concentrate on environmental protection and greenhouse gas reduction, we are steadily witnessing a shift in the transportation industry towards electromobility. There are, though, several issues that need to be addressed to encourage the adoption of EVs at a larger scale. To this end, we propose a solution capable of addressing multiple EV charging scheduling issues, such as congestion management, scheduling a charging station in advance, and allowing EV drivers to plan optimized long trips using their EVs. The smart charging scheduling system we propose considers a variety of factors such as battery charge level, trip distance, nearby charging stations, other appointments, and average speed. Given the scarcity of data sets required to train the Reinforcement Learning algorithms, the novelty of the recommended solution lies in the scenario simulator, which generates the labelled datasets needed to train the algorithm. Based on the generated scenarios, we created and trained a neural network that uses a history of previous situations to identify the optimal charging station and time interval for recharging. The results are promising and for future work we are planning to train the DQN model using real-world data.

Emerging molecular and computational techniques may allow construction and computational treatment of dynamic digital twins on the scale of populations to individuals. This approach may pave the way for predictive, preventive, and personalized treatment. Because digital twins are independent of geographical location, this may also contribute to global equitable health.

The prevalence of differential equations as a mathematical technique has refined the fields of control theory and constrained optimization due to the newfound ability to accurately model chaotic, unbalanced systems. However, in recent research, systems are increasingly more nonlinear and difficult to model using Differential Equations only. Thus, a newer technique is to use policy iteration and Reinforcement Learning, techniques that center around an action and reward sequence for a controller. Reinforcement Learning (RL) can be applied to control theory problems since a system can robustly apply RL in a dynamic environment such as the cartpole system (an inverted pendulum). This solution successfully avoids use of PID or other dynamics optimization systems, in favor of a more robust, reward-based control mechanism. This paper applies RL and Q-Learning to the classic cartpole problem, while also discussing the mathematical background and differential equations which are used to model the aforementioned system.

Background and motivation: Combining Deep Reinforcement Learning (Deep RL) and Health Systems Simulations has significant potential, for both research into improving Deep RL performance and safety, and in operational practice. While individual toolkits exist for Deep RL and Health Systems Simulations, no framework to integrate the two has been established. Aim: Provide a framework for integrating Deep RL Networks with Health System Simulations, and to ensure this framework is compatible with Deep RL agents that have been developed and tested using OpenAI Gym. Methods: We developed our framework based on the OpenAI Gym framework, and demonstrate its use on a simple hospital bed capacity model. We built the Deep RL agents using PyTorch, and the Hospital Simulation using SimPy. Results: We demonstrate example models using a Double Deep Q Network or a Duelling Double Deep Q Network as the Deep RL agent. Conclusion: SimPy may be used to create Health System Simulations that are compatible with agents developed and tested on OpenAI Gym environments. GitHub repository of code: https://github.com/MichaelAllen1966/learninghospital

Seismic vulnerability assessment in urban areas would in principle require the detailed modeling of each single building and the implementation of complex numerical calculations. This procedure is clearly difficult to apply at urban scale where many buildings have to be considered and therefore it is essential to have simplified, but at the same time reliable, approaches to vulnerability assessment. Among the proposed strategies, one of the most interesting concerns the application of machine learning algorithms, able to classify buildings according to their vulnerability on the basis of training procedures applied to existing datasets. In this paper machine learning algorithms will be applied to a dataset which collects and catalogs the structural characteristics of a large number of buildings and reports the damage observed in L’Aquila territory during the intense seismic activity occurred in 2009. A combination of a trained neural network and a random forest algorithm allows to identify an opportune “a-posteriori” vulnerability score, deduced from the observed damage, which is compared to an “a-priori” vulnerability one, evaluated taking into account characteristic indexes for building’s typologies. By means of this comparison an inverse approach to seismic vulnerability assessment, which can be extended to different urban centers, is proposed.

The increasing trend of studying the innate softness of robotic structures and amalgamating it with the benefits of the extensive developments in the field of embodied intelligence has led to sprouting of a relatively new yet extremely rewarding sphere of technology. The fusion of current deep reinforcement algorithms with physical advantages of a soft bio-inspired structure certainly directs us to a fruitful prospect of designing completely self-sufficient agents that are capable of learning from observations collected from their environment to achieve a task they have been assigned. For soft robotics structure possessing countless degrees of freedom, it is often not easy (something not even possible) to formulate mathematical constraints necessary for training a deep reinforcement learning (DRL) agent for the task in hand, hence, we resolve to imitation learning techniques due to ease of manually performing such tasks like manipulation that could be comfortably mimicked by our agent. Deploying current imitation learning algorithms on soft robotic systems have been observed to provide satisfactory results but there are still challenges in doing so. This review article thus posits an overview of various such algorithms along with instances of them being applied to real world scenarios and yielding state-of-the-art results followed by brief descriptions on various pristine branches of DRL research that may be centers of future research in this field of interest.

This last decade, the amount of data exchanged in the Internet increased by over a staggering factor of 100, and is expected to exceed well over the 500 exabytes by 2020. This phenomenon is mainly due to the evolution of high speed broadband Internet and, more specifically, the popularization and wide spread use of smartphones and associated accessible data plans. Although 4G with its long-term evolution (LTE) technology is seen as a mature technology, there is continual improvement to its radio technology and architecture such as in the scope of the LTE Advanced standard, a major enhancement of LTE. But for the long run, the next generation of telecommunication (5G) is considered and is gaining considerable momentum from both industry and researchers. In addition, with the deployment of the Internet of Things (IoT) applications, smart cities, vehicular networks, e-health systems, and Industry 4.0, a new plethora of 5G services has emerged with very diverging and technologically challenging design requirements. These include: high mobile data volume per area, high number of devices connected per area, high data rates, longer battery life for low-power devices, and reduced end-to-end latency. Several technologies are being developed to meet these new requirements. Among these we list ultra-densification, millimeter Wave usage, antennas with massive multiple-input multiple-output (MIMO), antenna beamforming to increase spacial diversity, edge/fog computing, among others. Each of these technologies brings its own design issues and challenges. For instance, ultra-densification and MIMO will increase the complexity to estimate channel condition and traditional channel state information (CSI) estimation techniques are no longer suitable due to the complexity of the new scenarios. As a result, new approaches to evaluate network condition such as by continuously collecting and monitoring key performance indicators become necessary. Timely decisions are needed to ensure the correct operation of such network. In this context, deep learning (DL) models could be seen as one of the main tools that can be used to process monitoring data and automate decisions. As these models are able to extract relevant features from raw data (images, texts, and other types of unstructured data), the integration between 5G and DL looks promising and one that requires exploring. As main contributions, this paper presents a systematic review about how DL is being applied to solve some 5G issues. We examine data from the last decade and the works that addressed diverse 5G problems, such as physical medium state estimation, network traffic prediction, user device location prediction, self network management, among others. We also discuss the main research challenges when using DL models in 5G scenarios and identify several issues that deserve further consideration.

Locomotor impairment is a high-prevalent and significant source of disability and significantly impacts a large population’s quality of life. Despite decades of research in human locomotion, the challenges of simulating human movement to study the features of musculoskeletal drivers and clinical conditions remain. Most recent efforts in utilizing reinforcement learning (RL) techniques are promising to simulate human locomotion and reveal musculoskeletal drives. However, these simulations often failed to mimic natural human locomotion because most reinforcement strategies have yet to consider any reference data regarding human movement. To address these challenges, in this study, we designed a reward function based on the trajectory optimization rewards (TOR), and bio-inspired rewards, which includes the rewards obtained from reference motion data captured by a single Intertial Moment Unit (IMU) sensor. The sensor was equipped on the participants’ pelvis to capture reference motion data. Also, we adapted the reward function by leveraging previous research in walking simulation for TOR. The experimental results showed that the simulated agents with the modified reward function performed better in mimicking the collected IMU data from participants, which means the simulated human locomotion was more realistic. Also, as this bio-inspired defined cost, IMU data enhanced the agent’s capacity to converge during the training process. As a result, the models’ convergence is faster than those developed without reference motion data. Consequently, human locomotion can be simulated more quicker and in a broader range of environments with a better simulation performance.

Learning to maintain postural balance while standing requires a significant fine coordination effort between the neuromuscular system and the sensory system. It is one of the key contributing factors towards fall prevention, especially in the older population. Using artificial intelligence (AI), we can similarly teach an agent to maintain a standing posture, and thus teach the agent not to fall. In this paper, we investigate the learning progress of an AI agent and how it maintains a stable standing posture through reinforcement learning. During training, the AI agent learnt three policies. First, it learnt to maintain the Centre-of-Gravity and Zero-Moment-Point in front of the body. Then, it learnt to shift the load of the entire body on one leg while using the other leg for fine tuning the balancing action. Finally, it started to learn the coordination between the two pre-trained policies. This study shows the potentials of using deep reinforcement learning in human movement studies. The learnt AI behaviour also exhibited attempts to achieve an unplanned goal because it correlated with the set goal (e.g. walking in order to prevent falling). The failed attempts to maintain a standing posture is an interesting by-product which can enrich the fall detection and prevention research efforts.

The popularity of deep reinforcement learning (DRL) methods in economics have been exponentially increased. DRL through a wide range of capabilities from reinforcement learning (RL) and deep learning (DL) for handling sophisticated dynamic business environments offers vast opportunities. DRL is characterized by scalability with the potential to be applied to high-dimensional problems in conjunction with noisy and nonlinear patterns of economic data. In this work, we first consider a brief review of DL, RL, and deep RL methods in diverse applications in economics providing an in-depth insight into the state of the art. Furthermore, the architecture of DRL applied to economic applications is investigated in order to highlight the complexity, robustness, accuracy, performance, computational tasks, risk constraints, and profitability. The survey results indicate that DRL can provide better performance and higher accuracy as compared to the traditional algorithms while facing real economic problems at the presence of risk parameters and the ever-increasing uncertainties.

Sparse reward long horizon task is a major challenge for deep reinforcement learning algorithm. One of the key barriers is data-inefficiency. Even in the simulation environment, it usually takes weeks to training the agent. In this study, a data-efficiency training framework is proposed, where a curriculum learning is design for the agent in the simulation scenario. Different distributions of the initial state are set for the agent to get more informative reward during the whole training process. A fine-tuning of the parameters in the output layer of the neural network for value function is conduct to bridge the gap between sim-to-real. An experiment of UAV maneuver control is conducted in the proposed training framework to verify the method more efficient. We demonstrate that data-efficiency is different for the same data in different training stages.

Recent years have witnessed significant advances in reinforcement learning (RL), which has registered remarkable success in solving various sequential decision-making problems in machine learning. Most of the successful RL applications, e.g., the games of Go and Poker, robotics, and autonomous driving, involve the participation of more than one single agent, which naturally fall into the realm of multi-agent RL (MARL), a domain with a relatively long history, and has recently re-emerged due to advances in single-agent RL techniques. Although, typically, the communication protocol between agents is manually specified and not altered during training, recently, some papers have shown signs of trying to emerge a communication between agents on the one hand and, on the other hand, to understand what is exchanged between agents. So, there is a growing body of literature on this topic which includes qualitative and quantitative studies and the ones that apply mixed methods. This study presents the scoping review of the methodological strategies undertaken in a total of 16 research articles. The results present the critical appraisal of quantitative methods in terms of validity and reliability and for qualitative methods considering four trustworthiness factors. In the end, relevant insights are further explored with implications and reflections on how they can benefit one's research in the field.

Digital Microfluidic Biochips (DMFBs), used in various kinds of fields like DNA analysis, clinical diagnosis, and PCR testing, have made biochemical experiments more compact, efficient, and user-friendly than previous ways. However, their reliability is often compromised by their inability to adapt to all kinds of errors. All errors in biochips can be categorized into two types: known errors and unknown errors. Known errors are detectable before the start of the routing process through sensors or cameras. Unknown errors, in contrast, become apparent only during the routing process and remain undetected by sensors or cameras, which is the biggest issue to unexpectedly stop the routing process and diminishes the reliability of biochips. This paper introduces a deep reinforcement learning-based routing algorithm designed to manage not only known errors but also unknown errors. Our experiments demonstrate that our algorithm outperforms previous ones in terms of the success rate of the routing in the scenario including both known errors and unknown errors. Additionally, our algorithm contributes to detecting unknown errors during the routing process and identifying the most efficient routing path with high probability.

We present a hierarchical reinforcement learning (RL) architecture that employs various low-level agents to act in the trading environment, i.e. the market. The highest level agent selects among a group of specialised agents, and then the selected agent decides when to sell or buy a single asset for some period. This period can be variable according to a termination function. We hypothesized that due to different market regimes, more than one single agent is needed when trying to learn from such heterogeneous data, and instead, multiple agents will perform better, with each one specialising in a subset of the data. We use $k-means$ clustering to partition the data and train each agent with a different cluster. Partitioning the input data also helps model-based RL (MBRL), where models can be heterogeneous. We also add two simple decision-making models to the set of low-level agents, diversifying the pool of available agents and thus increasing overall behaviour flexibility. We perform multiple experiments showing the strengths of a hierarchical approach and test various prediction models at both levels. We also use a risk-based reward at the high level, which transforms the overall problem into a risk-return optimization. This type of reward shows a significant reduction in risk while minimally reducing profits. Overall, the hierarchical approach shows significant promise, especially when the pool of low-level agents is highly diverse.

We propose a 10-Gb/s 64-quadrature amplitude modulation (QAM) signal-based Radio over Fiber (RoF) system for 50 km of standard single mode fiber length which utilizes Reinforcement Learning (RL) SARSA based decision method to indicate an effective decision which mitigates nonlinearity. By utilizing RL-SARSA algorithm, the results demonstrate that significant reduction can be obtained in terms of bit error rate.

Prostate cancer is the second most frequent malignancy (after lung cancer) in men worldwide. Prostate tissue biopsies are usually graded using scores according to the Gleason grading system. This Gleason score is the most popular prognostic marker that reveals the potential aggressiveness of the disease. However, inter-observer variability in rating by human assessors is a major limiting factor. Such variability could lead to missing a severe case or suggesting unnecessary treatments. This study explores the discriminative ability of artificial intelligence (deep learning models) for Gleason score assessment. The study was designed to use whole-slide images of digitized H&E-stained biopsies of prostate tissues to automate the grading process and provide a remotely accessible clinical decision support system. Custom convolutional neural network architectures were trained on 10,616 images of prostate tissues. Gaussian filters were applied to pre-process the images and improve model performance. Transfer learning was applied to train eight machine learning architectures namely: Xception, VGG16 & 19, ResNet101, MobileNet, DenseNet121, EfficientNetB5 & B7. Efficient NetB7 had the best performance 85.2% compared with ground-truth classification by experts. Performance improves as more data is available. The model was deployed and hosted as a web application API on Google cloud service to ensure remote access. Tissue biopsy images can be uploaded and the corresponding Gleason score recovered immediately. This system reduces diagnostics turnaround time, increase throughput, and compensate for limited skills, especially in low resource settings.

This article explores the application of machine learning techniques, specifically focusing on ensemble methods like Random Forests, for detecting fraudulent activities in digital financial transactions. Highlighting the evolution from traditional statistical approaches to modern machine learning models, it underscores the effectiveness of Random Forests in handling the inherent challenges of imbalanced datasets typical in fraud detection scenarios. Using a Kaggle dataset of credit card transactions, the study optimizes Random Forest parameters through rigorous parameter tuning, achieving significant improvements in model performance metrics such as Area Under the Curve (AUC). The findings underscore the critical role of machine learning in enhancing fraud detection capabilities, emphasizing the ongoing evolution and future potential of these methodologies in financial risk management.

Efficient triaging and referral assessment are critical in ensuring prompt medical intervention in the Community Health Care (CHC) system. However, the existing triaging systems in many Community Health Services are an intensive, time-consuming process and often lack accuracy, particularly for various symptoms that might represent heart failure or other health-threatening conditions. There is a noticeable limit of research papers describing AI technologies for triaging patients. This paper proposes a novel quantitative data-driven approach using machine learning (ML) modelling to improve the community clinical triaging process. Furthermore, this study aims to employ the feature selection process and machine learning power to reduce the triaging process’s waiting time and increase accuracy in clinical decision-making. The model was trained on medical records of “Heart Failure” patients’ dataset, which included demographics, past medical history, vital signs, medications, and clinical symptoms. A comparative study was conducted using the list of machine learning algorithms where XGBoost demonstrated the best performance among other ML models. The triage levels of 2,35,982 patients achieved an accuracy of 99.94%, precision of 0.9986, recall of 0.9958, and f1-score of 0.9972 within 0.059 seconds. The proposed diagnostic model can be implemented for the CHC decision system and be developed further for other medical conditions.

For a mobile robot, navigation in a densely crowded space can be a challenging and sometimes impossible task, especially with traditional techniques. In this paper, we present a framework to train neural controllers for differential drive mobile robots which must safely navigate a crowded environment while trying to reach a target location. To learn the robot’s policy, we train a convolutional neural network using two reinforcement learning algorithms, Deep Q-Networks (DQN) and Asynchronous Advantage Actor Critic (A3C), and develop a training pipeline that allows to scale the process to several compute nodes. We show that the asynchronous training procedure in A3C can be leveraged to quickly train neural controllers and test them on a real robot in a crowded environment.

Recently, the application of deep reinforcement learning in recommender system is flourishing and stands out by overcoming drawbacks of traditional methods and achieving high recommendation quality. The dynamics, long-term returns and sparse data issues in recommender system have been effectively solved. But the application of deep reinforcement learning brings problems of interpretability, overfitting, complex reward function design, and user cold start. This paper proposed a tag-aware recommender system based on deep reinforcement learning without complex function design, taking advantage of tags to make up for the interpretability problems existing in recommender system. Our experiment is carried out on MovieLens dataset. The result shows that, DRL based recommender system is superior than traditional algorithms in minimum error and the application of tags has little effect on accuracy when making up for interpretability. In addition, DRL based recommender system has excellent performance on user cold start problems.

We propose a deep reinforcement learning based manipulator path tracking method to solve the computationally difficult and non-unique problem of manipulator path tracking methods based on inverse kinematics. By transforming the path tracking task into a sequence decision problem, our method adopts an end-to-end learning method for closed-loop control and avoids the process of finding the inverse solution. We first explored the feasibility of the deep reinforcement learning method in the path tracking of the manipulator. After verifying the feasibility, the path tracking of the multi-degree-of-freedom(multi-DOF) manipulator was realized by combining the maximum entropy deep reinforcement learning algorithm. The experimental results show that our method has a good effect on the path tracking of the manipulator, which not only avoids the process of finding the inverse kinematics solution, but also requires no dynamic model. Therefore, we believe that our method has great significance in the study of manipulator path tracking.

Modeling is essential to understand better the generative mechanisms responsible for experimental observations gathered from complex systems. In this work, we are using such an approach to analyze the electrocardiogram (ECG). We present a systematic framework to decompose ECG signals into sums of overlapping lognormal components. We used reinforcement learning to train a deep neural network to estimate the modeling parameters from ECG recorded in babies of 1 to 24 months of age. We demonstrate this model-driven approach by showing how the extracted parameters vary with age. After correction for multiple tests, 10 of 24 modeling parameters showed statistical significance below the 0.01 threshold, with absolute Kendall rank correlation coefficients in the [0.27, 0.51] range. We presented a model-driven approach to the analysis of ECG. The impact of this framework on fundamental science and clinical applications is likely to be increased by further refining the modeling of the physiological mechanisms generating the ECG. By improving the physiological interpretability, this approach can provide a window into latent variables important for understanding the heart-beating process and its control by the autonomous nervous system.

The necessity for effective intrusion detection systems (IDS) in big data environments has grown critical due to the rising prevalence of big data systems and the rising amount of security threats. Applying conventional intrusion detection methods to the enormous and intricate data produced by big data ecosystems presents difficulties. Deep learning, has proven to be exceptionally adept at deciphering complex, large-scale data. This study proposes a Deep Learning-based Intrusion Detection System (IDS) created specifically for the Big Data environment. Traditional intrusion detection systems struggle to efficiently identify and prevent cyber threats due to the ever-increasing volume and complexity of data in current networks. We suggest an IDS that makes use of Deep Learning (CNN, LSTM, GAN, etc.), to address this problem. These cutting-edge neural network topologies give the system the ability to process and analyze massive amounts of data for precise intrusion detection. Data collection, data preprocessing, feature engineering, DL model training, intrusion detection, warning generation, and reaction are some of the crucial parts of the suggested IDS. The data collection module mines the Big Data environment for network traffic, system logs, and security event information. To make sure the data is suitable for analysis, it is preprocessed. To improve the detection abilities of the DL models, important features are extracted from the data using feature engineering approaches.

Multimodal remote sensing data classification can enhance the model’s ability to distinguish land features through multimodal data fusion. In this context, how to help models understand the relationship between multimodal data and target tasks is the focus of researchers. Inspired by human feedback learning mechanisms, causal reasoning mechanisms, and knowledge induction mechanisms, this paper integrates causal learning, reinforcement learning, and meta learning into a unified remote sensing data classification framework and proposed the causal meta-reinforcement learning (CMRL). First, based on feedback learning mechanisms, we have overcome the limitations of traditional implicit optimization of fusion features and customized a reinforcement learning environment for multimodal remote sensing data classification tasks. Through feedback interactive learning between agents and the environment, we help them understand the complex relationships between multimodal data and labels, thereby achieving full mining of multimodal complementary information. Second, based on the causal inference mechanism we designed causal distribution prediction action, classification reward, and causal intervention reward, capturing pure causal factors in multimodal data and cutting off false statistical associations between non-causal factors and class labels. Finally, based on the knowledge induction mechanism, we designed a bi-layer optimization mechanism based on meta-learning. By constructing a meta training task and meta validation task simulation model in the generalization scenario of unseen data, we helped the model induce cross-task shared knowledge, thereby improving its generalization ability for unseen multimodal data. The experimental results on multiple sets of multimodal datasets show that the proposed method achieved state-of-the-art performance in multimodal remote sensing data classification tasks.

In August 2020, the World Health Assembly launched a global initiative to eliminate cervical cancer by 2030, setting three primary targets. One key goal is to achieve a 70% screening coverage rate for cervical cancer, primarily relying on the precise analysis of Papanicolaou (Pap) or digital Pap smears. However, the responsibility of reviewing Pap smear samples to identify potentially cancerous cells primarily falls on pathologists, a task known to be exceptionally challenging and time-consuming. This paper proposes a solution to address the shortage of pathologists for cervical cancer screening. It leverages the OpenAI-GYM API to create a deep reinforcement learning environment utilizing liquid-based Pap smear images. By employing the Proximal Policy Optimization algorithm, autonomous agents navigate Pap smear images, identifying cells with the aid of rewards, penalties, and accumulated experiences. Furthermore, the use of a pre-trained convolutional neuronal network like Res-Net54 enhances the classification of detected cells based on their potential for malignancy. The ultimate goal of this study is to develop a highly efficient, automated Papanicolaou analysis system, ultimately reducing the need for human intervention in regions with limited pathologists.

Evolutionary algorithms find applicability in reinforcement learning of neural networks due to their independence from gradient-based methods. To achieve successful training of neural networks using evolutionary algorithms, careful considerations must be made to select appropriate algorithms due to the availability of various algorithmic variations. In Part1 and Part2, the author previously reported experimental evaluations on Genetic Algorithm and Evolution Strategy for reinforcement learning of neural networks, utilizing the Acrobot control task. This article constitutes Part3 of the series of comparative research. In this study, Differential Evolution is adopted as the third instance of major evolutionary algorithms. The experimental results show a statistically significant superiority of DE over both GA and ES (p < .01). In addition, DE exhibits its robustness to variations in hyperparameter configurations (the number of offsprings and generations). In the previous experiments, both ES and GA showed significant performance differences depending on the configurations, whereas in the experiment reported in this article, such differences are not detected for DE.

Evolutionary algorithms find applicability in reinforcement learning of neural networks due to their independence from gradient-based methods. To achieve successful training of neural networks using evolutionary algorithms, careful considerations must be made to select appropriate algorithms due to the availability of various algorithmic variations. The author previously reported experimental evaluations on Evolution Strategy for reinforcement learning of neural networks, utilizing the Acrobot control task. In this study, Genetic Algorithm is adopted as another instance of major evolutionary algorithms. Experimental results demonstrate that there was no statistically significant difference between the experimental performances of GA and ES, but the priority of generations and offsprings was different; GA performed better with a greater number of generations while ES performed better with a greater number of offsprings. Eight hidden units were the best among four variations (4, 8, 16 or 32 units), which aligns with previous study using ES.

Evolutionary algorithms find applicability in the reinforcement learning of neural networks due to their independence from gradient-based methods. To achieve successful training of neural networks using evolutionary algorithms, careful considerations must be made to select appropriate algorithms due to the availability of various algorithmic variations. The author previously reported experimental evaluations on Evolution Strategy for reinforcement learning of neural networks, utilizing the pendulum control task. In this study, the Acrobot control task is adopted as another task. Experimental results demonstrate that ES successfully trained a Multi-Layer Perceptron to achieve a remarkable height of 99.85% concerning the maximum height. However, the trained MLP failed to maintain the chain end in an upright position throughout an episode. In this study, it was observed that employing 8 hidden units in the neural network yielded better results with statistical significance compared to using 4, 16, or 32 hidden units. Furthermore, the findings indicate that a larger population size in ES led to a more extensive exploration of potential solutions over a greater number of generations, which aligns with the previous study.

Evolutionary algorithms and swarm intelligence algorithms find applicability in reinforcement learning of neural networks due to their independence from gradient-based methods. To achieve successful training of neural networks using these algorithms, careful considerations must be made to select appropriate algorithms due to the availability of various algorithmic variations. In Part1, the author previously reported experimental evaluations on Evolution Strategy　for reinforcement learning of binary neural networks, utilizing the Pendulum control task. This article constitutes Part2 of the series of comparative research. In this study, Genetic Algorithm is adopted as another evolutionary algorithm. Wilcoxon signed rank test revealed that there was no statistically significant difference between the fitness scores obtained using GA and those using ES. However, the p-value 0.11 indicated that GA worked better than ES on this training task. As the values of binary weights, {-1, 1} were significantly superior to {0, 1} (p < .01). The motion of the pendulum controlled by the binary MLP after the training showed that the binary MLP successfully swung the pendulum swiftly into an inverted position and maintained its stability after inversion.

Stagnation at local optima represents a significant challenge in bio–inspired optimization algorithms, often leading to suboptimal solutions. This paper addresses this issue by proposing a hybrid model that combines the Orca Predator Algorithm with Deep Q–Learning. Orca Predator Algorithm is an optimization technique that mimics the hunting behavior of orcas. It solves complex optimization problems by exploring and exploiting search spaces efficiently. Deep Q–Learning is a reinforcement learning technique that combines Q–Learning with deep neural networks. This integration aims to turn the stagnation problem into an opportunity for more focused and effective exploitation, enhancing the optimization technique’s performance and accuracy. The proposed hybrid model leverages the biomimetic strengths of Orca Predator Algorithm to identify promising regions nearby in the search space, complemented by the fine–tuning capabilities of Deep Q–Learning to navigate these areas precisely. The practical application of this approach is evaluated using the high–dimensional Heartbeat Categorization Dataset, focusing on the feature selection problem. This dataset, comprising complex electrocardiogram signals, provided a robust platform for testing the feature selection capabilities of our hybrid model. Our experimental results are encouraging, showcasing the hybrid strategy capability to identify relevant features without significantly compromising the performance metrics of machine learning models. This analysis was performed by comparing the improved method of Orca Predator Algorithm against its native version and a set of state–of–the–art algorithms.

This paper is primarily focused on the robust control of an inverted pendulum system based on the policy iteration in reinforcement learning. First, a mathematical model of the single inverted pendulum system is established through a force analysis of the pendulum and trolley. Second, based on the theory of robust optimal control, the robust control of the uncertain linear inverted pendulum system is transformed into an optimal control problem with an appropriate performance index. Moreover, for the uncertain linear and nonlinear systems, two reinforcement-learning control algorithms are proposed using the policy iteration method. Finally, two numerical examples are provided to validate the reinforcement learning algorithms for the robust control of the inverted pendulum systems.

Fast convergence routing is an important issue for LEO constellation network, due to its dynamical topology changing and time varying transmission requests. Most of existing research focus on the OSPF routing algorithm, which cannot handle the frequently links state changing of network. In this paper, we propose a Fast-Convergence Reinforcement Learning Satellite Routing Algorithm (FRL-SR) for LEO satellite networks, in which the satellite gets the network links status fast and adjusts its routing strategy. In FRL-SR, each satellite node is regarded as an agent. The agent selects the port for packet forwarding according to its own routing policy. When the satellite network state changes, agent would send ’hello’ packets to the neighbor node to update the neighbor node’s routing policy. Compared with traditional reinforcement learning, FRL-SR can perceive network information faster, and then converge faster. Also, FRL-SR can mask the dynamics of satellite network topology and adaptively adjust the forwarding strategy according to the link state. Various simulation is constructed, the results show that the proposed FRL-SR algorithm out performance the Dijkstra algorithm in performance of average delay, packet arriving ratio, network load balance.

We consider the problem of coordinating the charging of an entire fleet of electric vehicles (EV), using a model-free approach, i.e. purely data-driven reinforcement learning (RL). The objective of the RL-based control is to optimize charging actions, while fulfilling all EV charging constraints (e.g. timely completion of the charging). In particular, we focus on batch-mode learning and adopt fitted Q-iteration (FQI). A core component in FQI is approximating the Q-function using a regression technique, from which the policy is derived. Recently, a dueling neural networks architecture was proposed and shown to lead to better policy evaluation in the presence of many similar-valued actions, as applied in a computer game context. The main research contributions of the current paper are that (i)we develop a dueling neural networks approach for the setting of joint coordination of an entire EV fleet, and (ii)we evaluate its performance and compare it to an all-knowing benchmark and an FQI approach using EXTRA trees regression technique, a popular approach currently discussed in EV related works. We present a case study where RL agents are trained with an epsilon-greedy approach for different objectives, (a)cost minimization, and (b)maximization of self-consumption of local renewable energy sources. Our results indicate that RL agents achieve significant cost reductions (70--80%) compared to a business-as-usual scenario without smart charging. Comparing the dueling neural networks regression to EXTRA trees indicates that for our case study's EV fleet parameters and training scenario, the EXTRA trees-based agents achieve higher performance in terms of both lower costs (or higher self-consumption) and stronger robustness, i.e. less variation among trained agents. This suggests that adopting dueling neural networks in this EV setting is not particularly beneficial as opposed to the Atari game context from where this idea originated.

The integration of deep learning models into financial risk prediction and analysis has significantly transformed traditional approaches. While conventional quantitative methods often rely on simplistic metrics like maximum drawdown, the advent of deep learning necessitates a more nuanced evaluation, emphasizing the model's generalization ability, especially during market crises such as stock market crashes. This paper explores the critical aspects of evaluating deep learning models' risk control capabilities in finance, underscoring the importance of understanding both statistical metrics and generalization abilities, particularly in adverse market conditions. By examining deep learning models' performance in scenarios like stock market crashes and highlighting the significance of cross-validation techniques, this study aims to offer practitioners insights into constructing robust risk management systems. It advocates for a comprehensive approach integrating quantitative analysis with macroeconomic factors to enhance financial risk prediction and analysis in volatile markets. The experimentation reveals that different deep generative models excel in various aspects of financial time series analysis, with generative adversarial networks (GANs) demonstrating superior performance in predicting Value at Risk (VaR) and variational autoencoders (VAEs) excelling in return rate prediction. Moreover, integrating multiple models further enhances predictive performance, leveraging the strengths of each model to compensate for individual weaknesses. Overall, this paper underscores the potential and significance of deep generative models in financial time series analysis, offering a roadmap for improved risk management and decision-making in financial markets.

This study discusses the application of deep learning technology in network intrusion detection systems (IDS) and focuses on a new model named CNN-Focal. First, through the review of traditional IDS technology, it analyzes its limitations in dealing with complex network traffic. Then, the design principle of the CNN-Focal model is described in detail, which uses threshold convolution and SoftMax multi-class classification technology to effectively improve abnormal traffic detection's accuracy and efficiency. The experimental results show that CNN-Focal performs well on the open data set, demonstrating the potential and advantages of its application in the natural network environment and providing a new perspective and method for further research of deep learning in the field of network security in the future.

Goal-conditioned reinforcement learning (RL) holds promise for addressing intricate control challenges by enabling agents to learn and execute desired skills through separate decision modules. However, the irregular occurrence of required skills poses a significant challenge to effective learning. In this paper, we demonstrate the detrimental effects of this imbalanced skill(sub-goal) distribution and propose a novel training approach, Classified Experience Replay (CER), designed to mitigate this challenge. We demonstrate that adapting our method to conventional RL methods significantly enhances the performance of the RL agent. Considering the challenges inherent in tasks such as driving, characterized by biased occurrences of required sub-goals, our study demonstrates the improvement in trained outcomes facilitated by the proposed method. In addition, we introduce a specialized framework tailored for self-driving tasks on highways, integrating Model Predictive Control (MPC) into our RL trajectory optimization training paradigm. Our approach, utilizing CER with the suggested framework, yields remarkable advancements in trajectory optimization for RL agents operating in highway environments.

There is only a very short reaction time for people to find the best way out of a building in a fire outbreak. Software applications can be used to assist the rapid evacuation of victims; however, this is an arduous task, which requires an understanding of advanced technologies. Since well-known pathway algorithms (such as, Dijkstra, Bellman-Ford and A*) can lead to serious problems over performance, we decided to make use of deep reinforcement learning techniques. A wide range of strategies including a random initialization of replay buffer and transfer learning were assessed in three projects involving schools of different sizes. The results showed the proposal was viable and that in most cases the performance of transfer learning was superior. In addition, the study raised challenges that had to be faced in the future.

The robust control of high precision electromechanical systems, such as micropositioners, is challenging in terms of the inherent high nonlinearity, the sensitivity to external interference, and the complexity of accurate identification of the model parameters. To cope with these problems, this work investigates a disturbance observer-based deep reinforcement learning control strategy to realize high robustness and precise tracking performance. Reinforcement learning has shown great potential as optimal control scheme, however, its application in micropositioning systems is still rare. Therefore, embedded with the integral differential compensator (ID), deep deterministic policy gradient (DDPG) is utilized in this work with the ability to not only decrease the state error but also improves the transient response speed. In addition, an adaptive sliding mode disturbance observer (ASMDO) is proposed to further eliminate the collective effect caused by the lumped disturbances. The sterling performance is revealed with intensive tracking simulation experiments and demonstrates the improvement in the accuracy and response time of the controller.

As a new distributed machine learning (ML) approach, federated learning (FL) shows the great potential to preserve data privacy by enabling distributed data owners to collaboratively build a global model without sharing their raw data. However, the heterogeneity in terms of data distribution and hardware configurations make it hard to select participants from the thousands of nodes. In this paper, we propose a multi-objective node selection approach to improve time-to-accuracy performance while resisting malicious nodes. We firstly design a deep reinforcement learning assisted FL framework. Then the problem of multi-objective node selection under this framework is formulated as a Markov decision process (MDP), which aims to reduce the training time and improve model accuracy simultaneously. Finally, a deep Q-netwok (DQN) based algorithm is proposed to efficiently solve the optimal set of participants for each iteration. Simulation results show that the proposed method not only significantly improves the accuracy and training speed of FL, but has stronger robustness to resist malicious nodes.

The IEEE 802.11ah standard is introduced to address the growing scale of Internet of Things (IoT) applications. To reduce contention and enhance energy efficiency within the system, the restricted access window (RAW) mechanism is introduced at the medium access control (MAC) layer to manage the significant number of stations accessing the network. However, the RAW parameters need to be appropriately determined to achieve optimal network performance. In this paper, we optimize the configuration of RAW parameters in the IEEE 802.11ah-based IoT network. We analyze and propose an RAW parameters optimization problem with the objective of improving network throughput and formulate it as a Markov decision process. To cope with the complex and dynamic network conditions, we propose a proximal policy optimization (PPO)-based deep reinforcement learning (DRL) algorithm to find the optimal RAW parameters. We construct network environments with periodic and random traffic in the NS-3 simulator to validate the performance of the proposed PPO-based RAW parameters optimization algorithm. Simulation results reveal that using the PPO-based DRL algorithm, optimal RAW parameters can be obtained under different network conditions, thereby significantly improving network throughput.

In this review we summarize existing trends of flow control used to improve the aerodynamic efficiency of wings. We first discuss active methods to control turbulence, starting with flat-plate geometries and building towards the more complicated flow around wings. Then, we discuss active approaches to control separation, a crucial aspect towards achieving high aerodynamic efficiency. Furthermore, we highlight methods relying on turbulence simulation, and discuss various levels of modelling. Finally, we thoroughly revise data-driven methods, their application to flow control, and focus on deep reinforcement learning (DRL). We conclude that this methodology has the potential to discover novel control strategies in complex turbulent flows of aerodynamic relevance.

In this article, we propose a machine learning based fine-grained vehicle classification method VehiClassNet, which effectively solves the challenges of precise vehicle recognition and classification in intelligent transportation systems by combining multi-scale feature extraction, cross modal fusion, and attention mechanisms. Our model demonstrated excellent performance on the Stanford Cars-196 dataset. The innovation of this study lies in its advanced multi-scale feature extraction, cross modal fusion, and attention mechanism. It not only improves classification accuracy, but also lays the foundation for further research on intelligent transportation systems and autonomous vehicle technology.

Abstract—Online learning platforms provide diverse course resources, but this often result in the issue of information overload. Learners always want to learn courses that are appropriate for their knowledge level and preferences quickly and accurately. Effective course recommendation plays a key role in helping learners select appropriate courses and improving the efficiency of online learning. However, when a user is enrolled in multiple courses, Existing course recommendation methods face the challenge in accurately recommending the target course that is most relevant to the user, because of the noise courses. In this paper, we propose a novel reinforcement learning model named Actor-Critic Hierarchical Reinforcement Learning (ACHRL). The model incorporates the Actor-Critic method to construct the profile reviser. This can remove noise courses and make personalized course recommendation effectively. Furthermore, we propose a policy gradient based on temporal difference error to reduce the variance in the training process, to speed up the convergence of the model, and improves the accuracy of the recommendation. We evaluate the proposed model on two real datasets, and the experimental results show that the proposed model is significantly outperforms the existing recommendation models (improving 3.77% to 13.66% in terms of HR@5).

In this paper we present the exploration of indoor positioning technologies for UAVs, and navigation techniques for path planning and obstacle avoidance. The objective is to perform warehouse inventory tasks using a drone to search for barcodes or markers to identify the objects. For the indoor positioning techniques, we employ Visual-Inertial Odometry (VIO), Ultra-Wideband (UWB), AprilTag fiducial markers, and Simultaneous Localization and Mapping (SLAM). These algorithms encompass global positioning, local positioning, and pre-mapping positioning, comparing the merits and drawbacks of various techniques and trajectories. In UAV navigation, we combine SLAM-based RTAB-Map indoor mapping and navigation path planning of the ROS for indoor environments. This system enables precise drone positioning indoors and utilizes global and local path planners to generate flight paths that avoid dynamic, static, unknown, and known obstacles, demonstrating high practicality and feasibility. To achieve the warehouse inventory inspection, a reinforcement learning approach is proposed to recognize the markers by adjusting the UAV's viewpoint. We address several main problems in inventory management, including efficiently planning paths while ensuring a certain level of detection rate. Two reinforcement learning techniques, AC (Actor-Critic) and PPO (Proximal Policy Optimization), are implemented based on AprilTag identification. Testing is performed in both simulated and real-world environments, and the effectiveness of the proposed method is validated.Code is available at https://github.com/kellen080/Navigation and https://github.com/kellen080/Indoor\_Positioning.

Evolutionary algorithms and swarm intelligence algorithms find applicability in reinforcement learning of neural networks due to their independence from gradient-based methods. To achieve successful training of neural networks using these algorithms, careful considerations must be made to select appropriate algorithms due to the availability of various algorithmic variations. In Part1, 2 and 3, the author previously reported experimental evaluations on Evolution Strategy, Genetic Algorithm, and Differential Evolution for reinforcement learning of neural networks, utilizing the Acrobot control task. This article constitutes Part4 of the series of comparative research. In this study, Particle Swarm Optimization is adopted as an instance of major swarm intelligence algorithms. The experimental result shows that PSO performed worse than all of DE, GA and ES. The difference between PSO and DE was statistically significant (p<.01). In addition, PSO exhibited lower capability in exploring solutions in high-dimensional search spaces than DE, GA, and ES did. A larger swarm size compensated for the weakness of PSO in global exploration, thus making itself more beneficial than a larger number of swarm search iterations.

The author previously reported an experimental result of evolutionary reinforcement learning of neural network controllers for the pendulum task. In the previous work, a conventional multilayer perceptron was employed in which connection weights were real numbers. In this study, the author experimentally applies an evolutionary algorithm to the reinforcement training of binary neural networks. In both studies, the same task and the same evolutionary algorithm are utilized, i.e. the pendulum control problem and Evolution Strategy respectively. The only differences lie in the memory size per connection weight and the model size of the neural network. The findings from this study are (1) the performance of the binary MLP with 32 hidden units was inferior to that of the real-valued MLP with 16 hidden units; however, this difference was not statistically significant (p >.05); (2) the trained binary MLP successfully swung the pendulum swiftly into an inverted position and maintained its stability after inversion, as the real-valued MLP had done; and (3) the memory size required to record the binary MLP with 32 hidden units is 3.1% or 6.2% of the memory size required to record the real-valued MLP with 16 hidden units.

Modern power system integrates more and more new energy and use a large number of power electronic equipment. This makes it face more challenges in online optimization and real-time control. Deep reinforcement learning(DRL) has the ability of processing big data and high-dimensional features, as well as the ability of independently learning and optimizing decision-making in complex environments. In this paper, we explore DRL based online combination optimization method of grid section for large complex power system. In our method, to improve the convergence speed of the model, we propose to discretize the output action of the unit and simplify the action space. We also design a reinforcement learning loss function with strong constraints to further improve the convergence speed of the model and facilitate the algorithm to obtain the stable solution. Moreover, to avoid the local optimal solution problem caused by the discretization of the output action, we propose to use the annealing optimization algorithm to make the granularity of the unit output finer. We verify our method on IEEE 118-bus system. The experimental results show that our model has fast convergence speed and better performance, and can obtain stable solutions.

The brain being arguably the most complex organ in the human body, its detailed functioning is yet to be fully deciphered, let alone accurately modeled. Nonetheless, the medical community has gathered quite a remarkable amount of studies about the consequences of illnesses and injuries on its development and functioning. This bibliographic review aims to cover mostly the findings related to the changes in the physical distribution of neurons and their connections - the connectome -, both structural and functional, as well as their modelling. It does not intend to offer an extensive medical description of all injuries and diseases affecting the brain, rather presenting the most common ones succinctly so the need for accurate brain modelling can be fully understood and pondered, offering propositions to this aim.

The use of drones for various applications has become increasingly popular in recent years, and machine learning has played a significant role in this trend. In this paper, we provide a comprehensive survey of the classification and application of machine learning in drones. The paper begins with an overview of the different types of machine learning algorithms and their applications in drones, including supervised learning, unsupervised learning, and reinforcement learning. Next, we present a detailed analysis of various real-world applications of machine learning in drones, such as object recognition, route planning, obstacle avoidance, search area optimization, and autonomous search. The paper also discusses the challenges and limitations of using machine learning in drones, such as data privacy, data quality, and computational requirements. Finally, the paper concludes with a discussion of the future directions of machine learning in drones and its potential impact on various industries and fields. This paper provides a valuable resource for researchers, practitioners, and students interested in the intersection of machine learning and drones.

Abstract This study focuses on the toxicity assessment of various flavonoids, with particular emphasis on Hesperidin. Among the investigated compounds, Hesperidin exhibits notable potential and low toxicity in both short and long-term evaluations. The toxicity parameters for Hesperidin include a low Max. Tolerated Dose (Human) (0.525 log mg/kg/day), indicative of its minimal potential harm to humans. Additionally, its Oral Rat Acute Toxicity (LD50) (2.506 mol/kg) suggests a relatively low acute toxicity level. The Oral Rat Chronic Toxicity (LOAEL) (3.167 log mg/kg_bw/day) signifies a higher dose threshold for observing adverse effects over the long term. Furthermore, Hesperidin demonstrates low Minnow Toxicity (7.131 log mM), suggesting minimal impact on aquatic organisms. Overall, this comprehensive assessment positions Hesperidin as a promising flavonoid with favorable properties and minimal toxicity.

Continuous greenhouse gas emissions are causing global warming and impacting the habitats of many animals. Researchers in the field of electric power are making efforts to mitigate this situation. Operating and maintaining the power grid in an economic, low-carbon, and stable is challenging. To address the issue, we propose a grid dispatching technique that combines prediction technology, reinforcement learning, and optimization technology. Prediction technology can forecast future power demand and solar power generation, while reinforcement learning and optimization technology can make charging and discharging decisions for energy storage devices based on current and future grid conditions. In the power system, the aggregation of distributed energy resources increases uncertainty, particularly due to the fluctuating generation of renewable energy. This requires the use of advanced predictive control techniques to ensure long-term economic and decarbonization goals. In this paper, we present a real-time dispatching framework that integrates deep learning-based prediction, reinforcement learning-based decision-making, and stochastic optimization techniques. The framework can rapidly adapt to target uncertainty caused by various factors in real-time data distribution and control processes. The proposed framework achieved global Champion in the NeurIPS Challenge 2022 competition and demonstrated its effectiveness in practical scenarios of intelligent building energy management.

It is well-known that the canonical quantization of $\varphi^4_4$ fails. Here is how to fix it along with many other problems.

The increasing popularity of carbon-reinforced concrete (CRC) is attributed to its exceptional tensile properties, low density, no corrosion phenomenon, and remarkable flexibility, allowing it to be easily shaped into various forms. This research investigates the feasibility of using a special 2D Netzgitterträger (NetzGT) reinforcement system, featuring a net-shaped fabricated textile made of multiple diagonally offset rovings with overlapping edge strands, as a viable alternative to traditional steel reinforcement in concrete beams. This reinforcement is manufactured from carbon rovings with three different diagonal angles of 50⁰, 60⁰, and 70⁰ respectively. Laboratory experiments were conducted to assess the mechanical behavior of beams reinforced with the 2D NetzGT reinforcement. Bending and shear tests were performed on beams with varying numbers of overlapped edge roving and roving angles to evaluate the tensile capacity and failure characteristics of beams. The increase in the number of overlapped edge rovings led to a noticeable increase in the maximum tensile force. Tensile tests on strands were also performed with the increasing number of overlapped rovings to analyze their tensile strength. Additionally, single yarn pull-out tests were also conducted to examine the influence of the roving angle on the bond strength between the carbon textile roving and the concrete matrix

The modern supply chain (SC) is growing in terms of data, devices, users, and stakeholders, which introduced new security challenges and threats, especially with the reliance on centralized servers or cloud platforms. In addition, increased trust among system participants exposes the SC to a higher risk of vulnerabilities which require strong security measures. This article proposes a hybrid security framework for SC systems, BC-DRLzSC, that integrates Blockchain (BC) and Deep Reinforcement Learning (DRL) designed to operate in a zero trust (ZT) environment. In particular, we propose a decentralized BC-based approach integrated with smart contracts to manage system participant registration and authentication and to control access to system resources. BC-DRLzSC adopts a ZT architecture to reinforce SC security, which can be achieved with an advocate to verify each entity’s trustworthiness before granting or retaining access to system resources. Incorporating the ZT architecture, with BC and DRL, can potentially and significantly bolster SC system security. DRL is employed to develop a proactive attack detection model that continuously monitors the incoming traffic from authenticated nodes within the network and predicts any malicious actions. Finally, we evaluate the performance of our proposed DRL solution using the NSL-KDD dataset.

This paper presents a novel approach to designing beam codebooks for downlink multiuser hybrid multiple input multiple output (MIMO) wireless communication systems, leveraging multi-agent reinforcement learning (MARL). The primary objective is to develop an environment-specific beam codebook composed of non-interfering beams, learned by cooperative agents within the MARL framework. Machine learning (ML) based beam codebook design for downlink communications have been based on channel state information (CSI) feedback or only reference signal received power (RSRP) consisting of an offline training and user clustering phase. In massive MIMO, the full CSI feedback data is of large size and is resource-intensive to process, making it challenging to implement efficiently. RSRP alone for a stand-alone base station is not a good marker of the position of a receiver. Hence, in this work, uplink CSI estimated at the base station along with feedback of RSRP and binary acknowledgment of the accuracy of received data is utilized to design the beamforming codebook at the base station. Simulations using sub-array antenna and ray-tracing channel demonstrate the proposed system’s ability to learn topography-aware beam codebook for arbitrary beams serving multiple user groups simultaneously. The proposed method extends beyond mono-lobe and fixed beam architectures by dynamically adapting arbitrary shaped beams to avoid inter-beam interference, enhancing overall system performance. This work leverages MARL’s potential in creating efficient beam codebooks for hybrid MIMO systems, paving the way for enhanced multiuser communication in future wireless networks.

Abstract:The objects of fine-grained image categories(e.g., bird species) are various subclass under different categories. Because the differences between subclass are very subtle and most of them are concentrated in multiple local areas, the task of fine-grained image recognition is very challenging. At the same time, some fine-grained networks tend to focus on a certain region when judging the target category, resulting in the lack of other auxiliary regional features. To this end, Inception V3 is used as the backbone network, and an enhanced and complementary fine-grained image classification network is designed. While adopting the method of reinforcement learning to obtain more detailed fine grain image features, the complementary network can obtain the complementary discriminant area of the target through the method of attention erasure to increase the network's perception of the overall target. Finally, experiments are conducted on CUB-200-2011, FGVC Aircraft and Stanford dogs three open datasets. The experimental results show that the proposed model has better performance.

Electromobility has gained significant attention in the auto- motive industry, with car manufacturers striving to improve the charging experience for their users. In this study, we compare multiple realistic charging strategies in order to quantify the benefits arising from better planning of charging sessions. We base our evaluation on electricity price data from three regions: Sweden (SE3 region), Germany (DE region), and Denmark (DK2 region). This data is obtained from the ENTSO-E Transparency Platform, covering the two full years 2021 and 2022. We enhance this dataset with commercial price forecasts from the company StormGeo and benchmark it against a naive forecasting method. The primary aim of our work is to compare how the strategies of varying levels of complexity perform across the three different countries in terms of minimizing energy costs for electric car users.

In this work, several random Boolean networks (RBN) are generated and analyzed from two characteristics: their time evolution diagram and their transition diagram. To do this, its randomness is estimated using three metrics, of which algorithmic complexity is the only one capable of revealing both transitions towards the chaotic regime and the algorithmic contribution of certain states to the transition diagram and their relationship with the order they occupy in the temporal evolution of the respective RBN.

While researchers have made notable progress in bio-inspired swimming robot development, a persistent challenge lies in creating propulsive gaits tailored to these robotic systems. The California sea lion achieves its robust swimming abilities through a careful coordination of foreflippers and body segments. In this paper, reinforcement learning (RL) was used to develop a novel sea lion foreflipper gait for a bio-robotic swimmer using a numerically modelled computational representation of the robot. This model integration enabled reinforcement learning to develop desired swimming gaits in the challenging underwater domain. The novel RL gait outperformed the characteristic sea lion foreflipper gait in the simulated underwater domain. When applied to the real-world robot, the RL constructed novel gait performed as well or better than the characteristic sea lion gait in many factors. This work shows the potential for using complimentary bio-robotic and numerical models with reinforcement learning to enable the development of effective gaits and maneuvers for underwater swimming vehicles.

This study assesses the impact of incorporating an adaptive learning mechanism into an agent-based model (ABMS) simulating behavior on a university campus during the course of a pandemic outbreak, with a particular case on the COVID-19 pandemic. The aim is to reduce overcrowding and infections on campus through the use of Reinforcement Learning (RL). Our findings indicate that RL is a viable approach for effectively representing agents’ behavior within this context. The results reveal specific temporal patterns of overcrowding violations. While our study successfully mitigated campus crowding, it had limited influence on altering the course of the epidemic. This highlights the necessity for comprehensive epidemic control strategies that consider the role of individual decision-making influenced by adaptive learning, along with the implementation of targeted interventions. This research significantly contributes to our understanding of adaptive learning within complex systems and offers valuable insights for shaping future public health policies in similar community settings. Future research directions encompass exploring various parameter settings and updating representations of the disease’s natural history.

The application of space robotic manipulators and heightened autonomy for ios represents a paramount pursuit for leading space agencies, given the substantial threat posed by space debris to operational satellites and forthcoming space endeavors. This work presents a guidance algorithm based on drl to solve for the space manipulator path-planning during the motion synchronization phase with the mission target. The goal is the trajectory generation and control of a spacecraft equipped with a 7-dof robotic manipulator, such that its end effector remains stationary with respect to the target point of capture. The ppo drl algorithm is used to optimize the manipulator’s guidance law, and the autonomous agent generates the desired joints rates of the robotic arm, which are then integrated and passed to a model-based feedback linearization controller. The agent is first trained to optimize its guidance policy and then tested extensively to validate the results against a simulated environment representing the motion synchronization scenario of an ios mission.

In this study, reinforcement learning (RL) was used in factory simulation to optimize storage devices for use in Industry 4.0 and digital twins. First, we defined an RL environment, modeled it, and validated its ability to simulate a real physical system. Subsequently, we introduced a method to calculate reward signals and apply them to the environment to ensure the alignment of the behavior of the RL agent with the task objective. The stocker simulation model was used to validate the effectiveness of RL. The model is a storage device that simulates logistics in a manufacturing production area. The results revealed that RL is a useful tool for automating and optimizing complex logistics systems and increase the applicability of RL in logistics. We proposed a novel method for creating an agent through learning using the proximal policy optimization algorithm, and the agent was optimized by configuring various learning options. The application of reinforcement learning resulted in an effectiveness of 30% to 100%, and methods can be expanded to other fields.

Online reinforcement learning (RL) algorithms are increasingly used to personalize digital interventions in the fields of mobile health and online education. Common challenges in designing and testing an RL algorithm in these settings include ensuring the RL algorithm can learn and run stably under real-time constraints, and accounting for the complexity of the environment, e.g., a lack of accurate mechanistic models for the user dynamics. To guide how one can tackle these challenges, we extend the PCS (Predictability, Computability, Stability) framework, a data science framework that incorporates best practices from machine learning and statistics in supervised learning (Yu and Kumbier, 2020), to the design of RL algorithms for the digital interventions setting. Further, we provide guidelines on how to design simulation environments, a crucial tool for evaluating RL candidate algorithms using the PCS framework. We illustrate the use of the PCS framework for designing an RL algorithm for Oralytics, a mobile health study aiming to improve users' tooth-brushing behaviors through the personalized delivery of intervention messages. Oralytics will go into the field in late 2022.

A detailed literature review is performed in this study to address solutions for the full-body design and control of an aerial manipulator. Deep Reinforcement Learning methods are growing to be utilized recently to cope with various uncertainties. The pros and cons of these theories will be explained as well as introducing the advantages of Fuzzy Reinforcement Learning methods. State-of-the-Art, possible challenges, potential approaches, and a summary of desired precision devices are discussed in this study.

The steep rise in reinforcement learning (RL) in various applications in energy as well as the penetration of home automation in recent years are the motivation for this article. It surveys the use of RL in various home energy management system (HEMS) applications. There is a focus on deep neural network (DNN) models in RL. The article provides an overview of reinforcement learning. This is followed with discussions on state-of-the-art methods for value, policy, and actor–critic methods in deep reinforcement learning (DRL). In order to make the published literature in reinforcement learning more accessible to the HEMS community, verbal descriptions are accompanied with explanatory figures as well as mathematical expressions using standard machine learning terminology. Next, a detailed survey of how reinforcement learning is used in different HEMS domains is described. The survey also considers what kind of reinforcement learning algorithms are used in each HEMS application. It suggests that research in this direction is still in its infancy. Lastly, the article proposes four performance metrics to evaluate RL methods.

This paper presents an innovative multi-agent, data-driven reinforcement learning (RL) approach to develop and utilize the thermal equivalent network model that represents the motor's thermal dynamics. A multi-agent reinforcement learning is designed and trained to adjust the model parameters using data from several motor driving cycles. To ensure the incoming driving cycle matches the historical data before employing the pre-trained RL agents, offline statistical analysis and clustering techniques are developed and used. Numerical simulations highlighted the RL agent's ability to develop strategies that effectively address the variability of driving cycles. The proposed RL framework showed its capability to accurately reflect the motor's thermal behavior under various driving conditions.

Climate change and development have increased urban flooding, requiring modernization of stormwater infrastructure. Retrofitting standard passive systems with controllable valves/pumps is promising, but requires real-time control (RTC). One method of automating RTC is reinforcement learning (RL), a general technique for sequential optimization and control in uncertain environments. The notion is that an RL algorithm can use inputs of real-time flood data and rainfall forecasts to learn a policy for controlling the stormwater infrastructure to minimize measures of flooding. In real-world conditions, rainfall forecasts and other state information, are subject to noise and uncertainty. To account for these characteristics of the problem data, we implemented Deep Deterministic Policy Gradient (DDPG), an RL algorithm that is distinguished by its capability to handle noise in the input data. DDPG implementations were trained and tested against a passive flood control policy. Three primary cases were studied: (i) perfect data, (ii) imperfect rainfall forecasts, and (iii) imperfect water level and forecast data. Rainfall episodes (100) that caused flooding in the passive system were selected from 10 years of observations in Norfolk, Virginia, USA; 85 randomly selected episodes were used for training and the remaining 15 unseen episodes served as test cases. Compared to the passive system, all RL implementations reduced flooding volume by 70.5% on average, and performed within a range of 5%. This suggests that DDPG is robust to noisy input data, which is essential knowledge to advance the real-world applicability of RL for stormwater RTC.

This paper presents a study on the thermal efficiency improvement of the user equipment room in the district heating system based on reinforcement learning [1], and suggests a general method of constructing a learning network(DQN)[2] using deep Q learning[3], which is a reinforcement learning algorithm that does not specify a model. In addition, we introduce the big data platform system and the integrated heat management system for the energy field in the massive data processing from the IoT sensor installed in large number of thermal energy control facilities.

We prove through path integral Monte Carlo computer experiments that the affine quantization of the $\varphi_4^4$ scaled Euclidean covariant relativistic scalar field theory is a valid quantum field theory with a well defined continuum limit of the one- and two-point-function. Affine quantization leads to a completely satisfactory quantization of field theories using situations that involve scaled behavior leading to an unexpected, $\hbar^2/\varphi^2$ which arises only in the quantum aspects.

Attitude control of a novel regional truss-braced wing aircraft with low stability characteristics is addressed in this paper using Reinforcement Learning (RL). In recent years, RL has been increasingly employed in challenging applications, particularly, autonomous flight control. However, a significant predicament confronting discrete RL algorithms is the dimension limitation of the state-action table and difficulties in defining the elements of the RL environment. To address these issues, in this paper, a detailed mathematical model of the mentioned aircraft is first developed to shape an RL environment. Subsequently, Q-learning, the most prevalent discrete RL algorithm will be implemented in both the Markov Decision Process (MDP), and Partially Observable Markov Decision Process (POMDP) frameworks to control the longitudinal mode of the air vehicle. In order to eliminate residual fluctuations that are a consequence of discrete action selection, and simultaneously track variable pitch angles, a Fuzzy Action Assignment (FAA) method is proposed to generate continuous control commands using the trained Q-table. Accordingly, it will be proved that by defining an accurate reward function, along with observing all crucial states (which is equivalent to satisfying the Markov Property), the performance of the introduced control system surpasses a well-tuned Proportional–Integral–Derivative (PID) controller.

The Semantic-Enhanced Tree-LSTM (SeT-LSTM) network, a novel advancement in the realm of linguistic modeling, marks a significant step forward from traditional Tree-based Long Short Term Memory (LSTM) networks. By intricately weaving in the nuances of typed grammatical dependencies, SeT-LSTMs offer a more nuanced understanding of language semantics. Traditional models often overlook the semantic shift caused by variations in word or phrase roles, a gap this paper aims to fill by focusing on the types of grammatical connections, or typed dependencies, within sentences. Our proposed architecture, dubbed the Semantic Relationship-Guided LSTM (SRG-LSTM), leverages a control mechanism to model the interplay between sequence elements. Additionally, we present a novel Tree-LSTM variant, the Semantic Dependency Tree-LSTM (SDT-LSTM), which integrates dependency parse structures with dependency types for more robust sentence embedding. The SDT-LSTM demonstrates superior performance in Semantic Relatedness Scoring and Sentiment Analysis compared to its predecessors. Qualitatively, it shows resilience to changes in sentence voice and heightened sensitivity to nominal alterations, aligning well with human intuition. This research underlines the pivotal role of grammatical relationships in sentence understanding and paves the way for further exploration in this domain.

Historically, the military and non-religious, sub-continent’s intellectual elite Muslim men crafted state ideologies before partition, by actively contributing to sociocultural constructs prevalent to this day, in service of colonial patriarchal concepts. The purpose of this ideological collaboration has been to mould a Pakistani cis-gendered female child primarily into the ideal “Pakistan-Muslim” bride for marriage, carried out through her family and society. This has shaped the lives of cis-gendered female children in Pakistan through coerced, gendered control, implemented as prescriptive societal norms. These norms are rooted in colonial, patriarchal ideologies and are functions of capital and coercion intentionally perpetuated to erase her pluralistic identity, reducing her to a central resource within a family unit. To the military concept of nationalism, her mind and her body symbolize the unification and centralization of a statist ideology. Thus, the military state plays an essential role in shaping her purpose around the concept of nationalism. The intellectual elite, via the state, uphold colonial, patriarchal ideologies, where the primary identity of a cis-gendered female child is actualized through her marriage. The cis-gendered female child’s life is a vehicle to unify the state’s two-fold “Pakistani-Muslim” agenda, by implementing a ‘singular statist narrative’. A narrative that attempts to resolve two major tensions: First, it seeks to wipe out the possibility of a nation connecting with and exploring its own immense, internal ethnocultural diversity. It does this by imposing an imagined “homogenous Muslim” identity — which doesn’t exist — and is instead imported from multiple countries to reinforce a ‘singular statist narrative’, Second, it aims to control the mixing of local, sociocultural narratives by bridging any disjuncture between them by labelling colonial institutional patriarchy concepts as inherently “Pakistani”; in service of its preferred ‘singular statist narrative’.

Transcutaneous injection laryngoplasty is a well-known procedure for treating paralyzed vocal fold by injecting augmentation material into it. Hence, vocal fold localization plays a vital role in the preoperative planning as the fold location is required to determine the optimal injection route. In this communication, we propose a mirror environment based reinforcement learning (RL) algorithm for localizing the right and left vocal folds in preoperative neck CT. RL-based methods commonly showed noteworthy outcome in general anatomic landmark localization problem in the recent years. However, such methods suggest training individual agent for localizing each fold, though the right and left vocal folds are located in close proximity and have high feature-similarity. Utilizing the lateral symmetry between the right and left vocal folds, the proposed mirror environment allows for a single agent for localizing both the folds by treating the left fold as a flipped version of the right fold. Thus, localization of both folds can be trained using a single training session which utilizes the inter-fold correlation and avoids redundant feature learning. Experiment with 120 CT volumes showed improved localization performance and training efficiency of the proposed method compared with the standard RL method.

Aiming at the problem of model error and tracking dependence in the process of intelligent vehicle motion planning, an intelligent vehicle model transfer trajectory planning method based on deep reinforcement learning is proposed, which obtain an effective control action sequence directly. Firstly, an abstract model of the real environment is extracted. On this basis, Deep Deterministic Policy Gradient (DDPG) and vehicle dynamic model are adopted to jointly train a reinforcement learning model, and to decide the optimal intelligent driving maneuver. Secondly, the actual scene is transferred to equivalent virtual abstract scene by transfer model, furthermore, the control action and trajectory sequences are calculated according to trained deep reinforcement learning model. Thirdly, the optimal trajectory sequence is selected according to evaluation function in the real environment. Finally, the results demonstrate that the proposed method can deal with the problem of intelligent vehicle trajectory planning for continuous input and continuous output. The model transfer method improves the model generalization performance. Compared with the traditional trajectory planning, the proposed method output continuous rotation angle control sequence, meanwhile, the lateral control error is also reduced.

The novel coronavirus disease (COVID-19) pandemic has resulted in over 720 million confirmed cases and 7 million deaths worldwide, with insufficient treatment options. Innumerable efforts are being made around the world for faster identification of therapeutic agents to treat the deadly disease. Postacute sequelae of SARS-CoV-2 infection or COVID-19 (PASC), also called Long COVID, is still being understood and lacks treatment options as well. A growing list of drugs are being suggested by various in silico, in vitro and ex vivo models, however currently only two treatment options are widely used: the RNA-dependent RNA polymerase (RdRp) inhibitor remdesivir, and the main protease inhibitor nirmatrelvir in combination with ritonavir. Computational drug development tools and in silico studies involving molecular docking, molecular dynamics, entropy calculations and pharmacokinetics can be useful to identify new targets to treat COVID-19 and PASC, as shown in this work and our recent paper that identified alendronate as a promising candidate. We have now investigated all bisphosphonates which can bind competitively to nidovirus RdRp-associated nucleotidyl (NiRAN) transferase domain, and systematically down selected seven candidates (CHEMBL608526, CHEMBL196676, CHEMBL164344, CHEMBL4291724, CHEMBL4569308, CHEMBL387132, CHEMBL98211), two of which closely resemble the approved drugs minodronate and zoledronate. This work and our recent paper together provide an in silico mechanistic explanation for alendronate and zoledronate users having dramatically reduced odds of SARS-CoV-2 testing, COVID-19 diagnosis, and COVID-19-related hospitalizations, and indicate that similar observational studies with minodronate could be valuable.

In recent years, advancements in brain science and neuroscience have significantly influenced the field of computer science, particularly in the domain of reinforcement learning (RL). Drawing insights from neurobiology and neuropsychology, researchers have leveraged these findings to develop novel mechanisms for understanding intelligent decision-making processes in the brain. Concurrently, the emergence of quantum computing has opened new frontiers in artificial intelligence, leading to the development of quantum machine learning (QML). This study introduces a novel model that integrates Quantum Spiking Neural Networks (QSNN) and Quantum Long Short-Term Memory (QLSTM) architectures, inspired by the complex workings of the human brain. Specifically designed for reinforcement learning tasks in energy-efficient environments, our approach progresses through two distinct stages mirroring sensory and memory systems. In the initial stage, analogous to the brain’s hypothalamus, low-level information is extracted to emulate sensory data processing patterns. Subsequently, resembling the hippocampus, this information is processed at a higher level, capturing and memorizing correlated patterns. We conduct a comparative analysis of our model against existing quantum models such as Quantum Neural Networks and their classical counterparts, elucidating its unique contributions. Through empirical results, we aim to underscore the effectiveness of our approach in optimizing energy consumption in practical scenarios.

The space exploration landscape has transformed dramatically over the past several decades, with a significant rise in the number of annual launches. Key factors driving this increase include technological advancements, the miniaturization of satellites, and the surge in commercial satellite deployments for communication, Earth observation, and remote sensing. This paper examines the evolution of space activities from the launch of Sputnik 1 in 1957 to recent developments in commercial spaceflight and international collaborations. The paper discusses notable milestones in space exploration history, including early space missions, Apollo Moon landings, and the development of space stations like the International Space Station. It highlights the shift from state-led space programs to a more diverse landscape, characterized by private sector involvement and international partnerships. The "Space 4.0" phase, beginning around 2006, has seen the entry of commercial players like SpaceX, leading to a significant upsurge in orbital launches. Statistical data reveals an upward trend in launch activity, with over 200 launches in 2023 alone, driven by large satellite constellations like SpaceX's Starlink and the steady growth of China's space program. The increasing launch rate poses challenges such as space debris, emphasizing the need for mitigation strategies. Looking ahead, the paper projects a continued increase in launch activity, with over 150 launches expected annually by 2026 and 200-300 launches by 2040. Commercial space ventures, deep space exploration, lunar activities, and space tourism drive this growth. The paper concludes by emphasizing the importance of addressing space debris and promoting the long-term sustainability of space activities, which is crucial as humanity's presence in space continues to expand.

The collision avoidance mechanism adopted by the IEEE 802.11 standard is not optimal. The mechanism employs a binary exponential backoff (BEB) algorithm in the medium access control (MAC) layer. Such an algorithm increases the backoff interval whenever a collision is detected to minimize the probability of subsequent collisions. However, the expansion of the backoff interval causes degradation of the radio spectrum utilization (i.e., bandwidth wastage). That problem worsens when the network has to manage the channel access to a dense number of stations, leading to a dramatic decrease in network performance. Furthermore, a wrong backoff setting increases the probability of collisions such that the stations experience numerous collisions before achieving the optimal backoff value. Therefore, to mitigate bandwidth wastage and, consequently, maximize the network performance, this work proposes using reinforcement learning (RL) algorithms, namely Deep Q Learning (DQN) and Deep Deterministic Policy Gradient (DDPG), to tackle such an optimization problem. As for the simulations, the NS-3 network simulator is used along with a toolkit known as NS3-gym, which integrates a reinforcement-learning (RL) framework into NS-3. The results demonstrate that DQN and DDPG have much better performance than BEB for static and dynamic scenarios, regardless of the number of stations. Moreover, the performance difference is amplified as the number of stations increases, with DQN and DDPG showing a 27% increase in throughput with 50 stations compared to BEB. Furthermore, DQN and DDPG presented similar performances.