Evolutionary algorithms and swarm intelligence algorithms find applicability in reinforcement learning of neural networks due to their independence from gradient-based methods. To achieve successful training of neural networks using these algorithms, careful considerations must be made to select appropriate algorithms due to the availability of various algorithmic variations. In Part1, 2 and 3, the author previously reported experimental evaluations on Evolution Strategy, Genetic Algorithm, and Differential Evolution for reinforcement learning of neural networks, utilizing the Acrobot control task. This article constitutes Part4 of the series of comparative research. In this study, Particle Swarm Optimization is adopted as an instance of major swarm intelligence algorithms. The experimental result shows that PSO performed worse than all of DE, GA and ES. The difference between PSO and DE was statistically significant (p<.01). In addition, PSO exhibited lower capability in exploring solutions in high-dimensional search spaces than DE, GA, and ES did. A larger swarm size compensated for the weakness of PSO in global exploration, thus making itself more beneficial than a larger number of swarm search iterations.

Selective Paste Intrusion (SPI) is an additive manufacturing (AM) process in which thin layers of aggregates are selectively bonded by cement paste only where the structure is to be produced. In this way, concrete elements with complex geometries and structures can be created. Reinforcement is required to increase the flexural strength of the concrete elements and, thus, enable their applicability in practice. Integrating the reinforcement is a difficult task, particularly in the case of SPI due to the layer-wise printing method. Especially with respect to possible complex structures, the production of the reinforcement needs to be adapted to SPI, thereby offering a high degree of freedom. One concept for a reinforcement integration is combining the two additive manufacturing processes SPI and Wire and Arc Additive Manufacturing (WAAM). However, since the two processes serve different fields of application, their compatibility is not necessarily given. Ongoing investigations show that the temperatures caused by WAAM adversely affect both, the cement paste rheology required for sufficient paste penetration into the particle bed and the overall concrete strength. This paper provides an overview of ongoing research focusing on different cooling strategies and their effects on the compressive strength of SPI-printed concrete parts.

Titanium-based metal-matrix composites manufactured by additive manufacturing offer tremendous lightweighting opportunities. However, processing high reinforcement content remains challenging. This study reports an improved manufacturing process for Ti-TiC enabling high reinforcement content and significant fracture strain concurrently: mechanical blending, followed by laser powder bed fusion and a single heat treatment. As-built microstructure shows both un-melted TiC particles and sub-stoichiometric TiC dendrites resulting from a partial dissolution of TiC particles. The heat treatment is shown to fully convert TiC dendrites into equiaxed TiC grains. Reduction of the C/Ti ratio in TiC during the process results in an increase in the reinforcement content, from a nominal 12 vol% to an effective 21.5 vol%. Ti-TiC tensile samples reached fracture strains of up to 1.7%, Young’s moduli of up to 149 GPa and ultimate tensile strengths of up to 827 MPa. Lower TiC initial powder size distributions displayed the best mechanical performance.

The increasing popularity of carbon-reinforced concrete (CRC) is attributed to its exceptional tensile properties, low density, no corrosion phenomenon, and remarkable flexibility, allowing it to be easily shaped into various forms. This research investigates the feasibility of using a special 2D Netzgitterträger (NetzGT) reinforcement system, featuring a net-shaped fabricated textile made of multiple diagonally offset rovings with overlapping edge strands, as a viable alternative to traditional steel reinforcement in concrete beams. This reinforcement is manufactured from carbon rovings with three different diagonal angles of 50⁰, 60⁰, and 70⁰ respectively. Laboratory experiments were conducted to assess the mechanical behavior of beams reinforced with the 2D NetzGT reinforcement. Bending and shear tests were performed on beams with varying numbers of overlapped edge roving and roving angles to evaluate the tensile capacity and failure characteristics of beams. The increase in the number of overlapped edge rovings led to a noticeable increase in the maximum tensile force. Tensile tests on strands were also performed with the increasing number of overlapped rovings to analyze their tensile strength. Additionally, single yarn pull-out tests were also conducted to examine the influence of the roving angle on the bond strength between the carbon textile roving and the concrete matrix

This systematic literature review explores the burgeoning use of Artificial Intelligence (AI) in manufacturing systems, in line with the principles of Industry 4.0 and the growth of smart factories. In this landscape, AI is crucial in addressing the complexity and dynamism of contemporary manufacturing processes, including machine breakdowns, fluctuating orders and unpredictable job arrivals. This systematic literature review, conducted using the Scopus database and bibliometric tools, pursues two primary objectives. First, it identifies the prevailing trends in solving scheduling problems with AI and identifies the most commonly used AI techniques in the literature. Secondly, it analyses how authors have successfully employed AI to address production scheduling challenges in real-world industrial settings and assesses the benefits obtained by companies. The dynamic nature of manufacturing systems requires adaptive scheduling paradigms. AI, including Particle Swarm Optimization, Neural Networks, and Reinforcement Learning, is applied to optimize production processes, predict machine failures, and achieve substantial benefits. In real-world applications, these AI-driven solutions have led to reduced production costs, enhanced energy efficiency, and more efficient scheduling processes. AI is increasingly recognized as an essential tool in addressing the evolving challenges of modern manufacturing environments.

Deep reinforcement learning is one of the research hotspots in artificial intelligence and has been successfully applied in many research areas, however, the low training efficiency and high demand for samples are problems that limit the application To address these problems, a hierarchical episodic control model extending episodic memory to the domain of hierarchical reinforcement learning is proposed in this paper. The model is theoretically justified and employs a hierarchical implicit memory planning approach for counterfactual trajectory value estimation. Starting from the final step and recursively moving back along the trajectory, a hidden plan is formed within the episodic memory. Experience is aggregated both along trajectories and across trajectories, and the model is updated using a multi-headed backpropagation similar to bootstrapped neural networks. This model extends the parameterized episodic memory framework to the realm of hierarchical reinforcement learning and is theoretically analyzed to demonstrate its convergence and effectiveness. Experiments conducted in Four Room, Mujoco, and UE4-based active tracking , highlight that the hierarchical episodic control model effectively enhances training efficiency. It demonstrates notable improvements in both low-dimensional and high-dimensional environments, even in cases of sparse rewards.

While machine learning models are powering more and more everyday devices, there is a growing need for explaining them. This especially applies to the use of Deep Reinforcement Learning in solutions that require security, such as vehicle motion planning. In this paper, we propose a method of understanding what the RL agent’s decision is based on. The method relies on conducting statistical analysis on a massive set of state-decisions samples. It indicates which input features have an impact on the agent’s decision and the relationships between decisions, the significance of the input features, and their values. The method allows us for determining whether the process of making a decision by the agent is coherent with human intuition and what contradicts it. We applied the proposed method to the RL motion planning agent which is supposed to drive a vehicle safely and efficiently on a highway. We find out that making such analysis allows for a better understanding agent’s decisions, inspecting its behavior, debugging the ANN model, and verifying the correctness of input values, which increases its credibility.

In 1948, Skinner described the behavior of pigeons under response-independent schedules as “superstitious,” and proposed that the responses were reinforced by contiguous, adventitious food deliveries. Subsequently, response-independent schedules have been of interest to both basic and applied researchers, first to understand the mechanisms involved, and later, as “noncontingent reinforcement” (NCR) to reduce undesirable behavior. However, the potential superstitious effects produced by these schedules have been challenged, with some researchers arguing that antecedent variables play a significant role. This paper examines the evidence for adventitious reinforcement from both laboratory and applied research, the results of which suggest that antecedent, non-operant functions may be important in fully understanding the effects of NCR. We propose an applied-basic research synthesis, in which attention to potential non-operant functions could provide a more complete understanding of response-independent schedules. We conclude with a summary of the applied implications of the non-operant functions of NCR schedules.

We review some properties of a relativistic classical massless charged particle with spin interacting with an external electromagnetic field. We give in particular a proper definition of kinetic energy and total energy, the latter being conserved when the external field is stationary. We find that the particle’s velocity may differ from c as a result of the spin - electromagnetic field interaction, without jeopardizing Lorentz invariance.

The adoption of algorithms based on Artificial Intelligence (AI) has been rapidly increasing during the last years. However, some aspects of AI techniques are under heavy scrutiny. For instance, in many use cases, it is not clear whether the decisions of an algorithm are well-informed and conforming to human understanding. Having ways to address these concerns is crucial in many domains, especially whenever humans and intelligent (physical or virtual) agents must cooperate in a shared environment. In this paper, we introduce an application of an explainability method based on the creation of a Policy Graph (PG) based on discrete predicates that represent and explain a trained agent’s behaviour in a multi-agent cooperative environment. We show that from these policy graphs, policies for surrogate interpretable agents can be automatically generated. These policies can be used to measure the reliability of the explanations enabled by the PGs, through a fair behavioural comparison between the original opaque agent and the surrogate one. The contributions of this paper represent the first application of policy graphs in the context of explaining agent behaviour in collaborative multi-agent scenarios and presents experimental results that sets this kind of scenario apart from previous application in single-agent scenarios: when requiring collaborative behaviour, predicates that allow representing observations about the other agents are crucial to replicate the opaque agent’s behaviour and increase the reliability of explanations.

Reinforcement Learning (RL) has demonstrated considerable potential in solving challenges across various domains, notably in autonomous driving. Nevertheless, implementing RL in autonomous driving comes with its own set of difficulties, such as the overestimation phenomenon, extensive learning time, and sparse reward problems. Although solutions like Hindsight Experience Replay (HER) have been proposed to alleviate these issues, the direct utilization of RL in autonomous vehicles remains constrained due to the intricate fusion of information and the possibility of system failures during the learning process. In this paper, we present a novel RL-based autonomous driving system technology that combines Obstacle Dependent Gaussian (ODG) RL, Soft Actor-Critic (SAC), and meta-learning algorithms. Our approach addresses key issues in RL, including the overestimation phenomenon and sparse reward problems, by incorporating prior knowledge derived from the ODG algorithm. We evaluated our proposed algorithm on official F1 circuits, using high-fidelity racing simulations with complex dynamics. The results demonstrate exceptional performance, with our method achieving up to 89% faster learning speed compared to existing algorithms in these environments.

An experimental study was carried out to evaluate the ductility of reinforced concrete beams longitudinally reinforced with hybrid FRP-Steel bars. The specimens were fourteen reinforced concrete beams with and without hybrid reinforcement. The test variables were bars position, the ratio of longitudinal reinforcement, and the type of FRP bars. The beams were loaded up to failure using a four-point bending test. The performance of the tested beams was observed using the load-deflection curve obtained from the test. Numerical analysis using the fiber element model was used to examine the growth of neutral axis depth due to the effect of test variables. The neutral axis curves were then used to further estimate the neutral axis angle and neutral axis displacement index. The test results show that the position of the reinforcement greatly influences the flexural behavior of the beam with hybrid reinforcement. It was observed from the test that the flexural capacity of beams with hybrid reinforcement is 4% to 50% higher than that of the beams with conventional steel bars depending on bars position and the ratio of longitudinal reinforcement. The ductility decreases as the hybrid reinforcement ratio (Af/As) increases. This study also showed that a numerical model developed can predict the flexural behavior of beams with hybrid reinforcement with reasonable accuracy.

Adaptive bitrate (ABR) algorithms optimize the quality of streaming experiences for users in client-side video players especially in unreliable or slow mobile networks. Several rule-based heuristic algorithms can achieve stable performance, but they sometimes fail to adapt properly to changing network conditions. Fluctuating bandwidth may cause algorithms to default to behavior that creates a negative experience for the user. ABR algorithms can be generated with reinforcement learning, a decision-making paradigm in which an agent learns to make optimal choices through interactions with an environment. Training reinforcement learning algorithms for bitrate streaming requires building a simulator for an agent to experience interactions quickly; training an agent in the real environment is infeasible due to the long step times in real environments. This project explores using supervised learning to construct a world-model, or a learned simulator, from recorded interactions. A reinforcement learning agent trained inside of the learned model, rather than a simulator, can outperform rule-based heuristics. Furthermore, agents trained inside the learned world-model can outperform model-free agents in low sample regimes. This work highlights the potential for world-models to quickly learn simulators, and to be used to generate optimal policies.

In recent years, the viability of employing multi-agent reinforcement learning technology for adaptive traffic signal control has been extensively validated. However, owing to restricted communication among agents and the partial observability of the traffic environment, the process of mapping road network states to actions encounters numerous challenges. To address this problem, this paper proposes a multi-agent deep reinforcement learning model with an emphasis on communication content (CMARL). The model decouples the complex relationships between multi-signal agents through centralized training and decentralized execution. Specifically, we first pass the traffic state through an improved deep neural network to achieve the extraction of high-dimensional semantic information and the learning of the communication matrix. Then the agents selectively interact with each other based on the learned communication matrix and generate the final state features. Finally, the features are inputted to the QMIX network to achieve the final action selection. We compare the CMARL model with 6 other baseline algorithms in real traffic networks. The results show that CMARL can significantly reduce vehicle congestion, and run stably in various scenarios.

A calculation is presented for the mass relationships between electron, muon and the tau particle. It is shown that charged leptons with mass levels beyond the one of the tau particle cannot exist because of insufficient binding energy. The underlying mechanism of the relationship between the leptons is the isospin feature of Dirac particles, which appears being a substantial attribute next to nuclear angular spin. It is further shown that the energy of the elementary electric charge, such as embodied by the mass of an electron, is due to the interaction between the isospins of the pion’s quark and antiquark.

The non-symmetrical collapse of an empty cylindrical cavity is modelled using Smoothed Particle Hydrodynamics. The presence of a nearby surface produces an anisotropic pressure field generating a high velocity jet that hits the surface. The collapse follows a different dynamic based to the initial distance between the centre of the cavity and the surface. When the distance is greater than the cavity radius (detached cavity) the surface is hit by travelling shock waves. When the distance is less than the cavity radius (attached cavity) the surface is directly hit by the jet and later by other shock waves generated in the last stages of the of the collapse. The results show that the surface is hit by a stronger shock when distance between the centre of the cavity and the surface is zero while showing more complex double peaks behaviour for other distances.

In this study, a comprehensive review on production scheduling based on reinforcement learning (RL) techniques using bibliometric analysis has been carried out. The aim of this work is, among other things, to point out the growing interest in this domain and to outline the influence of RL as a type of machine learning on production scheduling. To achieve this, the paper explores production scheduling using RL by investigating the descriptive metadata of all pertinent publications contained in the Web of Science database. The study focuses on a wide spectrum of publications spanning the years between 1998 and 2023. The findings and recommendations of this study can serve as new insights for future research endeavors in the realm of production scheduling using RL techniques.

The problem addressed in this article consists of the motion control of a quadrotor, comprising model disturbances and uncertainties. In order to tackle model uncertainty, adaptive control based on reinforcement learning is used. The distinctive feature of this article, in comparison with other works on quadrotor control using reinforcement learning is the exploration of the underlying optimal control problem in which a quadratic cost and a linear dynamics allow an algorithm that runs in real time. Instead of identifying a plant model, adaptation is obtained by approximating the performance index given by the Q-function using directional forgetting recursive least squares that rely on a linear regressor build from quadratic functions of input/output data. The adaptive algorithm proposed is tested in simulation in a cascade control structure that drives a quadrotor. Simulations show the improvement in performance that results when the proposed algorithm is turned on.

Malaria is a long-standing disease and one of the top life-threatening diseases, yet its treatment has not changed, while the world has already embraced the Fourth Industrial Revolution (4IR). A wave of research on digitizing monitoring mechanisms of such a deadly disease has surfaced. Automated malaria screening is one of the detection processes which are gaining popularity in the research domain. However, the process needs to be coupled with other processes aiming a nationally or regionally contextualised malaria monitoring system. This paper proposes a digital malaria monitoring system in the context of an African country or region. One advantage of such a digital system is that is enables a novel disease spread forecasting model based on the dynamics of different malaria types. The architecture of the diagnosis system is described, and the disease spread model is mathematically modelled in terms of a SPITR (Susceptible- Protected- Infected-Treated- Recovered) epidemic model which is further analysed. The forecasting model is expressed and analysed whereas experiments are conducted using a Monte Carlo simulation method. The design of the monitoring system has inspired how predictions can be made in the complex cases such as mixed infections. Results show that reinforcing the model parameter makes a significant improvement on the disease prediction.

The need to develop novel lightweight materials and their manufacturing processes is sets out to meet the new aerospace, automotive and construction requirements. Within this context, this research work is proposed to develop a novel thermoplastic composite material with high mechanical properties. These composites will be based on thermoplastic matrixes made from polyamide and 35% short glass fiber filled-polyamide reinforced with different types of fabrics. As reinforcement, glass fiber fabrics will be used as the base. They will be treated with different processes, both chemical and physical, to promote adherence to the matrix. Textile overmoulding technology was selected for manufacturing these composites. This technology was primarily developed to manufacture aesthetic lined components and has achieved a great implantation. Once these new composites are manufactured, they will be submitted to different tests to evaluate their behavior regarding adhesion, impact strength and stiffness. It is expected an improvement on stiffness and impact absorption.

Most European cities are characterized by very large areas, often formed by buildings with low quality, from a series of point of view. The possibility of renovating them is strategic to improve both the quality of life and to the possibility of economic recovery for building companies. In the last decades, the attention of the scientific community has been addressed to the energy renovation, thanks to the strong activities of the European Community in this field. However, since a relevant part of the EC territory is at risk of earthquake, the possibility to combine both energy and seismic renovation actions may be strategic for many countries. In particular, Italy and Romania are linked by a common social tradition that springs from the Roman Empire. Nowadays, this link is stronger, thanks to common interests in social, cultural and business fields. Therefore, the investigation of possible synergies for seismic and energy renovation strategies may be really interesting for both countries. This paper represents the first step in this direction. After an overview of regulations and common practices for buildings with reinforced concrete structures, in both states, some key combined renovation interventions will be described and discussed, as well as advantages and perspectives of integrated renovation approaches.

The structural performance of reinforced concrete relies heavily on the bond between reinforcement and concrete. In nonlinear finite element analyses, bond is either modelled by merged, also called perfect bond, or coincident with slip, also called bond-slip, approaches. Here, the performance of these two approaches for the modelling of failure of reinforced concrete was investigated using a damage-plasticity constitutive model in LS-DYNA. Firstly, the influence of element size on the response of tension-stiffening analyses with the two modelling approaches was investigated. Then, the results of the two approaches were compared for plain and fibre reinforced tension stiffening and a drop weight impact test. It was shown that only the coincident with slip approach provided mesh insensitive results. However, both approaches were capable of reproducing the overall response of the experiments in the form of load and displacements satisfactorily for the meshes used.

Dynamic adaptive video streaming over HTTP (DASH) plays a crucial role in video transmission across networks. Traditional adaptive bitrate (ABR) algorithms adjust the quality of video segments based on network conditions and buffer occupancy. However, these algorithms rely on fixed rules within a complex environment, making it challenging to achieve optimal decisions considering the overall context. In this paper, we propose a novel Deep Reinforcement Learning-based approach for streaming DASH, focusing on maintaining consistent perceived video quality throughout the streaming session to enhance user experience. Our approach optimizes the Quality of Experience (QoE) by dynamically controlling the quality distance factor between consecutive video segments. We evaluate this approach through a simulation model that encompasses diverse wireless network environments and various video sequences. Additionally, we compare our proposed approach with state-of-the-art methods. The experimental results demonstrate significant improvements in QoE, ensuring users enjoy stable, high-quality video streaming sessions.

While the standard model accurately describes data at the electroweak scale without inclusion of gravity, beyond the standard model physics is increasingly intertwined with gravitational phenomena and cosmology. Thus gravity mediated breaking of supersymmetry in supergravity models lead to sparticles masses, which are gravitational in origin, observable at TeV scales and testable at the LHC, and supergravity also provides candidate for dark matter, a possible framework for inflationary models and for models of dark energy. Further, extended supergravity models, and string and D-brane models contain hidden sectors some of which may be feebly coupled to the visible sector resulting in heat exchange between the visible and hidden sectors. Because of the couplings between the sectors both particle physics and cosmology are effected. The above implies that particle physics and cosmology are intrinsically intertwined in the resolution of essentially all of the cosmological phenomena such as dark matter and dark energy and in the resolution of cosmological puzzles such as Hubble tension and EDGES anomaly. Here we give a brief overview of the intertwining and implications for the discovery of sparticles, and the resolution of the cosmological anomalies and identification of dark matter and dark energy as major challenges for the coming decades.

The fluid flow field at the upper airways is highly complex due to the complex structure of the airway. The inhaled particle flow, the air streamline and the interaction of the continuum and discrete phase could significantly affect the transport behaviour of the inhaled particles. A range of analytical, mathematical and computational fluid dynamics (CFD) models analyzed the airflow and particle transport in different idealized and asymmetric airway models. A precise understanding of the continuum and discrete phase interaction in realistic human airways is missing, and this study aims to develop a CFD-DEM model for particle transport in realistic airways. This study uses the CFD model for the continuum phase and the discrete element method (DEM) for the discrete phase. A soft sphere approach is used for the interaction of the discrete phase. Proper validation is performed for particle transport efficiency. The CFD-DEM model analyzed the particle transport in an idealized and realistic airway model, and different methods are used to analyze the transport behaviour. During the particle-particle interaction, a stagnation point and a high-pressure zone are observed at the airway model's carinal angle. The numerical results report higher deposition efficiency (DE) for particle-particle interaction than without interaction. The flow field becomes highly complex with the spring constant values, and higher DE is found for high spring constant values. The spring dashpot friction-dshf method shows higher deposition at the upper part of the airways than other interaction methods. The findings of this study and more case-specific analysis would improve the knowledge of aerosol transport in airways and the health risk assessment of the patient.

Double suction centrifugal pump installed along the Yellow River faces the serious sediment erosion due to the high sediment content which cause the poor operation efficiency of pump unit. The particle motion characteristics and erosion characteristics in the pump under different flow rates and different particle concentrations were numerically simulated based on the particle track model of solid-liquid two-phase flow. The results show that the flow rate has a significant effect on the particle tracks and the erosion caused by the particles in the impeller. The total erosion rate is positively correlated with the flow rate, and increases with the increase of flow rate. The vortex and secondary flow in the impeller have obvious influence on the particle trajectory, which increases the particle concentration at the trailing edge of the pressure surface and intensifies the impact erosion in this area. The particles carried by the vortex intensifies the local erosion. The particle concentration mainly affects the erosion rate, but has little effect on the erosion position. The in-fluence of flow rate on the pump erosion is greater than that of the particle properties. These results provide a reference for optimization of the design of anti-erosion blades of the double-suction pump and regulation-operation of pumping station.

Meshfree particle method, which is always regarded as a pure Lagrangian approach, is easily represented complicated domain topologies, moving boundaries, and multiphase media. Solving acoustic problems with the mesfree particle method forms a branch of the acoustic wave modeling field, namely, particle-based computational acoustics (PCA). The aim of this paper is to improve the accuracy of using the PCA method to solve two-dimensional acoustic problems, and realize the particle representation with a hybrid meshfree and finite-difference time-domain (FDTD) method for acoustic boundary conditions at both the plane and curved surface. As a widely used Lagrangian meshfree method, the smoothed particle hydrodynamics (SPH) based on the support domain and the kernel function has developed rapidly in recent years. The traditional SPH method is easily implements parallel processing and has been applied in sound wave simulation. As a corrective method with higher accuracy than SPH, the acoustic propagation and scattering in the time domain is simulated with the corrective smoothed particle method (CSPM). Moreover, a hybrid meshfree-FDTD boundary treatment technique is utilized to represent different acoustic boundaries in the Lagrangian approach. In this boundary treatment technique, the parameter value of virtual particles is obtained with the FDTD method, which concerns truncation errors based on the Tayler series expansion. Soft, rigid, and Mur’s absorbing boundary conditions are developed to simulate sound waves in finite and infinite domain. Results of modeling acoustic propagation and scattering show that CSPM is accurate and convergence with exact solutions, and different acoustic boundaries are validated to be effective in the computation.

To further enhance the intelligence level of coal mining faces and achieve autonomous derivation, learning, and optimization of shearer navigation cutting, this paper proposes the methods of shearer digital twin navigation cutting motion planning based on the concept of shearer autonomous navigation cutting technology and intelligent coal mining face digital twins. The study includes the digital twin theory and the construction method of the shearer digital twin navigation cutting motion planning system based on this theory. Based on the digital twin theory, a shearer digital twin navigation cutting motion planning system was constructed. This system supports the service functions of shearer cutting digital twin, dynamic navigation map digital twin, reinforcement learning environment construction, and motion planning through the physical perception layer, comprehensive data layer, and digital-model fusion analysis layer. Finally, by comparing the effects of the DQN-NAF and DDPG deep reinforcement learning algorithms in the shearer motion planning task within the constructed digital twin environment, the results show that the DQN-NAF algorithm demonstrates better performance and stability in solving the shearer digital twin motion planning task.

In this work, we propose an approach for the autonomous navigation of mobile robots using fusion of sensor data by a Double Deep Q-Network with collision avoidance by detecting moving people via computer vision techniques. We evaluate two data fusion methods for the proposed autonomous navigation approach: Interactive and Late fusion strategy. Both are used to integrate mobile robot sensors through the following sensors: GPS, IMU and, an RGB-D camera. The proposed collision avoidance module is implemented along with the sensor fusion architecture in order to prevent the autonomous mobile robot from colliding with moving people. The simulation results indicate a significant impact on the success of completing the proposed mission by the mobile robot with the fusion of sensors, indicating a performance increase (success rate) of 27% in relation to navigation without sensor fusion. With the addition of moving people in the environment, deploying the people detection and collision avoidance security module has improved about 14% the success rate when compared to that of the autonomous navigation approach without the security module. Video was developed with robot navigation using the DDQN-Late Fusion https://www.loom.com/share/684afa6a5b0148afadc9a200ab9f3483.

We discuss the implementation of a suite of virtual diagnostics at the FACET-II facility currently under commissioning at SLAC National Accelerator Laboratory. The diagnostics will be used for prediction of the longitudinal phase space along the linac, spectral reconstruction of the bunch profile and non-destructive inference of transverse beam quality (emittance) using edge radiation at the injector dogleg and bunch compressor locations. These measurements will be folded in to adaptive feedbacks and ML-based reinforcement learning controls to improve the stability and optimize the performance of the machine for different experimental configurations. In this paper we describe each of these diagnostics with expected measurement results based on simulation data and discuss progress towards implementation in regular operations.

The prevalence of differential equations as a mathematical technique has refined the fields of control theory and constrained optimization due to the newfound ability to accurately model chaotic, unbalanced systems. However, in recent research, systems are increasingly more nonlinear and difficult to model using Differential Equations only. Thus, a newer technique is to use policy iteration and Reinforcement Learning, techniques that center around an action and reward sequence for a controller. Reinforcement Learning (RL) can be applied to control theory problems since a system can robustly apply RL in a dynamic environment such as the cartpole system (an inverted pendulum). This solution successfully avoids use of PID or other dynamics optimization systems, in favor of a more robust, reward-based control mechanism. This paper applies RL and Q-Learning to the classic cartpole problem, while also discussing the mathematical background and differential equations which are used to model the aforementioned system.

The increasing trend of studying the innate softness of robotic structures and amalgamating it with the benefits of the extensive developments in the field of embodied intelligence has led to sprouting of a relatively new yet extremely rewarding sphere of technology. The fusion of current deep reinforcement algorithms with physical advantages of a soft bio-inspired structure certainly directs us to a fruitful prospect of designing completely self-sufficient agents that are capable of learning from observations collected from their environment to achieve a task they have been assigned. For soft robotics structure possessing countless degrees of freedom, it is often not easy (something not even possible) to formulate mathematical constraints necessary for training a deep reinforcement learning (DRL) agent for the task in hand, hence, we resolve to imitation learning techniques due to ease of manually performing such tasks like manipulation that could be comfortably mimicked by our agent. Deploying current imitation learning algorithms on soft robotic systems have been observed to provide satisfactory results but there are still challenges in doing so. This review article thus posits an overview of various such algorithms along with instances of them being applied to real world scenarios and yielding state-of-the-art results followed by brief descriptions on various pristine branches of DRL research that may be centers of future research in this field of interest.

Natural fibers are being increasingly used in different areas of engineering, including as composite reinforcement. Among these fibers, carauá stands out for its good mechanical properties and adherence to resin. Nevertheless, little is known about the behavior of this material in the manufacture of a composite or whether classic failure theory can be used in this case. In this context, the present study assesses the mechanical properties of two laminas made of unidirectional curauá fiber with volumetric fiber percentages of 30 % and 22 %, and compares the results with the values obtained for four failure criteria reported in the literature, using analysis of variance (ANOVA). To that end, tensile tests were conducted in the direction of the fiber and at other loading angles, in addition to iosipescu shear tests. The results show that the maximum stress criterion does not represent the failure behavior of these materials and that the best was the Hashin criterion.

Composite fiber materials are superior materials due to their high strength and light weight. Composites reflect the properties of their constituents, which is proportional to the volume fraction of each phase. There are different fiber reinforcement types and each affects it’s flexural, tensile and compression strength. When selecting a composite for a specific application, the forces excreted on the composite must be known in order to determine the reinforcement type. Unidirectional fiber reinforcement will allow very strong load resistance but only in one direction where as a random orientated fiber reinforcement can resist less load but can maintain this quota in all directions. These materials are said to be anisotropic. Certain composite fibers, taking into consideration there weights, are physically stronger than conventional metals. This research deals with the analysis of three composite materials with different reinforcement types, volume fraction and phase content. It was found that material A (glass epoxy) was the strongest in the longitudinal direction with a flexural strength of 534 MPa in the longitudinal and 420 MPa in the transverse direction. The flexural stresses of material B (glass silicone) and material C (glass polyester) where both found in the 120 to 135 MPa range. Differences were due to their differences in matrix composition and reinforcement type.

Abstract: As the policies and regulations currently in place concentrate on environmental protection and greenhouse gas reduction, we are steadily witnessing a shift in the transportation industry towards electromobility. There are, though, several issues that need to be addressed to encourage the adoption of EVs at a larger scale. To this end, we propose a solution capable of addressing multiple EV charging scheduling issues, such as congestion management, scheduling a charging station in advance, and allowing EV drivers to plan optimized long trips using their EVs. The smart charging scheduling system we propose considers a variety of factors such as battery charge level, trip distance, nearby charging stations, other appointments, and average speed. Given the scarcity of data sets required to train the Reinforcement Learning algorithms, the novelty of the recommended solution lies in the scenario simulator, which generates the labelled datasets needed to train the algorithm. Based on the generated scenarios, we created and trained a neural network that uses a history of previous situations to identify the optimal charging station and time interval for recharging. The results are promising and for future work we are planning to train the DQN model using real-world data.

The fundamental divergence in foundation of modern physics lies in the incompatibility between quantum theory and relativity. This study created a new mathematical tools, proposed the dark particle hypothesis and new equivalent principle, and established an ideal model on the cosmic continuum. According to this study, any cosmic system is a continuum composed of existence continuum relative to the wavelength of bosons energy waves and its dimension continuum, four fundamental interactions form a continuum relative to the wavelength of corresponding boson energy waves, respectively, the elementary particles and their existence fields are the basic units of cosmic continuum, the existence field of an existence body is equivalent to its dimension field, dark matter is a dark mass body composed of dark particles, singularity is a dark matter celestial body formed by collapse of black hole, and the singularity and its existence field constitute an existence continuum of singularity.

Automated systems, regulated by algorithmic protocols and predefined set-points for feedback control, require the oversight and fine-tuning of skilled technicians. This necessity is particularly pronounced in automated greenhouses, where optimal environmental conditions depend on the specialized knowledge of dedicated technicians, emphasizing the need for expert involvement during installation and maintenance. To address these challenges, this study proposes the integration of data acquisition technologies using Internet of Things (IoT) protocols and optimization services via Reinforcement Learning (RL) methodologies. The proposed model was tested in an industrial production greenhouse for the cultivation of industrial hemp, applying adapted strategies to the crop, and was guided by an agronomic technician knowledgeable about the plant. The expertise of this technician was crucial in transferring the RL model to a real-world automated greenhouse equipped with IoT technology. The study concludes that the integration of IoT and RL technologies is effective, validating the model's ability to manage and optimize greenhouse operations efficiently and adapt to different types of crops. Moreover, this integration not only enhances operational efficiency but also reduces the need for constant human intervention, thereby minimizing labor costs and increasing scalability for larger agricultural enterprises. Furthermore, the RL-based control has demonstrated its ability to maintain selected temperatures and achieve energy savings compared to classical control methods.

The current design of reinforcement learning methods demands exhaustive computing. Algorithms such as Deep Q-Network achieved outstanding results in the development of the area. However, the need for thousands of parameters and training episodes is still a problem. Thus, this document proposes a comparative analysis of the Q-Learning algorithm (the inception to create Deep Q Learning) and our proposed method termed M-Learning. The comparison among algorithms using Markov decision processes with delayed reward as a general testbench framework. Firstly, a full description of the main problems related to implementing Q-Learning, mainly about its multiple parameters. Then, the foundations of our proposed heuristic with its formulation and the whole algorithm were reported in detail. Finally, the methodology chosen to compare both algorithms was to train the algorithms in the Frozen Lake environment. The experimental results and an analysis of the best solutions found that our proposed algorithm highlights the differences in the number of episodes necessary and their standard variations. The code will be available on a GitHub repository once the paper is published.

Goal-conditioned reinforcement learning (RL) holds promise for addressing intricate control challenges by enabling agents to learn and execute desired skills through separate decision modules. However, the irregular occurrence of required skills poses a significant challenge to effective learning. In this paper, we demonstrate the detrimental effects of this imbalanced skill(sub-goal) distribution and propose a novel training approach, Classified Experience Replay (CER), designed to mitigate this challenge. We demonstrate that adapting our method to conventional RL methods significantly enhances the performance of the RL agent. Considering the challenges inherent in tasks such as driving, characterized by biased occurrences of required sub-goals, our study demonstrates the improvement in trained outcomes facilitated by the proposed method. In addition, we introduce a specialized framework tailored for self-driving tasks on highways, integrating Model Predictive Control (MPC) into our RL trajectory optimization training paradigm. Our approach, utilizing CER with the suggested framework, yields remarkable advancements in trajectory optimization for RL agents operating in highway environments.

This study investigated how positive (e.g., incentives) and negative (e.g., restrictions) reinforcement used in vaccination promotional messages and trust in government impact parental vaccine perceptions and decisions on vaccinating their children during a large-scale infectious disease crisis. Two hundred and eighty-six parents of unvaccinated teens participated in a 2 (reinforcement: positive vs. negative) × 2 (trust in government: high vs. low) between-subjects experimental survey. Positive reinforcement (e.g., gift cards for the vaccinated) was more effective for parents with low trust in the government to improve their attitudes and intentions toward child vaccination, while parents with high trust in the government were unaffected by reinforcement types. To the best of our knowledge, this study is among the first to examine the effects of both positive and negative reinforcement on vaccination perceptions and behavioral intentions through a controlled experiment and is the first study to do so in a large-scale infectious disease crisis setting. This research is expected to help public health professionals better prepare for and manage virus-induced public health crises in the future.

The conducted investigation encompassed the comprehensive integration of mechanical, envi-ronmental, chemical, and microstructural evaluations of a composite amalgamating sandy soil and a synthetic polymer at two distinct concentrations (2.5% and %) across multiple curing in-tervals (0, 1, 2, 4, 7, 15, 30, and 45 days). The studied soil originates from an environmentally sig-nificant protected area in Brazil. The implementation of mechanisms for soil improvement in the region must adhere to technical criteria without causing environmental harm. Direct shear testing was conducted, permeability was assessed, and microstructure analysis and XRD and XRF/EDX studies of both the soil and composites were conducted. It was observed that longer curing times yielded improved results in shear stress, friction angle, and cohesive intercept, with SP_5% exhibiting the highest values compared to the soil and SP_2.5%. Adding the polymeric so-lution to the soil contributed to cementation and cohesion gains in the substrate. Through mi-crostructural characterization, the polymer's role as a cementing agent for the grains is evident, forming a film on the grains and binding them together. Based on the analyses and studies con-ducted in the research, it can be concluded that there is technical feasibility for applying the pol-ymeric solution at both dosages in geotechnical projects.

Machine learning offers advanced tools for efficient management of radio resources in modern wireless networks. In this study, we leverage a multi-agent deep reinforcement learning (DRL) approach, specifically the Parameterized Deep Q-Network (DQN), to address the challenging problem of power allocation and user association in massive multiple-input multiple-output (M-MIMO) communication networks. Our approach tackles a multi-objective optimization problem aiming to maximize network utility while meeting stringent quality of service requirements in M-MIMO networks. To address the non-convex and nonlinear nature of this problem, we introduce a novel multi-agent DQN framework. This framework defines a large action space, state space, and reward functions, enabling us to learn a near-optimal policy. Simulation results demonstrate the superiority of our Parameterized Deep DQN (PD-DQN) approach when compared to traditional DQN and RL methods. Specifically, we show that our approach outperforms traditional DQN methods in terms of convergence speed and final performance. Additionally, our approach shows 72.2 % and 108.5 % improvement over DQN methods and RL method respectively in handling large-scale multi-agent problems in M-MIMO networks.

Shape Memory Alloy (SMA) have emerged as a revolutionary material that holds the potential to change the design and construction methods of buildings and other structures. One of the most promising applications of SMAs in civil engineering is their use as reinforcements for steel structures. This paper presents the latest research progress, opportunities and challenges of SMA in the field of steel structural reinforcement, both in terms of basic components and applications. In terms of components, the construction forms and working mechanisms of Fe-SMA strips, SMA/CFRP composite patches and SMA dampers are introduced. On this basis, the application of SMA in steel structures reinforcement is introduced, and its effect is analyzed from three aspects: crack restoration, seismic retrofitting and structural strengthening. Finally, the results of the current research are summarized and the shortcomings are analyzed, hoping to provide a reference for the research of SMA in the field of steel structures reinforcement.

Digital Microfluidic Biochips (DMFBs), used in various kinds of fields like DNA analysis, clinical diagnosis, and PCR testing, have made biochemical experiments more compact, efficient, and user-friendly than previous ways. However, their reliability is often compromised by their inability to adapt to all kinds of errors. All errors in biochips can be categorized into two types: known errors and unknown errors. Known errors are detectable before the start of the routing process through sensors or cameras. Unknown errors, in contrast, become apparent only during the routing process and remain undetected by sensors or cameras, which is the biggest issue to unexpectedly stop the routing process and diminishes the reliability of biochips. This paper introduces a deep reinforcement learning-based routing algorithm designed to manage not only known errors but also unknown errors. Our experiments demonstrate that our algorithm outperforms previous ones in terms of the success rate of the routing in the scenario including both known errors and unknown errors. Additionally, our algorithm contributes to detecting unknown errors during the routing process and identifying the most efficient routing path with high probability.

This report gives a comprehensive summary of two problems about graph convolutional networks (GCNs): over-smoothing and heterophily challenges, and outlines future directions to explore.

The demand for explainable and transparent models increases with the continued success of reinforcement learning. In this article, we explore the potential of generating shallow decision trees (DT) as simple and transparent surrogate models for opaque deep reinforcement learning (DRL) agents. We investigate three algorithms for generating training data for axis-parallel and oblique DTs with the help of DRL agents ("oracles") and evaluate these methods on classic control problems from OpenAI Gym. The results show that one of our newly developed algorithms, the iterative training, outperforms traditional sampling algorithms, resulting in well-performing DTs that often even surpass the oracle from which they were trained. Even higher dimensional problems can be solved with surprisingly shallow DTs. We discuss the advantages and disadvantages of different sampling methods and insights into the decision-making process made possible by the transparent nature of DTs. Our work contributes to the development of not only powerful but also explainable RL agents and highlights the potential of DTs as a simple and effective alternative to complex DRL models.

Locomotor impairment is a high-prevalent and significant source of disability and significantly impacts a large population’s quality of life. Despite decades of research in human locomotion, the challenges of simulating human movement to study the features of musculoskeletal drivers and clinical conditions remain. Most recent efforts in utilizing reinforcement learning (RL) techniques are promising to simulate human locomotion and reveal musculoskeletal drives. However, these simulations often failed to mimic natural human locomotion because most reinforcement strategies have yet to consider any reference data regarding human movement. To address these challenges, in this study, we designed a reward function based on the trajectory optimization rewards (TOR), and bio-inspired rewards, which includes the rewards obtained from reference motion data captured by a single Intertial Moment Unit (IMU) sensor. The sensor was equipped on the participants’ pelvis to capture reference motion data. Also, we adapted the reward function by leveraging previous research in walking simulation for TOR. The experimental results showed that the simulated agents with the modified reward function performed better in mimicking the collected IMU data from participants, which means the simulated human locomotion was more realistic. Also, as this bio-inspired defined cost, IMU data enhanced the agent’s capacity to converge during the training process. As a result, the models’ convergence is faster than those developed without reference motion data. Consequently, human locomotion can be simulated more quicker and in a broader range of environments with a better simulation performance.

With the rapid development of the Internet of Things (IoT), there are dramatic increasing number of devices in the network, which causes the challenge that only using infrastructure such as based station cannot provide service with all devices with high quality. Therefore, due to their flexibility and economy, unmanned aerial vehicles (UAV) are widely used to increase the performance of IoT networks. UAVs can not only provide communication services for IoT devices in the absence of a network, but they can also perform video surveillance, cargo transportation, pesticide spraying, and other specialized tasks. However, due to the complexity of the scenario and the need for real-time decision making, it is challenging to schedule UAVs in the network using traditional optimization methods, and growing attention has focused on using AI to optimize UAVs in the network. In this paper, we focus on the AI-enabled UAV optimization method in IoT networks and give a comprehensive scope on what and how to use AI-enabled methods to increase the performance of UAV-assisted IoT networks. Moreover, a brief analysis of the challenges of using AI methods in IoT networks and some potential research directions are given.

Background and motivation: Combining Deep Reinforcement Learning (Deep RL) and Health Systems Simulations has significant potential, for both research into improving Deep RL performance and safety, and in operational practice. While individual toolkits exist for Deep RL and Health Systems Simulations, no framework to integrate the two has been established. Aim: Provide a framework for integrating Deep RL Networks with Health System Simulations, and to ensure this framework is compatible with Deep RL agents that have been developed and tested using OpenAI Gym. Methods: We developed our framework based on the OpenAI Gym framework, and demonstrate its use on a simple hospital bed capacity model. We built the Deep RL agents using PyTorch, and the Hospital Simulation using SimPy. Results: We demonstrate example models using a Double Deep Q Network or a Duelling Double Deep Q Network as the Deep RL agent. Conclusion: SimPy may be used to create Health System Simulations that are compatible with agents developed and tested on OpenAI Gym environments. GitHub repository of code: https://github.com/MichaelAllen1966/learninghospital

Learning to maintain postural balance while standing requires a significant fine coordination effort between the neuromuscular system and the sensory system. It is one of the key contributing factors towards fall prevention, especially in the older population. Using artificial intelligence (AI), we can similarly teach an agent to maintain a standing posture, and thus teach the agent not to fall. In this paper, we investigate the learning progress of an AI agent and how it maintains a stable standing posture through reinforcement learning. During training, the AI agent learnt three policies. First, it learnt to maintain the Centre-of-Gravity and Zero-Moment-Point in front of the body. Then, it learnt to shift the load of the entire body on one leg while using the other leg for fine tuning the balancing action. Finally, it started to learn the coordination between the two pre-trained policies. This study shows the potentials of using deep reinforcement learning in human movement studies. The learnt AI behaviour also exhibited attempts to achieve an unplanned goal because it correlated with the set goal (e.g. walking in order to prevent falling). The failed attempts to maintain a standing posture is an interesting by-product which can enrich the fall detection and prevention research efforts.

The popularity of deep reinforcement learning (DRL) methods in economics have been exponentially increased. DRL through a wide range of capabilities from reinforcement learning (RL) and deep learning (DL) for handling sophisticated dynamic business environments offers vast opportunities. DRL is characterized by scalability with the potential to be applied to high-dimensional problems in conjunction with noisy and nonlinear patterns of economic data. In this work, we first consider a brief review of DL, RL, and deep RL methods in diverse applications in economics providing an in-depth insight into the state of the art. Furthermore, the architecture of DRL applied to economic applications is investigated in order to highlight the complexity, robustness, accuracy, performance, computational tasks, risk constraints, and profitability. The survey results indicate that DRL can provide better performance and higher accuracy as compared to the traditional algorithms while facing real economic problems at the presence of risk parameters and the ever-increasing uncertainties.

The widespread adoption of electromobility constitutes one of the measures designed to reduce air pollution caused by traditional fossil fuels. However, several factors are currently impending this process, ranging from insufficient charging infrastructure, battery capacity, long queueing and charging time, to psychological factors. On top of range anxiety, the frustration of the EV drivers is further fueled by the lack the uncertainty of finding an available charging point on their route. To address this issue, we propose a solution that comes to bypass the limitations of the Reserve now function of the OCPP standard, enabling drivers to make charging reservations for the upcoming days, especially when planning a longer trip. We created an algorithm that generates reservation intervals based on the charging station's reservation and transaction history. Subsequently, we ran a series of test cases that yielded promising results, with no overlapping reservations.

A new approach was presented in this study to simulate a stable atom along with subatomic particles in it. Subatomic particles in an atom are treated as conventional particles, exhibiting attraction and repulsion between them, giving a stable atom simulation. This new approach leads to developing new potentials for subatomic particles to simulate a stable atom. Subatomic particle dynamics (SPD) simulations were developed for within-atom simulations that give a stable atom structure for infinite time and provide many insights into electron trajectories and atom properties like energies and radius of the atom. The effect of protons and neutrons movement in the nucleus on electron trajectories was captured in SPD. Hydrogen and Carbon atom SPD were reported in this study. The Pilot-wave theory was implemented in this study to simulate the wave nature of subatomic particles in an atom. Electrons motion was guided by the interference pattern produced by the electron’s aether medium waves in SPD. SPD was implemented on molecules, resulting in a stable molecular structure giving insights into electron trajectories shared by multiple atoms.

In this work, we propose a variation of the cellular particle swarm optimization algorithm with differential evolution hybridization (CPSO-DE), to include constrained optimization in it, named Ts-CPD. It is implemented as a kernel of electronic design automation (EDA) tool capable of sizing circuit components considering a single-objective design with restrictions and constraints. The aim is to improve the optimization solutions in the sizing of analog circuits. To evaluate our proposal’s performance, we present the design of three analog circuits: a deferential amplifier, a two-stage operational amplifier, and a folded cascode operational transconductance amplifier. Numerical simulation results indicate that Ts-CPD can find better solutions, in terms of the design objective and the accomplishment of constraints, than those reported in previous works.

Magnetic Particle Imaging (MPI) is a new imaging modality that sensitively and specifically detects superparamagnetic iron oxide nanoparticles (SPIONs). Many labs have been developing cellular magnetic resonance imaging (MRI) tools using both SPIONs and fluorine-19 (19F)-based contrast agents for numerous important applications, including tracking of immune and stem cells used for cellular therapies. SPION-based MRI cell tracking has very high sensitivity, but low specificity. SPIONs produce negative contrast in MRI, or signal voids. SPIO is not directly detected by MRI, but indirectly through its relaxation effects on protons, therefore, it is not possible to reliably quantify the local tissue concentration of SPION particles and cell number cannot be determined. 19F based cell tracking uses perfluorocarbons (PFC) to label cells. The number of 19F atoms can be directly measured from 19F MR images and related to cell number. 19F MRI has high specificity, but low sensitivity. MPI cell tracking displays great potential for overcoming the challenges of MRI-based cell tracking allowing for both high cellular sensitivity and high specificity and quantification of SPIO labeled cell number. In this paper we describe nanoparticle and MPI system factors that influence MPI sensitivity and resolution, quantification methods and give our perspective on testing and applying MPI for cell tracking.

Measurements of optical properties have been used for decades to study particle distributions in the ocean. They have been found useful to constrain suspended mass concentration as well as particle-related properties such as size, composition, packing (particle porosity or density) and settling velocity. Optical properties, however, provide measurements that are biased, as certain particles (based on size, composition, shape or packing) contribute to a specific property more than others. Here we study this issue both theoretically as well as by contrasting different optical properties collected simultaneously in a bottom boundary layer, to highlight the utility of such measurements as well as the biases we are likely to encounter using different optical properties to study suspended particles. In particular, we investigate the possibility to infer settling velocity from vertical profiles of optical measurements, finding that the effects of aggregation dynamics can seldom be ignored.

This study aims to examine the non-volatile Particle Number (PN) emissions of a retrofitted Heavy-Duty Spark-Ignition (HD-SI) engine powered by liquefied petroleum gas (LPG) under both steady-state and transient conditions. The engine was tested under seven steady-state operating points to investigate the PN behavior and Particle Size Distribution (PSD) upstream and down-stream of the Three-Way Catalyst (TWC). This analysis intends to assess the impact of including particles with diameters ranging from 10 nm to 23 nm in the total particle count, a consideration for future regulations. The study employed the World Harmonized Transient Cycle (WHTC) for transient conditions to encompass the same engine working region used in the steady-state analysis. A Dekati FPS-4000 diluted the exhaust sample to measure the PSD and PN for particle diameters between 5.6 nm and 560 nm using the TSI-Engine Exhaust Particle Sizer (EEPS) 3090. Findings indicate that PN levels tend to increase downstream of the TWC under steady-state conditions in operating points with low exhaust gas temperatures and flows (equal to or less than 500 °C and 120 kg/h). Furthermore, the inclusion of particles with diameters between 10 nm and 23 nm leads to an increase in PN emissions by 17.70% to 40.84% under steady conditions and by an average of 40.06% under transient conditions, compared to measurements that only consider particles larger than 23 nm. Notably, in transient conditions, most PN emissions occur during the final 600 seconds of the cycle, linked to the most intense phase of the WHTC.

The production of anatase titanium dioxide particles plays a crucial role in the sulfate process used for manufacturing white pigment. A key factor in improving the quality of white pigments is enhancing the thermal mixing process within the precipitation tank. This improvement ensures the uniform dispersion of seed particles instead of their agglomeration, leading to the formation of particles with uniform sizes. The objective of this study was to enhance three-phase CFD simulations involving the mixing process of H₂SO₄ solution, steam as a gas phase, and solid seed particles. By analyzing the trajectories of the seed particles using CFD, the optimal injection position for the seed particles within the mixing process was determined. Subsequently, lab scale test and real field test were conducted based on the insights gained from the CFD simulations. The particle size distribution of two different types of seed inlets was analyzed and compared using Transmission Electron Microscopy (TEM). The findings of this study demonstrate that the developed multi-phase CFD simulation can be effectively utilized to enhance the precipitation process for the production of anatase titanium dioxide particles.

The shift towards autonomous excavators in construction and mining is a significant leap towards enhancing operational efficiency and ensuring worker safety. However, it presents challenges such as the need for sophisticated decision-making and environmental perception due to complex terrains and diverse conditions. Our study introduces the E-GTN framework, a novel approach tailored for autonomous excavation that leverages advanced multisensor fusion and a custom-designed convolutional neural network to address these challenges. Results demonstrate that GridNet effectively processes grid data, enabling the reinforcement learning algorithm to make informed decisions, thereby ensuring efficient and intelligent autonomous excavator performance. The study concludes that the E-GTN framework offers a robust solution for the challenges in unmanned excavator operations, providing a valuable platform for future advancements in the field.

Evolutionary algorithms and swarm intelligence algorithms find applicability in reinforcement learning of neural networks due to their independence from gradient-based methods. To achieve successful training of neural networks using these algorithms, careful considerations must be made to select appropriate algorithms due to the availability of various algorithmic variations. In Part1, the author previously reported experimental evaluations on Evolution Strategy　for reinforcement learning of binary neural networks, utilizing the Pendulum control task. This article constitutes Part2 of the series of comparative research. In this study, Genetic Algorithm is adopted as another evolutionary algorithm. Wilcoxon signed rank test revealed that there was no statistically significant difference between the fitness scores obtained using GA and those using ES. However, the p-value 0.11 indicated that GA worked better than ES on this training task. As the values of binary weights, {-1, 1} were significantly superior to {0, 1} (p < .01). The motion of the pendulum controlled by the binary MLP after the training showed that the binary MLP successfully swung the pendulum swiftly into an inverted position and maintained its stability after inversion.

Stagnation at local optima represents a significant challenge in bio–inspired optimization algorithms, often leading to suboptimal solutions. This paper addresses this issue by proposing a hybrid model that combines the Orca Predator Algorithm with Deep Q–Learning. Orca Predator Algorithm is an optimization technique that mimics the hunting behavior of orcas. It solves complex optimization problems by exploring and exploiting search spaces efficiently. Deep Q–Learning is a reinforcement learning technique that combines Q–Learning with deep neural networks. This integration aims to turn the stagnation problem into an opportunity for more focused and effective exploitation, enhancing the optimization technique’s performance and accuracy. The proposed hybrid model leverages the biomimetic strengths of Orca Predator Algorithm to identify promising regions nearby in the search space, complemented by the fine–tuning capabilities of Deep Q–Learning to navigate these areas precisely. The practical application of this approach is evaluated using the high–dimensional Heartbeat Categorization Dataset, focusing on the feature selection problem. This dataset, comprising complex electrocardiogram signals, provided a robust platform for testing the feature selection capabilities of our hybrid model. Our experimental results are encouraging, showcasing the hybrid strategy capability to identify relevant features without significantly compromising the performance metrics of machine learning models. This analysis was performed by comparing the improved method of Orca Predator Algorithm against its native version and a set of state–of–the–art algorithms.

Thin people who undergo implant insertion into the chest wall face a high risk of implant exposure to the external environment. Five months after an 8-year-old boy underwent implantable cardioverter-defibrillator (ICD) implantation in a subcutaneous pocket in the left anterolateral chest wall to manage long QT syndrome, ICD replacement became necessary owing to exposure risk from distal and lateral thinning of the ICD pocket. Pocket rupture and exposure would increase the risk of infection; therefore, we performed ICD removal and primary pocket closure. Two weeks later, a new suprafascial pocket was created, acellular dermal matrix (ADM) was attached to the inner wall to prevent ICD protrusion, and a new ICD was inserted. One year postoperatively, the ADM was engrafted, and no complications were observed. A thin subcutaneous layer increases the risk of ICD implantation complications. Inner wall strengthening with ADM can help prevent pocket rupture.

Evolutionary algorithms find applicability in reinforcement learning of neural networks due to their independence from gradient-based methods. To achieve successful training of neural networks using evolutionary algorithms, careful considerations must be made to select appropriate algorithms due to the availability of various algorithmic variations. In Part1 and Part2, the author previously reported experimental evaluations on Genetic Algorithm and Evolution Strategy for reinforcement learning of neural networks, utilizing the Acrobot control task. This article constitutes Part3 of the series of comparative research. In this study, Differential Evolution is adopted as the third instance of major evolutionary algorithms. The experimental results show a statistically significant superiority of DE over both GA and ES (p < .01). In addition, DE exhibits its robustness to variations in hyperparameter configurations (the number of offsprings and generations). In the previous experiments, both ES and GA showed significant performance differences depending on the configurations, whereas in the experiment reported in this article, such differences are not detected for DE.

This paper is primarily focused on the robust control of an inverted pendulum system based on the policy iteration in reinforcement learning. First, a mathematical model of the single inverted pendulum system is established through a force analysis of the pendulum and trolley. Second, based on the theory of robust optimal control, the robust control of the uncertain linear inverted pendulum system is transformed into an optimal control problem with an appropriate performance index. Moreover, for the uncertain linear and nonlinear systems, two reinforcement-learning control algorithms are proposed using the policy iteration method. Finally, two numerical examples are provided to validate the reinforcement learning algorithms for the robust control of the inverted pendulum systems.

Evolutionary algorithms find applicability in reinforcement learning of neural networks due to their independence from gradient-based methods. To achieve successful training of neural networks using evolutionary algorithms, careful considerations must be made to select appropriate algorithms due to the availability of various algorithmic variations. The author previously reported experimental evaluations on Evolution Strategy for reinforcement learning of neural networks, utilizing the Acrobot control task. In this study, Genetic Algorithm is adopted as another instance of major evolutionary algorithms. Experimental results demonstrate that there was no statistically significant difference between the experimental performances of GA and ES, but the priority of generations and offsprings was different; GA performed better with a greater number of generations while ES performed better with a greater number of offsprings. Eight hidden units were the best among four variations (4, 8, 16 or 32 units), which aligns with previous study using ES.

For Synthetic Aperture Radar (SAR) image registration, successive processes following feature extraction are required by both the traditional feature-based method and the deep learning method. Among these processes, the feature matching process—whose time and space complexity are related to the number of feature points extracted from sensed and reference images, as well as the dimension of feature descriptors—proves to be particularly time-consuming. Additionally, the successive processes introduce data sharing and memory occupancy issues, requiring an elaborate design to prevent memory leaks. To address these challenges, this paper introduces the OptionEM-based reinforcement learning framework to achieve end-to-end SAR image registration. This framework outputs registered images directly without requiring feature matching and calculation of the transformation matrix, leading to significant processing time savings. The Transformer architecture is employed to learn image features, while a correlation network is introduced to learn the correlation and transformation matrix between image pairs. Reinforcement learning, as a decision process, can dynamically correct errors, making it more efficient and robust compared to supervised learning mechanisms like deep learning. We present a hierarchical reinforcement learning framework combined with episodic memory to mitigate the inherent problem of invalid exploration in generalized reinforcement learning algorithms. This approach effectively combines coarse and fine registration, further enhancing training efficiency. Experiments conducted on three sets of SAR images, acquired by TerraSAR-X and Sentinel-1A, demonstrate that the proposed method’s average runtime is sub-second, achieving subpixel registration accuracy.

Evolutionary algorithms find applicability in the reinforcement learning of neural networks due to their independence from gradient-based methods. To achieve successful training of neural networks using evolutionary algorithms, careful considerations must be made to select appropriate algorithms due to the availability of various algorithmic variations. The author previously reported experimental evaluations on Evolution Strategy for reinforcement learning of neural networks, utilizing the pendulum control task. In this study, the Acrobot control task is adopted as another task. Experimental results demonstrate that ES successfully trained a Multi-Layer Perceptron to achieve a remarkable height of 99.85% concerning the maximum height. However, the trained MLP failed to maintain the chain end in an upright position throughout an episode. In this study, it was observed that employing 8 hidden units in the neural network yielded better results with statistical significance compared to using 4, 16, or 32 hidden units. Furthermore, the findings indicate that a larger population size in ES led to a more extensive exploration of potential solutions over a greater number of generations, which aligns with the previous study.

There is only a very short reaction time for people to find the best way out of a building in a fire outbreak. Software applications can be used to assist the rapid evacuation of victims; however, this is an arduous task, which requires an understanding of advanced technologies. Since well-known pathway algorithms (such as, Dijkstra, Bellman-Ford and A*) can lead to serious problems over performance, we decided to make use of deep reinforcement learning techniques. A wide range of strategies including a random initialization of replay buffer and transfer learning were assessed in three projects involving schools of different sizes. The results showed the proposal was viable and that in most cases the performance of transfer learning was superior. In addition, the study raised challenges that had to be faced in the future.

We propose a deep reinforcement learning based manipulator path tracking method to solve the computationally difficult and non-unique problem of manipulator path tracking methods based on inverse kinematics. By transforming the path tracking task into a sequence decision problem, our method adopts an end-to-end learning method for closed-loop control and avoids the process of finding the inverse solution. We first explored the feasibility of the deep reinforcement learning method in the path tracking of the manipulator. After verifying the feasibility, the path tracking of the multi-degree-of-freedom(multi-DOF) manipulator was realized by combining the maximum entropy deep reinforcement learning algorithm. The experimental results show that our method has a good effect on the path tracking of the manipulator, which not only avoids the process of finding the inverse kinematics solution, but also requires no dynamic model. Therefore, we believe that our method has great significance in the study of manipulator path tracking.

We present a hierarchical reinforcement learning (RL) architecture that employs various low-level agents to act in the trading environment, i.e. the market. The highest level agent selects among a group of specialised agents, and then the selected agent decides when to sell or buy a single asset for some period. This period can be variable according to a termination function. We hypothesized that due to different market regimes, more than one single agent is needed when trying to learn from such heterogeneous data, and instead, multiple agents will perform better, with each one specialising in a subset of the data. We use $k-means$ clustering to partition the data and train each agent with a different cluster. Partitioning the input data also helps model-based RL (MBRL), where models can be heterogeneous. We also add two simple decision-making models to the set of low-level agents, diversifying the pool of available agents and thus increasing overall behaviour flexibility. We perform multiple experiments showing the strengths of a hierarchical approach and test various prediction models at both levels. We also use a risk-based reward at the high level, which transforms the overall problem into a risk-return optimization. This type of reward shows a significant reduction in risk while minimally reducing profits. Overall, the hierarchical approach shows significant promise, especially when the pool of low-level agents is highly diverse.

Sparse reward long horizon task is a major challenge for deep reinforcement learning algorithm. One of the key barriers is data-inefficiency. Even in the simulation environment, it usually takes weeks to training the agent. In this study, a data-efficiency training framework is proposed, where a curriculum learning is design for the agent in the simulation scenario. Different distributions of the initial state are set for the agent to get more informative reward during the whole training process. A fine-tuning of the parameters in the output layer of the neural network for value function is conduct to bridge the gap between sim-to-real. An experiment of UAV maneuver control is conducted in the proposed training framework to verify the method more efficient. We demonstrate that data-efficiency is different for the same data in different training stages.

The robust control of high precision electromechanical systems, such as micropositioners, is challenging in terms of the inherent high nonlinearity, the sensitivity to external interference, and the complexity of accurate identification of the model parameters. To cope with these problems, this work investigates a disturbance observer-based deep reinforcement learning control strategy to realize high robustness and precise tracking performance. Reinforcement learning has shown great potential as optimal control scheme, however, its application in micropositioning systems is still rare. Therefore, embedded with the integral differential compensator (ID), deep deterministic policy gradient (DDPG) is utilized in this work with the ability to not only decrease the state error but also improves the transient response speed. In addition, an adaptive sliding mode disturbance observer (ASMDO) is proposed to further eliminate the collective effect caused by the lumped disturbances. The sterling performance is revealed with intensive tracking simulation experiments and demonstrates the improvement in the accuracy and response time of the controller.

This last decade, the amount of data exchanged in the Internet increased by over a staggering factor of 100, and is expected to exceed well over the 500 exabytes by 2020. This phenomenon is mainly due to the evolution of high speed broadband Internet and, more specifically, the popularization and wide spread use of smartphones and associated accessible data plans. Although 4G with its long-term evolution (LTE) technology is seen as a mature technology, there is continual improvement to its radio technology and architecture such as in the scope of the LTE Advanced standard, a major enhancement of LTE. But for the long run, the next generation of telecommunication (5G) is considered and is gaining considerable momentum from both industry and researchers. In addition, with the deployment of the Internet of Things (IoT) applications, smart cities, vehicular networks, e-health systems, and Industry 4.0, a new plethora of 5G services has emerged with very diverging and technologically challenging design requirements. These include: high mobile data volume per area, high number of devices connected per area, high data rates, longer battery life for low-power devices, and reduced end-to-end latency. Several technologies are being developed to meet these new requirements. Among these we list ultra-densification, millimeter Wave usage, antennas with massive multiple-input multiple-output (MIMO), antenna beamforming to increase spacial diversity, edge/fog computing, among others. Each of these technologies brings its own design issues and challenges. For instance, ultra-densification and MIMO will increase the complexity to estimate channel condition and traditional channel state information (CSI) estimation techniques are no longer suitable due to the complexity of the new scenarios. As a result, new approaches to evaluate network condition such as by continuously collecting and monitoring key performance indicators become necessary. Timely decisions are needed to ensure the correct operation of such network. In this context, deep learning (DL) models could be seen as one of the main tools that can be used to process monitoring data and automate decisions. As these models are able to extract relevant features from raw data (images, texts, and other types of unstructured data), the integration between 5G and DL looks promising and one that requires exploring. As main contributions, this paper presents a systematic review about how DL is being applied to solve some 5G issues. We examine data from the last decade and the works that addressed diverse 5G problems, such as physical medium state estimation, network traffic prediction, user device location prediction, self network management, among others. We also discuss the main research challenges when using DL models in 5G scenarios and identify several issues that deserve further consideration.

We propose a 10-Gb/s 64-quadrature amplitude modulation (QAM) signal-based Radio over Fiber (RoF) system for 50 km of standard single mode fiber length which utilizes Reinforcement Learning (RL) SARSA based decision method to indicate an effective decision which mitigates nonlinearity. By utilizing RL-SARSA algorithm, the results demonstrate that significant reduction can be obtained in terms of bit error rate.

Particles are considered temporarily identical if the interchange of these particles does not cause any observable physical effects within a time interval Δt. An indistinguishability hypothesis is proposed for such particles, suggesting that two particles (or systems) with nearly the same degree of internal freedom will actually interchange under certain circumstances, thereby affecting the subsequent dynamics. This indistinguishability implies that, theoretically, it is possible to capture an elementary particle with extraordinarily large internal freedom by trading it for an isolated composite system of ordinary matter particles with almost the same total internal freedom as the large-freedom particle. Following this, the ordinary particles will instantly disperse into the environment, while the large-freedom particle will initially be captured and then gradually delocalized into the environment, an observable phenomenon. If this hypothesis is proven true in the future, it might provide a novel strategy for detecting dark matter particles.

Hypoxia, associated with abnormal vessel growth, is a characteristic feature of many solid tumors that increases their metastatic potential and resistance to radiotherapy. Carbon ion radiation therapy, alone or in combination with other treatments, is one of the most promising treatments for hypoxic tumors because the oxygen enhancement ratio decreases with increasing particle LET. Nevertheless, the current clinical practice does not yet fully benefit from using carbon ions for tackling hypoxia. Here we give an overview of the existing experimental and clinical evidence supporting the efficacy of C-ion radiotherapy in overcoming hypoxia-induced radioresistance, followed by a discussion of the strategies proposed to enhance it, including different approaches to maximize the LET in the tumors.

Polyaniline or PANI is a conductive polymer (CPs) with flexibility in turning into conducting properties. The drawback is mainly in terms of limited mechanical properties improvement which attracted the interest in fabrication of PANI with other polymeric materials. PANI-PVL composites film was synthesized and fabricated in this work with chronological steps according to the objectives aligned in two different phases. The first phase determines the optimum parameters to synthesize nanosized PANI. The ideal initial molar ratio, r was 1, stirring speed of 600 rpm, purified by filtration, synthesis temperature of 25 oC, washed by dopant acid, acetone and distilled water. The usage of non-ionic surfactant, Triton X-100 (TX100) with 0.1 % concentration favours the formation of smaller particle size of PANI and good dispersibility compared to the usage of anionic sodium dodecyl sulphate (SDS). The synthesis of PANI in nanosized dimension has the potential to be applied for antimicrobial applications such as gloves and antimicrobial coating of frequent contact points in the near future.

We investigate the Friedmann–Lemaitre–Robertson–Walker cosmological models within the framework of Rastall gravity incorporating particle creation. The modified field equations for Rastall gravity are derived and exact solutions are obtained under various choices of scale factors. The qualitative behaviour of our solutions depends on the Rastall coupling parameter ψ=kλ. Following the literature, we have restricted the Rastall coupling parameter ψ(k=1) to the range −0.0001 < ψ < 0.0007 at 68% CL from CMB+BAO data. Further, we have discussed the distinct physical behaviour of the derived models in detail.

We define a spinor-Minkowski metric for SL(4,C). It is not a trivial generalization of the SL(2,C) metric and it involves the Minkowskian one. We define 4x4 version of the Pauli matrices and eight 4-component associated generalized eigenvectors that can be regarded as undotted covariant spinors. The 4-component spinors can be grouped into four categories. Each category transforms in its own way. The outer products of pairwise combinations of 4-component spinors can be associated with 4-vectors. Including the dotted covariant, undotted and dotted contravariant forms totally we have sixteen pairs of spinors. Eight of them live in the conjugate space which has no countepart in SL(2,C).

During the last decade, new unusual physical phenomena have been discovered in studying the optics of dielectric mesoscale particles of an arbitrary three-dimensional shape with the Mie size parameter near 10 (q ~ 10). The paper provides a brief overview of these phenomena from optics to terahertz, plasmonic and acoustic ranges. The different particle configurations (isolated, regular or Janus) are discussed, and the possible applications of such mesoscale structures are briefly reviewed herein in relation to the field enhancement, nanoparticle manipulation and super-resolution imaging. The number of interesting applications indicates to a new promising scientific direction emerged in optics, terahertz and acoustic ranges, and plasmonics. In this paper we present the authors' view of these problems.

In this letter, instead of choosing the Einstein Rosen bridge between two black holes as in ER=EPR, we consider a wormhole between a black hole and a closed edge of the wormhole. We assume that information in a black hole travels through a wormhole, turns to mass (dark matter) in the closed region. This study is in contradiction with the existence of the white hole in our Universe. We replace the notion of the white hole with the massive closed region. We prove the metric of the closed region by the Hopf fibration, this new metric generalizes the $AdS_{5}$\ metric.

In this paper, we present a technique to unify the Reissner–Nordstr¨om metric and the Kerr–Newman metric. We construct a specifific model and calculate the entanglement entropy of black horizon. We are interested in the entangled particle and antiparticle spinning on the black hole horizon. The two Reissner-Nordstr¨om horizons r±, are the results of the rotation of several entangled particle-antiparticle on the real horizon. The energy absorbed by a black hole is transformed into a kinetic energy of the entangled particle-antiparticles. This study provides a new type of black hole metric. We show that the rotation of an entangled system of a particle and an antiparticle can create a extremal black hole. We also explore some of the implications of this point of view for the black hole entanglement.

This study analyzes the correlation between the lifetime and the rest energy of the unstable particle states with a lifetime greater than the zeptosecond (10⁻²¹ s), using data available from the Particle Data Group. This set of states seems to be divided into three groups, in each of which the two quantities can be correlated through a remarkably accurate power law. Although this fact does not represent anything new compared to the predictions of the Standard Model, it nevertheless reveals an unexpected order structure in the set of particle decays, emerging from such predictions.

Both macro and microprojectiles (e.g., interplanetary, interstellar and even intergalactic material) are seen as important vehicles for the exchange of potential (bio)material within our solar system as well as between stellar systems in our Galaxy. Accordingly, this requires estimates of the impact probabilities for different source populations of projectiles, including for intergalactic meteor particles which have received relatively little attention since considered as rare events (discrete occurrences that are statistically improbable due to their very infrequent appearance). We employ the simple but comprehensive model of intergalactic microprojectile capture by the gravity of exoplanets which enables us to estimate the map of collisional probabilities for an available sample of exoplanets in habitable zones around host stars. The model includes a dynamical description of the capture adopted from Mautner model of interstellar exchange of microparticles and changed for our purposes. We use statistical and information metrics to calculate probability map of intergalactic meteorite particle capture. Moreover, by calculating the entropy index map we estimate the concentration of these rare events. We further adopted a model from immigration theory, to show that the time dependent distribution of single molecule immigration of material indicates high survivability of the immigrated material taking into account birth and death processes on our planet. At present immigration of material can not be observationally constrained but it seems reasonable to think that it will be possible in the near future, and to use it along other proposed parameters for life sustainability on some planet.

Current numerical methods for simulating biophysical processes in aquatic environments are typically constructed in a grid-based Eulerian framework or as an individual-based model in a particle-based Lagrangian framework. Often, the biogeochemical processes and physical (hydrodynamic) processes occur at different time and space scales, and changes in biological processes do not affect the hydrodynamic conditions. Therefore, it is possible to develop an alternative strategy to grid-based approaches for linking hydrodynamic and biogeochemical models that can significantly improve computational efficiency for this type of linked biophysical model. In this work, we utilize a new technique which links hydrodynamic effects and biological processes through a property-carrying particle model (PCPM) in a Lagrangian/Eulerian framework. The model is tested in idealized cases and its utility is demonstrated in a practical application to Sandusky Bay. Results show the integration of Lagrangian and Eulerian approaches allows for a natural coupling of mass transport (represented by particle movements and random walk) and biological processes in water columns which is described by a nutrient-phytoplankton-zooplankton-detritus (NPZD) biological model. This method is far more efficient than traditional tracer based Eulerian biophysical models for 3-D simulation, particularly for a large domain and/or ensemble simulations.

Transport of hydrophobic drugs in the human body exhibits complications due to the low solubility of these compounds. With the purpose of enhancing the bioavailability and biodistribution of such drugs, recent studies have reported the use of amphiphilic molecules, such as phospholipids, for synthesis of nanoparticles or nanocapsules. Given that phospholipids can self–assemble in liposomes or micellar structures, they are ideal candidates to function as vehicles of hydrophobic molecules. In this work, we report mesoscopic simulations of nanoliposomes, constituted by lecithin and coated with a shell of chitosan. The stability of such structure and the efficiency of encapsulation of capsaicin, as well as the internal and superficial distribution of capsaicin and chitosan inside the nanoliposome were analyzed. The characterization of the system was carried out through density maps and the potentials of mean force for the lecithin–capsaicin, lecithin–chitosan and capsaicin–chitosan interactions. The results of these simulations show that chitosan is deposited on the surface of the nanoliposome, as has been reported in some experimental works. It was also observed that a nanoliposome of approximately 18 nm in diameter is stable during the simulation. The deposition behavior was found to be influenced by pattern of N-acetylation of chitosan.

In this paper we present the exploration of indoor positioning technologies for UAVs, and navigation techniques for path planning and obstacle avoidance. The objective is to perform warehouse inventory tasks using a drone to search for barcodes or markers to identify the objects. For the indoor positioning techniques, we employ Visual-Inertial Odometry (VIO), Ultra-Wideband (UWB), AprilTag fiducial markers, and Simultaneous Localization and Mapping (SLAM). These algorithms encompass global positioning, local positioning, and pre-mapping positioning, comparing the merits and drawbacks of various techniques and trajectories. In UAV navigation, we combine SLAM-based RTAB-Map indoor mapping and navigation path planning of the ROS for indoor environments. This system enables precise drone positioning indoors and utilizes global and local path planners to generate flight paths that avoid dynamic, static, unknown, and known obstacles, demonstrating high practicality and feasibility. To achieve the warehouse inventory inspection, a reinforcement learning approach is proposed to recognize the markers by adjusting the UAV's viewpoint. We address several main problems in inventory management, including efficiently planning paths while ensuring a certain level of detection rate. Two reinforcement learning techniques, AC (Actor-Critic) and PPO (Proximal Policy Optimization), are implemented based on AprilTag identification. Testing is performed in both simulated and real-world environments, and the effectiveness of the proposed method is validated.Code is available at https://github.com/kellen080/Navigation and https://github.com/kellen080/Indoor\_Positioning.

The IEEE 802.11ah standard is introduced to address the growing scale of Internet of Things (IoT) applications. To reduce contention and enhance energy efficiency within the system, the restricted access window (RAW) mechanism is introduced at the medium access control (MAC) layer to manage the significant number of stations accessing the network. However, the RAW parameters need to be appropriately determined to achieve optimal network performance. In this paper, we optimize the configuration of RAW parameters in the IEEE 802.11ah-based IoT network. We analyze and propose an RAW parameters optimization problem with the objective of improving network throughput and formulate it as a Markov decision process. To cope with the complex and dynamic network conditions, we propose a proximal policy optimization (PPO)-based deep reinforcement learning (DRL) algorithm to find the optimal RAW parameters. We construct network environments with periodic and random traffic in the NS-3 simulator to validate the performance of the proposed PPO-based RAW parameters optimization algorithm. Simulation results reveal that using the PPO-based DRL algorithm, optimal RAW parameters can be obtained under different network conditions, thereby significantly improving network throughput.

Multimodal remote sensing data classification can enhance the model’s ability to distinguish land features through multimodal data fusion. In this context, how to help models understand the relationship between multimodal data and target tasks is the focus of researchers. Inspired by human feedback learning mechanisms, causal reasoning mechanisms, and knowledge induction mechanisms, this paper integrates causal learning, reinforcement learning, and meta learning into a unified remote sensing data classification framework and proposed the causal meta-reinforcement learning (CMRL). First, based on feedback learning mechanisms, we have overcome the limitations of traditional implicit optimization of fusion features and customized a reinforcement learning environment for multimodal remote sensing data classification tasks. Through feedback interactive learning between agents and the environment, we help them understand the complex relationships between multimodal data and labels, thereby achieving full mining of multimodal complementary information. Second, based on the causal inference mechanism we designed causal distribution prediction action, classification reward, and causal intervention reward, capturing pure causal factors in multimodal data and cutting off false statistical associations between non-causal factors and class labels. Finally, based on the knowledge induction mechanism, we designed a bi-layer optimization mechanism based on meta-learning. By constructing a meta training task and meta validation task simulation model in the generalization scenario of unseen data, we helped the model induce cross-task shared knowledge, thereby improving its generalization ability for unseen multimodal data. The experimental results on multiple sets of multimodal datasets show that the proposed method achieved state-of-the-art performance in multimodal remote sensing data classification tasks.

The author previously reported an experimental result of evolutionary reinforcement learning of neural network controllers for the pendulum task. In the previous work, a conventional multilayer perceptron was employed in which connection weights were real numbers. In this study, the author experimentally applies an evolutionary algorithm to the reinforcement training of binary neural networks. In both studies, the same task and the same evolutionary algorithm are utilized, i.e. the pendulum control problem and Evolution Strategy respectively. The only differences lie in the memory size per connection weight and the model size of the neural network. The findings from this study are (1) the performance of the binary MLP with 32 hidden units was inferior to that of the real-valued MLP with 16 hidden units; however, this difference was not statistically significant (p >.05); (2) the trained binary MLP successfully swung the pendulum swiftly into an inverted position and maintained its stability after inversion, as the real-valued MLP had done; and (3) the memory size required to record the binary MLP with 32 hidden units is 3.1% or 6.2% of the memory size required to record the real-valued MLP with 16 hidden units.

Modern power system integrates more and more new energy and use a large number of power electronic equipment. This makes it face more challenges in online optimization and real-time control. Deep reinforcement learning(DRL) has the ability of processing big data and high-dimensional features, as well as the ability of independently learning and optimizing decision-making in complex environments. In this paper, we explore DRL based online combination optimization method of grid section for large complex power system. In our method, to improve the convergence speed of the model, we propose to discretize the output action of the unit and simplify the action space. We also design a reinforcement learning loss function with strong constraints to further improve the convergence speed of the model and facilitate the algorithm to obtain the stable solution. Moreover, to avoid the local optimal solution problem caused by the discretization of the output action, we propose to use the annealing optimization algorithm to make the granularity of the unit output finer. We verify our method on IEEE 118-bus system. The experimental results show that our model has fast convergence speed and better performance, and can obtain stable solutions.

A detailed literature review is performed in this study to address solutions for the full-body design and control of an aerial manipulator. Deep Reinforcement Learning methods are growing to be utilized recently to cope with various uncertainties. The pros and cons of these theories will be explained as well as introducing the advantages of Fuzzy Reinforcement Learning methods. State-of-the-Art, possible challenges, potential approaches, and a summary of desired precision devices are discussed in this study.

For a mobile robot, navigation in a densely crowded space can be a challenging and sometimes impossible task, especially with traditional techniques. In this paper, we present a framework to train neural controllers for differential drive mobile robots which must safely navigate a crowded environment while trying to reach a target location. To learn the robot’s policy, we train a convolutional neural network using two reinforcement learning algorithms, Deep Q-Networks (DQN) and Asynchronous Advantage Actor Critic (A3C), and develop a training pipeline that allows to scale the process to several compute nodes. We show that the asynchronous training procedure in A3C can be leveraged to quickly train neural controllers and test them on a real robot in a crowded environment.

In this review we summarize existing trends of flow control used to improve the aerodynamic efficiency of wings. We first discuss active methods to control turbulence, starting with flat-plate geometries and building towards the more complicated flow around wings. Then, we discuss active approaches to control separation, a crucial aspect towards achieving high aerodynamic efficiency. Furthermore, we highlight methods relying on turbulence simulation, and discuss various levels of modelling. Finally, we thoroughly revise data-driven methods, their application to flow control, and focus on deep reinforcement learning (DRL). We conclude that this methodology has the potential to discover novel control strategies in complex turbulent flows of aerodynamic relevance.

In the past two decades, metaheuristic optimization algorithms (MOAs) have been increasingly popular, particularly in logistic, science, and engineering problems. The fundamental characteristics of such algorithms are that they are dependent on a parameter or a strategy. Some online and offline strategies are employed in order to obtain optimal configurations of the algorithms. Adaptive operator selection is one of them, and it determines whether or not to update a strategy from the strategy pool during the search process. In the filed of machine learning, Reinforcement Learning (RL) refers to goal-oriented algorithms, which learn from the environment how to achieve a goal. On MOAs, reinforcement learning has been utilised to control the operator selection process. Existing research, however, fails to show that learned information may be transferred from one problem-solving procedure to another. The primary goal of the proposed research is to determine the impact of transfer learning on RL and MOAs. As a test problem, a set union knapsack problem with 30 separate benchmark problem instances is used. The results are statistically compared in depth. The learning process, according to the findings, improved the convergence speed while significantly reducing the CPU time.

Recently, the application of deep reinforcement learning in recommender system is flourishing and stands out by overcoming drawbacks of traditional methods and achieving high recommendation quality. The dynamics, long-term returns and sparse data issues in recommender system have been effectively solved. But the application of deep reinforcement learning brings problems of interpretability, overfitting, complex reward function design, and user cold start. This paper proposed a tag-aware recommender system based on deep reinforcement learning without complex function design, taking advantage of tags to make up for the interpretability problems existing in recommender system. Our experiment is carried out on MovieLens dataset. The result shows that, DRL based recommender system is superior than traditional algorithms in minimum error and the application of tags has little effect on accuracy when making up for interpretability. In addition, DRL based recommender system has excellent performance on user cold start problems.

Climate change and development have increased urban flooding, requiring modernization of stormwater infrastructure. Retrofitting standard passive systems with controllable valves/pumps is promising, but requires real-time control (RTC). One method of automating RTC is reinforcement learning (RL), a general technique for sequential optimization and control in uncertain environments. The notion is that an RL algorithm can use inputs of real-time flood data and rainfall forecasts to learn a policy for controlling the stormwater infrastructure to minimize measures of flooding. In real-world conditions, rainfall forecasts and other state information, are subject to noise and uncertainty. To account for these characteristics of the problem data, we implemented Deep Deterministic Policy Gradient (DDPG), an RL algorithm that is distinguished by its capability to handle noise in the input data. DDPG implementations were trained and tested against a passive flood control policy. Three primary cases were studied: (i) perfect data, (ii) imperfect rainfall forecasts, and (iii) imperfect water level and forecast data. Rainfall episodes (100) that caused flooding in the passive system were selected from 10 years of observations in Norfolk, Virginia, USA; 85 randomly selected episodes were used for training and the remaining 15 unseen episodes served as test cases. Compared to the passive system, all RL implementations reduced flooding volume by 70.5% on average, and performed within a range of 5%. This suggests that DDPG is robust to noisy input data, which is essential knowledge to advance the real-world applicability of RL for stormwater RTC.

Joints between walls are very important for structural analysis of each masonry building at the global and local level. This issue was often neglected in case of traditional joints and relatively squat walls. Nowadays the issue of wall joints is becoming particularly important due to the continuous strive for simplifying structures, introducing new technologies and materials. Eurocode 6 and other standards (USA, Canadian, Chinese, and Japanese) recommend inspecting joints between walls, but no detail procedures have been specified. This paper presents our own tests on joints between walls made of autoclaved aerated concrete (AAC) masonry units. Tests included reference models composed of two wall panels joined perpendicularly with a masonry bond (6 models), traditional steel and modified connectors (12 models). A shape and size of test models and structure of a test stand were determined on the basis of the analysis of the current knowledge, pilot studies and numerical analyses of FEM. The analysis referred to the morphology and failure mechanism of models. Load-displacement relationships for different types of joints were compared and obtained results were referred to results for reference models. A mechanism of cracking and failure was found to vary, and clear differences in behaviour and load capacity of each type of joints were observed. Individual working phases of joints were determined and defined, and the empirical approach was suggested to determine forces and displacement of wall joints.

This paper presents a study on the thermal efficiency improvement of the user equipment room in the district heating system based on reinforcement learning [1], and suggests a general method of constructing a learning network(DQN)[2] using deep Q learning[3], which is a reinforcement learning algorithm that does not specify a model. In addition, we introduce the big data platform system and the integrated heat management system for the energy field in the massive data processing from the IoT sensor installed in large number of thermal energy control facilities.

In this study, we examine the relationship of bank level lending and borrowing decisions and the risk preferences on the dynamics of the interbank lending We develop an agent-based model that incorporates individual bank decisions using the temporal difference reinforcement learning algorithm with empirical data of 6600 S. banks. The model can successfully replicate the key characteristics of interbank lending and borrowing relationships documented in the recent literatur A key finding of this study is that risk preferences at individual bank level can lead to unique interbank market structures which are suggestive of the capacity that the market responds to surprising