The term artificial intelligence refers to the field of computer science that deals with the design and implementation of computer systems that mimic elements of human behavior that imply basic intelligence: learning, adaptability, inference, contextual understanding, and problem solving among others. Artificial intelligence is a point of intersection between multiple sciences such as computer science, psychology, philosophy, neurology, linguistics and engineering science, with the aim of synthesizing intelligent behavior, with elements of reasoning, learning and adaptation to the environment, while it is usually applied to specially made machines or computers.

The Mixture of Experts (MoE) architecture has evolved as a powerful and versatile approach for improving the performance and efficiency of deep learning models. This survey aims to provide the fundamental principles of MoE in details, a paradigm that harnesses the collective power of multiple specialized "expert" models working in concert to tackle complex tasks.Our exploration is a thorough analysis of the core components that constitute the MoE framework. We begin by dissecting the routing mechanism, a crucial element responsible for dynamically assigning input data to the most appropriate expert models. This routing process is pivotal in ensuring that each expert's specialized knowledge is optimally utilized. The survey places significant emphasis on expert specialization, a key feature that sets MoE apart from traditional architectures. We examine various strategies for developing and training specialized experts, exploring how this specialization enables MoE models to effectively handle diverse and multifaceted problems.Load balancing, another critical aspect of MoE systems, receives thorough attention. We discuss techniques for efficiently distributing computational resources among experts, ensuring optimal model performance while managing hardware constraints. This section provides insights into the delicate balance between model capacity and computational efficiency. Furthermore, we provide an in-depth discussion of the expert models themselves, examining their architectural designs, training methodologies, and how they interact within the larger MoE framework. This includes an analysis of different types of experts, from simple neural networks to more complex, task-specific architectures. The survey also navigates through diverse research avenues within the MoE landscape, highlighting recent advancements and innovative applications across various domains of machine learning. We pay particular attention to the burgeoning use of MoE in two rapidly evolving fields: computer vision and large language model (LLM) scaling. By providing this comprehensive overview, our survey aims to offer researchers and practitioners a deep understanding of MoE's capabilities, current applications, and potential future directions in the ever-evolving landscape of deep learning.The Mixture of Experts (MoE) architecture has evolved as a powerful and versatile approach for improving the performance and efficiency of deep learning models. This survey aims to provide the fundamental principles of MoE in details, a paradigm that harnesses the collective power of multiple specialized "expert" models working in concert to tackle complex tasks.Our exploration is a thorough analysis of the core components that constitute the MoE framework. We begin by dissecting the routing mechanism, a crucial element responsible for dynamically assigning input data to the most appropriate expert models. This routing process is pivotal in ensuring that each expert's specialized knowledge is optimally utilized. The survey places significant emphasis on expert specialization, a key feature that sets MoE apart from traditional architectures. We examine various strategies for developing and training specialized experts, exploring how this specialization enables MoE models to effectively handle diverse and multifaceted problems.Load balancing, another critical aspect of MoE systems, receives thorough attention. We discuss techniques for efficiently distributing computational resources among experts, ensuring optimal model performance while managing hardware constraints. This section provides insights into the delicate balance between model capacity and computational efficiency. Furthermore, we provide an in-depth discussion of the expert models themselves, examining their architectural designs, training methodologies, and how they interact within the larger MoE framework. This includes an analysis of different types of experts, from simple neural networks to more complex, task-specific architectures. The survey also navigates through diverse research avenues within the MoE landscape, highlighting recent advancements and innovative applications across various domains of machine learning. We pay particular attention to the burgeoning use of MoE in two rapidly evolving fields: computer vision and large language model (LLM) scaling. By providing this comprehensive overview, our survey aims to offer researchers and practitioners a deep understanding of MoE's capabilities, current applications, and potential future directions in the ever-evolving landscape of deep learning.

Livestock marketplaces consist of infrastructure and rules through which information is exchanged and transactions are completed. Livestock trading incurs significant transaction costs, beyond just commission fees paid to marketplace owners (e.g., se-lecting, loading and transporting animals). In addition, animals being transported, held and transacted in a market setting increases the likelihood of injury or disease. The Covid-19 pandemic contributed to large-scale changes across typically physical marketplaces as stay-at-home orders and travel restrictions were invoked. Livestock marketplaces pivoted online rapidly, facilitated by the deployment of video feeds, al-lowing buyers to view/purchase animals via mobile phone. This shift offered the promise of mitigating many of the transaction costs associated with a fully physical marketplace. This study evaluated how computer vision (CV) approaches can con-tribute to the evolution of livestock marketplaces. It used livestock video data for 240 cattle (bulls) weighing 320-740 Kg from 16 different breeds to elicit additional trade-relevant data (livestock weight) and can be considered as an intermediary step to better understand how video data can be used to further digitalize livestock trade, price formation and transaction completion via technology solutions. The dataset comes from three sales held on different dates in a single mart location, which allows a consistent camera setup throughout. CV technologies were employed to solve the problem of cattle weight estimation and a distinctive achievement lies in curating a dataset cap-turing cattle in motion within the context of a cattle auction thereby injecting realism and complexity into the CV weight estimation task. Employing advanced object detection algorithms, the study created an intricate data pipeline to generate an appropriate modelling dataset and devised new multi-channel architectures for deeper analysis of diverse image inputs. The significance of sizable datasets and the challenge of man-aging inherent input variability for accurate CV regression is highlighted. Top-down view predictions demonstrate comparatively better performance, although failing to achieve benchmark precision. Challenges arise from camera angle variations, roaming behaviors, and breed diversity. Likewise, side-view predictions face challenges due to increased variability and lack of standardization. Despite this, these predictions offer insights into the value of incorporating Region of Interest (ROI) coordinates and data augmentation for enhanced performance. The study concludes with a multi-input ar-chitecture combining top-down and side-view predictions, demonstrating moderate success highlighting potential synergy between both perspectives. The central con-clusion resonates strongly – using standardized and multi-view images encompassing controlled movement and consistent poses could bolster accurate CV weight estima-tion.

Regular inspections during construction work verify that the work completed is consistent with the plans and specifications and ensure that it is within the planned time and budget. This requires frequent physical site observations to independently measure and verify the completion percentage of the construction progress performed over periods of time. The current computer vision-based (CV) techniques for the measurement of as-built elements, predominantly use 3D laser scanning or 3D Photogrammetry modelling to determine the geometrical properties of as-built elements on construction sites. Both techniques require data acquisition from several positions and angles to generate sufficient information about the element’s coordinates making the deployment of these techniques on dynamic construction project sites a challenging task. In this paper, we propose a pipeline for automating the measurement of as-built components using artificial intelligence (AI) and computer vision (CV) techniques. The pipeline requires a single image obtained with a stereo-camera system to measure the size of selected objects or as-built components. We demonstrate our approach by measuring the size of concrete walls and columns. The novelty of this work is attributed to the fully automated CV-based method for measuring any given element using a single image only. The proposed solution is suitable for use in measuring the sizes of as-built components of built assets. It has the potential to be further developed and integrated with BIM models for use on construction projects for progress monitoring.

This work presents an approach for detecting mathematical formulas in scanned document images. The proposed approach is end-to-end trainable. Since many OCR engines cannot reliably work with the formulas, it is essential to isolate them to obtain the clean text for information extraction from the document. Our proposed pipeline comprises a hybrid task cascade network with deformable convolutions and a Resnext101 backbone. Both of these modifications help in better detection. We evaluate the proposed approaches on the ICDAR-2017 POD and Marmot datasets and achieve an overall accuracy of 96% for the ICDAR-2017 POD dataset. We achieve an overall reduction of error of 13%. Furthermore, the results on Marmot datasets are improved for the isolated and embedded formulas. We achieved an accuracy of 98.78% for the isolated formula and 90.21% overall accuracy for embedded formulas. Consequently, it results in an error reduction rate of 43% for isolated and 17.9% for embedded formulas.

In recent times, using image-based deep learning techniques has been very useful for classifying fruits. However, mango is a particular challenge because its shelf life is directly related to the degree of maturity at the time of harvest, which is determined mainly through the visual assessment of the operators when sorting them, doing it incorrectly, coupled with the blows and compressions exposed during storage and transport, trigger the deterioration in their quality, hence the importance of this operation. In this sense, the purpose of this study focuses on designing a computational model based on Deep Learning that uses a convolutional neural network (CNN) parameterized to select mangoes according to their color tone based on their maturity stage. A dataset was established from 201 labeled images, then a CNN architecture was built, obtaining a model on which 3 criteria were evaluated: the number of convolution and grouping layers, the resolution of the input images and the number of training epochs required. As a result, an input image resolution of 32 x 32 pixels was achieved, the process included a convolution layer followed by pooling that fed another layer that processed the results and ended in an output layer with 2 neurons under the One Hot Encoding representation. The model required 20 epochs to perfect its training, achieving 96.04 % accuracy in mango image selection. This study supports the potential of CNNs in quality analysis through mango color selection, even in uncertainty scenarios.

The paper presents a proposal for an in-house mathematical model of a liquid flat-plate solar collector. The proposed model is a one-dimensional distributed parameter model enabling simulations of the collector operation under arbitrarily variable boundary conditions. The model is based on the solution of energy balance equations for all components of the collector. The formulated differential equations are solved iteratively using an implicit difference scheme. In order to obtain a stable numerical solution, it is necessary to use appropriate steps of the time and spatial division. These were found by comparing the results obtained from the model with the results of the analytical solution available in the literature for the transient state, which constitutes the novelty of the present study. The accuracy of the results obtained from the model was verified experimentally by comparing the measured and the calculated history of the fluid temperature at the outlet of the collector. The calculation of the collector time constant is proposed in the paper as an example of the model practical application. The results of the time constant calculation were compared with the values obtained experimentally on the test stand. This is another novelty of the presented research. The analysed collector instantaneous efficiency was then calculated for selected outdoor conditions. The presented mathematical model can also be used to verify the correctness of the collector operation. Comparing, on an ongoing basis, the measured and the calculated values of the fluid temperature at the collector outlet, conclusions can be drawn about the process of solar glass fouling or glycol gelling. The model's simplicity and the low computational demands enable such comparisons in an online mode.

This paper presents a novel architecture for detecting mathematical formulas in document images, which is an important step for reliable information extraction in several domains. Recently, Cascade Mask R-CNN networks have been introduced to solve object detection in computer vision. In this paper, we suggest a couple of modifications to the existing Cascade Mask R-CNN architecture: First, the proposed network uses deformable convolutions instead of conventional convolutions in the backbone network to spot areas of interest better. Second, it uses a dual backbone of ResNeXt-101, having composite connections at the parallel stages. Finally, our proposed network is end-to-end trainable. We evaluate the proposed approach on the ICDAR-2017 POD and Marmot datasets. The proposed approach demonstrates state-of-the-art performance on ICDAR-2017 POD at a higher IoU threshold with an f1-score of 0.917, reducing the relative error by 7.8%. Moreover, we accomplished correct detection accuracy of 81.3% on embedded formulas on the Marmot dataset, which results in a relative error reduction of 30%.

Deep Neutral Networks (DNNs) were initially proposed towards the midst of the 20th century, motivated by the neural structure and mechanism of the human brain. Thanks to major ad-vancements in computational resources, during the past decade DNN-based systems have demonstrated unprecedented performance in terms of accuracy and speed. However, recent work has shown that such models may not be sufficiently robust during the inference process. Furthermore, due to the data-driven learning nature of DNNs, designing interpretable and gen-eralizable networks is a major challenge, especially in critical applications, such as medical Computer Aided Diagnostics (CAD) and other medical imaging tasks, including classification, regression and reconstruction. Within this context, a line of physics-driven approaches for deep learning has recently emerged, aimed at improving the stability and generalization capacity of DNNs for medical imaging applications. In this paper, we review recent work focused on phys-ics-driven or prior-information learning for a variety of imaging modalities and medical applica-tions. We discuss how the inclusion of domain-related knowledge into the learning process and networks’ design supports their stability and generalization capability. In addition, we highlight current and future challenges within this scope.

Daltonisation refers to the recolouring of images such that details normally lost by colour vision deficient observers become visible. This comes at the cost of introducing artificial colours. In a previous work, we presented a gradient-domain colour image daltonisation method that outperformed previously known methods both in behavioural and psychometric experiments. In the present paper, we improve the method by (i) finding a good first estimate of the daltonised image, thus reducing the computational time significantly, and (ii) introducing local linear anisotropic diffusion, thus effectively removing the halo artefacts. The method uses a colour vision deficiency simulation algorithm as an ingredient, and can thus be applied for any colour vision deficiency, and can even be individualised if the exact individual colour vision is known.

Despite the fact that COVID-19 is no longer a global pandemic due to development and integration of different technologies for the diagnosis and treatment of the disease. Technological advancement in the field of molecular biology, electronics, computer science, artificial intelligence, Internet of Things, nanotechnology etc. has led to the development of molecular approaches and computer aided diagnosis for the detection of COVID-19. This study provides a holistic approach on COVID-19 detection based on (1) molecular diagnosis which include RT-PCR, antigen-antibody and CRISPR-based biosensors and (2) computer aided detection based on AI-driven models which include Deep Learning and Transfer learning approach. The review also provide comparison between these 2 emerging technologies and open research issues for the development of smart-IoMT-enable platform for the detection of COVID-19.

Maze navigation using one or more robots has become a recurring challenge in scientific literature and real life practice, with fleets having to find faster and better ways to navigate environments such as a travel hub (e.g. airports) or to evacuate a disaster zone. Many methods have been used to solve this issue, including the implementation of a variety of sensors and other signal receiving systems. Most interestingly, camera-based techniques have increasingly become more popular in this kind of applications, given their robustness and scalability. In this paper, we have implemented an end-to-end strategy to address this scenario, allowing a robot to solve a maze in an autonomous way, by using computer vision and path planning. In addition, this robot shares the generated knowledge to another by means of communication protocols, having to adapt its mechanical characteristics to be able to solve the same challenge. The paper presents experimental validation of the four components of this solution, namely camera calibration, maze mapping, path planning and robot communication. Finally, we present the integration and functionality of these methods applied in a pair of NAO robots.

This study explores the integration of computer vision with topology optimization for additive manufacturing, with a focus on maximizing eigenfrequency in a design domain. Utilizing cus-tom-developed photogrammetry software, high-resolution images are processed to generate de-tailed 3D models, subsequently converted to STL files with precision. Adaptive meshing in COMSOL Multiphysics, controlled through a MATLAB API, ensures optimal mesh resolution. Prioritizing resource conservation in extraterrestrial environments, the original volume is reduced by 50% while preserving structural integrity. The design domain undergoes rigorous topology optimization in MATLAB, supported by COMSOL's advanced FEM simulation. The optimized design exhibits a 57% performance improvement and a 50% weight reduction, maintaining desired vibration characteristics, validating the efficacy of the modifications. Moreover, the case with an eccentric mass shows a significant 64% increase in eigenfrequency.

This article explores the efficiency of various clustering methods for image segmentation under different luminosity conditions. Image segmentation plays a crucial role in computer vision appli-cations, and clustering algorithms are commonly used for this purpose. The search for an adaptive clustering mechanism aims to ensure maximum symmetry of real objects with objects/segments in their digital representations. However, clustering methods performance can fluctuate with varying lighting conditions during image capture. Therefore, we assess the performance of several clustering algorithms — including k-means, K-Medoids, fuzzy c-means, Possibilistic C-Means, Gus-tafson-Kessel, Entropy-based Fuzzy, Riddler-Calvard, Kohonen Self-Organizing Maps, and Meanshift — across images captured under different illumination conditions. Additionally, we develop an adaptive image segmentation system utilizing empirical data. Conducted experiments highlight varied performance among clustering methods under different luminosity conditions. This research enhance better understanding of luminosity's impact on image segmentation and aiding in method selection for diverse lighting scenarios.

Existing methods for 3D human pose estimation mainly divide the task into two stages. The first stage identifies the 2D coordinates of the human joints in the input image, namely the 2D human joint coordinates. The second stage uses the results from the first stage as input to recover the depth information of human joints from the 2D human joint coordinates to achieve 3D human pose estimation. However, the recognition accuracy of the two-stage method relies heavily on the results of the first stage and includes too many redundant processing steps, which reduces the inference efficiency of the network. To address these issues, we propose the EDD, a fully End-to-end 3D human pose estimation method based on transformer architecture with Dual Decoders. By learning multiple human poses, the model can directly infer all 3D human poses in the image using a pose decoder, and then further optimize the recognition result using a joint decoder based on the kinematic relations between joints. With the attention mechanism, this method can adaptively focus on the most relevant features to the target joint, effectively overcoming the feature misalignment problem in the human pose estimation task and greatly improving the model performance. Any complex post-processing step, such as non-maximum suppression, is eliminated, further improving the efficiency of the model. The results show that the method achieves an accuracy of 87.4% on the MuPoTS-3D dataset, significantly improving the accuracy of end-to-end 3D human pose estimation methods.

Existing one-shot neural architecture search (NAS) methods have to conduct a search over a giant super-net, which leads to the huge computational cost. To reduce such cost, in this paper, we propose a method, called FTSO, to divide the whole architecture search into two sub-steps. Specifically, in the first step, we only search for the topology, and in the second step, we search for the operators. FTSO not only reduces NAS’s search time from days to 0.68 seconds, but also significantly improves the found architecture's accuracy. Our extensive experiments on ImageNet show that within 18 seconds, FTSO can achieve a 76.4% testing accuracy, 1.5% higher than the SOTA, PC-DARTS. In addition, FTSO can reach a 97.77% testing accuracy, 0.27% higher than the SOTA, with nearly 100% (99.8%) search time saved, when searching on CIFAR10.

Spike shape and morphometric characteristics are among the key characteristics of cultivated cereals associated with their productivity. Identification of the genes controlling these traits requires morphometric data at harvesting and analysis of numerous plants, which could be automatically done using technologies of digital image analysis. A method for wheat spike morphometry utilizing 2D image analysis is proposed. Digital images are acquired in two variants: a spike on a table (one projection) or fixed with a clip (four projections). The method identifies spike and awns in the image and estimates their quantitative characteristics (area in image, length, width, circularity, etc.). Section model, quadrilaterals, and radial model are proposed for describing spike shape. Parameters of these models are used to predict spike shape type (spelt, normal, or compact) by machine learning. The mean error in spike density prediction for the images in one projection is 4.61 (~18%) versus 3.33 (~13%) for the parameters obtained using four projections.

This paper presents a system that merges computer vision and surface electromyography techniques to carry out grasping tasks. To perform this, the vision-driven system is used to compute pre-grasping poses of the robotic system based on the analysis of tridimensional object features. Then, the human operator can correct the pre-grasping pose of the robot using surface electromyographic signals from the forearm during wrist flexion and extension. Weak wrist flexions and extensions allow a fine adjustment of the robotic system to grasp the object and finally, when the operator considers that the grasping position is optimal, a strong flexion is performed to initiate the grasping of the object. The system has been tested with several subjects to check its performance showing a grasping accuracy of around 95% of the attempted grasps which increases by around a 9% the grasping accuracy of previous experiments in which electromyographic control was not implemented.

This paper presents an analysis of the sensitivity of controlling an autonomous surface object among a group of encountered objects to the inaccuracy of state control process data. For this purpose, a safe and optimal model of the control process was defined. An algorithm for determining the optimal and safe trajectory of an object, based on the multi-object game model, was developed; for comparative analysis, an algorithm for calculating only the optimal trajectory, not taking into account the maneuverability of other objects, was additionally developed. First, simulation-based studies of the algorithms enabled a comparative analysis of a number of different acceptable strategies for optimally shifting the trajectory of maneuvering objects from a single initial direction. Thereafter, the main goal of this paper was implemented: an analysis of the sensitivity characteristics of safe control, assessed with the risk of collision, both on the inaccuracy of navigation data and on the number of possible control strategies. Finally, final conclusions and a plan for further research on the subject of the paper were formulated.

With recent focus on deep neural network architectures for development of algorithms for computer-aided diagnosis (CAD), we provide a review of studies within the last 3 years (2015-2017) reported in selected top journals and conferences. 29 studies that met our inclusion criteria were reviewed to identify trends in this field and to inform future development. Studies have focused mostly on cancer-related diseases within internal medicine while diseases within gender-/age-focused fields like gynaecology/pediatrics have not received much focus. All reviewed studies employed image datasets, mostly sourced from publicly available databases (55.2%) and few based on data from human subjects (31%) and non-medical datasets (13.8%), while CNN architecture was employed in most (70%) of the studies. Confirmation of the effect of data manipulation on quality of output and adoption of multi-class rather than binary classification also require more focus. Future studies should leverage collaborations with medical experts to aid future with actual clinical testing with reporting based on some generally applicable index to enable comparison. Our next steps on plans for CAD development for osteoarthritis (OA), with plans to consider multi-class classification and comparison across deep learning approaches and unsupervised architectures were also highlighted.

Geometrical assessments of human skull have been done based on the anatomical landmarks. Automatic detection of the landmarks, if developed, will be a great help not only medically, but also anthropologically. An automated system with multi-phased deep learning networks to predict three-dimensional coordinate values of cranio-facial landmarks, was developed. From a publicly available database, computed tomography images of craniofacial area were obtained. They were digitally reconstructed into three-dimensional models. Sixteen anatomical landmarks were plotted on each of the models and coordinate values of them were recorded. Three-phased regression deep learning networks were trained respectively with 90 training datasets. For evaluation, 30 testing datasets were employed. Three-dimensional error for the first phase, testing 30 data, was 11.60 pixels in average. (1 pixel = 500 / 512 mm) For the second phase, it was significantly improved to 4.66 pixels. For the third phase, it was significantly progressed to 2.88. This was comparable to the gaps between the landmarks, plotted by two experienced practitioners. Our proposing method of multi-phased prediction, coarse detection first and narrowing down the detection area, may be a possible solution, within the physical limitation of memory and computation.

This study introduces a sophisticated neural network structure for segmenting breast tumors. It achieves this by combining a pretrained Vision Transformer (ViT) model with a U-Net framework. The U-Net architecture, commonly employed for biomedical image segmentation, is further enhanced with Depthwise Separable Convolutional Blocks to decrease computational complexity and parameter count, resulting in better efficiency and less overfitting. The Vision Transformer, renowned for its robust feature extraction capabilities utilizing self-attention processes, efficiently captures the overall context within images, surpassing the performance of conventional convolutional networks. By using a pretrained ViT as the encoder in our U-Net model, we take advantage of its extensive feature representations acquired from extensive datasets, resulting in a major enhancement in the model’s ability to generalize and train efficiently. The suggested model has exceptional performance in segmenting breast cancers from medical images, highlighting the advantages of integrating transformer-based encoders with efficient U-Net topologies. This hybrid methodology emphasizes the capabilities of transformers in the field of medical image processing and establishes a new standard for accuracy and efficiency in activities related to tumor segmentation.

Deep learning is widely used in image segmentation, effectively identifying crops and weeds and contributing to the reduction of herbicide use. Traditional methods of applying pesticides over large areas are inefficient in dealing with weeds as needed, leading to massive waste of pesticides, high-cost production, and serious environmental pollution. This affects crop yield and quality. Although weed recognition using conventional deep learning methods has evolved over time, there are still challenges in weed extraction, detection, and segmentation. Accurate recognition and detection of weeds are essential prerequisites for implementing variable spraying. This paper proposes a semantic segmentation method based on UNet++ for complex environments where accurate identification of plants and weeds is challenging, targeting weeds in sugar beets, peas, and rice. An attention module is integrated into the upsampling process of UNet++, and UNet++ is used as a backbone network to effectively integrate multi-scale information, efficiently suppressing external noise interference. The UNet++ model with an integrated attention mechanism module achieves higher IOU than the general UNet++ model used in medical image analysis. This method effectively detects crops and weeds in complex backgrounds, providing reference material for the accurate application of robotic herbicides.

Amidst the burgeoning global concerns surrounding skin cancer, the quest for precise and efficient diagnostic methodologies has intensified. Conventional diagnostic approaches relying on subjective visual inspection by dermatologists often grapple with subjectivity and resource constraints. This predicament underscores the critical role of Artificial Intelligence (AI) in revolutionizing diagnostic paradigms by significantly enhancing accuracy and accessibility. This study embarks on an extensive exploration of skin cancer classification, specifically targeting the intricate task of 8-class skin cancer classification. Employing cutting-edge deep learning models on the ISIC2019 dataset, a comprehensive analysis was conducted utilizing a diverse array of pre-trained ImageNet architectures and Vision Transformer models. To counteract the inherent class imbalance in skin cancer datasets, a pioneering "Naturalize" augmentation technique was introduced, leading to the creation of indispensable tools—the Naturalized 2.4K ISIC2019 and groundbreaking Naturalized 7.2K ISIC2019 datasets—catalyzing advancements in classification accuracy. The pivotal role of AI in mitigating the risks of misdiagnosis and under-diagnosis in skin cancer becomes evident through this research. AI’s proficiency in analyzing vast datasets and discerning subtle patterns augments the diagnostic prowess of dermatologists significantly. Integration of AI-powered systems in skin cancer diagnosis holds immense promise in facilitating early detection and substantially improving patient outcomes. The meticulous evaluation employed quantitative measures such as confusion matrices, classification reports, and visual analyses using Score-CAM across diverse dataset variations. Astonishingly, the culmination of these endeavors culminated in an unprecedented achievement—100% average accuracy, precision, recall, and F1-Score—within the groundbreaking Naturalized 7.2K ISIC2019 dataset. This remarkable feat underscores the transformative potential of AI-driven methodologies, exemplifying their prowess in redefining the landscape of skin cancer diagnosis and management. In summary, this research represents a pivotal stride towards redefining dermatological diagnosis, showcasing the remarkable impact of AI-powered solutions in surmounting the challenges inherent in skin cancer diagnosis. The attainment of 100% across crucial metrics within the Naturalized 7.2K ISIC2019 dataset serves as a testament to the transformative capabilities of AI-driven approaches in reshaping the trajectory of skin cancer diagnosis and patient care.

The article presents the operation of the automatic pine sawn timber inspection system, which was developed at the Woodinspector company and offered commercially. The vision inspection system is used to detect various wood defects, including knots, blue stain, and mechanical damage caused by worms. Blue stain is a defect that is difficult to detect based on the color of the wood, because it can be easily confused with wood defects or dirt that do not impair its strength properties. In particular, the issues of detecting blue stain in wood, the use of artificial neural networks and improving the operation of the system in production conditions were discussed.

Space syntax can be potentially applied for the evaluation of quality of user flows particularly in solving problems of sustainable projects in spaces intended for users to move around. This study aims to analyze the concepts of pedestrian flows (open and closed) by space syntax based bibli-ographical approach on a global scale, demonstrating the capability of improvements in Sus-tainable Development Goals (SDGs) as applied to the architecture of sustainable flows. Scopus theoretical reference bases were used, which are directly related to the theme of space syntax in open and closed spaces. Frequency analyzes were carried out, applying the content analysis, to identify words with a degree of similarity, related to “space syntax: flow in urban environ-ments” and “space syntax in closed built systems”, in relations to the SDGs. The results show that pedestrian flows identified in the literature aid to understand the global production on space syntax in open and closed spaces, directed to user flows in the built environment, where many environments of flows in the built space analyzed become unsustainable because they do not present full flow efficiency. In our study on space syntax, the following central terms were identified: pedestrian movements (open urban systems) and space (closed built systems), which allows a better understanding of the flows, highlighting the importance of the urbanist architect on the functionality of user flows in sustainable architectural projects on a global scale. The most frequent categories for open environments were the terms space and form and for closed envi-ronments were space and flow.

The traditional understanding of brain function has predominantly focused on chemical and electrical processes. However, new research in fruit fly (Drosophila) binocular vision reveals ultrafast photomechanical photoreceptor movements significantly enhance information processing, thereby impacting a fly's perception of its environment and behaviour. The coding advantages resulting from these mechanical processes suggest that similar physical motion-based coding strategies may affect neural communication ubiquitously. The theory of neural morphodynamics proposes that rapid biomechanical movements and microstructural changes at the level of neurons and synapses enhance the speed and efficiency of sensory information processing, intrinsic thoughts, and actions by regulating neural information in a phasic manner. We propose that morphodynamic information processing evolved to drive predictive coding, synchronising cognitive processes across neural networks to match the behavioural demands at hand effectively.

Currently the majority of studies on vision-based measurement has been conducted under ideal environments so that an adequate measurement performance and accuracy is ensured. However, vision-based systems may face some adverse influencing factors such as illumination change and fog interference, which can affect the measurement accuracy. This paper develops a robust vision-based displacement measurement method which can handle the two common and important adverse factors given above and achieve sensitivity at the subpixel level. The proposed method leverages the advantage of high-resolution imaging incorporating spatial and temporal context aspects. To validate the feasibility, stability and robustness of the proposed method, a series of experiments was conducted on a two-span three-lane bridge in the laboratory. The illumination change and fog interference are simulated experimentally in the laboratory. The results of the proposed method are compared to conventional displacement sensor data and current vision-based method results. It is demonstrated that the proposed method gives better measurement results than the current ones under illumination change and fog interference.

The rise in both the frequency of natural disasters and the ubiquity of artificial intelligence has led to novel applications of new technologies in improving disaster response processes, such as the labor-intensive assessment of disaster damages. Assessment of residential and commercial structure damages, a precursory step to government agencies being able to provide most of their financial assistance, has benefited from aerial and satellite imagery-based computer vision models; however, limitations of using such imagery include the ground structures being obscured by clouds or smoke, as well as the lack of resolution to distinguish individual structures from others. Using a different data source, we propose a damage severity classification model using ground-level imagery, focusing on residential structures damaged by wildfires. This classifier, a Vision Transformer (ViT) model trained on over 18,000 professionally labeled images of damaged homes from the 2020-2022 California wildfires, has achieved an accuracy score of over 95%. Further, we have open sourced the training dataset–the first of its kind and scale–as well as built a publicly available web application prototype, which we demoed to and received feedback from disaster response officials, both of which further contribute to the broader literature beyond the proposed model.

Machine learning models facilitate the search for non-linear relationships when modeling hydrological processes, but they are equally effective for automation at the data preparation stage. The tasks whose automation was analyzed consisted in estimating changes in the roughness of a mountain streambed and the extent of floods from images. The Segment Anything Model (SAM) developed this year by Meta was used for this purpose. Images from many years from the Wielka Puszcza mountain stream located in the Polish Carpathians were used as the only input data. The model was not additionally trained for the described tasks. The model can be run in several modes, but the two most appropriate were used in this study. The first one is available in the form of a web application, while the second one is available in the form of a Jupyter notebook run in the Google Colab environment. Both methods do not require specialized knowledge and can be used by virtually any hydrologist. In the roughness estimation task, the average Intersection over Union (IoU) ranges from 0.55 for grass to 0.82 for shrubs/trees. Ultimately, it was possible to estimate the roughness coefficient of the mountain streambed between 0.027 and 0.059 based solely on image data. In the task of estimation of the flood extent, when selecting appropriate images, one can expect IoU at the level of at least 0.94, which seems to be an excellent result considering that SAM is a general-purpose segmentation model. It can therefore be concluded that SAM model can be a useful tool for a hydrologist.

In Mexico, the incorporation of deaf people into education has been lacking since only 14% of the deaf population in the age group between 3 and 29 years access education with the support of a hearing aid. Additionally, those who have been incorporated frequently face inappropriate educational strategies which poorly develop the use of Mexican Sign Language (MSL) and therefore academical success and opportunities for insertion in the workplace are difficult. This research explores a novel mexican sign language lexicon video dataset containing the dynamical gestures most frequently used by MSL. Each gesture consists of a set of different versions of videos under uncontrolled conditions. MX-ITESO-100 data set is composed of a lexicon of 100 gestures and 5,000 videos from three participants with different grammatical elements. Additionally, the data set is evaluated in a two-step neural network model with an accuracy greater than 99%. and thus serves as a benchmark for future training of machine learning models in computer vision systems. Finally, this research provides an inclusive environment within society and organizations in particular for people with hearing impairment.

Mice are one of the frequently used animal models in science research whose behavioral characteristics can provide much valuable information in biology, neuroscience, and pharmacology. Nowadays, artificial intelligence is widely used in mice behavior analysis. Integrated AI systems such as ChatGPT and VisualGPT are already available, and we discuss the feasibility of MiceGPT to help researchers identify and classify mouse behavior more easily. We review the applications of mice behavior analysis, analyze the tasks of deep learning on these applications based on an AI pyramid, and finally summarize the AI approaches to solve these tasks. Based on these summaries, we propose three MiceGPT architectures to demonstrate the theoretical feasibility of MiceGPT.

Objective: Radiomics and deep transfer learning are two popular technologies used to develop computer-aided detection and diadnosis (CAD) schemes of medical images. This study aims to investigate and compare advantages and potential limitations of applying these two technologies in developing CAD schemes. Methods: A relatively large and diverse retrospective dataset including 3,000 digital mammograms is assembled in which 1,496 images depict malignant lesions and 1,054 images depict benign lesions. Two CAD schemes are developed to classify breast lesions. The first scheme is developed using four steps namely, applying an adaptive multi-layer topographic region growing algorithm to segment lesions, computing initial radiomics features, applying a principal component algorithm to generate an optimal feature vector, and building a support vector machine classifier. The second CAD scheme is built based a pre-trained residual net architecture (ResNet50) as a transfer learning model to classify breast lesions. Both CAD schemes are trained and tested using a 10-fold cross-validation method. Several score fusion methods are also investigated to classify breast lesions. CAD classification performance is evaluated by the area under ROC curve (AUC). Results: ResNet50 model-based CAD scheme yields AUC = 0.85±0.02, which is significantly higher than radiomics feature-based CAD scheme with AUC = 0.77±0.02 (p < 0.01). Additionally, fusion of classification scores generated by two CAD schemes does not further improve classification performance. Conclusion: This study indicates that using deep transfer learning is more efficient to develop CAD schemes and enables to yield higher lesion classification performance than CAD schemes developed using radiomics-based technology.

This study presents a novel approach combining computer vision with topology optimization for additive manufacturing. Using specialized photogrammetry software, we capture high-resolution images of the design domain to create precise 3D models through detailed scanning. These models are converted into STL file format and remeshed using an adaptive algorithm within COMSOL 5.3 Multiphysics, managed by a custom MATLAB 2023. This integration allows optimal mesh resolution and accuracy in analyses. We applied this method to design a concrete pillar for 3D printing, aiming for a 75% volume reduction to enhance material efficiency and stability, crucial for extraterrestrial environments. The design, scanned with a 360-degree camera array, informed the MATLAB-based topology optimization process. Combining MATLAB’s optimization algorithms with COMSOL’s meshing and finite element solver solver, we explored material-efficient configurations. The results show significant volume reduction, particularly in the central design area, optimizing material use while maintaining structural stability. The optimization algorithm demonstrated rapid, stable convergence to near-optimal solutions within about 20 iterations, highlighting the method’s efficiency and robustness in complex design optimization.

Vision-based tactile sensors (VBTS) have become the de facto method of giving robots the ability to obtain tactile feedback from their environment. Unlike other solutions to tactile sensing, VBTS offers high spatial resolution feedback without compromising on instrumentation costs or incurring additional maintenance expenses. However, conventional cameras used in VBTS have a fixed update rate and output redundant data, leading to computational overhead downstream. In this work, we present a neuromorphic vision-based tactile sensor (N-VBTS) that employs observations from an event-based camera for contact angle prediction. Particularly, we design and develop a novel graph neural network, dubbed TactiGraph, that asynchronously operates on graphs constructed from raw N-VBTS streams exploiting their spatiotemporal correlations to perform predictions. Although conventional VBTS uses an internal illumination source, TactiGraph is reported to perform efficiently in both scenarios, with and without an internal illumination source. Rigorous experimental results revealed that TactiGraph achieved a mean absolute error of 0.62∘ in predicting the contact angle and was faster and more efficient than both conventional VBTS and other N-VBTS, with lower instrumentation costs. Specifically, N-VBTS requires only 5.5% of the compute-time needed by VBTS when both are tested on the same scenario.

Many sand scorpions are faithful to burrows they dig, however, it is unknown how these animals get back home after hunting excursions. Of the many mechanisms of homing that exist, path integration (PI) is one of the more common tools used by arachnids. In PI, an animal integrates its distance and direction while leaving its home, enabling it to compute an approximate home-bound vector for the return trip. The objective of our study was to test whether scorpions use PI to return home under absolute darkness in the lab. We first allowed animals to establish burrows in homing arenas. Then, after they left their burrow, we recorded the scorpion’s location in the homing arena before we transferred it to the center of a testing arena. We used overhead IR cameras to record its movements in the testing arena. If scorpions exhibited PI, we predicted they would follow a vector in the test arena that approximated the same angle and distance from the capture point to their burrow in their home arena. However, under the conditions of this experiment, we found no evidence that scorpions moved along such home-bound vectors. We speculate that scorpions may need a reliable reference cue to accommodate path integration.

Tables in the document image are one of the most important entities since they contain crucial information. Therefore, accurate table detection can significantly improve information extraction from tables. In this work, we present a novel end-to-end trainable pipeline, HybridTabNet, for table detection in scanned document images. Our two-stage table detector uses the ResNeXt-101 backbone for feature extraction and Hybrid Task Cascade (HTC) to localize the tables in scanned document images. Moreover, we replace conventional convolutions with deformable convolutions in the backbone network. This enables our network to detect tables of arbitrary layouts precisely. We evaluate our approach comprehensively on ICDAR-13, ICDAR-17 POD, ICDAR-19, TableBank, Marmot, and UNLV. Apart from the ICDAR-17 POD dataset, our proposed HybridTabNet outperforms earlier state-of-the-art results without depending on pre and post-processing steps. Furthermore, to investigate how the proposed method generalizes unseen data, we conduct an exhaustive leave-one-out-evaluation. In comparison to prior state-of-the-art results, our method reduces the relative error by 27.57% on ICDAR-2019-TrackA-Modern, 42.64% on TableBank (Latex), 41.33% on TableBank (Word), 55.73% on TableBank (Latex + Word), 10% on Marmot, and 9.67% on UNLV dataset. The achieved results reflect the superior performance of the proposed method.

Visualization of volumetric data holds great importance, particularly in medical and biomedical applications. Various imaging techniques, such as FMRI and 3D microscopy, are employed to generate volumetric data, used in medical practice, research, and teaching. Commonly utilized tools like 3D Slicer, Fiji, and MATLAB ® aid in rendering and analyzing 3D images. However, these tools may lack comprehensive rendering functionality and face challenges in handling computational demands as data sizes grow. To address these limitations, this work introduces a GPU-supported renderer with a MATLAB ® interface. This solution gives the user flexible control over rendering parameters and optimizes data transfer through a memory management system. By leveraging the computational power of NVIDIA GPUs, the renderer enables complex and high-quality renderings, enhancing speed and efficiency. It therefore facilitates the analysis and visualization of volumetric data within an integrated environment, namely MATLAB ®, streamlining their workflows. This advancement provides valuable opportunities for researchers and medical professionals to explore and comprehend volumetric data effectively.

Fall prediction using machine learning has become one of the most fruitful and socially relevant applications of computer vision in gerontological research. Since its inception in the early 2000s, this subfield has proliferated into a robust body of research underpinned by various machine learning algorithms (including neural networks, support vector machines, and decision trees) as well as statistical modeling approaches (Markov chains, Gaussian mixture models, and hidden Markov models). Furthermore, some advancements have been translated into commercial and clinical practice, with companies in various stages of development capitalizing on the aging population to develop new commercially available products. Yet despite the marvel of modern machine learning-enabled fall prediction, little research has been conducted to shed light on the security and privacy concerns that such systems pose for older adults. The present study employs an interdisciplinary lens in examining privacy issues associated with machine learning fall prediction and exploring the implications of these models in elderly care and the Internet of Medical Things (IoMT). Ultimately, a justice-informed set of best practices rooted in social geroscience is suggested to help fall prediction researchers and companies continue to advance the field while preserving elderly privacy and autonomy.

Uncontrolled and fast cell proliferation is the cause of brain tumors. Early cancer detection is vitally important in order to save many lives. Brain tumors can be divided into several categories depending on the kind, place of origin, pace of development, and stage of progression; as a result, tumor classification is crucial for targeted therapy. The aim of brain tumor segmentation is to accurately delineate the areas of the brain. A specialist with a thorough understanding of brain illnesses must manually identify the proper type of brain tumor. Additionally, processing a lot of images takes time and is tiresome. Therefore, automatic segmentation and classification techniques are required to speed up and enhance the diagnosis of brain tumors. Tumors can be quickly and safely detected by brain scans using imaging modalities, including computed tomography (CT), magnetic resonance imaging (MRI), and others. Machine learning (ML) and artificial intelligence (AI) have shown promise in the development of algorithms that aid in automatic classification and segmentation utilizing various imaging modalities. This review discussed various types of brain tumors, publicly accessible datasets, enhancement methods, segmentation, feature extraction, classification, machine learning techniques, and deep learning, learning through transfer, for the study of brain tumors.

Document deskewing is a fundamental problem in document image processing. While existing methods have limitations, such as Hough Line Transformation that can deskew images upside down, and Deep Learning models that require huge amounts of human labour and computational resources and still fail to deskew while taking care of orientation, OCR-based methods also struggle to read text when it is tilted. In this paper, we propose a novel, simple, cost-effective deep learning method for fixing the skew and orientation of documents. Our approach reduces the search space for the machine learning model to predict whether an image is upside down or not, avoiding the huge search space of predicting an angle between 0 and 360. We finetuned a MobileNetV2 model, which was pre-trained on imagenet, using only 200 images and achieve good results. This method is useful for automation-based tasks, such as data extraction using OCR technology, and can greatly reduce manual labour.

One of the critical tools for early detection and subsequent evaluation of the incidence of lung diseases is chest radiography. This study presents a real-world implementation of a convolutional neural network (CNN) based Carebot Covid app to detect COVID-19 from chest X-ray (CXR) images. Our proposed model takes the form of a simple and intuitive application. Used CNN can be deployed as a STOW-RS prediction endpoint for direct implementation into DICOM viewers. The results of this study show that the deep learning model based on DenseNet and ResNet architecture can detect SARS-CoV-2 from CXR images with precision of 0.981, recall of 0.962 and AP of 0.993.

Artificial intelligence (AI) and 3D printing (3DP) play critical roles in what is known as the 4th Industrial Revolution by developing data- and machine-intelligence-based integrated production technologies. This shift has led to increasingly complex design requirements and fabrication methods, posing several challenges for implementing design, fabrication process, and architectural quality standardization. This paper systematically reviews AI techniques applied in diverse stages of 3DP production in architecture. The research goals are to (1) offer a comprehensive critical analysis of the body of literature; (2) identify and categorize approaches to integrating AI in the production of 3D-printed structures; (3) identify and discuss challenges and opportunities of AI integration in architectural production of 3DP structures; and (4) identify gap and provide recommendations for future research. The findings indicate that topics related to AI applications are current and trending, receiving increasing attention from researchers in general and particularly those working in machine and deep learning, artificial neural networks, and pattern recognition. However, the potential of the application of AI in architectural production using additive manufacturing still needs to be explored. Considering this, the paper emphasizes the necessity of redefining traditional field boundaries and opening new opportunities in the sector.

In this paper, we present a multi-task model that predicts disparities and confidence levels in deep stereo matching simultaneously. We do this by combining its successful model for each separate task and obtaining a multitask model that can be trained with a proposed loss function. We show the advantages of this model compared to training and predicting disparity and confidence sequentially. This method enables an improvement of 15% to 30% in the area under the curve (AUC) metric when trained parallel rather than sequential. In addition, the effect of weighting the components in the loss function on the stereo and confidence performance is investigated. By improving the confidence estimate, the practicality of stereo estimators for creating distance images is increased.

This paper introduces an innovative approach to lipreading, leveraging a web application designed to generate subtitles for videos where the speaker's mouth is visible and a comprehensive literature review that precedes the discussion, encompassing a thorough examination of various lipreading methods employed over the past decade. Our method employs a powerful deep learning model, featuring a 3D-convolution network and bidirectional LSTM, enabling accurate sentence-level predictions based solely on visual lip movements. With an impressive accuracy of 97%, our model is trained using pre-segmented lips regions, transformed into animated GIFs for effective pre-training. This work stands as a significant contribution to the evolving landscape of lipreading research, offering a practical and accurate solution for real-world applications.

The point cloud is a form of three-dimensional data that comprises various detailed features at multiple scales. Due to this characteristic and its irregularity, point cloud analysis based on deep learning is challenging. While previous works utilize the sampling-grouping operation of PointNet++ for feature description and then explore geometry by means of sophisticated feature extractors or deep networks, such operations fail to describe multi-scale features effectively. Additionally, these techniques have led to performance saturation. And it is difficult for standard MLPs to directly "mine" point cloud. To address above problems, we propose the Detail Activation (DA) module, which encodes data based on Fourier transform after sampling and grouping. We activate the channels at different frequency levels from low to high in the DA module to gradually recover finer point cloud details. As training progresses, the proposed Point-MDA can uncover local and global geometries of point cloud progressively. Our experiments show that Point-MDA achieves superior classification accuracy, outperforming PointNet++ by 3.3% and 7.9% in terms of overall accuracy on the ModelNet40 and ScanObjectNN dataset, respectively. Furthermore, it accomplishes this without employing complicated operations, while exploring the full potential of PointNet++.

Auto checkout has received more and more attention in recent years and this system automatically generates a shopping bill by identifying the picture of the products purchased by the customers. However, the system is challenged by the domain adaptation problem, where each image of the training set contains only one commodity, whereas the test set is a collection of multiple commodities. The existing solution to this problem is to resynthesize the training images to enhance the training set. Then the composite images are rendered using CycleGAN to make the image distribution of the training set and the test set more similar. However, we find that the detection boxes given by the ground truth of the common dataset contain a large part of the background area, the area will affect the training process as noise. To solve this problem, we propose a mask data priming method. Specifically, we redo the large scale Retail Product Checkout (RPC) dataset and add segmentation annotation information to each item in the training set image based on the original dataset using pixel-level annotation. Secondly, a new network structure is proposed in which we train the network using joint learning of detectors and counters, and fine-tune the detection network by filtering out suitable images from the test set. Experiments on the RPC dataset have shown that our method yields better results. we used an approach that reached 81.87% compared to 56.68% for the baseline approach which demonstrates that pixel-level information helps to improve the detection results of the network.

Spike shape and morphometric characteristics are among the key characteristics of cultivated cereals associated with their productivity. Identification of the genes controlling these traits requires morphometric data harvesting and analysis for numerous plants, which is automatable using technologies of digital image analysis. A method for wheat spike morphometry utilizing 2D image analysis is proposed. Digital images are acquired in two variants: a spike on a table (one projection) or fixed with a clip (four projections). The method identifies spike and awns in the image and estimates their quantitative characteristics (area in image, length, width, circularity, etc.). Models of sections, quadrilaterals, and radial model are proposed for describing spike shape. Parameters of these models are used to predict spike shape type (spelt, normal, or compact) by machine learning. The mean error in spike density prediction for the images in one projection is 4.61 (~18%) versus 3.33 (~13%) for the parameters obtained using four projections. F1 measure in automated spike classification into three types is 0.78 using logistic regression (one projection) and 0.85 using random forest method (four projections). The proposed method is implemented in Java; examples of images and user guide are available at http://wheatdb.org/werecognizer.

Ultrasound (US) has become a widely used imaging modality in clinical practice, characterized by its rapidly evolving technology, advantages, and unique challenges such as low imaging quality and high variability. There is a critical need to develop advanced automatic US image analysis methods to enhance diagnostic accuracy and objectivity. Vision transformer, a recent innovation in machine learning, has demonstrated significant potential in various research fields, including general image analysis and computer vision, due to its capacity to process large datasets and learn complex patterns. Its suitability for automatic US image analysis tasks, such as classification, detection, and segmentation, has been recognized. This review provides an introduction to vision transformer and discusses its applications in specific US image analysis tasks, while also addressing the open challenges and potential future trends in its application in medical US image analysis. Vision transformer has shown promise in enhancing the accuracy and efficiency of ultrasound image analysis and is expected to play an increasingly important role in the diagnosis and treatment of medical conditions using ultrasound imaging as technology progresses.

The COVID-19 outbreak demonstrated the viability of various remote working solutions, telepresence robots (TPRs) being one of them. High-quality video transmission is one of the cornerstones of using such solutions, as most of the information about the environment is acquired through vision. This study aims to compare the camera capabilities of four models of popular telepresence robots using compact reduced LogMAR and Snellen optometry charts as well as text displayed on the projector screen. The symbols from the images are extracted using OCR (Optical Character Recognition) software and the results of the recognition are compared with the symbols on the charts. Double 3 TPR provides the best quality images of optometric charts, but the OCR results of measurements of the image on the projector do not show the clear advantage of one single model over the others. The results demonstrated by Temi 2 and Double 3 TPRs are generally better than the others, suggesting that these TPRs are more feasible to be used in teaching and learning scenarios.

Through the refinement of approaches to applying visual augmentation to virtual environments, the possibilities of utilising this process in robotics continues to be explored. However these endeavours have only yielded a collaborative relationship between the two technologies, where the augmented reality (AR) component is separate from the robot functionality. Regarding telepresence robots, research has shown that their perceived usefulness wanes in comparison to that of video conferencing tools as they do not offer a level of immersion and modified visual experience to justify their use. Therefore this paper contributes a novel virtually augmented robot (VAR) model that embeds AR functionality into telepresence, through the use of augmentation and facial recognition sub-models and algorithms. Experimental results show that using pulse width modulation (PWM), duty cycles, digital signals and accelerometer angles provides an approach for stabilising robot motion and the transference of human to robot head movement. The results also show the benefit of having a robot model with intrinsic AR functionality, as it offers a greater level of immersion and visual control of an environment.

Track fasteners play a pivotal role in infrastructure inspection for high-speed rail. Yet, images taken by drones often capture shadows cast by electrical towers flanking the high-speed rail tracks. These shadows can hinder the visibility of the track fasteners, thereby impacting detection efficiency and accuracy considerably. The present paper introduces an end-to-end shadow removal algorithm, rooted in generative adversarial network training. The comprehensive network framework is segmented into three sub-networks: pseudo-mask generation, shadow removal, and result refinement. We have integrated a Fourier convolutional residual module to bolster the feature extraction capability of the generator network. This integration ensures the network retains a global receptive field, even in its more superficial layers. By employing an overall weighted loss function, we enhance the quality of the images produced without shadows. Further, a perceptual loss function has been incorporated to retain the structural information of objects, setting the stage for subsequent defect detection. Our results highlight that Pse-ShadowNet adeptly eradicates fastener shadows while maintaining vital visual features, including object position, structure, texture, edges, and other key visual elements. Consequently, the reconstructed images are detailed and showcase superior image quality.

Human Action Recognition is widely used in multiple fields to recognize activities and extract spatial and temporal information. This paper focuses on analyzing various methods as well as provides extensive knowledge of the foundational concepts of the HAR. A dataset is also very crucial for any research study so we discussed the popular dataset and its features in this paper.

Long-term continuous time series of SO2 emissions are considered critical elements of both volcano monitoring and basic research into processes within magmatic systems. One highly successful framework for computing these fluxes involves reconstructing a representative time-averaged SO2 plume from which to estimate the SO2 source flux. Previous methods within this framework have used ancillary wind datasets from reanalysis or numerical weather prediction (NWP) to construct the mean plume and then again as a constrained parameter in the fitting. Additionally, traditional SO2 datasets from ultraviolet (UV) sensors lack altitude information which must be assumed to correctly calibrate the SO2 data and to capture the appropriate NWP wind level which can be a significant source of error. We have made novel modifications to this framework which do not rely on prior knowledge of the winds and therefore do not inherit errors associated with NWP winds. To perform the plume rotation, we modify a rudimentary computer vision algorithm designed for object detection in medical imaging to detect plume-like objects in gridded SO2 data. We then fit a solution to the general time-averaged dispersion of SO2 from a point source. We demonstrate these techniques using SO2 data generated by a newly developed probabilistic layer height and column loading algorithm designed for the Cross-track Infrared Sounder (CrIS), a hyperspectral infrared sensor aboard the Joint Polar Satellite System’s Suomi-NPP and NOAA-20 satellites. This SO2 data source is best suited to flux estimates at high-latitude volcanoes and at low-latitude, but high-altitude volcanoes. Of particular importance, IR SO2 data can fill an important data gap in the UV-based record: estimating SO2 emissions from high-latitude volcanoes through the polar winters when there is insufficient solar backscatter for UV sensors to be used.

The article describes a classification solution for amber stones. The problem of classifying amber is known for a long time among jewelers and artisans of amber art. Existing solutions can classify amber pieces according to color, but a need to classify by shape and texture is not satisfied up to now. The proposed solution is capable of classifying the gemstones according to a shape. Amber can be considered as a specific object since the form is difficult to define unambiguously. Data for amber experiments was gathered from amber art craftsmen. In the proposed solution amber form can be classified into 10 different classes (7 classes chosen during the experiment).

Ancient numismatics, that is, the study of ancient currencies (predominantly coins), is an interesting domain for the application of computer vision and machine learning, and has been receiving an increasing amount of attention in recent years. Notwithstanding the number of articles published on the topic, the variety of different methodological approaches described, and the mounting realisation that the relevant problems in the field are most challenging indeed, all research to date has entirely ignored one specific, readily accessible modality: colour. Invariably, colour is discarded and images of coins treated as being greyscale. The present article is the first one to question this decision (and indeed, it is a decision). We discuss the reasons behind the said choice, present a case why it ought to be reexamined, and in turn investigate the issue for the first time in the published literature. Specifically, we propose two new colour-based representations specifically designed with the aim of being applied to ancient coin analysis, and argue why it is sensible to employ them in the first stages of the classification process as a means of drastically reducing the initially enormous number of classes involved in type matching ancient coins (tens of thousands, just for Ancient Roman Imperial coins). Furthermore, we introduce a new data set collected with the specific aim of denomination-based categorisation of ancient coins, where we hypothesised colour could be of potential use, and evaluate the proposed representations. Lastly, we report surprisingly successful performances which goes further than confirming our hypothesis—rather, they convincingly demonstrate a much higher relevant information content carried by colour than even we expected. Thus we trust that our findings will be noted by others in the field and that more attention and further research will be devoted to the use of colour in automatic ancient coin analysis.

The study addresses the challenge of efficiently peeling pineapples, which have a distinct elliptical form, thick skin, and small eyes that are difficult to detect with conventional automated methods. This results in significant flesh waste. To improve the process, the researchers developed an integrated system combining an enhanced BlendMask method, termed SAAF-BlendMask, and a PCP method. SAAF-BlendMask improves the detection of small pineapple eyes, while PCP ensures accurate posture adjustment for precise path planning. The system uses 3D vision and deep learning technologies, achieving an average precision (AP) of 73.04% and a small object precision (APs) of 62.54% in eye detection, with a path planning success rate reaching 99%. The fully automated electromechanical system was tested on 110 real pineapples, demonstrating a reduction in flesh waste by 11.7% compared to traditional methods. This study highlights the potential of advanced machine vision and robotics in enhancing the efficiency and precision of food processing.

Imaging systems mounted to ground vehicles are used to image fruit tree canopies for estimation of fruit load, but frequently need correction for fruit occluded by branches, foliage or other fruits. This can be achieved using an orchard ‘occlusion factor’, estimated from a manual count of fruit load on a sample of trees (referred to as the reference method). It was hypothesised that canopy images could hold information related to the number of occluded fruit. Five approaches to correct for occluded fruit based on canopy images were compared using data of three mango orchards in two seasons. However, no attribute correlates to the number of hidden fruit were identified. Several image features obtained through segmentation of fruit and canopy areas, such as the proportion of fruit that were partly occluded, were used in training Random forest and multi-layered perceptron (MLP) models for estimation of a correction factor per tree. In another approach, deep learning convolutional neural networks (CNNs) were directly trained against harvest fruit count on trees. The supervised machine learning methods for direct estimation of fruit load per tree delivered an improved prediction outcome over the reference method for data of the season/orchard from which training data was acquired. For a set of 2017 season tree images (n=98 trees), a R2 of 0.98 was achieved for the correlation of the number of fruits predicted by a Random forest model and the ground truth fruit count on the trees, compared to a R2 of 0.68 for the reference method. The best prediction of whole orchard (n = 880 trees) fruit load, in the season of the training data, was achieved by the MLP model, with an error to packhouse count of 1.6% compared to the reference method error of 13.6%. However, the performance of these models on new season data (test set images) was at best equivalent and generally poorer than the reference method. This result indicates that training on one season of data was insufficient for the development of a robust model. This outcome was attributed to variability in tree architecture and foliage density between seasons and between orchards, such that the characters of the canopy visible from the interrow that relate to the proportion of hidden fruit are not consistent. Training of these models across several seasons and orchards is recommended.

Attempts to use computers to aid in the detection of breast malignancies date back more than 20 years. Despite significant interest and investment, this has historically led to minimal or no significant improvement in performance and outcomes with traditional computer aided detection. However, recent advances in artificial intelligence and machine learning are now starting to deliver on the promise of improved performance. There are at present more than 20 FDA approved AI applications for breast imaging, but adoption and utilization are widely variable and overall low. Breast imaging is unique and has aspects that create both opportunities and challenges for AI development and implementation. Breast cancer screening programs worldwide rely on screening mammography to reduce the morbidity and mortality of breast cancer and many of the most exciting research and available AI applications focus on cancer detection for mammography. There are however multiple additional potential applications for AI in breast imaging including decision support, risk assessment, breast density quantitation, workflow and triage, quality evaluation, response to neoadjuvant chemotherapy assessment, and image enhancement. In this review the current status, availability, and future directions of investigation of these applications are discussed, as well as the opportunities and barriers to more widespread utilization.

This paper presents a methodology to detect the erosion and progradation effects in the Atrato river delta, located in the Gulf of Urabá in Colombia using SAR (Synthetic Aperture Radar) data. Other authors have studied this area by using optical images, encountering difficulties given the persistent presence of clouds in this region. SAR imagery is rescaled, despeckled, composed, classified, registered and processed to obtain a heatmap that shows the overall changes during eight years (2016–2023) in one single image. Since this technology uses microwave active signals, the presence of clouds and other environmental phenomena was not a problem, and all the images from the selected dates could be processed and analyzed. Also, detection of the dynamic processes was obtained for every period considered in this study with respect to the previous one in time, calculating areas and detecting the regions subject to the erosion and progradation processes. The multitemporal analysis performed found that the newly created mouth is the most active area for these processes, coinciding with other studies. A comparison of these findings with the Oceanic Niño Index (ONI) showed a relative delayed coupling to the erosion process and a coupling of progradation with dry and wet seasons.

Table detection is a preliminary step in extracting reliable information from tables in scanned document images. We present CasTabDetectoRS, a novel end-to-end trainable table detection framework that operates on Cascade Mask R-CNN, including Recursive Feature Pyramid network and Switchable Atrous Convolution in the existing backbone architecture. By utilizing a comparatively lightweight backbone of ResNet-50, this paper demonstrates that superior results are attainable without relying on pre and post-processing methods, heavier backbone networks (ResNet-101, ResNeXt-152), and memory-intensive deformable convolutions. We evaluate the proposed approach on five different publicly available table detection datasets. Our CasTabDetectoRS outperforms the previous state-of-the-art results on four datasets (ICDAR-19, TableBank, UNLV, and Marmot) and accomplishes comparable results on ICDAR-17 POD. Upon comparing with previous state-of-the-art results, we obtain a significant relative error reduction of 56.36%, 20%, 4.5%, and 3.5% on the datasets of ICDAR-19, TableBank, UNLV, and Marmot, respectively. Furthermore, this paper sets a new benchmark by performing exhaustive cross-datasets evaluations to exhibit the generalization capabilities of the proposed method.

Intracortical microelectrodes that can be implanted have the ability to capture fast-changing neuron action potentials in the living brain. Despite this potential, many obstacles must still be overcome for reliable, long-term, high-quality recordings and accurate analysis of brain activity. These challenges include improving recording quality, enhancing recording stability, increasing recording capacity, and adding multi-functional features. One potential approach to enhance implantable microelectrodes involves the advancement of materials, refinement of implantation methodologies, and augmentation of the number of sites for recording. However, these challenges are still being actively researched, and advancements in microelectrode technology are underway. The difficulties with implantable microelectrodes are evaluated, and solutions are presented from different perspectives. The latest advancements in microelectrode technology are reviewed and condensed, and future possibilities are explored.

Accurate crop diseases classification is crucial for ensuring food security and enhancing agricultural productivity. However, existing crop disease classification algorithms primarily focus on a single image modality and typically require a large number of samples. Our research counters these issues by using pre-trained Vision-Language Models (VLMs), which enhances multimodal synergy for better crop disease classification than traditional unimodal approaches. Firstly, we apply the multimodal model Qwen-VL to generate meticulous textual descriptions for representative disease images selected through clustering from the training set, which will serve as prompt text for generating classifier weights. Compared to using solely the language model for prompt text generation, this approach better captures and conveys fine-grained and image-specific information, thereby enhancing prompt quality. Secondly, we integrate the cross-attention and the SE (Squeeze-and-Excitation) attention into the training-free mode VLCD (Vision-Language model for Crop Diseases classification) and the training-required mode VLCD-T (VLCD-Training) respectively for prompt text processing, enhancing classifier weights by emphasizing key text features. Experimental outcomes conclusively prove our method’s heightened classification effectiveness in few-shot crop disease scenarios, tackling data limitations and intricate disease recognition issues. It offers a pragmatic tool for agricultural pathology and reinforces the smart farming surveillance infrastructure.

Recent progress in microfabrication technique allowed the rapid development of neural implants. They are getting categorized as effective tools for clinical practice, especially to treat traumatic and neurodegenerative disorders. Microelectrode arrays already have been used in numerous neural interface devices. Basically, almost all neural implants have been developed based on BCI (Brain Computer Interface) system. When BCI system falls under invasive technique, it is referred as BMI or Brain Machine Interface. BMIs hold promises for neurorehabilitation of motor and sensory function, cognitive state evaluation and treatment of neurological chaos. A directed overview of the field of neural implants is discussed in this article. The aim of this review is to give a brief introduction of neural prosthetics as well as their exciting applications in treating neurological disorders and a deep discussion on their functionality are mentioned. BCI system and their different types, their functionality, their pros and cons, how other neural implants developed, and their present status have been covered. Different possibilities and possible future of deep brain stimulation (DBS), Neuralink, motor and sensory neural prosthetics are further discussed.

This work addresses the challenge of EEG visual multiclass classification into 40 classes for Brain-Computer interface applications, by using deep learning architectures. The visual multiclass classification approach offers BCI applications a significant advantage, since it allows the supervision of more than one BCI interaction, considering that each class label supervises a BCI task. However, because of the nonlinearity and nonstationarity of EEG signals, performing multiclassifiers based on EEG features remains a significant challenge for BCI systems. In the present work, mutual information-based discriminant channel selection and Minimum-Norm Estimate algorithms are implemented to select discriminant channels and enhance EEG data. Hence, deep EEGNet and Convolutional-recurrent neural networks are implemented separately to classify EEG data of image visualization into 40 labels. By using the k-fold cross-validation approach, average classification accuracies of 94.8% and 89.8% were obtained by implementing the aforementioned network architectures, respectively. Satisfactory results obtained with this method offer a new implementation opportunity for multi-task BCI applications by utilizing a reduced number of channels (&lt;50%), compared to those presented in the related literature where the whole set of channels is used.

Identification of plants is a challenging task which aims to identify the family, genus, and species level according to morphological features. Automated deep learning-based computer vision algorithms are widely used for identifying plants and can help users to narrow down the possibilities. However, numerous morphological similarities between and within species make the classification difficult. In this paper, we tested a custom convolution neural network (CNN) and vision transformer (ViT) based models using the PyTorch framework to classify plants. We used a large dataset of 88K and 16K images for classifying plants at genus and species levels respectively. Our results show that for classifying plants at the genus level, ViT models perform better compared to CNN-based models ResNet50 and ResNet-RS-420, and other state-of-the-art CNN-based models suggested in previous studies on a similar dataset. The ViT model achieved top accuracy of 83.3% for classifying plants at the genus level. ViT models also perform better for classifying plants at the species level compared to CNN-based models ResNet50 and ResNet-RS-420, with a top accuracy of 92.5%. We show that the correct set of augmentation techniques plays an important role in classification success.

Image clustering and classification are fundamental tasks in computer vision, critical for applications overlap image retrieval, object recognition, and image classification. However, the robustness of clustering algorithms against adversarial attacks remains interesting topic. In this paper, we investigate how adversarial attacks on image classification algorithms impact Image Clustering, similarity obtained using the Dot Product, KNN, HNSW algorithms and model Gradient-Weighted Class Activation Mapping (Grad-CAM). In our work was proposed a targeted study of the impact of adversarial attacks on the clustering ability of ResNet50 under various adversarial scenarios. Was used ResNet50 as the basis for the experiments, a widely used architecture known for its effectiveness in image classification. This network was subjected to various adversarial attacks in order to understand how these perturbations affect its clustering capabilities. By thoroughly examining the resultant clustering outcomes under different attack scenarios, we aim to uncover vulnerabilities and nuances inherent in clustering algorithms and similarity metrics when confronted with adversarial input.

Within environmental perception, automatic navigation and object detection, computer vision is a crucial and demanding field with many applications in modern industries, such as multi-target long-term visual tracking in automated production, defect detection, and driverless robotic vehicles with sensing ability. The performance of computer vision has greatly improved recently thanks to developments in deep learning algorithms and hardware computing capabilities, which has spawned the creation of a large number of related applications. This paper presents the results of a detailed review of over 50 papers published over the course of two decades (1999–2024), with a primary focus on the technical advancement of computer vision. To elucidate the foundational principles, an examination of typical visual algorithms based on traditional correlation filtering was initially conducted. Subsequently, a comprehensive overview of the most recent advancements in deep learning-based computer vision techniques was compiled. Furthermore, a comparative analysis between conventional and novel algorithms was undertaken to discuss the future trends and directions of computer vision. Lastly, the feasibility of employing visual SLAM (Simultaneous Localization and Mapping) algorithms in the context of autonomous vehicles with sensing capability was explored. Furthermore, we explored a thorough and efficient architecture for utilizing these techniques in autonomous robotic vehicles into unmanned green and low-carbon smart factories, underlining novel advances as well as potential prospects for future research.

Reconstructing a 3D model of a scene from a set of multiple views poses a significant challenge in the field of computer vision. The advent of NeRF has marked a major breakthrough in this domain. However, there is a need to extend the capabilities of the NeRF model to enable editing tasks such as relighting and deformation, as well as to enhance its training efficiency. Neural reflectance fields introduce the concept of the reflectance equation into the NeRF framework to achieve relighting of the NeRF model. We leverage the TensoRF approach, which incorporates tensor decomposition and employs multiple tensors to store features, to expedite the training process of the NeRF model. Our novel method, called StensorR, combines the reflectance equation and tensor decomposition within the radiation field model framework. Differing from previous approaches, we employ a single tensor to store scene features and render the surface color of our scene using a simplified reflectance equation. This approach accelerates model convergence and enables relighting of the NeRF model. Experimental results demonstrate that our method achieves a 50% faster convergence rate compared to existing relighting radiation field models, while successfully enabling relighting and improving the quality of synthesized images from new viewpoints.

The contamination on the insulators may increase its surface conductivity and, as a consequence, electrical discharges occur more frequently, which can lead to interruptions in the power supply. To maintain reliability in the electrical distribution power system, components that have lost their insulating properties must be replaced. Identifying the components that need maintenance, is a difficult task as there are several levels of contamination that are hardly noticed during inspections. To improve the quality of inspections, this paper proposes to use the k-nearest neighbours (k-NN) to classify the levels of insulator contamination, based on the image of insulators at various levels of contamination simulated in the laboratory. Using computer vision features such as mean, variance, asymmetry, kurtosis, energy, and entropy are used for training the k-NN. To assess the robustness of the proposed approach, statistical analysis and a comparative assessment with well-consolidated algorithms such as decision tree, ensemble subspace, and support vector machine models are presented. The k-NN showed results of up to 85.17 % accuracy using the k-fold cross-validation method, with an average accuracy higher than 82 % for multi-classification of the contamination of the insulators, being superior to the compared models.

We have an unprecedented ability to map the Earth’s surface as deep learning technologies are applied to an abundance of high-frequency Earth observation data. Simple, free, and effective methods are needed to enable a variety of stakeholders to use these tools to improve scientific knowledge and decision making. Here we present a trained U-Net model that can map and delineate ground mounted solar arrays using publicly available Sentinel-2 imagery, and that requires minimal data pre-processing and no feature engineering. By using label overloading and image augmentation during training, the model is robust to temporal and spatial variation in imagery. The trained model achieved a precision and recall of 91.5% each and an intersection over union of 84.3% on independent validation data from two distinct geographies. This generalizability in space and time makes the model useful for repeatedly mapping solar arrays. We use this model to delineate all ground mounted solar arrays in North Carolina and the Chesapeake Bay watershed to illustrate how these methods can be used to quickly and easily produce accurate maps of solar infrastructure.

This paper presents a state machine based architecture which enhances flexibility and reusability of industrial robots, more concretely dual-arm multisensor robots. The proposed architecture, in addition to allowing absolute control of the execution, eases the programming of new applications by increasing the reusability of the developed modules. Through an easy-to-use graphical user interface, operators are able to create, modify, reuse and maintain industrial processes increasing the flexibility of the cell. Moreover, the proposed approach is applied in a real use case in order to demonstrate its capabilities and feasibility in industrial environments. A comparative analysis is presented for evaluating presented approach versus traditional robot programming techniques.

We propose a computer-generated hologram method for near-eye virtual reality and augmented reality 3D display. A 3D object located near the holographic plane is projected onto a projection plane to obtain a plurality of projected images with different angles. The hologram is calculated by superposition of projected image convolution with a specific PSF through the interference of reference wave with object wave. Holographic 3D display systems with LED as illumination, 4f optical filtering system and lens as eyepiece for near-eye VR display and HOE as combiner for near-eye AR display are designed. The results show that the proposed method is about 38 times faster than the conventional point cloud method. The proposed method is flexible to produce speckle noise free high-quality VR and AR images with efficient focus and defocus capabilities.

A fast calculation method for the full parallax high-resolution hologram is proposed based on the elemental light field image (EI) rendering. A 3D object located near the holographic plane is firstly rendered as multiple EIs with a pinhole array. Each EI is interpolated and multiplied by a divergent sphere wave and interfered with a reference wave to form a hogel. Parallel acceleration is used to calculate the high-resolution hologram because calculation of each hogel is independent. A high-resolution hologram with the resolution of 20,0000×20,0000 pixels is calculated only within 8 minutes. Full parallax high-resolution 3D displays are realized by optical reconstructions.

ORB (Oriented FAST and Rotated Brief) features are the most commonly used features in visual SLAM and visual odometry that have high computational speed, and rotation scale invariance. However, due to the homogenization of ORB features, the corner properties of specific features are poor in some artificial environments, and it can lead to poor matching performance when the environmental texture is not rich. The underlying reason is that the pixel grayscale changes in the less-richly textured regions on the image are not obvious, leading to a certain degree of mis-matching in the matching process. At the same time, when the camera motion speed is low, there is much overlap between adjacent frames. This results in minimal or almost no changes in the projection of feature points, and the system is highly sensitive to errors, In this case, even small errors can cause significant fluctuations in the calculation results. In response to the above issues, an improved feature point method for stereo vision mileage calculation is proposed to solve the problem of system stability deterioration when cameras move at low speeds in artificial envi-ronments. Firstly, weight calculation based on different texture regions is used to solve the problem of low corner properties of feature points caused by environmental texture differences. Then, keyframes are used for motion model estimation to improve the system's stability under low-speed motion. The experimental results of the dataset showed that the key frame rate was optimal between 10% and 12%, and the positioning accuracy was higher than these open-source systems compared with three common open-source VO systems.

Efficient processing of end-of-life lithium-ion batteries of electric vehicles is important and a pressing challenge for a circular economy. Regardless of whether the processing strategy is recycling, repurposing or remanufacturing, the first processing step would usually involve disassembly. As battery disassembly is a dangerous task, efforts have been made to robotise it. In this paper, a robotic disassembly platform using four industrial robots is proposed to automate the non-destructive disassembly of a plug-in hybrid electric vehicle battery pack into modules. The work was conducted as a case study to demonstrate the concept of autonomous disassembly of an electric vehicle battery pack. A two-step object localisation method based on visual information is used to overcome positional uncertainties from different sources and is validated by experiments. Also, the unscrewing system is highlighted, and its functions, such as handling untightened fasteners, loosening jammed screws, and changing the nutrunner adapters with square drives, are detailed. Furthermore, the time required for each operation is compared with that taken by human operators. Finally, the limitations of the platform are reported and future research directions are suggested.

Quality control is a critical component in industrial manufacturing, directly influencing efficiency, product reliability, and ultimately, customer satisfaction. In the dynamic environment of industrial manufacturing, traditional methods of inspection may not adequately meet the evolving complexity, necessitating innovative approaches to bolster precision and productivity. In this study, we explore the application of mixed reality (MR) technology for real-time quality control in the assembly process. Our methodology involved the integration of smart glasses with a server-based image recognition system, designed to conduct real-time component analysis. The innovative aspect of our study lies in the harmonization of MR and computer vision algorithms, providing immediate visual feedback to inspectors and thereby improving the speed and accuracy of defect detection. YOLOv8 have been adopted in this study for detection object model. The project implementation occurred in a controlled environment to enable a comprehensive evaluation of the system functionality, the identification of possible problems and improvements in the system performance. The results indicated the viability of mixed reality as a powerful tool for enhancing traditional inspection processes. The fusion of MR and computer vision offers possibilities for future advancements in industrial quality control, paving the way for more efficient and reliable manufacturing ecosystems.

Studying and understanding the behavior of people and vehicles on public roads can be of utmost importance for supporting the activities of many institutional stakeholders. It may allow automated supervision of the ongoing situation in a given place, with warnings or alarms raised in case of anomalies. It may be used to plan their interventions on road and town organization. It may provide them with advanced support to decision-making. The number of involved entities and places to manage makes it infeasible to manually handle all the traffic-related tasks. Moreover, the complexity of the tasks to be carried out requires the adoption of advanced approaches. Many AI solutions are nowadays mature to support these requirements. In some cases, the motivations and objectives of traffic management require the AI outcomes to be understandable, interpretable and explainable. In this paper, we propose TrAnSIT (TRaffic ANalysis Supervision and Interpretation Tool), an AI-based framework that combines several modules, each aimed at tackling a specific traffic-related task, so as to cover a wide landscape of traffic-related issues, from overall urban or suburban traffic management to surveying specific road segments that fall under the scope of one camera. Most of these modules are based on AI techniques that support a human-level understanding of the outcomes.

The abundance of information in academic articles, reports, and studies can make it challenging for researchers to gain insights from the existing literature. To address this issue, there is a growing demand for tools that can help researchers effectively parse and analyze large volumes of data. One such tool is DataDiscoveryLab, a software system that utilizes computer vision algorithms and NLP techniques to parse academic articles into text and figures, creating three separate databases. These databases allow researchers to quickly identify articles that may be relevant to their research questions, gain a deeper understanding of the research presented, and analyze visual data. The integration of article mining and computer vision in the DataDiscoveryLab software system provides researchers with a powerful tool for navigating the vast amount of scientific literature available today. Yet, as we will discuss in the latter papers these databases’ purpose is to create a bridge between researchers’ data and practically unlimited scientific publications. Yet, in this article, we will discuss how we plan to do that, and our efforts on integrating deep learning modes. After all, unlike already existing AI models, DataDiscoveryLab can be their combination and the first Generative AI in academia that can encompass every part of the natural sciences.

As machine learning and computer vision techniques and methods continue to advance, the collection of naturalistic traffic data from video feeds is becoming more and more feasible. That is especially true for the case of bicycles, for which the collection of naturalistic data is not achievable in the traditional vehicle approach. This study describes a research effort that aims to extract naturalistic cycling data from a video dataset for use in safety and mobility applications. The used videos come from a dataset collected in a previous Virginia Tech Transportation Institute study in collaboration with SPIN in which continuous video data at a non-signalized intersection on the Virginia Tech campus was recorded. The research team applied computer vision and machine learning techniques to develop a comprehensive framework for the extraction of naturalistic cycling trajectories. In total, this study resulted in the collection and classification of 619 bicycle trajectories based on their type of interactions with other road users. The results confirm the success of the proposed methodology in relation to extracting the locations, speeds, and accelerations of the bicycles at a high level of precision. Furthermore, preliminary insights into the acceleration and speed behavior of bicyclists around motorists are determined.

Animal welfare is not only an ethically important consideration in good animal husbandry, but can also have a significant effect on an animal’s productivity. The aim of this paper is to show that a reduction in animal welfare, in the form of increased stress, can be identified in pigs from frontal images of the animals. We train a Convolutional Neural Network (CNN) using a leave-one-out design and show that it is able to discriminate between stressed and unstressed pigs with an accuracy of >90% in unseen animals. Grad-CAM is used to identify the animal regions used, and these support those used in manual assessments such as the Pig Grimace Scale. This innovative work paves the way for further work examining both positive and negative welfare states with a view to the development of an automated system that can be used in precision livestock farming to improve animal welfare.

This study investigated the capability of vocal-identity-cloning Artificial Intelligence (AI) to encode human-specific confident, doubtful, and neutral-intending emotive states. Linear mixed-effects models and machine learning classification with eXtreme Gradient Boosting were employed to examine the underlying acoustic signatures from 2,700 audio clips, comprising of sentences spoken by human speakers and two sets of equivalences (AI-Geography/AI-Trivia, based on the trained text) generated by voice-cloning models designed to clone human speakers’ identities. Compared with neutral-intending voice, human speakers lengthened their vocal tract, raised the fundamental frequency, and increased Chroma constant-Q transform when they intended to be confident; An opposite pattern was shown when they intended to be doubtful. The two sets of AI sounds displayed a similar pattern to human speech, suggesting a shared mechanism for encoding vocal expression across sources. The 1,000 times training-testing classification models reported an in-group advantage for AI sources. The algorithms, trained on AI-Geography/AI-Trivia, resulted in higher accuracies when tested within these AI sources than when tested on human audio. All between-source classifications reported above-chance-level (1/3) accuracies. These findings highlighted that voice-cloning AI, the widely used conversational agent, can learn and generate human-specific vocally-expressed confidence.

Это исследование сосредоточено на разработке системы искусственного зрения для гибкого дельта-робота-манипулятора и ее интеграции с межмашинной связью для оптимизации взаимодействия устройств в реальном времени. Эта интеграция направлена ​​на увеличение скорости роботизированной системы и улучшение ее общей производительности. Предлагаемое сочетание системы искусственного зрения с межмашинной связью позволяет обнаруживать и распознавать цели с высокой точностью в режиме реального времени в ограниченном пространстве, предназначенном для позиционирования, дальнейшей локализации и выполнения производственных процессов, таких как сборка или сортировка деталей. . В данном исследовании RGB-изображения используются в качестве входных данных для алгоритма MASK-R-CNN, а результаты обрабатываются в соответствии с особенностями прототипа дельта-манипулятора робота. Данные, полученные от MASK-R-CNN, адаптированы для использования в системе управления дельта-роботом с учетом ее уникальных характеристик и требований к позиционированию. Технология M2M позволяет манипулятору робота быстро реагировать на изменения, например, на перемещение объектов или изменение их положения, что имеет решающее значение для задач сортировки и упаковки. Система была протестирована в условиях, близких к реальным, чтобы оценить ее производительность и надежность.

Deep eutectic solvents (DESs) are one of the most rapidly evolving types of solvents, appearing in a broad range of applications such as nanotechnology, electrochemistry, biomass transformation, pharmaceuticals, membrane technology, biocomposite development, modern 3D-printing, and many others. The range of their applicability continues to expand, which demands the development of new DESs with improved properties. To do so requires an understanding of the fundamental relationship between the structure and properties of DESs. Computer simulation and machine learning techniques provide a fruitful approach as they can provide predictions, reveal physical mechanisms and readily be linked to experiments. This review is devoted to the computational research of DESs and describes technical features of DES simulations and the corresponding perspectives on various DES applications. The aim is to demonstrate the current frontiers of computational research of DESs and discuss future perspectives.

Nowadays, emerging technologies have changed the places of work through computers and ICT tools, which have revolutionized teaching and learning environments in different ways. In spite of the fact that computers as ICT tools have become part and progressively instrument for teachers and instructors used in teaching and learning, but most educators can't incorporate them into their teaching and learning process, which results in students being ill-equipped or lacking some necessary skills in the world of work, which leads to low performance and poor handling of tools whose lead to low production. To tackle this issue, it is essential to develop the technical and vocational education and training (TVET) system by determining the quality of technical and vocational education (TVE) teachers. In this paper, the literature concerning the competence required by TVET teachers towards computer-related instructional tech-nology for classroom teaching and learning was examined through the technological pedagogical content knowledge (TPACK) model. Sixty (60) questionnaires were ad-ministered to TVE teachers within six technical colleges of education in north-eastern Nigeria. The data was quantitatively analyzed using traditional linear models, namely multilinear regression (MLR) and artificial intelligence (AI) models, namely artificial neural networks (ANN) and adaptive neuro-fuzzy inference systems (ANFIS), which were developed using MATLAB 9.3 (R2019a), while the classical linear MLR model was developed using SPSS software. The results from our classical study indicated that TVE teachers are competent in computer and some instructional technology usage and show a high correlation between competence and teaching experience and a lower correlation between competence and gender. The goodness of fit shows a good fit of the model. Future studies should examine the application of other linear and non-linear AI techniques.

In the last decades we have experienced great social, cultural, and educational changes, spending more time performing tasks in near vision and less hours in outdoor activities, having more overexposure to digital screens, especially the youngest and increasingly from younger ages. It is common to see cases when performing tasks in near vision the posture or position of the head is not adequate, especially the youngest, being essential to have a correct posture of the head to avoid visual, muscular, or articular problems. We present a procedure to try to control the correct position of the head in real time when performing close-up activities. The aim is to detect the inclination of the head and to know if a bad position is adopted due to a visual problem or simply to inadequate visio-postural habits, warning us of the postural anomaly to correct it.

To date, the development of unmanned technologies using autonomous underwater vehicles (AUV) has become an urgent demand for solving the problem of inspection of industrial sub-sea structures. A key issue here is precise localization of AUVs relative to underwater objects. However, the well-known challenges of recognizing and coordinating underwater objects (impossibility to use GPS, undercurrents, etc.) do not allow high-precision navigation if based solely on a standard suite of AUV navigation tools (sonars, etc.). An alternative is a technology of processing of optical images that, at short distances, can provide higher accuracy of AUV navigation compared to a technology of acoustic measurement processing. In this study, in the context of the problem of subsea production system inspection, we propose a method to recognize underwater objects and provide coordinate reference of AUV to them, based on stereo-image processing. Its distinctive features are the use of a non-standard technique for generating a geometric model of an object from its views (foreshortening) taken from positions of a pre-made overview trajectory, the use of various characteristic geometric elements when recognizing objects, and the original algorithms for comparing visual data of the inspection trajectory with an a priori model of the object.

Identifying punch types as part of punch kinematic analysis is a key component for both coaches and players, providing critical insights into the variety and effectiveness of punches, which are essential for refining overall performance. Existing research indicates that punch type classification typically relies on data from either wearables or video data. Both approaches have their merits and demerits but require a significant amount of labeled data for modeling the classification algorithm. In this work, we propose a novel approach that significantly reduces the labeling effort by one-sixth by leveraging video data to enhance the training information of sensor data. This enriched training phase enables more accurate punch classification with reduced labeled data. Subsequently, the real-time punch identification and classification use only the wearable data. On a real dataset, with 15% of the labeling efforts, our approach achieved an accuracy rate of 91.41% for rear hand punch recognition and 91.91% for lead hand punch recognition, along with 92.33% and 94.56% for punch classification, respectively. Through the development of our Smart Boxer system, we aim to revolutionize punch analytics in boxing, providing comprehensive insights while enhancing the overall training and competitive experience for athletes and enriching fan engagement with the sport.

In recent years, Synthetic Aperture Radar (SAR) has received attention in the Automatic Target Recognition (ATR) field due to its ability to acquire images both day and night, combined with the superior resolution offered by the latest sensors. However, despite advances, there remains a deficiency in dataset quality and quantity. This has necessitated the use of simulated data to overcome challenges associated with obtaining large quantities of high-quality images of targets and their subsequent labeling. In this study, we trained a Convolutional Neural Network (CNN) using a simulated dataset, subsequently employing it to classify real images from the MSTAR dataset. Traditional training methodologies often struggle with generalization when confronted with domain disparities between simulated and real data. To mitigate this, we exploited Margin Disparity Discrepancy (MDD), a domain adaptation technique not previously explored in SAR context. Findings from this study showcased a 13% increase in classification accuracy and improved generalization using MDD. Through Explainable AI (XAI) techniques such as t-SNE and other metrics like Kullback Leibler Divergence (KLD), we revealed a more distinct alignment of feature spaces between the two domains after domain adaptation, emphasizing MDD’s role in enhancing model accuracy and generalization.

Manual visual inspections typically conducted after an earthquake are high-risk, subjective, and time-consuming. Delays from inspections often exacerbate the social and economic impact of the disaster on affected communities. Rapid and autonomous inspection using images acquired from unmanned aerial vehicles offer the potential to reduce such delays. Indeed, a vast amount of re-search has been conducted toward developing automated vision-based methods to assess the health of infrastructure at the component and structure level. Most proposed methods typically rely on images of the damaged structure, but seldom consider how the images were acquired. To achieve autonomous inspections, methods must be evaluated in a comprehensive end-to-end manner, incorporating both data acquisition and data processing. In this paper, we leverage recent advances in computer generated imagery (CGI) to construct a 3D synthetic environment for simulation of post-earthquake inspections that allows for comprehensive evaluation and valida-tion of autonomous inspection strategies. A critical issue is how to simulate and subsequently render the damage in the structure after an earthquake. To this end, a high-fidelity nonlinear finite element model is incorporated in the synthetic environment to provide a representation of earthquake-induced damage; this finite element model, combined with photo-realistic rendering of the damage, is termed herein a physics-based graphics models (PBGM). The 3D synthetic en-vironment with PBGMs provide a comprehensive end-to-end approach for development and validation of autonomous post-earthquake strategies using UAVs, including: (i) simulation of path planning of virtual UAVs and image capture under different environmental conditions; (ii) au-tomatic labeling of captured images, potentially providing an infinite amount of data for training deep neural networks; (iii) availability of the ground truth damage state from the results of the finite-element simulation; and (iv) direct comparison of different approaches to autonomous as-sessments. Moreover, the synthetic data generated has the potential to be used to augment field datasets. To demonstrate the efficacy of PBGMs, models of reinforced concrete moment-frame buildings with masonry infill walls are examined. The 3D synthetic environment employing PBGMs is shown to provide an effective testbed for development and validation of autonomous vision-based post-earthquake inspections that can serve as an important building block for ad-vancing autonomous data to decision frameworks.

The relationship between the street environment and the health, education, mobility, and criminal behaviors of its citizens has long been investigated by economists, sociologists and urban planners. Home buyers were found to pay a premium for better street appearance. Prior studies considering streetscapes mainly focus on objective measures such as the number of nearby trees, the tree canopy area, or the view index of physical features such as greenery, sky or building. However, subjective perceptions may have complex or subtle relationships to physical features, individual physical features or simply summing them up do not capture people’s comprehensive perception. In contrast, this study proposed a new approach for the urban-scale application to quantify both subjectively and objectively measured streetscape scores for six important perception qualities, namely Greenness, Walkability, Safety, Imageability, Enclosure, and Complexity. Built on prior quantitative studies in urban design quality and emerging applications in deep learning and open source street view imagery for urban perceptions, we integrated existing frameworks to (1) effectively collect and evaluate both subjectively and objectively- measured perceptions; (2) investigate the coherence and divergence in ML-predicted subjective scores and formula-derived objective scores; and (3) compare their effects in affecting house prices taking Shanghai as a case study using a large-scale dataset on home transactions. The results implied: first, the percentage increase in sales price attributable to street scores is significant for both subjective and objective measurements. In general subjective scores explained more variance over structural attributes and objective scores in hedonic price model. Particularly, objective Greenness, subjective Safety and Imageability scores positively affected house prices. Second, for Greenness and Imageability scores, the subjective and objective measures exhibited opposite signs in affecting house prices, which implied that there might be mechanisms related to the psychological, social-demographical characteristics of street users that have not been fully incorporated by objective measures that taking view indices or recombination of them. In addition, certain objective measure might outperform subjective counterpart when the connotation of the perception is self-evident and not complicated, for example the Greenness. For those concepts were not familiar to the average person, subjective framework exhibits better performance. This is the first study comprehensively expanding hedonic price method with both subjectively and objectively measured streetscape qualities. It suggested that city authorities could levy a street environment tax to compensate the public budget invested in street environment where developers secured benefits from a price premium. This study enriches our understanding of the economic values of the subjective and objective measures street qualities. It sheds light on promising future study areas where the coherence and divergence of the two measurements should be further stressed.

In order to design efficient and sustainable office spaces and to automate lighting, heating and air circulation in these facilities, solving the challenge of occupancy prediction is crucial. In office spaces where large areas need to be observed, multiple sensors must be used for full coverage. In these cases, it is normally important to keep the costs low, but also to make sure that the privacy of the people who use such environments are preserved. Low-cost and low-resolution heat (thermal) sensors can be very useful to build solutions that address these concerns. However, they are extremely sensitive to noise artifacts which might be caused by heat prints of the people who left the space or by other objects which are either using electricity or exposed to sunlight. There are some earlier solutions for occupancy prediction that employ low-resolution heat sensors, however, they have not addressed nor compensate for such heat artifacts. Therefore, in this paper, we present a low-cost and low-energy consuming smart space implementation to predict the number of people in the environment based on whether their activity is static or dynamic in time. We use a low-resolution (8×8) and non-intrusive heat sensor to collect data from an actual meeting room. We propose two novel workflows to predict the occupancy; one based on computer vision and one based on machine learning. Besides comparing the advantages and disadvantages of these different workflows, we use several state-of-the-art explainability methods in order to provide a detailed analysis of the algorithm parameters and how the image properties influence the resulting performance. Furthermore, we analyze noise resources which affect the heat sensor data. We hope that our analysis brings light into understanding how to handle very low-resolution heat images in these environments. The presented workflows could be used in various domains and applications other than smart offices, where occupancy prediction is essential, e.g., for elderly care.

Research on Traffic Light Recognition (TLR) has grown in recent years, primarily driven by the growing interest in autonomous vehicles development. Machine Learning algorithms have been widely used to that purpose. Mainstream approaches, however, require large amount of data to properly work, and as a consequence, a lot of computational resources. In this paper we propose the use of Expert Instruction (IE) as a mechanism to reduce the amount of data required to provide accurate ML models for TLR. Given an image of the exterior scene taken from the inside of the vehicle, we stand the hypothesis that the picture of a traffic light is more likely to appear in the central and upper regions of the image. Frequency Maps of traffic light location were thus constructed to confirm this hypothesis. The frequency maps are the result of a manual effort of human experts in annotating each image with the coordinates of the region where the traffic light appears. Results show that EI increased the accuracy obtained by the classification algorithm in two different image datasets by at least 15%. Evaluation rates achieved by the inclusion of EI were also higher in further experiments, including traffic light detection followed by classification by the trained algorithm. The inclusion of EI in the PCANet achieved a precision of 83% and recall of 73% against 75.3% and 51.1%, respectively, of its counterpart. We finally presents a prototype of a TLR Device with that expert model embedded to assist drivers. The TLR uses a smartphone as a camera and processing unit. To show the feasibility of the apparatus, a dataset was obtained in real time usage and tested in an Adaptive Background Suppression Filter (AdaBSF) and Support Vector Machines (SVMs) algorithm to detect and recognize traffic lights. Results show precision of 100% and recall of 65%.

The purpose of this study is to ​​explore the use of a custom Transformer model in brain-computer interfaces (BCIs) that translate the neural activity present when an individual with limited verbal and fine-motor skills attempts to handwrite. As found in previous studies, Transformers have performed better than recurrent neural networks (RNNs) in translation tasks which are similar to its usage here in decoding neural signals into intended handwritten text. Due to the known benefits of a Transformer, the hypothesis was that the Transformer would show promise in the context of a BCI through the recorded metrics. The neural signals of a tetraplegic individual, when they attempted to handwrite, were provided by existing research. Four trials were conducted using data with or without augmentation which the model used when training to separately minimize training loss and validation loss. When comparing the results of a BCI with the implementation of a Transformer model with the original RNN BCI (the original data source), the Transformer model performed less favorably across all four trials. Although the results do not indicate that the Transformer model currently outperforms an RNN BCI, it is important to note that further testing of the model's capabilities (such as training it with a larger and more preferable dataset and/or for longer, comparing training times between the RNN and Transformer, and/or seeing how the Transformer is improved with an offline autocorrect feature) is necessary before determining whether Transformers can enhance communication in this manner.

The relative position of the orchard robot to the rows of fruit trees is an important parameter for achieving autonomous navigations. The current methods for estimating the position parameters between rows of orchard robots obtain low parameter accuracy, and to address this problem, this paper proposes a machine vision-based method for detecting the relative position of orchard robots and fruit tree rows. Firstly, the fruit tree trunk is identified based on the improved YOLOv4 model; secondly, the camera coordinates of the tree trunk are calculated from the principle of binocular camera triangulation, and the ground projection coordinates of the tree trunk are obtained through coordinate conversion; finally, the midpoints of the projection coordinates of different sides are combined and the navigation path is obtained by linear fitting with the least squares method, and the position parameters of the orchard robot are obtained through calculation. The experimental results show that the average accuracy and average recall of the improved YOLOv4 model for fruit tree trunk detection are 97.05% and 95.42%, respectively, which are 5.92 and 7.91 percentage points higher than those of the original YOLOv4 model. The average errors of heading angle and lateral deviation estimates obtained based on the method in this paper are 0.57° and 0.02 m. The method can accurately calculate heading angle and lateral deviation values at different positions between rows, and can provide a reference for autonomous visual navigation of orchard robots.

Neuromorphic computing, a brain inspired non-Von Neumann computing system, addresses the challenges posed by the Moore’s law memory wall phenomenon. It has the capability to increasingly enhance performance while maintaining power efficiency. Neuromorphic chip architecture requirements vary depending on the application and optimizing it for large-scale applications remains to be a challenge. Neuromorphic chips are programmed using spiking neural networks which provide them with important properties such as parallelism, asynchronism, and on-device learning. Widely used spiking neuron models include the Hodgkin-Huxley Model, Izhikevich model, integrate-and-fire model and spike response model. Hardware implementation platforms of the chip follow three approaches: analog, digital, or a combination of both. Each platform can be implemented using various memory topologies which interconnects with the learning mechanism. Current neuromorphic computing systems typically use the unsupervised learning spike timing-dependent plasticity algorithms. However, algorithms such as voltage-dependent synaptic plasticity have the potential to enhance performance. This review summarizes the potential neuromorphic chip architecture specifications and highlights which applications they are suitable for.

Surface defect detection is a crucial step in the process of automotive wheel production. However, the task possesses challenges due to complex background and a wide range of defect types. In order to detect the defects on the wheel surface accurately and quickly, this paper proposes a YOLOv5-based algorithm for automotive wheel surface defect detection. The algorithm trains and tests the YOLOv5s model using the self-created automotive wheel surface defect dataset, which contains four kinds of defects: linear, dotted, sludge, pinhole. The extensive experimental results demonstrate that the deep learning network trained by our method can achieve an average accuracy of 71.7% and 57.14 FPS. Our findings prove that this detection algorithm performs better than other common target detection algorithms and meets the real-time requirements of industrial applications.

It's challenging to achieve robust lane detection depending on single frame when considering complicated scenarios. In order to detect more credible lane markings by using sequential frames, a novel approach to fusing vision and Inertial Measurement Unit (IMU) is proposed in this paper. The hough space is employed as the space where lane markings are stored and it's calculated by three steps. Firstly, a basic hough space is extracted by Hough Transform and primary line segments are extracted from it. In order to measure the possibility about line segments belong to lane markings, a CNNs based classifier is introduced to transform the basic hough space into a probabilistic space by using the networks outputs. However, this probabilistic hough space based on single frame is easily disturbed. In the third step, a filtering process is employed to smooth the probabilistic hough space by using sequential information. Pose information provided by IMU is applied to align hough spaces extracted at different times to each other. The final hough space is used to eliminate line segments with low possibility and output those with high confidence as the result. Experiments demonstrate that the proposed approach has achieved a good performance.

Nowadays, brain signal processing is performed rapidly in various brain-computer interface (BCI) applications. Most researchers focus on developing new methods for the future or improving the basic implemented models to identify the optimum standalone feature set. Our research focuses on four ideas. One of them introduces future communication models, and the others are for improving old models or methods. These are: 1) new communication imagery model instead of speech imager using the mental task: Due to speech imagery is very difficult, and it is impossible to imagine sound for all of the characters in all of the languages. Our research introduces a new mental task model for all languages that call Lip-sync imagery. This model can use for all characters in all languages. This paper implemented two lip-sync for two sounds, characters or letters. 2) New combination Signals: Selecting an inopportune frequency domain can lead to inefficient feature extraction. Therefore, domain selection is so important for processing. This combination of limited frequency ranges proposes a preliminary for creating Fragmentary Continuous frequency. For the first model, two s intervals of 4 Hz as filter banks were examined and tested. The primary purpose is to identify the combination of filter banks with 4Hz (scale of each filter bank) from the 4Hz to 40Hz frequency domain as new combination signals (8Hz) to obtain well and efficient features using increasing distinctive patterns and decreasing similar patterns of brain activities.3) new supplement bond graph classifier for SVM classifier: When SVM linear uses in very noisy, the performance is decreased. But we introduce a new bond graph linear classifier to supplement SVM linear in noisy data. 4) a deep formula recognition model: it converts the data of the first layer into a formula model (formula extraction model). The main goal is to reduce the noise in the subsequent layers for the coefficients of the formulas. The output of the last layer is the coefficients selected by different functions in different layers. Finally, the classifier extracts the root interval of the formulas, and the diagnosis does based on the root interval. For all of the ideas achieved the results of implementing methods. The results are between 55% to 98%. Less result is 55% for the deep detection formula, and the highest result is 98% for new combination signals.

Learning programming is a complex and challenging task for many students. It in-volves both understanding theoretical concepts and acquiring practical skills. Hence, analyzing learners’ data from online learning environments alone fails to capture the full breadth of stu-dents’ actions if part of their learning process takes place elsewhere. Moreover, existing studies on learning analytics applied to programming education have mainly relied on frequency analysis to classify students according to their approach to programming or to predict academic achieve-ment. However, frequency analysis provides limited insights into the individual time-related characteristics of the learning process. The current study examines students’ strategies when learning programming, combining data from the learning management system and from an au-tomated assessment tool. To gain an in-depth understanding of students’ learning process as well as of the types of learners, we used learning analytics methods that account for the temporal order of learning actions. Our results show that students have special preferences for specific learning resources when learning programming, namely slides that support search, and copy and paste. We also found that videos are relatively less consumed by students, especially while working on programming assignments. Lastly, students resort to course forums to seek help only when they struggle.