Survey of Deep Learning Accelerators for Edge and Emerging Computing

1. Introduction

Artificial intelligence, and in particular deep learning, is becoming increasingly popular in edge devices and systems. Deep learning algorithms require significant amounts of computations ranging from a few million to billions of operations based on the depth of the Deep Neural Network (DNN) models, and thus, there is an urgent need to process these efficiently. As shown in Figure 1, two possible approaches for processing deep learning inference on edge devices are directly on the device using highly efficient processors, fog, or cloud computing. A key benefit of fog/cloud-based processing is that large, complex models can be run without overburdening the edge device. The drawbacks of this approach are the need for a reliable communications channel, communications cost, communications delay, and potential loss of privacy.

In situations where a rapid response is needed, privacy is paramount, or a reliable communications channel may not always be available, processing of the deep learning network on the edge device or system is the only option [1,2,3]. As a result, a large amount of academic and industrial research is being done to develop efficient deep learning edge processors [3]. Several companies have already announced or have started selling such processors. This paper provides details on these commercial deep learning edge processors and compares their performances based on manufacturer provided information. Additionally, the paper delves into the frameworks and applications related to these processors. The scope of edge computing includes end devices and edge nodes [4]. End devices include smartphones, wearables, autonomous cars, gadgets, and many more. Edge nodes are switches, routers, micro data centers, and servers deployed at the edge [5,6]. Table 1 lists some of the key characteristics of edge deep learning processors that are considered in this paper.

There are multiple types of AI accelerators enabling DNN computing: Central Processing Unit (CPU), Graphics Processing Unit (GPU), Tensor Processing Unit (TPU), Application Specific Integrated Circuit (ASIC), System on-Chip (SoC), Processing in-Memory (PIM), and Neuromorphic processor. ASIC, SoC, TPU, PIM, and Neuromorphic systems are mainly targeted for low-power AI applications in edge and IoT devices. Google introduced different versions of the TPU that are used in the Google cloud and in the edge for training and inference [7]. Neuromorphic processors are non-Von Neumann computing systems that mimic human cognitive information processing systems. They generally utilize spiking neural networks (SNN) for processing [8,15]. Several tech companies, including Intel and IBM [8,9,10] have developed brain-inspired neuromorphic processors for edge applications. PIM is another non-Von Neumann computing paradigm that eliminates the data transfer bottleneck by having the computation take place inside a memory array in a highly parallel fashion [16,17,18,19,20].

PIM technology reduces the data movement and latency compared to traditional architectures and makes the computations significantly more efficient. Edge processors usually perform inference with highly optimized DNN models. The models are often compressed to reduce the number of computations, and the weight precision is usually quantized from the floating-point (FP) format normally used in training. The quantized integer (INT) and brain-float (BF) are used in inference processors. Typically, INT4, INT8, INT16, FP16, or BF16 numerical precision is used in the inference processor. However, recently released processors from multiple startups can compute with very low precision while trading off accuracy to some extent [21].

The current trend of computing technology is to enable data movement faster for higher speed and more efficient computing. To achieve this, AI edge processors need some essential prerequisites: lower energy consumption, smaller area, and higher performance. Neuromorphic and PIM processors are becoming more popular for their higher energy efficiency and lower latency [9,10,19,20]. However, a single edge processor usually does not support all types of DNN networks and frameworks. There are multiple types of DNN models, and each usually excels at particular application domains. For example, Recurrent Neural Networks (RNN), Long-Short-Term-Memory (LSTM), and Gated Recurrent Unit (GRU) are suitable for natural language processing [22,23,24,25,26,27,28], but Convolutional Neural Networks (CNN), Residual Neural Network (ResNet), and Visual Geometry Group (VGG) networks are better for detection and classification [29,30,31].

The CMOS technology node used for fabricating each device has a significant impact on its area, energy consumption, and speed. TSMC currently uses 3nm extreme Ultraviolet (UV) technology for the Apple A17 processor [32]. TSMC is currently aspiring to develop 2nm technology by 2025 for higher performance and highly energy-efficient AI computing processors [33]. Samsung’s smartphone processor Exynos 2200 is in the market developed with 4nm technology [34]. Intel utilized its Intel-4/7nm technology for its Loihi 2 neuromorphic processor [9].

This article provides a comprehensive review of commercial deep learning edge processors. Over 100 edge processors are listed along with their key specifications. We believe this is the most comprehensive technical analysis at present. The main contributions of this review are:

• It provides a comprehensive and easy to follow description of the state-of-the-art edge devices and their underlying architecture.
• It reviews the supported programming frameworks of the processors and general model compression techniques to enable edge computing.
• The study has analyzed the technical details of the processors for edge computing and provides charts on hardware parameters.

This paper is arranged as follows: section 2 describes key deep learning algorithms very briefly. Section 3 describes model compression techniques commonly used to optimize deep learning networks for edge applications. Section 4 discusses the frameworks available for deep learning AI applications. Section 5 describes the frameworks for developing AI applications on SNN processors. The processors are reviewed briefly in section 6. Section 7 discusses the data on the processors and performs a comparative analysis. A brief summary of this review study is presented in section 8.

2. Deep Learning Algorithms in Edge Application

Deep learning (DL) is a subset of AI and machine learning. It consists of multilayered artificial neural network architectures that optimize the network learning parameters to recognize the patterns and sequences for numerous applications. The networks can be trained for specific tasks, such as speech recognition [35], image recognition [36,37], security [38], anomaly detection [39], and fault detection [40]. Deep learning algorithms can be classified into the following categories: supervised, semi-supervised, unsupervised, and deep reinforcement learning [41,42].

This study is focused on AI accelerators for edge/IoT applications. Supervised and semi-supervised DL categories are usually trained on high-performance computing systems and then deployed to edge devices. Supervised learning models utilize labeled data samples. These models usually extract key features from incoming data samples and use the features to classify the sample. One of the most popular categories of supervised DL networks is CNNs [42]. Some common CNN architectures include VGG [43], ResNet [44], and GoogleNet [45]. Semi-supervised neural networks use a few labels to learn the categories and could be generative models or time-based sequence learning models. The semi-supervised topologies include GAN, GRU, RNN, and LSTM. The internal layers of these NN models are composed of CNN and fully connected network topologies. A number of edge processors support the semi-supervised network models for automation applications. For example, DeepVision (now Kinara) introduced ARA-1 (2020) and ARA-2 (2022) [46], which target autonomous applications, such as robotics, autonomous vehicles, smart tracking, and autonomous security systems. Kneron introduced KL720 in 2021, which supports semi-supervised network topologies for a wide range of applications [47]. In 2021, Syntiant released a new PIM AI processor for extreme edge applications, accommodating supervised and semi-supervised network topologies and supporting CNN, GRU, RNN, and LSTM topologies [20].

The computational complexity of DL models is a barrier to implementing these models for resource constrained edge or IoT devices. For edge applications, the deep neural network should be designed in an optimized way that is equally efficient without losing accuracy significantly. Common deep learning application areas in the edge include [48,49,50,51,52,53,54,55]: image classification, object detection, object tracking, speech recognition, health care, and natural language processing (NLP). This section will discuss some lightweight DL models for edge applications to perform classification and object detection.

i. Classification

Classification is probably the most popular use of CNNs and is one of the key applications in the computer vision field [56,57,58]. While larger networks with higher accuracies are utilized in desktop and server systems, smaller and more highly efficient networks are typically used for edge applications.

SqueezeNet [59,60] utilizes a modified convolutional model that is split into squeeze and expand layers. Instead of 3x3 convolution operations seen in typical CNNs, a much simpler 1x1 convolution operation is used. SqueezeNet achieves AlexNet levels of accuracy with 50x fewer network parameters [60]. Using model compression techniques, SqueezeNet can be compressed to 0.5 MB, which is about 510x smaller than AlexNet.

MobileNet [61] was created by Google and is one of the most popular DL models for edge applications. MobileNet substitutes the traditional convolution operation with a more flexible and efficient depthwise separable operation, significantly reducing computational costs. The depthwise separable technique performs two operations: depthwise convolution and pointwise convolution. There are three available versions of MobileNet networks: MobileNet v1 [61], MobileNet v2 [62], and MobileNet v3 [63]. MobileNet v2 builds on MobileNet v1 by adding a linear bottleneck and an inverted residual block at the end. The latest MobileNet v3 utilizes NAS (Neural Architecture Search) and NetAdapt to design a more accurate and efficient network architecture for inference applications [63].

ShuffleNet [64] utilizes group convolution and channel shuffle to reduce computation complexity. It increases accuracy by retraining with minimal computational power. There are two versions of ShuffleNet, ShuffleNet v1 and ShuffleNet v2 [64,65].

EfficientNet is a family of the convolutional network model scaled from other models. It can uniformly scale all the network dimensions, such as width, depth, and resolution by using a compound coefficient [66]. The scaling method facilitates the development of a family of networks. Unlike other DL models, the EfficientNet model focuses not only on accuracy but also on the efficiency of the model.

ii. Ditection

Object detection is an important task in computer vision that identifies and localizes all the objects in an image. This application has a wide range of applications, including autonomous vehicles, smart cities, target tracking, and security systems [67]. The broad range of object detection and DL network applications are discussed in [68,69]. DL networks for object detection can be categorized into two types: i) single-stage (such as SSD, YOLO, and CenterNet) and ii) two-stage (such as Fast/Faster RCNN). There are multiple criteria for choosing the right architecture for the edge application. Single-stage detectors are computationally more efficient than two-stage architecture, making them a better choice for edge applications. For example, YOLO v5 demonstrates better performance compared to Faster-RCNN-ResNet-50 [67].

iii. Speech Recognition and Natural Language Processing

Speech recognition and natural language processing are becoming increasingly important applications of deep learning. Speech emotion and speech keyword recognition are the objectives of speech recognition. The process includes multiple state-of-the-art research fields, such as AI, pattern recognition, signal processing, and information theory. Apple's Siri and Google's Alexa illustrate the potential applications of speech recognition and manifest better computer-human interfacing. RNN based neural networks and time delay DNN (TDNN) are popular choices for speech recognition [70]. Combined networks, such as TDNN-LSTM [71] or RNN-LSTM, are also popular choices for speech recognition [72].

Detailed analysis of deep neural networks for NLP can be found in [73,74]. Important applications of NLP are machine translation, named entity recognition, question answering system, sentiment analysis, spam detection, and image captioning. An early NLP model was sequence2sequence learning, based on RNNs. More recently, NLP was boosted by the advent of the transformer model, BERT [75]. BERT utilized an attention mechanism that learned contextual relations between words [75]. Other state-of-the-art NLP models are GPT-2 [76], GPT-3 [77], GPT-4 [78], and switch transformer [79]. However, these models run on HPC systems and are thus not compatible with edge devices. DeFormer [80], MobileBERT [81], and EdgeBERT [82] are some of the examples of NLP models targeted for edge devices. A more detailed discussion on NLP models for edge devices can be found in [83].

Syntiant [20] has recently been building tiny AI chips for voice and speech recognition and has attracted attention in the tech industry. Syntiant’s Neural Decision Processors (NDPs) are certified by amazon for use in Alexa-based devices [84]. Other voice recognition AI chips include NXP’s i.MX8, i.MX9x [85,86,87] and M1076 from Mythic [88,89,90]. LightSpear 2803S from Gyrfalcon can be utilized for NLP [91,92]. IBM unveiled its NorthPole edge processor for NLP applications at the HotChips 2023 conference [299].

3. Model Compression

Unoptimized DL models contain considerable redundancy in parameters and are generally designed without consideration of power or latency. Lightweight and optimized DL models enable AI application on edge devices. Designing effective models for running on resource-constrained systems is challenging. DNN model compression techniques are utilized to convert unoptimized models to forms that are suitable for edge devices. Model compression techniques are studied extensively and discussed in [93,94,95,96,97,98]. The techniques include parameter pruning, quantization, low-rank factorization, compact filtering, and knowledge distillation. In this section, we will discuss some of the key model compression techniques.

i. Quantization

Quantization is a promising approach to optimize the DNN models for edge devices. Data quantization for edge AI has been studied extensively in [94,95,96,97,98,99,100,101]. Parameter quantization takes a DL model and compresses its parameters by changing the floating-point weights to a lower precision to avoid costly floating-point computations. As shown in Table 2, most edge inference engines support INT4, 8, or 16 precisions. Quantization techniques can be taken to the limit by developing Binary Neural Networks (BNN) [101]. The BNN uses a single bit to represent activations and reduces memory requirements. Leapmind is the pioneer of low precision computations in their edge processor, Efficiera [21]. It is an ultra-low power edge processor and can perform AI computations with 1 bit weights and 2 bit activations.

Recent hardware studies show that lower precision does not have a major impact on inference accuracy. For example, Intel and Tsinghua University have presented QNAP [102], where they utilize 8 bits for weights and activations. They show an inference accuracy loss of only 0.11% and 0.40% for VGG-Net and GoogleNet, respectively, when compared to a software baseline with the ImageNet dataset. Samsung and Arizona State University have experimented with extremely low precision inference in PIMCA [103], where they utilize 1 bit for weights and activations. They show that VGG-9 and ResNet-18 had accuracy losses of 3.89% and 6.02% respectively.

Lower precision increases the energy and area efficiency of a system. PIMCA can compute 136 and 35 TOPS/W in 1 and 2 bit precision, respectively for ResNet-18. TSMC [104] has studied the impact of low precision computations on area efficiency. They show 221 and 55 TOPS/mm² area efficiency in 4- and 8-bit precision. Thus, with 4-bit computation, they achieve about 3.5x higher computation throughput per unit area compared to 8-bit computation.

Brain-Float-16 (or BF-16) [105] is a limited precision floating point format that is becoming popular for AI applications in edge devices. BF16 combines certain components of FP32 and FP16. From FP16, the BF16 utilizes 16 bits overall. From FP32, BF16 utilizes 8 bits for the exponent field (instead of 5 bits for FP16). A key benefit of BF16 is the format gets the same dynamic range and inference accuracy as of FP32 [106]. BF16 speeds up the MAC operation in edge devices to enable faster AI inference on the edge devices. Both the GDDR6-AiM from SK Hynix [107] and Pathfinder-1600 from Blaize [108,109] support BF16 for AI applications. The supported precision levels of various edge processors are presented in Table 2.

ii. Pruning

Pruning is the technique to remove unnecessary network connections to make the network lightweight for deploying on edge processors. Several studies [94,95,96,97,98,99,100,101,110,111,112] show that upto 91% of weights in AlexNet can be pruned with minimal accuracy reduction. Various training methods have been proposed to apply pruning to pre-trained networks [101]. Pruning however has drawbacks, such as creating sparsity in the weight matrices. This sparsity leads to unbalanced parallelism in the computation and irregular access to the on-chip memory. Several techniques have been developed [113,114] to reduce the sparsity.

iii. Knowledge Distillation

Knowledge distillation, introduced by B. Christian et al. [115], is a technique where the knowledge of an ensemble of larger networks is transferred to a smaller network without loss of validity. This can reduce the computational load significantly. The effectiveness of knowledge distillation is studied extensively in [94,95,96,97,98,99,100,101,116,117,118,119,120], where the authors show that the distillation of knowledge from a larger regularized model into a smaller model works effectively. Various algorithms have been proposed to improve the process of transferring knowledge, such as adversarial distillation, multi-teacher distillation, cross-modal distillation, attention-based distillation, quantized distillation, and NAS based distillation [121]. Although knowledge distillation techniques are mainly used for classification applications, they are also applied to other applications, such as object detection, semantic segmentation, language modeling, and image synthesis [122].

4. Framework for Deep Learning Networks

At present, the majority of edge AI processors are designed for inference only. Network training is typically carried out on higher performance desktop or server systems. There are a large variety of software frameworks to train deep networks and also to convert them into lightweight, suitable for edge devices. Popular DNN frameworks include Tensorflow (TF) [123], Tensorflow Lite (TFL) [124], PyTorch [125], PyTorch mobile [126], Keras [127], Caffe2 [128], OpenCV [129], ONNX [130], and MXNet [131]. Some of these frameworks support a broad class of devices, such as android, iOS, or Linux systems.

TFL was developed by Google and supports interfacing with many programming languages (such as Java, C++, Python). It can take a trained model from TensorFlow and apply model compression to reduce the amount of computations needed for inference.

ONNX was developed by the PyTorch team to represent traditional machine learning and state-of-the-art deep learning models [130]. The framework is interoperable across popular development tools, such as PyTorch, Caffe2, and Apache MXNet. Many of the current AI processors support the ONNX framework, such as Qualcomm SNPE, AMD, ARM, and Intel [132].

PyTorch mobile was developed by Facebook and allows a developer to train AI models for edge applications. The framework provides a node-to-node workflow that enables the clients to have a privacy-preserving learning environment via collaborative or federated learning [125,126]. It supports XNNPACK floating point kernel libraries for ARM CPUs and integrates QNNPACK for quantized INT8 kernels [126].

Caffe2 is a lightweight framework developed by Facebook [128]. This framework supports C++ and Python APIs, which are interchangeable and helps to develop prototypes quickly that could potentially be optimized later. Caffe2 integrates with Android Studio and Microsoft Visual Studio for mobile development [128]. Caffe2Go is developed to embed in mobile apps for applying a full-fledged deep learning framework for real-time capture, analysis, and decision making without the help of a remote server [133].

Facebook uses Pytorch Mobile, Caffe2 and ONNX for developing their products. Pytorch is used for the experiment and rapid development, Caffe2 is developed for aiming at the production environment, while ONNX helps to share the models between the two frameworks [130].

MXNet is a fast and scalable framework developed by the Apache Software Foundation [131]. This framework supports both training and inference with a concise API for AI applications in edge devices. MXNet supports Python, R, C++, Julia, Perl, and many other languages and can be run on any processor platform for developing AI applications [131]. As shown in Table 4, TFL, ONNX, and Caffe2 are the most widely used frameworks for AI edge applications.

Some edge processors are compatible only with their in-home frameworks. For example, Kalray's MPPA3 edge processor is compatible with KaNN (Kalray Neural Network), so any trained deep network must be converted to KaNN to run on the MPPA3 processor [13]. CEVA introduced its own software framework CEVA-DNN for converting pre-trained network models and weights from offline training frameworks (such as Caffe, TensorFlow) for inference applications on the CEVA processors [134,135,136]. CEVA added a retrain feature in CEVA-DNN for the Neuro-Pro processor to enable a deployed device to be updated without uploading a database to the server [134]. The developer can also use CEVA-DNN tools on a simulator or test device and then transfer the updated model to edge devices [136].

5. Framework for Spiking Neural Networks

Spiking neural networks (SNN) utilize brain inspired computing primitives, where a neuron accumulates a potential and fires only when a threshold is crossed [137]. This means in spiking neural networks, the neurons have outputs sporadically. Thus, SNNs have much fewer neuron to neuron communications compared to deep neural networks, where all neurons always send outputs. The net result of this is that SNNs can be dramatically more power efficient than DNNs and could potentially implement a task with far fewer operations. Thus, an SNN processor with the same operations per second capability as a DNN processor could theoretically have a much higher task level throughput.

To get the highest efficiency from SNN processors, it is best to use algorithms that are developed from the ground up to use spiking neurons. Examples of such algorithms include constraint satisfaction problems [138] and genetic algorithms [139]. Several studies have examined how to implement DNNs using SNNs [140]. Davidson et al. [141] show through modeling of energies that this should not result in higher efficiency than the original DNN using the same underlying silicon technology. However, P. Blouw et al. [142] implemented keyword spotting on several hardware platforms and showed that the Loihi was about 5x more energy efficient than the Movidius deep learning processor. The remainder of this section describes some of the key frameworks for implementing SNN architectures for spiking neuromorphic processors.

Nengo is a Python based framework developed by Applied Brain Research for spiking neurons. It supports multiple types of processors, including Loihi [143] and Spinnaker [144]. Nengo is very flexible in writing code and simulating SNNs. The core framework is the Nengo ecosystem, which includes Nengo objects and NumPy based simulators. The Nengo framework has Nengo GUI for model construction and visualization tools and NengoDL for simulating deep learning models using SNNs [145].

Meta-TF [146] is a framework developed by BrainChip for edge application in the Akida neuromorphic chips [147,148,149]. Meta-TF takes advantage of the Python scripting language and associated tools, such as Jupyter notebook and NumPy. Meta-TF includes three Python packages [146]: 1) The Akida Python package works as an interface to the Akida neuromorphic SoC. 2) the CNN2SNN tool provides an environment to convert a trained CNN network into SNNs. Brainchip embeds the on-chip training capability in the Akida processor, and thus, the developers can train SNNs on the Akida processor directly [149]. iii) Akida Model Zoo contains pre-created network models, which are built with the Akida sequential API and the CNN2SNN tool by using quantized Keras models.

Lava is a framework currently being developed by Intel to build SNN models and map them to neuromorphic platforms [150]. The current version of the Lava framework supports the Loihi neuromorphic chips [9]. Lava includes Magma which helps to map and execute neural network models and sequential processes to neuromorphic hardware [150]. Magma also helps to estimate performance and energy consumption on the platform. Lava has additional properties, including offline training, integration with other frameworks, a Python interface, and being an open-source framework (with proper permissions). The Lava framework supports online real-time learning, where the framework adopts plasticity rules. However, the learning is constrained to access only locally available process information [150].

6. Edge Processors

At present, GPUs are the most popular platform for implementing DNNs. These, however, are usually not suitable for edge computing (except the NVIDIA Jetson systems) due to their high-power consumption. A large variety of AI hardware has been developed, many of which target edge applications. Several articles have reviewed AI hardware in broad categories, giving an overall idea of the current trend in AI accelerators [151,152,153]. Earlier works [2,154,155,156] have reviewed a small selection of older edge AI processors.

This paper presents a very broad coverage of edge AI processors and PIM processors from the industry. This includes processors already released, processors that have been announced, and processors that have been published about in research venues (such as the ISSCC and the VLSI conferences). This section is divided into four subsections: subsection (i) describes dataflow processors, subsection (ii) describes neuromorphic processors, and subsection (iii) describes PIM processors. All of these sections describe industrial products that have been announced or released. Finally, subsection (iv) describes the processors in industrial research.

Table 2 describes the key hardware characteristics of the commercial edge-AI and PIM-AI processors. Table 3 lists the same key characteristics for the processors from industrial research. Table 4 describes the key software/application characteristics of the processors in Table 2 .

i. Dataflow Edge Processor

This section describes the latest dataflow processors from the industry. Dataflow processors are custom designed for neural network inference and, in some cases, training computations. The processors are listed in alphabetical order based on the manufacturer name. The data provided is from the publications or websites of the processors.

Apple released the bionic SoC A16 with an NPU unit for the iPhone 14 [157]. The A16 processor exhibits about 20% better performance with the same power consumption as their previous version, A15. It is embedded with a 6-core ARM8.6a CPU, 16-core NPU, and 8-core GPU [157]. The Apple M2 processor was released in 2022 primarily for the Macbooks, and then optimized for iPads. This processor includes a 10-core GPU and 16-core NPU [158]. M1 performs 11 TOPS with 10 W of power consumption [159]. M2 has 18% and 35% more powerful CPU and GPU for faster computations.

ARM recently announced the Ethos-N78 with an 8-core NPU for automotive applications [160]. Ethos-N78 is an upgraded version of Ethos-N77. Both NPUs support INT8 and INT16 precision. Ethos-N78 performs more than 2x better than the earlier version. The most significant improvement of Ethos-N78 is enabling a new data compression method that reduces the bandwidth and improves performance and energy efficiency [161].

Blaize released its Pathfinder P1600, El Cano AI inference processor. This processor integrates 16 graph streaming processors (GSP) that deliver 16 TOPS at its peak performance [162]. It uses a dual Cortex-A53 for running the operating system at up to 1GHz. Blaize GSP processors integrate data pipelining and support up to INT-64 and FP-8-bit operations [163].

AIMotive [164] introduced the inference edge processor Apache5, which supports a wide range of DNN models. The system has an aiWare3p NPU with an energy efficiency of 2 TOPS/W. Apache5 supports INT8 MAC and INT32 internal precision [165]. This processor is mainly targeted for autonomous vehicles [166].

CEVA [134] released the Neupro-S on-device AI processor for computer vision applications. Neupro comprises two separate cores. One is the DSP-based Vector Processor Unit (VPU), and the other is the Neupro Engine. VPU is the controller, and the Neupro Engine does most of the computing work with INT8 or INT16 precision. A single processor performs up to 12.5 TOPS, while the performance can be scaled to 100 TOPS with multicore clusters [134,135]. The deep learning edge processors are mostly employed for inference tasks. CEVA added a retraining capability to its CDNN (CEVA DNN) framework for online learning on client devices [136].

Cadence introduced the Tensilica DNA 100, which is a comprehensive SoC for domain-specific on-device AI edge accelerators [167]. It has low, mid, and high-end AI products. Tensilica DNA 100 offers 8 GOPS to 32 TOPS AI processing performance currently and predicts 100 TOPS in future releases [168]. The target application of the DNA 100 is IoTs, intelligent sensors, vision, and voice application. The mid and high-end applications include smart surveillance and autonomous vehicles, respectively.

Table 2. Commercial Edge Processors with Operation Technology, Process Technology, and Numerical Precision.

Company	Latest Chip	Power (W)	Process (nm)	Area (mm2)	Precision INT/FP		Performance (TOPS)	E. Eff. (TOPS/W)		Architecture	Reference
Apple	M1	10	5	119	64		11	1.1		Dataflow	[159]
Apple	A14	6	5	88	64		11	1.83		Dataflow	[242]
Apple	A15	7	5		64		15.8	2.26		Dataflow	[242]
Apple	A16	5.5	4		64		17	3		Dataflow	[157]
*AIStorm	AIStorm	0.225			8		2.5	11		Dataflow	[243]
*AlphaIC	RAP-E	3			8		30	10		Dataflow	[244]
aiCTX	Dynap-CNN	0.001	22	12	1		0.0002	0.2		Neuromorphic	[15,213]
*ARM	Ethos78	1	5		16		10	10		Dataflow	[160,161]
*AIMotive	Apache5 IEP	0.8	16	121	8		1.6-32	2		Dataflow	[164,165]
*Blaize	Pathfinder, EI Cano	6	14		64, FP-8, BF16		16	2.7		Dataflow	[162]
*Bitman	BM1880	2.5	28	93.52	8		2	0.8		Dataflow	[245,246]
*BrainChip	Akida1000	2	28	225	1,2,4		1.5	0.75		Neuromorphic	[147,148]
*Cannan	Kendrite K210	2	28		8		1.5	1.25		Dataflow	[247,248]
*CEVA	CEVA-Neuro-S		16		2, 5, 8, 12, 16		12.7			Dataflow	[134]
*CEVA	CEVA-Neuro-M	0.83	16		2, 5, 8, 12, 16		20	24		Dataflow	[135]
*Cadence	DNA100	0.85	16		16		4.6	3		Dataflow	[167,168]
*Deepvision	ARA-1	1.7	28		8,16		4	2.35		Dataflow	[169]
*Deepvision	ARA-2		16							Dataflow	[170]
*Eta	ECM3532	0.01	55	25	8		0.001	0.1		Dataflow	[249]
*FlexLogic	InferX X1	13.5	7	54	8		7.65	0.57		Dataflow	[250]
*Google	Edge TPU	2	28	96	8, BF16		4	2		Dataflow	[176,177]
*Gyrfalcon	LightSpeer 2803S	0.7	28	81	8		16.8	24		PIM	[224]
*Gyrfalcon	LightSpeer 5801	0.224	28	36	8		2.8	12.6		PIM	[224]
*Gyrfalcon	Janux GS31	650/900	28	10457.5	8		2150	3.30		PIM	[225]
*GreenWaves	GAP9	0.05	22	12.25	FP-(8,16,32)		0.05	1		Dataflow	[180,181]
*Horizon	Journey 3	2.5	16		8		5	2		Dataflow	[171]
*Horizon	Journey5/5P	30	16		8		128	4.8		Dataflow	[172,173]
*Hailo	Hailo 8 M2	2.5	28	225	4,8,16		26	2.8		Dataflow	[174,175]
Intel	Loihi 2	0.1	7	31	8		0.3	3		Neuromorphic	[9]
Intel	Loihi	0.11	14	60	1-9		0.03	0.3		Neuromorphic	[9,218]
*Intel	Intel® Movidius	2	16	71.928	16		4	2		Dataflow	[186]
IBM	TrueNorth	0.065	28	430	8		0.0581	0.4		Neuroorphic	[10,218]
IBM	NorthPole	74	12	800	2,4,8		200 (INT8)	2.7		Dataflow	[299,304]
*Imagination	PowerVR Series3NX				FP-(8,16)		0.60			Dataflow	[182,183]
*Imagination	IMG 4NX MC1	0.417			4,16		12.5	30		Dataflow	[184]
*Imec	DIANA		22	10.244	2	29.5 (A), 0.14 (D)			14.4	PIM+Digital	[222,223]
*Kalray	MPPA3	15	16		8,16	255			1.67	Dataflow	[13]
*Kneron	KL720 AI	1.56	28	81	8,16	1.4			0.9	Dataflow	[191]
*Kneron	KL530	0.5			8	1			2	Dataflow	[192]
*Koniku	Konicore									Neuromorphic	[12]
*LeapMind	Efficiera	0.237	12	0.422	1,2,4,8,16,32	6.55			27.7	Datalow	[21]
Memryx	MX3	1	--	--	4,8,16 (W) BF16	5			5	Dataflow	[297]
*Mythic	M1108	4		361	8	35			8.75	PIM	[89]
*Mythic	M1076	3	40	294.5	8	25			8.34	PIM	[18,88,90]
*mobileEye	EyeQ5	10	7	45	4,8	24			2.4	Dataflow	[193,194,195]
*mobileEye	EyeQ6	40	7		4,8	128			3.2	Dataflow	[196]
*Mediatek	i350		14			0.45				Dataflow	[251]
*NVIDIA	Jetson Nano B01	10	20	118	FP16	1.88			0.188	Dataflow	[197]
NVIDIA	AGX Orin	60	7	--	8	275			3.33	Dataflow	[199]
*NXP	i.MX 8M+		14	196	FP16	2.3				Dataflow	[86,87]
*NXP	i.MX9	4x10-6	12							Dataflow	[85]
*Perceive	Ergo	0.073	5	49	8	4			55	Dataflow	[252]
TSU & Polar Bear Tech	QM930	12	12	1089	4,8,16	20 (INT8)			1.67	Dataflow	[302]
Qualcomm	QCS8250		7	157.48	8	15				Dataflow	[200,201]
Qualcomm	Snapdragon 888+	5	5		FP32	32			6.4	Dataflow	[202,203,204]
Qualcomm	Snapdragon 8 Gen2		4		4,8,16, FP16	51				Dataflow	[303]
*RockChip	rk3399Pro	3	28	729	8, 16	3			1	Dataflow	[253]
Rokid	Amlogic A311D		12			5				Dataflow	[254]
Samsung	Exynos 2100		5			26				Dataflow	[205,206]
Samsung	Exynos 2200		4		8,16, FP16					Dataflow	[255]
Samsung	HBM-PIM	0.9	20	46.88		1.2			1.34	PIM	[226,227]
Sima..ai	MLSoC	10	16	175.55	8	50			5	Dataflow	[300,301]
Synopsis	EV7x		16		8, 12, 16,	2.7				Dataflow	[209,210]
*Syntiant	NDP100	0.00014	40	2.52		0.000256			20	PIM	[228,229]
*Syntiant	NDP101	0.0002	40	25	1,2, 4,8	0.004			20	PIM	[228,231]
*Syntiant	NDP102	0.0001	40	4.2921	1, 2, 4, 8	0.003			20	PIM	[228,235]
*Syntiant	NDP120	0.0005	40	7.75	1,2,4,8	0.0019			3.8	PIM	[228,234]
*Syntiant	NDP200	0.001	40		1,2,4,8	0.0064			6.4	PIM	[228,232]
Think Silicon	NEMA®|pico XS	0.0003	28	0.11	FP16,32	0.0018			6	Dataflow	[256]
Tesla/Samsung	FSD Chip	36	14	260	8, FP-8	73.72			2.04	Dataflow	[211]
Videntis	TEMPO									Neuromorphic	[11]
Verisilicon	VIP9000		16		16, FP16	0.5-100				Dataflow	[207,208]
Untether	TsunAImi	400	16		8	2008			8	PIM	[236,237]
UPMEM	UPMEM-PIM	700	20		32, 64	0.149				PIM	[238,239,240,241]

*Processors are available for purchase; **Integer Precision is indicated by only precision number(s). Floating point precision is mentioned as FP in the precision column.

Table 2. Commercial Edge Processors with Operation Technology, Process Technology, and Numerical Precision.

Company	Latest Chip	Power (W)	Process (nm)	Area (mm2)	Precision INT/FP		Performance (TOPS)	E. Eff. (TOPS/W)		Architecture	Reference
Apple	M1	10	5	119	64		11	1.1		Dataflow	[159]
Apple	A14	6	5	88	64		11	1.83		Dataflow	[242]
Apple	A15	7	5		64		15.8	2.26		Dataflow	[242]
Apple	A16	5.5	4		64		17	3		Dataflow	[157]
*AIStorm	AIStorm	0.225			8		2.5	11		Dataflow	[243]
*AlphaIC	RAP-E	3			8		30	10		Dataflow	[244]
aiCTX	Dynap-CNN	0.001	22	12	1		0.0002	0.2		Neuromorphic	[15,213]
*ARM	Ethos78	1	5		16		10	10		Dataflow	[160,161]
*AIMotive	Apache5 IEP	0.8	16	121	8		1.6-32	2		Dataflow	[164,165]
*Blaize	Pathfinder, EI Cano	6	14		64, FP-8, BF16		16	2.7		Dataflow	[162]
*Bitman	BM1880	2.5	28	93.52	8		2	0.8		Dataflow	[245,246]
*BrainChip	Akida1000	2	28	225	1,2,4		1.5	0.75		Neuromorphic	[147,148]
*Cannan	Kendrite K210	2	28		8		1.5	1.25		Dataflow	[247,248]
*CEVA	CEVA-Neuro-S		16		2, 5, 8, 12, 16		12.7			Dataflow	[134]
*CEVA	CEVA-Neuro-M	0.83	16		2, 5, 8, 12, 16		20	24		Dataflow	[135]
*Cadence	DNA100	0.85	16		16		4.6	3		Dataflow	[167,168]
*Deepvision	ARA-1	1.7	28		8,16		4	2.35		Dataflow	[169]
*Deepvision	ARA-2		16							Dataflow	[170]
*Eta	ECM3532	0.01	55	25	8		0.001	0.1		Dataflow	[249]
*FlexLogic	InferX X1	13.5	7	54	8		7.65	0.57		Dataflow	[250]
*Google	Edge TPU	2	28	96	8, BF16		4	2		Dataflow	[176,177]
*Gyrfalcon	LightSpeer 2803S	0.7	28	81	8		16.8	24		PIM	[224]
*Gyrfalcon	LightSpeer 5801	0.224	28	36	8		2.8	12.6		PIM	[224]
*Gyrfalcon	Janux GS31	650/900	28	10457.5	8		2150	3.30		PIM	[225]
*GreenWaves	GAP9	0.05	22	12.25	FP-(8,16,32)		0.05	1		Dataflow	[180,181]
*Horizon	Journey 3	2.5	16		8		5	2		Dataflow	[171]
*Horizon	Journey5/5P	30	16		8		128	4.8		Dataflow	[172,173]
*Hailo	Hailo 8 M2	2.5	28	225	4,8,16		26	2.8		Dataflow	[174,175]
Intel	Loihi 2	0.1	7	31	8		0.3	3		Neuromorphic	[9]
Intel	Loihi	0.11	14	60	1-9		0.03	0.3		Neuromorphic	[9,218]
*Intel	Intel® Movidius	2	16	71.928	16		4	2		Dataflow	[186]
IBM	TrueNorth	0.065	28	430	8		0.0581	0.4		Neuroorphic	[10,218]
IBM	NorthPole	74	12	800	2,4,8		200 (INT8)	2.7		Dataflow	[299,304]
*Imagination	PowerVR Series3NX				FP-(8,16)		0.60			Dataflow	[182,183]
*Imagination	IMG 4NX MC1	0.417			4,16		12.5	30		Dataflow	[184]
*Imec	DIANA		22	10.244	2	29.5 (A), 0.14 (D)			14.4	PIM+Digital	[222,223]
*Kalray	MPPA3	15	16		8,16	255			1.67	Dataflow	[13]
*Kneron	KL720 AI	1.56	28	81	8,16	1.4			0.9	Dataflow	[191]
*Kneron	KL530	0.5			8	1			2	Dataflow	[192]
*Koniku	Konicore									Neuromorphic	[12]
*LeapMind	Efficiera	0.237	12	0.422	1,2,4,8,16,32	6.55			27.7	Datalow	[21]
Memryx	MX3	1	--	--	4,8,16 (W) BF16	5			5	Dataflow	[297]
*Mythic	M1108	4		361	8	35			8.75	PIM	[89]
*Mythic	M1076	3	40	294.5	8	25			8.34	PIM	[18,88,90]
*mobileEye	EyeQ5	10	7	45	4,8	24			2.4	Dataflow	[193,194,195]
*mobileEye	EyeQ6	40	7		4,8	128			3.2	Dataflow	[196]
*Mediatek	i350		14			0.45				Dataflow	[251]
*NVIDIA	Jetson Nano B01	10	20	118	FP16	1.88			0.188	Dataflow	[197]
NVIDIA	AGX Orin	60	7	--	8	275			3.33	Dataflow	[199]
*NXP	i.MX 8M+		14	196	FP16	2.3				Dataflow	[86,87]
*NXP	i.MX9	4x10-6	12							Dataflow	[85]
*Perceive	Ergo	0.073	5	49	8	4			55	Dataflow	[252]
TSU & Polar Bear Tech	QM930	12	12	1089	4,8,16	20 (INT8)			1.67	Dataflow	[302]
Qualcomm	QCS8250		7	157.48	8	15				Dataflow	[200,201]
Qualcomm	Snapdragon 888+	5	5		FP32	32			6.4	Dataflow	[202,203,204]
Qualcomm	Snapdragon 8 Gen2		4		4,8,16, FP16	51				Dataflow	[303]
*RockChip	rk3399Pro	3	28	729	8, 16	3			1	Dataflow	[253]
Rokid	Amlogic A311D		12			5				Dataflow	[254]
Samsung	Exynos 2100		5			26				Dataflow	[205,206]
Samsung	Exynos 2200		4		8,16, FP16					Dataflow	[255]
Samsung	HBM-PIM	0.9	20	46.88		1.2			1.34	PIM	[226,227]
Sima..ai	MLSoC	10	16	175.55	8	50			5	Dataflow	[300,301]
Synopsis	EV7x		16		8, 12, 16,	2.7				Dataflow	[209,210]
*Syntiant	NDP100	0.00014	40	2.52		0.000256			20	PIM	[228,229]
*Syntiant	NDP101	0.0002	40	25	1,2, 4,8	0.004			20	PIM	[228,231]
*Syntiant	NDP102	0.0001	40	4.2921	1, 2, 4, 8	0.003			20	PIM	[228,235]
*Syntiant	NDP120	0.0005	40	7.75	1,2,4,8	0.0019			3.8	PIM	[228,234]
*Syntiant	NDP200	0.001	40		1,2,4,8	0.0064			6.4	PIM	[228,232]
Think Silicon	NEMA®|pico XS	0.0003	28	0.11	FP16,32	0.0018			6	Dataflow	[256]
Tesla/Samsung	FSD Chip	36	14	260	8, FP-8	73.72			2.04	Dataflow	[211]
Videntis	TEMPO									Neuromorphic	[11]
Verisilicon	VIP9000		16		16, FP16	0.5-100				Dataflow	[207,208]
Untether	TsunAImi	400	16		8	2008			8	PIM	[236,237]
UPMEM	UPMEM-PIM	700	20		32, 64	0.149				PIM	[238,239,240,241]

*Processors are available for purchase; **Integer Precision is indicated by only precision number(s). Floating point precision is mentioned as FP in the precision column.

Deepvision has updated their edge inference coprocessor ARA-1 for applications to autonomous vehicles and smart industries [46. It includes eight compute engines with 4 TOPS and consumes 1.7-2.3 W of power [169]. The computing engine supports INT8 and INT16 precision. Deepvision has recently announced its second-generation inference engine, ARA-2, which will be released later in 2022 [170]. The newer version will support LSTM and RNN neural networks in addition to the networks supported in ARA-1.

Horizon announced its next automotive AI inference processor Journey 5/5P [171], which is the updated version of Journey 3. The mass production of Journey 5 will be starting in 2022. The processor exhibits a performance of 128 TOPS, and has a power of 30 W, giving an energy efficiency of 4.3 TOPS/W [172,173].

Hailo released its Hailo-8 M-2 SoC for various edge applications [174]. The computing engine supports INT8 and INT16 precision. This inference engine is capable of 26 TOPS and requires 2.5 W of power. The processor can be employed as a standalone or coprocessor [175].

Google introduced its Coral Edge TPU, which comprises only 29% of the floorpan of the original TPU for edge applications [176]. The Coral TPU shows high energy efficiency in DNN computations compared to the original TPUs which are used in cloud inference applications [178]. Coral Edge TPU supports INT8 precision and can perform 4 TOPS with 2 Watts of power consumption [176].

Google released its Tensor processor for mobile applications, coming with its recent Pixel series mobile phone [179]. Tensor is an 8-core cortex CPU chipset fabricated with 5 nm process technology. The processor has a 20-core Mali-G78 MP20 GPU with 2170 GFLOPS computing speed. The processor has a built-in NPU to accelerate AI models with a performance of 5.7 TOPS. The maximum power consumption of the processor is 10W.

GreenWaves announced their edge inference chip GAP9 [180]. It is a very low-cost, low-power device that consumes 50 mW and performs 50 GOPS at its peak. GAP9 provides hearable developments through DSP, AI accelerator, and ultra-low latency audio streaming on IoT devices. GAP9 supports a wide range of computing precision, such as INT8, 16, 24, 32, and FP16, 32 [181].

IBM introduced the NorthPole [299], a non-Von Neumann deep learning inference engine, at the HotChips 2023 conference. The processor shows massive parallelism with 256 cores. Each core has 768KB of near-compute memory to store weights, activations, and programs. The total on-chip memory capacity is 192 MB. The NorthPole processor does not use off-chip memory to load weights or store intermediate values during deep learning computations. Thus, it dramatically improves latency, throughput, and energy consumption, which helps outperform existing commercial deep learning processors. The external host processor works on three commands: write tensor, run network, and read tensor. The NorthPole processor follows a set of pre-scheduled deterministic operations in the core array. It is implemented in 12nm technology and has 22 billion transistors taking up 800 mm² of chip area. The performance data released on the NothPole processor are computed based on frame/sec. The performance metrics of operations/sec in integer or floating point are unavailable in the public domain currently. However, the operation per cycle is available for different data precisions. In vector-matrix multiplication, 8, 4, and 2-bit can perform 2048, 4096, and 8192 operations/cycles. The FP16 can compute 256 operations/cycle (the number of cycles/s is not released at this time). NorthPole can compute 800, 400, and 200 TOPS with INT 2, 4, and 8 precision. The processor can be applied to a broad area of applications and can execute inference with a wide range of network models applied in classification, detection, segmentation, speech recognition, and transformer models in NLP.

Imagination introduced a wide range of edge processors with targeted applications in IoTs to autonomous vehicles [182]. The edge processor series is categorized as the PowerVR Series3NX and can achieve up to 160 TOPS with multicore implementations. For ultra-low power applications, one can choose PowerVR AX3125, which has a 0.6 TOPS computing performance [183]. IMG 4NX MC1 is a single-core Series 4 processor for autonomous vehicle applications and performs at 12.5 TOPS with less than 0.5 W of power consumption [184].

Intel released multiple edge AI processors such as Nirvana Spring Crest NNP-I [185] and Movidious [186]. Recently, they have announced a scaleable 4th generation Xeon processor series that can be used for desktop to extreme edge devices [187]. The power consumption for an ultra-mobile processor is around 9W while computed with INT8 precision. The development utilizes the SuperFin fabrication technology with 10nm process technology. Intel is comparing its core architecture to the Skylake processor, and it claims an efficient core achieves 40% better performance with 40% less power.

IBM developed the Artificial Intelligence Unit (AIU) based on their AI accelerator used in the 7-nanometer Telum chip that powers its z16 system [188]. AIU is a scaled version developed on a 5 nm process technology and features a 32-core design with a total of 23 billion transistors. AIU uses IBM’s approximate computing frameworks where the computing executes with FP16 and FP32 precisions [189].

Leapmind has introduced the Efficiera for edge AI inference implemented in FPGA or ASIC [21]. Efficiency is for ultra-low power applications. The computations are typically performed in 8-, 16-, or 32-bit precision. However, the company claims that 1-bit weight and 2-bit activation can be achieved while still maintaining accuracy for better power and area efficiency. They show 6.55 TOPS at 800MHz clock frequency with an energy efficiency of 27.7 TOPS/W [190].

Kneron released its edge inference processor, KL 720, for various applications, such as autonomous vehicles and smart industry [191]. The KL 720 is an upgraded version of the earlier KL 520 for similar applications. The revised version performs at 0.9 TOPS/W and shows up to 1.4 TOPS. The neural computation supports INT8 and INT16 precisions [191]. Kneron's most up-to-date heterogeneous AI chip is KL 530 [192]. It is enabled with a brand new NPU, which supports INT4 precision and offers 70% higher performance than that of INT8. The maximum power consumption of KL 530 is 500 mW and can deliver up to 1 TOPS [192].

Table 3. Processors, Supported Neural Network Models, Deep Learning Frameworks, And Application Domains.

Company	Product	Supported Neural Networks	Supported Frameworks	Application/benefits
Apple	Apple A14	DNN	TFL	iPhone12 series
Apple	Apple A15	DNN	TFL	iPhone13 series
aiCTX-Synsense	Dynap-CNN	CNN, RNN, Reservoir Computing	SNN	High-speed aircraft, IoT, security, healthcare, mobile
ARM	Ethos78	CNN and RNN	TF, TFL,Caffe2,PyTorch, MXNet, ONNX	Automotive
AIMotive	Apache5 IEP	GoogleNet, VGG16, 19, Inception-v4, v2, MobileNet v1, ResNet50, Yolo v2	Caffe2	Automotives, pedestrian detection, vehicle detection, lane detection, driver status monitoring
Blaize	EI Cano	CNN, YOLO v3	TFL	Fit for industrial, retail, smart-city, and computer-vision systems
BrainChip	Akida1000	CNN in SNN, Mobilenet	MetaTF	Online learning, data analytics, security
BrainChip	AKD500, 1500, 2000	DNN	MetaTF	Smart home, Smart health, Smart City and smart transportation
CEVA	Neuro-s	CNN, RNN	TFL	IoTs, smartphones, surveillance, automotive, robotics, medical
Cadence	Tensilica DNA100	FCC, CNN, LSTM	ONNX, Caffe2, TensorFlow	IoT, smartphones, AR/VR, smart surveillance, Autonomous vehicle
Deepvision	ARA-1	Deep Lab V3, Resnet-50, Resnet-152, MobileNet-SSD, YOLO V3, UNET	Caffe2, TFL, MXNET, PyTorch	Smart retail, robotics, industrial automation, smart cities, autonomous vehicles, and more
Deepvision	ARA-2	Model in ARA-1 and LSTM, RNN,	TFL, Pytorch	Smart retail, robotics, industrial automation, smart cities,
Eta	ECM3532	CNN, GRU, LSTM	---	Smart home, consumer products, medical, logistics, smart industry
Gyrfalcon	LightSpeer 2803S	CNN based, VGG, ResNet, MobileNet;	TFL, Caffe2	High performance audio and video processing
Gyrfalcon	LightSpeer 5801	CNN based, ResNet, MobileNet and VGG16,	TFL, PyTorch & Caffe2	Object Detection and Tracking,NLP, Visual Analysis
Gyrfalcon Edge Server	Janux GS31	VGG, REsNet, MobileNet	TFL, Caffe2, PyTorch	Smart cities, surveillance, object detection, recognition
GreenWaves	GAP9	CNN, mobileNet v1		DSP application
Horizon	Journey 3	CNN, mobilenet v2, efficient net	TFL, Pytorch, ONNX, mxnet, Caffe2	Automotive
Horizon	Journey5/5P	Resnet18, 50, MobileNet v1-v2, ShuffleNetv2, EfficientNet FasterRCNN, Yolov3	TFL, Pytorch, ONNX, mxnet, Caffe2	Automotive
Hailo	Hailo 8 M2	YOLO 3, YOLOv4, CenterPose, CenterNet, ResNet-50	ONNX, TFL	Edge vision applications
Intel	Loihi 2	SNN based NN	Lava, TFL, Pytorch	Online learning, sensing, robotics, healthcare
Intel	Loihi	SNN based NN	Nengo	Online learning, robotics, healthcare and many more
Imagination	PowerVR Series3NX	Mobilenet v3, CNN	Caffe, TFL	smartphones, smart cameras, drones, automotives, wearables,
Imec & GF	DIANA	DNN	TFL, Pytorch	Analog computing in Edge inference
KoniKu	Konicore	synthetic Biology+Silicon	--	Chemical detection, aviation, security
Kalray	MPPA3	Deep network converted to KaNN	Kalray's KANN	Autonomous vehicles, surveillance, robotics, industry, 5G
Kneron	KL720 AI	CNN, RNN, LSTM	ONNX, TFL, Keras, Caffe2	wide applications from automotive to home appliances
Kneron	KL520	Vgg16, Resnet, GoogleNet, YOLO, Lenet, MobileNet, FCC	ONNX, TFL, Keras, Caffe2	Automotive, home, industry and so on.
LeapMind	Efficiera	CNN, YOLO v3, Mobilenet-v2, Lmnet	Blueoil, Python & C++ API	Home, Industrial machinery, surveillance camera, robots
Memryx	MX3	CNN	Pytorh, ONNX, TF, Keras	Automation, surveillance, agriculture, financial
Mythic	M1108	CNN, large complex DNN, Resnet50, YOLO v3, Body25	Pytorch, TFL, and ONNX	Machine Vision, Electronics, Smart Home, UAV/Drone, Edge Server
Mythic	M1076	CNN, Complex DNN, Resnet50, YOLO v3	Pytorch, TFL, and ONNX	Surveillance, Vision, voice, Smart Home, UAV, Edge Server
MobileEye	EyeQ5	DNN		Autonomous driving
MobileEye	EyeQ6	DNN		Autonomous driving
Mediatek	i350	DNN	TFL	Vision and voice, Biotech and Bio-metric measurements
NXP	i.MX 8M+	DNN	TFL, Arm NN, ONNX	Edge Vision
NXP	i.MX9	CNN, Mobilenet v1	TFL, Arm NN, ONNX	Graphics, image, display, audio
NVIDIA	AGX Orin	DNN	TF, TFL, Caffe, Pytorch	Robotics, Retail, Traffic, Manufacturing
Qualcomm	QCS8250	CNN, GAN, RNN	TFL	smartphone, tablet, support 5G, video and image processing
Qualcomm	Snapdragon 888+	DNN	TFL	Smartphone, tablet, 5G, gaming, video upscaling, image & video processing,
RockChip	rk3399Pro	VGG16, ResNEt50, Inception4	TFL, Caffe, mxnet, ONNX, darknet	Smart Home, City, and Industry; face recognition, driving monitoring,
Rokid	Amlogic A311D	Inception V3, YoloV2, YOLOV3	TFL, Caffe2 Darknet	High-performance multimedia
Samsung	Exynos 2100	CNN	TFL	Smartphone, tablet, advanced image signal processing (ISP), 5G
Samsung	HBM-PIM	DNN see youtube to write on int	Pytorch, TFL	Supercomputer and AI application
Synopsis	EV7x	CNN, RNN, LSTM	OpenCV, OpenVX and OpenCL C, TFL, Caffe2	Robotics, autopilot car, vision, SLAM, and DSP algorithms
Syntiant	NDP100	DNN	TFL	Mobile phones, hearing equipment, smartwatches, IoT, remote controls ​
Syntiant	NDP101	CNN, RNN, GRU, LSTM	TFL	Mobile phones, smart homes, remote controls, smartwatches, IoT
Syntiant	NDP102	CNN, RNN, GRU, LSTM	TFL	Mobile phones, smart homes, remote controls, smartwatches, IoT
Syntiant	NDP120	CNN, RNN, GRU, LSTM	TFL	Mobile phones, smart home, wearables, PC, IoT endpoints, media streamers, AR/VR
Syntiant	NDP200	FC, Conv, DSConv, RNN-GRU, LSTM	TFL	Mobile phones, smart homes, security cameras, video doorbells
Think Silicon	Nema PicoXS	DNN	----	Wearable and embedded devices
Tesla	FSD	CNN	Pytorch	Automotive
Verisilicon	VIP9000	All modern DNN	TF, Pytorch, TFL, DarkNet, ONNX	Can perform as intelligent eye and intelligent ear at the edge
Untether	TsunAImi	DNN, ResNet-50, Yolo, Unet, RNN, BERT, TCNs, LSTMs	TFL, Pytorch	NLP, Inference at the edge server or data center
UPMEM	UPMEM-PIM	DNN	-----	Sequence alignment: DNA or protein; Genome assembly: Metagenomic analysis

Table 3. Processors, Supported Neural Network Models, Deep Learning Frameworks, And Application Domains.

Company	Product	Supported Neural Networks	Supported Frameworks	Application/benefits
Apple	Apple A14	DNN	TFL	iPhone12 series
Apple	Apple A15	DNN	TFL	iPhone13 series
aiCTX-Synsense	Dynap-CNN	CNN, RNN, Reservoir Computing	SNN	High-speed aircraft, IoT, security, healthcare, mobile
ARM	Ethos78	CNN and RNN	TF, TFL,Caffe2,PyTorch, MXNet, ONNX	Automotive
AIMotive	Apache5 IEP	GoogleNet, VGG16, 19, Inception-v4, v2, MobileNet v1, ResNet50, Yolo v2	Caffe2	Automotives, pedestrian detection, vehicle detection, lane detection, driver status monitoring
Blaize	EI Cano	CNN, YOLO v3	TFL	Fit for industrial, retail, smart-city, and computer-vision systems
BrainChip	Akida1000	CNN in SNN, Mobilenet	MetaTF	Online learning, data analytics, security
BrainChip	AKD500, 1500, 2000	DNN	MetaTF	Smart home, Smart health, Smart City and smart transportation
CEVA	Neuro-s	CNN, RNN	TFL	IoTs, smartphones, surveillance, automotive, robotics, medical
Cadence	Tensilica DNA100	FCC, CNN, LSTM	ONNX, Caffe2, TensorFlow	IoT, smartphones, AR/VR, smart surveillance, Autonomous vehicle
Deepvision	ARA-1	Deep Lab V3, Resnet-50, Resnet-152, MobileNet-SSD, YOLO V3, UNET	Caffe2, TFL, MXNET, PyTorch	Smart retail, robotics, industrial automation, smart cities, autonomous vehicles, and more
Deepvision	ARA-2	Model in ARA-1 and LSTM, RNN,	TFL, Pytorch	Smart retail, robotics, industrial automation, smart cities,
Eta	ECM3532	CNN, GRU, LSTM	---	Smart home, consumer products, medical, logistics, smart industry
Gyrfalcon	LightSpeer 2803S	CNN based, VGG, ResNet, MobileNet;	TFL, Caffe2	High performance audio and video processing
Gyrfalcon	LightSpeer 5801	CNN based, ResNet, MobileNet and VGG16,	TFL, PyTorch & Caffe2	Object Detection and Tracking,NLP, Visual Analysis
Gyrfalcon Edge Server	Janux GS31	VGG, REsNet, MobileNet	TFL, Caffe2, PyTorch	Smart cities, surveillance, object detection, recognition
GreenWaves	GAP9	CNN, mobileNet v1		DSP application
Horizon	Journey 3	CNN, mobilenet v2, efficient net	TFL, Pytorch, ONNX, mxnet, Caffe2	Automotive
Horizon	Journey5/5P	Resnet18, 50, MobileNet v1-v2, ShuffleNetv2, EfficientNet FasterRCNN, Yolov3	TFL, Pytorch, ONNX, mxnet, Caffe2	Automotive
Hailo	Hailo 8 M2	YOLO 3, YOLOv4, CenterPose, CenterNet, ResNet-50	ONNX, TFL	Edge vision applications
Intel	Loihi 2	SNN based NN	Lava, TFL, Pytorch	Online learning, sensing, robotics, healthcare
Intel	Loihi	SNN based NN	Nengo	Online learning, robotics, healthcare and many more
Imagination	PowerVR Series3NX	Mobilenet v3, CNN	Caffe, TFL	smartphones, smart cameras, drones, automotives, wearables,
Imec & GF	DIANA	DNN	TFL, Pytorch	Analog computing in Edge inference
KoniKu	Konicore	synthetic Biology+Silicon	--	Chemical detection, aviation, security
Kalray	MPPA3	Deep network converted to KaNN	Kalray's KANN	Autonomous vehicles, surveillance, robotics, industry, 5G
Kneron	KL720 AI	CNN, RNN, LSTM	ONNX, TFL, Keras, Caffe2	wide applications from automotive to home appliances
Kneron	KL520	Vgg16, Resnet, GoogleNet, YOLO, Lenet, MobileNet, FCC	ONNX, TFL, Keras, Caffe2	Automotive, home, industry and so on.
LeapMind	Efficiera	CNN, YOLO v3, Mobilenet-v2, Lmnet	Blueoil, Python & C++ API	Home, Industrial machinery, surveillance camera, robots
Memryx	MX3	CNN	Pytorh, ONNX, TF, Keras	Automation, surveillance, agriculture, financial
Mythic	M1108	CNN, large complex DNN, Resnet50, YOLO v3, Body25	Pytorch, TFL, and ONNX	Machine Vision, Electronics, Smart Home, UAV/Drone, Edge Server
Mythic	M1076	CNN, Complex DNN, Resnet50, YOLO v3	Pytorch, TFL, and ONNX	Surveillance, Vision, voice, Smart Home, UAV, Edge Server
MobileEye	EyeQ5	DNN		Autonomous driving
MobileEye	EyeQ6	DNN		Autonomous driving
Mediatek	i350	DNN	TFL	Vision and voice, Biotech and Bio-metric measurements
NXP	i.MX 8M+	DNN	TFL, Arm NN, ONNX	Edge Vision
NXP	i.MX9	CNN, Mobilenet v1	TFL, Arm NN, ONNX	Graphics, image, display, audio
NVIDIA	AGX Orin	DNN	TF, TFL, Caffe, Pytorch	Robotics, Retail, Traffic, Manufacturing
Qualcomm	QCS8250	CNN, GAN, RNN	TFL	smartphone, tablet, support 5G, video and image processing
Qualcomm	Snapdragon 888+	DNN	TFL	Smartphone, tablet, 5G, gaming, video upscaling, image & video processing,
RockChip	rk3399Pro	VGG16, ResNEt50, Inception4	TFL, Caffe, mxnet, ONNX, darknet	Smart Home, City, and Industry; face recognition, driving monitoring,
Rokid	Amlogic A311D	Inception V3, YoloV2, YOLOV3	TFL, Caffe2 Darknet	High-performance multimedia
Samsung	Exynos 2100	CNN	TFL	Smartphone, tablet, advanced image signal processing (ISP), 5G
Samsung	HBM-PIM	DNN see youtube to write on int	Pytorch, TFL	Supercomputer and AI application
Synopsis	EV7x	CNN, RNN, LSTM	OpenCV, OpenVX and OpenCL C, TFL, Caffe2	Robotics, autopilot car, vision, SLAM, and DSP algorithms
Syntiant	NDP100	DNN	TFL	Mobile phones, hearing equipment, smartwatches, IoT, remote controls ​
Syntiant	NDP101	CNN, RNN, GRU, LSTM	TFL	Mobile phones, smart homes, remote controls, smartwatches, IoT
Syntiant	NDP102	CNN, RNN, GRU, LSTM	TFL	Mobile phones, smart homes, remote controls, smartwatches, IoT
Syntiant	NDP120	CNN, RNN, GRU, LSTM	TFL	Mobile phones, smart home, wearables, PC, IoT endpoints, media streamers, AR/VR
Syntiant	NDP200	FC, Conv, DSConv, RNN-GRU, LSTM	TFL	Mobile phones, smart homes, security cameras, video doorbells
Think Silicon	Nema PicoXS	DNN	----	Wearable and embedded devices
Tesla	FSD	CNN	Pytorch	Automotive
Verisilicon	VIP9000	All modern DNN	TF, Pytorch, TFL, DarkNet, ONNX	Can perform as intelligent eye and intelligent ear at the edge
Untether	TsunAImi	DNN, ResNet-50, Yolo, Unet, RNN, BERT, TCNs, LSTMs	TFL, Pytorch	NLP, Inference at the edge server or data center
UPMEM	UPMEM-PIM	DNN	-----	Sequence alignment: DNA or protein; Genome assembly: Metagenomic analysis

Memryx [297] released an inference processor, MX3. This processor computes deep learning models with 4, 8, or 16 bit weight and BF16 activation functions. MX3 consumes about 1 W of power and computes with 5 TFLOPS. This chip stores 10 million parameters on a die, and thus needs more chips for implementing larger networks.

MobileEye and STMicroelectronics released EyeQ 5 SoC for autonomous driving [193]. EyeQ 5 is 4 times faster than their earlier version, EyeQ 4. It can produce 2.4 TOPS/W and goes up to 24 TOPS with 10 W of power [194]. Recently, MobileEye has announced their next generation processor, EyeQ6, which is around 5x faster than EyeQ5 [195]. For INT8 precision, EyeQ5 performs 16 TOPS, and EyeQ6 shows 34 TOPS [196].

NXP introduced their edge processor i.MX 8M+ for the targeted applications in vision, multimedia, and industrial automations [86]. The system includes a powerful Cortex-A53 processor integrated with an NPU. The neural network performs 2.3 TOPS with 2W of power consumption. The neural computation supports INT16 precision [87]. NXP is scheduled to launch its next AI processor, iMX9, in 2023 with more features and efficiency [85].

NVIDIA released the Jetson Nano, which is able to run multiple applications in parallel, such as image classification, object detection, segmentation, and speech processing [197]. This developer kit is supported by the NVIDIA JetPack SDK and is able to run state-of-the-art AI models. The Jetson Nano consumes around 5-10 W of power and computes 472 GFLOPS in FP16 precision. The new version of Jetson Nano B01 can perform 1.88 TOPS 198].

NVIDIA released Jetson Orin, which includes specialized development hardware, AGX Orin. It is embedded with 32GB of memory, has a 12-core CPU, and can exhibit a computing performance of 275 TOPS while using INT8 precision [199]. The computing is powered by NVIDIA ampere architecture with 2048 cores, 64 tensor cores, and 2 NVDLA v2.0 accelerators for deep learning [199].

Qualcomm developed the QCS8250 SoC for intensive camera and edge applications [200]. This processor supports wifi and 5G for the IoTs. A quad hexagon vector extension V66Q with hexagon DSP is used for machine learning. An integrated NPU is used for advanced video analysis. The NPU supports INT8 precision and runs at 15 TOPS [201]. Qualcomm has released the Snapdragon 888+ 5G processor for use in smartphones. It takes the smartphone experience to a new level with AI-enhanced gaming, streaming, and photography [202]. It includes a 6th generation Qualcomm AI engine with the Qualcomm Hexagon780 CPU [203,204]. The throughput of the AI engine is 32 TOPS with 5 W of power consumption [203]. The Snapdragon 8 Gen2 mobile platform was presented at the HotChips 2023 conference and exhibited 60% better energy efficiency than the Snapdragon 8 in INT4 precision.

Samsung announced the Exynos 2100 AI edge processor for smartphones, smartwatches, and automobiles [205]. Exynos supports 5G network and performs on-device AI computations with triple NPUs. They fabricate using 5nm extreme UV technology. The Exynos 2100 consumes 20% lower power and delivers 10% higher performance than Exynos 990. Exynos 2100 can perform up to 26 TOPS, and it is 2 times more power efficient than the earlier version of Exynos [206]. A more powerful mobile processor, Exynos 2200, was released recently.

SiMa.ai [300] introduced the MLSoC for computer vision applications. MLSoc is implemented on TSMC 16nm technology. The accelerator can compute 50 TOPS while consuming 10 W of power. MLSoC uses INT8 precision in computation. The processor has 4 MB of on-chip memory for the deep learning operations. The processor is 1.4x more efficient than Orin, measured in frames/W.

Tsinghua and Polar Bear Tech released their QM930 accelerator consisting of seven chiplets. The chiplets are organized as one hub chiplet and six side chiplets, forming a Hub-Side processor. The processor is implemented in 12nm CMOS technology. The total area for the chiplets is 209 mm² for seven chiplets. However, the total substrate area of the processor is 1089 mm². The processor can compute with INT4, INT8, and INT16 precision, showing peak performances of 40, 20, and 10 TOPS, respectively. The system energy efficiency is 1.67 TOPS/W while computed in INT8. The power consumption can be varied from 4.5 to 12 W.

Verisilicon brought VIP 9000 for face and voice recognition. It adopts Vivante's latest VIP V8 NPU architecture for processing neural networks [207]. The computing engine supports INT8, INT16, FP16, and BF16. The performance can be scaled from 0.5 to 100 TOPS [208].

Synopsis developed the EV7x multi-core processor family for vision applications [209]. The processor integrates vector DSP, vector FPU, and a neural network accelerator. Each VPU supports a 32-bit scalar unit. The MAC can be configured for INT8, INT16, or INT32 precisions. The chip can achieve up to 2.7 TOPS in performance [210].

Tesla designed the FSD processor which was manufactured by Samsung for autonomous vehicle operations [211]. The SoC processor includes 2 NPUs and one GPU. The NPUs support INT8 precision, and each NPU can compute 36.86 TOPS. The peak performance of the FSD chip is 73.7 TOPS. The total TDP power consumption of each FSD chip is 36 W [211].

Several other companies have also developed edge processors for various applications but did not share hardware performance details on their websites or through publicly available publications. For instance, Ambarella [307] has developed various edge processors for automotive, security, consumer, and IoTs for industrial and robotics applications. Ambarella’s processors are SoC types, mainly using ARM processors and GPUs for DNN computations.

ii. Neuromorphic Edge AI Processor

In 2022, the global market value of neuromorphic chips was 3.7 billion, and by 2028, the estimated market value is projected to be $ 27.85 Billion [212]. The neuromorphic processors described in this section utilize spike-based processing.

Synsense (formerly AICTx) has introduced a line of ultra-low power neuromorphic processors: DYNAP-CNN, XYLO, DYNAP-SE2, and DYNAP-SEL [15]. Of these, we were able to find performance information on only the DYNAP-CNN chip. This processor is fabricated on a 22 nm process technology and has a die area of 12 mm². Each chip can implement up to a million spiking neurons, and a collection of DYNAP-CNN chips can be utilized to implement a larger CNN architecture. The chip utilizes asynchronous processing circuits [213].

BrainChip introduced the Akida line of spiking processors. The AKD1000 has 80 NPUs, 3 pJ/synaptic operation, and around 2 W of power consumption [147]. Each NPU consists of eight neural processing engines that run simultaneously and control convolution, pooling, and activation (ReLu) operations [148]. Convolution is normally carried out in INT8 precision, but it can be programmed for INT 1, 2, 3 or 4 precisions while sacrificing 1-3% accuracy. BrainChip has announced future releases of smaller and larger Akida processors under the AKD500, AKD1500, and AKD2000 labels [148]. A trained DNN network can be converted to SNN by using the CNN2SNN tool in the Meta-TF framework for loading a model into an Akida processor. This processor also has on-chip training capability, thus allowing the training of SNNs from scratch by using the Meta-TF framework [146].

GrAI Matters Lab (GML) developed and optimized a neuromorphic SoC processor named as VIP for computer vision application. VIP is a low power and low latency AI processor, with 5-10 W of power consumption, and the latency is 10x less than the NVIDIA nano [214]. The target applications are for audio/video processing on the end devices.

IBM developed the TrueNorth neuromorphic spiking system for real-time tracking, identification, and detection [10]. It consists of 4096 neurosynaptic cores and 1 million digital neurons. The typical power consumption is 65 mW, and the processor can execute 46 GSOPS/W, with 26 pJ per synaptic operation [10,215]. The total area of the chip is 430 mm², which is almost 14x bigger than that of Intel’s Loihi 2.

Innatera announced a neuromorphic chip that is fabricated using TSMC’s 28 nm process [216]. When tested with audio signals [217], each spike event consumed about 200 fJ, while the chip consumed only 100 uW for each inference event. The target application areas are mainly audio, healthcare, and radar voice recognition [217].

Intel released the Loihi [9], a spiking neural network chip in 2018, and an updated version, the Loihi 2 [9], in 2021. The Loihi 2 is fabricated using Intel’s 7nm technology and has 2.3 billion transistors with a chip area of 31mm². This processor has 128 neuron cores and 6 low power x86 cores. It can evaluate up to 1 million neurons and 120 million synapses. The Loihi chips support online learning. Loihi processors support INT8 precision. Loihi 1 can deliver 30 GSOPS with 15 pJ per synaptic operation [218]. Both Loihi 1 and Loihi 2 consume similar amounts of power (110mW and 100mW, respectively [219]). However, the Loihi 2 outperforms the Loihi 1 by 10 times. The chips can be programmed through several frameworks, including, Nengo, NxSDK, and Lava [150]. The latter is a framework developed by Intel and is being pushed as the primary platform to program the Loihi 2.

IMEC develops a RISC-V processor based digital neuromorphic processor with 22nm process technology in 2022 [220]. They have implemented an optimized BF-16 processing pipeline inside the neural process engine. The computation can also support INT4 and INT8 precision. They have used 3-layer memory to reduce the chip area.

Koniku combines biological machines with silicon devices to design a micro electrode array system core [12]. They are developing the hardware and algorithm that mimics the smell sensory receptor that is found in some animal noses. However, the detailed device parameters are not publicly available. The device is mainly used in security, agriculture, and safe flight operation [221].

iii. PIM Processor

PIM processors are becoming an alternative for AI application due to the low latency, high energy efficiency, and reduced memory requirements. PIMs are the analog, and in-place computing architecture thus, it reduces the burden of additional storage modules. However, there are some digital presents the schematic representation of a common PIM computing architecture. It consists of the crossbar array (NxM) of the popular storage devices. The crossbar array performs as the weight storage and analog multiplier. The storage devices could be SRAM, RRAM, PCM, STT-MRAM or a flash memory cell. The computing array is equipped with the peripheral circuits, a data converter (ADC or DAC), sensing circuits, and a write circuit for the crossbar. Some of the PIM processors are discussed in this section.

Imec and GlobalFoundries have developed DIANA, a processor that includes both digital and analog cores for DNN processing. The digital core is employed for widely parallel computation, whereas the analog in-memory computing (AiMC) core enables much higher energy efficiency and throughput. The core uses a 6T-SRAM array with a size of 1152x512. Imec developed the architecture, while the chip is fabricated using GlobalFoundries' 22FDX solution [222]. It is targeted for a wide range of edge applications, from smart speakers to self-driving vehicles. The analog component (AiMC) computes at 29.5 TOPS with and the digital core computes at 0.14 TOPS. The digital and analog components have efficiencies of 4.1 TOPS/W and 410 TOPS/W, respectively in isolation. The overall system energy efficiency of DIANA is 14.4 TOPS/W for Cifar-10 [223].

Gyrfalcon has developed multiple PIM processors, including the Lightspeeur 5801, 2801, 2802, and 2803 [24]. The architecture uses digital AI processing in-memory units that compute a series of matrices for CNN. The Lightspeeur 5801 has a performance of 2.8 TOPS at 224 mW and can be scaled up to 12.6 TOPS/W. The Lightspeeur 2803S is their latest PIM processor for the advanced edge, desktop, and data center deployments [19]. Each Lightspeeur 2803S chip performs 16.8 TOPS while consuming 0.7W of power, giving an efficiency of 24 TOPS/W. Lightspeeur 2801 can compute 5.6 TOPS with an energy efficiency of 9.3 TOPS/W. Gyrfalcon introduced its latest processor, Lightspeeur 2802, using TSMC’s magnetoresistive random access memory technology. Lightspeeur 2802 exhibits an energy efficiency of 9.9 TOPS/W. Janux GS31 is the edge inference server which is built with 128 Lightspeeur 2803S chips [225]. It can perform 2150 TOPS and consumes 650 W.

Mythic has announced its new analog matrix processor, M1076 [18]. The latest version of Mythic’s PIM processor reduced its size by combining 76 analog computing tiles, while the original one (M1108) uses 108 tiles. The smaller size offers more compatibility to implant on edge devices. The processor supports 79.69M on-chip weights in the array of flash memory, and 19,456 ADCs for parallel processing. There is no external DRAM storage required. The DNN models are quantized from FP32 to INT8 and retrained in Mythic’s analog compute engine. A single M1076 chip can deliver up to 25 TOPS while consuming 3 W of power [90]. The system can be scaled for high performance up to 400 TOPS by combining 16 of M1076 chips which requires 75 W [88,89].

Samsung has announced its HBM-PIM machine learning-enabled memory system with PIM architecture [16]. This is the first successful integration of the PIM architecture of high bandwidth memory. This technology incorporates the AI processing function into Samsung HBM2 Aquabolt to speed up high-speed data processing in supercomputers. The system delivered 2.5x performance with 60% lower energy consumption than the earlier HBM1 [16]. Samsung LPDDR5-PIM memory technology for mobile device technology is targeted to bring the AI capability in the mobile device without connecting to the data center [226]. The HBM-PIM architecture is different from the traditional analog PIM architecture as outlined in Figure 2. It does not require data-conversion and sensing circuits as the actual computation is taking place in the near-computing module in the digital domain. Instead, it places a GPU surrounded by HBM stacks to realize the parallel processing and minimize data movement [227]. Therefore, this is similar to a dataflow processor

Syntiant has developed a line of flash-memory array based edge inference processors, such as NDP10x, NDP120, NDP200 [228]. Syntiant's PIM architecture is very energy efficient, and it combines with an edge optimized training pipeline. A Cortex-M0 is embedded in the system that runs the NDP firmware. The NDP10x processors can hold 560k weights of INT4 precision and perform MAC operation with an INT8 activation. The training pipeline can build neural networks for various applications according to the specifications with optimized latency, memory size, and power consumption [228]. Syntiant released five different versions of application processors. NDP 100 is their first AI processor, updated in 2020 with a tiny little dimension of 2.52 mm² and ultra-low power consumption, less than 140 μW [229]. Syntiant continues to provide more PIM processors named NDP 101, 102, 120, and NDP 200 [230,231,232]. The application domains are mainly smartphone, wearable, and hearable pieces of equipment, remote controls, IoT endpoints. The neural computations are supported by INT 1, 2, 4, and 8 precision. The energy efficiency of the NDP 10x series is 2 TOPS/W [233], which includes NDP100, NDP 101, and NDP 102. NDP 120 [235] and NDP 200 exhibit 1.9 GOPS/W and 6.4 GOPS/W [232], respectively.

Untether has developed its PIM AI accelerator card TsunAImi [236] for inference at the data center or in the server. The heart of the TsunAImi is four runAI200 chips which are fabricated by TSMC in standard SRAM arrays. Each runAI200 chip features 511 cores and 192MB of SRAM memory. runAI200 computes in INT8 precision and performs 502 TOPS at 8 TOPS/W, which is 3x more than NVIDIA’s Ampere A100 GPU. The resulting performance of TsunAImi system is 2008 TOPS with 400 W [237].

UPMEMP PIM innovatively placed thousands of DPU units within the DRAM memory chips [238]. The DPUs are controlled by high-level applications running on the main CPU. Each DIMM consists of 16 PIM-enabled chips. Each PIM has 8 DPUs, making 128 DPUs of total DPUs for each UPMEM [239].

However, the system is massively parallel, and up to 2560 DPUs units can be assembled as a unit server with 256 GB PIM DRAM. The computing power is 15x of x86 server with the main CPU. The throughput is benchmarked for INT32 bit addition is 58.3 MOPS/DPU [240]. This system is suitable for DNA sequencing, Genome comparison; Phylogenetics; Metagenomic analysis, and more [241].

iv. Processor in the Industrial Research

The PIM computing paradigm is still in its rudimentary stage, however a very promising system for efficient MAC operation and low power edge application. A good number of industries and industry-academic research collaborations are escalating the development of PIM technologies and architectures. In this section, the PIM processors in the industrial and industry-university collaboration have been briefly discussed. The recent developments of the PIM research are tabulated in Table 4.

Table 4. Edge Processors in Industrial Research with Technology, Process Technology, and Numerical Precision.

Research Group	Name	Power (W)	Process (nm)	Area (mm2)	Precision INT/FP*	Performance (TOPS)	E. Eff. (TOPS/W)	Architecture	Reference
TSMC+ NTHU		2.13E-03	22	6	2,4,8	4.18E-01	195.7	PIM	[259]
TSMC		0.037	22	0.202	4,8,12,16	3.3	89	PIM	[257]
TSMC		0.00142	7	0.0032	4	0.372	351	PIM	[258]
Samsung+GIT	FORMS	66.36	32	89.15	8	0.0277		PIM	[262]
IBM + U Patra	HERMES	9.61E-02	14	0.6351	8	2.1	21.9	PIM	[260]
Samsung+ASU	PIMCA	0.124		20.9	1,2	4.9	588	PIM	[261]
Intel+Cornell U	CAPE		7	9	4			PIM	[263]
SK Hynix	AiM			6.08		1		PIM	[264]
TSMC	DCIM	0.0116	5	0.0133	4,8	2.95	254	PIM	[265]
Samsung		0.3181	4	4.74	4,8,16, FP16	39.3	11.59	Dataflow	[266]
Alibaba + FU		0.0212	28	8.7	3	0.97	32.9	Dataflow	[267]
Alibaba + FU		0.072	65	8.7	3	1	8.6	Dataflow	[267]
Alibaba		0.978	55	602.22	8			Dataflow	[268]
TSMC+ NTHU		0.00227	22	18	2,4,8	0.91	960.2	PIM	[269]
TSMC + NTHU		0.00543	40	18	2,4,8	3.9	718	PIM	[270]
TSMC+GIT		0.000350	40	0.027		0.0092	26.56	PIM	[271]
TSMC+GIT		0.131	40	25	1-8,1-8,32	7.989	60.64	PIM	[272]
Intel+UC		0.0090	28	0.033	1,1	20	2219	PIM	[273]
Intel+UC		0.0194	28	0.049	1-4,1	4.8	248	PIM	[274]
TSMC+ NTHU	nvCIM	0.00398	22	6	2,4	5.12	1286.4	PIM	[275]
Pi2star +NTHU		0.00841.	65	12	1-8	3.16	75.9	PIM	[276]
Pi2star +NTHU		0.00652	65	9	4,8	2	35.8	PIM	[277]
Tsing+NTHU		0.273	28	6.82	12	4.07	27.5	Dataflow	[278]
Samsung		0.381	4	4.74	4,8,FP16	19.7	11.59	Dataflow	[279]
Renesas Electronics		4.4	12			60.4	13.8	Dataflow	[280]
IBM		6.20	7	19.6	2,4,FP(8,16,32)	102.4	16.5	Dataflow	[281]
Intel + IMTU	QNAP	0.132	28	3.24	8	2.3	17.5	Dataflow	[282]
Samsung		0.794	5	5.46	8,16	29.4	13.6	Dataflow	[283]
Sony		0.379	22	61.91	8,16,32	1.21	4.97	Dataflow	[284]
Mediatek		1.05	7	3.04		3.6	13.32	Dataflow	[285]
Pi2star		0.099	65	12	8	1.32	13.3	Dataflow	[286]
Mediatek		0.0012	12			0.102	86.24	PIM	[287]
TSMC+NTHU		0.10	22	8.6	8,8,8	6.96	68.9	PIM	[288]
TSMC+NTHU		0.099	22	9.32	8,8,8	24.8	251	PIM	[289]
ARM+Harvard		0.04	12		FP4	0.734	18.1	Dataflow	[290]
ARM+Harvard		0.045	12		FP8	0.367	8.24	Dataflow	[291]
TSMC + NTHU		0.0037	22	18	8,8,22	0.59	160.1	Dataflow	[292]
STMircroelectronics		0.738	18	4.24	1,1	229	310	Dataflow	[293]
STMircroelectronics		0.740	18	4.19	4,4	57	77	Dataflow	[294]
MediaTek		0.711	12	1.37	12	16.5	23.2	PIM	[309]
TSMC+ NTHU			16		8		98.5	PIM	[308]
Renesas Electronics		5.06	14		8	130.55	23.9	Dataflow	[310]

* Integer Precision is indicated by only precision number(s). Floating point precision is mentioned as FP in the precision column.

Table 4. Edge Processors in Industrial Research with Technology, Process Technology, and Numerical Precision.

Research Group	Name	Power (W)	Process (nm)	Area (mm2)	Precision INT/FP*	Performance (TOPS)	E. Eff. (TOPS/W)	Architecture	Reference
TSMC+ NTHU		2.13E-03	22	6	2,4,8	4.18E-01	195.7	PIM	[259]
TSMC		0.037	22	0.202	4,8,12,16	3.3	89	PIM	[257]
TSMC		0.00142	7	0.0032	4	0.372	351	PIM	[258]
Samsung+GIT	FORMS	66.36	32	89.15	8	0.0277		PIM	[262]
IBM + U Patra	HERMES	9.61E-02	14	0.6351	8	2.1	21.9	PIM	[260]
Samsung+ASU	PIMCA	0.124		20.9	1,2	4.9	588	PIM	[261]
Intel+Cornell U	CAPE		7	9	4			PIM	[263]
SK Hynix	AiM			6.08		1		PIM	[264]
TSMC	DCIM	0.0116	5	0.0133	4,8	2.95	254	PIM	[265]
Samsung		0.3181	4	4.74	4,8,16, FP16	39.3	11.59	Dataflow	[266]
Alibaba + FU		0.0212	28	8.7	3	0.97	32.9	Dataflow	[267]
Alibaba + FU		0.072	65	8.7	3	1	8.6	Dataflow	[267]
Alibaba		0.978	55	602.22	8			Dataflow	[268]
TSMC+ NTHU		0.00227	22	18	2,4,8	0.91	960.2	PIM	[269]
TSMC + NTHU		0.00543	40	18	2,4,8	3.9	718	PIM	[270]
TSMC+GIT		0.000350	40	0.027		0.0092	26.56	PIM	[271]
TSMC+GIT		0.131	40	25	1-8,1-8,32	7.989	60.64	PIM	[272]
Intel+UC		0.0090	28	0.033	1,1	20	2219	PIM	[273]
Intel+UC		0.0194	28	0.049	1-4,1	4.8	248	PIM	[274]
TSMC+ NTHU	nvCIM	0.00398	22	6	2,4	5.12	1286.4	PIM	[275]
Pi2star +NTHU		0.00841.	65	12	1-8	3.16	75.9	PIM	[276]
Pi2star +NTHU		0.00652	65	9	4,8	2	35.8	PIM	[277]
Tsing+NTHU		0.273	28	6.82	12	4.07	27.5	Dataflow	[278]
Samsung		0.381	4	4.74	4,8,FP16	19.7	11.59	Dataflow	[279]
Renesas Electronics		4.4	12			60.4	13.8	Dataflow	[280]
IBM		6.20	7	19.6	2,4,FP(8,16,32)	102.4	16.5	Dataflow	[281]
Intel + IMTU	QNAP	0.132	28	3.24	8	2.3	17.5	Dataflow	[282]
Samsung		0.794	5	5.46	8,16	29.4	13.6	Dataflow	[283]
Sony		0.379	22	61.91	8,16,32	1.21	4.97	Dataflow	[284]
Mediatek		1.05	7	3.04		3.6	13.32	Dataflow	[285]
Pi2star		0.099	65	12	8	1.32	13.3	Dataflow	[286]
Mediatek		0.0012	12			0.102	86.24	PIM	[287]
TSMC+NTHU		0.10	22	8.6	8,8,8	6.96	68.9	PIM	[288]
TSMC+NTHU		0.099	22	9.32	8,8,8	24.8	251	PIM	[289]
ARM+Harvard		0.04	12		FP4	0.734	18.1	Dataflow	[290]
ARM+Harvard		0.045	12		FP8	0.367	8.24	Dataflow	[291]
TSMC + NTHU		0.0037	22	18	8,8,22	0.59	160.1	Dataflow	[292]
STMircroelectronics		0.738	18	4.24	1,1	229	310	Dataflow	[293]
STMircroelectronics		0.740	18	4.19	4,4	57	77	Dataflow	[294]
MediaTek		0.711	12	1.37	12	16.5	23.2	PIM	[309]
TSMC+ NTHU			16		8		98.5	PIM	[308]
Renesas Electronics		5.06	14		8	130.55	23.9	Dataflow	[310]

* Integer Precision is indicated by only precision number(s). Floating point precision is mentioned as FP in the precision column.

Alibaba has developed SRAM and DRAM-based digital CIM and PNM systems for low precision edge applications [267,268]. The CIM architecture uses multiple chiplet modules (MCM) to solve the complex problem instead of a single SoC. The CIM architecture in [267] proposes Computing-on-Memory Boundary (COMB), which is a compromise between in-memory and near-memory computation. This technique exhibits high macro computing energy efficiency and low system power overhead. This CIM architecture demonstrated scaleable MCM systems using a COMB NN processor. The layerwise pipeline mapping schemes are utilized to deploy different sizes of NN for the required operation. The chip operation is demonstrated with keyword spotting, CIFAR-10 image classification, and object detection with tiny-YOLO NN using one, two, and four chiplets.

IBM and the University of Patra together presented their PCM-based CIM processor, HERMES [260]. This CIM is a 256×256 in-memory compute core fabricated in 14nm CMOS process technology for edge inference. HERMES is demonstrated for image classification operation on MNIST and CIFAR-10 datasets.

Samsung technology has been working on various CIM architectures for AI applications in edge to the datacenter. The company has released HBM-PIM recently [226,227]. HBM-PIM is for high-speed memory access, which is fabricated with DRAM in a 20nm process. Samsung and Arizona State University (ASU) presented a PIMCA chip for AI inference [261]. PIMCA chip for AI inference [261]. PIMCA consumes a very low amount of power (124 mW). PIMCA is highly energy efficient, 588 TOPS/W as shown in Table 2. TSMC has designed and fabricated analog [257,258] and digital [265] CIM systems for inference.

Besides TSMC’s own research, the company has multiple CIM research projects on various emerging memory devices such as ReRAM [259], STT-MRAM [269], PCM [270], RRAM [271] and RRAM-SRAM [272] in collaboration with various research groups in the academia. The performance of these macro inference chips has been demonstrated in various high-tier conferences or scientific forums very recently. The best performance is demonstrated in ISSCC 2022 with PCM devices, and it exhibits 5.12 TOPS in 2-bit precision [275], which is 1286.4 TOPS/W. This CIM processor supports INT2 and 4 bit computing precision. The digital CIM system is fabricated with FinFET in 5nm process technology, and it performs 2.95 TOPS and 254 TOPS/W [265].

Besides the AI accelerators introduced above, there are a handful of companies that are working on edge processors. The companies working on neuromorphic processors are MemComputing [213,295], GrAI [214], and iniLabs [296]. Memryx is a recently formed startup which is building high performance and energy efficient AI processors for a wide range of applications, such as transportation, IoT, and industry [297]. It can compute Bfloat16 activation with 4/8/16-bit weight and performs about 5TFLOPS.

7. Performance Analysis of the Edge Processors

This section discusses the performance analysis of the edge processors described earlier. The discussion is focused on different architectures for edge processors. At first, the overall performance is discussed based on the computing performance, power consumption, chip area, and computing precision. Then only PIM processors are discussed. At the end of this section, we have focused on the devices still under research and development or waiting for commercially available.

i. Overall Analysis of the AI Edge Processor

We compare all the edge AI processors listed in the previous section using the following key metrics:

• Performance: tera-operations per second (TOPS).
• Energy efficiency: TOPS/W.
• Power: Watt (W).
• Area: square millimeter (mm²).

Performance: Figure 3 plots the performance vs. power consumption, with different labels for dataflow, neuromorphic, and PIM processors. The processors within a power consumption range of 1W to 60W have a performance of 1 to 275 TOPS. These are geared towards comparatively high-power applications such as surveillance systems, autonomous vehicles, industries, smart cities, and UAVs. The highest throughput processors in this list are the EyeQ6 from MobileEye, the Journey 5 from Horizon, and the Jetson Orin from Nvidia. The Jetson Orin is about 2.15 times faster than both the EyeQ6 and Journey 5. From the company datasheet [], the Jetson Orin has 275 TOPS at INT8 precision for 60W of power. The Orin consumes about 1.5 and 2 times more power than the EyeQ6 and Journey 5, respectively. The processors with a power consumption of less than 1W have a performance from 0.2 GOPS to 17 TOPS.

The IBM NorthPole has 200 TOPS for INT8 precision at 60 W (based on a discussion with IBM). However, the NorthPole can have higher TOPS of 400 and 800 at 4 and 2 bit precision respectively. According to a recent NorthPole article, the maximum power consumption of the NorthPole processor is 74 W [304].

These are targeted for extreme edge and IoT applications. The least power is needed for PIM processors of the NDP series by Syntiant, which are flash-memory based PIM processors [20].

Among neuromorphic processors, Loihi 2 outperforms other neuromorphic processors, except for the Akida AKD1000. The AKD1000 however consumes 20x more power than the Loihi 2 (see Table 2). Although the neuromorphic processors seem less impressive in terms of TOPS vs. W, it is important to note that they generally need far fewer synaptic operations to perform a task, if the task is performed with an algorithm that is natively spiking (i.e., not a deep network implemented with spiking neurons) [298].

The neuromorphic processors consume significantly less energy than other processors for inference tasks [142]. For example, the Loihi processor consumes 5.3x less energy than the Intel Movidious and 20.5x less energy than the Nvidia Jetson Nano [142]. Figure 3 shows that higher performance PIM processors (such as the M1076, M1108, LS-2803S, and AnIA), exhibit similar computing speeds as dataflow or neuromorphic processors at the same range of power consumption (0.5 to 1.5 W).

Precision: Data precision is an important consideration when comparing processor performances. Figure 5 presents the precision of the processors from Figure 3. Figure 5 shows the distribution of precision and total number of processors for each architecture category. A processor may support more than one type of computing precision. Figure 3 and Figure 4 are based on the highest precision supported for each processor.

Among dataflow processors, INT8 is the most widely supported precision for DNN computations. NVIDIA’s Orin achieves 275 TOPS with INT8 precision, the maximum computing speed for INT8 precision in Figure 5. However, some processors utilize INT1 (Efficiara), INT64 (A15, A14, and M1), FP16 (ARA-1, DNA100, Jetson Nano, Snapdragon 888+), and INT16 (Ethos78, and Movidius). Neuromorphic and PIM processors mainly support INT1 to INT8 data precisions. Lower computing precisions generally reduce the inference accuracy. According to [185], VGG-9 and ResNet-18 have accuracy losses of 3.89% and 6.02%, respectively, for inference while computed with INT1 precision. A more in-depth discussion of the relationship between quantization and accuracy is presented in Section 3(i). A higher precision provides better accuracy but incurs more computing costs. Figure 5 shows that the most common precision in the processors examined is INT8. This provides a good balance between accuracy and computational costs.

As shown in Figure 4, and 5, almost all the neuromorphic processors use INT8 for synaptic computations. The exception to this is the AKD1000, which uses INT4 and shows the best performance among neuromorphic processors in terms of operations per second (1.5 TOPS). It however consumes around 18x more power than Loihi processors. At INT8 precision, the Loihi 1 performs 30 GSOPS using 110 mW [218,219], whereas Loihi 2 surpasses this throughput by 10x, with a similar power consumption [9].

As shown in Figure 4,5 PIM processors primarily support precisions of INT1 to INT8. Figure 5 shows the performance of PIM processors in INT4 and INT8 precisions due to the unavailability of data for all supported precisions. Mythic processors (M1108 and M1076) manifest the best performance among PIM processors. Mythic and Syntiant have developed their PIM processors with flash-memory devices. However, Mythic processors require significantly higher power to compute DNNs in INT8 precision with its 76 computing tiles. Syntiant processors use INT4 precision and compute with about 13,000x lower throughput than Mythic M1076 while consuming about 6000x less power. The Syntiant processors are limited to smaller networks with up to 64 classes in NDP10x. On the other hand, Mythic processors can handle 10x more weights with greater precision [233]. The Samsung DRAM architecture-based PIM processor uses computing modules near the memory banks and supports INT64 precision [16].

Energy Efficiency:Figure 6 presents the performance vs. energy efficiency of dataflow, PIM and neuromorphic processors. The efficiency determines the computing throughput of a processor per watt. The energy efficiency of all PIM processors is located within 1 to 16 TOPS/W, whereas most of the dataflow processors are located in 0.1 to 55 TOPS/W. The PIM architecture reduces latency by executing the computation inside the memory modules, which increases computing performance and reduces power consumption. Loihi 2 manifests best energy efficiency among all neuromorphic processors. Energy Efficiency vs. power consumption, as shown in Figure 7, gives us a better understanding about the processors. Loihi 2 shows better energy efficiency than many high performances edge AI processors while it consumes very low power. Ergo is the most energy efficient processor among all dataflow processors, which shows 55 TOPS/W. Ergo is the most energy efficient processor among all dataflow processors, which processors, which shows 55 TOPS/W.

Chip Area: The area is an important factor for choosing a processor for AI applications on edge devices. Modern processor technologies are pushing the boundaries to fabricate very high density and superior performance at the same time. The smaller die area and lower power consumption is very important for battery powered edge devices. The chip area is related to the cost of the silicon fabrication and also defines the application area. A smaller chip with high performance is desirable for edge applications.

Figure 7 and Figure 8 present the power consumption and performance, respectively vs. the chip area. It can be observed that in general, both the power consumption and performance increase with chip area. Based on the available chip sizes, the NothPole has the largest chip size of 800 mm² and performs 200 TOPS in INT8. The lowest area chips have a dataflow architecture. Figure 9 shows the energy efficiency vs. area as the combined relationship of Figures 8, and 9. In this Figure, the PIM processors form a cluster. The overall energy efficiency of this PIM cluster is higher than dataflow and neuromorphic processors of similar chip area. Some of the dataflow processors (such as Nema Pico, Efficiera, and IMG 4NX) exhibit higher energy efficiency and better performance vs area than other processors.

ii. AI Edge Processor with PIM Architecture

While Figure 3, Figure 4, Figure 5, Figure 6, Figure 7, Figure 8, Figure 9, Figure 10 and Figure 11 describe processors of all types, Figure 12 shows the relation between only PIM processors that are either announced as products or are still in the industrial research. The research processors are presented in the conferences, such as ISSCC and VLSI. The PIM processors at the lower right corner of Figure 11 are candidates for data center and intensive computing applications [236,237,238,239,240,241]. The PIM processors with higher energy efficiency are suitable for edge and IoT applications because of their smaller size, lower power consumption, and higher energy efficiency. From Figure 12 we can see that most of the PIM processors under industrial research show higher energy efficiency than announced processors. This indicates that future PIM processors are likely to have much better performances and efficiencies.

The PIM processors compute the MAC operation inside the memory array thus reducing the data transfer latency. Generally, PIM processors compute in lower integer/fixed-point precision. A PIM processor generally supports INT 1-16 precision. However, according to our study, we found around 59% of the PIM processors support INT8 precision for MAC operation, as shown in Figure 5. The low precision computation is faster and requires lower power consumption compared to dataflow processors. The PIM edge processors consume 0.0001 to 4 W for deep learning inference applications, as presented in Table 2 and Figure 3. However, the dataflow processors suffer from high memory requirements and latency issues and consume higher power than most of the PIM processors to get the same performance as we see in Figure 3-5.

From Figure 3-4, Syntient’s NDP200 consumes below 1mW power and shows the highest performance for extreme edge applications. Mythic M1108 consumes 4W and exhibits the highest performance (35 TOPS) than all dataflow and neuromorphic processors that consumes below 10 W of power. For the same chip area M1108 consumes 9x less power than Tesla’s dataflow processor FSD, while FSD computes 2x faster than M1108 as presented in Figure 8-9.

The processors below 100 mm², Gyrfalcon’s LS2803 shows the highest performance except EyeQ5. However, EyeQ5 consumes about 14x higher power and performs 1.4x better than LS2803. The benefit of deploying PIM processors for edge applications is its high performance with less power consumption, and the PIM processors reduce the computing latency significantly as the MAC operations perform inside the memory array.

iii. Edge Processor in the Industrial Research

Several companies, along with their collaborators, are developing edge computing architectures and infrastructures with state-of-the-art performance. Figure 13 shows the power consumption vs. energy efficiency of the industrial research processors which were presented at high tier conferences (such as ISSCC, VLSI). The chart includes both PIM [259,260,261,262,263,264,265,269,270,271,272,273,274,275,276,277,287,288,289] and dataflow [266-268,278-286.290-294] processors.

Renesas Electronics presented a near-memory system in ISSCC 2024 developed in a 14nm process that achieved 130.55 TOPS with 23.9TOPS/W [310]. TSMC and National Tsing Hua University presented one near memory system in a 22nm CMOS process in ISSCC 2023 that computes 0.59 TOPS and 160 TOPS/W in 8-8-26 bit (input-weight-output) precision [292]. This system showed the highest energy efficiency amongst the near-memory and dataflow processors. The energy efficiency is achieved by a 90% input sparsity, while a 50% input sparsity gives an energy efficiency of 46.4 TOPS/W [292]. Alibaba and Fudan University [267] presented a processing near memory system in ISSCC 2022 with 0.97 TOPS and 32.9 TOPS/W energy efficiency while computing with INT3 precision. This accelerator is a SRAM based near-memory computing architecture. Tsing Microelectronics and Tsinghua University [278] demonstrated a dataflow processor in ISSCC 2022 for NLP and computer vision applications, which shows an energy efficiency of 27.5 TOPS/W in INT12 precision. Renesas Electronics [280] exhibited 13.8 TOPS/W in INT8 computing precision. Many other companies, such as IBM [281], Sony [284], Mediatek [285], and Samsung [279,283] have also demonstrated their research work on the dataflow edge processors with energy efficiencies around 11 to 18 TOPS/W

PIM processors generally manifest better energy efficiencies than dataflow processors. TSMC and National Tsing Hua University presented a PIM system in a 16nm CMOS process in ISSCC 2024 that achieved 98.5 TOPS/W in 8-8-23 precision (input-weight-output) [308]. Mediatek and TSMC presented a digital PIM system in ISCC 2024 developed in a 3nm process that achieved 23.2 TOPS/W with 16.5 TOPS performance [309]. Intel and Columbia University demonstrated a PIM processor [273] in ISSCC 2022 that shows the performance and energy efficiency of 2219 TOPS/W and 20 TOPS respectively, which is around 33x more efficient than the processor mentioned in [269].

However, the former processor uses extremely lower precision (INT1). TSMC and Tsinghua University [288] presented a PIM accelerator in ISSCC 2023 with 6.96 TOPS and 68.9 TOPS/W, which is about 12x faster than the near memory computing system presented in [292], while computing in INT8 precision. STMicroelectronics presented a PIM accelerator that computes 57 TOPS and 77 TOPS/W in INT4 precision [294] and performs about 25x better than near memory computing presented in ISSCC 2021 [282]. TSMC and Tsinghua University [270] presented a PCM based processor in ISSCC 2022, which shows 65 TOPS/W in INT8 precision, and is around 5x better than [292]. Samsung and Arizona State University [261] demonstrated PIMCA in VLSI’ 2021 and showed an energy efficiency of 437 TOPS/W computed in INT1 precision. Other companies such as TSMC and collaborators [259,269,270,271,272,275,288,292], Samsung and collaborators [261], Intel and collaborators [263,273,274,282], and HK Hynix [264] have demonstrated their PIM processors in recent ISSCC and VLSI conferences.

8. Summary

This article reviewed different aspects and paradigms of recent AI edge processors released or announced recently by various tech companies. About 100 edge processors were examined. This work however did not cover the DNN algorithms, HPC computing processors, or cloud computing. We categorized state-of-the-art edge processors and analyzed their performance, area, and energy efficiency to support the research community in edge computing. Multiple processing architectures including dataflow, neuromorphic, and PIM were examined. The performance and power consumption were analyzed for narrowing down the edge AI processors for specific applications. The deep neural networks and software frameworks supported were discussed and presented in tables.

Several of the edge processors offer on-chip retraining in real-time. This enables retraining of networks without having to send sensitive data to the cloud, thus increasing security and privacy. Intel’s Loihi 2 and Brainchip’s Akida processor can be retrained on local data for personalized applications and faster response rates.

This study found the power consumption and performance of processors varies in different architectures, and application domains. For extreme wearable edge devices, the power consumption ranges from 100 μW to a few mW and computethroughput is around 1 GOPS. We found that many applications require higher computing performance, such as video processing, and autonomous car operations. These high-performance applications consume a higher amount of power than extreme edge processors. For example, IBM's NorthPole computes at 200 TOPS with INT8, while consuming 60 W of power. This study found that for the same range of power consumption and chip size, PIM architectures perform better than dataflow or neuromorphic processors. This review found that the PIM processors show significant energy efficiency and consume less power compared to dataflow and neuromorphic processors. For example, the Mythic M1108 is a PIM processor and has the highest performance (35 TOPS), among dataflow and neuromorphic processors that consume less than 10W of power. Neuromorphic processors are highly efficient for performing computation with less synaptic operations but may not be ideal for deep learning applications yet.

There are several emerging deep learning applications that are receiving significant interest. This includes generative AI models, such as transformer models used in ChatGPT and DALL-E for automated art generation. Transformer models are taking the AI world so aggressively, manifested by their super-intelligent chatbot and search queries. Generative AI models are also taking place in image and creative art generation. Transformer engines are mainly designed for data centers or cloud applications, but some processors, such as NVIDIA Hopper H100 [305], can be used for edge workloads. Samsung has released digital PIM for generative AI applications in the data center and edge [306]. ResNet, GoogleNet, and YOLO models are also being used in various industries for facial recognition, lane keeping assistance, and surveillance. Deep reinforcement learning is becoming popular for autonomous learning models in dynamic environments. All of these applications could benefit from highly efficient specialized processors that could run the applications locally, without the need for cloud access. Future directions for industry could be to implement these algorithms in emerging non-Von Neumann computing paradigms for low power computing on edge devices. Current dataflow processors, such as the NVIDIA Orin, or the IBM NorthPole would probably be able to handle these applications without any changes. More emerging architectures, such as PIM and neuromorphic technologies, may need more enhancements to enable these applications to run on edge devices.