1. Introduction

2. Background

3. Proposed System Architecture

4. Theoretical Model for MLP-Based Classifiers

• $max({\displaystyle \frac{1}{{\theta }_{project}^{switch}}},{\displaystyle \frac{k}{b}})={\displaystyle \frac{k}{b}}$ and $max({\displaystyle \frac{1}{{\theta }_{project}^{server}}},{\displaystyle \frac{m}{b}})={\displaystyle \frac{m}{b}}$. Since $k<m$ is the fundamental setting of random projection and $b>0$, we have ${\displaystyle \frac{k}{b}}<{\displaystyle \frac{m}{b}}$. Therefore inequality 7 obviously holds.
• $max({\displaystyle \frac{1}{{\theta }_{project}^{switch}}},{\displaystyle \frac{k}{b}})={\displaystyle \frac{k}{b}}$ and $max({\displaystyle \frac{1}{{\theta }_{project}^{server}}},{\displaystyle \frac{m}{b}})={\displaystyle \frac{1}{{\theta }_{project}^{server}}}$. In this case, ${\displaystyle \frac{k}{b}}<{\displaystyle \frac{m}{b}}\le {\theta }_{project}^{server}$. Therefore, similar to the previous case, inequality 7 still holds.
• $max({\displaystyle \frac{1}{{\theta }_{project}^{switch}}},{\displaystyle \frac{k}{b}})={\displaystyle \frac{1}{{\theta }_{project}^{switch}}}$ and $max({\displaystyle \frac{1}{{\theta }_{project}^{server}}},{\displaystyle \frac{m}{b}})={\displaystyle \frac{1}{{\theta }_{project}^{server}}}$. In other words, the random projection time is the bottleneck of both systems. In this case, the inequity holds if ${\theta }_{project}^{switch}>{\theta }_{project}^{server}$, which holds as long as DSPs on the FPGA have a speed advantage over the server in terms of matrix multiplication.
• $max({\displaystyle \frac{1}{{\theta }_{project}^{switch}}},{\displaystyle \frac{k}{b}})={\displaystyle \frac{1}{{\theta }_{project}^{switch}}}$ and $max({\displaystyle \frac{1}{{\theta }_{project}^{server}}},{\displaystyle \frac{m}{b}})={\displaystyle \frac{m}{b}}$. In this case, whether inequality 7 holds is more data-dependent and hardware-dependent. However, we found that the case is very unlikely to appear practically.

5. Experiment Setup and Performance Analysis for MLP-Based Classifier

Figure 6. Experimental results showing the impact of the proposed system on training time and accuracy.

Figure 6. Experimental results showing the impact of the proposed system on training time and accuracy.

6. Dataset-Specific Case Study for MLP-Based Classifier

7. Experiment Setup and Performance Analysis for kNN-Based Classifier

7.3. Dataset-Specific Case Study

The epsilon dataset is a dense dataset with high dimensionality, the original dataset contains 400000 entries with 2000 dimensions. To reduce the simulation time, a portion of the dataset is taken for simulation. In this case study, 7000 data entries are randomly sampled from the dataset for training and testing, and a train-test split portion of 0.8 is used.

To calculate $T_{single}$: According to Equation 11, the transmission time is simply the time spent forwarding the packet by the switch. This time is determined by the design of the network switch and the length of the packet. For simplicity, we assume that the packet is of fixed size, and the network switch operates the same as the NetFPGA-SUME [40] base design, but to match the state-of-the-art FPGA, we assume it is implemented on an AMD Alveo U280 FPGA. Based on [44], $T_{trans}$ contributes little to the total time due to the domination of the RP process. Another reason to ignore this parameter is that both the base case and our proposed design contain packet forwarding, reducing their impact on the speed up which is calculated by the ratio of the two cases. Therefore, $T_{single}$ is reduced to $T_{cos}+T_{sort}$. This gives us $T_{single}$ = 0.29270s. Note that $T_{single}$ includes the time taken for the fitting process, this is to ensure a fair comparison with the $T_{ensemble}$.

To calculate $T_{ensemble}$: As stated in Equation 12 and 13, there are two situations for $T_{ensemble}$, which can be further divided into two phases, $T_{store}$ and $T_{query}$, both sharing the $T_{rp}$ and $T_{bin}$ processes as illustrated in Figure 10.

Table 5. Experiment results

Table 5. Experiment results

Name	# entry	M	m	sp	Lib Time CPU Cosine/s	Estimated time/s FPGA	FPGA speedup	Total memory reduction	Trigger size Terabytes
epsilon [32]	7000	2000	1000	0.9	0.2927	0.095917	3.05	3.95	1.138
catsndogs	10000	512	100	0.6	0.3065	0.103337	2.97	21.01	4.369
AD [33]	3279	1558	200	0.6	0.1111	0.012718	8.74	27.70	6.647
stock	10074	2000	1000	0.8	0.4916	0.151872	3.24	26.67	1.280
realsim [34]	3614	20958	8000	0.8	1.5200	0.652816	2.33	6.99	1.677
gisette [35]	6000	5000	1000	0.8	0.5027	0.078037	6.44	22.22	3.200
REJAFDA [36]	1996	6824	700	0.6	0.2153	0.012023	17.91	519.92	8.318
qsar_oral [37]	8992	1024	800	0.8	0.3953	0.111949	3.53	12.8	0.819

Situation 1 — when the on-board memory can hold the dataset. In this case, there is no packet transmissions between the FPGA and the workers, removing one level of data transmission. The Store and Query in Figure 10 will both run on the FPGA-based switch. However, we do need to consider the DDR RAM latency for storing and reading. For worst-case analysis, we assume the read and write latency takes 200 cycles = 1 us. Since it is smaller than $T_{rp}$, it can be hidden within the 2.5us window. Therefore $T_{DDR}^{fpga}$ can be replaced by the initial latency of 1us, assuming read and write have the same latency.

For the Store phase, with the random projection being a time-consuming part of the design and to support different RP matrices, we design our system to choose between two types of matrix multiplication modules. A DSP-based multiplier for the RP matrices with large fixed-point values, or an adder tree-based multiplier for RP matrices contains only 0, -1 and 1. To avoid over utilising the resources, we assume that each switch will implement only two copies of the RP module.

With the DSP-based multiplier already discussed in [44], we implemented an adder tree with 2000 input of type 16-bit fixed-point using Vivado HLS. It only uses 5% LUT on the U280 board. Since the epsilon dataset has an input dimension equal to 2000, we allocate 4000 DSPs or two 2000 input adder trees for random projection. Using FPGA to binarise a vector can be simply done by extracting the sign bit of each element in the vector, this is included in our adder tree design. The overhead of this operation is trivial, thus adding this functionality to the DSP-based multiplier will not affect the estimation outcome. Therefore, $T_{rp}$ = 0.5*1000*5ns = 2.5us, meaning the projected data will be generated every 2.5us. The total time to generate the entire projected dataset in the store phase is $T_{store}$ = 0.9 * 0.8 * 7000 * 2.5us = 12.60 ms.

In the Query Phase, the query data is first projected to the same latent space using the database, which takes 2.5us. Then this projected query will be compared with the entire database, computing the Hamming distance along the way; Burst reading from the DRAM can be initiated within this 2.5us window, and by the time the projection is ready, the entire database will be ready to use. To compute the Hamming distance, another adder tree is implemented. This adder tree only takes as input a single bit, therefore significantly reducing resource usage. This circuit is much simpler than the adder tree for random projection, and our implementation result shows less than one per cent of LUT is used on the target platform.

If pipelined, the initial latency can be calculated as $lo{g}_{2}N$, where N is the length of the input bit vector. For an input of 1000 bits, the initial latency is around 11 cycles. The initial latency can be amortised with a large input dataset. For the sake of simplicity, we will ignore this latency in our estimation as the datasets used in our experiments are relatively large. When reading the data from DRAM, with a reading port data width of 1024, it is possible to read one data from the DRAM every cycle. Therefore, computing the Hamming distance of a single vector can be achieved every cycle, and the total time is taken $n_{data}$ cycles. For the epsilon dataset, this is $T_{ham}^{fpga}$ = 0.9 * 0.8 * 7000 * 5ns = 25.2us.

Finally, for the sorting time, we need to sort an integer vector of length $n_{data}$. We implemented an iterative merge sorter using Vivado HLS with 6300 inputs using a short data type (signed 16-bit), the latency result is shown in Table 6. Since the size of each input is bounded by the number of elements in the input vector, this allows further optimisation of the input data width of the design. We use this simple design as a proof of concept, since designing a high-performance sorter is not the main object of this paper. We are aware that better designs for retrieving top-k exist, this design serves as the upper bound of taking the top-k element. The implementation result shows little resource usage and is able to run under 200MHz. In fact, it is designed to return the result in log2(N) stages, so the output latency is deterministic given the running frequency.

In practice, our pipelined implementation with minimum optimisation (array partition) using Vivado HLS has an initial latency of 0.410ms and an interval latency of 6304 cycles = 31.52 us, as shown in Table 6. In theory, the interval latency (N) is the latency of a single stage and the initial latency ($Nlog2\left(N\right)$) is the total length of the pipeline and the latency. The implementation summary shows the actual latency corresponds to the theoretical value. The resource usage is shown in Table 7. Putting everything together, the total time taken by test data is $T_{query}$ = 7000 * (1 - 0.8) * (2.5us + 25.2us + 31.52us) + 0.410ms = 83.318ms Adding the time for the two phases gives the $T_{ensemble}$ = 12.6 ms + 83.318 = 95.917 ms

Situation 2 — when all data are stored in the worker. Under this condition, the speed-up will be dominated by the CPU computation time. KNeighborClassifier uses SciPy’s hamming distance which returns a normalised distance with type double. This does not fit our design purpose which requires only a small non-normalised integer distance as the output type. As a result, we implemented our own version of kNN that can take our own hamming distance function as input. Re-running the experiments on our implementation and the speed up under this condition is 7.5. Although this speedup appears to be higher than it is in Situation 1, it does not reflect the real behaviour of our design. This is because our own implementation does not leverage parallel computing on the CPU. The problem then becomes memory bound, meaning the time difference is dominated by the memory retrieval time of a full-precision dataset and a hamming-encoded one, which outweighs the benefit of computing RP on the FPGA. Getting the same comparison of Situation 1 requires the exact same software implementation of the Scikit learning library, which is beyond the scope of this study. Fortunately, it requires the dataset to be massive to trigger Situation 2; this is presented in the last column of Table 5. The computation detail is shown in Section 8.2.2.

In respect of classification accuracy, our simulation results show that by configuring the experiment with [projection dimension=1000, sample portion=0.9, number of split=18], we could recover the accuracy loss caused by the RP and binarisation process. In terms of memory usage, depending on the data type of the original dataset, there is a fixed compression ratio W using the hamming space i.e., the bit width of the data type to 1. For example, W=32 for the epsilon dataset that uses floating-point numbers. The total memory reduction (the ratio of the original memory size to the compressed memory size) can be calculated as:

$$R={\displaystyle \frac{M}{m}}\times {\displaystyle \frac{W}{n\_split\times sp}}$$

(16)

where ${\displaystyle \frac{M}{m}}$ is the ratio between the original dimension and the projected dimension, $sp$ is the sample portion, and $n_{split}$ is the size of the ensemble to recover the accuracy.

7.3.1. Quantifying the Gap between Forwarding Latency and Computation Latency

Based on the official NetFPGA-SUME wiki [14], the average port-to-port latency of the reference switch is 823ns. This corresponds to $T_{trans}^{fpga}$ in Equation 12 and Equation 13. For the epsilon dataset, $T_{rp}^{fpga}$ takes more than 3 times longer than $T_{trans}^{fpga}$. As long as the total transmission latency is shorter than $T_{rp}^{fpga}$, it can be hidden under the random projection latency, therefore, we could keep increasing the levels of the hierarchy until $T_{trans}^{fpga}$ saturates $T_{rp}^{fpga}$. In addition, if a certain level of buffering is enabled in the hierarchy, then it is able to control the packet arriving interval by sending packets back-to-back, with a controlled burst interval. As long as the burst interval is smaller than $T_{rp}^{fpga}$, then it can also be hidden under $T_{rp}^{fpga}$. Therefore we only need to add the initial latency to the whole model, which is 823ns times the number of levels in the hierarchy. Compared to $T_{ensemble}$ which is in millisecond scale, this is still small.

8. Discussion

8.2. kNN-Based Classifier

Table 5 summarises the experiment results for all the datasets; it also contains all the information required to compute the speed up and the memory reduction. The # entry column represents the total number of data instances used to compute kNN. Column $sp$ is the minimum sample portion to successfully recover the accuracy given the ensemble size. The data shows that our FPGA-based design offers speedup for datasets of different sizes and dimensionality. Notice that for the REJAFDA dataset, the speedup is around 17 times, this is due to the high sparsity of the dataset, which results in a very high $M/m$ ratio. To generalise our performance model, we have created a function that takes as input the first four columns of the table and returns the estimated FPGA execution time.

8.2.1. Relationship between Speed-up and Projected Dimension

Figure 11 presents the speed-up under different projected dimensions. The dotted line at the top of each figure represents the based line execution time for the original dataset. The markers at the bottom of each figure represent the execution time of our system with different projected dimensionalities (the lower the better). The triangle markers in the middle of each figure show the speed-up for each projected dimension.

In general, the speed-up for FPGA decreases linearly with the projected dimension. With a few exceptions such as Figure 11b and Figure 11c, where a constant speed-up for the FPGA can be observed. This is because the projected dimension is smaller than the input size of the matrix multiplier. With the current design, all dimensions smaller than the input threshold will experience a constant speed-up. In terms of prediction accuracy, all the presented dimensions are able to achieve the same accuracy as the baseline. If speed-up is vital, it is possible to further reduce the project dimension to gain higher speed-up, with the sacrifice of the prediction accuracy.

8.2.2. The Effect of Memory Reduction on System Scalability

Equation 16 presents the memory reduction ratio after summing all copies of data stored in all members of the ensemble. Multiply Equation 16 by $n\_split$ gives the memory reduction ratio ($R_s$) for a single copy of the dataset. Taking the epsilon dataset which utilises 18 splits as an example. $R_s$ = 18 * 3.95 = 71.11, which means for a single FPGA board, we will be able to store 71.11 times more data. If we only consider off-chip memory for storage, for an FPGA board with 32GB available DDR RAM, based on the assumption that each FPGA board holds two copies of the dataset, which means 16GB for each copy. With the memory reduction, we are able to store data with sizes up to 1.138TB on a single FPGA node, which is enough for most available datasets online.

Table 8. Comparison with previous designs

Table 8. Comparison with previous designs

Designs	Platform	Metrics	Input Data dimension
[45]	AP	Euclidean	unspecified
[28]	AP	Hamming	64 - 256
CHIP-kNN [24]	Cloud FPGA	Euclidean	2 - 128
PRINS [46]	In-storage	Euclidean	1 - 384
kNN-MSDF [47]	FPGA SoC	Euclidean	4 - 17
[48]	FPGA SoC	Hamming	64
[29]	FPGA	Manhattan	16384
Rosetta [49]	FPGA	Hamming	unspecified
[50]	FPGA	Hamming	<24
kNN-STUFF [51]	FPGA	Euclidean	<50
[25]	FPGA	Euclidean	64
[52]	FPGA	Euclidean	1 - 16
[26]	FPGA	Euclidean	16
[53]	FPGA	Hamming	<50
This work	FPGA network switch	Hamming	<20958

9. Conclusion and Future Works

References

1. Cooke, R.A.; Fahmy, S.A. A model for distributed in-network and near-edge computing with heterogeneous hardware. Future Generation Computer Systems 2020, 105, 395–409. [Google Scholar] [CrossRef]
2. Amazon. Amazon EC2 F1 Instances.
3. Firestone.; et al. Azure Accelerated Networking:{SmartNICs} in the Public Cloud. In Proceedings of the NSDI, 2018.
4. N.McKeown. PISA: Protocol independent switch architecture. https://opennetworking.org/wp-content/uploads/2020/12/p4-ws-2017-p4-architectures.pdf, 2015.
5. Hemmatpour, M.; Zheng, C.; Zilberman, N. E-commerce bot traffic: In-network impact, detection, and mitigation. In Proceedings of the 2024 27th Conference on Innovation in Clouds, Internet and Networks (ICIN). IEEE, 2024, pp. 179–185.
6. Datta, S.; Kotha, A.; Venkanna, U.; et al. XNetIoT: An Extreme Quantized Neural Network Architecture For IoT Environment Using P4. IEEE Transactions on Network and Service Management 2024. [CrossRef]
7. Zheng, C.; Xiong, Z.; Bui, T.T.; Kaupmees, S.; Bensoussane, R.; Bernabeu, A.; Vargaftik, S.; Ben-Itzhak, Y.; Zilberman, N. IIsy: Hybrid In-Network Classification Using Programmable Switches. IEEE/ACM Transactions on Networking 2024. [CrossRef]
8. Nguyen, H.N.; Nguyen, M.D.; Montes de Oca, E. A Framework for In-network Inference using P4. In Proceedings of the Proceedings of the 19th International Conference on Availability, Reliability and Security, 2024, pp. 1–6.
9. Paolini, E.; De Marinis, L.; Scano, D.; Paolucci, F. In-Line Any-Depth Deep Neural Networks Using P4 Switches. IEEE Open Journal of the Communications Society 2024. [CrossRef]
10. Zhang, K.; Samaan, N.; Karmouch, A. A Machine Learning-Based Toolbox for P4 Programmable Data-Planes. IEEE Transactions on Network and Service Management 2024. [CrossRef]
11. Cooke, R.A.; Fahmy, S.A. Quantifying the latency benefits of near-edge and in-network FPGA acceleration. In Proceedings of the EdgeSys, 2020.
12. Li.; et al. Accelerating distributed reinforcement learning with in-switch computing. In Proceedings of the ISCA. IEEE, 2019.
13. Meng, J.; Gebara, N.; Ng, H.C.; Costa, P.; Luk, W. Investigating the Feasibility of FPGA-based Network Switches. In Proceedings of the 2019 IEEE 30th International Conference on Application-specific Systems, Architectures and Processors (ASAP). IEEE, 2019, Vol. 2160, pp. 218–226.
14. NetFPGA. NetFPGA SUME: Reference Learning Switch Lite. https://github.com/NetFPGA/NetFPGA-SUME-public/wiki/NetFPGA-SUME-Reference-Learning-Switch-Lite, 2021.
15. Dai, Z.; Zhu, J. Saturating the transceiver bandwidth: Switch fabric design on FPGAs. In Proceedings of the FPGA, 2012, pp. 67–76.
16. Papaphilippou, P.; et al. Hipernetch: High-Performance FPGA Network Switch. TRETS 2021. [CrossRef]
17. Rebai, A.; Ojewale, M.A.; Ullah, A.; Canini, M.; Fahmy, S.A. SqueezeNIC: Low-Latency In-NIC Compression for Distributed Deep Learning. In Proceedings of the Proceedings of the 2024 SIGCOMM Workshop on Networks for AI Computing, 2024, pp. 61–68.
18. Le, Y.; Chang, H.; Mukherjee, S.; Wang, L.; Akella, A.; Swift, M.M.; Lakshman, T. UNO: Uniflying host and smart NIC offload for flexible packet processing. In Proceedings of the Proceedings of the 2017 Symposium on Cloud Computing, 2017, pp. 506–519.
19. Tork, M.; Maudlej, L.; Silberstein, M. Lynx: A smartnic-driven accelerator-centric architecture for network servers. In Proceedings of the Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems, 2020, pp. 117–131.
20. Papaphilippou.; et al. High-performance FPGA network switch architecture. In Proceedings of the FPGA, 2020.
21. Achlioptas, D. Database-friendly random projections: Johnson-Lindenstrauss with binary coins. JCSS 2003. [CrossRef]
22. Li, P.; Hastie, T.J.; Church, K.W. Very sparse random projections. In Proceedings of the SIGKDD, 2006.
23. Fox, S.; Tridgell, S.; Jin, C.; Leong, P.H. Random projections for scaling machine learning on FPGAs. In Proceedings of the 2016 International Conference on Field-Programmable Technology (FPT). IEEE, 2016, pp. 85–92.
24. Lu.; et al. CHIP-KNN: A configurable and high-performance k-nearest neighbors accelerator on cloud FPGAs. In Proceedings of the ICFPT. IEEE, 2020.
25. Pu, Y.; et al. An efficient knn algorithm implemented on FPGA based heterogeneous computing system using opencl. In Proceedings of the FCCM. IEEE, 2015.
26. Song, X.; Xie, T.; Fischer, S. A memory-access-efficient adaptive implementation of kNN on FPGA through HLS. In Proceedings of the ICCD. IEEE, 2019.
27. An, F.; Mattausch, H.J.; Koide, T. An FPGA-implemented Associative-memory Based Online Learning Method. IJMLC 2011. [Google Scholar] [CrossRef]
28. Lee.; et al. Similarity search on automata processors. In Proceedings of the IPDPS. IEEE, 2017.
29. Stamoulias, I.; et al. Parallel architectures for the kNN classifier–design of soft IP cores and FPGA implementations. TECS 2013. [Google Scholar] [CrossRef]
30. Charikar, M.S. Similarity estimation techniques from rounding algorithms. In Proceedings of the STOC, 2002.
31. Indyk, P.; Motwani, R. Approximate nearest neighbors: towards removing the curse of dimensionality. In Proceedings of the STOC, 1998.
32. LIBSVM Data: Classification (Binary Class). https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/binary.html#epsilon.
33. Kushmerick, N. Internet Advertisements. UCI Machine Learning Repository, 1998. [CrossRef]
34. LIBSVM Data: Classification (Binary Class). https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/binary.html#real-sim.
35. Guyon, I.; Gunn, S.; Ben-Hur, A.; Dror, G. Gisette. UCI Machine Learning Repository, 2008. [CrossRef]
36. Pinheiro, R.; et al. REJAFADA. UCI Machine Learning Repository, 2023. [CrossRef]
37. QSAR oral toxicity. UCI Machine Learning Repository, 2019. [CrossRef]
38. Dua, D.; Graff, C. UCI Machine Learning Repository, 2017.
39. Chollet, F.; et al. Keras. https://keras.io, 2015.
40. Zilberman, N.; et al. NetFPGA SUME: Toward 100 Gbps as research commodity. IEEE micro 2014. [Google Scholar] [CrossRef]
41. Kushmerick, N. Learning to Remove Internet Advertisements. In Proceedings of the Agents, 1999, p. 175.
42. Xilinx. Alveo U280 Data Center Accelerator Card.
43. Radford.; et al. Learning transferable visual models from natural language supervision. In Proceedings of the ICML. PMLR, 2021.
44. Meng.; et al. Fast and accurate training of ensemble models with FPGA-based switch. In Proceedings of the ASAP. IEEE, 2020.
45. Lee.; et al. Application codesign of near-data processing for similarity search. In Proceedings of the IPDPS. IEEE, 2018.
46. Kaplan, R.; et al. PRINS: Processing-in-storage acceleration of machine learning. IEEE Transactions on Nanotechnology 2018. [Google Scholar] [CrossRef]
47. Gorgin.; et al. kNN-MSDF: A Hardware Accelerator for k-Nearest Neighbors Using Most Significant Digit First Computation. In Proceedings of the SOCC. IEEE, 2022.
48. de Araújo.; et al. Hardware-Accelerated Similarity Search with Multi-Index Hashing. In Proceedings of the DASC/PiCom/CBDCom/CyberSciTech. IEEE, 2019.
49. Zhou.; et al. Rosetta: A realistic high-level synthesis benchmark suite for software programmable FPGAs. In Proceedings of the FPGA, 2018.
50. Ito.; et al. A Nearest Neighbor Search Engine Using Distance-Based Hashing. In Proceedings of the ICFPT. IEEE, 2018.
51. Vieira, J.; Duarte, R.P.; Neto, H.C. kNN-STUFF: kNN STreaming unit for FPGAs. IEEE Access 2019. [Google Scholar] [CrossRef]
52. Hussain, H.; et al. An adaptive FPGA implementation of multi-core K-nearest neighbour ensemble classifier using dynamic partial reconfiguration. In Proceedings of the FPL. IEEE, 2012.
53. Jiang, W.; Gokhale, M. Real-time classification of multimedia traffic using FPGA. In Proceedings of the ICFPL. IEEE, 2010.