Version 1
: Received: 14 June 2024 / Approved: 17 June 2024 / Online: 17 June 2024 (10:29:04 CEST)
How to cite:
Zhang, Y.; Zhao, J.; Liao, Z.; Wu, W.; Michieli, U.; Cannistraci, C. V. Brain-Inspired Sparse Training in MLP and Transformers with Network Science Modeling via Cannistraci-Hebb Soft Rule. Preprints2024, 2024061136. https://doi.org/10.20944/preprints202406.1136.v1
Zhang, Y.; Zhao, J.; Liao, Z.; Wu, W.; Michieli, U.; Cannistraci, C. V. Brain-Inspired Sparse Training in MLP and Transformers with Network Science Modeling via Cannistraci-Hebb Soft Rule. Preprints 2024, 2024061136. https://doi.org/10.20944/preprints202406.1136.v1
Zhang, Y.; Zhao, J.; Liao, Z.; Wu, W.; Michieli, U.; Cannistraci, C. V. Brain-Inspired Sparse Training in MLP and Transformers with Network Science Modeling via Cannistraci-Hebb Soft Rule. Preprints2024, 2024061136. https://doi.org/10.20944/preprints202406.1136.v1
APA Style
Zhang, Y., Zhao, J., Liao, Z., Wu, W., Michieli, U., & Cannistraci, C. V. (2024). Brain-Inspired Sparse Training in MLP and Transformers with Network Science Modeling via Cannistraci-Hebb Soft Rule. Preprints. https://doi.org/10.20944/preprints202406.1136.v1
Chicago/Turabian Style
Zhang, Y., Umberto Michieli and Carlo Vittorio Cannistraci. 2024 "Brain-Inspired Sparse Training in MLP and Transformers with Network Science Modeling via Cannistraci-Hebb Soft Rule" Preprints. https://doi.org/10.20944/preprints202406.1136.v1
Abstract
Dynamic sparse training is an effective strategy to alleviate the training and inference demands of artificial neural networks. However, current sparse training methods face a challenge in achieving high levels of sparsity while maintaining performance comparable to that of their fully connected counterparts. The Cannistraci-Hebb training (CHT) method produces an ultra-sparse advantage compared to fully connected training in various tasks by using a gradient-free link regrowth method, which relies solely on the network topology. However, its rigid selection based on link prediction scores may lead to epitopological local minima, especially at the beginning of the training process when the network topology might be noisy and unreliable. In this article, we introduce the Cannistraci-Hebb training soft rule (CHTs), which applies a flexible approach to both the removal and regrowth of links during training, fostering a balance between exploring and exploiting network topology. Additionally, we investigate the network topology initialization using several approaches, including the bipartite scale-free and bipartite small-world network models. Empirical results show that CHTs can surpass the performance of fully connected networks with MLP architecture by using only 1% of the connections (99% sparsity) on the MNIST, EMNIST, and Fashion MNIST datasets and can provide remarkable results with only 0.1% of the links (99.9% sparsity). In some MLPs for image classification tasks, CHTs can reduce the active neuron network size to 20% of the original nodes (neurons), demonstrating a remarkable ability to generalize better than fully connected architectures, reducing the entire model size. This represents a relevant result for dynamic sparse training. Finally, we present evidence from larger network models such as Transformers, with 10% of the connections (90% sparsity), where CHTs outperform other prevalent dynamic sparse training methods in machine translation tasks.
Keywords
dynamic sparse training; network science; Cannistraci-Hebb theory
Subject
Computer Science and Mathematics, Artificial Intelligence and Machine Learning
Copyright:
This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
