MS, A.; VG, J.; PS, D. Efficient Hybrid Inference for LLMs: Reward-Based Token Modelling with Selective Cloud Assistance. Preprints2024, 2024091208. https://doi.org/10.20944/preprints202409.1208.v1
APA Style
MS, A., VG, J., & PS, D. (2024). Efficient Hybrid Inference for LLMs: Reward-Based Token Modelling with Selective Cloud Assistance. Preprints. https://doi.org/10.20944/preprints202409.1208.v1
Chicago/Turabian Style
MS, A., Jithin VG and Ditto PS. 2024 "Efficient Hybrid Inference for LLMs: Reward-Based Token Modelling with Selective Cloud Assistance" Preprints. https://doi.org/10.20944/preprints202409.1208.v1
Abstract
Large language models (LLMs) are known for their exceptional performance across a range of natural language processing tasks, but their deployment comes at a high computational and financial cost. On the other hand, smaller language models (SLMs), which can be deployed on lower-cost edge devices, struggle to match the performance of their larger counterparts. This paper presents a novel hybrid inference approach that leverages the strengths of both model types while minimizing reliance on costly cloud-based LLMs. Unlike existing methods that route entire queries to either an SLM or a cloud LLM, our approach introduces a reward-based mechanism to dynamically determine the involvement of the cloud LLM during token generation. Specifically, each token predicted by the SLM is evaluated against a reward score, and only when this score falls below a certain threshold is the cloud LLM consulted for assistance in the next token prediction. This method not only reduces the traffic to the cloud LLM, thereby lowering costs, but also allows for flexible control over response quality depending on the reward score threshold. Experimental results demonstrate that our approach significantly reduces cloud LLM usage with minimal impact on overall response quality, offering a cost-effective solution for deploying high-performance language models.
Keywords
llm; hybrid-llm; reward-token
Subject
Computer Science and Mathematics, Artificial Intelligence and Machine Learning
Copyright:
This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
