MobileLLM

MobileLLM is a small language model optimized for mobile devices, focusing on designing high-quality LLMs with less than a billion parameters to adapt to the practicality of mobile deployment. Contrary to conventional wisdom, this study emphasizes the importance of model architecture in small LLMs. Through a deep and thin architecture, combined with embedding sharing and grouped query attention mechanisms, MobileLLM achieves significant improvements in accuracy and proposes a block-level weight sharing method that does not increase model size and has small latency overhead. Additionally, the MobileLLM family of models showed significant improvements over previous small models in the chat benchmark and approached LLaMA-v2 7B correctness in the API call task, highlighting the capabilities of small models in common device use cases.

MobileLLM is targeted at developers and researchers who need to deploy efficient language models on mobile devices. Due to its small number of parameters, it is suitable for resource-constrained environments such as mobile devices and edge computing devices. In addition, MobileLLM provides an effective solution for enterprises and developers who want to reduce cloud costs and latency.