The deployment of Large Language Models (LLMs) on edge devices represents a paradigm shift in artificial intelligence, transitioning from cloud-centric dependence to pervasive, privacy-preserving on-device intelligence. However, enabling multi-billion parameter models on battery-powered devices (e.g., AR glasses, mobile agents) faces severe bottlenecks in memory bandwidth, energy efficiency, and thermal dissipation. This survey presents a comprehensive taxonomy of efficient inference techniques, rigorously categorizing recent advancements in post-training quantization, structural pruning, speculative decoding, and heterogeneous system scheduling. Unlike prior reviews, we formalize the trade-offs between model fidelity and hardware constraints through the lens of the Roofline Model and thermodynamic limits. We further identify critical industrial challenges in enabling “always-on” agents, proposing a roadmap for future hardware-algorithm co-design.
References
[1]
OpenAI (2023) GPT-4 Technical Report.
[2]
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., et al. (2023) Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv:2307.09288.
[3]
Horowitz, M. (2014) 1.1 Computing’s Energy Problem (and What We Can Do about It). 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC), San Francisco, 9-13 February 2014, 10-14.
[4]
Williams, S., Waterman, A. and Patterson, D. (2009) Roofline: An Insightful Visual Performance Model. Communications of the ACM, 52, 65-76. https://doi.org/10.1145/1498765.1498785
[5]
Frantar, E., Ashkboos, S., Hoefler, T. and Alistarh, D. (2023) GPTQ: Accurate Post-Training Quantization for Generative Pre-Trained Transformers. arXiv:2210.17323.
[6]
Dettmers, T., Lewis, M., Belkada, Y. and Zettlemoyer, L. (2022) LLM.int8(): 8-Bit Matrix Multiplication for Transformers at Scale. arXiv:2208.07339.
[7]
Lin, J., Tang, J., Tang, H., Yang, S., Chen, W., Wang, W., et al., (2023) AWQ: Activation-Aware Weight Quantization for LLM Compression and Acceleration. arXiv:2306.00978.
[8]
Huang, R. (2026) Soft-Vote Ensemble Distillation: Aggregation Strategy Comparison for Sub-50K Edge Students. arXiv Preprint.
[9]
Leviathan, Y., Kalman, M. and Matias, Y. (2023) Fast Inference from Transformers via Speculative Decoding. arXiv:2211.17192.
[10]
Huang, R. (2026) ConfSched: Confidence-Threshold Routing for Efficient Edge-Cloud Activity Recognition. arXiv Preprint.
[11]
Gu, A. and Dao, T. (2023) Mamba: Linear-Time Sequence Modeling with Selective State Spaces. arXiv:2312.00752.
[12]
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C., Gonzalez, J.E., et al. (2023) Efficient Memory Management for Large Language Model Serving with PagedAttention. arXiv:2309.06180.
[13]
Meta AI (2024) ExecuTorch: A PyTorch-Based Library for On-Device Inference.
[14]
Erickson, J.A., Botelho, A.F., Peng, Z., Huang, R., Kasal, M.V. and Heffernan, N.T. (2021) “Is It Fair? Automated Open Response Grading. Proceedings of the 14th International Conference on Educational Data Mining, 29 June-2 July 2021, 682-687.
