Since transformer-based language models were introduced in 2017, they have been shown to be extraordinarily effective across a variety of NLP tasks including but not limited to language generation. The introduction and widespread adoption of these LLMs, which encode extremely high-dimensional semantic spaces, comes at a significant cost in terms of system and computational resource requirements, requirements that have reshaped the entire chip (GPU) and data center industry as hardware, cloud, and infrastructure providers try to keep up with the demand. This has motivated the research community to develop a variety of design strategies that optimize the use of these resources; however, computational requirements continue to grow in proportion to model size and complexity. With this study, we introduce a framework called Layer Cake for the precise measurement of the relative computational resource requirements necessary for text classification using a variety of classifiers from across the Machine Learning (ML) and Deep Learning (DL) landscape, leveraging different language model families and focusing on the forms of language representation used in different test scenarios. We find that while LLMs do yield the best results across classifiers on average, these improvements come at a significant computational overhead. For example, from a Macro-F1 score perspective, LLM-based classifiers outperform their static embedding language model counterparts (Word2Vec, FastText & GloVe), even when encapsulated in DL architectures such as Convolutional Neural Networks or Long-Short-Term Networks by 8.87% on average, and perform 12.73% better than ML classifiers such as Support Vector Machines and Logistic Regression models. However, this uptick in model performance comes at a computational overhead cost of 4398.07% compared to the GPU requirements of static word embedding DL classifiers, and a 4126.02% increase in computation time relative to ML classifiers, the latter of which are CPU, rather than GPU, bound.
References
[1]
Mikolov, T., Chen, K., Corrado, G.S. and Dean, J. (2013) Efficient Estimation of Word Representations in Vector Space. arXiv: 1301.3781.
[2]
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł. and Polosukhin, I. (2017) Attention Is All You Need. In: Guyon, I., Von Luxburg, U., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S. and Garnett, R., Eds., Advances in Neural Information Processing Systems, Volume 30, Curran Associates, Inc., 6000-6010.
[3]
Naveed, H., Khan, A.U., Qiu, S., Saqib, M., Anwar, S., Usman, M., etal. (2025) A Comprehensive Overview of Large Language Models. ACMTransactionsonIntelligentSystemsandTechnology. https://doi.org/10.1145/3744746
[4]
Fan, L., Li, L., Ma, Z., Lee, S., Yu, H. and Hemphill, L. (2024) A Bibliometric Review of Large Language Models Research from 2017 to 2023. ACMTransactionsonIntelligentSystemsandTechnology, 15, 1-25. https://doi.org/10.1145/3664930
[5]
Capraro, V., Lentsch, A., Acemoglu, D., Akgun, S., Akhmedova, A., Bilancini, E., etal. (2024) The Impact of Generative Artificial Intelligence on Socioeconomic Inequalities and Policy Making. PNASNexus, 3, 191. https://doi.org/10.1093/pnasnexus/pgae191
[6]
Sevag Kertechian, K. and El-Farr, H. (2024) Dissecting the Paradox of Progress: The Socioeconomic Implications of Artificial Intelligence. In: El-Farr, H., Ed., The ChangingLandscapeofWorkplaceandWorkforce, IntechOpen. https://doi.org/10.5772/intechopen.1004872
[7]
Worth, P.J. (2023) Word Embeddings and Semantic Spaces in Natural Language Processing. InternationalJournalofIntelligenceScience, 13, 1-21. https://doi.org/10.4236/ijis.2023.131001
[8]
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D. and Sutskever, I. (2019) Language Models Are Unsupervised Multitask Learners.
[9]
Devlin, J., Chang, M.W., Lee, K. and Toutanova, K. (2019) BERT: Pretraining of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference oftheNorthAmericanChapteroftheAssociationforComputationalLinguistics: HumanLanguageTechnologies, Volume 1 (LongandShortPapers), Minneapolis, June 2019, 4171-4186.
[10]
Liu, Y.H., Ott, M., Goyal, N., Du, J.F., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., etal. (2019) Roberta: A Robustly Optimized Bert Pre-Training Approach. arXiv: 1907.11692.
[11]
McCallum, A. and Nigam, K. (1998) A Comparison of Event Models for Naive Bayes Text Classification. Technical Report CMU-CS-98-137, Carnegie Mellon University.
[12]
Genkin, A., Lewis, D.D. and Madigan, D. (2007) Large-Scale Bayesian Logistic Regression for Text Categorization. Technometrics, 49, 291-304. https://doi.org/10.1198/004017007000000245
[13]
Joachims, T. (1998) Text Categorization with Support Vector Machines: Learning with Many Relevant Features. In: Nédellec, C. and Rouveirol, C., Eds., Machine Learning: ECML-98. ECML 1998, Springer, 137-142. https://doi.org/10.1007/bfb0026683
[14]
Pennington, J., Socher, R. and Manning, C. (2014) GloVe: Global Vectors for Word Representation. Proceedings of the 2014 Conference on Empirical Methods inNaturalLanguageProcessing (EMNLP), Doha, October 2014, 1532-1543. https://doi.org/10.3115/v1/d14-1162
[15]
Bojanowski, P., Grave, E., Joulin, A. and Mikolov, T. (2017) Enriching Word Vectors with Subword Information. Transactions of the Association for Computational Linguistics, 5, 135-146. https://doi.org/10.1162/tacl_a_00051
[16]
Zhang, X., Zhao, J.B. and LeCun, Y. (2015) Character-Level Convolutional Networks for Text Classification. Proceedings of the 29th International Conference on Neural Information Processing Systems—Volume 1, Montreal, 7-12 December 2015, 649-657.
[17]
Kim, Y. (2014) Convolutional Neural Networks for Sentence Classification. Proceedings ofthe 2014 ConferenceonEmpiricalMethodsinNaturalLanguageProcessing (EMNLP), Doha, October 2014, 1746-1751. https://doi.org/10.3115/v1/d14-1181
[18]
Hochreiter, S. and Schmidhuber, J. (1997) Long Short-Term Memory. NeuralComputation, 9, 1735-1780. https://doi.org/10.1162/neco.1997.9.8.1735
[19]
Houlsby, N., Giurgiu, A., et al. (2019) Parameter-Efficient Transfer Learning for NLP. Proceedingsofthe 36thInternationalConferenceonMachineLearning, Long Beach, 9-15 June 2019, 2790-2799.
[20]
Li, X.L. and Liang, P. (2021) Prefix-Tuning: Optimizing Continuous Prompts for Generation. arXiv: 2101.00190.
[21]
Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y.Z., Wang, S., Wang, L. and Chen, W.Z. (2021) LoRA: Low-Rank Adaptation of Large Language Models. arXiv: 2106.09685.
[22]
Dettmers, T., Pagnoni, A., Holtzman, A. and Zettlemoyer, L. (2023) QLoRA: Efficient Finetuning of Quantized LLMs. arXiv: 2305.14314.
[23]
Zhang, Q.R., Chen, M.S., Bukharin, A., et al. (2023) AdaLoRA: Adaptive Budget Allo-cation for Parameter-Efficient Fine-Tuning. arXiv: 2303.10512.
[24]
Sheng, Y., Cao, S.Y., Li, D.C., et al. (2024) S-LoRA: Serving Thousands of Concurrent LoRA Adapters. arXiv: 2311.03285.
[25]
Luo, M.R., Kuang, F.H., Wang, Y., Liu, Z.R. and He, T.X. (2025) Sc-LoRA: Balancing Efficient Fine-Tuning and Knowledge Preservation via Subspace-Constrained LoRA. arXiv: 2505.23724.
[26]
Moreo, A., Esuli, A. and Sebastiani, F. (2021) Word-Class Embeddings for Multiclass Text Classification. DataMiningandKnowledgeDiscovery, 35, 911-963. https://doi.org/10.1007/s10618-020-00735-3
Greene, D. and Cunningham, P. (2006) Practical Solutions to the Problem of Diagonal Dominance in Kernel Document Clustering. Proceedingsofthe 23rdInternationalConferenceonMachineLearning—ICML’06, Pittsburgh, 25-29 June 2006, 377-384. https://doi.org/10.1145/1143844.1143892
[29]
Lewis, D., Yang, Y.M., Russell-Rose, T. and Li, F. (2004) Rcv1: A New Bench-Mark Collection for Text Categorization Research. JournalofMachineLearningResearch, 5, 361-397.
[30]
Lang, K. (1995) NewsWeeder: Learning to Filter Netnews. In: Prieditis, A. and Russell, S., Eds., MachineLearningProceedings 1995, Elsevier, 331-339. https://doi.org/10.1016/b978-1-55860-377-6.50048-7
[31]
Sinha, A., Shen, Z., Song, Y., Ma, H., Eide, D., Hsu, B., etal. (2015) An Overview of Microsoft Academic Service (MAS) and Applications. Proceedingsofthe 24thInternationalConferenceonWorldWideWeb, Florence, 18-22 May 2015, 243-246. https://doi.org/10.1145/2740908.2742839
[32]
Farhangi, A., Sui, N., Hua, N., Bai, H., Huang, A. and Guo, Z. (2022) Protoformer: Embedding Prototypes for Transformers. In: Gama, J., Li, T., Yu, Y., Chen, E., Zheng, Y. and Teng, F., Eds., Advances in Knowledge Discovery and Data Mining, Springer, 447-458. https://doi.org/10.1007/978-3-031-05933-9_35
[33]
Sanderson, M. (2010) Test Collection Based Evaluation of Information Retrieval Systems. FoundationsandTrendsinInformationRetrieval, 4, 247-375. https://doi.org/10.1561/1500000009
[34]
Maas, A., Daly, R., Pham, P., Huang, D., Ng, A. and Potts, C. (2011) Learning Word Vectors for Sentiment Analysis. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Portland, June 2011, 142-150.
[35]
Sanh, V., Debut, L., Chaumond, J. and Wolf, T. (2020) Distilbert, a Distilled Version of Bert: Smaller, Faster, Cheaper and Lighter. arXiv: 1910.01108.
[36]
Hinton, G., Vinyals, O. and Dean, J. (2015) Distilling the Knowledge in a Neural Network. arXiv: 1503.02531.
[37]
Yang, Z.L., Dai, Z.H., Yang, Y.M., Carbonell, J., Salakhutdinov, R. and Le, Q.V. (2019) Xlnet: Generalized Autoregressive Pretraining for Language Understanding. Proceedings of the 33rd International Conference on Neural Information Processing Systems, Vancouver, 8-14 December 2019, 5753-5763.
[38]
Dai, Z.H., Yang, Z.L., Yang, Y.M., Carbonell, J., Le, Q.V. and Salakhutdinov, R. (2019) Transformer-XL: Attentive Language Models beyond a Fixed-Length Context. arXiv: 1901.02860.
[39]
Touvron, H., Minervini, P., Pappagari, V., etal. (2023) LLaMA: Open and Efficient Foundation Language Models. arXiv: 2302.13971.
[40]
Bi, X., Chen, D., Chen, G., et al. (2024) DeepSeek LLM: Scaling Open-Source Language Models with Longtermism. arXiv: 2401.02954.
[41]
Guo, D., Zhang, S.L., Liu, Z.Z., etal. (2025) DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv: 2501.12948.
[42]
Ouyang, L., Wu, J., Jiang, X., etal. (2022) Training Language Models to Follow Instructions with Human Feedback. arXiv: 2203.02155.
[43]
Wei, J.S., Wang, X.Z., Schuurmans, D., Bosma, M., Chi, E., Le, Q. and Zhou, D. (2022) Chain of Thought Prompting Elicits Reasoning in Large Language Models. arXiv: 2201.11903.
Liu, G. and Guo, J. (2019) Bidirectional LSTM with Attention Mechanism and Convolutional Layer for Text Classification. Neurocomputing, 337, 325-338. https://doi.org/10.1016/j.neucom.2019.01.078
[46]
Sparck Jones, K. (1972) A Statistical Interpretation of Term Specificity and Its Application in Retrieval. JournalofDocumentation, 28, 11-21. https://doi.org/10.1108/eb026526
[47]
Pearson, K. (1901) LIII. on Lines and Planes of Closest Fit to Systems of Points in Space. TheLondon, Edinburgh, andDublinPhilosophicalMagazineandJournalofScience, 2, 559-572. https://doi.org/10.1080/14786440109462720
[48]
Hotelling, H. (1933) Analysis of a Complex of Statistical Variables into Principal Components. JournalofEducationalPsychology, 24, 498-520. https://doi.org/10.1037/h0070888
[49]
Deerwester, S., Dumais, S.T., Furnas, G.W., Landauer, T.K. and Harshman, R. (1990) Indexing by Latent Semantic Analysis. JournaloftheAmericanSocietyforInformationScience, 41, 391-407. https://doi.org/10.1002/(sici)1097-4571(199009)41:6<391::aid-asi1>3.0.co;2-9
[50]
Salton, G. (1971) The SMART Retrieval System: Experiments in Automatic Document Processing. Prentice-Hall, Inc.
[51]
Chomsky, N. (1956) Three Models for the Description of Language. IEEETransactionsonInformationTheory, 2, 113-124. https://doi.org/10.1109/tit.1956.1056813
[52]
Chomsky, N. (1957) Syntactic Structures. De Gruyter Mouton.
[53]
Chomsky, N. (1965) Aspects of the Theory of Syntax. MIT Press.
[54]
Harris, Z.S. (1954) Distributional Structure. WORD, 10, 146-162. https://doi.org/10.1080/00437956.1954.11659520
[55]
Firth, J.R. (1957) A Synopsis of Linguistic Theory 1930-1955. In: StudiesinLinguisticAnalysis, Blackwell, 1-32. https://api.semanticscholar.org/CorpusID:208093066
[56]
Bahdanau, D., Cho, K. and Bengio, Y. (2016) Neural Machine Translation by Jointly Learning to Align and Translate. arXiv: 1409.0473.
[57]
Schuster, M. and Nakajima, K. (2012) Japanese and Korean Voice Search. 2012 IEEEInternationalConferenceonAcoustics, SpeechandSignalProcessing (ICASSP), Kyoto, 25-30 March 2012, 5149-5152. https://doi.org/10.1109/icassp.2012.6289079
[58]
Wu, Y.H., Schuster, M., Chen, Z.F., et al. (2016) Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation. arXiv: 1609.08144.
[59]
Sennrich, R., Haddow, B. and Birch, A. (2016) Neural Machine Translation of Rare Words with Subword Units. Proceedingsofthe 54thAnnualMeetingoftheAssociationforComputationalLinguistics (Volume 1: LongPapers), Berlin, August 2016, 1715-1725. https://doi.org/10.18653/v1/p16-1162
[60]
Raffel, C., Shazeer, N., Roberts, A., et al. (2019) Exploring the Limits of Transfer Learning with a Unified Text-To-Text Transformer. arXiv: 1910.10683.
[61]
Lan, Z.Z., Chen, M.D., Goodman, S., Gimpel, K., Sharma, P. and Soricut, R. (2019) Albert: A Lite Bert for Self-Supervised Learning of Language Representations. arXiv: 1909.11942.
[62]
Kudo, T. and Richardson, J. (2018) SentencePiece: A Simple and Language Independent Subword Tokenizer and Detokenizer for Neural Text Processing. Proceedingsofthe 2018 ConferenceonEmpiricalMethodsinNaturalLanguageProcessing: SystemDemonstrations, Brussels, November 2018, 66-71. https://doi.org/10.18653/v1/d18-2012
[63]
Clark, K., Luong, M.T., Le, Q.V. and Manning, C.D. (2020) ELECTRA: Pre-Training Text Encoders as Discriminators Rather than Generators. arXiv: 2003.10555.
[64]
Zhang, Q., Song, B., Lin, J., Liu, T., Li, C. and Lang, X. (2022) A Deep Learning-Based Entity-Relationship Extraction Method in the Field of Electric Power Public Opinion. In: AdvancesinTransdisciplinaryEngineering, IOS Press, 884-892. https://doi.org/10.3233/atde221110
[65]
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., etal. (2018) Deep Contextualized Word Representations. Proceedingsofthe 2018 ConferenceoftheNorthAmericanChapteroftheAssociationforComputationalLinguistics: HumanLanguageTechnologies, Volume 1 (LongPapers), New Orleans, June 2018, 2227-2237. https://doi.org/10.18653/v1/n18-1202
[66]
Qiu, X., Sun, T., Xu, Y., Shao, Y., Dai, N. and Huang, X. (2020) Pre-Trained Models for Natural Language Processing: A Survey. ScienceChinaTechnologicalSciences, 63, 1872-1897. https://doi.org/10.1007/s11431-020-1647-3
[67]
Minaee, S., Kalchbrenner, N., Cambria, E., Nikzad, N., Chenaghlu, M. and Gao, J. (2021) Deep Learning-Based Text Classification. ACMComputingSurveys, 54, 1-40. https://doi.org/10.1145/3439726
[68]
Trust, P. and Minghim, R. (2024) A Study on Text Classification in the Age of Large Language Models. MachineLearningandKnowledgeExtraction, 6, 2688-2721. https://doi.org/10.3390/make6040129
[69]
Sebastiani, F. (2002) Machine Learning in Automated Text Categorization. ACMComputingSurveys, 34, 1-47. https://doi.org/10.1145/505282.505283
[70]
Zhang, Y. (2021) Research on Text Classification Method Based on LSTM Neural Network Model. 2021 IEEEAsia-PacificConferenceonImageProcessing, ElectronicsandComputers (IPEC), Dalian, 14-16 April 2021, 1019-1022. https://doi.org/10.1109/ipec51340.2021.9421225
[71]
Brown, T., Mann, B., Ryder, N., et al. (2020) Language Models Are Few-Shot Learners. arXiv: 2005.14165.
[72]
Li, S., Zhao, Y.L., Varma, R., et al. (2020) PyTorch Distributed: Experiences on Accelerating Data Parallel Training. arXiv: 2006.15704.
[73]
Zebari, R.R., Mohsin Abdulazeez, A., Zeebaree, D.Q., van Asaad Zebari, D. and Saeed, J.N. (2020) A Comprehensive Review of Dimensionality Reduction Techniques for Feature Selection and Feature Extraction. JournalofAppliedScienceandTechnologyTrends, 1, 56-70.
[74]
Van Der Maaten, L., Postma, E. and Van den Herik, J. (2009) Dimensionality Reduction: A Comparative Review. Journal of Machine Learning Research, 10, 66-71.
[75]
Tifrea, A., Becigneul, G. and Ganea, O. (2018) Poincaré GloVe: Hyperbolic Word Embeddings. arXiv: 1810.06546.
[76]
Leimeister, M. and Wilson, B.J. (2019) Skip-Gram Word Embeddings in Hyperbolic Space. arXiv: 1905.09791.
[77]
Valentino, M., Carvalho, D. and Freitas, A. (2023) Multi-Relational Hyperbolic Word Embeddings from Natural Language Definitions. arXiv: 2305.07303.
[78]
Lee, J., Tang, R. and Lin, J. (2019) What Would Elsa Do? Freezing Layers during Transformer Fine-Tuning. arXiv: 1911.03090.
[79]
Ding, N., Qin, Y., Yang, G., Wei, F., Yang, Z., Su, Y., etal. (2023) Parameter-efficient Fine-Tuning of Large-Scale Pre-Trained Language Models. NatureMachineIntelligence, 5, 220-235. https://doi.org/10.1038/s42256-023-00626-4
[80]
Liang, M. and Niu, T. (2022) Research on Text Classification Techniques Based on Improved TF-IDF Algorithm and LSTM Inputs. ProcediaComputerScience, 208, 460-470. https://doi.org/10.1016/j.procs.2022.10.064
[81]
Zhou, H. (2022) Research of Text Classification Based on TF-IDF and CNN-LSTM. JournalofPhysics: ConferenceSeries, 2171, Article ID: 012021. https://doi.org/10.1088/1742-6596/2171/1/012021
[82]
Kamyab, M., Liu, G. and Adjeisah, M. (2021) Attention-Based CNN and BI-LSTM Model Based on TF-IDF and Glove Word Embedding for Sentiment Analysis. AppliedSciences, 11, Article 11255. https://doi.org/10.3390/app112311255
[83]
Appendix: Global Classifier Performance Data
[84]
Table A1. Global Macro-F1 Classifier Performance Summary (single-label datasets).
[85]
Table A2. Global Macro-F1 Classifier Performance Summary (multi-label datasets).
[86]
Table A3. Global Micro-F1 Classifier Performance Summary (single-label datasets).
[87]
Table A4. Global Micro-F1 Classifier Performance Summary (multi-label datasets).
