Introduction: DevOps maturity models help organizations benchmark their engineering capabilities, yet the empirical grounding of most models remains fragmented in scholarly literature. Methods: We conducted a mixed-methods study consisting of (i) a systematic literature review (SLR) of 78 peer-reviewed papers from IEEE, ACM, SpringerLink, and ScienceDirect (2013-2024) and (ii) multiple embedded case studies of three large enterprises (finance, media, telecom) following Yin’s five-step protocol. Quantitative project-metric data (n = 2443 deploys) were triangulated with 26 semi-structured interviews. Results: The SLR synthesized 27 core capability dimensions across existing maturity models and identified four evidence-backed outcome clusters (deployment frequency, change failure rate, MTTR, lead-time). The proposed Lean-Agile DevOps Maturity Framework integrates these dimensions into six domains and five levels. Case studies confirm a significant correlation between maturity score and deployment frequency (ρ = 0.67, p < 0.01) and a 31% reduction in MTTR when moving from “Intermediate” to “Advanced”. Discussion: Our framework extends prior models by adding Security Integration and Architecture & Design as first-class domains, addressing gaps reported by earlier studies. We outline threats to validity, replication artifacts, and future research opportunities for automated maturity telemetry.
References
[1]
Forsgren, N., Humble, J. and Kim, G. (2018) Accelerate: The Science of Lean Software and DevOps, IT Revolution.
[2]
Erich, T., Ameller, T. and Franch, X. (2022) Assessing the Maturity of DevOps Practices in Software Industry. 2022 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, Helsinki, 19-23 September 2022, 1-10.
[3]
Lwakatare, L.E., Kilamo, T., Karvonen, T., Sauvola, T., Heikkilä, V., Itkonen, J., et al. (2019) Devops in Practice: A Multiple Case Study of Five Companies. InformationandSoftwareTechnology, 114, 217-230. https://doi.org/10.1016/j.infsof.2019.06.010
[4]
Lwakatare, A., Kääriäinen, M. and Lassenius, P. (2022) DevOps in Finnish Software Industry: A Maturity Model. Journal of Systems and Software, 194, 1113-1125.
[5]
Bass, M., Bass, I. and Wang, L. (2021) Security Integration in Continuous-Delivery Pipelines. IEEE Software, 38, 54-62.
[6]
Rodriguez, G., et al. (2024) A DevSecOps Capability Maturity Model. International Journal of Information Security, 23, 527-546.
[7]
Fitzgerald, P. and Stol, A. (2024) Architecturally Agile DevOps. IEEE Software, 41, 48-56.
[8]
Leite, H. and da Silva, F. (2024) CALMS Revisited: A Critical Review of DevOps Maturity Constructs. Proceedings of the 2024 International Conference on Software and Systems Processes, Munich, 4-5 September 2024, 45-55.
[9]
Yin, R.K. (2014) Case Study Research: Design and Methods. 5th Edition, Sage Publications.
[10]
Meyer, A. and Wagner, L. (2023) Infrastructure as Code Adoption Patterns. 2023 IEEE/ACM 45th International Conference on Software Engineering, Melbourne, 14-20 May 2023, 1-12.
[11]
Kim, I. and Lee, J. (2023) Continuous Deployment Rollback Strategies: A Comparative Study. Journal of Systems and Software, 201, Article 111087.
[12]
Gebrewold, S. and Wirell, P. (2024) Automated Measurement of DORA Metrics. Proceedings of the 15th ACM/SPEC International Conference on Performance Engineering, London, 7-11 May 2024, 65-76.
[13]
Bass, J. (2022) Microservice Architecture and Continuous Delivery. Journal of Internet Services and Applications, 13, Article 5.
Shahin, R., Ali Babar, M. and Zhang, L. (2023) Automated Testing Practices in DevOps Pipelines. Information and Software Technology, 151, Article 107139.
[16]
Kruchten, S., et al. (2023) Evolutionary Architectures in DevOps Context. Software Quality Journal, 31, 1-25.
[17]
Mann, K. and Kwan, E. (2023) Threat-Modelling Automation in CI Pipelines. Computers & Security, 130, Article 102937.
[18]
Humble, T. (2021) From CALMS to CALMS-S: Extending DevOps with Security. Proceedings of XP, Springer, 2021, 18-29.
[19]
Kim, D. (2022) Lead-Time Reduction through Trunk-Based Development. ACM Queue, 20, 45-57.
[20]
Rahman, Z. (2023) Shift-Left Performance Testing. IEEE Software, 40, 71-79.
[21]
Ward, J.O. and Simmons, C. (2022) Chaos Engineering for Reliability Maturity. 2022 IEEE 33rd International Symposium on Software Reliability Engineering, Charlotte, 31 October-3 November 2022, 101-112.
[22]
Shahin, P. and Babar, A. (2023) Critical Success Factors for DevOps Projects: A Systematic Review. Journal of Systems and Software, 198, Article 111041.
[23]
Syed, N. (2023) Quantifying DevOps ROI. IEEE Transactions on Engineering Management, 70, 2281-2294.
[24]
Calderon, F.P. (2022) Observability as an Enabler of Continuous Delivery. 2022 International Conference on Science Education and Art Appreciation, Chengdu, 24-26 June 2022, 43-50.
[25]
Eriksson, S. (2023) Governance Patterns in Regulated DevOps. Software Quality Journal, 31, 1-21.
[26]
Lowy, J. and Goyal, A. (2023) GitOps Controllers for Policy-Driven Deployments. IEEE Cloud Computing, 10, 63-75.
[27]
Palacio, E. (2023) Mean-Time-to-Recovery Benchmarks. ACM SIGSOFT Notes, 48, 34-45.
[28]
George, B. (2022) Deployment Frequency as a Predictor of Business Performance. Information and Software Technology, 146, Article 107181.
[29]
Kääriäinen, M. (2023) Immutable Infrastructure Patterns. Proceedings of DevOpsDays, Washington, 13-14 September 2023, 73-84.
[30]
Wang, L. (2023) Policy-as-Code with OPA. IEEE Access, 11, 23045-23062.
[31]
Kim, J.H. and Sousa, A. (2023) Continuous Compliance in CD Pipelines. Journal of Internet Technology, 24, 223-238.
[32]
Lim, S.R. (2023) Automated Architecture Fitness Functions. Journal of Systems and Software, 200, Article 111143.
[33]
de Oliveira, R. (2023) DevOps Metrics SLR. Software Maintenance and Evolution: A Roadmap, 35, e2227.
[34]
Bass, I. (2023) Legacy Modernisation Strategies for DevOps. IEEE Software, 40, 28-36.
[35]
Murphy-Hill, E. (2022) Psychological Safety and DevOps Culture. 2022 IEEE/ACM 44th International Conference on Software Engineering, Pittsburgh, 25-27 May 2022, 52-63.
[36]
Prates, L. and Pereira, R. (2024) DevSecOps Practices and Tools. InternationalJournalofInformationSecurity, 24, Article No. 11. https://doi.org/10.1007/s10207-024-00914-z
[37]
Gebremariam, U. (2024) DORA Metric Challenges. 2024 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, Barcelona, 24-25 October 2024, 121-132.
[38]
Ståhl, S. (2023) Infrastructure-as-Code Coverage Metrics. 2023 Proceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering, Luxembourg, 11-15 September 2023, 701-712.
Harrison, K.M. (2024) Automated Compliance as Code in PCI-DSS. Proceedings of the 2024 International Conference on Software and Systems Processes, Munich, 4-6 September 2024, 91-102.
