Determining the causal effect of special education is a critical topic when making educational policy that focuses on student achievement. However, current special education research is facing challenges from persistent selection bias and complex confounding. Bayesian Additive Regression Trees (BART) is employed in this study to provide a flexible estimation of the academic performance. Targeted Maximum Likelihood Estimation (TMLE) is also integrated into the BART model, supporting doubly robust estimation of the special education effect. This study extracted survey data from the Early Childhood Longitudinal Study, Kindergarten Class (ECLS-K), to estimate the causal impact of special education status on students’ combined mathematics and reading achievement scores. The analysis results of the BART-TMLE model show that children receiving special education services demonstrated approximately 9 points lower scores on average for combined math and reading scores, even adjusting for a considerable number of covariates, compared to their peers who did not receive these services. The estimated negative treatment effect persists after controlling for observed covariates that are closely correlated to the combined test score. The negative effect likely reflects unobserved factors, such as the underlying severity of learning disabilities, parent involvement and other potential traits, which are actual factors that determine the placement of special education status, rather than indicating the ineffectiveness of special education service. The achievement gap in academic performance reflects the current observable status of special education. The estimated effect could be improved by future research incorporating educational domain knowledge, allowing the model to be constructed more accurately.
References
[1]
Blackorby, J., Wagner, M., Cameto, R., Davies, E., Levine, P., Newman, L., Marder, C., Cardoso, D. and Garza, N. (2005) Engagement, Academics, Social Adjustment, and Independence: The Achievements of Elementary and Middle School Students with Disabilities. National Center for Special Education Research. https://www.seels.net/designdocs/engagement/All_SEELS_outcomes_10-04-05.pdf
[2]
Wagner, M., Newman, L., Cameto, R., Garza, N. and Levine, P. (2005) After High School: A First Look at the Postschool Experiences of Youth with Disabilities. National Center for Special Education Research. https://files.eric.ed.gov/fulltext/ED494935.pdf
[3]
Hanushek, E.A., Kain, J.F. and Rivkin, S.G. (2002) Inferring Program Effects for Special Populations: Does Special Education Raise Achievement for Students with Disabilities? Review of Economics and Statistics, 84, 584-599. https://doi.org/10.1162/003465302760556431
[4]
Shifrer, D., Muller, C. and Callahan, R. (2011) Disproportionality and Learning Disabilities: Parsing Apart Race, Socioeconomic Status, and Language. Journal of Learning Disabilities, 44, 246-257. https://doi.org/10.1177/0022219410374236
[5]
Feng, Y. (2024) Assessing the Effectiveness of Special Education Services on Fifth Grade Math Scores: Using Traditional and Machine Learning Methods with ECLS-K Data. Applied and Computational Engineering, 45, 7-16. https://doi.org/10.54254/2755-2721/45/20241019
[6]
Setoguchi, S., Schneeweiss, S., Brookhart, M.A., Glynn, R.J. and Cook, E.F. (2008) Evaluating Uses of Data Mining Techniques in Propensity Score Estimation: A Simulation Study. Pharmacoepidemiology and Drug Safety, 17, 546-555. https://doi.org/10.1002/pds.1555
[7]
Rosenbaum, P.R. and Rubin, D.B. (1983) The Central Role of the Propensity Score in Observational Studies for Causal Effects. Biometrika, 70, 41-55. https://doi.org/10.1093/biomet/70.1.41
[8]
Stuart, E.A. (2010) Matching Methods for Causal Inference: A Review and a Look Forward. Statistical Science, 25, 1-21. https://doi.org/10.1214/09-sts313
[9]
Austin, P.C. (2011) An Introduction to Propensity Score Methods for Reducing the Effects of Confounding in Observational Studies. Multivariate Behavioral Research, 46, 399-424. https://doi.org/10.1080/00273171.2011.568786
[10]
Hill, J.L. (2011) Bayesian Nonparametric Modeling for Causal Inference. Journal of Computational and Graphical Statistics, 20, 217-240. https://doi.org/10.1198/jcgs.2010.08162
[11]
Tourangeau, K., Nord, C., Lê, T., Pollack, J.M. and Atkins-Burnett, S. (2006) Early Childhood Longitudinal Study, Kindergarten Class of 1998-99 (ECLS-K), Com-Bined User’s Manual for the ECLS-K Fifth-Grade Data Files and Electronic Code-Books (NCES Publication No. 2006-032). U.S. Department of Education, National Center for Education Statistics. https://nces.ed.gov/pubs2006/2006032_2.pdf
[12]
Imai, K. and Ratkovic, M. (2014) Covariate Balancing Propensity Score. Journal of the Royal Statistical Society Series B: Statistical Methodology, 76, 243-263. https://doi.org/10.1111/rssb.12027
[13]
Morgan, P.L., Frisco, M.L., Farkas, G. and Hibel, J. (2010) A Propensity Score Matching Analysis of the Effects of Special Education Services. The Journal of Special Education, 43, 236-254. https://doi.org/10.1177/0022466908323007
[14]
Imberman, S.A. (2014) The Effect of Special Education Services on Student Achievement. Journal of Public Economics, 118, 98-110. https://doi.org/10.2139/ssrn.1031693
[15]
Sullivan, A.L. and Field, S. (2013) Does Special Education Improve Preschoolers’ Academic Skills? CASTL Research Brief. Curry School of Education, University of Virginia. https://files.eric.ed.gov/fulltext/ED544029.pdf
[16]
Chipman, H.A., George, E.I. and McCulloch, R.E. (2010) BART: Bayesian Additive Regression Trees. The Annals of Applied Statistics, 4, 266-298. https://doi.org/10.1214/09-aoas285
[17]
Tourangeau, K., Nord, C., Lê, T., Sorongon, A.G. and Najarian, M. (2009) Early Childhood Longitudinal Study, Kindergarten Class of 1998-99 (ECLS-K), Com-Bined User’s Manual for the ECLS-K Eighth-Grade and K-8 Full Sample Data Files and Electronic Codebooks (NCES Publication No. 2009-004). National Center for Education Statistics, Institute of Education Sciences, U.S. Department of Education. https://nces.ed.gov/ecls/data/eclsk_k8_manual_part1.pdf
[18]
Kapelner, A. and Bleich, J. (2016) BartMachine: Machine Learning with Bayesian Additive Regression Trees. Journal of Statistical Software, 70, 1-40. https://doi.org/10.18637/jss.v070.i04
[19]
Dorie, V., Hill, J., Shalit, U., Scott, M. and Cervone, D. (2019) Automated versus Do-It-Yourself Methods for Causal Inference: Lessons Learned from a Data Analysis Competition. Statistical Science, 34, 43-68. https://doi.org/10.1214/18-sts667
[20]
Green, D.P. and Kern, H.L. (2012) Modeling Heterogeneous Treatment Effects in Survey Experiments with Bayesian Additive Regression Trees. Public Opinion Quarterly, 76, 491-511. https://doi.org/10.1093/poq/nfs036
[21]
Friedman, J.H. (2001) Greedy Function Approximation: A Gradient Boosting Machine. The Annals of Statistics, 29, 1189-1232. https://doi.org/10.1214/aos/1013203451
[22]
Breiman, L. (2001) Random Forests. Machine Learning, 45, 5-32. https://doi.org/10.1023/a:1010933404324
[23]
Chipman, H.A., George, E.I. and McCulloch, R.E. (1998) Bayesian CART Model Search. Journal of the American Statistical Association, 93, 935-948. https://doi.org/10.1080/01621459.1998.10473750
[24]
Tan, Y.V. and Roy, J. (2019) Bayesian Additive Regression Trees and the General BART Model. Statistics in Medicine, 38, 5048-5069. https://doi.org/10.1002/sim.8347
[25]
George, E.I. and McCulloch, R.E. (1993) Variable Selection via Gibbs Sampling. Journal of the American Statistical Association, 88, 881-889. https://doi.org/10.1080/01621459.1993.10476353
[26]
Rubin, D.B. (1974) Estimating Causal Effects of Treatments in Randomized and Nonrandomized Studies. Journal of Educational Psychology, 66, 688-701. https://doi.org/10.1037/h0037350
[27]
Hahn, P.R., Murray, J.S. and Carvalho, C.M. (2017) Bayesian Regression Tree Models for Causal Inference: Regularization, Confounding, and Heterogeneous Effects. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.3048177
[28]
Pearl, J. (2009) Causality. 2nd Edition, Cambridge University Press. https://doi.org/10.1017/cbo9780511803161
[29]
Linero, A.R. (2018) Bayesian Regression Trees for High-Dimensional Prediction and Variable Selection. Journal of the American Statistical Association, 113, 626-636. https://doi.org/10.1080/01621459.2016.1264957
[30]
Ročková, V. and van der Pas, S. (2020) Posterior Concentration for Bayesian Regression Trees and Forests. The Annals of Statistics, 48, 2108-2131. https://doi.org/10.1214/19-aos1879
[31]
Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., et al. (2018) Double/Debiased Machine Learning for Treatment and Structural Parameters. The Econometrics Journal, 21, C1-C68. https://doi.org/10.1111/ectj.12097
[32]
Tibshirani, R. (1996) Regression Shrinkage and Selection via the Lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology, 58, 267-288. https://doi.org/10.1111/j.2517-6161.1996.tb02080.x
[33]
Zou, H. and Hastie, T. (2005) Regularization and Variable Selection via the Elastic Net. Journal of the Royal Statistical Society Series B: Statistical Methodology, 67, 301-320. https://doi.org/10.1111/j.1467-9868.2005.00503.x
[34]
Belloni, A., Chernozhukov, V. and Hansen, C. (2014) Inference on Treatment Effects after Selection among High-Dimensional Controls. The Review of Economic Studies, 81, 608-650. https://doi.org/10.1093/restud/rdt044
[35]
Wasserman, L. and Roeder, K. (2009) High-Dimensional Variable Selection. The Annals of Statistics, 37, 2178-2201. https://doi.org/10.1214/08-aos646
[36]
Fithian, W., Sun, D. and Taylor, J. (2014) Optimal Inference after Model Selection. arXiv:1410.2597. https://doi.org/10.48550/arXiv.1410.2597
[37]
Rinaldo, A., Wasserman, L. and G’Sell, M. (2019) Bootstrapping and Sample Splitting for High-Dimensional, Assumption-Lean Inference. The Annals of Statistics, 47, 3438-3469. https://doi.org/10.1214/18-aos1784
[38]
Berk, R., Brown, L., Buja, A., Zhang, K. and Zhao, L. (2013) Valid Post-Selection Inference. The Annals of Statistics, 41, 802-837. https://doi.org/10.1214/12-aos1077
[39]
Gruber, S. and van der Laan, M.J. (2012) tmle: An R Package for Targeted Maximum Likelihood Estimation. Journal of Statistical Software, 51, 1-35. https://doi.org/10.18637/jss.v051.i13
[40]
Van der Laan, M.J. and Rose, S. (2011) Targeted Learning: Causal Inference for Observational and Experimental Data (pp. 89-135, 165-210). Springer. https://doi.org/10.1007/978-1-4419-9782-1
[41]
Porter, K.E., Gruber, S., van der Laan, M.J. and Sekhon, J.S. (2011) The Relative Performance of Targeted Maximum Likelihood Estimators. The International Journal of Biostatistics, 7, 1-34. https://doi.org/10.2202/1557-4679.1308
[42]
Schuler, M.S. and Rose, S. (2017) Targeted Maximum Likelihood Estimation for Causal Inference in Observational Studies. American Journal of Epidemiology, 185, 65-73. https://doi.org/10.1093/aje/kww165
[43]
Zheng, W. and van der Laan, M.J. (2011) Cross-Validated Targeted Minimum-Loss-Based Estimation. In: van der Laan, M.J. and Rose, S., Eds., Targeted Learning, Springer, 459-474. https://doi.org/10.1007/978-1-4419-9782-1_27
[44]
Imbens, G.W. and Rubin, D.B. (2015) Causal Inference for Statistics, Social, and Biomedical Sciences. Cambridge University Press. https://doi.org/10.1017/cbo9781139025751
[45]
Hahn, J. (1998) On the Role of the Propensity Score in Efficient Semiparametric Estimation of Average Treatment Effects. Econometrica, 66, 315.-331 https://doi.org/10.2307/2998560
[46]
Hirano, K., Imbens, G.W. and Ridder, G. (2003) Efficient Estimation of Average Treatment Effects Using the Estimated Propensity Score. Econometrica, 71, 1161-1189. https://doi.org/10.1111/1468-0262.00442
[47]
Belloni, A., Chernozhukov, V., Chetverikov, D., Hansen, C. and Kato, K. (2017) High-Dimensional Econometrics and Regularized GMM. arXiv:1806.01888. https://doi.org/10.48550/arXiv.1806.01888
[48]
Athey, S. and Imbens, G.W. (2019) Machine Learning Methods That Economists Should Know about. Annual Review of Economics, 11, 685-725. https://doi.org/10.1146/annurev-economics-080217-053433
[49]
Wainwright, M.J. (2019) High-Dimensional Statistics. Cambridge University Press. https://doi.org/10.1017/9781108627771
[50]
Kurz, A., Elliott, S.N., Wehby, J.H. and Smithson, J.L. (2010) Alignment of the Intended, Planned, and Enacted Curriculum in General and Special Education and Its Relation to Student Achievement. The Journal of Special Education, 44, 131-145. https://doi.org/10.1177/0022466909341196
[51]
Burke, M.A. and Sass, T.R. (2013) Classroom Peer Effects and Student Achievement. Journal of Labor Economics, 31, 51-82. https://doi.org/10.1086/666653
