In the current programming
education, in order to assess the true ability of learners, instructors still have no choice but to monitor their answering
process, standing by them. However, this is impractical for freshman training
in educational institutions and newcomer training in companies. Because of the
practicality, a large number of learners are assessed at once using written
tests or Web tests. They usually inquire of learners whether they know
algorithms and grammar. If not, they assess only the behavior of source codes
they submit, at best. Under the training based on such assessment, in reality,
not a few learners fail to acquire the skill of writing source codes. It
implies that the attainment of programming skills cannot be assessed only by
tests on knowledge and submitted source codes. This paper proposes a method for analyzing learners’ understanding that
focuses on their thinking process of programming. The proposed method
focuses on a code puzzle in which learners arrange fragments of a program code
to satisfy given requirements. It aims to estimate the learner’s perspective on
how fragments are built up to achieve the requirements. Learners with low
understanding are assumed to be different from those with high in terms of the
consistency of arranging ways to compose code fragments for specific blocks in
source codes. For the discrimination, the method builds a model using a hidden
Markov model. The internal state obtained from this model would help
instructors grasp the learner’s understanding level. The results of an
experiment present that the hidden Markov model produces meaningful values,
which enable instructors to interpret the understanding of individual learners.
References
[1]
Asai, S. (2019). Identification of Factors Affecting Cognitive Load in Programming Learning with Decision Tree. Journal of Computers, 14, 624-633. https://doi.org/10.17706/jcp.14.11.624-633
[2]
Baum, L. E., & Petrie, T. (1966). Statistical Inference for Probabilistic Functions of Finite State Markov Chains. The Annals of Mathematical Statistics, 37, 1554-1563. https://doi.org/10.1214/aoms/1177699147
[3]
Blikstein, P., Worsley, M., Piech, C., Sahami, M., Cooper, S., & Koller, D. (2014). Programming Pluralism: Using Learning Analytics to Detect Patterns in the Learning of Computer Programming. Journal of the Learning Sciences, 23, 561-599. https://doi.org/10.1080/10508406.2014.954750
[4]
Crow, T., Luxton-Reilly, A., & Wuensche, B. (2018). Intelligent Tutoring Systems for Programming Education. In R. Mason, & Simon (Eds.), Proceedings of the 20th Australasian Computing Education Conference (pp. 53-62). Association for Computing Machinery. https://doi.org/10.1145/3160489.3160492
[5]
Futamura , Y., Kawai, T., Horikoshi, H., & Tsutsumi, M. (1981). Development of Computer Programs by Problem Analysis Diagram (PAD). In S. Jeffrey, & L. G. Stucki (Eds.), Proceedings of the 5th International Conference on Software Engineering (pp. 325-332). Association for Computing Machinery.
[6]
Guo, P. J. (2015). Codeopticon: Real-Time, One-To-Many Human Tutoring for Computer Programming. In C. Latulipe, B. Hartmann, & T. Grossman (Eds.), Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology (pp. 599-608). Association for Computing Machinery. https://doi.org/10.1145/2807442.2807469
[7]
Hellas, A., Ajanovski, V. V., Knutas, A., Ihantola, P., Gutica, M., Leinonen, J., Liao, S. N., Petersen, A., Hynninen, T., & Messom, C. (2019, May 20). Predicting Academic Performance: A Systematic Literature Review. In G. Rossling, & B. Scharlau (Eds.), Proceedings Companion of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education (pp. 175-199). Association for Computing Machinery. https://doi.org/10.1145/3293881.3295783 https://research.monash.edu/en/publications/predicting-academic-performance-a-systematic-literature-review
[8]
Ihantola, P., Sorva, J., & Vihavainen, A. (2014). Automatically Detectable Indicators of Programming Assignment Difficulty. In B. Rutherfoord, L. Li, S. Van de Ven, & A. Settle (Eds.), Terry Steinbach, Proceedings of the 15th Annual Conference on Information Technology Education (pp. 33-38). Association for Computing Machinery. https://doi.org/10.1145/2656450.2656476
[9]
Ito, H., Shimakawa, H., & Harada, F. (2021). Advanced Comprehension Analysis Using Code Puzzle. In E. Ziemba, & W. Chmielarz (Eds.), Information Technology for Management: Towards Business Excellence (pp. 45-64). Springer. https://doi.org/10.1007/978-3-030-71846-6_3
[10]
Jadud, M. C. (2006). Methods and Tools for Exploring Novice Compilation Behaviour. In R. J. Anderson, S. Fincher, & M. Guzdial (Eds.), Proceedings of the 2006 2nd International Workshop on Computing Education Research (pp. 73-84). Association for Computing Machinery. https://doi.org/10.1145/1151588.1151600
[11]
Johnson, W. L., & Gladwin, L. A. (1987). Intention-Based Diagnosis of Novice Programming Errors. IEEE Expert, 2, 94. https://doi.org/10.1109/MEX.1987.4307101
[12]
Kaplan, A. (2021). Higher Education at the Crossroads of Disruption: The University of the 21st Century. Emerald Publishing. https://doi.org/10.1108/9781800715011
[13]
Kato, T., Kambayashi, Y., Terawaki, Y., & Kodama, Y. (2018). Analysis of Students’ Behaviors in Programming Exercises Using Deep Learning. In V. Uskov, R. Howlett, & L. Jain (Eds.), Smart Education and e-Learning 2017 (pp. 38-47). Springer. https://doi.org/10.1007/978-3-319-59451-4_4
[14]
Lane, H. C., & VanLehn, K. (2005). Intention-Based Scoring: An Approach to Measuring Success at Solving the Composition Problem. ACM SIGCSE Bulletin, 37, 373-377. https://doi.org/10.1145/1047344.1047471
[15]
Luo, J., & Wang, T. (2020). Analyzing Students’ Behavior in Blended Learning Environment for Programming Education. In Proceedings of the 2020 2nd World Symposium on Software Engineering (pp. 179-185). Association for Computing Machinery. https://doi.org/10.1145/3425329.3425346
[16]
Marion, B., Impagliazzo, J., St. Clair, C., Soroka, B., & Whitfield, D. (2007). Assessing Computer Science Programs. In I. Russell, S. M. Haller, J. D. Dougherty, & S. H. Rodger (Eds.), Proceedings of the 38th SIGCSE Technical Symposium on Computer Science Education (pp. 131-132). Association for Computing Machinery. https://doi.org/10.1145/1227310.1227358
[17]
McCracken, M., Almstrum, V., Diaz, D., Guzdial, M., Hagan, D., Kolikant, Y. B.-D., Laxer, C., Thomas, L., Utting, I., & Wilusz, T. (2001). A Multi-National, Multi-Institutional Study of Assessment of Programming Skills of First-Year CS Students. ACM SIGCSE Bulletin, 33, 125-180. https://doi.org/10.1145/572139.572181
[18]
Mysore, A., & Guo, P. J. (2018). Porta: Profiling Software Tutorials Using Operating-System-Wide Activity Tracing. In P. Baudisch, A. Schmidt, & A. Wilson (Eds.), Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology (pp. 201-212). Association for Computing Machinery. https://doi.org/10.1145/3242587.3242633
[19]
Nesbit, J., Liu, L., Liu, Q., & Adesope, O. (2015). Work in Progress: Intelligent Tutoring Systems in Computer Science and Software Engineering Education. In 2015 ASEE Annual Conference and Exposition Proceedings (pp. 26.1754.1-26.1754.12). American Society for Engineering Education. https://doi.org/10.18260/p.25090
[20]
Nikula, U., Gotel, O., & Kasurinen, J. (2011, October 31). A Motivation Guided Holistic Rehabilitation of the First Programming Course. ACM Transactions on Computing Education, 11, Article No. 24. https://doi.org/10.1145/2048931.2048935 https://eric.ed.gov/?id=EJ958645
[21]
Parsons, D., & Haden, P. (2006). Parson’s Programming Puzzles: A Fun and Effective Learning Tool for First Programming Courses. In D. Tolhurst, & S. Mann (Eds.), Proceedings of the 8th Australasian Conference on Computing Education (Vol. 52, pp. 157-163). Australian Computer Society, Inc.
[22]
Scaradozzi, D., Cesaretti, L., Screpanti, L., & Mangina, E. (2020). Identification of the Students Learning Process during Education Robotics Activities. Frontiers in Robotics and AI, 7, Article No. 21. https://doi.org/10.3389/frobt.2020.00021
[23]
Schnotz, W., & Kürschner, C. (2007). A Reconsideration of Cognitive Load Theory. Educational Psychology Review, 19, 469-508. https://doi.org/10.1007/s10648-007-9053-4
[24]
Tabanao, E. S., Rodrigo, M. M., & Jadud, M. C. (2011). Predicting At-Risk Novice Java Programmers through the Analysis of Online Protocols. In K. Sanders, M. E. Caspersen, & A. Clear (Eds.), Proceedings of the Seventh International Workshop on Computing Education Research (pp. 85-92). Association for Computing Machinery. https://doi.org/10.1145/2016911.2016930
[25]
Villamor, M. M. (2020). A Review on Process-Oriented Approaches for Analyzing Novice Solutions to Programming Problems. Research and Practice in Technology Enhanced Learning, 15, Article No. 8. https://doi.org/10.1186/s41039-020-00130-y
[26]
Watson, C., Li, F. W. B., & Godwin, J. L. (2013). Predicting Performance in an Introductory Programming Course by Logging and Analyzing Student Programming Behavior. In 2013 IEEE 13th International Conference on Advanced Learning Technologies (pp. 319-323). Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/ICALT.2013.99
[27]
Watson, C., Li, F. W. B., & Godwin, J. L. (2014). No Tests Required: Comparing Traditional and Dynamic Predictors of Programming Success. In J. D. Dougherty, K. Nagel, A. Decker, & K. Eiselt (Eds.), Proceedings of the 45th ACM Technical Symposium on Computer Science Education (pp. 469-474). Association for Computing Machinery. https://doi.org/10.1145/2538862.2538930
