This paper presents a system for an interactive multimodal environment able (i) to train the listening comprehension in various populations of pupils, both Italian and immigrants, having different disabilities and (ii) to assess speech production and discrimination. The proposed system is the result of a research project focused on pupils with sensorial, neuropsychological, and relational impairments. The project involves innovative technological systems that the users (speech terabits psychologists and preprimary and primary schools teachers) could adopt for training and assessment of language and speech. Because the system is used in a real scenario (the Italian schools are often affected by poor funding for education and teachers without informatics skills), the guidelines adopted are low-cost technology; usability; customizable system; robustness. 1. Introduction Learning systems providing user interaction within physical spaces have been carried out over the years. However the high cost and the high complexity of the technologies used have always implied that their use by pupils in real context was limited to occasional visits or short periods of experimentation. Our aim is to provide an interactive multimodal environment (developed in C++) that can be integrated with the ordinary educational activities within the school. For this purpose, we use common technologies—such as webcams, microphones, and Microsoft Kinect sensors—in order (i) to provide tools that allow teachers to adapt or create autonomously the educational activities content to be carried out with the system and (ii) to implement a user interface for the management software that does not require specific computer skills. Our system implements the five different types of interaction stated by Moreno and Mayer [1]: (1) dialogue, (2) control, (3) manipulation, (4) search, and (5) navigation. Indeed, these five levels are very familiar during the everyday learning activity. Here are some examples: (1) a comparison/oral discussion in which the exchange of information is not unilateral, but the opportunities to the students to ask questions and express their opinions are given influencing the content of the lesson; (2) oral exposure in which the student has the ability to control the speed and to stop the explanation in order to benefit from the educational content at their own pace; (3) a scientific experiment that leaves the possibility for the student to test different parameters and see what happens; (4) the ability to independently seek information on a certain subject within a
References
[1]
R. Moreno and R. Mayer, “Interactive multimodal learning environments: special issue on interactive learning environments: contemporary issues and trends,” Educational Psychology Review, vol. 19, no. 3, pp. 309–326, 2007.
[2]
D. H. Jonassen, “Computers as mindtools for engaging critical thinking and representing knowledge,” in Proceedings of the Educational Technology Conference and Exhibition (EdTech '99), 1999.
[3]
L. Vygotsky, Thought and Language, MIT Press, Boston, Mass, USA, 1986.
[4]
S. Price and Y. Rogers, “Let's get physical: rhe learning benefits of interacting in digitally augmented physical spaces,” Computers and Education, vol. 43, no. 1-2, pp. 137–151, 2004.
[5]
J. Bruner, Toward a Theory of Instruction, Belknap Press of Harvard University Press, 1966.
[6]
A. Karmiloff-Smith, “Nativism versus neuroconstructivism: rethinking the study of developmental disorders,” Developmental Psychology, vol. 45, no. 1, pp. 56–63, 2009.
[7]
S. Bhupinder, K. Rupinder, D. Nidhi, and K. Ramandeep, “The process of feature extraction in automatic speech recognition system for computer machine interaction with humans: a review,” International Journal of Advanced Research in Computer Science and Software Engineering, vol. 2, no. 2, 7 pages, 2012.
[8]
S. Ahmadi and A. S. Spanias, “Cepstrum-based pitch detection using a new statistical V/UV classification algorithm,” IEEE Transactions on Speech and Audio Processing, vol. 7, no. 3, pp. 333–338, 1999.
[9]
M. Puckette and T. Apel, “Real-time audio analysis tools for pd and MSP,” in Proceedings of the International Computer Music Conference, pp. 109–112, San Francisco, Calif, USA, 1998.
[10]
C. Sinthanayothin, N. Wongwaen, and W. Bholsithi, “Skeleton tracking using kinect sensor & displaying in 3D virtual scene,” International Journal of Advancements in Computing Technology, vol. 4, no. 11, pp. 213–223, 2012.
[11]
K. Akita, “Image sequence analysis of real world human motion,” Pattern Recognition, vol. 17, no. 1, pp. 73–83, 1984.
[12]
J. O'Rourke and N. I. Badler, “Model-based image analysis of human motion using constraint propagation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 2, no. 6, pp. 522–536, 1980.
[13]
C. Bregler and J. Malik, “Tracking people with twists and exponential maps,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8–15, Santa Barbara, Calif, USA, June 1998.
[14]
T. Horprasert, I. Haritaoglu, C. Wren, D. Hardwood, L. Davis, and A. Pentland, “Real-time 3D motion capture,” in Proceedings of the 2nd Workshop on Perceptual User Interfaces, 1998.
[15]
H. D. Yang and S. W. Lee, “Reconstruction of 3D human body pose from stereo image sequences based on top-down learning,” Pattern Recognition, vol. 40, no. 11, pp. 3120–3131, 2007.
[16]
J. Rodgers, D. Anguelov, H. C. Pang, and D. Koller, “Object pose detection in range scan data,” in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR '06), pp. 2445–2452, June 2006.
[17]
V. Ganapathi, C. Plagemann, D. Koller, and S. Thrun, “Real time motion capture using a single time-of-flight camera,” in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR '10), pp. 755–762, San Francisco, Calif, USA, June 2010.
[18]
C. Plagemann, V. Ganapathi, D. Koller, and S. Thrun, “Real-time identification and localization of body parts from depth images,” in Proceedings of the IEEE International Conference on Robotics and Automation (ICRA '10), pp. 3108–3113, Anchorage, Alaska, USA, May 2010.
[19]
D. Hogg, “Model-based vision: a program to see a walking person,” Image and Vision Computing, vol. 1, no. 1, pp. 5–20, 1983.
[20]
L. Snidaro, G. L. Foresti, and L. Chittaro, “Tracking human motion from monocular sequences,” International Journal of Image and Graphics, vol. 8, no. 3, pp. 455–471, 2008.
[21]
C. R. Wren, A. Azarbayejani, T. Darrell, and A. P. Pentland, “P finder: real-time tracking of the human body,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 19, no. 7, pp. 780–785, 1997.
[22]
I. Haritaoglu, D. Harwood, and L. S. Davis, “W4: real-time surveillance of people and their activities,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 22, no. 8, pp. 809–830, 2000.
[23]
L. E. Baum, T. Petrie, G. Soules, and N. Weiss, “A maximization technique occurring in the statistical analysis of probabilistic functions of Markov Chains,” Annals of Mathematical Statistics, vol. 41, no. 1, pp. 164–171, 1970.
[24]
A. Quattoni, M. Collins, and T. Darrell, “Conditional random fields for object recognition,” in Proceedings of the NIPS, pp. 1097–1104, 2004.
[25]
L. Xia, C. Chen, and J. K. Aggarwal, “Human detection using depth information by Kinect,” in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW '11), pp. 15–22, Colorado Springs, Colo, USA, June 2011.
[26]
H. M. Zhu and C. M. Pun, “Movement tracking in real-time hand gesture recognition,” in Proceedings of the 9th IEEE/ACIS International Conference on Computer and Information Science (ICIS '10), pp. 240–245, Yamagata, Japan, August 2010.
[27]
M. R. Malgireddy, J. J. Corso, S. Setlur, V. Govindaraju, and D. Mandalapu, “A framework for hand gesture recognition and spotting using sub-gesture modeling,” in Proceedings of the 20th International Conference on Pattern Recognition (ICPR '10), pp. 3780–3783, Istanbul, Turkey, August 2010.
[28]
Y. Gu and C. Yuan, “Human action recognition for home sensor network,” in Proceedings of the International Conference on Information Technology and Software Engineering, vol. 212 of Lecture Notes in Electrical Engineering, pp. 643–656, 2013.
[29]
G. Bradski and A. Kaehler, Learning OpenCV: Computer Vision with the OpenCV Library, O'Reilly Media, 2008.
[30]
T. Lei, Y. Wang, Y. Fan, and J. Zhao, “Vector morphological operators in HSV color space,” Science China Information Sciences, vol. 56, no. 1, pp. 1–12, 2013.
[31]
N. Falco, M. D. Mura, F. Bovolo, J. A. Benediktsson, and L. Bruzzone, “Change detection in VHR images based on morphological attribute profiles,” IEEE Geoscience and Remote Sensing Letters, vol. 10, no. 3, pp. 636–640, 2013.
[32]
S. Jabri, Z. Duric, H. Wechsler, and A. Rosenfeld, “Detection and location of people in video images using adaptive fusion of color and edge information,” in Proceedings of the 15th IEEE International Conference on Pattern Recognition, vol. 4, pp. 627–630, 2000.
[33]
R. Ramanathan, Combining egocentric and exocentric views: enhancing the virtual worlds interface [Master's Project], Department of Urban and Regional Planning, University of Illinois at Urbana-Champaign, 1999.
[34]
M. Obrist, F. F？rster, D. Wurhofer, M. Tscheligi, and J. Hofst？tter, “Evaluating first experiences with an educational computer game: a multi-method approach,” Interaction Design and Architecture, vol. 11-12, pp. 26–36, 2011.
[35]
M. Steward, “Learning through research: an introduction to the main theories of learning,” JMU Learning and Teaching Press, vol. 4, no. 1, pp. 6–14, 2004.
[36]
M. Roch and M. C. Levorato, “Simple view of reading in Down's syndrome: the role of listening comprehension and reading skills,” International Journal of Language and Communication Disorders, vol. 44, no. 2, pp. 206–223, 2009.
[37]
C. Fadel and C. Lemke, “Multimodal learning through media: what the research says,” Tech. Rep., Cisco Systems, 2008.
[38]
S. Zanolla, S. Canazza, A. Rodà, and G. L. Foresti, “Learning by means of an interactive multimodal environment,” in Proceedings of the SETECEC Conference, Alaipo and Ainci, Eds., pp. 167–176, Venice, Italy, March 2012.
[39]
S. Zanolla, S. Canazza, A. Rodà, A. Camurri, and G. Volpe, “Entertaining listening by means of the Stanza Logo-Motoria: an interactive multimodal environment,” Entertainment Computing, 2013.
