The Construction Cycle and Quality Controls for Training Transfer Evaluations (CCQCTTE) is an assessment method that results of collaboration between University of Quebec in Montreal (UQAM) Canada and the University of Teacher Education of State of Vaud (HEP Vaud) Switzerland. The main objective of CCQCTTE project is to design and field test a method for building high quality training transfer assessments (level 3 of Kirkpatrick’s model). In relation with this goal, we defined five sub-objectives: easy way of use; implementation of best practices; cyclical quality approach; taking into account of transfer factors; diagnostic feedbacks. CCQCTTE consists of eight steps: 1) analysis of the training objectives and of the factors influencing the transfer; 2) assessment design; 3) items writing; 4) information about the assessment; 5) collection of transfer data; 6) processing of results; 7) feedbacks and 8) macro-regulation. The end product of the first step is a table of specifications and a list of transfer factors. Once the evaluation plan is defined in step two, we can move on to step three of item development. During the fourth step, all the stakeholders are informed. During step five data collection takes place in the training environment and in the workplace. Data are processed to extract information during step six. The seventh step concerns elaboration and sending of personalized feedback to the trainees and the stakeholders. Finally, the eighth and final step is a “macro-regulation” that consists of learning from all the previous steps in order to improve future transfer assessments cycles. During the first-year, we made a preliminary field testing and the second-year, a series of main field tests of the CCQCTTE. During the third year, the method was implemented in Montreal and in Lausanne. The three years international CCQCTTE project has made it possible to develop the method during construction of several transfer assessments for lifelong training programs. We highlighted a real added value of step 8 that transforms the cycle in a kind of spiral of quality. In terms of limitation, we note that step 1 “Analysis” remains time-consuming and that it is difficult to start without the accompaniment of an experienced expert of the CCQCTTE. As part of this paper, we will describe the CCQCTTE method and its quality approach, the circumstances in which it was developed and field tests results.
References
[1]
AERA, APA, ACME (1999). Standards for International and Psychological Testing. Washington: American International Research Association.
[2]
Alvarez, K., Salas, E., & Garofano, C. M. (2004). An Integrated Model of Training Evaluation and Effectiveness. Human Resource Development Review, 3, 385-416.
https://doi.org/10.1177/1534484304270820
[3]
Anderson, L. W. (2002). Curricular Alignment: A Re-Examination. Theory into Practice, 41, 255-260. https://doi.org/10.1207/s15430421tip4104_9
[4]
Antonioni, D. (1994). The Effects of Feedback Accountability on upward Appraisal Ratings. Personnel Psychology, 47, 349-356.
https://doi.org/10.1111/j.1744-6570.1994.tb01728.x
[5]
Atkinson, R. C. (1972). Ingredients for a Theory of Instruction. American Psychologist, 27, 921-931. https://doi.org/10.1037/h0033572
[6]
Baldwin, T. T., & Ford, J. K. (1988). Transfer of Training: A Review and Directions for Future Research. Personnel Psychology, 41, 63-105.
https://doi.org/10.1111/j.1744-6570.1988.tb00632.x
[7]
Bates, R., Holton, E. F., & Hatala, J. P. (2012). A Revised Learning Transfer System Inventory: Factorial Replication and Validation. Human Resource Development International, 15, 549-569. https://doi.org/10.1080/13678868.2012.726872
[8]
Best, M., & Neuhauser, D. (2006). Heroes and Martyrs of Quality and Safety—Walter A Shewhart, 1924, and the Hawthorne Factory. BMJ Quality & Safety, 15, 142-143.
https://doi.org/10.1136/qshc.2006.018093
[9]
Borg, W. R., & Gall, M. D. (1989). Educational Research. Longman.
[10]
Carnevale, A. P., & Desrochers, D. (1999). Training in the Dilbert Economy. Training & Development, 53, 32-36.
[11]
Cheng, E. W. L., & Hampson, I. (2007). Transfer of Training: A Review and New Insights. International Journal of Management Reviews, 10, 327-341.
https://doi.org/10.1111/j.1468-2370.2007.00230.x
[12]
Chochard, Y. (2012). Les variables influencant le rendement des formations managériales: Une étude de cas multiple suisse par la méthode de l’analyse de l’utilité. Université de Fribourg. http://doc.rero.ch/record/30448
[13]
Chochard, Y. (2018). Les apports de la formation continue offerte aux organismes communautaires: étude de cas d’une formation. La revue canadienne pour l’étude de l’éducation des adultes, 50, 32-46.
[14]
CHUV (2017). Rapport d’activités 2017 du Centre hospitalier universitaire vaudois. Lausanne: CHUV.
[15]
Cox, R. C., & Vargas, J. S. (1966, April). A Comparison of Item Selection Techniques for Norm-Referenced and Criterion-Referenced Tests. In Annual Conference of the National Council on Measurement in Education. Chicago.
[16]
D’Hainaut, L. (1973). étude d’une nouvelle variable pour l’analyse statistique des expériences pédagogiques. Bulletin de psychologie, 26, 622-630.
[17]
Deming, W. E. (1986). Out of the Crisis (p. 88). MIT Press.
[18]
Deming, W. E. (1993). The New Economics for Industry, Government, and Education. MIT Press.
[19]
Downing, S. M. (2011). Twelve Steps for Effective Test Development. In S. M. Downing, & T. M. Haladyna (Eds.), Handbook of Test Development (pp. 3-24). Routlege.
[20]
Dunberry, A., & Péchard, C. (2007). L’évaluation de la formation dans l’entreprise-état de la question et perspectives.
https://www.cirdef.uqam.ca/wp-content/uploads/2015/03/CIRDEF-PSRA-Dunberry_EvaluationEntrepriseLitt2007.pdf
[21]
Fives, H., & Didonato-Barnes, N. (2013). Classroom Test Construction: The Power of a Table of Specifications. Practical Assessment, Research, and Evaluation, 18, 1-7.
http://pareonline.net/getvn.asp?v=18&n=3
[22]
Fleishman, E. (1953). Leadership Climate, Human Relations Training, and Supervisory Behavior. Personnel Psychology, 6, 205-222.
https://doi.org/10.1111/j.1744-6570.1953.tb01040.x
[23]
Formaeva (2011). Les pratiques d’évaluation des formations des entreprises francaises en 2010. Formaeva.
[24]
Formaeva (2012). Comparaison des pratiques d’évaluation des formations en France et au Québec. Formaeva.
[25]
Frash, R., Kline, S., Almanza, B., & Antun, J. (2008). Support for a Multi-Level Evaluation Framework in Hospitality Training. Journal of Human Resources in Hospitality & Tourism, 7, 197-218. https://doi.org/10.1080/15332840802156949
[26]
Georgenson, D. L. (1982). The Problem of Transfer Calls for Partnership. Training & Development Journal, 36, 75-78.
[27]
Gilles, J.-L. (2002). Qualité spectrale des tests standardisés universitaires—Mise au point d’indices édumétriques d’analyse de la qualité spectrale des évaluations des acquis des étudiants universitaires et application aux épreuves MOHICAN check up ’99 (Thèse de doctorat en sciences de l’éducation). Université de Liège, Faculté de psychologie et des sciences de l’éducation.
[28]
Gilles, J.-L. (2017). Origin, Foundation, Objectives, and Original Aspects of the PEERS Program Linking Research and Training in Internationalization of Teacher Education. In J.-L. Gilles (Ed.), Linking Research and Training in Internationalization of Teacher Education with the PEERS Program: Issues, Case Studies and Perspectives (pp. 25-48). Peter Lang. https://doi.org/10.3726/b11697
[29]
Gilles, J.-L., & Lovinfosse, V. (2004). Utilisation du cycle SMART de gestion qualité des évaluations standardisées dans le contexte d’une Haute Ecole: Regard critique en termes de validité, fidélité, sensibilité des mesures, diagnosticité, praticabilité, équité, communicabilité et authenticité. In S. Arzola, R. Vizcarra, A. Cornejo, & G. Undurraga (Eds.), World Education for Educational Research (WAER)—XIV Congreso Mundial de Ciencias de la Education: Educatores para una Nueva Cultura—Summary of papers abstracts. Pontifica Universidad Catolica de Chile, Facultad de Education.
[30]
Gilles, J.-L., & Tinnirello, S. (2017). DOCIMO: An Online Platform Dedicated to the Construction and Quality Management of Learning and Impact Assessments in the Digital Age. Poster Presented at the The #dariahTeach Open Resources Conference.
http://hdl.handle.net/20.500.12162/107
[31]
Gilles, J.-L., Gutmann, C., & Tedesco, B. (2012). Colloque “PEERS Projects, a New Key Component for the Internationalization of Teacher Education”. In G. Baillat (Ed.), Livre des résumés du 17e Congrès de l’Association mondiale des sciences de l’éducation (pp. 108-111). Université de Reims. http://hdl.handle.net/20.500.12162/182
[32]
Gilles, J.-L., Chochard, Y., Berset, T., Bieri, S., El Guermai, J., Graveline, A. et al. (2017). Cycle de construction et de gestion qualité d’évaluation du transfert (CGQET). Université du Québec à Montréal. http://hdl.handle.net/20.500.12162/122
[33]
Gilles, J.-L., Piette, S.-A., Detroz, P., Tinnirello, S., Pirson, M., Dabo, M., & Lê, H. (2005). The Electronic Construction and Quality Control in Standardized Testing Platform Project (e-C&QCST). In A. Demetriou, & E. Dochy (Eds.), Proceedings of European Association for Research on Learning and Instruction (EARLI) (p. 429).
http://hdl.handle.net/20.500.12162/1807
[34]
Hale, J. (2002). Performance-Based Evaluation. Jossey-Bass/Pfeiffer.
[35]
Hedge, J. W., Borman, W. C., & Birkeland, S. A. (2001). History and Development of Multisource Feedback as a Methodology. In D. W. Bracken, C. W. Timmreck, & A. H. Church (Eds.), The Handbook of Multisource Feedback (pp. 15-32). Jossey Bass.
[36]
Holton III, E. F., Bates, R. A., & Ruona, W. E. A. (2000). Development of a Generalized Learning Transfer System Inventory. Human Resource Development Quarterly, 11, 333-359. https://doi.org/10.1002/1532-1096(200024)11:4%3C333::AID-HRDQ2%3E3.0.CO;2-P
[37]
Holton, E. F. (1996). The Flawed Four-Level Evaluation Model. Human Resource Development Quarterly, 7, 5-21. https://doi.org/10.1002/hrdq.3920070103
[38]
Jehanzeb, K., & Bashir, N. A. (2013). Training and Development Program and its Benefits to Employee and Organization: A Conceptual Study. European Journal of Business and Management, 5, 243-252.
[39]
Kirkpatrick, D. L. (1959). Techniques for Evaluating Training Programs. Journal of the American Society of Trainings Directors, 13, 21-26.
[40]
Kirkpatrick, D. L. (1996). Great Ideas Revisited. Training and Development, 50, 54-59.
[41]
Kirkpatrick, D. L. (1998). Evaluating training Programs (2nd ed.). Barrett-Koehler Publishers Inc.
[42]
Kirkpatrick, D. L. (2006). Seven Keys to Unlock the Four Levels of Evaluation. Performance Improvement, 45, 5-8. https://doi.org/10.1002/pfi.2006.4930450702
[43]
Kirkpatrick, D. L., & Kirkpatrick, J. D. (2008). Implementing the Four Levels: A Practical Guide for Effective Evaluation of Training Programs (153 p). Barret-Koehler Publishers, Inc.
[44]
Loiselle, J. (2001). La recherche développement en éducation: Sa nature et ses caractéristiques. In M. Anadón (Ed.), Nouvelles dynamiques de recherche en éducation (pp. 77-97). Les Presses de l’Université de Laval.
[45]
Mac Guigan, F. J. (1967). The G Statistics, An Index of Amount Learned. N.S.P. Journal, 6, 14-16. https://doi.org/10.1002/pfi.4180060905
[46]
Mathews, B. P., Ueno, A., Kekale, T., Repka, M., Pereire, Z. L., & Silva, G. (2001). Quality Training: Needs and Evaluation-Findings from a European Survey. Total Quality Management, 12, 483-490. https://doi.org/10.1080/09544120123778
[47]
Mckenna, J.-F. (1990). Take the “A” Training. Industry Week, 239, 22-29.
[48]
Meignant, A. (2009). Manager la formation (8e éd). Liaisons.
[49]
MESS (2001). Politique gouvernementale—L’action communautaire: Une contribution essentielle à l’exercice de la citoyenneté et au développement social du Québec: Portail 46 Chochard. “Les apports de la formation continue” Québec.
http://www4.gouv.qc.ca/fr/Portail/citoyens/programme-service/Pages/Info.aspx?sqctype=sujet&sqcid=2236
[50]
Morissette, D. (1996). Guide pratique de l’évaluation sommative—Gestion des épreuves et des examens. Editions du renouveau pédagogique.
[51]
Mwema, N. W., & Gachunga, H. G. (2014). The Influence of Performance Appraisal on Employee Productivity in Organizations: A Case Study of Selected WHO Offices in East Africa. International Journal of Social Sciences and Entrepreneurship, 1, 324-337.
[52]
Nightingale, P., & O’Neil, M. (1994). Achieving Quality Learning in Higher Education. Kogan Page.
[53]
Nitko, A., & Brookhart, S. (2007). Educational Assessment of Students (5th ed.). Pearson Merrill Prentice Hall.
[54]
Phillips, J. J., & Stone, R. D. (2002). How to Measure Training Results. McGraw-Hill.
[55]
Pottiez, J. (2017). L’évaluation de la formation—Pilotez et maximisez l’efficacité de vos formations (2e éd). Dunod.
[56]
Ramsden, P. (1991). A Performance Indicator of Teaching Quality in Higher Education: The Course Experience Questionnaire. Studies in Higher Education, 16, 129-150.
https://doi.org/10.1080/03075079112331382944
[57]
Rodriguez, J., & Walters, K. (2017). The Importance of Training and Development in Employee Performance and Evaluation. World Wide Journal of Multidisciplinary Research and Development, 3, 206-212.
[58]
Saks, A. M. (2002). So What Is a Good Transfer of Training Estimate? A Reply to Fitzpatrick. The Industrial-Organizational Psychologist, 39, 29-30.*
[59]
Segers, M., Dochy, F., & Cascallar, E. (2003). Optimising New Modes of Assessment: In Search of Qualities and Standards. Kluwer Academic Publishers.
https://doi.org/10.1007/0-306-48125-1
[60]
Shewhart, W. A. (1931). Economic Control of Quality of Manufactured Product (501 p). Van Nostrand.
[61]
Shewhart, W. A. (1958). Nature and Origin of Standards of Quality. Bell System Technical Journal, 37, 1-22. https://doi.org/10.1002/j.1538-7305.1958.tb03867.x
[62]
Shewhart, W. A. (1989). Les fondements de la maitrise de la qualité. Economia.
[63]
Van Buren, M. E., & Erskine, W. (2002). The 2002 ASTD State of the Industry Report (Vol. 8). American Society for Training & Development.
[64]
Van den Bossche, P., Segers, M., & Jansen, N. (2010). Transfer of Training: The Role of Feedback in Supportive Social Networks. International Journal of Training and Development, 14, 81-94. https://doi.org/10.1111/j.1468-2419.2010.00343.x
[65]
Van der Maren, J.-M. (2003). Chapitre 5. La recherche de développement. In La recherche appliquée en pédagogie (2e éd., pp. 107-124). De Boeck Supérieur.
https://doi.org/10.3917/dbu.maren.2003.01
[66]
Yamnill, S. & McLean, G. N. (2001). Theories Supporting Transfer of Training. Human Resource Development Quarterly, 12, 195-208. https://doi.org/10.1002/hrdq.7
[67]
Zink, K. J., & Schmidt, A. (1995). Measuring Universities against the European Quality Award Criteria. Total Quality Management, 6, 547-561.
https://doi.org/10.1080/09544129500000011
