全部 标题 作者
关键词 摘要

OALib Journal期刊
ISSN: 2333-9721
费用:99美元

查看量下载量

相关文章

更多...

Incremental Learning Based on Data Translation and Knowledge Distillation

DOI: 10.4236/ijis.2023.132003, PP. 33-47

Keywords: Incremental Domain Learning, Data Translation, Knowledge Distillation, Cat-astrophic Forgetting

Full-Text   Cite this paper   Add to My Lib

Abstract:

Recently, deep convolutional neural networks (DCNNs) have achieved remarkable results in image classification tasks. Despite convolutional networks’ great successes, their training process relies on a large amount of data prepared in advance, which is often challenging in real-world applications, such as streaming data and concept drift. For this reason, incremental learning (continual learning) has attracted increasing attention from scholars. However, incremental learning is associated with the challenge of catastrophic forgetting: the performance on previous tasks drastically degrades after learning a new task. In this paper, we propose a new strategy to alleviate catastrophic forgetting when neural networks are trained in continual domains. Specifically, two components are applied: data translation based on transfer learning and knowledge distillation. The former translates a portion of new data to reconstruct the partial data distribution of the old domain. The latter uses an old model as a teacher to guide a new model. The experimental results on three datasets have shown that our work can effectively alleviate catastrophic forgetting by a combination of the two methods aforementioned.

References

[1]  Hsu, Y.C., Liu, Y.C., Ramasamy, A. and Kira, Z. (2018) Re-Evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines. Preprint.
https://arxiv.org/abs/1810.12488
[2]  van de Ven, G.M. and Tolias, A.S. (2019) Three Scenarios for Continual Learning.
https://arxiv.org/abs/1904.07734
[3]  McClosk ey, M. and Cohen, N.J. (1989) Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem. Psychology of Learning and Motivation, 24, 109-165.
https://doi.org/10.1016/S0079-7421(08)60536-8
[4]  Kirkpatrick, J., Pascanu, R., Rabinowitz, N.C., Veness, J., Desjardins, G., Rusu, A.A., Milan, K., Quan, J., Ramalho, T., Grabska Barwinska, A., et al. (2016) Overcoming Catastrophic Forgetting in Neural Networks. Proceedings of the National Academy of Sciences of the United States of America, 114, 3521-3526.
https://doi.org/10.1073/pnas.1611835114
[5]  Zenke, F., Poole, B. and Ganguli, S. (2017) Continual Learning through Synaptic Intelligence. Proceedings of the 34th International Conference on Machine Learning, Sydney, 6-11 August 2017, 3987-3995.
[6]  Aljundi, R., Babiloni, F., Elhoseiny, M., Rohrbach, M. and Tuytelaars, T. (2018) Memory Aware Synapses: Learning What (Not) to Forget. Proceedings of the Computer Vision-ECCV 2018-15th European Conference, Munich, 8-14 September 2018, 144-161.
https://doi.org/10.1007/978-3-030-01219-9_9
[7]  Delange, M., Aljundi, R., Masana, M., Parisot, S., Jia, X., Leonardis, A., Slabaugh, G., Tuytelaars, T. (2021) A Continual Learning Survey: Defying Forgetting in Classification Tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44, 3366-3385.
https://doi.org/10.1109/TPAMI.2021.3057446
[8]  Parisi, G.I., Kemker, R., Part, J.L., Kanan, C., Wermter, S. (2019) Continual Lifelong Learning with Neural Ntworks: A Review. Neural Networks, 113, 54-71.
https://doi.org/10.1016/j.neunet.2019.01.012
[9]  Belouadah, E., Popescu, A. and Kanellos, I. (2020) A Comprehensive Study of Class Incremental Learning Algorithms for Visual Tasks. Neural Networks, 135, 38-54.
https://doi.org/10.1016/j.neunet.2020.12.003
[10]  Zhao, B., Xiao, X., Gan, G., Zhang, B. and Xia, S.T. (2020) Fairness in Class Incremental Learning. Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, 13-19 June 2020, 13205-13214.
https://doi.org/10.1109/CVPR42600.2020.01322
[11]  Hou, S., Pan, X., Loy, C.C., Wang, Z. and Lin, D. (2019) Learning a Unified Classifier Incrementally via Rebalancing. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, 16-20 June 2019, 831-839.
https://doi.org/10.1109/CVPR.2019.00092
[12]  Mittal, S., Galesso, S. and Brox, T. (2021) Essentials for Class Incremental Learning. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, Nashville, TN, 19-25 June 2021, 3508-3517.
https://doi.org/10.1109/CVPRW53098.2021.00390
[13]  Volpi, R., Larlus, D. and Rogez, G. (2020) Continual Adaptation of Visual Representations via Domain Randomization and Meta-Learning. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, 20-25 June 2021, 4441-4451.
https://doi.org/10.1109/CVPR46437.2021.00442
[14]  Fernando, C., Banarse, D., Blundell, C., Zwols, Y., Ha, D., Rusu, A.A., Pritzel, A. and Wierstra, D. (2017) PathNet: Evolution Channels Gradient Descent in Super Neural Networks.
https://arxiv.org/abs/1701.08734
[15]  Serrà, J., Suris, D., Miron, M. and Karatzoglou, A. (2018) Overcoming Catastrophic Forgetting with Hard Attention to the Task. Proceedings of the Proceedings of the 35th International Conference on Machine Learning, Stockholm, 10-15 July 2018, 4555-4564.
[16]  Rusu, A.A., Rabinowitz, N.C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R. and Hadsell, R. (2016) Progressive Neural Networks.
https://arxiv.org/abs/1606.04671
[17]  Xu, J. and Zhu, Z. (2018) Reinforced Continual Learning. Proceedings of the Advances in Neural Information Processing Systems, Montréal, 3-8 December 2018, 907-916.
[18]  Yan, S., Xie, J. and He, X. (2021) DER: Dynamically Expandable Representation for Class Incremental Learning. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, 20-25 June 2021, 3013-3022.
https://doi.org/10.1109/CVPR46437.2021.00303
[19]  Rajasegaran, J., Khan, S., Hayat, M., Khan, F.S. and Shah, M. (2020) iTAML: An Incremental Task-Agnostic Meta-learning Approach. Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, 13-19 June 2020, 13585-13594.
https://doi.org/10.1109/CVPR42600.2020.01360
[20]  Wang, Q., Fink, O., Van Gool, L., et al. (2022) Continual Test-Time Domain Adaptation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, LA, 18-24 June 2022, 7201-7211.
https://doi.org/10.1109/CVPR52688.2022.00706
[21]  Lee, K., Lee, K., Shin, J. and Lee, H. (2019) Overcoming Catastrophic Forgetting With Unlabeled Data in the Wild. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision, Seoul, 27 October-2 November 2019, 312-321.
https://doi.org/10.1109/ICCV.2019.00040.
[22]  Peng, X., Bai, Q., Xia, X., Huang, Z., Saenko, K. and Wang, B. (2019) Moment Matching for Multi-Source Domain Adaptation. Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision, Seoul, 27 October-2 November 2019, 1406-1415.
https://doi.org/10.1109/ICCV.2019.00149
[23]  Pan, S.J., Tsang, I.W., Kwok, J.T. and Yang, Q. (2011) Domain Adaptation via Transfer Component Analysis. IEEE Transactions on Neural Networks, 22, 199-210.
https://doi.org/10.1109/TNN.2010.2091281
[24]  Hinton, G.E., Vinyals, O. and Dean, J. (2015) Distilling the Knowledge in a Neural Network.
https://arxiv.org/abs/1503.02531
[25]  Addepalli, S., Nayak, G.K., Chakraborty, A. and Radhakrishnan, V.B. (2020) DeGAN: Data-Enriching GAN for Retrieving Representative Samples from a Trained Classifier. Proceedings of the The Thirty-Fourth AAAI Conference on Artificial Intelligence, New York, NY, 7-12 February 2020, 3130-3137.
[26]  Kim, J., Jeong, J. and Shin, J. (2020) M2m: Imbalanced Classification via Major-to-Minor Translation. Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, 13-19 June 2020, 13893-13902.
https://doi.org/10.1109/CVPR42600.2020.01391.
[27]  LeCun, Y., Bottou, L., Bengio, Y. and Haffner, P. (1998) Gradient-Based Learning Applied to Document Recognition. Proceedings of the IEEE, 86, 2278-2324.
https://doi.org/10.1109/5.726791
[28]  Ganin, Y. and Lempitsky, V.S. (2015) Unsupervised Domain Adaptation by Backpropagation. Proceedings of the Proceedings of the 32nd International Conference on Machine Learning, Lille, 6-11 July 2015, 1180-1189.
[29]  Xu, R., Chen, Z., Zuo, W., Yan, J. and Lin, L. (2018) Deep Cocktail Network: Multi-Source Unsupervised Domain Adaptation with Category Shift. Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, 18-22 June 2018, 3964-3973.
https://doi.org/10.1109/CVPR.2018.00417
[30]  Krizhevsky, A., Hinton, G., et al. (2009) Learning Multiple Layers of Features from Tiny Images.
http://www.cs.utoronto.ca/~kriz/learning-features-2009-TR.pdf
[31]  Deng, J., Dong, W., Socher, R., Li, L., Li, K. and Li, F. (2009) ImageNet: A Large-Scale Hierarchical Image Database. Proceedings of the 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Miami, 20-25 June 2009, 248-255.
https://doi.org/10.1109/CVPR.2009.5206848
[32]  Sutskever, I., Martens, J., Dahl, G.E. and Hinton, G.E. (2013) On the Importance of Initialization and Momentum in Deep Learning. Proceedings of the 30th International Conference on Machine Learning, Atlanta, GA, 16-21 June 2013, 1139-1147.
[33]  Krizhevsky, A., Sutskever, I. and Hinton, G.E. (2012) ImageNet Classification with Deep Convolutional Neural Networks. Proceedings of the Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012, Nevada, 3-6 December 2012, 1106-1114.
[34]  He, K., Zhang, X., Ren, S. and Sun, J. (2016) Deep Residual Learning for Image Recognition. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, 27-30 June 2016, 770-778.
https://doi.org/10.1109/CVPR.2016.90.

Full-Text

Contact Us

service@oalib.com

QQ:3279437679

WhatsApp +8615387084133