Extracting fine-grained information from social media is traditionally a challenging task, since the language used in social media messages is usually informal, with creative genre-specific terminology and expression. How to handle such a challenge so as to automatically understand the opinions that people are communicating has become a hot subject of research. In this paper, we aim to show that leveraging the pre-learned knowledge can help neural network models understand the creative language in Tweets. In order to address this idea, we present a transfer learning model based on BERT. We fine-turned the pre-trained BERT model and applied the customized model to two downstream tasks described in SemEval-2018: Irony Detection task and Emoji Prediction task of Tweets. Our model could achieve an F-score of 38.52 (ranked 1/49) in Emoji Prediction task and 67.52 (ranked 2/43) and 51.35 (ranked 1/31) in Irony Detection subtask A and subtask B. The experimental results validate the effectiveness of our idea.
References
[1]
Rosenthal, S., Farra, N. and Nakov, P. (2017) SemEval-2017 Task 4: Sentiment Analysis in Twitter. Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), Vancouver, Canada, August 2017, 502-518.
https://doi.org/10.18653/v1/S17-2088
[2]
Nakov, P., Ritter, A., Rosenthal, S., Sebastiani, F. and Stoyanov, V. (2016) SemEval-2016 Task 4: Sentiment Analysis in Twitter. Proceedings of the 10th International Workshop on Semantic Evaluation (Semeval-2016), San Diego, CA, June 2016, 1-18. https://doi.org/10.18653/v1/S16-1001
[3]
Van Hee, C., Lefever, E. and Hoste, V. (2018) Semeval-2018 Task 3: Irony Detection in English Tweets. Proceedings of The 12th International Workshop on Semantic Evaluation, New Orleans, LA, June 2018, 39-50.
https://doi.org/10.18653/v1/S18-1005
[4]
Wu, C., Wu, F., Wu, S., Liu, J., Yuan, Z. and Huang, Y. (2018) THU_NGN at Semeval-2018 Task 3: Tweet Irony Detection with Densely Connected LSTM and Multi-Task Learning. Proceedings of The 12th International Workshop on Semantic Evaluation, New Orleans, LA, June 2018, 51-56.
https://doi.org/10.18653/v1/S18-1006
[5]
Baziotis, C., Nikolaos, A., Kolovou, A., Paraskevopoulos, G., Ellinas, N. and Potamianos, A. (2018) NTUA-SLP at SemEval-2018 Task 2: Predicting Emojis Using RNNs with Context-Aware Attention. Proceedings of The 12th International Workshop on Semantic Evaluation, New Orleans, LA, June 2018, 438-444.
https://doi.org/10.18653/v1/s18-1069
[6]
Paetzold, G. (2018) UTFPR at IEST 2018: Exploring Character-to-Word Composition for Emotion Analysis. Proceedings of the 9th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, Brussels, Belgium, October 2018, 176-181. https://doi.org/10.18653/v1/W18-6224
[7]
Felbo, B., Mislove, A., Søgaard, A., Rahwan, I. and Lehmann, S. (2017) Using Millions of Emoji Occurrences to Learn Any-Domain Representations for Detecting Sentiment, Emotion and Sarcasm. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark, September 2017, 1615-1625. https://doi.org/10.18653/v1/D17-1169
[8]
Kiros, R., Zhu, Y., Salakhutdinov, R.R., Zemel, R., Urtasun, R., Torralba, A. and Fidler, S. (2015) Skip-Thought Vectors. In: Advances in Neural Information Processing Systems, 3294-3302.
[9]
West, J., Ventura, D. and Warnick, S. (2007) Spring Research Presentation: A Theoretical Foundation for Inductive Transfer. Brigham Young University, College of Physical and Mathematical Sciences, 32.
[10]
Whlson, T., Kozareva, Z., Nakov, P., Rosenthal, S., Stoyanov, V. and Ritter, A. (2013) SemEval-2013 Task 2: Sentiment Analysis in Twitter. Proceedings of the International Workshop on Semantic Evaluation, Atlanta, GA.
[11]
Mohammad, S., Bravo-Marquez, F., Salameh, M. and Kiritchenko, S. (2018) Semeval-2018 Task 1: Affect in Tweets. Proceedings of the 12th International Workshop on Semantic Evaluation, New Orleans, LA, June 2018, 1-17.
https://doi.org/10.18653/v1/S18-1001
[12]
Zhu, X., Kiritchenko, S. and Mohammand, S. (2014) Nrc-Canada-2014: Recent Improvements in the Sentiment Analysis of Tweets. Proceedings of the 8th International Workshop on Semantic Evaluation, Dublin, Ireland, August 2014, 443-447.
https://doi.org/10.3115/v1/S14-2077
[13]
Tang, D., Wei, F., Qin, B., Liu, T. and Zhou, M. (2014) Coooolll: A Deep Learning System for Twitter Sentiment Classification. Proceedings of the 8th International Workshop on Semantic Evaluation, Dublin, Ireland, August 2014, 208-212.
https://doi.org/10.3115/v1/S14-2033
[14]
Cappallo, S., Mensink, T. and Snoek, C.G. (2015) Image2emoji: Zero-Shot Emoji Prediction for Visual Media. Proceedings of the 23rd ACM International Conference on Multimedia, Brisbane, Australia, 26-30 October 2015, 1311-1314.
https://doi.org/10.1145/2733373.2806335
[15]
Barbieri, F., Ballesteros, M. and Saggion, H. (2017) Are Emojis Predictable? Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, 2, 105-111. https://doi.org/10.18653/v1/E17-2017
[16]
Barbieri, F., Camacho-Collados, J., Ronzano, F., Anke, L.E., Ballesteros, M., Basile, V., Saggion, H., et al. (2018) SemEval 2018 Task 2: Multilingual Emoji Prediction. Proceedings of The 12th International Workshop on Semantic Evaluation, New Orleans, LA, June 2018, 24-33. https://doi.org/10.18653/v1/S18-1003
[17]
Bouazizi, M. and Ohtsuki, T.O. (2016) A Pattern-Based Approach for Sarcasm Detection on Twitter. IEEE Access, 4, 5477-5488.
https://doi.org/10.1109/ACCESS.2016.2594194
[18]
Van Hee, C., Lefever, E. and Hoste, V. (2016) Exploring the Realization of Irony in Twitter Data. Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), 1794-1799.
[19]
Radford, A., Narasimhan, K., Salimans, T. and Sutskever, I. (2018) Improving Language Understanding with Unsupervised Learning. Technical Report, OpenAI.
[20]
Devlin, J., Chang, M.W., Lee, K. and Toutanova, K. (2019) BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 1, 4171-4186.
[21]
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R. and Le, Q.V. (2019) XLNet: Generalized Autoregressive Pretraining for Language Understanding. arXiv preprint arXiv:1906.08237.
[22]
Baziotis, C., Pelekis, N. and Doulkeridis, C. (2017) Datastories at Semeval-2017 Task 4: Deep LSTM with Attention for Message-Level and Topic-Based Sentiment Analysis. Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), Vancouver, Canada, August 2017, 747-754.
https://doi.org/10.18653/v1/S17-2126
[23]
Çöltekin, Ç. and Rama, T. (2018) Tübingen-Oslo at SemEval-2018 Task 2: SVMs Perform Better than RNNs in Emoji Prediction. Proceedings of The 12th International Workshop on Semantic Evaluation, New Orleans, LA, June 2018, 34-38.
https://doi.org/10.18653/v1/S18-1004
[24]
Liu, M. (2018) EmoNLP at SemEval-2018 Task 2: English Emoji Prediction with Gradient Boosting Regression Tree Method and Bidirectional LSTM. Proceedings of The 12th International Workshop on Semantic Evaluation, New Orleans, LA, June 2018, 390-394. https://doi.org/10.18653/v1/S18-1059
[25]
Beaulieu, J. and Owusu, D.A. (2018) UMDuluth-CS8761 at SemEval-2018 Task 2: Emojis: Too Many Choices? Proceedings of The 12th International Workshop on Semantic Evaluation, New Orleans, LA, June 2018, 400-404.
https://doi.org/10.18653/v1/S18-1061
[26]
Baziotis, C., Nikolaos, A., Papalampidi, P., Kolovou, A., Paraskevopoulos, G., Ellinas, N. and Potamianos, A. (2018) NTUA-SLP at SemEval-2018 Task 3: Tracking Ironic Tweets Using Ensembles of Word and Character Level Attentive RNNs. Proceedings of The 12th International Workshop on Semantic Evaluation, New Orleans, LA, June 2018, 613-621. https://doi.org/10.18653/v1/S18-1100
[27]
Rohanian, O., Taslimipoor, S., Evans, R. and Mitkov, R. (2018) WLV at SemEval-2018 Task 3: Dissecting Tweets in Search of Irony. Proceedings of The 12th International Workshop on Semantic Evaluation, New Orleans, LA, June 2018, 553-559.
https://doi.org/10.18653/v1/S18-1090
[28]
Ghosh, A. and Veale, T. (2018) IronyMagnet at SemEval-2018 Task 3: A Siamese Network for Irony Detection in Social Media. Proceedings of The 12th International Workshop on Semantic Evaluation, New Orleans, LA, June 2018, 570-575.
https://doi.org/10.18653/v1/S18-1093
[29]
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Polosukhin, I., et al. (2017) Attention Is All You Need. 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, 5998-6008.
