|
|
基于Span-BERT的实体关系联合抽取的研究
|
Abstract:
实体抽取和关系分类是自然语言处理领域其他任务的基石，二者效果直接或间接影响其他任务的效果。近年来得益于预训练语言模型在自然语言处理应用的巨大成功，实体关系联合抽取发展迅速，而当下使用BERT进行预训练基于Span抽取方式的联合抽取模型在解决实体重叠等问题的同时，依旧存在面对长文本效果变差，模型泛化性较差等问题。本文提出一个基于预训练语言模型Span-BERT进行微调的联合抽取模型，以Span为单位实现实体关系的联合抽取，在抽取训练过程中引入负样本采样策略，并在Span-BERT中进行有效提取，以此来增强模型性能和鲁棒性。实验结果和消融实验表明了该方法的有效性，通过不同程度的噪音化数据集SciERC，证明了本模型具有良好的鲁棒性，同时在ADE、CoNLL2004和SciERC个基准数据集上取得了不错的结果。
Entity extraction and relation classification serve as the cornerstones of various tasks in the field of natural language processing, with their effectiveness directly or indirectly impacting the outcomes of other tasks. In recent years, owing to the remarkable success of pre-trained language models in natural language processing applications, joint extraction of entities and relations has rapidly evolved. However, current pre-training methods utilizing BERT based on Span extraction suffer from challenges such as diminished performance on long texts and poor model generalization, despite addressing issues like entity overlap. This paper proposes a joint extraction model fine-tuned on the pre-trained language model Span-BERT. It achieves joint extraction of entities and relations at the Span level, introducing a negative sample sampling strategy during the extraction training process. Effective extraction is carried out within Span-BERT to enhance model performance and robustness. Experimental results and ablation studies demonstrate the effectiveness of this approach. Evaluated on different levels of noisy datasets such as SciERC, the model exhibits robustness. Moreover, it achieves promising results on benchmark datasets, including ADE, CoNLL2004, and SciERC.
| [1] | Wei, Z.P., Su, J.L., Wang, Y., Tian, Y. and Chang, Y. (2020) A Novel Cascade Binary Tagging Framework for Relational Triple Extraction. |
| [2] | Vinyals, O., Fortunato, M. and Jaitly, N. (2015) Pointer Networks. Advances in Neural Information Processing Systems, 28, 1-9. |
| [3] | Joshi, M., Chen, D.Q., Liu, Y.H., Weld, D.S., Zettlemoyer, L. and Levy, O. (2020) Spanbert: Improving Pre-Training by Representing and Predicting Spans. Transactions of the Association for Computational Linguistics, 8, 64-77. https://doi.org/10.1162/tacl_a_00300 |
| [4] | Socher, R., Huval, B., Manning, C.D. and Ng, A.Y. (2012) Semantic Compositionality through Recursive Matrix-Vector Spaces. Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, Jeju, 12-14 July 2012, 1201-1211. |
| [5] | Zhang, D.X. and Wang, D. (2014) Relation Classification via Convolutional Deep Neural Network. International Conference on Computational Linguistics, Dublin, August 2014, 2335-2344. |
| [6] | Xu, Y., Mou, L.L., Li, G., Chen, Y.C., Peng, H. and Jin, Z. (2015) Classifying Relations via Long Short Term Memory Networks along Shortest Dependency Paths. Association for Computational Linguistics, Lisbon. https://doi.org/10.18653/v1/D15-1206 |
| [7] | Zheng, S.C., Wang, F., Bao, H.Y., Hao, Y.X., Zhou, P. and Xu, B. (2017) Joint Extraction of Entities and Relations Based on a Novel Tagging Scheme. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, 1, 1227-1236. https://doi.org/10.18653/v1/P17-1113 |
| [8] | Yuan, Y., Zhou, X.F., Pan, S.R., Zhu, Q.N., Song, Z.L. and Guo, L. (2021) A Relation-Specific Attention Network for Joint Entity and Relation Extraction. In: Zhou, Z.H., Ed., International Joint Conference on Artificial Intelligence, Yokohama, 7-15 January 2021, 4054-4060. https://doi.org/10.24963/ijcai.2020/561 |
| [9] | Dai, D., Xiao, X.Y., Lyu, Y.J., Dou, S., She, Q.Q. and Wang, H.F. (2019) Joint Extraction of Entities and Overlapping Relations Using Position-Attentive Sequence Labeling. The AAAI Conference on Artificial Intelligence, Honolulu, 27 January-1 February 2019, 6300-6308. https://doi.org/10.1609/aaai.v33i01.33016300 |
| [10] | Zhang, H. (2023) SpanRE: Entities and Overlapping Relations Extraction Based on Spans and Entity Attention. |
| [11] | Luan, Y., Wadden, D., He, L.H., Shah, A., Ostendorf, M. and Hajishirzi, H. (2019) A General Framework for Information Extraction Using Dynamic Span Graphs. https://doi.org/10.18653/v1/N19-1308 |
| [12] | Wang, Y.C., Yu, B.W., Zhang, Y.Y., Liu, T.W., Zhu, H.S. and Sun, L.M. (2020) TPLinker: Single-Stage Joint Extraction of Entities and Relations through Token Pair Linking. Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, December 2020, 1572-1582. https://doi.org/10.18653/v1/2020.coling-main.138 |
| [13] | Eberts, M. and Ulges, A. (2021) Span-Based Joint Entity and Relation Extraction with Transformer Pre-Training. |
| [14] | Wang, Y., Liu, X.X., Hu, W.X. and Zhang, T. (2022) A Unified Positive-Unlabeled Learning Framework for Document-Level Relation Extraction with Different Levels of Labeling. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Abu Dhabi, December 2022, 4123-4135. https://doi.org/10.18653/v1/2022.emnlp-main.276 |
| [15] | Ye, Z.-X. and Ling, Z.-H. (2019) Multi-Level Matching and Aggregation Network for Few-Shot Relation Classification. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, July 2019, 2872-2881. |
| [16] | Dutordoir, V., Saul, A., Ghahramani, Z. and Simpson, F. (2022) Neural Diffusion Processes. |
| [17] | Lee, K., Salant, S., Kwiatkowksi, T., Parikh, A., Das, D. and Berant, J. (2017) Learning Recurrent Span Representations for Extractive Question Answering. |
| [18] | Devlin, J., Chang, M.-W., Lee, K. and Toutanova, K. (2019) Bert: Pre-Training of Deep Bidirectional Transformers for Language Understanding. Proceedings of NAACL-HLT 2019, Minneapolis, 2-7 June 2019, 4171-4186. |
| [19] | He, C.Y., Annavaram, M. and Avestimehr, S. (2020) Group Knowledge Transfer: Federated Learning of Large Cnns at the Edge. Advances in Neural Information Processing Systems, 33, 14068-14080. |
| [20] | Leng, Z.Q., Tan, M.X., Liu, C.X., Cubuk, E.D., Shi, X.J., Cheng, S.Y., Anguelov, D., et al. (2022) Polyloss: A Polynomial Expansion Perspective of Classification Loss Functions. 10th International Conference on Learning Representations (ICLR 2022), 25-29 April 2022, 1-16. |
| [21] | Gurulingappa, H., Rajput, A.M., Roberts, A., Fluck, J., Hofmann-Apitius, M. and Toldo, L. (2012) Development of a Benchmark Corpus to Support the Automatic Extraction of Drug-Related Adverse Effects from Medical Case Reports. Journal of Biomedical Informatics, 45, 885-892. https://doi.org/10.1016/j.jbi.2012.04.008 |
| [22] | Roth, D. and Yih, W.-T. (2004) A Linear Programming Formulation for Global Inference in Natural Language Tasks. In: Proceedings of the 8th Conference on Computational Natural Language Learning (CoNLL-2004) at HLT-NAACL 2004, Association for Computational Linguistics, Boston, 1-8. |
| [23] | Luan, Y., He, L.H., Ostendorf, M. and Hajishirzi, H. (2018) Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Construction. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, October-November 2018, 3219-3232. https://doi.org/10.18653/v1/D18-1360 |
| [24] | Theodoropoulos, C., Henderson, J., Coman, A.C. and Moens, M.-F. (2021) Imposing Relation Structure in Language-Model Embeddings Using Contrastive Learning. Proceedings of the 25th Conference on Computational Natural Language Learning, November 2021, 337-348. https://doi.org/10.18653/v1/2021.conll-1.27 |
| [25] | Ye, D.M., Lin, Y.K., Li, P. and Sun, M.S. (2021) Packed Levitated Marker for Entity and Relation Extraction. |
| [26] | Lan, Y.S., He, G.L., Jiang, J.H., et al. (2023) Complex Knowledge Base Question Answering: A Survey. IEEE Transactions on Knowledge and Data Engineering, 35, 11196-11215. https://doi.org/10.1109/TKDE.2022.3223858 |
| [27] | Qiao, D., Dai, C.C., Ding, Y.Y., Li, J.T., Chen, Q. and Chen, W.L. (2022) SelfMix: Robust Learning against Textual Label Noise with Self-Mixup Training. Proceedings of the 29th International Conference on Computational Linguistics, Gyeongju, October 2022, 960-970. |
