Improving Machine Reading Comprehension through A Simple Masked-Training Scheme
Publication Name
IJCNLP-AACL 2023 - 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, Findings of the Association for Computational Linguistics: IJCNLP-AACL 2023
Abstract
Extractive Question Answering (EQA) is a fundamental problem in Natural Language Understanding, aiming at answering given questions via extracting a contiguous sequence or span of words from a passage. Recent work on EQA has achieved promising performance with the help of pre-trained language models, for which Masked Language Modeling (MLM) is usually adopted as a pre-training task to predict masked tokens. This paper revisits MLM and proposes a simple yet effective method to improve the EQA performance, termed the [Mask]-for-Answering method (M4A). Specifically, three masking strategies are first introduced, which produce masked copies of the original passages. Instead of predicting masked tokens as in MLM, both original samples and masked copies are utilized simultaneously for training the EQA model. Importantly, a discrepancy loss is further incorporated to ensure that masked copies remain semantically close to the originals. As such, M4A is able to produce robust embeddings for both original and masked samples and infer correct answers even with masked context. Experimental study on several highly-competitive benchmarks consistently demonstrates the superiority of our proposed method over existing methods. M4A also achieves strong performance in low-resource settings and out-of-domain generalization.
Open Access Status
This publication is not available as open access
First Page
222
Last Page
232
Funding Number
DP210101426
Funding Sponsor
Australian Research Council