Towards Robust Token Embeddings for Extractive Question Answering
Publication Name
Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Abstract
Extractive Question Answering (EQA) tasks have gained intensive attention in recent years, while Pre-trained Language Models (PLMs) have been widely adopted for encoding purposes. Yet, PLMs typically take as initial input token embeddings and rely on attention mechanisms to extract contextual representations. In this paper, a simple yet comprehensive framework, termed perturbation for alignment (PFA), is proposed to investigate variations towards token embeddings. A robust encoder is further formed being tolerant against the embedding variation and hence beneficial to subsequent EQA tasks. Specifically, PFA consists of two general modules, including the embedding perturbation (a transformation to produce embedding variations) and the semantic alignment (to ensure the representation similarity from original and perturbed embeddings). Furthermore, the framework is flexible to allow several alignment strategies with different interpretations. Our framework is evaluated on four highly-competitive EQA benchmarks, and PFA consistently improves state-of-the-art models.
Open Access Status
This publication is not available as open access
Volume
14306 LNCS
First Page
82
Last Page
96
Funding Number
888/008/268
Funding Sponsor
Australian Research Council