Document Type
Conference Paper
Publication Date
2024
DOI
10.18653/v1/2024.findings-naacl.241
Publication Title
2024 Findings of the Association for Computational Linguistics: NAACL 2024
Pages
3795-3809
Conference Name
2024 Annual Conference of the North American Association for Computational Linguistics, 16-21 June 2024, Mexico City, Mexico
Abstract
Pre-trained language models (PLMs) have consistently demonstrated outstanding performance across a diverse spectrum of natural language processing tasks. Nevertheless, despite their success with unseen data, current PLM-based representations often exhibit poor robustness in adversarial settings. In this paper, we introduce RobustSentEmbed, a self-supervised sentence embedding framework designed to improve both generalization and robustness in diverse text representation tasks and against a diverse set of adversarial attacks. Through the generation of high-risk adversarial perturbations and their utilization in a novel objective function, RobustSentEmbed adeptly learns high-quality and robust sentence embeddings. Our experiments confirm the superiority of RobustSentEmbed over state-of-the-art representations. Specifically, Our framework achieves a significant reduction in the success rate of various adversarial attacks, notably reducing the BERTAttack success rate by almost half (from 75.51% to 38.81%). The framework also yields improvements of 1.59% and 0.23% in semantic textual similarity tasks and various transfer tasks, respectively.
Rights
© 2024 Association for Computational Linguistics
Licensed on a Creative Commons Attribution 4.0 International (CC BY 4.0) License.
Original Publication Citation
Asl, J. R., Panzade, P., Blanco, E., Takabi, D., & Cai, Z. (2024). RobustSentEmbed: Robust sentence embeddings using adversarial self-supervised contrastive learning. In K. Duh, H. Gomez, & S. Bethard (Eds.), 2024 Findings of the Association for Computational Linguistics: NAACL 2024 (pp. 3795-3809). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.findings-naacl.241
Repository Citation
Asl, J. R., Panzade, P., Blanco, E., Takabi, D., & Cai, Z. (2024). RobustSentEmbed: Robust sentence embeddings using adversarial self-supervised contrastive learning. In K. Duh, H. Gomez, & S. Bethard (Eds.), 2024 Findings of the Association for Computational Linguistics: NAACL 2024 (pp. 3795-3809). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.findings-naacl.241
ORCID
0000-0003-0447-3641 (Takabi)