Efficient Domain Adaptation of Sentence Embeddings Using Adapters

Tim Schopf, Dennis N. Schneider, Florian Matthes

Publikation: Beitrag in Buch/Bericht/KonferenzbandKonferenzbeitragBegutachtung

Abstract

Sentence embeddings enable us to capture the semantic similarity of short texts. Most sentence embedding models are trained for general semantic textual similarity tasks. Therefore, to use sentence embeddings in a particular domain, the model must be adapted to it in order to achieve good results. Usually, this is done by fine-tuning the entire sentence embedding model for the domain of interest. While this approach yields state-of-the-art results, all of the model's weights are updated during finetuning, making this method resource-intensive. Therefore, instead of fine-tuning entire sentence embedding models for each target domain individually, we propose to train lightweight adapters. These domain-specific adapters do not require fine-tuning all underlying sentence embedding model parameters. Instead, we only train a small number of additional parameters while keeping the weights of the underlying sentence embedding model fixed. Training domain-specific adapters allows always using the same base model and only exchanging the domain-specific adapters to adapt sentence embeddings to a specific domain. We show that using adapters for parameter-efficient domain adaptation of sentence embeddings yields competitive performance within 1% of a domainadapted, entirely fine-tuned sentence embedding model while only training approximately 3.6% of the parameters.

OriginalspracheEnglisch
TitelInternational Conference Recent Advances in Natural Language Processing, RANLP 2023
UntertitelLarge Language Models for Natural Language Processing - Proceedings
Redakteure/-innenGalia Angelova, Maria Kunilovskaya, Ruslan Mitkov
Herausgeber (Verlag)Incoma Ltd
Seiten1046-1053
Seitenumfang8
ISBN (elektronisch)9789544520922
DOIs
PublikationsstatusVeröffentlicht - 2023
Veranstaltung2023 International Conference Recent Advances in Natural Language Processing: Large Language Models for Natural Language Processing, RANLP 2023 - Varna, Bulgarien
Dauer: 4 Sept. 20236 Sept. 2023

Publikationsreihe

NameInternational Conference Recent Advances in Natural Language Processing, RANLP
ISSN (Print)1313-8502

Konferenz

Konferenz2023 International Conference Recent Advances in Natural Language Processing: Large Language Models for Natural Language Processing, RANLP 2023
Land/GebietBulgarien
OrtVarna
Zeitraum4/09/236/09/23

Fingerprint

Untersuchen Sie die Forschungsthemen von „Efficient Domain Adaptation of Sentence Embeddings Using Adapters“. Zusammen bilden sie einen einzigartigen Fingerprint.

Dieses zitieren