Easy-to-use combination of POS and BERT model for domain-specific and misspelled terms

Alexandra Benamar (1) , Meryl Bothua (2) , Cyril Grouin (1) , Anne Vilnat (1)

1 ILES - Information, Langue Ecrite et Signée
2 EDF R&D - EDF R&D

In this paper, we present BERT-POS, a simple method for encoding syntax into BERT embeddings without retraining or finetuning data, based on Part-Of-Speech (POS). Although fine-tuning is the most popular method to apply BERT models on domain datasets, it remains expensive in terms of training time, computing resources, training data selection and retraining frequency. Our alternative works at the preprocessing level and relies on POS tagging sentences. It gives interesting results for words similarity regarding out-of-vocabulary both in terms of domain-specific words and misspellings. More specifically, the experiments were done on French language, but we believe that they would be similar on others.

Mots clés: Natural Language Processing; Language Models; Semantic Similarity; Out-of-Vocabulary Words; Part-Of-Speech

Domaines: Traitement du texte et du document

Alexandra Benamar, Meryl Bothua, Cyril Grouin, Anne Vilnat. Easy-to-use combination of POS and BERT model for domain-specific and misspelled terms. NL4IA Workshop Proceedings, Nov 2021, Milan, Italy. ⟨hal-03474696⟩