Agnostic debiasing of static embeddings: An approach to fairness in language models

Abstract

Word vector representations were the initial building block that started the current state-of-the-art methods for several NLP tasks. Bias metrics and debiasing methods for static embeddings have been studied with moderate success, achieving some bias reductions for specific groups and metrics. However, these methods often fail to improve multiple metrics simultaneously or to meaningfully impact extrinsic tasks. Recent research in debiasing has mainly shifted its focus towards contextual embeddings and large language models (LLMs). Here we argue that static embeddings provide a simpler and more controlled setting for testing hypotheses and techniques, which can then be extended to more complex models. We introduce a method that captures multiple demographic dimensions (gender, race, age, etc.) in static embeddings simultaneously, eliminating the need for specialized tasks or demographic-specific vocabulary.