Exploration and mitigation of gender bias in word embeddings from transformer-based language models
This thesis is submitted in partial fulfillment of the requirements for the degree of Bachelor of Science in Computer Science and Engineering, 2023.
| Auteurs principaux: | , , , , |
|---|---|
| Autres auteurs: | |
| Format: | Thèse |
| Langue: | English |
| Publié: |
Brac University
2024
|
| Sujets: | |
| Accès en ligne: | http://hdl.handle.net/10361/23457 |
| id |
10361-23457 |
|---|---|
| record_format |
dspace |
| spelling |
10361-234572024-06-13T21:04:51Z Exploration and mitigation of gender bias in word embeddings from transformer-based language models Hossain, Ariyan Haque, Rakinul Hannan, Khondokar Mohammad Ahanaf Rafa, Nowreen Tarannum Musarrat, Humayra Sadeque, Farig Yousuf Department of Computer Science and Engineering, Brac University Natural Llnguage processing Gender bias Debiasing Continued pretraining Natural language processing (Computer science) This thesis is submitted in partial fulfillment of the requirements for the degree of Bachelor of Science in Computer Science and Engineering, 2023. Cataloged from PDF version of thesis. Includes bibliographical references (pages 60-66). Machine learning has the potential to uncover data biases resulting from human error when it’s implemented without proper restraint. However, this complexity arises from word embedding, which is a prominent technique for capturing textual input as vectors applied in different machine learning and natural language processing tasks. Word embeddings are biased because they are trained on text data, which frequently incorporates prejudice and bias from society. These biases may become deeply established in the embeddings, producing unfair or biased results in AI applications. There are efforts made to recognise and lessen certain prejudices, but comprehensive bias elimination is still a difficult task. In Natural Language Processing (NLP) systems, contextualized word embeddings have taken the place of traditional embeddings as the preferred source of representational knowledge. It is critical to evaluate biases contained in their replacements as well since biases of various kinds have already been discovered in standard word embeddings. Our focus is on transformer-based language models, primarily BERT, which produce contextual word embeddings. To measure the extent to which gender biases exist, we apply various methods like cosine similarity test, direct bias test and ultimately detect bias through probability of filling MASK by the models. Based on this probability, we develop a novel metric called MALoR to observe bias. Finally, to mitigate the bias, we continue pretraining these models on a gender balanced dataset. Gender balanced dataset is created by applying Counterfactual Data Augmentation (CDA). To ensure consistency, we perform our experiments on different gender pronouns and nouns - “he-she”, “his-her” and “male names-female names”. These debiased models can then be used across several applications. Ariyan Hossain Rakinul Haque Khondokar Mohammad Ahanaf Hannan Nowreen Tarannum Rafa Humayra Musarrat B.Sc in Computer Science 2024-06-13T11:33:47Z 2024-06-13T11:33:47Z 2023 2023-09 Thesis ID 20101099 ID 20101290 ID 20101079 http://hdl.handle.net/10361/23457 en Brac University theses are protected by copyright. They may be viewed from this source for any purpose, but reproduction or distribution in any format is prohibited without written permission. 76 pages application/pdf Brac University |
| institution |
Brac University |
| collection |
Institutional Repository |
| language |
English |
| topic |
Natural Llnguage processing Gender bias Debiasing Continued pretraining Natural language processing (Computer science) |
| spellingShingle |
Natural Llnguage processing Gender bias Debiasing Continued pretraining Natural language processing (Computer science) Hossain, Ariyan Haque, Rakinul Hannan, Khondokar Mohammad Ahanaf Rafa, Nowreen Tarannum Musarrat, Humayra Exploration and mitigation of gender bias in word embeddings from transformer-based language models |
| description |
This thesis is submitted in partial fulfillment of the requirements for the degree of Bachelor of Science in Computer Science and Engineering, 2023. |
| author2 |
Sadeque, Farig Yousuf |
| author_facet |
Sadeque, Farig Yousuf Hossain, Ariyan Haque, Rakinul Hannan, Khondokar Mohammad Ahanaf Rafa, Nowreen Tarannum Musarrat, Humayra |
| format |
Thesis |
| author |
Hossain, Ariyan Haque, Rakinul Hannan, Khondokar Mohammad Ahanaf Rafa, Nowreen Tarannum Musarrat, Humayra |
| author_sort |
Hossain, Ariyan |
| title |
Exploration and mitigation of gender bias in word embeddings from transformer-based language models |
| title_short |
Exploration and mitigation of gender bias in word embeddings from transformer-based language models |
| title_full |
Exploration and mitigation of gender bias in word embeddings from transformer-based language models |
| title_fullStr |
Exploration and mitigation of gender bias in word embeddings from transformer-based language models |
| title_full_unstemmed |
Exploration and mitigation of gender bias in word embeddings from transformer-based language models |
| title_sort |
exploration and mitigation of gender bias in word embeddings from transformer-based language models |
| publisher |
Brac University |
| publishDate |
2024 |
| url |
http://hdl.handle.net/10361/23457 |
| work_keys_str_mv |
AT hossainariyan explorationandmitigationofgenderbiasinwordembeddingsfromtransformerbasedlanguagemodels AT haquerakinul explorationandmitigationofgenderbiasinwordembeddingsfromtransformerbasedlanguagemodels AT hannankhondokarmohammadahanaf explorationandmitigationofgenderbiasinwordembeddingsfromtransformerbasedlanguagemodels AT rafanowreentarannum explorationandmitigationofgenderbiasinwordembeddingsfromtransformerbasedlanguagemodels AT musarrathumayra explorationandmitigationofgenderbiasinwordembeddingsfromtransformerbasedlanguagemodels |
| _version_ |
1814308502029467648 |