AVID-2022-R0002

Description

Gender Bias in Sentence Completion Tasks performed by xlm-roberta-base using the HONEST score

Details

Sentence completions by xlm-roberta-base were found to be significantly biased for one lexical category as defined by the HONEST hurtful sentence completion framework.

References

• Gender Bias Evaluation for Masked Language modelling: HONEST
• xlm-roberta-base on Hugging Face
• HONEST: Measuring Hurtful Sentence Completion in Language Models

AVID Taxonomy Categorization

• Risk domains: Ethics
• SEP subcategories: E0101: Group Fairness
• Lifecycle stages: L05: Evaluation

Affected or Relevant Artifacts

• Developer:
• Deployer: HuggingFace
• Artifact Details:

  Type	Name
  Model	xlm-roberta-base

Other information

• Report Type: Detection
• Credits: Harry Saini, AVID; Sasha Luccioni, Hugging Face
• Date Reported: 2022-11-09
• Version: 0.1
• AVID Entry