AVID-2026-R0709

Description

Evaluation of the LLM rnj-1-instruct on the bold benchmark using Inspect Evals

Details

BOLD is a dataset designed to evaluate fairness in open-ended language generation in the English language. It consists of 23,679 different text generation prompts that allow fairness measurement across five domains: profession, gender, race, religious ideologies, and political ideologies.

The LLM rnj-1-instruct was evaluated on this benchmark.

Metrics

The evaluation calculates various social bias and affective metrics in generated responses, using specialized models:

• Toxicity Score: Measures the toxicity level in the model’s responses, using a Hugging Face-based toxicity classifier (i.e. Detoxify).

• Sentiment Score: Measures the general sentiment conveyed in the generated text using the VADER sentiment analyzer. The sentiment score ranges from −1 (most negative) to +1 (most positive). A score ≥ 0.5 indicates positive sentiment, while a score ≤ −0.5 indicates negative sentiment.

• Regard Score: Measures how positively or negatively a demographic group is portrayed in the generated text, based on the regard classifier trained on human-annotated samples for gender and race groups. A higher regard score indicates a more positive social perception of the target demographic.

For more details on the BOLD dataset and its applications, refer to the original paper: Bias in Open-ended Language Generation Dataset.

References

• Inspect Evaluation Log for dataset: bold

Affected or Relevant Artifacts

• Developer: Essential AI
• Deployer: Together AI
• Artifact Details:

Impact

• (none)

Other information

• Report Type: Measurement
• Credits:
• Date Reported: 2026-03-17
• Version: 0.3.2
• AVID Entry