Unmasking Implicit Bias: Evaluating Persona-Prompted LLM Responses in Power-Disparate Social Scenarios

Hide Additional Info ×

About This Visualisation & Study

This tool explores potential implicit biases based on this paper. LLMs responded in simulated social scenarios as a Responder ('Blake') to a Subject ('Alex'), assigned different demographic identities (e.g., Race).

Metrics Explored:

• Cosine Distance (Sensitivity): Measures semantic change vs. baseline. Calculated as 1 - cosine_similarity(demog_prompted_response, non_demog_prompted_response). Higher values = greater semantic distance from non-demographically-prompted response. Demographic combinations leading to low-cosine-distance responses are are considered an LLM's 'default'.
• Win Rate (Quality): Perceived quality vs. baseline (judged by LLM on Helpfulness, Honesty, Harmlessness).
  • 1: Demog. response preferred.
  • 0.5: Draw.
  • 0: Non-Demog. Baseline preferred.
  Values > 0.5 suggest comparatively better quality with demog.; values < 0.5 suggest a comparative drop. Different demographic combinations produce responses with varying win rates and hence varying quality.

How to Use:

1. Use dropdowns for Model, Metric, and Power Disparity.
2. Optionally explore Scenarios, Demographics, Overall Findings.
3. Heatmap shows average metric for Subject (Y-axis) vs Responder (X-axis) demographics.
4. Hover cells for values/counts. Click cells for detailed examples pop-up.
5. The table above the controls dynamically summarises heatmap extremes.

Hide Findings ×

Key Findings From The Paper

These are general trends observed across the models and scenarios studied:

• Default Persona Bias: LLM responses often implicitly lean towards a "default persona" resembling a middle-aged, able-bodied, native-born, Caucasian, atheistic male with centrist political views. This means responses might change less (lower Cosine Distance) when personas match this default.
• Response Quality Variation: Interactions involving certain specific demographics were associated with comparatively lower-quality responses (lower Win Rate vs. baseline).
• Impact of Power Disparity:
  • The presence of power disparities between the Subject (SUB) and Responder (RES) tends to increase the variability in both response meaning (Cosine Distance) and quality (Win Rate) across different demographic groups.
  • This suggests that implicit biases might be more pronounced or activated under conditions of unequal power.

Note: These are general findings based on the study's overall dataset. Explore the heatmap and specific interpretations below for model-specific details under different conditions.