1
Conclusions from figure 5 seem inconsistent
Viewed 182 times

Figure 5 shows the acceptance rate of Github pull requests based on:

  1. Whether the person was an "insider" vs an "outsider" to the project
  2. Whether the person was actually a man or a woman (based on linkages between github information and publicly-available social media information)
  3. Whether the github account had gender-identifiable information in it (gender-neutral vs gendered).

You conclude that "For insiders, we observe little evidence of bias when we compare women with gender-neutral profiles and women with gendered profiles, since both have about equivalent acceptance rates" but "For outsiders, we see evidence for gender bias: women’s acceptance rates are 71.8% when they use gender neutral profiles, but drop to 62.5% when their gender is identifiable."

However, you don't seem to address something important: for outsiders, men's acceptance rates also drop when using gendered profiles, from about 68% (gender-neutral) to 64% (gendered). (Eyeballing the graph, since you don't include hard figures.)

In the "gendered outsider" category, women indeed have a 62% acceptance rate vs a 64% acceptance rate for men. But in the "gendered insider" category, women have what looks like a 88% acceptance rate, vs 86% for men.

If the 62% ~ 64% difference merits the conclusion that "Women have lower acceptance rates as outsiders when they are identifiable as women", doesn't the 88% ~ 86% difference merit the conclusion "Women have higher acceptance rates as insiders when they are identifiable as women"?

Or alternately, if the 88% ~ 86% difference merits the conclusion, "For insiders, we observe little evidence of bias", doesn't the 62% ~ 64% difference also merit the conclusion, "For outsiders, we observe little evidence of bias"?

Given just the data you present here, it looks to me like the correct conclusion is, "We observe little evidence of bias either for insiders or outsiders."

waiting for moderation