Conclusions about effect of avatar and gender bias

This is a very interesting study and I really like the idea of using GitHub data. I have a question regarding the effect of gendered vs gender-neutral avatars. One of your conclusions is that female outsiders experience gender bias. However, this conclusion seems to rest on the assumption that female coders with gendered avatars are as competent as female coders with gender neutral avatars. Did you somehow test this assumption? This is really important because it is possible that outsiders with gendered avatars are simply less competent and in this case the lower acceptance rates would simply reflect that objectively.

I think your data gives us reason to believe that this alternative explanation is correct. You say (page 16): “Women have lower acceptance rates as outsiders when they are identifiable as women.” However, Fig. 5 shows that the same thing is true for men. Outsider men with gendered avatar had lower acceptance rates than outsider men with gender-neutral avatars. So gendered avatars had a negative effect on acceptance regardless of sex. Now, you could argue that the effect of avatar was bigger in women than in men and that this reflects gender bias but given your confidence intervals I doubt that this difference is significant.

I think there is also another problem with your logic. When we look at the data for insiders, we see that the avatar had no effect for women but it did have an effect for men. Applying your logic, we'd have to conclude that there is gender bias against insider men. And this evidence is actually stronger than the evidence that you provide for anti-female gender bias because insider women did not show any effect of avatar, only men did.

I would be interested to hear your thoughts on that.

waiting for moderation
1 Answer
Accepted answer

Great point about the assumption: yes, it does assume that (for outsiders) gender-neutral and gendered women are similar. We did not validate this assumption. Your suggestion about doing it by comparing competence, via, for instance, lines of code in the average accepted PR, is an excellent one. We’ll add it to our tasks for the next draft! This might help shed some light on some of the other interesting parts of Figure 5, as you point out.

waiting for moderation

Longitudinally, one could possibility that the kind of person (man or woman) who reveals their gender makes worse PRs by observing users who update their profiles (e.g. profile pics) and seeing if their acceptance rates show a (relative) drop.

Also, as Titus notes, you do not have an explicit test for your claim that "For outsiders, we see evidence for gender bias: women’s acceptance rates are 71.8% when they use gender neutral profiles, but drop to 62.5% when their gender is identifiable. There is a similar drop for men, but the effect is not as strong". While the effect is less strong in your sample, you have to present an inferential test for the interaction you claim. See http://www.ncbi.nlm.nih.gov/pubmed/21878926

Note here the difference in sample size: the confidence interval for men who reveal their gender is much smaller.

waiting for moderation