Home › 2021 › November

Month: November 2021

By Posted on Posted in Sex No Comments

The Journal of Race, Ethnicity, and Politics published Nelson 2021 "You seem like a great candidate, but…: Race and gender attitudes and the 2020 Democratic primary".

Nelson 2021 is an analysis of racial attitudes and gender attitudes that makes inferences about the effect of "gender attitudes" using measures that ask only about women, without any appreciation of the need to assess whether the effect of gender attitudes about women are offset by the effect of gender attitudes about men.

But Nelson 2021 has another element that I thought worth blogging about. From pages 656 and 657:

Importantly, though, I hypothesized that the respondent's race will be consequential for whether these race and gender attitudes matter—specifically, that I expect it is white respondents who are driving these relationships. To test this hypothesis, I reran all 16 logit models from above with some minor adjustments. First, I replaced the IVs "Black" and "Latina/o/x" with the dichotomous variable "white." This variable is coded 1 for those respondents who identify as white and 0 otherwise. I also added interaction terms between the key variables of interest—hostile sexism, modern sexism, and racial resentment—and "white." These interactions will help assess whether white respondents display different patterns than respondents of color...

This seems like a good research design: if, for instance, the p-value is less than p=0.05 for the "Racial resentment X White" interaction term, then we can infer that, net of controls, racial resentment associated with the outcome among White respondents differently than racial resentment associated with the outcome among respondents of color.

---

But, instead of reporting the p-value for the interaction terms, Nelson 2021 compared the statistical significance for an estimate among White respondents to the statistical significance for the corresponding estimate among respondents of color, such as:

In seven out of eight cases where racial resentment predicts the likelihood of choosing Biden or Harris, the average marginal effect for white respondents is statistically significant. In those same seven cases, the average marginal effect for respondents of color on the likelihood of choosing Biden or Harris is insignificant...

But the problem with comparing statistical significance for estimates is that a difference in statistical significance doesn't permit an inference that the estimates differ.

For example, Nelson 2021 Table A5 indicates that, for the association of racial resentment and the outcome of Kamala Harris's perceived electability, the 95% confidence interval among White respondents is [-.01, -.001]; this 95% confidence interval doesn't include zero, so that's a statistically significant estimate. The corresponding 95% confidence interval among respondents of color is [-.01, .002]; this 95% confidence interval includes zero, so that's not a statistically significant estimate.

But the corresponding point estimates are reported as -0.01 among White respondents and -0.01 among respondents of color, so there doesn't seem to be sufficient evidence to claim that these estimates differ from each other. Nonetheless, Nelson 2021 counts this as one of the seven cases referenced in the aforementioned passage.

Nelson 2021 Table 1 indicates that the sample had 906 White respondents and 466 respondents of color. The larger sample for Whites than respondents of color biases the analysis toward a better chance of detecting statistical significance among White respondents than among respondents of colors.

---

Table A5 provides sufficient evidence that some interaction terms had a p-value less than p=0.05, such as for the policy outcome for Joe Biden, with non-overlapping 95% confidence intervals for hostile sexism of [-.02, .0004] for respondents of color and [.002, .02] for White respondents.

But I'm not sure how much this matters, without evidence about how well hostile sexism measured gender attitudes among White respondents, compared to how well hostile sexism measured gender attitudes among respondents of color.

Tagged with: , ,

By Posted on Posted in Sex No Comments

PLOS ONE recently published Gillooly et al 2021 "Having female role models correlates with PhD students' attitudes toward their own academic success".

Colleen Flaherty at Inside Higher Ed quoted Gillooly et al 2021 co-author Amy Erica Smith discussing results from the article. From the Flaherty story, with "she" being Amy Erica Smith:

"When we showed students a syllabus with a low percentage of women authors, men expressed greater confidence than women in their ability to do well in the class" she said. "When we showed students syllabi with more equal gender representation, men's self-confidence declined, but women and men still expressed equal confidence in their ability to do well. So making the curriculum more fair doesn't actually hurt men relative to women."

Figure 1 of Gillooly et al 2021 presented evidence of this male student backlash, with the figure note indicating that the analysis controlled for "orientations toward quantitative and qualitative methods". Gillooly et al 2021 indicated that these "orientation" measures incorporate respondent ratings of their interest and ability in quantitative methods and qualitative methods.

But the "Grad_Experiences_Final Qualtrics Survey" file indicates that these "orientation" measures appeared on the survey after respondents received the treatment. And controlling for such post-treatment "orientation" measures is a bad idea, as discussed in Montgomery et al 2018 "How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It".

The "orientation" items were located on the same Qualtrics block as the treatment and the self-confidence/self-efficacy item, so it seems possible that these "orientation" items might have been intended as outcomes and not as controls. I didn't find any preregistration that indicates the Gillooly et al plan for the analysis.

---

I used the Gillooly et al 2021 data to assess whether there is sufficient evidence that this "male backlash" effect occurs in straightforward analyses that omit the post-treatment controls. The p-value is about p=0.20 for the command...

ologit q14recode treatment2 if female==0, robust

...which tests the null hypothesis that male students' course-related self-confidence/self-efficacy as measured on the five-point scale did not differ by the difference in percentage of women authors on the syllabus.

See the output file below for more analysis. For what it's worth, the data provided sufficient evidence at p<0.05 that, among men students, the treatment affected responses to three of the four items that Gillooly et al 2021 used to construct the "orientation" controls.

---

NOTES

1. Data. Stata code. Output file.

2. Prior post discussing a biased benchmark in research by two of the Gillooly et al 2021 co-authors.

3. Figure 1 of Gillooly et al 2021 reports 76% confidence intervals to help assess a p<0.10 difference between estimates, and Figure 2 of Gillooly et al 2021 reports 84% confidence intervals to help assess a p<0.05 difference between estimates. I would be amazed if this p=0.05 / p=0.10 variation was planned before Gillooly et al analyzed the data.

Tagged with: , , , ,

By Posted on Posted in Sex No Comments

PS: Political Science & Politics published Utych 2020 "Powerless Conservatives or Powerless Findings?", which responded to arguments in my 2019 "Left Unchecked" PS symposium entry. From Utych 2020:

Zigerell (2019) presented arguments that research supporting a conservative ideology is less likely to be published than research supporting a liberal ideology, focusing on the most serious accusations of ideological bias and research malfeasance. This article considers another less sinister explanation—that research about issues such as anti-man bias may not be published because it is difficult to show conclusive evidence that it exists or has an effect on the political world.

I wasn't aware of the Utych 2020 PS article until I saw a tweet that it was published, but the PS editors kindly permitted me to publish a reply, which discussed evidence that anti-man bias exists and has an effect on the political world.

---

One of the pieces of evidence for anti-man bias mentioned in my PS reply was the Schwarz and Coppock meta-analysis of candidate choice experiments involving male candidates and female candidates. This meta-analysis was accepted at the Journal of Politics, and Steve Utych indicated on Twitter that it was a "great article" and that he was a reviewer of the article. The meta-analysis detected a bias favoring female candidates over male candidates, so I asked Steve Utych whether it is reasonable to characterize the results from the meta-analysis as reasonably good evidence that anti-man bias exists and has an effect in the political realm.

I thought that the exchange that I had with Steve Utych was worth saving (archived: https://archive.is/xFQvh). According to Steve Utych, this great meta-analysis of candidate choice experiments "doesn't present information about discrimination or biases". In the thread, Steve Utych wouldn't describe what he would accept as evidence of anti-man bias in the political realm, but he was willing to equate anti-man bias with alien abduction.

---

Suzanne Schwarz, who is the lead author of the Schwarz and Coppock meta-analysis, issued a series of tweets (archived: https://archive.is/pFSJ0). The thread was locked before I could respond, so I thought that I would blog about my comments on her points, which she labeled "first" through "third".

Her first point, about majority preference, doesn't seem to be relevant about whether anti-man bias exists and has an effect in the political realm.

For her second point, that voting in candidate choice experiments might differ from voting in real elections, I think that it's within reason to dismiss results from survey experiments, and I think that it's within reason to interpret results from survey experiments as offering evidence about the real world. But I think that each person should hold no more than one of those positions at a given time.

So if Suzanne Schwarz doesn't think that the meta-analysis provides evidence about voter behavior in real elections, there might still be time for her and her co-author to remove language from their JOP article that suggests that results from the meta-analysis provide evidence about voter behavior in real elections, such as:

Overall, our findings offer evidence against demand-side explanations of the gender gap in politics. Rather than discriminating against women who run for office, voters on average appear to reward women.

And instead of starting the article with "Do voters discriminate against women running for office?", maybe the article could instead start by quoting language from Suzanne Schwarz's tweets. Something such as:

Do "voters support women more in experiments that simulate hypothetical elections with hypothetical candidates"? And should anyone care, given that this "does not necessarily mean that those voters would support female politicians in real elections that involve real candidates and real stakes"?

I think that Suzanne Schwarz's third point is that a person's preference for A relative to B cannot be interpreted as an "anti" bias against B, without information about that person's attitudinal bias, stereotypes, or animus regarding B.

Suzanne Schwarz claimed that we would not interpret a preference for orange packaging over green packaging as evidence of an "anti-green" bias, but let's use a hypothetical involving people, of an employer who always hires White applicants over equally qualified Black applicants. I think that it would be at least as reasonable to describe that employer as having an anti-Black bias, compared to applying the Schwarz and Coppock language quoted above, to describe that employer as "appear[ing] to reward" White applicants.

---

The Schwarz and Coppock meta-analysis of 67 survey experiments seems like it took a lot of work, was published in one of the top political science journals, and, according to its abstract, was based on an experimental methodology that "[has] become a standard part of the political science toolkit for understanding the effects of candidate characteristics on vote choice", with results that add to the evidence that "voter preferences are not a major factor explaining the persistently low rates of women in elected office".

So it's interesting to see the "doesn't present information about discrimination or biases" and "does not necessarily mean that those voters would support female politicians in real elections that involve real candidates and real stakes" reactions on Twitter archived above, respectively from a peer reviewer who described the work as "great" and from one of the co-authors.

---

NOTES

1. Zach Goldberg and I have a manuscript presenting evidence that anti-man bias exists and has a political effect, based on participant feeling thermometer ratings about men and about women in data from the 2019 wave of the Democracy Fund Voter Study Group VOTER survey. Zach tweeted about a prior version of the manuscript. The idea for the manuscript goes back at least to a Twitter exchange from March 2020 (Zach, me).

Steve Utych reported on the 2019 wave of this VOTER survey in his 2021 Electoral Studies article about sexism against women, but neither his 2021 Electoral Studies article or his PS article questioning the idea of anti-man bias reported results from the feeling thermometer ratings about men and about women.

Tagged with: ,