Tuesday, September 30, 2008
Bad Reporting
This article-post on the Freakonomics blog was indicative of the poverty of reporting on psychological and economics findings. I found it dreadfully uninformative. How can it simply be said that 'chauvinists are more likely to earn more'? Could it not be that individuals who are successful end up as chauvinists, rather than that chauvinists end up successful? What were the other controls? What other variables did they consider? What did they rule out? Why did the article adopt causative language? In my opinion, there is probably so much going on in the formation of beliefs about 'gender' that just isolating one belief is basically arbitrary. I turned to the original article to find out more.
Barring the comment in the post on the fact that the sample was constituted of Americans, most of the subsequent phrasing makes it seem as though, in general, it 'pays' to be chauvinistic. This is not necessarily a general relationship as it is for a specific sample of US individuals. See my correlation-causation fallacy comment above.
Next point, in the paper the range of ages is 8 years (those 14-22 in 1979) because of the way the sample is constructed. Thus this is a measure of gender bias for this specific generation, but does not necessarily inform us as to the gender bias of later generations or age cohorts. Again, generalizing from this age cohort is perilous. Thus, the above shouldn't say 'Men' it should say, 'Men in the US who are currently between the ages of 43 and 51'. The Freakonomics post wasn't specific at all. The article was substantially more specific, though it too over-generalized on occasion.
Also, don't forget that the paper reports how, '[M]ore educated people and more intelligent individuals were less likely to have a traditional gender role orientation' (p.1002) and that 'individuals raised in religious households were more likely to have a traditional gender role orientation' (ibid). Which, to me, indicates that there are, quite possibly, problems of endogeneity in the model. What does this mean? Well, the consequence of this kind of thing is that the standard errors of the coefficients (the betas) are meaningless. Why? Well, if you have a variable which is meant to be measuring some effect, but there is a relationship between this variable and the error term in your regression (through bidirectional causality, simultaneity, or missing variables) then what you are actually measuring is the bias as a consequence of one of these effects. I, for one, do not believe that the index constructed in the paper actually identifies anything [see this wiki article on instrumental variables, 2SLS and identification for basic econometric protocols in this kind of situation].
Consequently, we do not know whether the coefficients are actually significant. What does this imply? For one, that the actual end result is meaningless - we don't know whether there is a positive and strong relationship involved because we could be measuring anything that is related to the index that they construct and the error term. What does this mean? Well, a host of other variables could actually be driving the higher income variable any of which could be itself correlated with chauvinism. So they are not identifying anything causative!
Now, why does this kind of reporting incense me? The main reason is that an uninformed and unskeptical chauvinist pratt could read the article and use it as justification for their own behaviour. They could internalize the idea that 'chauvinists earn more' (when we can't actually conclude anything of the sort from this research) and they walk away believing their chauvinism justified. Notwithstanding this, scientifically the article was a mess. It was unspecific, it overgeneralized and it misrepresented the outcomes from the article. It reminds me, once again, that so much education is needed for the populace at large in order to ensure that readers of such a post don't get dumbed down by it and read it at face value.
Currently have 0 comments:
Post a Comment