I heard on Twitter today about a paper stating that after the Snowden leaks, people looked up terrorism-related topics less on Wikipedia.
I'm not worried about deliberate fabrication of data, but I'm worried that poor quality statistical analysis may have happened. I'm not expert enough to look at the paper's statistical analyses myself and determine whether they're appropriate, so instead I want to know whether rigorous peer review has taken place.
The journal involved, the Berkeley Technology Law Journal, doesn't seem to mention peer review either on its about page or the page about the journal. The page about submissions says "Evaluation involves scoring by multiple different editors, and rarely lasts fewer than 3-4 weeks.", which suggests that it's evaluated only by editors, not by academic peers. Also, evaluation and scoring sounds different to what I'd expect to happen in peer review - it sounds like they're only evaluating whether it's a good fit for the journal and of enough interest.
How do I determine if a paper has undergone anonymous peer review of its statistical analysis, short of contacting the editor of the journal?
Related question: How do I recognise a peer-reviewed journal article? but the answers seem to focus on science related subjects, as opposed to the humanities.
Let's tease apart two issues here:
- Has a paper undergone peer review of its technical content (which will include the statistical content)?
- Should you believe the statistics in the paper?
If it's in a well-respected peer-reviewed journal, then you can assume that it's undergone reasonable peer review. That means it's unlikely that the content is blatantly and obviously crazy, and that in fact it's likely to be fairly solid. It does not mean, however, that it is correct, particularly when it comes to something as subtle and frequently mis-applied as statistical analysis.
Bad statistical analysis is extremely common in every field of science, for a number of different reasons. Ignorance is a really big one, though, and one that's likely to be shared by the peer reviewers. Thus, if you want to know whether to trust statistical analysis, you need to look at it yourself.