Shoulder Tapping

Let's say that for some reason Justin Bieber picks up the habit of tapping people on the shoulder. Then, of course, in a few days every kid in America is tapping everyone they see on the shoulder. The next logical step is some parents get concerned about the health effects of shoulder tapping and make sure that scientists studying shoulder tapping get funded.

Let's say that 200 studies are started. Each study picks a different health effect that seems like it could be connected (impact on bone structure, passing of germs, social anxiety...) and checks if it is correlated with shoulder tapping. These are real scientists who know what they are doing so they properly account for external factors and other correlations and thanks to all the funding are able to get such a large sample size that each study has a confidence level of 99%.

Then a journalist or study of studies scientist comes along and asks every scientist what they found. Turns out that with 99% confidence shoulder tapping causes peripheral vision loss.

There's no theory why shoulder tapping causes peripheral vision loss, but if there's a study showing a chance it is true then best to try to kill the new trend right?

Well, actually, even if there is no correlation between shoulder tapping and any of the studied health effects, the odds that 1 or more studies out of 200 would show a correlation with 99% confidence is 87%. (1-0.99^200). It is actually rather likely that there would be a false positive (even with only 69 studies there's a 50% chance of false positive, and with only 10 studies there's a 10% chance of false positive, much smaller, but still 10x worse than the quoted 99% confidence level).

While the individual studies were done with proper statistics the news story/study of studies was actually done with very poor statistics. Now if a scientist read the news article and then repeated the study of peripheral vision loss and again found a correlation with 99% confidence that would be worth reporting. Of course there would still be some question as to causation, but at least it would be true that there is very likely a correlation.