Significance Testing – Three Concerns


Some words of caution about significance testing by Kevin Gray:

“I’ve long had three major concerns about significance testing. First, it assumes probability samples, which are rare in most fields. For example, even when probability sampling (e.g., RDD) is used in consumer surveys, because of (usually) low response rates, we don’t have true probability samples. Secondly, it assumes no measurement error. Measurement error can work in mysterious ways but generally weakens relationships between variables. Lastly, like automated modeling, it passes the buck to the machine and provides a pseudo-scientific excuse for not thinking. I think researchers should think.”