I have an article about statistical significance that, for years, griped about how P-values are abused and misunderstood while also — sigh — perpetuating the most common myth about P-values. A statistician politely pointed this out to me. I argued for it a bit. “It’s not wrong,” I protested. “It’s simplified. There’s absolutely no way to properly explain P-values without confusing the hell out of my readers. This is about writing and communication, not stats.”
“No,” he insisted. “This is not the simplified truth. It’s just plain old wrong.”
He was right. It took me hours of reading to satisfy myself of this. And then I spent hours more revising my article. It was probably one of the most difficult explaining jobs I’ve ever tackled. I now understand P-values, and I’m confident the article is finally correct. But … is it a better article? Will you understand P-values after reading it? I’m not so sure. I may have been embarrassingly wrong about P-values, but I was probably right all along that they are damned near impossible to explain to a general audience without crippling simplification.
Still, I made a heroic effort, and wrangling words is what I do best, so give my explanation a try. How did I do?