Saturday, December 25, 2010

Resolving Debates in Statistics

Occasionally there are debates in science about what models are formally appropriate for thinking reasoning about scientific concepts. These include:

- The existence of race as a meaningful concept in light of current knowledge about human genetic diversity. See Richard Lewontin's article "The apportionment of human diversity" (1972) and Lewontin's Fallacy for more information.
- Quantum and classical physics.
- Bayesian and Frequentist methods for statistical inference.

Scientists have chosen sides in these debates based on education, philosophical preference, and sometimes practical concerns. Usually these debates can be reduced down to the appropriateness of certain calculations in solving scientific problems. For example the Bayesian-Frequentist debate largely centers on the incorporation of prior information into inferential calculations. In "Confidence Intervals vs Bayesian Intervals," by ET Jaynes suggests that all of these debates over methods can be resolved by benchmarking methods on a set of focused problems:

I suggest we apply the same criterion in statistics: the merits of any statistical method are determined by the results it gives when applied to specific problems. 

Earlier in the same piece:

This is an exciting time in physics, because recent advances in technology (lasers, fast computers, etc.) have brought us to the point where issues which have been debated fruitlessly on the philosophical level for 45 years, are at last reduced to issues of fact, and experiments are now underway testing controversial aspects of quantum theory that have never been accessible to direct check. We have the feeling that, very soon now, we are going to know the real truth, the long debate can end at last, one way or another; and we will be able to turn a great deal of energy to more constructive things. Is there any hope that the same can be done for statistics? 
I think there is, and history points the way. It is to Galileo that we owe the first demonstration that ideological conflicts are resolved, not by debate, but by observations of fact. But we also recall that he ran into some difficulties in selling this idea to his contemporaries. Perhaps the most striking thing about his troubles was not his eventual physical persecution, which was hardly uncommon in those days; but rather the quality of logic that was used by his adversaries. For example, having turned his new telescope to the skies, Galileo announced discovery of the moons of Jupiter. A contemporary scholar ridiculed the idea, asserted that hsi theology had proved there could be no moons around Jupiter; and steadfastly refused to look through Galileo's telescope. But to everyone who did take a look, the evidence of his own eyes somehow carried more convicing power than did any amount of theology. 

I find his take on this matter very compelling. Also, I think that similar debates rage in other fields of study, such the Editor War between Emacs and VI in computer programming.