Adding Art to the Rigor Of Statistical Science
Converts Emerge for the Controversial Idea That Experience Must Temper New Data

By DAVID LEONHARDT

About a decade ago, in the moun tainous Grampian region of Scot land, a team of medical researchers began studying a drug meant to help people survive a heart attack. Doc tors would give the drug, called a clotbuster, to some patients before they had even reached the emergen cy room and then compare their sur vival rate with that of patients who had not taken the drug. The results were astonishing. In a paper published in The British Medi cal Journal in 1992, the researchers reported that the drug hadreduced the death rate of heart attack vic tims by about 50 percent. The sample size was small - about 300 patients - and the Grampian region was remote, meaning that it often took patients more than an hour to get to a hospital. Still, the results suggested that the clotbuster might be more effective than almost any similar drug to come before it. A pair of British statisticians were skeptical, however. "The amount of data wasn't that overwhelming," said Stuart Pocock, one of the pair and a professor at the London School of Hygiene and Tropical Medicine. "So there was a residual doubt: was it too good to be true?"

To test theiir doubt, Mr. Pocock and his colleague, David Spiegelhalter, reached back two and a half centu ries to a mathematical technique de vised by a Nonconformist minister named Thomas Bayes who died be fore his seminal paper was ever pub lished. Bayes's formula allows scien tists to combine new data with their prior beliefs about how the world works. It is an idea that amounts to heresy in much of the statistical world. After all, the method requires individuals to make subjective deci sions about how strongly to weigh prior beliefs. As a result, many scien tists say it sullies pure data with bias and outside information.

But its adherents, who call them selves Bayesians, argue that the formula is an extraordinarily powerful tool that permits scientists to keep new information in the proper per spective. If a study, even a statisti cally significant one suggests that Pigs can fly, Bayes's theorem allows researchers to combine the studys results mathematically with hun dreds of years of knowledge about the travel habits of swine.

Sure enough, when Mr. Pocock and Mr. SpiegethaIter analyzed the Grampian results in the context of other work on clotbusters, they found that the drug was likely to reduce mortality by about 27 per cent, not 50 percent. They sent a letter with their findings to The Brit ish Medical Journal, and, as is the case in many academic debates, the matter seemed to end there. Then, in May of last year, The Journal, of the American Medical As sociation published a review of many studies on the clotbuster. It conclud ed that the drug seemed to reduce death by about 17 percent - far closer to the professors' results than to the Grampian study's. Bayesians around the world re joiced. "It's a real-life example," said Robert A. J. Matthews, a physi cist at Aston University in Birminham England, Who gave a lecture On the Grampi an case this month in Vienna. "People could n't say, 'Well, You just altered your prior beliefs until they matched what we now know is the truth.,,,

The victory may have been a small one in the Struggle of Bayesian academics, but it came when Bayes was gaining a devoted following, particularly among many of the world's largest companies. With computers far more powerful than they were even a decade ago, the intricate calculations re quired for many Bayesian problems no longer seem so daunting. As a result, Pfizercoei, the pharmaceuti cal company, is melding existing knowledge about broad drug categories with results from, early animal tests to design optimal drug trials for human beings. Microsoftcoei has created a Bayesian computer program that can analyze incoming e-mail and phone calls to determine which are urgent and Which can be temporarily ignored.

And a small British company called Autonomy, whose founder wrote his his doctoral Thesis on Bayesian methods, sells software that helps computers intelligently sort through the vast amounts of information that large organizations now produce on any given day. Autonomy's clients include the United States Departments of Defense and Energy, General Motors, Procter & Gamble and the Knight-Ridder newspaper chain.

Even in academia - where the method remains the exception to the classical, or "frequentist," statistical approach - it has gained some footholds. Astronomers have used it to estimate what faraway stars look like. Some philosophy classes include units on Bayesian theories. And last year Boise State University opened a center for the study of Bayesian methods.

At the core of Bayes's long-delayed popu larity is an admission that the world is rife with uncertainty and often not suited to clean statistical tests. When repeated ex periments are not feasible, Bayesians be lieve, sweeping generalization should not rest on a small amount of new data. Instead, they want to bring to bear knowledge that people have accumulated over many years, assign a weight to each part of that knowl edge and then consider the new data.

When the new information is both strong and widespread, it overwhelms the old con sensus. But when a relatively small amount of data contradicts a wealth of prior knowl edge, the old consensus changes only slight ly. "What Bayes says is, 'You're making outrageous claims here - you're going to need outrageous evidence to back that up,'

Think of how. you might estimate the average weight of a potato. A Bayesian would combine a sample taken from a local vegetable stand with the expert opinions of food researchers. A frequentist statistician, on the other hand, would want to look only at the sample data - and reserve judgment unless the sample was large enough. "The Bayesian will say, 'I essentially have a larger sample,' and the non-Bayes ian will say, 'You have a biased result,", said Judea Pearl, a computer scientist at the University of California at Los Angeles.

A Bayesian might also be willing to stretch the limits of statistics and estimate the odds that Al Gore will run for.president again in 2004, said Michael 1. Jordan, a professor at the University of California at Berkeley. The Bayesian would study the behavior of past nominees who have lost close elections and would analyze Mr. Gore's recent fund-raising behavior.

The result of a Bayesian calculation is usually not a single number but a series of probabilities. "Part of the Bayesian spirit is that you always have uncertainty about everything," Mr. Jordan said, "and you always want to report your uncertainty."

In many ways, the approach is similar to a thought process that people use every day. When walking in a city, they decide when it is safe to cross a street based on the cars they see (the new data) and what they know about traffic flow (their prior beliefs). When fixing a broken refrigerator, ,they combine information about a leak they can see with their accumulated knowledge about what tends to malfunction in the appliance.

"Subconsciously, the human brain does a lot of this Bayesian processing without us realizing it," said Dr. Mike R. Lynch, the chief executive and founder of Autonomy.

To many scientists, however, the intuitive nature of Bayesian methods adds little to their appeal. In fact, the scientists say, by assigning mathematical weights to some body's subjective beliefs, Bayesians corrupt the data they are working with. "The fre quentist would say it's better to have no answer at all than one you can't have confi dence in, "said Richard Lowry, a psycholo gist at Vassar who is skeptical of Bayesian methods.

In this way, Bayes's theory is both conser vative and radical: conservative because it tends to cast doubt on unexpected findings and radical because it questions the ortho doxy that each experiment should stand on its own. And in its duality, the idea is a fitting testament to its inventor.

Born in London in 1702, Thomas Bayes followed his father, Joshua, into the minis try. Both were Nonconformists, and one of the few works Thomas published during his lifetime was a defense of Issac Newton against a bishop who had attacked the logic of his calculus, according to the Encyclopae dia Britannica.

Thomas Bayes was successful enough as a mathematician to win election to the Roy al Society of London, but he would be long forgotten today had it not been for a friend and fellow minister named Richard Price, who inherited Bayes's papers when he died in 1761. Mr. Price, himself famous for devis ing one of the first actuarial, tables, came across the "Essay Towards Solving a Prob lem in the Doctrine of Chances," and in 1763 the Royal Society published it.

Statisticians have long recognized the rule's importance, and some high school classes use it to solve straightforward prob ability problems. But once both data and beliefs enter the picture, the math can be come stupefyingly complex. Over the last 10 or 15 years, however, computers have be come powerful enough to handle Bayesian calculations with relative ease, and the method has won a following. "We now have the tools," said Dr. Andy P. Grieve, a senior statistical consultant at Pfizer. "We are able to use Bayesian methods."

Indeed, some of Bayses's biggest fans can, Me Dr. Grieve, be found in the corpo rite sector. For example, the NCP, Corpora tion, which makes A.T.M.'s and bar-code scanners, and a consortium of big banks run a research center in London that is trying to use Bayesian methods to figure out which loans or investments different types of peo ple are most likely to want.

Few companies are making a bigger push than Microsoft, which is trying to design software that would become a virtual secre tary. The software, now being tested, com bines facts about who has sent an e-mail message and the words it contains with information about a user's habits. The program then decides whether the user can wait a few minutes, or even hours, to see the message.

"We want to understand how to work with people in the way an insightful colleague would," said Dr. Eric Horvitz, who oversees a team of 20 researchers at Microsoft. "We're still a ways off from that, but that's the goal."

All this effort springs from an essay that is older not only than Microsoft or the computer itself but the United States or the harnessing, of electricity as well. As Dr. Lynch said, "It's taken 250 years for the rest of us to realize just how intelligent Thomas Bayes was."