Newspaper takes issue with statistics in Star AIDS series
By Mark Azzara
© 2000 Republican-American
Editor's note: The following story appeared on the front page of the Feb. 4 edition of the Waterbury (Conn.) Republican-American daily newspaper. It is reprinted with permission.
WATERBURY, Conn. - Roman Catholic priests may not be dying of AIDS at a rate four times the national average after all, polling experts say.
The extraordinary death rate for priests was among the findings reported in a Kansas City Star series, "AIDS in the Priesthood." The series appeared nationally Sunday.
Since then, nationally known polling experts contacted by The Republican-American have raised serious questions about the methods of The Star's four-year investigation and a survey it conducted of American priests.
Mark Zieman, editor and vice president of The Star, said Thursday that the newspaper and its editors "fully stand by our findings."
On Wednesday the Center for Media and Public Affairs challenged the death-rate claim.
The center said on its Web site - www.newswatch.org - that data from the 1998 Statistical Abstract of the United States puts the AIDS-related death rate among adult males at about 4 per 10,000. That is the same rate The Star estimated among priests, who are all adult males.
The Star had said the AIDS-related death-rate among priests was four times higher than that of the "general population" - which, the center said, would indicate that The Star included the death rates for women and children, rather than comparing death rates of adult males.
Other questions also have emerged.
The Star's claim of the higher-than-normal death rate was based on an estimate of at least 300 AIDS-related deaths among priests. The estimate was derived from interviews with medical experts and priests and from an analysis of health statistics.
Tom W. Smith, director of the General Social Survey at the National Opinion Research Center at the University of Chicago, said the data appears to be compared to public statistics, which may have been compiled differently.
Smith said that when information is not collected in the same way, "you can't do a legitimate comparison. This is a classic apples-and-oranges thing," he said. "The most charitable thing to say is there is not enough information on their method to know if they're right or wrong."
Take, for example, death certificates.
The Star examined hundreds in several states. It found instances when causes other than AIDS were listed for priests even when AIDS was the real cause. However, the series made no claim to have done a similar study for all deaths in those states - even though priests might not be the only ones who would seek to conceal AIDS as the cause of death.
Additionally, Smith said The Star's death-rate statistics for priests may not have been based on accurate base numbers.
The number of priests was in excess of 58,000 until the 1990s, according to the Official Catholic Directory, but Smith suspects the priests' death rate for various years may have been calculated against the present census of 46,352 priests as counted nationally. If the smaller number were used, the death rate could be artificially inflated.
Zieman, responding for The Star to questions raised by The Republican-American, said readers and potential critics should consider the broader picture:
"The main point of our series - that hundreds of priests have died of AIDS and hundreds living believe the Church can help address this issue - does not seem to be addressed at all in your questions," he said. "I mention it only because I believe those points are unassailable and I wouldn't want them to get lost in your story. That would be a great disservice not only to our staff, but to the many people who are quoted in the series and shared their personal stories with The Star."
THE SURVEY, Zieman said, "was meant only to show how priests themselves felt about the issue, not to support numbers from AIDS researchers, death certificates or additional research done by The Star."
The Star sent its questionnaire to 3,013 priests randomly in all 50 states. It received 801 replies - a 27 percent response.
The survey reported that 15 percent of replying priests were homosexual and 5 percent were bisexual.
The Star included a caveat, saying the paper could not "ensure that the priests responding are demographically and geographically representative of all Roman Catholic priests. The priests who chose to respond ... may be different from those who opted not to reply."
Under the circumstances, Smith said, the newspaper was "not as responsible as it should be" in publishing the survey's results.
"They kind of say 'We don't know if it's representative,' but then they go on and report the numbers as if it is representative," he said.
Several experts said the survey may have skewed results by encouraging a higher-than-average response from homosexuals.
Smith said gay priests may have seen the survey as an opportunity "to speak out about their role - their ambiguous and marginal role in the church."
Said Mary Gautier, senior research associate at the Center for Applied Research in the Apostolate: "There is a pretty good probability of a higher response rate from homosexuals."
Polling experts interviewed by The Republican-American were unanimous in saying that The Star's 27-percent response rate was low for a mailed survey, and that the results appear to be based on a single mailing. "If you mail it out once, only the most motivated are going to reply," Smith said.
If The Star had sent out a second or third letter, it would have gotten a higher response that would have lessened the likelihood of bias, said Michael Traugott, professor and research scientist at the Institute for Social Research at the University of Michigan and president of the American Association for Public Opinion Research.
But Richard Williams, associate professor and chairman of the sociology department at the University of Notre Dame, said good follow-up efforts could have undermined the newspaper's promise of anonymity.
Zieman, of the Star, said the poll elicited a satisfactory response.
"Every poll has its limitations, and our intent was to follow regularly accepted polling methods, but not to track down every priest and force them to respond to what is admittedly a very private issue," Zieman said.
"The fact that so many did respond is actually remarkable and very high for such a survey; if you or your sources disagree, then you're free to share your facts with your readers as we have shared ours."
The Rev. Lawrence F. Carew of Trumbull was among the 3,013 priests who received the survey, but he threw it out.
"When I read it over it seemed like it was engineered to come out with a predetermined conclusion that this was a massive problem," said Carew, a priest who has counseled troubled clergy as founder of the Healing the Shepherds retreats, and who is spiritual director of the Greater Danbury Christian Counseling Center.
Carew said he's not surprised by the results.
"My reaction was they got just what they were looking for, and I feel like it's very distorted and way out of proportion to what the truth probably is," he said. "I just felt really sad because I felt like the people were getting hit with a piece of propaganda rather than an objective survey."
The Star said in its series that the responses, based on a comparison of diocesan vs. religious-order priests and the age range of priests, is similar to the national figures provided by Gautier's organization.
But Smith said that, with only two standard subgroupings to rely on for comparison, the other numbers in the survey are "weakly, if at all, correlated" to who Catholic priests are in general.
FURTHERMORE, because there is no indication of the geographical location of the respondents, there's no way of knowing the origin of responses.
"Gays are not randomly distributed across the country. They are overly concentrated in big cities," Smith said.
Traugott raised similar concerns.
"What you know from what they tell you looks like the population (of priests) on the basis of age and assignment," Traugott said.
But, he said, there are no existing studies against which to compare The Star's survey. As a result, the conclusions are "a matter of inference. It seems plausible, but you can't be certain" because of the possibility of a biased response and the low response rate.
The wording of the questions also raised questions about whether the answers were valid.
For example, priests were asked if they were heterosexual, homosexual or bisexual. Smith said "bisexual" is not used by his organization because "we find people don't correctly understand the meaning of that term." Smith said the 2 percent who said "other" in identifying their sexual orientation is a fairly high number, and he said other terms (asexual or non-sexual, for instance) should have been used.
Smith and Traugott said the wording of the Star's cover letter could have played a significant role in determining how - or even if - priests answered.
They said they were concerned about the letter's tone and whether it suggested an agenda. Traugott said because the letter came from a news organization, it might prompt priests not to respond.
Zieman said the limitations his newspaper faces in wading through "more than 1,000 e-mails, phone calls and letters from around the world," in addition to inquiries from other news organizations, prevented him from answering all of the questions posed by The Republican-American Thursday.
Randy Smith, assistant managing editor of The Star, said the methodology and objectives of the study would be addressed in a future article. The series has appeared on The Star's Web site, www.kcstar.com.
Other concerns raised by experts contacted by the Republican-American included:
- Survey results were listed as percentages without whole numbers.
Smith said there's no way to know whether large numbers of priests may have skipped one or more questions, for reasons known only to them - possibly because priests were offended by questions about their sexual orientation. If just 10 percent refused to answer a question "that could heavily shift the result," Smith said.
- One question asked of the priests did not use the same wording in the results.
The Star reported answers to the question: "How was sexuality addressed during your theological training?" But priests were actually asked, "In your religious theological training, how do you feel that sexuality was addressed as an integral part of your psycho/social/spiritual being?"
Smith said it was "bad form" not to report questions exactly as asked. Traugott questioned the answers because each respondent had to define the word "integral" and the term "psycho/social/spiritual being."
- There was no breakdown of results by the two standard subgroups (assignment and age).
Traugott suggested that younger priests might have answered questions differently than older priests, and diocesan priests might have responded differently than those in religious orders. That would, among other things, be a quick way to determine if younger priests received a better education on sexuality.
Traugott said the survey demands a lot from older priests, who were expected to recall details of seminary training after decades.
- The margin of error might be significantly higher than The Star reports.
The Star said that, given the sample size (801 responses out of 3,013 surveys mailed), "the poll's margin of error is 3.5 percentage points, meaning that if the same poll were conducted 100 times, 95 of those times the results would be no more than 3.5 percentage points higher or lower." Traugott said standard polling rules for using that error rate require a high response rate, which wasn't found in The Star's survey.
"The sample size is too small, so it probably under represents the error rate. But there's no way to calculate," he said.