Thursday, December 10, 2009

Does Advertising Skew Wine Spectator Wine Reviews?

We live in a world with many urban myths, so-called "truths" that are often accepted with a lack of any real evidence. It is always curious why such urban myths spread and are accepted. It may partially be due to such myths appealing to our prejudices.

As an example, some people have accused magazines like Wine Spectator of being biased towards wineries that advertise in their magazine. They seem to think that the large, full page ads cause reviewers to favor those wines. Yet these allegations are never supported by actual evidence. So what is the truth? Are the accuser simply prejudiced against Wine Spectator?

An actual study has now been conducted on this issue and the report has just been issued. Check out Does Advertising Bias Product Reviews? An Analysis of Wine Ratings by Jonathan Reuter (Journal of Wine Economics, Volume 4, Issue 2, Winter 2009, Pages 125–151). Reuter is a local person, working at the Department of Finance of the Carroll School of Management at Boston College.

To test this issue, Reuter compared two wine publications, Wine Spectator and Wine Advocate. As the Wine Advocate does not accept advertising, he felt it would be appropriate to compare its wine scores with those of the Wine Spectator, which does accept advertising. I think it might have been a better, and more comprehensive, study if Reuter had also compared the Wine Enthusiast, or other major wine magazines, which accept advertising.

His basic conclusion shatters the urban myth: "Overall, the tests for biased ratings and biased awards produce little consistent evidence that Wine Spectator favors advertisers." So will this myth die out? Probably not, as some prefer to believe this rather than confront the facts.

Taken in the worst light against Wine Spectator, the study suggested that wines from advertisers may score almost one point higher than wines from nonadvertisers, as compared to the wine scores in the Wine Advocate. But Reuter also stated this "..is also consistent with the two publications evaluating wine using different standards, perhaps because they cater to different consumer tastes." There are also other reasons why that might be the case. In addition, less than one point certainly is not a very significant difference, and could well be in an acceptable margin of error.

Though some people may try to cling to this less than one point difference to further the urban myth, they are holding onto something very shaky, as well as ignoring numerous other factors. It is significant that Reuter found that "..,Wine Spectator is no more likely to bestow awards upon advertisers." This helps support the conclusion that advertising does not bias Wine Spectator reviews.

This might also be a wake-up call to advertisers who felt they were getting better reviews based on all of their advertising dollars. Their money was not buying them anything extra.

Let us continue to shatter wine's urban myths!

3 comments:

Anonymous said...

Very eye opening story and test.

Josh

The Wine Whore said...

I agree that advertising doesn't necessarily influence reviews but what about lists like the Top 100 of WS? I always find it funny when the wines in the top 20 also have paid ads in the magazine... since these lists aren't based solely on scores, it just seems like there's a lot of "fudge" room.

Anonymous said...

Thank you for your fair-minded look at this study.

For more background, I'd like to point out that it is based on reviews of only 700 wines from the year 2000, and that advertised wines actually received lower scores from Wine Spectator than non-advertised wines. In addition, advertised wines actually received higher scores from Wine Advocate than from Wine Spectator.

Founded in 1976, Wine Spectator has become the most widely-read wine publication in the world, according to independent research, with more than 2.4 million readers in the U.S. (MRI, Fall 2009). That success results from the credibility Wine Spectator has earned from readers, which, in turn, stems from their belief in the publication’s expertise and integrity.

The integrity rests on our Code of Ethics and our tasting methodology (both posted in detail on our Web site, WineSpectator.com). At Wine Spectator, we will continue to uphold the highest standards of truth and fairness, and to work diligently to avoid any bias or conflicts of interest.

Thomas Matthews
Executive editor
Wine Spectator