The reliability of user–generated hotel reviews

July 06, 2011 | Hotel Marketing

Mail article Print article Facebook Twitter

If online review sites were both unreliable and mistrusted, they would quickly become irrelevant. Even if online review sites were trusted by consumers but were significantly unreliable, they would not provide any value to customers and would quickly lose their audience. A new study shows now that neither is the case.

As online reviews have grown in popularity, attacks regarding their authenticity and trustworthiness have increased. TripAdvisor, the leading online review site for hotels, has been criticized for allowing reviews to be posted by anyone about any hotel, without needing supporting evidence. In an attempt to evaluate the legitimacy of these concerns, we compared results from TripAdvisor reviews with the Market Metrix Hospitality Index (MMHI), a well known hotel customer satisfaction panel in operation since 2001. Customer evaluations for 67 hotels were directly compared using TripAdvisor and MMHI data for 12 months. The results indicated that these sources provide consistent results by property although more variability was identified in the TripAdvisor reviews. This study should help address questions raised about the credibility of user–generated reviews and review sites.

Several years ago TripAdvisor partnered with Market Metrix to create a customer satisfaction scoreboard designed for the needs of hotel managers. The key metric used is the Customer Satisfaction Index (CSI), which is a new score that rates each TripAdvisor review from 0–100 based on 7 questions from the TripAdvisor survey. The CSI score makes it easy to compare results to local benchmark sets and determine which reviews require immediate action. Different from TripAdvisor's Popularity Index, the CSI score allows reviews to be compared to any competitors and trended over time. A satisfaction scoreboard for each hotel is displayed on the property's hotel manager page on TripAdvisor.com. For this research study the CSI score was the metric used for TripAdvisor Reviews.

The hotels used in this comparison were selected on the basis of their location, brand and type to achieve a diverse mix. Each hotel also must have received a sufficient number of responses. Statistical analysis was conducted to compare the similarity of scores for each hotel, the distribution of the data, and the change in scores.

The most significant conclusion of this study is that the mean scores of hotels track very consistently and closely between TripAdvisor (CSI) and MMHI. This indicates that, when taken as a whole, the reviews for a particular hotel are a reliable measure of average customer satisfaction of that hotel, given adequate sample size. However, the variability of scores was found to be slightly greater among TripAdvisor hotels. TripAdvisor reviews were more spread out with more high scores and more low scores. While some persons may appreciate reading a wider range of reviews about a hotel, extreme observations may be distracting or even distort consumer perception.

Why would TripAdvisor CSI Scores have higher variance than MMHI? Differences in survey methodology (number of questions, scoring, and sample selection) all play a role in differences in variance between these two sources. The fact that TripAdvisor surveys come from customers who are not part of a sample–controlled panel but are motivated by many different experiential and psychographic factors could be a source of variance. For example, it is logical that individuals who have had either an exceptionally good or bad experience would be more inclined to post a review, but individuals across the satisfaction scale would likely participate in a sample–controlled panel study. This would lead to higher variance among TripAdvisor reviews.

If online review sites were both unreliable and mistrusted, they would quickly become irrelevant. Even if online review sites were trusted by consumers but were significantly unreliable, they would not provide any value to customers and would quickly lose their audience. Our findings show that neither is the case. Review sites are gaining in popularity as customers are clamoring for more peer–to–peer support.

Consumers, in general, trust established review sites. And at least TripAdvisor, based on this study, is a reliable source for obtaining customer satisfaction information of a given hotel. Therefore, UGC will increasingly reduce the impact of traditional marketing and sales efforts and add value to strategies that will increase the customer experience and ultimately the scores of hotel reviews.

Related Link: Market Metrix