Given the wide interest in whiskies – and the apparently dizzying complexity of offerings – it is not surprising that a large of number of professional and amateur whisky reviewers have become established.

There are numerous ways that whisky enthusiasts have tried to add value to your personal selection process (some with more success than others). As all would agree, there can certainly be idiosyncratic taste preferences and biases – which can certainly be highly personal. But it turns out that you can calibrate these personal assessments against one another, and draw meaningful inferences to classify whisky flavour and quality.

The goal of this site is to perform the most scientifically-valid way of integrating all the subjective views of experienced tasters – garnered from the well-known whisky reviewers described below – and provide you with a meaningful classification system and quality guide.

Reviewer Selection Criteria

The criteria used in my selection of reviewers involve:

• Particularly active since 2010
• Easily accessible tasting notes and scores available
• Well-known (or at least, well-Google ranked) for the style of whisky they review
• Have reviewed a significant number of whiskies in my dataset
• Appear to be independent in their scoring
• Show a similar distribution and range in their whisky scoring, compared to other established reviewers

Scroll down for a current list of the reviewers I track. But first, an explanation to how I compile and validate the scores.

To start, when a particular expression has been repeatedly sampled by a given reviewer over time, I have limited this analysis to scores done in the last five years. This is to ensure the dataset contains only the most modern versions of any given whisky expression (i.e., ones you are likely able to buy).

Note that there is no way to really ascertain how independent these reviews actually are (hence the “apparent” qualifier above). As a product reviewer myself in the flashlight field, I know how easy it is to be swayed by the opinions of other reviewers (simple solution for me: I don’t read other reviews of a product I’m testing until after my review is posted). I also know the pressure manufacturers/dealers can place on reviewers for positive reviews. Again, that is simple for me as I already have to decline more requests for reviews than I can accept (due to time limitations). So no problem if someone wants to stop sending me flashlights to test (and at their risk to do so, as flashlight forum members tend to notice when I stop reviewing certain common brands). 😉 In my experience, most well-established reviewers have similar policies. Correlation analyses, as described on my Meta-Critic methodology page, also help in identifying potential confounds.

Anyway, this does lead us to the next question, how did I select the whiskies in my dataset?

Truthfully, this dataset began life as a list of whiskies I had tried, or was curious to try. I added more and more reviewers over time, in the hope of finding one(s) who matched my relative preferences (and thus could serve as a general guide to future tries). In the process, these reviewers’ catalogues started to shape the database – by stimulating the types of whiskies I grew interested in. Eventually, I realized that a proper statistical meta-analysis could be done, and so more systematically built up a full dataset of whiskies and reviewers.

In building this larger dataset, I have generally maintained a focus on commonly available whisky expressions. In particular, I eventually tracked most of the whiskies that were available for sale through the LCBO (which is the sole state-controlled source of liquor in my Province of Ontario). To that dataset I have added base expressions and popular higher-end expressions for many of the common whisky brands, even if not available locally. I have also added a focus on International malts and blends, as I have some interest in this area.

Preparing for the Meta-Analysis

As I was building this dataset, I kept an eye on how the individual scores were distributing within each reviewer subset of whiskies. In any sort of meta-analysis involving inferential statistics (particularly those using parametric measures), any deviation from normality is problematic. To put that in simple terms, you can’t easily compare reviewers if they don’t review similar ranges of products. Where necessary, I added more budget whiskies into the dataset to ensure a reasonably good normal distribution for each reviewer.

The complete dataset used at the time of this site launch contained ~380 whiskies, >20 reviewers, and >2600 individual score data points (all manually curated, to ensure accuracy and consistency of reporting). As the math suggests, this meant that I had on average ~7 reviews per whisky. However, I will report on any whisky that has at least 3 reviews, for reasons I will explain on the more detailed statistical methodology pages.

The site continues to expand and grow – both for number of whiskies and number of reviewers. Please see my Understanding Reviewer Scoring page from an introduction to how the Meta-Critic Scores are assembled.

Value of Reviewer Commentaries

Note that not all online review websites are equally helpful in guiding your selection process – including those used in the analysis here. Again, I picked these reviewers because they had a good number of consistent scores relevant for my dataset. Some of these provide a better resource than others (and there are plenty of good ones not included here).

Please see my Understanding Flavour Commentaries page for some general observations and suggestions for making sense of reviewer flavour notes.

Finally, I also recommend you check out my Review Bases and Limitations page, to see some of the general limitations inherent in this analysis.

Reviewers

With those caveats in mind, let’s meet the reviewers used for the Meta-Critic score analysis here.

Note that for collectives that involve multiple reviewers, I actually need to have enough data on each individual reviewer, in order to build an appropriate statistical model. As such, I am identifying the individual reviewers that I track by name, where feasible.

• Best Shot Whisky Reviews – Jan van den Ende
  
• Breaking Bourbon – Eric, Nick & Jordan
• Canadian Whisky – Davin de Kergommeaux
• Diving for Pearls Blog – Michael Kravitz
  
• Dramming (Pour Me Another One) – Oliver Klimek
• Globe and Mail – Beppi Crossariol
• In Search of Elegance – Jason Hambrey
• Japan Whisky Reviews – Michio Hayashi
  
• Japanese Whisky Review – Brian Love (aka Dramtastic)
• Malt Maniacs – various Malt Maniacs
• My Annoying Opinions
• New Bourbon Drinker – Peter Brush
• Quebec Whisky – André Girard, Martin Vigneault, Patrick, RV and Eli
  
• Ralfy – Ralfy Mitchell
• Reddit – various Reddit reviewers (currently >50 individually tracked)
• RumHowler Blog – Chip Dykstra
• Scotch Noob – Nathan Keeny
• Whiskey Jug – Josh Peters
• Whiskey Reviewer – Richard Thomas et al
• Whiskey Wash – Aaron Knapp, Margarett Waterbury, Whisky Kirk, Joshua St. John, and Savannah Weinstock
• Whisky Advocate – John Hansell, Dave Broom, Jonny McCormick, Gavin Smith, Lew Bryson, Davin de Kergommeaux, et al
  
• Whisky Bible – Jim Murray
• Whisky.buzz – Mark Bylok
• Whisky Fun – Serge Valentin
• Whisky Notes – Ruben
• Whisky Saga – Thomas Øhrbom

And since the second year of this site, my own scores are added to the mix. 🙂

Note that there is no greater weighing for any one individual reviewer. There are cases though where multiple individuals are combined into a reduced number of reviewer categories. This could be due to relatively low numbers for the individual reviewers (e.g. Whiskey Wash), or when reviewers specialize in the whiskies they review and don’t overlap (e.g. Whisky Advocate). Sometimes individuals clusters into groups with defined characteristics (e.g. Reddit). But of course, there are also cases where each individual reviewer has enough data to be considered separately (e.g. Quebec Whisky).

I don’t have space to get into the relative merits of each of the review sites above, but some general comments on certain stand-out examples, and ones where I had had to make some adjustments for data entry purposes. In no particular order:

André, Patrick, Martin, RV and Eli of QuebecWhisky.com – http://www.quebecwhisky.com/

It is a shame that this site is not better known. Each whisky is typically fully reviewed by multiple members, and described with succinct tasting/nosing notes, personal impressions, and numerical scores by each reviewer. The catch? The site is in French only.

The five main reviewers constantly crack me up with their take-no-prisoners, call-it-like-it-is attitude (mixed with a healthy dose of sass). It helps that I’m a native Quebecker, and so can relate to and appreciate their approach. 🙂 For English-only readers, I suppose you could try the Google translate feature. You should get the major flavour profiles coming through fairly clearly (but will likely lose a lot of the humorous nuances).

Well-organized, and easily searchable, the site has logged over 2000 individual whiskies since it came online in the summer of 2011. It is true that the members of this site don’t meet the standard for true independence. Indeed, one interesting feature is that they seem to sample from the same bottle (removing that source of variability). They also provide pictures of each bottle, which is a definite aid to ensuring you know the expression they are talking about. However, they also seem to be aware of each other’s reviews, which may affect their own commentary and scores. But this gives you a good opportunity to statistically measure what happens when reviewers are clearly aware of each others’ reviews – see my detailed discussion later for more info.

Davin de Kergommeaux, Whisky Advocate reviewer and host of canadianwhisky.org – http://www.canadianwhisky.org/

Davin is probably the most recognizable name in Canadian whisky reviewing, thanks to his canadianwhisky.org website and his seminal book: Canadian Whisky – The Portable Expert. Despite the title, the book is really the most thoroughly researched and detailed history of Canadian distilling that I am aware of.

On WhiskyAdvocate.com, Davin provides numerical scores and very brief (but colourful) overviews of Canadian whiskies. As with the other WhiskyAdvocate reviewers however, there is typically a minimal amount of flavour profile description available to the reader.

His canadianwhisky.org website has exceedingly detailed reviews of whiskies, including a star rating system (i.e., no star, or 3 to 5 stars, with half star measures in-between – making a total of 6 categories). There is also some good industry background commentary, especially for the Canadian whisky scene.

Using the numerical scores from Whisky Advocate to work from, I have been able to translate his Canadian Whisky star ratings into a consistent set of score ranges. As with the Scotch Noob, I assigned the mean score of every given range to any star-only rated whisky – unless additional commentary allowed me to rank a given whisky slightly higher or lower within a given numerical range.

Nathan Keeny, aka The Scotch Noob – http://scotchnoob.com/

Nathan’s whisky review site has been around since November 2010, and focuses mainly on “affordable” Scotch-style whiskies. This typically means scotches that are available in the US for <$100 (often well uner). Roughly a quarter of his review catalogue is US-made bourbons, and there are a good number of international whiskies as well (including a few entry-level Canadian ones).

I recommend Nathan’s site as a top pick resource for budding whisky fans, as he provides a lot of good background overviews and recommendation lists – all in very clear language, geared for the newcomer to whisky. His individual reviews are generally entertaining reading, with a nicely balanced mix of distillery background, tasting notes, and personal commentary.

Note that Nathan provides only category labels for his whisky ratings (on a 6-category scale). I initially converted these into a numerical scale range for preliminary comparisons, adjusted by his relative commentaries. For example, a whisky got a slightly higher score (within a given numerical range) if he described it as his favourite in a certain class, or if he listed it as a recommendation in a background article, etc. This approach is perfectly justified, as the meta-analysis re-equalizes individual reviewers scores in statistically valid manner later.

However, in following up with Nathan, I discovered that he adjusts his rankings depending on price (i.e., really expensive whiskies get lower category rankings). For full consistency to other reviewers used here, Nathan has provided me with his actual raw scores, which I have directly incorporated into the database.

Jason Hambrey of In Search of Elegance – http://www.InSearchofElegance.net

Formerly known as WhiskyWon, Jason’s whisky site is another great resource for those looking for reviews of modern whiskies. Located in Ontario (as I am), Jason generally draws from the same LCBO well that is available to me for tastings/purchase – but supplemented by his wide network of contacts. While the site has only been up for a couple of years, Jason has an impressive number of detailed reviews posted.

Each review has a picture of the bottle, detailed tasting notes, background on the whisky, and scores (broken down by category and summed for a total score). Somewhat like Nathan the Scotch Noob, Jason is another modern reviewer who I think has got the mix largely right for producing an informative and entertaining review site.

John Hansell, Dave Broom, Jonny McCormick, Gavin Smith and Lew Bryson, of WhiskyAdvocate.com

Whisky Advocate is great resource of whisky information, news and reviews (I will admit to being a subscriber). It has a number of dedicated reviewers/commentators (including Davin de Kergommeaux, already covered earlier). A numerical quality score is provided, along with very brief overviews of each whisky – in an easily searchable public database.

An unusual feature of this site is that the reviewers do NOT do competing reviews of a given spirit (i.e., only one member of the group officially reviews it). The presumption therefore is that they have worked to calibrate their scoring systems, to provide reasonably good concordance between individual reviewers.

For the purposes of my meta-analysis, I separated Davin out of the group (since I had additional data from his own personal site). But you could effectively consider these five remaining reviewers as one reviewer category (as they never provide competing reviewers).

Josh Peters, of The Whiskey Jug – http://thewhiskeyjug.com/

With a roughly equal split between American whiskies and Scottish single malts and blends (with a few international and Canadian whiskies thrown in), Josh’s extensive review site provides a good resource to help round out my whisky database (particularly on the bourbon side). It is good to have someone who has a different perspective, based in part on local availability and experience.

Josh’s reviews follow a traditional format, with a score right up front (next the actual picture of the bottle reviewed). Very methodical, with succinct and crisp tasting notes. Josh is also part of the Reddit review community – but I keep his reviews in separate category from the rest of the group.

Jim Murray, author of The Whisky Bible

Mr. Murray likely needs no introduction to whisky enthusiasts – especially fans of Scottish single malts and blends. For the purposes of this comparison, I have relied predominantly on the latest 2015 and 2016 editions of his Whisky Bible.

This poses some difficulty for my meta-analysis, as he has been reviewing for many years, and it is hard to separate new content in his book from the old (and potentially outdated) entries. The guide thoughtfully annotates which is the newest review of a given expression, but it is hard to know what to make of the others (unless you can cross-reference to earlier editions). It can also be difficult to locate a given whisky in the guide, especially outside of the main Scottish distilleries (i.e., the American whiskey section has various expressions of the same distilleries appearing in different subsections). Given the lack of pictures or any form of searchable index, it can thus be hard to know if he’s referring to the specific expression you have in front of you.

I try to avoid commenting on how reviewer’s scores correlate to one another – but you don’t have to look too hard online to find a lot of commentary on this issue in regards to Mr. Murray. I will grant that he has been extremely prolific, but is unusual among reviewers for having personally favoured both complex and delicate whiskies with high scores – including some very basic budget blends. See for example this discussion of his scoring on my Crown Royal Northern Harvest Rye review.  Variation among reviewers is an important point that I will return to in the step-by-step guide to how to best interpret the results of the meta-analysis results here.

Ralf Mitchell, aka Ralfy the video blogger – http://ralfy.com/

I doubt any modern whisky enthusiast (with an internet connection) is unfamiliar with the video whisky blogger Ralfy. While not the first one to perform direct-to-camera overviews of his favourite whiskies, I doubt anyone else out there has amassed as large of a fan following (>70,000 subscribers on his YouTube channel, when last I checked).

I suspect Ralfy’s casual “chatty” style, wealth of knowledge, and highly personal delivery are all part of his considerable charm (the native Scottish accent probably doesn’t hurt either). In addition to very extensive comments about each of the individual whiskies he vlogs about (his preferred term), you also get a lot of background on the individual distillers – and on the general field and history of whisky-making in Scotland.

At the end of each review, he provides a numerical score – starting at 8.0 (he doesn’t review whiskies that don’t meet this personal cut-off level). As with a couple of other reviewers included here, this means that some adjustment has to made in the meta-analysis for the “raised floor” of minimum quality in the whiskies he reviews.

Ruben – WhiskyNotes.be

WhiskyNotes is one of the early whisky blog sites – and one that continues to be very active to this day. Run by a Belgian native, Ruben has extensive experiences with many classes of whisky – and most especially single malts. The site is also a great place to get news and updates of upcoming releases – especially for some of the rarer and higher-end products from well-known distilleries.

Ruben’s individual whisky reviews contain just the right mix of distillery background, clear flavour descriptions, and overall quality assessment. The site is well organized, easy to search, and with a compelling visual style.

Richard Thomas et al – WhiskeyReviewer.com

A self-described modern “Web Magazine”, WhiskeyReviewer has rapidly become for one of the more highly-ranked sources of whisky news and reviews online.  With a core staff of 7 members (and several guest writers), they regularly produce original news, reviews and commentaries about whisky and its production. Like some of the other more active sites listed here, they typically produce new material every weekday (including at least one whisky review).

WhiskeyReviewer uses a letter-based rating system, but this can easily be converted into a corresponding numerical score. And as it turns out, the distribution of review grades/scores on this site closely matches the other reviewers presented here (I will explain the distribution characteristics of reviewers on my next methodology page). Note that multiple writers/editors can sometimes review the same whiskies – in those cases, I average the ratings across members on this site.

Serge Valentin – http://www.whiskyfun.com/

Serge has been (and continues to be) an influential presence in the whisky world. He provides personalized tasting notes and scores on his own site, WhiskyFun.com. Serge has detailed an impressive array of whiskies – including many versions of the same expression over time.

I appreciate that the origin and age of all the expressions profiled on WhiskyFun are clearly delineated, typically with pictures of the actual bottles (which is always helpful). That said, it can be a tricky to search through the extensive dataset on the site, so you will want to pay attention to these details to ensure you are getting comments for the expression you are interested in. Serge’s tasting notes are fairly succinct, with some personal observations peppered throughout.

Thomas Øhrbom – http://www.whiskysaga.com/

Run by a self-described “whisky nerd” from Norway, whiskysaga.com is both a great review site and thoughtful distillery travelogue. The site began in January 2012, and has been published in English since April 2014 (you will need to run earlier Norwegian posts through Google translate). Most of the content is provided by Thomas, but with additional commentaries from guest bloggers.

Succinct tasting notes are provided on each whisky reviewed, along with an appropriate level of distillery background. Thomas has had access to a wide-range of higher-end whiskies from around the world, and has reviewed many whiskies that I have never heard of (especially the new Nordic ones). Extremely prolific, with a very active posting schedule – I find Thomas is often among the first to review new expressions. His enthusiasm for the topic is also quite infective – I find myself wanting to visit all the places he describes! The Whisky Saga site is well worth the visit.

Malt Maniacs – http://www.maltmaniacs.net/

The Malt Maniacs are well known in the whisky industry. Originally started by a core group of enthusiasts who met over the internet, when last I checked there were 31 “certified” members in the group (who presumably try to be consistent in their scoring approach). They are well-distributed geographically, which helps to ensure a very good distribution of whiskies in their dataset.

While active for a number of years – their publicly accessible Malt Monitor database (http://www.whisky-monitor.com/home.jsp) had over 17,000 distinct bottlings at the time of this site’s launch – they no longer seem to be active. I have looked at the breakdowns between Malt Maniac scores across a number of whiskies, and find the variability to be unusually low (i.e., they are tend to cluster around the average score for a given expression, with the exception of a few outlier members). As a result, I have opted to provide a single average score across all the members who have reviewed a given whisky (across all sampling for expressions tasted in the last five years). No greater weight is assigned depending on reviewer number, just the single average score. While not ideal, I feel this is the simplest way to incorporate their highly consistent scores into my analysis. It would be too Herculean a task to track down the individual full catalogues of each member, and their low variability in scoring suggests you probably wouldn’t want/need to anyway.

Note that the Malt Maniacs Oliver Klimek and Serge Valentin are included separately in my database, as they maintain independent websites with their own tasting notes and scores. When feasible, I have removed any entries for these members.

Reddit Reviewers – https://www.reddit.com/r/scotch

Reddit has extremely active members forum on many topics. The “Scotchit” user group on Reddit meets many of the reviewer criteria here, including being very active in the last few years with openly available reviews. While the main Scotchit site can be a bit daunting to navigate, the full open-access review archive is freely available (with over 13,000 reviews at the time of this site’s launch). The main challenge is that individual Scotchit user reviews (and scoring) can vary widely in quality, experience and consistency, as I discuss in this commentary post.

After some effort, I have managed to identify top Reddit reviewers who individually meet the level of the expert reviewer selection described above – save for independence. Like the Quebec Whisky site, these reviewers are aware of each others reviews, and actually swap/trade samples (indeed, I am one of them). But they show even higher levels of correlation among themselves, with some atypical (but consistent) views on certain brands and expressions – as described in this follow-up commentary post.

I have continued to revise this analysis, and can now divide the Reddit community into three broad classes of reviewers. Please see the Update section at the end of the commentary linked above. To summarize, I normalized the scoring for each Reddit reviewer individually (currently 46 of them), but then combine them into one of five identified subgroups. These subgroups then move through the second-pass normalization and inclusion the Meta-Critic database along with the other individual expert reviewers. In other words, the top Reddit reviewers are integrated into five separate reviewer categories, and thus provide (collectively) up to 5 data points on each whisky.

The Scotchit group on Reddit has a lot of members with an impressive knowledge of whisky. It is definitely worth checking out if you have questions.

With that introduction out of the way, I recommend you move along to my Understanding Reviewer Scoring page.