Methodology – Introduction
Doing the sort of scientifically-valid meta-analysis envisaged here properly is a very labour intensive process. Most online “metacritic” sites are simple aggregators that just average a subset of scores on a specific topic (often of variable quality). The goal here is a lot more ambitious – to weigh the quality of evidence from multiple sources, and provide statistically valid extrapolations. I plan to do this for both quality assessment metrics, as well as underlying flavour profiles. This involves the extensive manual curation of data from numerous sources, which I have been working on for many months.
As mentioned earlier, you might think that it is not possible to provide objective classifications and rankings from purely subjective data. But that too is a question that can be empirically tested using statistical means – and one that you can easily verify yourself with the right tests. Note that when it comes to a flavour assessment, I am not the first one to think of doing this. I have the advantage of building on the best scientific analysis performed to date on whisky flavour (which I will describe on these pages). But what is new here is the meta-analysis approach I’ve taken to quantifying whisky quality. This is something I honestly did not expect to work out as well as it has, and I will need to take some pains to explain that methodology and the results in detail.
Please check out the subheadings of this section for more information on how I have done the meta-analysis here, with some example of the data validations.