The following statement presupposes that these “show elements” merit a show being rated more highly, which is a highly debatable proposition—

“If the deviation and entropy measures have successfully identified the raters who contribute the most information and least bias, then weighted show ratings should be even more correlated with show elements relative to the current system. Technically, this analytical approach is a test of convergent validity.”

Who is to say that a poorly performed but major “bust out” always merits a higher rating, for example? The Baker’s Dozen Izabella was a huge bust out but pretty poorly performed. The first Tela I ever saw made me depressed because of how not like the 90s it was (subsequent ones were much better).

It seems to me the entire endeavor to ascertain a “reliable” rating system described in these series of posts itself presupposes that (1) there is an objective, verifiable way to measure the quality of a show and (2) the rating system should be used for that purpose instead of to reflect the aggregate of the assessments of the people who have heard or experienced the show.

I would say both of those assumptions are wrong, and the whole effort is misguided in that sense.

To me, a useful rating system should tell us (1) what did the people who attended this particular show think (ie, the fun factor) and (2) what do the people who regularly listen to and attend Phish concerts think about the quality of this show (ie, replay quality). (If someone went to a show and got puked on and had a bad time, it’s not “wrong” for him to rate it poorly.)

Distinguishing ratings on those two dimensions would help a lot.

A third may be to distinguish quality of jamming from quality of composed performances. For me there are shows where the jamming is better and others where the compositions are played perfectly, and they don’t always coincide.

Allowing a 10 point rating scale would also help for the reasons mentioned above—most phish shows are amazing, and no one wants to rate one a 3/5. 6 or 7 out of ten seems more generous even when it technically is not. In my view most phish shows should be in the 6-10 out of 10 range.

How to verify that the ratings submitted by users reflect genuine beliefs is a separate issue and I applaud the thinking to develop some sort of control. Weighting rankings could be one way although similar to others here it rubs me the wrong way.