[We would like to thank Paul Jakus (@paulj) of the Dept. of Applied Economics at Utah State University for this summary of research presented at the 2024 Phish Studies Conference. -Ed.]

Numerous Phish.Net Forum threads have hypothesized about the effect of people with extreme ratings behavior on show ratings, so this post will focus on the behavior of (anonymous) individual raters. The most well-known of extreme raters are “bombers”, or those who rate all shows as a ‘1’. “Fluffers”—those who rate every show a ‘5’—have received less attention than bombers.

The ratings database has 88 people who qualify as pure bombers and, adding them all up, this group has submitted a total of only 268 ratings out of 343,241 total ratings (0.08%). The database contains 5,499 pure fluffers, or people who have rated shows using only ‘5’. Clearly, fluffers are far more prevalent in the data, and account for 15,757 of all show ratings (4.59%).

But that’s not all. Remember Rater B from the first post in this series, with those 947 performances rated as ‘5’ and one show as a ‘4’? They’re not a pure fluffer, so if we are concerned about extreme rating behavior we must look beyond pure bombers and fluffers. Further, the terms “bomber” and “fluffer” imply deliberate manipulation of ratings. We simply cannot know if a submitted rating is designed to manipulate, or if it reflects true beliefs. Thus, the product marketing literature refers to these people as “anomalous” raters, and has developed a number of statistics designed to reveal them in large databases. We’ll look at how two of these measures—average deviation and entropy—work with the ratings database.

Consider the ratings profiles of several real (but anonymous) raters pulled from the database. Raters A and B first appeared in the first blogpost (A gave the most ‘1s’ whereas B gave the most ‘5s’). Rater C is a pure fluffer (all ‘5s’) and Rater D (all ‘1s’) is a pure bomber. Raters E and F are broadly representative of the typical user in the database—a lot of ‘5s’, and a few ‘4s’ and maybe a couple of ‘3s’. Rater G tends to use either a ‘5’ or a ‘1’, and only rarely uses ratings in between (i.e., a simple “thumbs up/down” scale.)

Average Deviation. This measures the average difference between a person’s show rating and the mean show rating of all raters, for all shows a person has rated. In our ratings database, large average deviations can occur only when a person uses a lot of ‘1s’ relative to their other ratings. Rater D, a pure bomber over 35 shows, has a large average deviation (2.858). Raters A and G have lower average deviations (despite a lot of ‘1s’) because of their ‘4s’ and ‘5s’.

Entropy. Entropy measures the average expected value of information provided by a rater. The formula for entropy can be found [here, Wikipedia link below]. Smaller values of entropy are associated with less informational value, and the intuition is straightforward: Look at Rater C and make your best guess as to what their next show rating will be? If C provides anything other than a ‘5’, we’d be surprised. Similarly, what do you think the next rating from Rater D will be? Again, anything other than a ‘1’ would be a surprise.

The expected average value of information (entropy) is zero for both pure fluffers (Rater C) and pure bombers (Rater D). From an informational perspective, fluffers and bombers are identical, and the motivation for their response (honest or manipulative) is irrelevant. We can predict perfectly what their next rating will be, regardless of the quality of Phish’s next performance, so they are of no help in assessing show quality.

Entropy suggests the expected informational value of Rater B’s next rating is a very low (0.008) because they are almost always a ‘5’. In contrast, Rater A uses the full scale, and so has a very high entropy score: there’s likely a lot of informational value in their rating. The same holds true for Raters E, F and G.

Part 3 of this series will address issues of statistical bias in the database, and how anomalous raters (identified by average deviation and entropy) may contribute to that bias.

WIKI LINK: https://en.wikipedia.org/wiki/Entropy_(information_theory)

If you liked this blog post, one way you could "like" it is to make a donation to The Mockingbird Foundation, the sponsor of Phish.net. Support music education for children, and you just might change the world.