Since its initial release in 2013, the “FaSoLa Minutes” iOS app has become an important component of how many singers learn and engage with Sacred Harp songs, singers, and singings. The app includes data from the Minutes of Sacred Harp Singings since 1995. A review of the app appeared in vol. 3, no. 2 of this newsletter. Among many useful features described in that review, the app can be used to answer statistical questions such as:
- Which songs in The Sacred Harp: 1991 Edition have been led the most often?
- How has the popularity of each song varied over time?
- How many times has each leader led since 1995, and how many different songs?
- What is each leader’s entropy number?
…and this is the point in many discussions of the “FaSoLa Minutes” app where the puzzled expressions appear, because the entropy measure is unfamiliar to most Sacred Harp singers. You may have some additional questions:
- Why is the entropy measure included with the other statistics in the app?
- What does a leader’s entropy indicate about that leader?
- How is the entropy number calculated?
- How does my choice of songs change my entropy number?
- Does my entropy measure matter—and should I work to change it?
We will answer these questions with both nontechnical explanations and formulas for readers who are so inclined. In the following sections, we describe what the entropy number is intended to measure and how to interpret it. Then we explain the formulas used to define entropy with simple examples that illustrate how song choice affects the entropy measurement. We conclude with thoughts on how song leaders can use the entropy number, with tables that illustrate how entropy is affected based on a leader’s choice of songs.
How to Interpret the Entropy Number
The “Help” page in the “FaSoLa Minutes” app describes entropy as “a measure from 1 to 0 of a singer’s unpredictability when leading.” An entropy number near 0 indicates that a leader is very predictable, while a number near 1 indicates high unpredictability.
The entropy measure is built from the frequency that a leader has called each song in the book, so it provides more information about how predictable the song choices of that leader are than the other song count information in the app. For example, picture two hypothetical leaders, Adam and Bertrand, who have each led 100 times and have led 20 different songs. However, Adam has led each of his 20 songs 5 times, while Bertrand has led 19 songs 1 time each, and “Rose of Sharon” (p. 254 in The Sacred Harp) 81 times. At the next singing Adam and Bertrand attend, the arranging committee will have a harder time predicting what Adam will lead. The entropy measure for each leader reflects this by assigning a higher entropy number to Adam than to Bertrand.
Note that the entropy number does not indicate whether someone is a good leader. For example, one of the qualities of a good leader is the ability to choose an appropriate song considering such factors as the time of day and the energy of the class. A leader who is unpredictable to the point of being completely random is unlikely to do this well. Also, the statistics in the app are based on song choices in the minutes, and therefore don’t reflect other recognized qualities of a good leader, such as familiarity with the song one chooses to lead, selection of appropriate verses and tempo, and good communication of the leader’s intentions to the class. There are excellent leaders with relatively high entropy, and there are other excellent leaders with relatively low entropy.
History and Mathematical Formula for EntropyThe formula for entropy was developed by Claude Shannon in the 1940s. Shannon was a mathematician at Bell Labs, the research and development branch of the Bell Telephone Company. His ideas of information theory became part of the foundation of computer science. Shannon defined the smallest unit of information as a “bit,” which can only have one of two values: or . He described how any message can be encoded as a sequence of bits, and developed the entropy formula as a way to measure the amount of information in a message. Shannon entropy “” is defined by this formula:
is the probability of a particular term appearing in the message, out of all the possible terms—in our case, the probability of leading a particular song (song ) from The Sacred Harp.
is a logarithm to base 2. (Logarithms are the reverse of exponentiation. We say that the logarithm of 8 to base 2, written , is equal to 3 because .) Shannon
used base 2 in this formula because the “bit” has two possible values.
The symbol (the Greek letter Sigma) is mathematical shorthand for a sum of a number of terms, where in this case each term is with the index value “.” So the entropy formula could also be written as:
including a separate term for each song, but that would take much longer to write out.
For the entropy calculation in the app, we make the assumption that the frequency each song has been lead in the past is likely to predict future behavior. So, for example, if you’ve led 100 times in the minutes, and 20 of those were “Lenox” (p. 40), then
the frequency with which you led “Lenox” was 0.2, or 20 percent, and we would use 20 percent as the probability for “Lenox.”
For each leader, the probabilities of leading each song in the book need to be added together to calculate that leader’s entropy. If you’ve never led a particular song, it neither adds to nor subtracts from your entropy.
The maximum possible entropy comes from choosing any of the 554 songs in The Sacred Harp: 1991 Edition, with equal probability. In other words, the class has no way of predicting which song you are about to call, so the leader with maximum entropy is completely unpredictable. In math terms, all the ’s would be the same, 1/554. The Shannon entropy of the completely unpredictable leader would be:
In the “FaSoLa Minutes” app, someone who is completely predictable (i.e. always leads the same song) has an entropy of 0, and the completely unpredictable leader has an entropy of 1.0. This is because entropy reported in the app has been “normalized” by dividing by . (Normalization is frequently used in statistics, because it makes it easier to identify where a statistical value falls between the minimum (0 or 0%) and maximum (1 or 100%) possible values.) A reported (“relative”) entropy of 0.600 is actually a raw entropy of 5.47, divided by 9.114, the maximum possible entropy.
“FaSoLa Minutes” app normalized entropy
Example: How Song Choice Affects Entropy Number
Say a leader, Catherine, has led 2 different songs 3 times: “Lenox” twice, and “New
Britain” (p. 45t) once.
Then in the calculation of the entropy formula, “Lenox” has a probability of 2/3, and “New Britain” has a probability of 1/3, so the entropy number is 0.1008. (If you look through the leaders listed in the app, you will find many leaders who have led two different songs three times and therefore have this entropy number.)
Now Catherine is going to lead for a fourth time, and is choosing a song. How will her entropy number change based on her choice?
Option 1: If she leads “Lenox” again, its probability rises to 3/4, and “New Britain’s” drops to 1/4. This makes her song choice more predictable than before, so her new entropy number goes down to 0.0890.
Option 2: If she leads “New Britain” again, then its probability rises to 2/4, and “Lenox’s” probability falls to 2/4. Her entropy rises to 0.1097, because her song choice is less predictable than before, even though she has still only led 2 songs.
Option 3: She picks a new song to lead—for the formula it doesn’t matter which of the other 552 songs in The Sacred Harp she chooses. Then “Lenox’s” probability drops to
2/4, “New Britain’s” drops to 1/4, and the new song has probability 1/4. As a result, her entropy rises to 0.1646, because her song choice is much less predictable than before.
In general, leading a song that was previously led could either increase or decrease the entropy number, depending on how it changes the probabilities of the songs already led, but leading a new song always results in the largest change in entropy, and it always increases the entropy number.
(Homework for the interested reader: Using some algebra and the properties of logarithms, the formula for “FaSoLa Minutes” app entropy shown above simplifies to , which uses common logarithms instead of the logarithm to base 2. Common logarithms are a common function on scientific calculators, so this makes it possible to work out simple examples such as the ones above with a calculator. Give it a try!)
How Song Leaders Use Entropy
So what does entropy mean for song leaders? One way to think about it is in terms of the number of songs a singer seems to be leading. If a leader has led 20 different songs one time each, her entropy would be 0.48. If she has led 50 different songs, her entropy would be 0.63. See Table 1 below for some sample figures. If a leader led the number of songs on the left, one time each, that leader would have the entropy number on the right.
Table 2 shows the equivalent number of songs, led one time each, which correspond to a given entropy number in the app. This gives a sense of the minimum number of songs/times an individual would have to lead to arrive at the corresponding entropy.
Your entropy number can be interpreted as a measure of the range of songs you are likely to select. Repeating a previously-led song will result in a lower entropy than leading a new song, as described in the previous section. The number of songs listed in the tables serves as a kind of “functional minimum” on the possible number of songs a leader is expected to have available. Looking at the table, a leader with an entropy of 0.700 has at least 83 song choices available, but likely many more. For example: After leading five different songs, I would have an entropy of 0.255. If I then choose to lead a repeat song, my new entropy will be 0.247. But if I choose to a lead a new song, my new entropy will be 0.284. As the entropy number rises, the differences will become smaller. Any repetition of a song makes you more “predictable,” and lowers your entropy. At the other end of the spectrum, if you only lead two songs in perfect alternation, your entropy number will oscillate around 0.11, no matter how many times you lead.
Note that because almost all leaders have repeated songs, and most leaders lead some songs more often than other songs, most leaders who have led 20 different songs will have an entropy lower than the maximum 0.48; this difference between the actual entropy for a leader and the highest possible entropy in these tables indicates indirectly how often a leader has chosen to repeat a previously-led song, as opposed to trying a new one.
You’ll notice from the tables that the result isn’t “linear”: as the entropy number increases, it takes a lot more songs to get the same amount of change in entropy.
The best way to raise your entropy number is to lead a lot of songs at singings recorded in the minutes, and to lead a new song each time. But the fact that the app measures entropy doesn’t mean this needs to be your goal. While leading many different songs might be a good way to become a more proficient leader, we should always think about how a song might fit with the texture of the singing when choosing. This should be something we consider whether we’re leading a new song or not. For example, it’s not a sign of a proficient leader to choose “Rose of Sharon” at the end of a day-long singing, regardless of the leader’s entropy number. In addition, the only way to improve your leading of a particular song is to lead it multiple times.
We hope that this article improves understanding of the entropy statistic in the “FaSoLa Minutes” app, and appreciation for all the components that make one a good leader. Attaining a high entropy number is not a useful end in itself, but the entropy measure provides information that can help us think about how we choose songs to lead.
The iOS application “FaSoLa Minutes” was developed by Mark Godfrey, and designed by Lauren Bock. It is available for $4.99 from the iPhone App Store, with proceeds going to the Sacred Harp Musical Heritage Association.
|Minimum Number of Songs||Entropy|
|Entropy||Minimum Number of Songs|
I’m wondering whether there could be a different way to measure probability. If I’ve led only one or two songs, the app’s entropy measure assumes that those are the only songs I might lead in the future. But in reality, if I’ve only led once, that gives no information about my future probability—I might never lead them again. It’s only when I’ve begun to lead more often that it becomes clear that the probabilities of those songs are actually higher.
So if I’ve led 5 songs 5 times, the proportion of songs led to times led implies that my probability of leading those songs again is actually lower. The problem is quantifying this intuition.
You pose good questions about an intuitive understanding of entropy in a series of song choices. Rather than thinking of our ability to predict someone’s next lesson (since they may choose any song in the book if it hasn’t been led that day), think of how much information about their choices is contained in the record of prior lessons. There is no entropy in one lesson of one song (entropy equals zero), since the frequency of that lesson is one. Likewise, there is no entropy in leading the same song over and over, no matter how many times (again the frequency of that song will be one). A more illuminating case is the one you cite: What If I lead five songs, one time each? This is the maximally entropic way to lead five times, with an entropy of 0.255. Any repeats among the songs, for a total of five lessons, would lead to a lower entropy measure, and convey the intuition that the leader is a bit more predictable, i.e., less “random.”
Note again that there is a maximal attainable entropy for any particular number of lessons, i.e., by leading a new song every time. Those maxima are detailed in the Tables. Your sense is correct that we know more about a leader after they have led fifty times than we do after they have led five times: as the number of lessons increases, the range of possible outcomes increases. Try comparing the entropy of a leader to their maximum attainable entropy for that total number of lessons. This gives a better relative sense of their predictability.
This is good advice! Somehow I missed the tables on my first reading. I’m wondering how valuable it would be to “normalize” entropy by listing the quotient of actual entropy/possible entropy for number songs. Do you think this would be a good intuitive measurement?
I think that quotient is a good way to make fair comparisons. I don’t know that that means it will be added to the app.
At the risk of appearing to be a smarty pants, I’ll point out that Claude Shannon’s formula for entropy of information is essentially identical to an older formula for what is called “entropy of mixing” in chemistry—except for the base of the logarithm.
It wasn’t lost on Shannon and his colleagues in the 1940s that their formulation of information entropy resembled entropy in statistical mechanics, as worked out by Gibbs, Boltzmann, and others in the nineteenth century. This is probably why they adopted the term.
Anecdotally, Shannon was told that using the term entropy was ideal, because nobody really understood entropy anyway, and he could say whatever he wanted without fear of contradiction.
I would love it if something like this were available for desktop computer users at some point.