Expecting Rain

A frequent topic on these discussions is the variation of set lists.

In an attempt to measure this variation objectively, I’ve developed a predictability index, a measure for each concert of how it varied from those that came before. Here, is how it works.

(i) Each song is given a score of 1 if it is on the set-list or zero if it isn’t.
(ii) An exponentially weighted moving average of these scores over all previous concerts gives the song’s persistency measure (PM).
(iii) A simple average of the persistency measures of the songs in the set-list gives the predictability index (PI) for the concert.

Those of you who don’t have some maths background may want to skip this paragraph. I have used a smoothing factor of 0.05. Thus a song’s PM for any gig is equal to 0.05 times its score for that gig plus 0.95 times its PM value for the previous gig. The system has a half-life of approximately13.5 which ensures a relatively long memory and results in a fairly stable series of PM values for each song – if it’s dropped from the set-list for just one night, it’s PM value doesn’t change a lot, nor will it shoot up suddenly when it gets played after an absence. When a song is played for the first time ever, it’s PM goes from zero to 0.05. If it remains in the set-list thereafter, the PM value will gradually increase towards the limiting value of 1. Okay that’s technical stuff – I don’t want to lose my readership entirely.

To give readers a flavour of how this system works out, here are the persistency values for the last set-last of 2011 (Hammersmith, 21 November):


Leopard-Skin Pill-Box Hat 0.90
It's All Over Now, Baby Blue 0.27
Things Have Changed 0.94
Spirit On The Water 0.24
Honest With Me 0.61
Forgetful Heart 0.33
The Levee's Gonna Break 0.37
Man In The Long Black Coat 0.21
Highway 61 Revisited 0.99
Desolation Row 0.48
Thunder On The Mountain 0.99
Ballad Of A Thin Man 1.00
All Along The Watchtower 0.98
Like A Rolling Stone 1.00
Forever Young 0.08

Songs such as Ballad of A Thin Man and Like A Rolling Stone which have been nearly ever present in recent set-lists now have persistency values of practically 1. All Along The Watchtower and Highway 61 Revisited are also very high. On the other hand Desolation Row, which was in and out of the set last year, has a value of 0.48 while Forever Young which was played even less frequently has a value of 0.08.

Now if we average the values in the above table we get 0.63. This is the value of the Predictability Index (PI) for the final night at the Hammersmith last November. Below is a table showing how it varied during the course of 2011:



3-Apr-11 Taipei, Taiwan 0.43
6-Apr-11 Beijing, China 0.47
8-Apr-11 Shanghai, China 0.49
10-Apr-11 Ho Chi Minh City, Vietnam 0.47
12-Apr-11 Hong Kong 0.52
13-Apr-11 Hong Kong 0.49
15-Apr-11 Singapore 0.58
17-Apr-11 Fremantle, Western Australia 0.56
19-Apr-11 Adelaide, South Australia 0.51
20-Apr-11 Melbourne, Australia 0.55
21-Apr-11 Melbourne, Victoria, Australia 0.53
23-Apr-11 Wollongong, New South Wales 0.52
25-Apr-11 Byron Bay, New South Wales 0.58
26-Apr-11 Byron Bay, New South Wales 0.58
27-Apr-11 Sydney, Australia 0.53
28-Apr-11 Sydney, Australia 0.54
30-Apr-11 Auckland, New Zealand 0.54
16-Jun-11 Cork, Ireland 0.55
18-Jun-11 London, England 0.56
20-Jun-11 Tel Aviv, Israel 0.58
22-Jun-11 Milan, Italy 0.42
24-Jun-11 Sursee, Switzerland 0.48
25-Jun-11 Mainz, Germany 0.47
26-Jun-11 Hamburg, Germany 0.50
27-Jun-11 Odense, Denmark 0.44
29-Jun-11 Bergen, Norway 0.55
30-Jun-11 Oslo, Norway 0.55
2-Jul-11 Borlänge, Sweden 0.61
14-Jul-11 Santa Barbara, California 0.58
15-Jul-11 Costa Mesa, California 0.58
16-Jul-11 Las Vegas, Nevada 0.60
18-Jul-11 Phoenix, Arizona 0.59
19-Jul-11 Tucson, Arizona 0.55
21-Jul-11 Albuquerque, New Mexico 0.57
23-Jul-11 Thackerville, Oklahoma 0.61
24-Jul-11 Canyon Lake, Texas 0.63
26-Jul-11 New Orleans, Louisiana 0.66
27-Jul-11 Pensacola, Florida 0.62
28-Jul-11 Atlanta, Georgia 0.63
30-Jul-11 Memphis, Tennessee 0.63
1-Aug-11 Nashville, Tennessee 0.64
2-Aug-11 Evansville, Indiana 0.65
3-Aug-11 Toledo, Ohio 0.63
5-Aug-11 Kettering, Ohio 0.60
6-Aug-11 Cleveland, Ohio 0.62
7-Aug-11 Rochester Hills, Michigan 0.66
9-Aug-11 Canandaigua, New York 0.62
10-Aug-11 Scranton, Pennsylvania 0.67
12-Aug-11 Bethel, New York 0.64
13-Aug-11 Wantagh, New York 0.65
14-Aug-11 Asbury Park, New Jersey 0.68
16-Aug-11 Columbia, Maryland 0.68
17-Aug-11 Philadelphia, Pennsylvania 0.69
19-Aug-11 Gilford, New Hampshire 0.69
20-Aug-11 Bangor, Maine 0.67
21-Aug-11 Boston, Massachusetts 0.67
6-Oct-11 Dublin, Ireland 0.68
8-Oct-11 Glasgow, Scotland 0.62
9-Oct-11 Glasgow, Scotland 0.65
10-Oct-11 Manchester, England 0.68
11-Oct-11 Nottingham, England 0.45
13-Oct-11 Cardiff, Wales 0.56
14-Oct-11 Bournemouth, England 0.58
16-Oct-11 Lille, France 0.58
17-Oct-11 Paris, France 0.65
19-Oct-11 Antwerp, Belgium 0.53
20-Oct-11 Rotterdam, The Netherlands 0.59
21-Oct-11 Esch-sur-Alzette, Luxembourg 0.63
23-Oct-11 Oberhausen, Germany 0.59
25-Oct-11 Mannheim, Germany 0.62
26-Oct-11 Munich, Germany 0.60
27-Oct-11 Leipzig, Germany 0.64
29-Oct-11 Berlin, Germany 0.60
31-Oct-11 Hamburg, Germany 0.58
2-Nov-11 Herning, Denmark 0.66
3-Nov-11 Malmo, Sweden 0.61
4-Nov-11 Stockholm, Sweden 0.62
6-Nov-11 Hannover, Germany 0.58
7-Nov-11 Nürnberg, Germany 0.66
8-Nov-11 Innsbruck, Austria 0.62
9-Nov-11 Padova, Italy 0.68
11-Nov-11 Florence, Italy 0.67
12-Nov-11 Rome, Italy 0.61
14-Nov-11 Assago, Italy 0.69
15-Nov-11 Geneva, Switzerland 0.65
16-Nov-11 Zurich, Switzerland 0.61
19-Nov-11 London, England 0.60
20-Nov-11 London, England 0.68
21-Nov-11 London, England 0.63

From this it can be seen tha the most interesting (in the sense of set-list variation) concerts last year were those at Milan (22-Jun), Taipei (3-Apr), Odense (27-Jun), Nottingham (11-Oct), Beijing (6-Apr) and Mainz (25-Jun)

I have done this analysis for all years going back to 1974. But before posting any of this, I will let this much out first and see what the comments are like.

P.S. I must thank Bjorner and Bill Pagel for making setlists readily available.

Great idea and (as far as I can tell) great job pulling it off!!!!!

Wow, Tom! First of all, a HUGE hats off for sharing this tremendous effort. The discipline involved in putting this type of statistical analysis data together is admirable. The challenge going forward will be keeping it up to date and real-time. I'll bet the folks over at the Pool will be all over this. Nice work and thanks for giving us a preview. Thanks again!

There are not many ways math can be made fascinating to me, but you've certainly done it with this, tompius! THANKS!

My conclusion is that per performance Mr. Dylan is predictably unpredictable.

June 22
1. Leopard-Skin Pill-Box Hat (Bob on keyboard)
2. When I Paint My Masterpiece (Bob on guitar)
3. 'Til I Fell In Love With You (Bob on guitar and harp)
4. I Don't Believe You (She Acts Like We Never Have Met)
(Bob center stage on harp)
5. Summer Days (Bob on keyboard)
6. Spirit On The Water (Bob on keyboard and harp)
7. Tweedle Dee & Tweedle Dum (Bob on guitar)
8. Can't Wait (Bob center stage on harp)
9. The Levee's Gonna Break (Bob on keyboard)
10. Visions Of Johanna (Bob on keyboard)
11. Highway 61 Revisited (Bob on keyboard)
12. Forgetful Heart (Bob center stage on harp, Donnie on viola)
13. Thunder On The Mountain (Bob on keyboard)
14. Ballad Of A Thin Man (Bob center stage on harp)
   
  (encore)
15. Like A Rolling Stone (Bob on keyboard)
16. All Along The Watchtower (Bob on keyboard)
17. Blowin' In The Wind (Bob on keyboard, Donnie on violin)

   
April 3
1. Gotta Serve Somebody (Bob on keyboard)
2. It Ain't Me, Babe (Bob on guitar)
3. Things Have Changed (Bob on guitar)
4. Sugar Baby (Bob center stage on harp)
5. Cold Irons Bound (Bob center stage on harp)
6. Simple Twist Of Fate (Bob on guitar)
7. Honest With Me (Bob on keyboard)
8. Desolation Row (Bob on keyboard, Donnie on electric mandolin)
9. Tweedle Dee & Tweedle Dum (Bob on keyboard, then center stage on harp)
10. Forgetful Heart (Bob center stage on harp, Donnie on violin)
11. Highway 61 Revisited (Bob on keyboard)
12. Tryin' To Get To Heaven (Bob on keyboard)
13. Jolene (Bob on keyboard)
14. Ballad Of A Thin Man (Bob center stage on harp)
   
  (encore)
15. Like A Rolling Stone (Bob on keyboard)
16. Blowin' In The Wind (Bob on keyboard, then center stage on harp)


Maybe June 22 and April 3 are birthdays of ladies he knows. If so, it would appear Ms. June is lovely, a vision, while Ms. April is less than intelligent




   

October 11, 2011

1. Leopard-Skin Pill-Box Hat (Bob on keyboard)
2. Love Sick (Bob center stage with harp)
3. Just Like Tom Thumb's Blues (Bob on guitar)
4. Can't Wait (Bob center stage)
5. Watching The River Flow (Bob on keyboard)
6. Not Dark Yet (Bob center stage with harp)
7. High Water (For Charley Patton)
(Bob center stage with harp, Donnie on banjo)
8. Visions Of Johanna (Bob on keyboard)
9. Highway 61 Revisited (Bob on keyboard)
10. Nettie Moore (Bob center stage, Donnie on viola)
11. Thunder On The Mountain (Bob on keyboard)
12. Ballad Of A Thin Man (Bob center stage with harp)
13. All Along The Watchtower (Bob on keyboard)
14. Like A Rolling Stone (Bob on keyboard)
   

This set list is a statistical anomaly mainly because of the lack of an encore that evening.

June 27, 2011

1. Rainy Day Women #12 & 35 (Bob on keyboard, Stu on acoustic guitar)
2. The Man In Me (Bob on keyboard then center stage on harp)
3. Things Have Changed (Bob center stage on harp, Stu on acoustic guitar)
4. I'll Be Your Baby Tonight (Bob on guitar, Stu on acoustic guitar)
5. Beyond Here Lies Nothin' (Bob on guitar, Donnie on electric mandolin)
6. Boots Of Spanish Leather
(Bob on guitar, Donnie on violin, Tony on standup bass)
7. Ballad Of Hollis Brown
(Bob center stage, Donnie on banjo, Tony on standup bass)
8. Every Grain Of Sand (Bob on keyboard and harp, Stu on acoustic guitar)
9. Tweedle Dee & Tweedle Dum (Bob on guitar)
10. A Hard Rain's A-Gonna Fall
(Bob on keyboard, Donnie on electric mandolin, Tony on standup bass)
11. Highway 61 Revisited (Bob on keyboard)
12. Forgetful Heart (Bob center stage, Donnie on viola, Tony on standup bass)
13. Thunder On The Mountain (Bob on keyboard)
14. Ballad Of A Thin Man (Bob center stage on harp)
   
  (encore)
15. Like A Rolling Stone (Bob on keyboard)
16. All Along The Watchtower (Bob on guitar then center stage on harp)
17. Blowin' In The Wind (Bob center stage, Donnie on violin, Stu on acoustic guitar)


Just wow.    

Thanks, good stuff. Kind of mathematically proves my feeling when I went to the shows, Milan being a big surprise, London x 3 being a bit of a letdown, variantion-wise.

Here are some more stats on the Prediction Index. The average values of the PI for each year of the NET are as follows:

Year Average Year Average
1988 0.42 2000 0.47
1989 0.40 2001 0.45
1990 0.41 2002 0.46
1991 0.52 2003 0.51
1992 0.46 2004 0.47
1993 0.51 2005 0.36
1994 0.50 2006 0.47
1995 0.42 2007 0.54
1996 0.42 2008 0.47
1997 0.47 2009 0.47
1998 0.53 2010 0.52
1999 0.47 2011 0.59


Clearly the trend is towards greater predictability as the years pass with 2011 coming out as easily the most predictable year thus far. I don't think this will surprise anyone who has been watching the set lists over recent years.

The year 2005 is a notable blip in this general trend. Somehow I don't expect surges of unpredictability in the years ahead. For one thing, Bob doesn't seem to want to do cover versions any more. Then again ... who knows what Bob is going to do.

Ah. damn! I hadn't noticed that my tables loose their formatting when I post. Does anyone know how to get tables to display properly, i.e. get tabs to stay.

Only on other forums, where I can do my own HTML. Your figures are still very legible.

Not sure for here. Please let us know on this thread where your data will live in case it doesn't end up here. You have enough stuff here that it could be linked to your own site, really. Just please promise to let us know the link or location, OK? I like this project that you're doing. Oh, and thanks again!

tompius, that is a very cool project (and I say that as someone who regards math as his arch-nemesis)!

If you haven't already done so, I recommend creating a website with all the stats. This could be a great resource for Dylan fans.

I don't have a web site and have no idea how to go about creating one. Even though I can do maths, I'm a complete technophobe.

All my analysis exists as an excel workbook. It's best presented as a series of graphs. If somebody is interested in putting it somewhere on the web, I'd be more than happy to pass on what I have.

Meanwhile here is the list of the 25 most interesting NET concerts, ranked in increasing order of Predictability Index.


16-Nov-93 New York City (Supper Club 1) 0.057
16-Nov-93 New York City (Supper Club 2) 0.085
12-Jan-90 New Haven, Connecticut (Toad's set 1) 0.111
07-Jun-04 New York City (Apollo Theatre) 0.116
17-Nov-93 New York City (Supper Club 3) 0.120
17-Nov-93 New York City (Supper Club 4) 0.132
16-Jul-05 Seattle, Washington 0.166
23-Sep-95 Fort Lauderdale, Florida 0.175
12-Jan-90 New Haven, Connecticut (Toad's set 4) 0.179
12-Jan-90 New Haven, Connecticut (Toad's set 2) 0.181
12-Jan-90 New Haven, Connecticut (Toad's set 3) 0.191
10-Jun-88 Berkeley, California 0.203
09-Jun-88 Sacramento, California 0.207
17-Apr-05 Boston, Massachusetts 0.227
14-Jan-90 Penn State, Pennsylvania 0.234
07-Jun-88 Concord, California 0.246
13-Apr-05 Buffalo, New York 0.247
17-Jun-88 St Louis, Missouri 0.249
11-Jun-88 Mountain View, California 0.250
29-Apr-05 New York, New York 0.251
28-Apr-05 New York, New York 0.253
16-Apr-05 Boston, Massachusetts 0.258
13-Oct-89 New York City 0.260
15-Jun-88 Denver, Colorado 0.261
23-Nov-03 London 0.263

The four Super Club gigs and the four sets at Toads Place dominate the upper echelons. And nothing from the post 2005 era makes it into the top 25. Also interesting is that all are USA venues apart from London in 25th place. I was under the impression that Bob played more interesting sets when he toured outside the USA. But, seemingly not. The New York (7 Jun 2004) is an oddity: it was part of the Jazz at Lincoln Center's 2004 Spring Gala and consisted of just two songs:

It Takes A Lot To Laugh, It Takes A Train To Cry
Don't Think Twice, It's All Right

So maybe the top regular concert in the ranking is Seattle (16 Jul 2005) . Although that's not so regular either, being limited to just nine songs:

Maggie's Farm (0.20)
Tell Me That It Isn't True (0.09)
I'll Be Your Baby Tonight (0.16)
Lay Lady Lay (0.14)
Most Likely You Go Your Way (0.31)
Blind Willie McTell (0.17)
Watching The River Flow (0.14)
Ballad Of A Thin Man (0.18)
I Shall Be Released (0.11)

(persistence measure in parentheses).

Great! I guess I could create a site for this. However, this would take some time, as I will be quite busy the next couple of weeks.

Wow, no wonder 1988 scores so well. Mr. Dylan had to be madly in love in the worst sense of that term. What a month June was that year to be a concert-goer and a Dylan listener!

He performed the beautiful Irish balled, The Lakes of Pontchartrain, for the first time ever that year, and only 18 times total in his career. In June alone, the month of brides, as he kicked off his summer season, he performed this song on the 7th, 10th, 24th, & 30th. He also did Barbara Allen on the 13th, 18th & 22nd; Eileen Aroon on the 15th, 17th & 30th; and Johnny Cash's Give My Love to Rose on June 28th, along with a couple helpings of Two Soldiers, and Mama, You Been On My Mind.

It's amazing what music and verse can do with emotions that have no other outlet.

During the summer of 2005 Dylan played virtually the same setlist every night with very little variation or creativity. By the end of the summer I would look at the setlists for the week (rather than daily) to see if I'd missed anything.

Just curiosus...how did all this data get entered? It seems like a task on par with building a pyramid.

Smoke,

You want to know how I put all this data together?

Thanks to cut and paste techniques, it wasn't as arduous as it might seem. I copied the set-lists from Bjorner into excel. Then I set up a template for doing the calculations. Click and drag made much of that easy. Even so I don't want to think about how many hours I've wasted on this. Much of it done when I was being paid to do something else.

I started doing it in about 2008 when there seemed to be endless discussion on these forums about whether the set-list was getting dull or not. Anecdotal evidence can be cited to support all sides of an argument. So I considered ways to tackle the argument objectively. What I have come up with is outlined in my opening post. I was about ready to post this in 2009 but then I moved address and didn't have any internet until about a year ago. Then I updated my lists through last year and was ready to post again once the tour ended in London last November. But never seemed to get the time to write up a description of what I'd done. Didn't want to scare people with a maths-heavy preamble but still felt the need to explain the system. Finally, this being a holiday weekend here in Ireland, I sat down and got it sorted.

Thanks for the show of interest. I'll post some more summary stuff in the next day or two. Got to go out and meet somebody now.

It's only way to get anything done these days. Hats off to you!!!

I love that, tompius

tompius - you have proven what bob said - its all mathematical!

great work! thanks!

I'm cornfused...save Tell Me That It Isn't True those are all genuine staples, and even that one he'd been playing in the previous year or so.

Well let's take a look to see if the low PI makes sense.

It's true that all those titles look like staples. That's in general. However, at the time of that specific concert, they weren't all that regular. The concert in question was no. 71 in the year 2005. So the PI value is chiefly determined by the frequency of performance over these 71 concerts. Those of 2004 have some effect too but not a lot. So let's focus on what happened in 2005. The number of times each of the songs had featured up to and including Seattle are as follows:

Maggie's Farm (14)
Tell Me That It Isn't True (4)
I'll Be Your Baby Tonight (11)
Lay Lady Lay (7)
Most Likely You Go Your Way (12)
Blind Willie McTell (9)
Watching The River Flow (12)
Ballad Of A Thin Man (9)
I Shall Be Released (4)

As you can see none of them were staples up to July 2005. (Maggie's Farm became quite regular for the remainder of the year, being played at 32 of the remaining 42 gigs.) It's also the case that none of these songs were performed at the concert immediately prior to Seattle: it was a completely new set-list from that point of view. (This fact alone marks it out as being an unusual concert.) Also Watching The River Flow hadn't been played at any of the previous 20 concerts.

I hope this helps explain how the Predictability Index dropped to such a relatively low value for the Seattle concert.

Just adds fodder to the reasoning behind getting those Supper Club gigs into an official Bootleg Series box. Tom, this just became my favorite thread until the tour officially resumes. I look forward to your next posts. Your textual interpretation of the data is fun as well. Thanks again and a big handshake to whoever it was that paid for this monumental task.