Expecting Rain

I notice that lots of folks who are discussing the Top Ten NET Years over in the rare recordings room mention 1988 in their lists. Based on tompious' numbers and what little research I did on set lists, that makes perfect sense. 2005, the year that "scores" a .36, was also on a number of lists on that thread.

Here is a link to that thread for cross-reference - http://expectingrain.com/discussions/viewtopic.php?f=12&t=68636

It is interesting to me to compare the objective numbers with the subjective opinions as to which shows or years stand above the rest.

interesting stuff!

thanks, tompius!

Previously I posted the 25 most interesting NET concerts. Now I'm going to the other end of the scale and listing the 25 with the highest Predictability Indices. Here they are ranked in descending order.

17-Jul-93 Bern, Switzerland 0.712
14-Nov-11 Assago, Italy 0.691
25-Oct-98 Chicago 0.687
17-Aug-11 Philadelphia, Pennsylvania 0.686
19-Aug-11 Gilford, New Hampshire 0.685
16-Aug-11 Columbia, Maryland 0.683
10-Oct-11 Manchester, England 0.682
06-Oct-11 Dublin, Ireland 0.681
14-Aug-11 Asbury Park, New Jersey 0.681
20-Nov-11 London, England 0.679
09-Nov-11 Padova, Italy 0.676
13-Jul-93 Cascais, Portugal 0.674
20-Aug-11 Bangor, Maine 0.673
11-Nov-11 Florence, Italy 0.672
22-Feb-98 Fairfax, Virginia 0.668
10-Aug-11 Scranton, Pennsylvania 0.668
21-Aug-11 Boston, Massachusetts 0.667
07-Aug-11 Rochester Hills, Michigan 0.662
07-Nov-11 Nürnberg, Germany 0.659
23-Oct-98 Minneapolis, Minnesota 0.659
26-Jul-11 New Orleans, Louisiana 0.657
02-Nov-11 Herning, Denmark 0.656
15-Nov-03 London 0.655
09-Oct-11 Glasgow, Scotland 0.653
20-Feb-98 Bristol, Tennessee 0.652


Note the preponderance of gigs from last year in this list: 18 of the 25. This makes a good case for the argument that the NET has become a bit predictable as far as set-list variation goes.

Still there's no reason not to be hopeful of seeing Lily Rosemary and The Jack of Hearts on this year's sets.

Is there any way to factor in the length of the set lists? I think the predictability is higher as the set lists have gotten shorter, but.... that is due in no small part to the typical encores and the shorter set lists.

Two thoughts:

1) Can we do this without the encores?
2) Can we factor in the length of the set list?

I'll love this thread either way.

always something else, i'd call that pretty predictable... just kidding

Tompius,
When people want more, it's a sign that they dig what you're doing and understand it. That's good. And should be taken positively as a compliment. Right now we are looking at straight set lists only. Not musical quality or different arrangement or much else. I was at some of these predictable set list shows and came away more amazed by the quality of the music and the freshness of the arrangements. There are a million ways to go with this and I am sure that you are quite aware of all this. Thanks again!

Funny, I was thinking the same thing. I was at the Chicago show third from the top...I didn't like it as much as some other shows but I'd brought some non-fan friends, and I couldn't have asked for a better show for them. Everything was tight, everything was dead on...even when they couldn't catch the words they knew they were seeing something.

I'm not sure if PI values are influenced by the length of the set. In my list of the least predictable concerts many had relatively short sets - Supper Club, Toads Place.

It depends on how short and long sets are put together. There are two views:

(i) Bob first makes a core set from the tried and tested staples and then racks his brain for a few rarely played numbers which he adds to this core set. The longer the set, the more rarities.

(ii) He starts with the rare songs and then adds the staples to bump it up to whatever length suits the occasion. The longer the set, the more regulars.

If he works by (i) then short sets will have proportinately more staples and will accordingly have higher PI values. If he goes by method (ii), the opposite is true: it's the long set that will have the preponderance of staples and it will have the higher PI value.
My instinct is that he doesn't use either method consistently. Sometimes he'll base a short set from the rare songs, sometimes from the regulars.

Well there's one way to check my instinct: measure the correlation between set length and PI. If it's more or less zero, my instinct is right. I'll do this later today and get back to you with the result.


Yes. Encores mostly - though not always - consist of familiar songs. So if we ignore the encore, the PI value will be lower. I could calculate three values for each concert: one for the main set, one for the encore and one for the set overall. However I'm not that keen on this. For one thing I'd have to do a fair bit of work to get all these calculations done. Also it complicates the picture even more. I'm trying to keep it as simple as possible. Part of the inspiration for doing the project was to try to put some general framework on the NET which, with 2397 concerts to date, can be overwhelming to contemplate.



As I've stated above I'm inclined to think set length does not affect the PI value. But if, when I do the correlation test, it comes out that there is a significant effect, then I'll give the matter some more thought. I'm reluctant to complicate what is already complicated.

Thanks for your interest.

PI values only measure set-list variation. The quality of the performance is another matter. And, as we know, qualitative assertions are subjective. One can view the matter in two ways:

(i) Practice makes perfect: playing the same set night after night leads to better and better performances.

(ii) Repetition is dull: playing the same songs over and over leads to carelessness and general boredom for the performer.

I'm inclined to think that when Bob shakes up the set-list, he also inspires himself to give a greater performance. On the other hand if he has become bored with the tour, he'll be unlikely to bother changing anything. So low PI values should pinpoint better performances. This is only my opinion and I haven't tested it at all.

Thanks for your very positive comments on my work. And can I thank everybody else too for their positive reception of the PI. When I began the thread, I thought the first reply would be from Thickboy with his "how dare you start a thread without getting my permission first, you handful-of-posts greenhorn low-life" rant. And he hasn't commented at all. Almost disappointed, I am.

The relationship between set length and PI.

I said earlier that I could calculate the correlation between set length (number of songs) and Predictability Index. Well it turns out to be about 0.00043 which is practically zero. So there is no relationship between the two: Short sets are just as likely as long sets to have a high (or low) PI.

I also calculated a few more statistics. If we take concerts that feature in the all-time top 75 (as voted a few years back), the average PI for those is 0.389 which is relatively low. The average for those recommended by Bjorner is 0.440, by Monicas dude is 0.409 and those by John Howells is 0.418.

The average for all NET concerts is 0.470 (Did I not report this already? It should have been on my first post.)

This is great stuff. I'm hoping that the Deadheads over at their site don't try to scoop you up too soon. By the way, are you a stockbroker or something?

So do you feel like predicting the first South American set list, or more, just for kicks, Mr. Pius?

Got your spreadsheets all revved up tompius? It won't be long now. A week from Sunday is the Rio 2012 opener.

I'm guessing that tonight's show is considered a 0.68, tompius?

Tompius says 0.63 for last night's Rio show.

Watch this thread for more mind-numbing stats in the next few days.

There should be a sticky on this thread. It provides interesting insight into the setlists and helps to quantify what we are hearing. It also provides valuable perspective.

Kudos to Tompius for his hard work. I look forward to any updates that you have time for.

Thank you, tompius.

I agree that a sticky is in order.

Barbara Allen and Wild Mountain Thyme on the 22nd. It was a short show, hotter than he11 but very memorable.

Thank you for sharing!

One of the things I am most interested in (on this already interesting thread) is seeing or hearing how tompius' numbers line up next to folks' personal experiences and the actual mood of the songs on the set list for that show.

I want to know how it's all interrelated

For those interested in making set-list forecasts, here follows a list of songs ranked by their PM value (see first post on this thread for definition of PM).


1.000 Ballad Of A Thin Man
1.000 Like A Rolling Stone
0.995 Highway 61 Revisited
0.994 Thunder On The Mountain
0.984 All Along The Watchtower
0.944 Things Have Changed
0.909 Leopard-Skin Pill-Box Hat
0.653 Tangled Up In Blue
0.580 Honest With Me
0.505 Desolation Row
0.405 The Levee's Gonna Break
0.398 Mississippi
0.386 Summer Days
0.367 Forgetful Heart
0.298 Blowin' In The Wind
0.288 A Hard Rain's A-Gonna Fall
0.283 Simple Twist of Fate
0.259 It's All Over Now, Baby Blue
0.251 Don't Think Twice, It's All Right
0.232 Spirit On The Water
0.228 It Ain't Me, Babe
0.221 Beyond Here Lies Nothin'
0.204 Tryin' To Get To Heaven
0.201 Man In The Long Black Coat
0.198 High Water (for Charlie Patton)
0.186 Blind Willie McTell
0.139 Jolene
0.131 Not Dark Yet
0.131 The Lonesome Death Of Hattie Carroll
0.127 Ballad Of Hollis Brown
0.125 Visions Of Johanna
0.099 John Brown
0.099 Make You Feel My Love
0.092 Love Sick
0.085 Girl From The North Country
0.080 Forever Young
0.067 Sugar Baby
0.065 Rainy Day Women # 12 & 35
0.061 To Ramona
0.051 Cold Irons Bound
0.043 Shooting Star
0.039 Tweedle Dee & Tweedle Dum
0.034 Nettie Moore
0.032 Just Like Tom Thumb's Blues
0.031 Rollin' And Tumblin'
0.029 Can't Wait
0.025 Gonna Change My Way of Thinking
0.024 Boots Of Spanish Leather
0.023 Watching The River Flow
0.019 Workingman's Blues #2
0.018 I'll Be Your Baby Tonight
0.016 This Wheel's On Fire
0.014 If You Ever Go To Houston
0.006 Just Like A Woman
0.006 Love Minus Zero/No Limit
0.005 I Don't Believe You
0.005 Stuck Inside Of Mobile With The Memphis Blues Again
0.004 Señor (Tales Of Yankee Power)
0.003 The Man In Me
0.003 My Wife's Hometown
0.002 Every Grain Of Sand
0.002 Masters Of War
0.002 When The Deal Goes Down
0.001 Til I Fell In Love With You
0.001 When I Paint My Masterpiece
0.001 Ain't Talkin'
0.001 I Dreamed I Saw St. Augustine
0.001 Lay Lady Lay
0.001 Shelter From The Storm
0.001 Most Likely You Go Your Way
0.001 Gotta Serve Somebody

I wouldn't expect more than one song with a PM value below 0.2 to be on tonight's show. So choose 15 from the first 24 listed and one wild card. In fact it might be wiser to just choose all 16 from the first 24. The first six look like permanent parts of the set now.

Then again, who knows when a big change will come.

Yes, as petitioned here, whoever is in charge of assigning stickiness, we implore you to please put glue on this one. Thank you good moderators!

The PI for last night's show is 0.607, down from the 0.629 of the first night. If it hadn't been for the fact that he played Rainy Day Women, it would have gone up to 0.638. Yes, Rainy Day... is now a relatively rare song - see my list of PM values in earlier post.

By the way the updated ranking of songs by PM value (most likely to be played based on past frequency) is as follows:

1.000 Ballad Of A Thin Man
1.000 Like A Rolling Stone
0.995 Highway 61 Revisited
0.994 Thunder On The Mountain
0.985 All Along The Watchtower
0.947 Things Have Changed
0.913 Leopard-Skin Pill-Box Hat
0.670 Tangled Up In Blue
0.601 Honest With Me
0.479 Desolation Row
0.417 Summer Days
0.384 The Levee's Gonna Break
0.378 Mississippi
0.349 Forgetful Heart
0.324 A Hard Rain's A-Gonna Fall
0.319 Simple Twist of Fate
0.288 Don't Think Twice, It's All Right
0.283 Blowin' In The Wind
0.270 Spirit On The Water
0.260 Beyond Here Lies Nothin'
0.246 It's All Over Now, Baby Blue
0.226 Blind Willie McTell
0.216 It Ain't Me, Babe
0.194 Tryin' To Get To Heaven
0.191 Man In The Long Black Coat
0.189 High Water (for Charlie Patton)
0.132 Jolene
0.124 Not Dark Yet
0.124 The Lonesome Death Of Hattie Carroll
0.120 Ballad Of Hollis Brown
0.119 Visions Of Johanna
0.094 John Brown
0.094 Make You Feel My Love
0.087 Love Sick
0.080 Girl From The North Country
0.076 Forever Young
0.064 Sugar Baby
0.062 Rainy Day Women # 12 & 35
0.058 To Ramona
0.049 Cold Irons Bound
0.040 Shooting Star
0.037 Tweedle Dee & Tweedle Dum
0.032 Nettie Moore
0.031 Just Like Tom Thumb's Blues
0.029 Rollin' And Tumblin'
0.027 Can't Wait
0.024 Gonna Change My Way of Thinking
0.023 Boots Of Spanish Leather
0.022 Watching The River Flow
0.018 Workingman's Blues #2
0.017 I'll Be Your Baby Tonight
0.015 This Wheel's On Fire
0.014 If You Ever Go To Houston
0.005 Just Like A Woman
0.005 Love Minus Zero/No Limit
0.005 I Don't Believe You
0.004 Stuck Inside Of Mobile With The Memphis Blues Again
0.004 Señor (Tales Of Yankee Power)
0.003 The Man In Me
0.002 My Wife's Hometown
0.002 Every Grain Of Sand
0.002 Masters Of War
0.001 When The Deal Goes Down
0.001 Til I Fell In Love With You
0.001 When I Paint My Masterpiece
0.001 Ain't Talkin'
0.001 I Dreamed I Saw St. Augustine
0.001 Lay Lady Lay
0.001 Shelter From The Storm
0.001 Most Likely You Go Your Way

This was shocking...until I read you did this at work. I could have created a new civilization with the time I spend doing 'other things' at work.

Another night, another ho hum set list. The PI value is 0.618. The PM values (as they were prior to the concert) for the individual songs are as follows:

Leopard-Skin Pill-Box Hat (0.913)
It's All Over Now, Baby Blue (0.246)
Things Have Changed (0.947)
Tangled Up In Blue (0.670)
Beyond Here Lies Nothin' (0.260)
Spirit On The Water (0.270)
High Water (for Charlie Patton) (0.189)
Desolation Row (0.479)
Honest With Me (0.601)
Simple Twist Of Fate (0.319)
Highway 61 Revisited (0.995)
Man In The Long Black Coat (0.191)
Thunder On The Mountain (0.994)
Ballad Of A Thin Man (1.000)
Like A Rolling Stone (1.000)
All Along The Watchtower (0.985)
Rainy Day Women # 12 & 35 (0.112)

Again Rainy Day Women is the freshest long in the set. Although I've a feeling that the PM value for it will rise steadily in the days ahead as it becomes the fixed encore.