Thursday, August 1, 2013

Putting the Anal in Analysis. Why SIA's newly released snowsports participation study is a bit crap.

 

I'm a bit concerned that the people who analyse the snowsports industry are a bit crap at their jobs. I guess if they were good they'd be working in banking...

Earlier this week SIA (Snowsports Industry America) released their yearly participation study which you can buy now for a hefty $899. This was opening section and chart from their press release:

The snowfall got a late start this season, really getting underway after the Christmas holiday. The lack of early season snow affected all snow sports except freeski and telemark, which both finished the season with more participants than previous years. Ultimately, overall snow sports participation was down 3% to 19.3M participants across all disciplines; alpine ski had a 19% drop in participation but still brought the most to the mountain with 8.2M, followed by snowboard with 7.4M, freeski with 5.4M, snowshoe with 4M, cross country with 3.3 million and telemark with 2.8M participants.


Here's another way of looking at those results...

The growth in freeskiing, and telemarking is great news for those specialities but not much use for skiing as a whole, because they both grew at the expense of traditional downhill skiing. Taken as a whole snowboarding and skiing (the dotted line) are showing pretty much the same trend with a slight drop since the 2009/2010 season.

The data they used here was also released a few months ago by the Outdoor Foundation who used the same survey to look at the US participation across all types of sports and activities (it's why my chart above has an extra season of data). Here's how those snowsports increases and decreases fit into the bigger picture.

Hunting with handguns is up 17%. Apparently there are now almost as many people running about the US firing off handguns at anything that moves, as there are surfing. 

Small numbers provide less accurate results

The SIA press release doesn't mention that it's just a subsection of this much larger study and it makes a difference. Here's how the original study was conducted:

During January and February of 2013 a total of 42,363 online interviews were carried out with a nationwide sample of individuals and households from the US Online Panel of over 1 million people operated by Synovate/IPSOS. A total of 15,770 individual and 26,593 household surveys were completed. The total population used was 287,138,000 people ages six and older. 

42,363 sounds like a thorough survey and for the benefit of not turning into the most pedantic wanker ever I'm going to assume the survey itself was well conducted. When you get into the individual sports though, the results are less robust. Let's take one of the sports from the study, freeskiing, as an example.

Freestyle skiing last season apparently had 5,357,000 participants, that's 1.87% of the US population. With a survey of 42,363 that means that just 792 people surveyed participated in freeskiing. Each person in the survey accounts for 6,778 people in the real world and it doesn't take a lot of people to make a huge difference in these figures. The difference between the 2010/11 and 2011/12 season figures, for example, is just one person. All it takes is a small difference in the group being sampled each year for the results in these specific sports to vary a great deal.

Just under 4,000 people in the survey were involved in any of these snowsports. How accurate does that make the report when you focus the results just down to those sports? A lot less than the 42,363 you think you're getting for your 900 bucks.


Correlation is not causation

So there are the numbers and even if it's a smaller survey than you might think you're getting it is still enough to provide some useful guidance, as long as you ignore the analysis provided by SIA that is. Going back to the opening sentence of that press release:

The snowfall got a late start this season, really getting underway after the Christmas holiday. The lack of early season snow affected all snow sports except freeski and telemark, which both finished the season with more participants than previous years. 

Why would late snowfall reduce the number of alpine skiers but increase the number of freestyle skiers or telemarkers? That's a pretty big guess to make on a survey of 4,000 people. Late snowfall might account for something like this overall drop, but it doesn't explain the redistribution. Did the seasonal snowfall really have any impact at all?

Talking about bankers, here's a little story from the day that Saddam Hussein was captured.

Immediately after he was captured Bloomberg News released this headline: US Treasuries Rise; Hussein capture may not curb terrorism

Half an hour later the market moved and the analysis completely changed: US Treasuries Fall; Hussein capture boosts allure of risky assets

This was a classic case of applying a nice and simple story to a complicated situation, two analyses that both couldn't be true at the same time. People have a tendency to look at numbers and create stories even when the situation is far too complicated to do so and this is what SIA have done here.

To put it another way, a friend of mine illustrated this problem because he wanted to explain this concept to his clients who were making the same mistake. I'm going to use his graphic because I'm too lazy to create a snowboard specific version.

There is a correlation between the number of ice creams sold (snowfall) and the number of shark attacks (people participating in snowsports), but the increase in ice creams sold (a change to a late snowfall) didn't cause the increase in shark attacks (decrease in snowsports participation). There are a whole load of factors not shown in these two lines on a graph and it's the same in the SIA results.

Snowsports in general could have fallen for a whole number of factors that sound reasonable - They could have fallen because of the weather conditions (although the significant late season snow would have pushed this factor the other way too), but they could have also fallen because of the higher costs of transport, higher ticket prices, fewer children getting involved, the current economic climate, the high price of getting started, increased competition from alternative sports, fewer women getting involved, skiers and snowboarders getting older, the population becoming more obese and so on. I could come up with a plethora of other factors that correlate with the fall in snowsports participation, but they didn't necessarily cause the problem and they certainly weren't the only factor. In reality any significant change was most likely due to a raft of interacting factors that would be too difficult for one person to ever get their head around. It's nice to be able to find a simple explanation, but it's very misleading when you get it wrong.


In reality, all we can tell from this report there probably isn't much change overall, while skiing is diversifying so that freestyle skiing and telemarking are becoming more popular styles. That's it. If you plan to  run your business on anything more than that you need to know that you are just working from someone else's finger-in-the-air guesses.

Maybe the full report isn't so bad, but all the signs from the press release say otherwise. If they can't get that brief synopsis right, it makes me worry about the full-length version and I won't be spending $899 to find out.

Enough of that. Here's the history of freestyle skiing in 2 minutes.
How the fuck did that only come in 16th place?



You Might Also Like...

The new site design. It took bloody ages, so if you don't like the change feel free to keep it to yourself.

Reports of the death of snowboard are greatly exaggerated. Some even worse industry analysis by RRC Associates.

2 comments:

  1. And you asshole cant even rewrite the numbers correctly .. according to SIA report the snowshoers for 2008/2009 is 2.9 not 4.9 .... dumbass

    ReplyDelete
  2. Those numbers and that chart are copied directly from SIA's press release (the link is just above the chart). Please feel free to direct your unbridled fury in their direction.



    Also, who really gives a shit? This isn't an article about snowshoeing. Feel free to do one.

    ReplyDelete

 
© 1896. Design by Main-Blogger - Tinkering by Zhang - Colouring in by Illicit