Episode Notes
What’s involved in forecasting the Federal Budget, COVID-19 daily case numbers, or Australia’s electricity needs?
Join expert Professor Rob Hyndman as he explains the art of prediction. This episode also features guests Jehan Ratnatunga (Who Gives A Crap) and leading economist Stephen Koukoulas.
Jehan Ratnatunga (00:00):
It was early March, 2020, and the world was wondering what was going on with coronavirus. And we started noticing our toilet paper sales going up and up and up.
Ginger Gorman (00:13):
Remember that? It was one of the stranger reactions to COVID-19. Australia’s first COVID-19 death was reported on the 1st of March, 2020. And then the next day, we saw our first two cases of community transmission. This was no longer a thing happening to people overseas. And what did we do? We panicked and bought pretty much every single available roll of toilet paper in the country.
Jehan Ratnatunga (00:41):
Initially, we thought, “Oh, today’s going to be a big day. It looks like it’s going to be maybe double our usual day.” Then, we kept an eye on it. A few hours later, it looks like it was going to be triple, quadruple. And we suddenly had a massive spike. We were more than 10X our usual sales day. And we were completely sold out in all our sites across the world within the day.
Ginger Gorman (01:04):
That’s Jehan Ratnatunga, one of the co-founders of Australian social enterprise, Who Gives A Crap.
Jehan Ratnatunga (01:11):
I think it was all happening in parallel with what was going on in the supermarkets. And in fact, I think there was a little bit of a feedback loop going on there where people would go to the supermarkets and suddenly see everything was out of stock and then jump online. So we saw a huge spike in search traffic coming to our site.
Ginger Gorman (01:27):
It seemed as though there was many people outraged by the panic buying as there were people actually doing the panic buying. And seriously, why toilet paper? It’s not as though COVID-19 was a gastric flu or something. But a toilet paper retailer probably has a pretty good insight into how their customers think. So was Jehan surprised?
Jehan Ratnatunga (01:52):
Initially, we were as confused as anyone else, which was like why of all things would toilet paper be running out right now? But as we thought about it more, it’s sort of we can understand there was a few things going on. The first one was because it’s so big, it would be out of stock on the shelf really quickly because it’s hard to put a lot of items on the shelf. So it looks like there’s less of it than there really is. And secondly, it’s one of those staple items that feels a little bit more on the survivalist shopping list than other things. And so, people obviously wanted some stability. And at that point in time, nobody really knew what was … how all of the lockdowns are going to unfold. So it sort of started to make sense why toilet paper was getting so much air time.
Jehan Ratnatunga (02:39):
In normal circumstances, the way we operate is we make sure we have enough supply at our various warehouses to last a few extra weeks, but generally, we try not to be overstocked because that would really put a strain on our cashflow. So we do manage it to a level and we’re looking at that every week.
Ginger Gorman (02:58):
What Jehan’s talking about there is the so-called just-in-time model of forecasting. It’s a common practice, particularly for businesses that sell physical stuff. The last thing they want to do is order too much stuff because storing stuff costs money or maybe it’s perishable or trends change so it might just go out of favor with customers. Instead, they use forecasting to predict how much of an item they’ll need for a limited amount of time.
Ginger Gorman (03:28):
So really just like when you do your grocery shopping, dry pasta and tinned tomatoes don’t perish, but they do take up space in your pantry. So if you make spaghetti bolognese once a week, for example, but it’s also your go-to meal when you can’t decide what else to make, you might then stock up on a few extra tins of tomatoes and bags of pasta just so it’s handy at any time you want it. But you’re not going to buy four bags of pasta every single week if all you ever use is one or two bags a week.
Ginger Gorman (04:03):
So retailers use forecasting and households use forecasting, but who else needs it? And just how much does forecasting predict and mould the world that we live in? This is Seriously Social. I’m Ginger Gorman. And on the podcast today, we are exploring the dark art of forecasting.
Professor Rob Hyndman (04:32):
Going back, many centuries back in the days of Constantine, forecasters were banned. And even only a few centuries ago, about 300 years ago in England, it was illegal to charge money to make forecasts. And the idea was that they were trying to stamp out sort of charlatans in fair grounds who were ripping people off by making money out of their forecasts. But actually, what I do is I do forecasting and some people pay me [inaudible 00:05:02]. So I’m glad those days are over.
Ginger Gorman (05:06):
Rob Hyndman is a professor of statistics at Monash University and also a fellow of the Academy of the Social Sciences in Australia. When Rob goes to dinner parties and tells people he’s a forecaster, most people think he’s a meteorologist, but forecasting is vital, not just in retail, but in pretty much every aspect of our lives. Take energy, for example.
Professor Rob Hyndman (05:29):
I’ve done quite a lot of work on energy forecasting so that’s forecasting the amount of electricity that people will use. That’s an interesting problem because it’s affected by human behavior. It’s also affected by changes in technology as people use different sorts of electrical devices or that there’s different sorts of generation capacity. And it’s also affected by the weather. On a hot day, people are using air conditioning and so energy rates go up. So there’s a few different things that contribute to that, which makes it interesting I think from a forecasting point of view to try to untangle the various inputs that might affect energy demand.
Ginger Gorman (06:06):
Why do you think it’s necessary in terms of society? Why can’t we just trod along without knowing how things are going to pan out without the mathematical models that you’re making?
Professor Rob Hyndman (06:16):
Well, in terms of energy demand, it’s important both in terms of having the capacity in the short term, making sure that enough energy is being generated so that tomorrow, people won’t have blackouts, but in the long term, it’s also necessary in terms of planning generation capacity like do we need to build another wind farm or is there going to be a gap in the supply in 20 years’ time based on our current projections?
Ginger Gorman (06:41):
Another really valuable application of forecasting and one that Rob has been working on recently relates to COVID-19, but Rob isn’t helping retailers and companies like Who Gives A Crap determine how much toilet paper they need in their stores. He’s actually helping forecast the number of cases for Australia’s state and territory governments.
Professor Rob Hyndman (07:02):
Yeah, that’s a very interesting problem because we’re needing to forecast daily case numbers in each state up to a few weeks ahead. We have a team of people, most of whom are epidemiologists, but includes people like me who’s an expert in forecasting, but not epidemiology. And we have three different models that are being used. One is a epidemiological model based around the standard SEIR approach to epidemics.
Ginger Gorman (07:31):
Okay, stop there for a second. SEIR stands for Susceptible-Exposed-Infectious-Recovered. Got it?
Professor Rob Hyndman (07:39):
Then we have a model which is agent-based, which is sort of trying to model individual behavior. And then, we have a time series model, which is tries to build a model based on all of the data from other countries around the world, but doesn’t take into account the dynamics of the pandemic. So it uses more data, but it’s not so smart in some ways. And the three models capture different parts, different aspects of how COVID-19 is evolving. And we put them together in a forecasting ensemble so that we get the benefit of all of the three different perspectives on it. And the forecast ensemble is what goes to the state governments and the national government every week to advise them on where we think things are moving and so that they can take appropriate policy responses.
Ginger Gorman (08:25):
Have you got a terrible feeling with something like a pandemic because it’s so serious and the outcomes are potentially so serious that you might get the modeling wrong?
Professor Rob Hyndman (08:33):
Well, I mean, when you’re forecasting, there’s always the chance that you’ve screwed it up somewhere. And when you’re doing short-term forecasting, you will know if you’ve screwed it up because the reality will come to pass very soon. In the case of the COVID-19 forecasts, even the large Victorian outbreak was within what we thought was possible. So we don’t think we’ve screwed it up. We think we’ve actually done a reasonable job at predicting the range of possibilities that could occur in each state over the last year.
Ginger Gorman (09:04):
So these are a couple of really valuable uses of forecasting, but how do forecasters plan for unpredictable stuff like once in a century floods or the global financial crisis or this global pandemic, the stuff we like to call Black Swan events?
Professor Rob Hyndman (09:25):
With a pandemic, it’s not a true Black Swan event because there have been other pandemics in the past. There was the Spanish flu back in 1918-1919. There’s been the various smaller epidemics such as swine flu and bird flu and so on over the last 20 years. So, it should’ve been expected. The problem is that a lot of planning ignored the fact that there could be such a thing as what we’re currently experiencing.
Ginger Gorman (09:52):
So that’s interesting, what you’re saying, that a lot of forecasting doesn’t actually take into account possible uncertainties or possible huge shocks or rare events.
Professor Rob Hyndman (10:06):
Exactly. The same thing happened in the global financial crisis. What happened should not have been unexpected. There have been the Great Depression. There’ve been big shocks to the financial systems in the past, but the problem is a lot of people were modeling based on small data sets that didn’t take account of things like a major financial shock. And so, they weren’t taking that into account in their forecasts and weren’t taking action based on the fact that such a thing could occur.
Ginger Gorman (10:36):
I know you also have done a lot of modeling in regard to the Pharmaceutical Benefits Scheme and the federal budget. And I think what’s interesting about that is politicians want a certain outcome. So in those sorts of circumstances, are you getting forecasting that’s not based on mathematical models, but maybe based upon something else?
Professor Rob Hyndman (11:00):
So parts of the budget are done very well. These days, the Pharmaceutical Benefits Scheme is forecast extremely well. It wasn’t the case 20 years ago. There was serious under-estimates of the amount of money that needed to be put aside to pay for the PBS. I got involved with that in about 2002 and developed some new models that have been used ever since and we now have pretty reliable forecasts. They’re genuine forecasts in that they’re the average of what could happen in a range of [futures 00:11:29].
Professor Rob Hyndman (11:29):
But other parts of the budget are clearly not true forecasts. If you look at the forecasts of the balance of payments, for example, against what actually happened for each of the last 15 years, they’re almost always optimistic forecasts and so it would seem that the Treasury has been asked to provide a level of optimism rather than a genuine forecast. I call these hopecasts rather than forecasts. It’s what the government hopes would happen rather than what they genuinely believe will happen.
Stephen Koukoulas (12:04):
There’s this tricky balance between forecasting what we think will happen based on assumptions on the Chinese economy, the US, on interest rates and all these other things versus getting an optimistic slant on it. I won’t call it spin, but an optimistic slant so that the treasurer can stand up on budget night and say, “Hey, aren’t we good economic managers? We’re delivering these sorts of numbers.”
Ginger Gorman (12:28):
That’s Stephen Koukoulas, formerly the chief economist at Citibank and a senior economic advisor to Prime Minister Julia Gillard, and now an economics and political commentator. He’s been following the hopecasting phenomenon for years.
Stephen Koukoulas (12:45):
You’d probably say that it was the Swan budgets in around 2010 or 2011, so just after the GFC. When we had the recovery, Mr. Swan as treasurer sort of got up there and said these four budget surpluses I deliver “on budget night” and lo and behold, within a few months of that, the economy took a bit of a dip down. That cost a lot of revenue to the bottom line of the budget. Commodity prices actually fell as well. So the things that drive a lot of government revenue, which helped create a good budget and helped you forecast a good budget position very quickly unraveled.
Stephen Koukoulas (13:24):
And that continued with Joe Hockey when he was treasurer in 2013, ’14, ’15 and, of course, probably in the very recent past most famously when Treasurer Frydenberg said we’re back in the black and he had the coffee mugs that were printed with the Back in the Black symbols, forecasting 10 years of budget surpluses, and they have zero net government debt by 2030, and lo and behold, here we are a little bit out from the next federal budget on May, the 11th and those numbers look like we’re heading for government debt of over a trillion dollars.
Ginger Gorman (13:57):
But the thing is you can’t really forecast what a global pandemic is going to do to a budget.
Stephen Koukoulas (14:05):
That’s correct, of course, and I think we need to for both Mr. Swan and Josh Frydenberg, things outside their control came along. This is the interesting thing about economic forecasting or any forecasting for that matter. If you put your hand on your heart and you say, “This is my best estimate for GDP growth, for unemployment rate, for inflation, for wages and, therefore, company profits and the budget bottom line, and along comes a pandemic, along comes a crashing iron ore prices because China does something with its economy or that undermines even the best intentioned forecasts that are made.
Ginger Gorman (14:44):
Well, you’ve worked in politics. You’ve worked inside government. Are these kind of pressures brought to bear on someone like yourself who’s sort of a serious economist, and you’re trying to do a decent and true forecast, but then you’ve got the politics of the day. They may want you to under forecast, let’s call it, or hopecast?
Stephen Koukoulas (15:08):
Yeah, that hopecasting is an interesting concept, but I quite like the notion of it, too. But look, in my experience, and I spent a few years in treasury, a few years in Prime Minister Gillard’s office. And my observation was that there was not much pressure at all. They wanted to get the numbers right. And if you get the forecast wrong as the treasurer, you look pretty silly. So the pressure, it has not been overtly strong to say, “Hang on. Let’s just put in a strong number here and make sure that we are forecasting a surplus.”
Ginger Gorman (15:40):
But what about that circumstance where, in fact, politicians might under forecast and say they might ask their numbers people to predict a very small surplus. So in fact, when there’s a big surplus, it looks fantastic as a headline in all the newspapers?
Stephen Koukoulas (15:56):
Yes. And curiously enough, we might even have to go back to Peter Costello was treasurer, because if we think back to that period when mining boom mark one started in about 2002-2003, the budget had just got into approximate balance. There were tiny surpluses. It was basically a balanced budget. And all of a sudden, commodity prices were very, very strong. The Chinese economy was growing at 10 or 11% per annum. Wow. And the nature of their growth meant that they required a lot of raw materials. It was an industrial boom, not a consumption boom, if you like. So all of the construction needed iron ore, energy, steel, the stuff that we produced.
Stephen Koukoulas (16:40):
And what that meant to the budget numbers were always better and I think that as we think back to that election campaign in 2004, and even in 2007, the one that [Costello Howard 00:16:49] lost, they were talking about aren’t we good economic managers ’cause we’re only forecasting a surplus of a few billion dollars, and they came in at 20 billion and were giving you a tax cut to boot. So, in a sense, if you were just a casual observer, which most of the electorate are and you saw tax cut, tick, big budget surplus, tick, and even a bigger budget surplus than we expected or were forecasting, tick. And that’s how that narrative, I think, got into our economy where we unfortunately to this day, even though we’re a little bit less obsessed than we used to, we’ve still got this obsession with debt and deficit and concerns that big budget deficits are bad, budget surpluses are good, which of course is wrong.
Ginger Gorman (17:36):
While he says it doesn’t really make sense to pressure government forecasters to favor optimism over accuracy, he does see it happen within the lobbying industry.
Stephen Koukoulas (17:46):
That’s, and actually, which I find quite disgraceful, to be honest, most of the time, there are a few exceptions, but for people who are wanting to lobby the government for a particular policy or a tax break or subsidies, whatever the cause may be, but it’s basically getting money into their pockets of the people who sponsor them, there’s been a trend in recent years for those lobbyists to get economics firms to be producing economic forecasts that will give them the outcome that they want.
Stephen Koukoulas (18:20):
So for example, what’s the economic consequences of putting a price on carbon, for example? Well, some people could come and say, if you’re in the coal industry and you got one of the wonderful economic consultancies to prepare a report, you would bet your bottom dollar that they’ll come up with a report that says, “Oh, the carbon price is very bad for the economy. And therefore, we shouldn’t have it,” even though the coal industry sponsored that research and those forecasts. And again, imagine going to the coal company going to a consultancy firm in that instance, and the consultancy firm came back and said, “Well, actually, it’s going to benefit the economy, we have a carbon price.” The coal company would say, “We didn’t pay you to produce that report. We paid you to produce a different report.”
Ginger Gorman (19:04):
How much impact do you think reporting like that has on government policy?
Stephen Koukoulas (19:12):
I think it has some influence. And I look back on the lobbying done for the Mining Resource Rent Tax. Think back to the Rudd government, Super Profits Tax I think it was called at one stage when they were talking about it. The mining industry got all of these consultancy firms to produce all these reports saying, “Oh, it’s going to mean the end of the mining industry. We’ll be unprofitable. We’ll lose thousands of jobs and it’s going to really hurt the industry.”
Stephen Koukoulas (19:39):
Whereas we know just recently, the Parliamentary Budget Office just in the last few weeks has presented a report that said that if the mining tax … Of course, the mining tax was abolished when Mr. Abbott won the election in 2013. If the mining tax was still in place and we know at the moment, the iron ore price is staggeringly high, about 160, 170 US dollars a ton, if that tax was still in place, the federal government, even after allowing for payment of company taxes and these sorts of things would’ve had an extra $33 billion.
Stephen Koukoulas (20:12):
And at the moment, when we’re arguing about childcare payments, the Job Seeker level. We’re arguing about even the budget bottom line, the deficit, $33 billion, that’s a staggering amount of money. So that’s where one lobby group through what you might call dodgy forecasting, dodgy economic modeling, produced a report that struck the nerve of both the politicians, obviously. But I remember seeing the ads on TV, oh, the mining industry, a few people with hard hats and [fluorite 00:20:43] jackets on saying how bad that Super Profits Tax was. So it can have a big effect on the community and it can have a big effect on the government. And for the sake of a $30 million advertising campaign for the mining companies, they’re roughly a decade ago, they’ve saved themselves $30 billion.
Ginger Gorman (21:07):
If everything from the federal coffers to COVID cases to energy use can be forecast, does that mean that there are a multitude of different types of mathematical models out there that forecasters use? Can there ever be a one size, fits all approach to forecasting?
Professor Rob Hyndman (21:24):
You do need to build a set of models that are appropriate for the thing you’re trying to forecast, but it’s actually surprising how generic some forecasting methods are. So the methods that I developed for the Pharmaceutical Benefits Scheme are now used by Coles, for example, for retail sales. Both situations have trend and seasonality patterns. And so, as you can describe those mathematically and then project them forward, you get a decent forecasting. But other things are very different, like the models we’re using for COVID-19 forecasting are very different from other models because they’re particularly designed to handle the sort of the dynamics of a pandemic rather than the dynamics of sales or fertility rates or something like that.
Ginger Gorman (22:10):
You are really passionate about forecasting for social good. And I know you’re doing some really interesting work, for example, at the moment around ambulances and how many ambulances different communities might need at different times.
Professor Rob Hyndman (22:26):
The work with ambulance is interesting. We’re doing some work with the Welsh Ambulance Service at the moment in trying to forecast that daily demand for different types of call-outs. And we’re hoping that we can improve the way the ambulance service is run by providing them with better forecasts.
Ginger Gorman (22:43):
Why haven’t ambulances got good forecasting? We’ve needed ambulances forever, as long as there’ve been wheels, basically.
Professor Rob Hyndman (22:51):
So a lot of organizations use sort of old methods that have been developed internally over time and have maybe been good enough, but are not necessarily the best they could do. And so, one of the things we like to do is to identify problems where they maybe haven’t got the resources allocated to that particular area of their business and where we think that the methods that are currently in use could easily be improved. And so, we identified emergency call-outs as an area where we think there’s good data available, but the data is not being used as effectively as we think it can be.
Professor Rob Hyndman (23:32):
The idea is that we want to be able to forecast the demand in different regions and for different types of problems so that they can have the right ambulances in the right places at the right time. And because we’re looking at the range of possible futures that could occur, we’re looking at how likely is it that you would need three MICA ambulances in this region on the weekend? So we’re looking at the probabilities and then you’d try to allocate resources as effectively as possible to minimize the risk to the community.
Ginger Gorman (24:06):
And are you building into that modeling something like a pandemic, which could have a sudden shock and a lot of sick people that you haven’t previously expected?
Professor Rob Hyndman (24:18):
That’s a really good question. At the moment, we’re just using the data that they’ve given us, which is only a few years, but obviously if there’s a major natural disaster, there could be a mining disaster in Wales, there could be an earthquake in some other regions of the world, then there’s going to be a whole different range of resources needed that our model’s probably not going to deal with.
Ginger Gorman (24:48):
For some of the more generalized forecasting activities, Rob has developed open source software. Basically, he built the software when he was doing his consulting work on the PBS, and it occurred to him that it might be useful to others in the forecasting community. Well, maybe he should’ve put a price tag on it because it’s been downloaded more than three million times and is used throughout the world.
Professor Rob Hyndman (25:14):
Over time, it’s become more and more popular to the point where I think it’s probably the most widely used forecasting software in the world. It contains the sort of methods that you can apply to a range of situations, but it doesn’t contain highly specialized methods that have been developed for, for example, for COVID-19. But the sort of methods that are in there work great for retail sales or for energy demand or for demand for different services. And so, any organization that hasn’t got the specialist skills to build their own forecasting models can just take that software and apply methods that have been very widely tested over a very long period of time and know they’ll get pretty reliable forecasts.
Ginger Gorman (25:56):
So why did you give it away for free given that you’d probably be a millionaire by now if you’d ask people to pay for it?
Professor Rob Hyndman (26:03):
I don’t know that people would have bought it. Everybody who needs to forecast should be able to do so without needing a huge amount of money. So organizations who may be nonprofit or people who are working in developing countries that are maybe not well-resourced financially can still have access to high quality forecasting resources.
Ginger Gorman (26:29):
So how do you forecast for the short term in unpredictable times? It’s something that Jehan and the team at Who Gives A Crap had to figure out very quickly when their customer base shot up during the toilet paper shortage. But at the same time, it really gave the company a chance to test themselves, to see if they really can be as big as the Aussie startup aspires to be.
Jehan Ratnatunga (26:56):
We have high aspirations for the business. And so we always have our sights on getting to larger numbers of customers. That’s the foundation behind the business is if you can take a product like toilet paper, which many, many people use and donate a portion of that to charity, then you really do need the scale to be able to solve these big problems in the world. So that really is part of our thinking.
Jehan Ratnatunga (27:21):
So there we were, we had just sold out in every warehouse around the world, and we knew that something completely unprecedented was going on. And the first thing we did was we put up a message on our website to try and reduce the panic that was going on. It was just saying, “Hey, we’re out of toilet paper right now, but join our wait list. We’ll let you know as soon as we have new toilet paper and PS, if you’re a subscriber to Who Gives A Crap, we’ve made sure we’ve saved enough for you, guys.” And then, we watched this email list slowly grow, and we thought maybe we’ll get a couple of thousand people signing up to it. We ended up getting almost half a million people signing up to that wait list.
Jehan Ratnatunga (28:03):
And the next challenge became how do you then manage this waiting list as we got new stock coming into the warehouse eventually? That became a really interesting challenge for us. We built out what we think is a sophisticated model, but I’m pretty sure there’s far more sophisticated modeling going on there that looked at a few different constraints that we had, the first one being how much toilet paper we have in each site, each warehouse; the second one being what our logistics team can actually handle. There was sort of a ceiling there that we didn’t want to put too much strain on them; and the third one being how much our customer support team could handle because we knew that if we have a lot of spike in sales, that would generate a lot of customer support tickets, and we wouldn’t have enough staff to manage those tickets.
Jehan Ratnatunga (28:49):
So we had these three constraints that we were managing, too. And we would then take the amount of toilet paper we had for that particular week and we would email our customers from our wait just enough so that it would generate enough sales to not trip any of those three constraints. And we did this, it was sort of a daily routine for almost six weeks I think we were operating the business like that, which I do not recommend to anyone. That was a time when we asked our company to really dig deep and do some really difficult things. But for us, we all recognized as a company that this is a chance to really have a big impact and end the year with a really, really large donation.
Jehan Ratnatunga (29:30):
And that really rallied everyone in the company behind this moment, despite all the uncertainty that they were experiencing in their everyday lives. So the thing that we learned and the thing that we’re most proud to do is that this mission to make sure everyone in the world has a toilet, the impact behind our business is both like a north star in the good times, but it’s also a really strong anchor in the more uncertain times. And that was really something we never thought about it in that way and really came through during the panic buying period.
Ginger Gorman (29:59):
Thanks for listening to Seriously Social. I’m Ginger Gorman. If you’re enjoying the podcast, make sure you check out our website, seriouslysocial.org.au for more content like articles and videos on the amazing work of Australia’s leaders in the social sciences. That’s it for this season, but I will be back in a few weeks with a four-part mini-series revisiting some of our older episodes from 2020 to find out how the information from those experts holds up with 2021 hindsight. See you soon.
Useful Links
- Who Gives A Crap
- What’s the pandemic forecast? Australian Research Council
- Forecasting: Principles and Practice (3rd Ed) Rob J Hyndman and George Athanasopoulos
- The Kouk Stephen Koukoulas
BONUS EP: Recession revisited
