According to Intensity’s latest US recession start date forecast, there is a 50% probability of a recession starting sometime in the January to February 2019 period.  And a 97% probability of it starting sometime within the next 6 months.

Their “point estimate” of a recession start is January 2019.

Like, as in, right now!

If true, it will take awhile for the impacts to start showing up in the official government statistics.  But the stock market sell-off last quarter may be a harbinger of things to come.

Intensity, an economics and data science firm based in San Diego, CA, developed and back-tested a machine learning prediction algorithm for its clients.  The firm started releasing a monthly forecast of the next US recession start date to the public starting in March 2018.

Over the course of the last 11 months, it has been interesting following the updates to their forecast as economic conditions changed.

Intuitively, one would expect that the forecast would “settle down”, the closer the expected start date became.

And it got me thinking about what the best way is to visualize these changing forecasts.

Visualizing Forecast Updates Over Time

The forecasted recession start date is not linear with time.  For example, in March 2018, the next recession was forecasted by Intensity to start in April 2019.  But in April 2018, the forecast was revised, and the recession was to start 6 months earlier in October 2018.

Plotting the month of the forecast on the x-axis and the forecasted month of the recession start on the y-axis yields a “traditional time series” view as shown below.

As time progresses from left to right, we can see the forecasted recession start date fluctuating up and down, settling on January 2019, the most recent forecasted start date.

However, another way to visualize this is to show the progression of time vertically, from bottom to top.  In this case the forecasted recession start date would fluctuate horizontally, left and right, as shown below.

I don’t know about you, but I find this second view more appealing.  Maybe it is the old economist in me, trained on the Phillips Curve in graduate school.  But for me, the vertical, “up-down” orientation makes the variation in the forecasted recession start date “pop” more than in the horizontal, “left-to-right” view.

So, Recession in 2019?

It will be very interesting to see if Intensity sticks to its January 2019 point estimate.  Prior to the unexpectedly positive December 2018 jobs report, the consensus seemed to be a recession starting some time in 2019 or 2020.  For example, Gary Shilling recently tossed his hat into the recession ring with a predicted 66% chance of a recession in 2019.

However, the positive jobs report apparently has many economists now softening their stance on a recession this year.  And there is talk of policy makers being able to sidestep a recession.

Only time will tell…so stay tuned!

Plotting Ordered Times Series in Tableau

By the way, these charts were made in Tableau.  And it was not as straight forward as flipping the axes to get the vertical view.  Tableau’s default inclination is to “connect the dots” from left to right when time is involved.

Fortunately, there is an easy way to get Tableau to connect the dots vertically.  This makes use of the Path property in the Marks card.  I simply added a field to my raw data that indicated the order of my data, which, of course was calendar order.

Then dropping this field on the Path property in the Marks card tells Tableau to connect the dots (or “Marks” in Tableau-speak) in this order.  With the date of the forecast on the vertical, y-axis, Tableau connects the dots from bottom to top.

Very slick!

The post How to Visualize Changing Recession Start Date Forecasts appeared first on KDD Analytics.

Predicting the future is an exercise in probability rather than certainty. As we have mentioned several times over the course of these articles, your forecast model will be wrong.

It is just a matter of how useful it might be.

A time series model will forecast a path through the forecast horizon, a “point forecast.” But this path is just one of the paths your forecast can take based on your estimated model.

Providing a sense of the uncertainty surrounding your forecast is an essential part of your job as a forecaster.

Forecast intervals

The standard approach is to provide the “forecast interval” for your forecast.

Typically, this is cast in terms of a 95% prediction interval. That is, 95 times out of 100, the actual value will fall within the specified range. (Note that there is a difference between a “confidence” interval and a “forecast” interval.)

Sources of forecast uncertainty

There are at least two sources of forecast uncertainty over the forecast horizon.

The first results from our ignorance of what the model’s error will be in the forecast horizon. So, we must rely on how well the model did in the recalibration sample (estimation + holdout) as an estimate.

The second source of uncertainty results from the model’s coefficients (or parameters) being estimates of their true values. As estimates, they have their own “confidence” interval.

As a result, the forecast interval can be quite large (as shown above). And, due to error compounding over time, the forecast interval widens the further into the forecast horizon you go.

In our example above, during the first month of the forecast horizon, the forecast interval is plus or minus 0.63% of the forecasted value. By month 6, this spread widens to plus or minus 2.95%.

Even accounting for forecast error and parameter uncertainty, these forecast intervals may still be too narrow.

What about meta forecasts?

In an earlier article we discussed combining forecasts into a meta forecast. The challenge in terms of a meta prediction interval is that it is not a simple matter to combine the prediction intervals of the constituents’ forecasts.

One approach is to simply show the extreme upper and lower forecast paths along with the meta forecast path, which will lie somewhere between the two extremes.

And then to caution the consumer of your forecast that this is just to give a sense of the possible forecast range, which will likely be too narrow (since the upper and lower forecast will each have their own prediction interval).

Probability-based assessment of forecast uncertainty

Another approach is to couch your forecast uncertainty in terms of a probability.

For example, based on your SALES forecast, what are the chances of hitting a certain level of sales by a certain date? If you are forecasting procurement needs for a warehouse, what is the chance of running out of inventory by a certain date? If you are a macroeconomist forecasting GDP, what are the chances of the economy falling into a recession by a certain date?

Suppose you are tasked with forecasting daily SALES over the next year.

Management has targeted a certain level of SALES and wants to know when that target will be hit. You can use the forecast uncertainty produced by your model to generate the following chart:

The vertical axis is the chance of hitting the SALES target by a certain date (in this case, days into the next year). So, 160 days into the year, there is a 10% chance of hitting the sales target.

By day 192, a month later, the chance has grown to 30%. And by day 218, there is a 50/50 chance the sales target will be reached.

Stating these chances in terms of odds may be an easier way to present this:

By day 160, the odds of hitting the target would be 9 to 1. By day 192 it would be a little over 2 to 1. And by day 218, it would be 1 to 1…a flip of the coin.

Bottom line

Uncertainty is a fact of life and your forecasts will be “wrong.”

But quantifying how wrong they can be will go a long way towards making them “useful.”

Part 1 – Practical Time Series Forecasting – Introduction

Part 2 – Practical Time Series Forecasting – Some Basics

Part 3 – Practical Time Series Forecasting – Potentially Useful Models

Part 4 – Practical Time Series Forecasting – Data Science Taxonomy

Part 5 – Practical Time Series Forecasting – Know When to Hold ’em

Part 6 – Practical Time Series Forecasting – What Makes a Model Useful?

Part 7 – Practical Time Series Forecasting – To Difference or Not to Difference

Part 8 – Practical Time Series Forecasting – Know When to Roll ’em

Part 9 – Practical Time Series Forecasting – Meta Models

The post Practical Time Series Forecasting – Bounding Uncertainty appeared first on KDD Analytics.

After an extensive model building and vetting process, along the lines we previously discussed here and here, the practical forecaster may still be left with several strong performing models.

These models perform similarly in the holdout sample tests. They retain their statistical properties when recalibrated on the full historical sample. But they yield different forecast paths over the forecast horizon.

Any one of the models could be easily defended. But the fact that the models yield different forecasts should make the forecaster pause.

An example

Below is an example of 3 short-run monthly forecasts:

The 3 models perform similarly in the holdout sample. One of the models is a least squares model. The other 2 are ARIMA models.

One model produces a steeply declining forecast. Another a slightly declining forecast. The third model produces an increasing forecast.

What should the forecaster do?

How can this happen?

Models are just that – models. They are abstractions from reality. And no single model will “fit” the holdout sample perfectly.

Two models, especially of different types (e.g. least squares vs. ARIMA), could have very similar holdout sample performance but differ dramatically in their forecast over the forecast horizon.

The holdout sample MAPE (mean absolute percentage error) could be very similar for these models. But the MAPE is an average error across the holdout sample. And the models could have arrived at their MAPEs by focusing on different aspects of the time series in the holdout sample.

Projecting these differences into the forecast horizon can result in very different forecasts.

Solutions

When there is no clear “champion” model, one solution is to combine the forecasts into one. We call this a “meta” forecast.

There are several ways this can be accomplished.

Checkpoint

But first, check to make sure the models to be combined are not “nested.” That is, one model is not a subset of another. If models are nested there usually is no advantage to combining their forecasts into a meta forecast.

In fact, a meta forecast will more likely be superior the greater the differences between the constituent models.

A meta forecast based on a least squares model and an ARIMA model will likely yield a smaller forecast error than that associated with either of the two models. However, if the two models were both least squares models, the superiority of a meta forecast might be questionable (Granger, 1989).

Solution 1

The simplest approach to arriving at a meta forecast is to simply average the forecasts of the individual models.

This essentially assumes that each model’s forecast is equally important in the meta forecast (i.e. receives equal weighting). This is a quick and uncomplicated way to generate a meta forecast.

Solution 2

Another approach makes use of each model’s holdout sample performance measures of forecast accuracy and bias. A weighting for each model’s forecast can be calculated using each model’s MAPE and MPE (mean percentage error) relative to that of all the models combined.

The meta forecast would then be a weighted average of the individual model forecasts. Models with lower MAPE and MPE would receive higher weights and contribute more to the meta forecast.

Solution 3

A third approach is to use regression to estimate the weights.

Using the holdout sample, or if too small, the full sample, regress the actual value on the forecasted value from each model. The goal is to find a regression with no constant and all regression coefficients positive and statistically significant.

The regression coefficients should then sum very close to one. These coefficients then become the weights by which forecasts are combined into a meta forecast (see Wilson and Keating).

Back to our example

The forecaster could go with candidate 3 since it “splits the difference.” However, the forecaster is still left with the task of defending why the other two equally plausible models were not chosen.

Alternatively, a meta forecast can be used. As an example, we created a simple average forecast across the 3 candidate models. As discussed above, this assumes an equal weighting across the 3 short-run forecasts. A more sophisticated approach would have been to estimate the weights using a regression approach.

Not surprisingly, the meta forecast is quite like the essentially flat forecast of candidate 3 (which lies almost half way between candidate 1’s and 2’s forecast). But not all cases will be like this.

If a regression approach to estimating the weights was used, the meta forecast could be quite different from that of candidate 3.

Yes, the meta forecast will lie between the two forecast extremes. But the assumed or estimated weights will dictate where the meta forecast will lie.

Bottom line

Combining forecasts from equally strong models is intuitively appealing since each model has its strengths and weaknesses.

Combining models’ forecasts in a complementary fashion should lead to more robust and accurate short-run forecasts.

Part 1 – Practical Time Series Forecasting – Introduction

Part 2 – Practical Time Series Forecasting – Some Basics

Part 3 – Practical Time Series Forecasting – Potentially Useful Models

Part 4 – Practical Time Series Forecasting – Data Science Taxonomy

Part 5 – Practical Time Series Forecasting – Know When to Hold ’em

Part 6 – Practical Time Series Forecasting – What Makes a Model Useful?

Part 7 – Practical Time Series Forecasting – To Difference or Not to Difference

Part 8 – Practical Time Series Forecasting – Know When to Roll ’em

The post Practical Time Series Forecasting – Meta Models appeared first on KDD Analytics.

Holdout samples are a key component to estimating a “useful” forecasting model. Set aside data at least equal in length to your forecast horizon (“holdout sample”). Build your models on the remaining data (“modeling sample”). And compare the candidate models’ forecast performance over the holdout sample.

At a minimum, a single holdout sample should be used.

But to get a better sense of a model’s future performance, consider using multiple holdout samples.

This guards against basing your model on a holdout sample that is unrepresentative of the overall characteristics of the time series.

One way to achieve this is to use “rolling” holdout samples.

Rolling analysis

A rolling analysis of a time series is generally used to test a model’s stability. That is, are a model’s parameters stable across time or do they change, especially in a systematic way?

This is important for a forecasting model. We don’t want a forecasting model whose parameters are changing during the forecast horizon in an unexpected (i.e. unmodeled) manner.

Suppose our forecast horizon is 6 months.

Under a single holdout sample, we would set aside the last 6 months of data as the holdout sample. Then using the remaining data as the modeling sample, estimate models, forecast over the single holdout sample and compare the models’ performance.

This will help narrow down the pool of candidate models.

Rolling holdout samples

But under a rolling holdout approach, also called “time series cross-validation,”  we would set aside a longer sample of data, say, the last 12 months. Then:

Step 1:  Estimate a model and forecast over the first 6-months of this 12-month period (“roll 1”);

Step 2:  Then add one 1 month to the tail-end of the estimation sample, recalibrate the model, and forecast over the subsequent 6-months (“roll 2”);

Step 3:  Then add another month to the estimation sample, recalibrate and forecast over the subsequent 6-months (“roll 3”);

Step 4:  Repeat until there are no more 6-month periods (“rolls”) remaining in the 12-month period.

So, in this example, we would have recalibrated our model 7 times (each with a modeling sample that is one additional month longer than the previous). And we would have made 7 forecasts over the rolling holdout periods.

The last “roll,” it turns out, is the same 6-month period we would have used under a single 6-month holdout sample case. So, we generate the stats for a standard single holdout sample during the course of this rolling holdout approach.

If we are examining multiple candidate models, this process can generate a lot of data. Below is an example of the rolling forecasts for one model.

Summary roll statistics

We could generate a similar chart for every model we are testing. But it is easier to work with measures of forecast accuracy and bias, such as MAPE and MPE.

For each roll forecast, we can calculate the MAPE and MPE and observe how they change across the rolling forecasts.

Are the MAPE and MPE constant? Fluctuate with no apparent trend? Or exhibit some systematic trend?

Doing this for every candidate model we are testing generates charts like this which can quickly show any areas of concern:

In this example, candidate models 18 and 15 may be worth further inspection since their MAPEs are much higher than the rest in a recent roll period (roll 6).

What else makes a model useful?

So, with respect to the guidelines for whittling down a pool of candidate models we listed in an earlier article, we can add the following from a rolling holdout analysis:

Stability – The model’s parameters should retain their statistical significance and not vary too much across the rolling periods; and the model’s residuals should remain “white noise” across the rolls;

Consistency of Performance – The model’s forecast accuracy and bias should not exhibit any strong trends, especially trends in the “wrong” direction (i.e. getting progressively worse) as the more recent time period is approached.

Strong Rolling Holdout Sample Performance – The model’s forecast accuracy and bias, averaged across all the rolls, should be high and low respectively. That is both the average MAPE and MPE should be low.

Benefits of Rolling

The primary benefit of a rolling analysis is that we get to see how a model performs forecast-wise over multiple time spans equal in length to our forecast horizon; instead of relying on performance in just one holdout sample.

A rolling analysis also addresses the issue of a short holdout sample (e.g. short forecast horizon) possibly not being representative of the general character of the time series.

In addition, a rolling analysis can be used as a check for the “best” model chosen using a single holdout sample. That is, would you pick the same model using the rolling holdout approach? If not, why?

In sum, a model that is persistently better at holdout sample forecasting over a longer time frame is likely to be more robust.

So, let ‘em roll!

Part 1 – Practical Time Series Forecasting – Introduction

Part 2 – Practical Time Series Forecasting – Some Basics

Part 3 – Practical Time Series Forecasting – Potentially Useful Models

Part 4 – Practical Time Series Forecasting – Data Science Taxonomy

Part 5 – Practical Time Series Forecasting – Know When to Hold ’em

Part 6 – Practical Time Series Forecasting – What Makes a Model Useful?

Part 7 – Practical Time Series Forecasting – To Difference or Not to Difference

The post Practical Time Series Forecasting – Know When to Roll ‘em appeared first on KDD Analytics.

A time series can have a very strong trend.

Visually, we often can see it. Gross domestic product (GDP) per person increasing year after year.

When a “shock” occurs to the process generating GDP, due to a recession for example, GDP gets knocked off its long-run growth path.

But can we expect GDP to bounce back and return to its original long-run growth path? Or will it start growing again but along a different path?

If the former, then the trend in GDP is said to be “deterministic.” And adding TIME to a time series forecasting model is one way to capture this trend.

On the other hand, if GDP starts a new trend after a recession, its trend is said to be “stochastic,” driven by random shocks. The standard approach to time series forecast modeling in this case is to “difference” the data before modeling.

The challenge as a forecaster is that it is not always easy to tell if the trend in a time series is deterministic or stochastic.

And your answer and the subsequent modeling choice will have important implications for the resulting forecast.

Deterministic vs. stochastic trends

Consider the time series shown below.

Suppose you were tasked with generating a 2-year forecast starting December 2003 (at the end of the shown time series history).

Is there a deterministic trend in this series? That is, do you suspect that the series will bounce back to the trend exhibited before January 2001?

Or has there been a fundamental change to the process generating this series and a new trend will start (i.e. the trend is stochastic)?

Deterministic trend

If you opt for a deterministic trend, then your forecasting model will be in “levels.” If we are talking about SALES, then it is the value of SALES at any given point in time. So, when we have a deterministic trend, we can model SALES as:

SALES_t = b₀ + b₁*TIME + ε_t

Of course, we could also account for seasonality by adding seasonal dummy variables as well as any hidden dynamics (cycles) by modeling the error term u_t as an ARMA process. But the key characteristic is the inclusion of a TIME variable (May 1993 = 1, June 1993 =2, etc.) and possibly TIME² and/or TIME³ depending on the series.

An ARMA process models SALES as being based on past SALES as well as on unobservable shocks. Such models can include two types of components: An autoregressive (AR) component captures the effect of past SALES on current SALES while a moving average (MA) component captures random shocks to the SALES series. 

Stochastic trend

If you opt for a stochastic trend, then the standard methodology is to difference your data (to remove the trend) and model the differences. This is known as ARIMA modeling. An ARIMA process is like an ARMA process except that the dynamics of the differenced series are modeled (see here).

Forecast differences

The forecast implications of this choice are shown in the following chart. We estimated a deterministic and a stochastic model and generated a forecast from each starting in December 2003. Specifically,

Deterministic Trend Model:  Y_t = b₀ + b₁*TIME + b₂*AR(1) + b₃*AR(2) + b₄*MA(3) + ε_t

Stochastic Trend Model:  Y_t – Y_t-1 = b₀ + b₁*AR(1) + b₂*AR(3) + ε_t

The forecast based on a deterministic model is shown by the orange line while the one based on the stochastic model is shown by the gray line. Also shown is what actually happened to the time series.

Hindsight is 20/20. In this case, the stochastic model would have been the better choice.

It does appear that some fundamental change occurred in the time series generation process. That is, the time series did not revert to its pre-2001 historical trend (at least during the forecast horizon).

The stochastic model yields a better forecast error (MAPE = 2.0%) than the deterministic model (MAPE = 5.6%) over the forecast horizon.

But at the time we had to make the forecast, all we had available were data through December 2003.

So, how do we pick between a deterministic and a stochastic forecasting model?

Holdout sample

From a practical perspective, unless we have very strong evidence of a stochastic process, the best course of action is to use a holdout sample.

Yes, there are techniques for testing whether a time series is “stationary” (i.e. has no trend) when visually it is not obvious.

But pragmatically, we are concerned about short-run forecast accuracy. And one way to compare competing models is by their performance in a holdout sample.

As we discussed in an earlier article, hold out a period of time at least equal to your forecast horizon from the data used to estimate a model. In this case, 2 years (January 2001 – December 2003).

Then build your models on data prior to January 2001 and compare the models’ forecast performance over the holdout sample.

In this case, such a holdout sample does not include any data from the strong trend period (pre-May 2001). So, likely a stochastic model would have performed better in the holdout sample as well.

But suppose we do this and have two (or more) models that perform equally well in the holdout sample?

We’ll cover this possibility in a subsequent article.

Part 1 – Practical Time Series Forecasting – Introduction

Part 2 – Practical Time Series Forecasting – Some Basics

Part 3 – Practical Time Series Forecasting – Potentially Useful Models

Part 4 – Practical Time Series Forecasting – Data Science Taxonomy

Part 5 – Practical Time Series Forecasting – Know When to Hold ’em

Part 6 – Practical Time Series Forecasting – What Makes a Model Useful?

The post Practical Time Series Forecasting – To Difference or Not to Difference appeared first on KDD Analytics.

So, you have estimated a bunch of forecasting models and realize (kudos to you!) that they are “all wrong” (ala George Box).

But your forecasting deadline is looming, and you need to find some useful models on which to base a forecast.

How do you decide which models make it to the next round?

Model building process

First, let’s review the forecast model build process:

Step 1:  Determine what is the business need;

Step 2:  Collect and examine your data; clean and adjust (e.g. frequency change) as necessary;

Step 3:  Determine your forecast horizon (i.e. align with the business need);

Step 4:  Determine and set aside your holdout sample;

Step 5:  Estimate models using the non-holdout portion of your time series (i.e. the “modeling sample”);

Step 6:  Gauge each model’s performance in the holdout sample;

Step 7:  Recalibrate each model using the full historical sample;

Step 8:  Make your forecast for the forecast horizon.

At the end of this process, you should have a few models that “pass muster,” that are potentially useful models.

But how do you whittle down all the models you tried to this select few?

Guidelines for selecting useful models

Here are some guidelines we follow:

Statistically Significant Parameters – Although one can argue that it is the prediction that matters, we still like to see model coefficients that are statistically significant with signs that can be explained. You may be asked to defend your model.

White Noise Residuals – When you estimate your model using the modeling sample, the residuals (difference between the actual and predicted values in the modeling sample) should have no apparent pattern to them. That is, there is no additional variation in the time series that can be explained by your model. What is left over is random or “white” noise.

Strong Holdout Sample Performance – Your model should produce low forecast error and exhibit low systematic bias in the holdout sample.

Robustness – When you recalibrate your model using the entire historical sample (modeling + holdout sample), your model should retain its statistical properties. That is, parameters are still significant with plausible signs and the residuals are still white noise.

Parsimony – If two models are equal in all performance respects except one is more complex than the other, we generally opt for the simpler model. Experience suggests that simpler models perform better when forecasting over the forecast horizon. And they are easier to interpret and explain to business decision makers.

Forecast Plausibility – The forecast produced by your model over the forecast horizon should be consistent with the available knowledge concerning the relevant business environment. In other words, the forecast needs to make sense. It is possible, following the steps above, to arrive at a high performing model which produces a counter intuitive forecast (e.g. declining SALES when the trend in SALES has been nothing but up).

At the end of this model building and testing process, you may have more than 1 model that can be used to generate your forecast. In a later article we will address what you can do in this situation.

The art of forecasting

Our experience is consistent with the opinion of others that there is still quite a bit of “art” to time series forecasting. Especially if you want it to meet a specific business need. Automated forecast routines exist. But we recommend that the process be closely supervised by a human to ensure a reasonable forecast.

Part 1 – Practical Time Series Forecasting – Introduction

Part 2 – Practical Time Series Forecasting – Some Basics

Part 3 – Practical Time Series Forecasting – Potentially Useful Models

Part 4 – Practical Time Series Forecasting – Data Science Taxonomy

Part 5 – Practical Time Series Forecasting – Know When to Hold ’em

The post Practical Time Series Forecasting – What Makes a Model Useful? appeared first on KDD Analytics.

Holdout samples are a mainstay of predictive analytics.

Set aside a portion of your data (say, 30%). Build your candidate models. Then “internally validate” your models using the holdout sample.

More sophisticated methods like cross validation use multiple holdout samples. But the idea is to see how well your models predict using data the model has not “seen” before. Then go back and fine tune to improve the models’ predictive accuracy.

Time series holdout samples

The truest test of your models is when they are applied to “new” data. Data from a fresh marketing campaign, a new set of customers, a more recent time period (“external validation”).

But you may not have access to such data when building your models. You certainly will not have access to future data.

So, a holdout sample needs to be crafted from the historical data at your disposal.

When building predictive models for, say, a marketing campaign or for loan risk scoring, there is usually a large amount of data to work with. So, holding out a sample for testing still leaves lots of data for model building.

However, the situation can be much different when working with time series data.

Depending on the frequency of the series, the amount of data points available to work with can be limited. 50 years of annual data is just 50 data points. 5 years of monthly data is just 60 data points.

Obviously the greater the frequency of data, the greater the number of data points available to work with…5 years of daily data is 1,825 data points. But these time series sample sizes usually pale against the large customer sets used to fuel marketing campaigns, which can run into the hundreds of thousands.

So, does this mean that holdout samples shouldn’t be used to test time series forecasting models?

Absolutely not!

You still need a way to whittle down your candidate models. You just need to be careful in how you select and use your holdout sample.

Holdout sample length

How much data should you set aside for a holdout sample? The rule of thumb we go by is to choose a holdout sample length that is at least (a) equal to the length of your forecast horizon or (b) equal to the length of time needed for your business to make a change.

Suppose you need a 12-month forecast to support a business plan. And you wish to forecast monthly sales for the 12 months starting November 1, 2017.

Then, your holdout sample should be at least the 12 months pertaining to November 2016 through October 2017. And your estimation sample should be all months prior to November 2016.

Remember, the time series methods we are addressing are best used for short-run forecasting. Most business forecasting needs are for short-run forecasts. The next few months or few years. Not the next 5 to 10 years.

Alternatively, suppose your business only needs 8 months to make a change (maybe it is getting more salespeople on line). Then your holdout sample should be at least 8 months.

Holdout sample performance

Once you estimate a model, you apply it to the holdout sample to see how well it predicts. There are several measures you can use to gauge how well your model performs. We focus on measures of accuracy and bias.

To measure forecast accuracy:

If the business cost of a forecast error is high, then the Mean Square Error (MSE) or Root Mean Square Error (RMSE) will magnify it since forecast errors are squared. MSE is the average of (predicted – actual)².

If the business cost of a forecast error is average, then the Mean Absolute Percent Error (MAPE) can be used. MAPE is simply the average of the absolute value of [(predicted – actual)/actual]. However, care should be taken if “0” values are possible as MAPE would be undefined.

See here for a discussion of forecast accuracy measures.

To measure forecast bias:

The Mean Percent Error (MPE) will indicate if there is a systematic bias to the forecast. If positive, then the model is over predicting; if negative it is underpredicting. And the further from 0, the greater the bias. MPE is the average of [(predicted – actual)/actual].

An alternative measure is Theil’s measure of systematic error, the “bias-proportion” of Theil’s inequality coefficient. This measures the extent to which average values of the forecasted and actual values deviate from each other, the larger the value, the greater the systematic bias.

In general, in the holdout sample, a good performing model will exhibit low overall error (high accuracy) and low systematic bias.

The chart below shows an example of such a model using a 5-month holdout sample. On average, the model’s error is between 0.28% and 1.85% while exhibiting a very small positive bias of 0.10%.

Note that there is no absolute criterion for what constitutes a “low” error, for example, MSE.

Measures of forecast error are to be judged relative to the context of the forecast you are making. In some cases, your models may be averaging an error in the 30%’s; in others it could be in the single digits.

Length of estimation sample

A related issue is how much data do you use for model estimation?

Often, there is not a choice. After setting aside a holdout sample, there may be just a bare minimum amount of data left for modeling (i.e. need more data points than model parameters to be estimated).

In general, the fewer the number of model parameters and the less “noisy” the data (i.e. less random), the fewer the number of data points needed. Typically, though, we look for at least 40 data points.

If you have a high frequency time series (monthly, daily, hourly) you may have room to consider whether the choice of the estimation sample length can affect model performance.

One can argue that the modeling sample should be reflective of the characteristics of the forecast horizon. That is the next year, say, is more likely to be like the past several years, not like 20 years ago. So, limit the estimating sample to more recent years.

Consider the time series shown below. Clearly the time path of this series has not been consistent. Rather than estimating a model using the entire historical sample, maybe limit it to the more recent period.

The trade-off is that there is less experiential history upon which to base a model. Maybe the dynamics associated with that turning point in early 2000 and subsequent recovery could prove to be fertile ground for training your model.

But this is testable proposition!

Because you have already set aside a holdout sample, you can test whether a model estimated on the full (non-holdout) sample performs better in the holdout sample than one based on a more recent sample.

Data frequency compression

Another use for a holdout sample is to test for whether changes to the frequency of the time series will improve predictive accuracy.

The frequency of the time series could be reduced to help match a desired forecast horizon. For example, suppose management wants a 3-year forecast. And you are working with monthly SALES. Yes, you could produce a 36 period (month) forecast. But that might be pushing the limits of your methodology, especially if there is not a strong trend.

Alternatively, by converting to a quarterly series, you would lessen the variability in your data and forecast only 12 periods. This might yield a more accurate forecast.

But again, this is testable using a holdout sample!

Bottom line

Holdout samples are a critical component of a time series forecasting methodology.

In a later article we will address using multiple holdout samples…to help guard against basing a model on a single, unrepresentative holdout sample (i.e. we found a great model just because we got lucky!).

Part 1 – Practical Time Series Forecasting – Introduction

Part 2 – Practical Time Series Forecasting – Some Basics

Part 3 – Practical Time Series Forecasting – Potentially Useful Models

Part 4 – Practical Time Series Forecasting – Data Science Taxonomy

The post Practical Time Series Forecasting – Know When to Hold ‘em appeared first on KDD Analytics.

(*It’s about the analytics.)

Machine Learning. Deep Learning. Data Science. Artificial Intelligence. Big Data.

Not a day goes by that one or all of these buzzwords stream past in our business news feeds.

Data analytics has become mainstream. And you better jump on board or risk being left at the station!

Just within the last year or so, searches of these topics have taken off. In fact, according to Google, in early 2017, search interest in one of these topics, machine learning, has eclipsed that of big data:

So, how do time series methods for forecasting fit into the taxonomy that currently defines the data science field?

Data science taxonomy

Key data science terms that are related to time series methods for forecasting are data mining, predictive analytics, machine learning (supervised and unsupervised), regression, structured and unstructured data.

These are not necessarily mutually exclusive. At the risk of incurring the wrath of the data science gods, here is our simplification:

Structured vs. unstructured data

Structured data are organized into “rows and columns” (spreadsheet); unstructured data are not (text in a book).

Time series methods use structured data.

Data mining

Data mining seeks to find patterns in data, whether structured or unstructured.

Time series methods seek to find patterns that repeat over time.

Predictive analytics

Predictive analytics seeks to find a relationship between a variable of interest (e.g. customer churn) and multiple dimensions (e.g. age, length of contract, zip code). These dimensions can be used to predict the likelihood of a customer churning (in our example).

Typically, predictive analytics is not based on time series data but “cross-sectional” data like a customer set. Additionally, time series methods use only a very limited set of dimensions, the primary one being past behavior of the variable being forecasted (e.g. sales).

Time series methods typically use the past behavior of the variable being forecasted as the primary dimension.

Machine learning

Machine learning means that a computer is using a program (algorithm) to “connect the dots” in the data. If you run a regression model in Excel you are engaging in machine learning.

However, supervised machine learning does not mean you are keeping watch over Excel as it does its stuff!

This is NOT what “supervised” machine learning means!

Supervised machine learning means that the computer is seeking to find a relationship between a single variable (e.g. churn) and many dimensional variables (e.g. age, length of contract, zip code).

Unsupervised machine learning means that the computer is seeking to find a relationship between many dimensions (e.g. age, length of contract, zip code) so that customers can, for example, be clustered into a small number of groups or tribes with similar characteristics.

Time series methods are a type of supervised machine learning since they attempt to find a relationship between present and past behavior.

Regression

Regression is one way a machine finds relationships between a single variable and a few (or many) dimensional variables or past values of the variable itself. There are several flavors of regression.

Time series models typically use least squares regression or maximum likelihood.

Bottom line

So, when you use time series methods for forecasting you are probably mining structured data using supervised, regression- or maximum likelihood-based, machine learning.

Part 1 – Practical Time Series Forecasting – Introduction

Part 2 – Practical Time Series Forecasting – Some Basics

Part 3 – Practical Time Series Forecasting – Potentially Useful Models

The post Practical Time Series Forecasting – Data Science Taxonomy appeared first on KDD Analytics.

This quote pretty well sums up time series forecasting models.

Any given model is unlikely to be spot on. And some can be wildly off.

But through a careful methodical process, we can whittle the pool of candidate models down to a set of useful models, if not a single preferred model.

When all is said and done, though, our guiding principle when building forecasting models is…how well the model predicts!

In practice, what this means for the types of models we consider is that we don’t rule anything out.

Yes, we have specific things we look for in an acceptable model (which we will cover later). But we don’t rule out a simple TIME trend model simply because it is too “simple.”

Our focus is on finding a forecasting model that can yield defensible short-run forecasts in a cost-effective manner.

Potentially useful models

So what kind of models do we typically examine?

As discussed in a previous article, a time series such as monthly sales (SALES) can have 3 components: trend, seasonal and cyclical. So, the type of model we consider depends on the extent to which 1, 2 or all 3 of these dynamics are present.

There are 3 classes of models that we typically consider. We will use a bit of math here to describe these models…think back to the formula of a line you learned in algebra: Y = a + bX.

Regression models

First are least squares regression models. Using SALES as our example, we could have a TIME trend model with, say, quarterly seasonality if we were examining SALES by quarter:

SALES_t = b₀ + b₁*TIME + b₂*Q1 + b₃*Q2 + b₄*Q3 + ε_t

Or a lagged least squares model with quarterly seasonality:

SALES_t = b₀ + b₁*SALES_t-1 + b₂*SALES_t-2 + b₃*Q1 + b₄*Q2 +b₅*Q3 +ε_t

In these model formulae, b₀ is the “intercept.” b₁, b₂,…etc. indicate the incremental effect (i.e. slope) on sales of a change in the value of a “right hand side” variable. ε_t is “residual” SALES, what is left “unexplained” by the model. And t is the time period, whether it is months, quarters, years, etc.

ARMA models

The second class of models are ARMA models.

An ARMA process models SALES as being based on past SALES as well as on unobservable shocks to SALES over time. Such models can include two types of components:

An autoregressive (AR) component captures the effect of past SALES on current SALES while a moving average (MA) component captures random shocks to the SALES series. These are typically estimated using a maximum likelihood technique.

We could have a model that is a pure ARMA model, for example:

SALES_t = b₀ + b₁*AR(1) + b₂*AR(2) + b₃*MA(1) +ε_t

Or a mixed regression-ARMA model, sometimes called “regression with ARMA errors,” like this:

SALES_t = b₀ + b₁*TIME + b₂*Q1 + b₃*Q2 + b₄*Q3 + b₄*AR(1) + b₅*MA(1) +ε_t

ARIMA models

A third class of models is related to the ARMA models above: ARIMA. According to standard Box-Jenkins methodology, if you know the underlying trend in SALES is “stochastic” (i.e. random), remove it by differencing SALES. Then model the differenced series as an ARMA process. For example:

SALES_t – SALES_t-1 = b₀ + b₁*AR(1) + b₂*MA(1) + b₃*MA(2) +ε_t

However, “it is sometimes very difficult to decide whether trend is best modeled as deterministic or stochastic, and the decision is an important part of the science – and art – of building forecasting models.” (Diebold,  Elements of Forecasting, 1998)

We will revisit this issue in a later article.

Other considerations

In addition to these 3 general classes of models we typically also try these variations:

• ARCH/GARCH models.

These models address heteroscedasticity in the residuals (ε_t). ARCH/GARCH models are used in the financial arena to help model return and risk where market volatility can fluctuate in a predictable manner.

• Inclusion of additional “right hand side variables.”

In the case of least squares and mixed regression-ARMA models, if the data are available, we often consider whether additional variables will improve predictive accuracy. In the case of SALES, for example, we could consider adding lagged values of advertising spending (AD SPEND). But if we are tasked with forecasting out 6 months, for example, then we cannot use lags of AD SPEND (in this example) shorter than 5 months. Else we would also have to forecast AD SPEND.

• Transformations.

For example, using the natural log of SALES can help model non-linear trends and/or dampen variation in SALES over time which may help to improve predictive accuracy.

Bottom line

There are many “specifications,” many potentially useful models that we estimate.

But not all end up in a final “pool” of candidates for the forecasting model. Each estimated model must pass certain tests to stay in the candidate pool.

In a later article we will cover the tests we use to help whittle down the pool of candidates to a set of truly useful models.

Part I – Practical Time Series Forecasting – Introduction

Part II – Practical Time Series Forecasting – Some basics

The post Practical Time Series Forecasting – Potentially Useful Models appeared first on KDD Analytics.

Forecasting the future is an exercise in uncertainty. And the further out one looks, the more uncertain the forecast becomes.

Most businesses are keenly focused on the next quarter, 6 months, year or at most next few years. Hence, our focus in this series is on time series methods for “short-run” forecasting.

The nature of time series

We are all familiar with charts like this:

showing a sequence of numbers ordered by time, across equally spaced periods of time. That is, a “time series” (e.g. closing stock price per day, sales per month, GDP per quarter, average global temperature per year).

Some time series exhibit little variability (up/down) from time period to time period (except for an overall trend) like the one above.

Others exhibit considerable variability across time with a much less apparent trend, like this:

An oftentimes unique characteristic of time series data, relative to non-time series data, is that successive values are not independent of each other. Although it may not be apparent from looking at a chart, today’s value is usually related in some way to yesterday’s value. And possibly to that of the day and/or several days before. This makes time series model estimation more complicated than in other areas.

A time series chart holds a unique fascination for us. Because we are constantly aware of the progression of time, our natural reaction when we see such charts is, “I wonder what’s going to happen next?”

Components of a time series

A successful forecasting model will account for each of 3 components that may exist in a time series: trend, seasonality and cycles.

Trend

Trend, when present, can be (but not always) visually apparent. For example, US real GDP (below) exhibits a persistent upward trend since the Great Depression.

Trend is a long-run phenomenon and reflects, in business, “slowly evolving preferences, technologies, institutions and demographics.” (Diebold, Elements of Forecasting)

Trend comes in two flavors.

If GDP, for example, was knocked off its long-run growth path by a recession but returned to the same path afterwards, then trend is said to be “deterministic.” Adding a TIME dimension to a model can go a long way to capturing such “deterministic” trend.

On the other hand, if GDP started a new growth path after the recession, then trend is said to be “stochastic.”

This distinction (between deterministic and stochastic trend) has important modeling and forecasting consequences which we will address in a later article.

Seasonality

A seasonal pattern repeats with calendar regularity.

The annual uptick in sales that occur during the November and December holiday season is an example. Higher airline passenger counts during the summer months is another example (see below). Adding seasonal indicators (“dummy variables“) to a model can capture such seasonality.

Cycles

A cyclic component can also be present. Cycles are much less rigid than seasonal patterns. One example is the business cycle, from a recession low to an expansion high.

A time series can contain one cycle (e.g. the daily cycle of body temperature) or multiple cycles (e.g. bicycle traffic patterns can exhibit daily, weekly and annual cycles). More broadly, a cyclic component is any dynamic not accounted for by trend or seasonality.

Modeling cycles takes us into the world of ARMA and ARIMA models which we’ll cover later.

Methods for forecasting

There are numerous methods for forecasting a time series, ranging from simple to complex.

Simple

The simplest is some type of smoothing routine, like moving averages or exponential smoothing. Moving averages , especially a 200-day moving average, are commonly used in technical analysis of stock price movements:

Complex

More complex econometric methods seek to model the relationship between, say, sales over time, and several dimensions that could affect sales, such as advertising spending.

Econometric models can consist of multiple interrelated equations (one for sales, one for ad spending) which would be estimated jointly, typically using a multiple regression methodology. Such models are used to model the US economy and to generate long-run forecasts of macroeconomic variables such as GDP and employment.

Also on the sophisticated end of the spectrum are techniques like spectral analysis, deep learning and neural networks. These methods require an elevated level of expertise on the part of a data scientist to implement and fine tune the models.

Middle of the road

In between the simpler and more complex forecasting methods is what we refer to as “time series methods.” These methods primarily rely on (but not always) the series’ historical behavior to inform the future. “Univariate modeling” is sometimes used to describe these methods.

A distinguishing feature of time series methods is that they explicitly account for the key characteristics of a time series: trend, seasonality and cycles.

The workhorses of time series methods are single equation, least squares regression and ARIMA models.

Least squares regression models can use a TIME trend, seasonal indicators and either lagged values of the series being modeled or an ARMA representation of the cyclic component to model a time series. They can also include other related lagged variables (e.g., advertising expenditures in a SALES forecasting model) but usually only if the lags are long.

If the trend of the series is “stochastic” (i.e. when the series is bumped off its trend path, it starts a new trend path), then ARIMA models may provide the best forecast.

Back to the short-run

The time series methods we will cover in this series of articles use the estimated dynamics and trend of the series to forecast a future path over the “forecast horizon.”

But since the forecasts will most likely ultimately revert to the underlying trend in the series, the best use of these time series methods is for “short-run” forecasts.

Although there is a more “technical” definition based on the type of model used, we generally define the “short run” as the period of time that matches most business’ forecast needs.  So, we are talking about anywhere from the next day to the next few years.

Part 1 – Practical Time Series Forecasting – Introduction

The post Practical Time Series Forecasting – Some Basics appeared first on KDD Analytics.