After an extensive model building and vetting process, along the lines we previously discussed here and here, the practical forecaster may still be left with several strong performing models.

These models perform similarly in the holdout sample tests. They retain their statistical properties when recalibrated on the full historical sample. But they yield different forecast paths over the forecast horizon.

Any one of the models could be easily defended. But the fact that the models yield different forecasts should make the forecaster pause.

An example

Below is an example of 3 short-run monthly forecasts:

The 3 models perform similarly in the holdout sample. One of the models is a least squares model. The other 2 are ARIMA models.

One model produces a steeply declining forecast. Another a slightly declining forecast. The third model produces an increasing forecast.

What should the forecaster do?

How can this happen?

Models are just that – models. They are abstractions from reality. And no single model will “fit” the holdout sample perfectly.

Two models, especially of different types (e.g. least squares vs. ARIMA), could have very similar holdout sample performance but differ dramatically in their forecast over the forecast horizon.

The holdout sample MAPE (mean absolute percentage error) could be very similar for these models. But the MAPE is an average error across the holdout sample. And the models could have arrived at their MAPEs by focusing on different aspects of the time series in the holdout sample.

Projecting these differences into the forecast horizon can result in very different forecasts.

Solutions

When there is no clear “champion” model, one solution is to combine the forecasts into one. We call this a “meta” forecast.

There are several ways this can be accomplished.

Checkpoint

But first, check to make sure the models to be combined are not “nested.” That is, one model is not a subset of another. If models are nested there usually is no advantage to combining their forecasts into a meta forecast.

In fact, a meta forecast will more likely be superior the greater the differences between the constituent models.

A meta forecast based on a least squares model and an ARIMA model will likely yield a smaller forecast error than that associated with either of the two models. However, if the two models were both least squares models, the superiority of a meta forecast might be questionable (Granger, 1989).

Solution 1

The simplest approach to arriving at a meta forecast is to simply average the forecasts of the individual models.

This essentially assumes that each model’s forecast is equally important in the meta forecast (i.e. receives equal weighting). This is a quick and uncomplicated way to generate a meta forecast.

Solution 2

Another approach makes use of each model’s holdout sample performance measures of forecast accuracy and bias. A weighting for each model’s forecast can be calculated using each model’s MAPE and MPE (mean percentage error) relative to that of all the models combined.

The meta forecast would then be a weighted average of the individual model forecasts. Models with lower MAPE and MPE would receive higher weights and contribute more to the meta forecast.

Solution 3

A third approach is to use regression to estimate the weights.

Using the holdout sample, or if too small, the full sample, regress the actual value on the forecasted value from each model. The goal is to find a regression with no constant and all regression coefficients positive and statistically significant.

The regression coefficients should then sum very close to one. These coefficients then become the weights by which forecasts are combined into a meta forecast (see Wilson and Keating).

Back to our example

The forecaster could go with candidate 3 since it “splits the difference.” However, the forecaster is still left with the task of defending why the other two equally plausible models were not chosen.

Alternatively, a meta forecast can be used. As an example, we created a simple average forecast across the 3 candidate models. As discussed above, this assumes an equal weighting across the 3 short-run forecasts. A more sophisticated approach would have been to estimate the weights using a regression approach.

Not surprisingly, the meta forecast is quite like the essentially flat forecast of candidate 3 (which lies almost half way between candidate 1’s and 2’s forecast). But not all cases will be like this.

If a regression approach to estimating the weights was used, the meta forecast could be quite different from that of candidate 3.

Yes, the meta forecast will lie between the two forecast extremes. But the assumed or estimated weights will dictate where the meta forecast will lie.

Bottom line

Combining forecasts from equally strong models is intuitively appealing since each model has its strengths and weaknesses.

Combining models’ forecasts in a complementary fashion should lead to more robust and accurate short-run forecasts.

Part 1 – Practical Time Series Forecasting – Introduction

Part 2 – Practical Time Series Forecasting – Some Basics

Part 3 – Practical Time Series Forecasting – Potentially Useful Models

Part 4 – Practical Time Series Forecasting – Data Science Taxonomy

Part 5 – Practical Time Series Forecasting – Know When to Hold ’em

Part 6 – Practical Time Series Forecasting – What Makes a Model Useful?

Part 7 – Practical Time Series Forecasting – To Difference or Not to Difference

Part 8 – Practical Time Series Forecasting – Know When to Roll ’em

The post Practical Time Series Forecasting – Meta Models appeared first on KDD Analytics.

(*It’s about the analytics.)

Machine Learning. Deep Learning. Data Science. Artificial Intelligence. Big Data.

Not a day goes by that one or all of these buzzwords stream past in our business news feeds.

Data analytics has become mainstream. And you better jump on board or risk being left at the station!

Just within the last year or so, searches of these topics have taken off. In fact, according to Google, in early 2017, search interest in one of these topics, machine learning, has eclipsed that of big data:

So, how do time series methods for forecasting fit into the taxonomy that currently defines the data science field?

Data science taxonomy

Key data science terms that are related to time series methods for forecasting are data mining, predictive analytics, machine learning (supervised and unsupervised), regression, structured and unstructured data.

These are not necessarily mutually exclusive. At the risk of incurring the wrath of the data science gods, here is our simplification:

Structured vs. unstructured data

Structured data are organized into “rows and columns” (spreadsheet); unstructured data are not (text in a book).

Time series methods use structured data.

Data mining

Data mining seeks to find patterns in data, whether structured or unstructured.

Time series methods seek to find patterns that repeat over time.

Predictive analytics

Predictive analytics seeks to find a relationship between a variable of interest (e.g. customer churn) and multiple dimensions (e.g. age, length of contract, zip code). These dimensions can be used to predict the likelihood of a customer churning (in our example).

Typically, predictive analytics is not based on time series data but “cross-sectional” data like a customer set. Additionally, time series methods use only a very limited set of dimensions, the primary one being past behavior of the variable being forecasted (e.g. sales).

Time series methods typically use the past behavior of the variable being forecasted as the primary dimension.

Machine learning

Machine learning means that a computer is using a program (algorithm) to “connect the dots” in the data. If you run a regression model in Excel you are engaging in machine learning.

However, supervised machine learning does not mean you are keeping watch over Excel as it does its stuff!

This is NOT what “supervised” machine learning means!

Supervised machine learning means that the computer is seeking to find a relationship between a single variable (e.g. churn) and many dimensional variables (e.g. age, length of contract, zip code).

Unsupervised machine learning means that the computer is seeking to find a relationship between many dimensions (e.g. age, length of contract, zip code) so that customers can, for example, be clustered into a small number of groups or tribes with similar characteristics.

Time series methods are a type of supervised machine learning since they attempt to find a relationship between present and past behavior.

Regression

Regression is one way a machine finds relationships between a single variable and a few (or many) dimensional variables or past values of the variable itself. There are several flavors of regression.

Time series models typically use least squares regression or maximum likelihood.

Bottom line

So, when you use time series methods for forecasting you are probably mining structured data using supervised, regression- or maximum likelihood-based, machine learning.

Part 1 – Practical Time Series Forecasting – Introduction

Part 2 – Practical Time Series Forecasting – Some Basics

Part 3 – Practical Time Series Forecasting – Potentially Useful Models

The post Practical Time Series Forecasting – Data Science Taxonomy appeared first on KDD Analytics.

This quote pretty well sums up time series forecasting models.

Any given model is unlikely to be spot on. And some can be wildly off.

But through a careful methodical process, we can whittle the pool of candidate models down to a set of useful models, if not a single preferred model.

When all is said and done, though, our guiding principle when building forecasting models is…how well the model predicts!

In practice, what this means for the types of models we consider is that we don’t rule anything out.

Yes, we have specific things we look for in an acceptable model (which we will cover later). But we don’t rule out a simple TIME trend model simply because it is too “simple.”

Our focus is on finding a forecasting model that can yield defensible short-run forecasts in a cost-effective manner.

Potentially useful models

So what kind of models do we typically examine?

As discussed in a previous article, a time series such as monthly sales (SALES) can have 3 components: trend, seasonal and cyclical. So, the type of model we consider depends on the extent to which 1, 2 or all 3 of these dynamics are present.

There are 3 classes of models that we typically consider. We will use a bit of math here to describe these models…think back to the formula of a line you learned in algebra: Y = a + bX.

Regression models

First are least squares regression models. Using SALES as our example, we could have a TIME trend model with, say, quarterly seasonality if we were examining SALES by quarter:

SALES_t = b₀ + b₁*TIME + b₂*Q1 + b₃*Q2 + b₄*Q3 + ε_t

Or a lagged least squares model with quarterly seasonality:

SALES_t = b₀ + b₁*SALES_t-1 + b₂*SALES_t-2 + b₃*Q1 + b₄*Q2 +b₅*Q3 +ε_t

In these model formulae, b₀ is the “intercept.” b₁, b₂,…etc. indicate the incremental effect (i.e. slope) on sales of a change in the value of a “right hand side” variable. ε_t is “residual” SALES, what is left “unexplained” by the model. And t is the time period, whether it is months, quarters, years, etc.

ARMA models

The second class of models are ARMA models.

An ARMA process models SALES as being based on past SALES as well as on unobservable shocks to SALES over time. Such models can include two types of components:

An autoregressive (AR) component captures the effect of past SALES on current SALES while a moving average (MA) component captures random shocks to the SALES series. These are typically estimated using a maximum likelihood technique.

We could have a model that is a pure ARMA model, for example:

SALES_t = b₀ + b₁*AR(1) + b₂*AR(2) + b₃*MA(1) +ε_t

Or a mixed regression-ARMA model, sometimes called “regression with ARMA errors,” like this:

SALES_t = b₀ + b₁*TIME + b₂*Q1 + b₃*Q2 + b₄*Q3 + b₄*AR(1) + b₅*MA(1) +ε_t

ARIMA models

A third class of models is related to the ARMA models above: ARIMA. According to standard Box-Jenkins methodology, if you know the underlying trend in SALES is “stochastic” (i.e. random), remove it by differencing SALES. Then model the differenced series as an ARMA process. For example:

SALES_t – SALES_t-1 = b₀ + b₁*AR(1) + b₂*MA(1) + b₃*MA(2) +ε_t

However, “it is sometimes very difficult to decide whether trend is best modeled as deterministic or stochastic, and the decision is an important part of the science – and art – of building forecasting models.” (Diebold,  Elements of Forecasting, 1998)

We will revisit this issue in a later article.

Other considerations

In addition to these 3 general classes of models we typically also try these variations:

• ARCH/GARCH models.

These models address heteroscedasticity in the residuals (ε_t). ARCH/GARCH models are used in the financial arena to help model return and risk where market volatility can fluctuate in a predictable manner.

• Inclusion of additional “right hand side variables.”

In the case of least squares and mixed regression-ARMA models, if the data are available, we often consider whether additional variables will improve predictive accuracy. In the case of SALES, for example, we could consider adding lagged values of advertising spending (AD SPEND). But if we are tasked with forecasting out 6 months, for example, then we cannot use lags of AD SPEND (in this example) shorter than 5 months. Else we would also have to forecast AD SPEND.

• Transformations.

For example, using the natural log of SALES can help model non-linear trends and/or dampen variation in SALES over time which may help to improve predictive accuracy.

Bottom line

There are many “specifications,” many potentially useful models that we estimate.

But not all end up in a final “pool” of candidates for the forecasting model. Each estimated model must pass certain tests to stay in the candidate pool.

In a later article we will cover the tests we use to help whittle down the pool of candidates to a set of truly useful models.

Part I – Practical Time Series Forecasting – Introduction

Part II – Practical Time Series Forecasting – Some basics

The post Practical Time Series Forecasting – Potentially Useful Models appeared first on KDD Analytics.

Forecasting the future is an exercise in uncertainty. And the further out one looks, the more uncertain the forecast becomes.

Most businesses are keenly focused on the next quarter, 6 months, year or at most next few years. Hence, our focus in this series is on time series methods for “short-run” forecasting.

The nature of time series

We are all familiar with charts like this:

showing a sequence of numbers ordered by time, across equally spaced periods of time. That is, a “time series” (e.g. closing stock price per day, sales per month, GDP per quarter, average global temperature per year).

Some time series exhibit little variability (up/down) from time period to time period (except for an overall trend) like the one above.

Others exhibit considerable variability across time with a much less apparent trend, like this:

An oftentimes unique characteristic of time series data, relative to non-time series data, is that successive values are not independent of each other. Although it may not be apparent from looking at a chart, today’s value is usually related in some way to yesterday’s value. And possibly to that of the day and/or several days before. This makes time series model estimation more complicated than in other areas.

A time series chart holds a unique fascination for us. Because we are constantly aware of the progression of time, our natural reaction when we see such charts is, “I wonder what’s going to happen next?”

Components of a time series

A successful forecasting model will account for each of 3 components that may exist in a time series: trend, seasonality and cycles.

Trend

Trend, when present, can be (but not always) visually apparent. For example, US real GDP (below) exhibits a persistent upward trend since the Great Depression.

Trend is a long-run phenomenon and reflects, in business, “slowly evolving preferences, technologies, institutions and demographics.” (Diebold, Elements of Forecasting)

Trend comes in two flavors.

If GDP, for example, was knocked off its long-run growth path by a recession but returned to the same path afterwards, then trend is said to be “deterministic.” Adding a TIME dimension to a model can go a long way to capturing such “deterministic” trend.

On the other hand, if GDP started a new growth path after the recession, then trend is said to be “stochastic.”

This distinction (between deterministic and stochastic trend) has important modeling and forecasting consequences which we will address in a later article.

Seasonality

A seasonal pattern repeats with calendar regularity.

The annual uptick in sales that occur during the November and December holiday season is an example. Higher airline passenger counts during the summer months is another example (see below). Adding seasonal indicators (“dummy variables“) to a model can capture such seasonality.

Cycles

A cyclic component can also be present. Cycles are much less rigid than seasonal patterns. One example is the business cycle, from a recession low to an expansion high.

A time series can contain one cycle (e.g. the daily cycle of body temperature) or multiple cycles (e.g. bicycle traffic patterns can exhibit daily, weekly and annual cycles). More broadly, a cyclic component is any dynamic not accounted for by trend or seasonality.

Modeling cycles takes us into the world of ARMA and ARIMA models which we’ll cover later.

Methods for forecasting

There are numerous methods for forecasting a time series, ranging from simple to complex.

Simple

The simplest is some type of smoothing routine, like moving averages or exponential smoothing. Moving averages , especially a 200-day moving average, are commonly used in technical analysis of stock price movements:

Complex

More complex econometric methods seek to model the relationship between, say, sales over time, and several dimensions that could affect sales, such as advertising spending.

Econometric models can consist of multiple interrelated equations (one for sales, one for ad spending) which would be estimated jointly, typically using a multiple regression methodology. Such models are used to model the US economy and to generate long-run forecasts of macroeconomic variables such as GDP and employment.

Also on the sophisticated end of the spectrum are techniques like spectral analysis, deep learning and neural networks. These methods require an elevated level of expertise on the part of a data scientist to implement and fine tune the models.

Middle of the road

In between the simpler and more complex forecasting methods is what we refer to as “time series methods.” These methods primarily rely on (but not always) the series’ historical behavior to inform the future. “Univariate modeling” is sometimes used to describe these methods.

A distinguishing feature of time series methods is that they explicitly account for the key characteristics of a time series: trend, seasonality and cycles.

The workhorses of time series methods are single equation, least squares regression and ARIMA models.

Least squares regression models can use a TIME trend, seasonal indicators and either lagged values of the series being modeled or an ARMA representation of the cyclic component to model a time series. They can also include other related lagged variables (e.g., advertising expenditures in a SALES forecasting model) but usually only if the lags are long.

If the trend of the series is “stochastic” (i.e. when the series is bumped off its trend path, it starts a new trend path), then ARIMA models may provide the best forecast.

Back to the short-run

The time series methods we will cover in this series of articles use the estimated dynamics and trend of the series to forecast a future path over the “forecast horizon.”

But since the forecasts will most likely ultimately revert to the underlying trend in the series, the best use of these time series methods is for “short-run” forecasts.

Although there is a more “technical” definition based on the type of model used, we generally define the “short run” as the period of time that matches most business’ forecast needs.  So, we are talking about anywhere from the next day to the next few years.

Part 1 – Practical Time Series Forecasting – Introduction

The post Practical Time Series Forecasting – Some Basics appeared first on KDD Analytics.

It goes without saying that every business is keenly interested in knowing what the future will bring.

Will sales grow next year? By how much? Will suppliers increase their prices? How fast will be the adoption of a new IoT product? How much warehouse capacity is needed for the next holiday period? Will some international event plunge the global economy into a recession?

Predicting the future is an exercise in probability rather than certainty. Businesses engage in various levels of sophistication in trying to bound the likelihood of future states to support their business plans.

Some have teams of economists and data scientists tasked with building complex forecasting models.

Many businesses, however, likely rely on less sophisticated means centered on spreadsheet models, trends and moving averages (or even educated guesses).

Time series methodology is a moderately sophisticated yet cost effective way to generate forecasts. It is a statistical approach which bases forecasts on the past behavior of the data series in question (e.g. monthly sales).

And it accounts for other characteristics of a time series which can yield a more accurate forecast than, say, a simple straight-line trend model.

More time, more data

We have all heard the forecasts about data growth, the proverbial “hockey stick.”

By one account, human and machine-generated data is growing at 10x the rate of traditional business data. And machine-generated data is growing at 50X the rate.

A good portion of this machine-generated data has a time dimension. Internet of Things (IoT) devices are proliferating, each of which has a potential to collect data over time.

A washing machine can monitor, collect and post performance data to the cloud. Using these data to forecast product failure can lead to a pro-active maintenance visit by your friendly but lonely Maytag repairman.

Similarly, a household’s electricity usage can be monitored, modeled and forecasted leading to cost-savings suggestions by an energy service provider under a time-of-day pricing scheme.

The electric utility company itself can use the data from all the IoT appliances in households to generate better residential load forecasts and help better manage the electricity grid.

Even if a more traditional businesses source like sales, inventories, deliveries, workforce utilization, IT usage and the like, advances in data collection, storage and proliferation are making time series data more readily accessible.

Thus, there will be an increased demand for product managers, economists, statisticians and data scientists to make use of these data and tell us what will happen next.

Time series methods

The premise of time series methods (and of most quantitatively-based forecasting methods) is that the future will be much like the past.

If sales have been growing at a consistently healthy rate with strong seasonal variation (e.g. holiday periods) for the last year, then it is likely the next year will be similar, all else constant. If done correctly, the methodology can yield a defensible forecast of likely sales each month during the “forecast horizon.”

But, as with all forecasting methodologies, there are pitfalls of which one should be aware.

Practical time series methods

This is the first of a series of articles on practical time series methods for short-run business forecasting.

There are abundant, excellent resources covering the basics of business forecasting including time series methods, ranging from blog posts to online courses to open-source textbooks.

And time series methods are a mainstay of advanced courses in econometrics and business forecasting (resources we recommend are Elements of Forecasting by Diebold and Econometric Models and  Economic Forecasts by Pindyck and Rubinfeld).

Rather than being a treatise on forecasting, this series of articles will present a practical methodology and some of the lessons we have learned performing time series forecasting for clients.

The post Practical Time Series Forecasting – Introduction appeared first on KDD Analytics.