Technical Addendum: Principles and Best Practices for Regression Analysis

The key issue in developing out of sample forecasts from autoregressive models is whether and how to include the autoregressive term. Since there is no actual data from which to calculate a forecasting error, there would seem to be no basis for including it in the forecasting equation. This guidance generally suggests ignoring the autoregressive term in out of sample forecasts. However, various techniques have been developed for using the autoregressive term in out of sample forecasts, but whether employing these terms is preferable to simply using the forecasting equation above, is unclear. This topic is complex and beyond the scope of this guidance, so the reader is referred to a good econometric textbook if more information is desired. Nevertheless, the use of the autoregressive term in the model provides superior estimates for the coefficients for the constant term and employment, so the use of these coefficients from the autoregressive model should be used to generate the out of sample forecast.

An example forecast is presented below. The results highlight the benefits of using a regression model with causal variables. Since retail sales is causally related to employment in the model, information about the growth in employment in the near term allows the forecast for retail sales to adjust accordingly. The simple trend analysis provides no quantitative basis for adjusting the forecast.

In the example in Table 8-28, employment growth in 2002 was assumed to be zero to reflect the slowing economy, then continue its past trend in future years. The regression based forecast for 2002 is about $33 million less under this scenario than the forecast that simply extrapolates past trends. The cumulative difference in the forecasts is about $1.2 billion over the forecast period. The much larger differences over the long term reflect the importance of incorporating updated information as quickly as possible. Forecasting errors in early years compound over time and become much larger as the length of the forecast increases.

Principle 1: For regression analysis, use real rather than nominal variable constructions.

Generally speaking, all variables in a regression equation should be adjusted for inflation. The reason is that inflation is not a "real" factor. Rather it is purely a monetary scaling of all variables that are expressed in dollar amounts. If a monetary series exhibits a trend, the analyst does not know whether this is the result of actual growth or a purely nominal phenomenon that can be attributed to inflation, unless inflation is removed from the equation by expressing all monetary variables in constant dollars.

If inflation is not removed from the analysis, regression equations tend to have higher than justified R². The reason being that the portion of the variation of the dependent variable that is attributable to inflation is known with certainty. That is, that portion of the variation in the explanatory variables that is attributable to inflation is perfectly correlated with the inflation in the dependent variable. The effect of that perfect correlation is to inflate the R² statistic. A regression analysis for revenue forecasting should express all forecasts in constant dollars. At the end of the forecasting process, these constant dollar revenue forecasts are inflated based on the inflation rate assumptions from the economic forecast and included in the agency financial plan, which is expressed in inflated dollars.

Principle 2: Fewer variables is better (most of the time).

Zvi Griliches, "Comments on Sims," in Frontiers of Quantitative Economics, vol.2, 1974, p. 335.

In most cases, a regression equation should have as few explanatory variables as possible. Simplicity in modeling is a virtue. Simple models have more clearly defined statistical relationships among variables and are easier to validate. Simple models are also easier to explain to non-modelers. Most importantly, models with too many variables run the risk of including variables that are correlated with each other. This problem is termed multi-collinearity. One of the assumptions of regression analysis is that all explanatory variables are independent of one another. If they are in fact, correlated, the coefficients on the explanatory variables become unstable and forecasts will be biased.

Before including a set of variables in an equation, the analyst should produce a correlation matrix to test which variables are correlated with each other and by how much. Generally speaking, most variables will display some amount of correlation, but if the correlation coefficient comes close to or exceeds 0.7, the two variables in question should not usually appear in the same equation. The result of a regression analysis when the explanatory variables are correlated can be unsettling. Coefficients can display the wrong sign, the t-statistics on the explanatory variables can be impossibly large, removing or adding a variable to the regression can make all the coefficients change dramatically. These impacts can make forecasts based on a multi-collinear regression unstable and theoretically unsound.

The most common method of dealing with this problem is to drop explanatory variables from the regression equation. This works well, as long as the regression equation still performs well and the model is fully specified. If the model is degraded, the other option is to transform some of the variables to remove the source of the collinearity. One common example of this approach is to transform an income variable to per capita income so that it can be used in an equation with population. By dividing income by population to get a per capita income value, one source of the collinearity with population is removed. This approach only works in specific cases where the source of the collinearity is clearly identifiable.

Principle 3: Use logarithmic transformations.

To ease the interpretation of results, economists often transform all their data series’ in a regression equation using natural logs. The usual revenue forecasting equation is a non-linear regression of the form:

where

When natural logs are applied to both sides of this equation, the result is:

which is linear in its components. This model is called the log-linear model since it is a non-linear regression that is linear in log form. In this specification, the coefficients are interpretable as elasticities:

which can be interpreted to mean the percent change in the dependent variable y given a percent change in the explanatory variable X. This is the definition of an elasticity.

A good test of a regression model forecasting retail sales from population and income would be to estimate the log-linear form of the regression model and ensure that the coefficients on population and income are in the neighborhood of 1. A coefficient of 1.0 for population means that a 1 percent increase in population leads to a 1 percent increase in retail sales, which is expected. A coefficient of 1.0 for per capita income means that a 1 percent increase in average income leads to a 1 percent increase in retail sales, which is also expected. The analyst will expect some deviation from 1.0 for population due to the reality that population growth may occur among groups that consume more or less than average. A similar explanation can be offered for average income where people’s propensity to consume taxable items from income growth may be more or less than the sample average.

A side note to logarithmic transformations is that they reduce the variance for each of the variables in the model. The effect of this is to raise the R² statistic in the final regression model. This benefit is illusory because to get usable forecasts, the analyst must exponentiate the model results to generate forecasts in actual dollars. This reintroduces the wider variances and the final forecasts are not any better than they would be if they were prepared without the log transformation.

Other non-linear functional forms can be checked, but they will generally not be suitable for revenue forecasting where the relationships are often consistent and stable under the assumptions of the linear model. A good econometrics textbook should be consulted for suggestions about functional form. In addition, some econometric software packages will have a variety of non-linear regression specifications built into them.

Principle 4: The model should make theoretical sense.

Regression models that are used to forecast demand, such as the retail sales example, should have a sound theoretical basis behind the functional form. This means applying the basic principles of economics to the development of the functional form. For instance, demand is a function of the number of consumers, their income and the price of the product. This suggests that a model to forecasts new car sales (for a registration fee revenue forecast) would include population, income per capita, and the average price of a new car to conform to what economists believe about the structure of demand.

Once a model is defined that makes economic sense and performs well, the coefficients that are estimated should also make sense. The magnitude of the impact of a change in an explanatory variable should be reasonable and the coefficients must have the correct sign. If either of these problems arises, the most likely problem is multi-collinearity (see Principle 2). Other potential problems can be poor data or mistakes made when constructing the data series.

Many models should be estimated without a regression constant. The regression constant indicates the level of the dependent variable if the explanatory variables were zero. The dependent variable in many tax revenue forecasting models would be zero if, for instance, population and income were zero. The regression equation should be tested without the constant to attempt to find a specification that performs well without it. If a constant term is required to achieve adequate fit with the data, the likely problem is a non-linearity in the modeling relationships that occurs well outside the available data set. Non-linear regressions can be estimated, but in many cases, a constant term will need to be included despite theoretical misgivings. The modeler should ensure that if a constant term is included, it is statistically significant at the 95 percent level.

Principle 5: Test lagged variable specifications.

Sometimes, financial reactions to economic indicators are delayed. The most common instance of delayed reaction is related to investment, whether business investment or real estate, in response to changes in interest rates. The delayed reaction is most evident if quarterly data are available, but yearly data can also display lagged reactions. Lagged variable specifications should be tested for a variety of different periods by using the prior period value of the explanatory variables in the regression.

Often, the best explanatory variable in a time series regression is the lagged dependent variable. The rationale for this type of regression is that the best predictor of future conditions is often current conditions. Cyclical impacts can be modeled by using more than one lagged dependent variable. The following regression model was developed to forecast construction activity (const), as a tax base for development fees:

Where _–1 indicates the prior period value and _–2 indicates a value from two periods earlier. This model postulates that construction activity depends on the prime interest rate from the prior period, construction activity in the two previous periods, and a constant term. This model was found to explain 95 percent of past variation in construction activity in one metropolitan area.

Principle 6: Account for serial correlation.

One of the assumptions of multiple regression analysis is that the error terms of the regression equations are uncorrelated with each other through time. If this assumption is violated, the model is said to display serial correlation of the disturbances. In simple terms, the error terms display an observable pattern, and are therefore, not random.

The Durbin-Watson (DW) statistic, which is calculated automatically in the regression routine of all statistical software packages, is used to detect the existence of serial correlation. The rule of thumb is that when error terms are uncorrelated with each other, the DW statistic equals 2. The analyst should consult the table of critical values for the DW statistic for the relevant confidence interval given the specific regression model in question.

Time series forecasting models often violate this assumption. Modelers account for this problem by including an "autoregressive" form of the error term as follows:

This formulation states that the error term is a function of the error term from the previous period plus a random component. This is called a first-order autoregressive term. A class of models called ARIMA (autoregressive integrated moving average) models combine several auto regressive and moving average terms to generate models that rely on no explanatory variables except constructions of the dependent variable itself and the error terms of the regression. These models are a rather complex form of trend analysis.

Principle 7: Test for structural stability.

Structural change in the relationships estimated in a regression model can be the source of bias in the forecasting process. Two common tests of structural instability are the Chow breakpoint test and the Chow forecast test. These tests are performed by splitting the data at some year T and performing tests on the structural change between these two periods.

In the Chow breakpoint test, separate regressions are estimated using all data before year T and after year T. The hypothesis that the coefficients are equal in both time periods is tested using a F-test. There is no hard and fast rule for choosing T except to choose years where the analyst suspects some change in the economic relationships may have occurred, such as recessions or periods of economic instability. Often, the midpoint of the time series is chosen in case there is a consistent structural change that is occurring over the entire period. Some statistical software packages can perform the Chow breakpoint test automatically.

The Chow forecast test is similar to the Chow breakpoint test. The data before year T is used to prepare a forecast for the years after year T. An F-test is used to test the hypothesis that the forecast values are equal to the actual values. The test compares the prediction errors to the variance that is expected if the null hypothesis were true. Failure to reject the null hypothesis (the forecast errors are zero) suggests that the model shows no evidence of structural instability over the forecast period.

An example of structural instability in revenue forecasting can be seen in certain forecasts of retail sales used to estimate sales tax revenue. Sales taxes are generally levied on only a subset of goods and services. Most services, such as medical care or legal services, are untaxed. If the percentage of income spent on taxable items is changing over time, this effect can introduce structural instability into sales tax revenue forecasts. The best way to deal with structural instability is to include a variable that accounts for the effect. In this case, the percent of personal income spent on goods (as opposed to services), can be calculated from economic data and most detailed economic forecasts. Including the percent of personal income spent on goods as an explanatory variable generally removes the structural instability from sales tax forecasting models.

Another source of structural instability might be long-run changes in the age distribution of regional population, which can affect sales and income tax revenues. Changes in vehicle ownership rates can affect forecasts of licensing or registration fees. All revenue sources can be impacted by shifts in the regional economic base. Forecasts are improved when these trends are identified and incorporated into the financial forecasts.