Residual Diagnostics and Multicollinearity

Overview

When we dive into multiple regression analysis (introduced in Section 14.1 of this volume), we often focus on producing the best-fitting model we can find. But once that model is in place, it’s not always sunshine and rainbows—residuals can misbehave, and predictor variables can interact in tricky (and sometimes sneaky) ways. Two key issues that demand our attention are (1) the behavior of residuals (do they look random, or are they quietly revealing a problem with our model’s assumptions?) and (2) the phenomenon of multicollinearity (when two or more explanatory variables move together so tightly that interpreting individual coefficients becomes a headache).

In this section, we’ll walk through the fundamentals of residual diagnostics, then explore multicollinearity in detail. I remember the first time I realized I’d been ignoring weird patterns in my own regression residuals. Let’s just say: the model sure looked great on paper, but the underlying error structure was basically screaming that something was wrong. Hopefully, by the end of this reading, you’ll be equipped to avoid (or at least address) such pitfalls.

Residual Diagnostics

Every regression analysis relies on certain assumptions, which we typically label as “classical OLS assumptions.” When we talk about residual diagnostics, we’re basically checking whether these assumptions appear to hold in practice—especially assumptions like linearity, constant variance (homoskedasticity), independence of errors, and normality of errors.

Visual Residual Analysis

One of the easiest yet most powerful ways to check for violations is to just look at your residuals. Plot the residuals against fitted values or against each independent variable in your model. If the model is doing its job properly in a standard linear regression setting, you’d ideally see a random cloud of points scattered around zero.

• If you see a pattern such as a curve, that might hint that your linear model is missing a relevant nonlinear term.
• If the variance of the residual cloud grows or shrinks as you move along the x-axis, you might be dealing with heteroskedasticity (violating the constant variance assumption).
• If the points appear to be systematically positive or negative in specific ranges, you may have omitted variables or a structural break.

Here’s a simple conceptual diagram (in Mermaid) that shows the steps we typically take when performing a residual check:

    flowchart LR
	    A["Collect residuals <br/> from fitted model"] --> B["Plot residuals <br/> vs. fitted values"]
	    B --> C["Look for any <br/> patterns or clusters"]
	    C --> D["Check for <br/> constant variance"]
	    D --> E["Conclude if <br/> model is adequate <br/> or if re-specification <br/> is needed"]

Normality of Residuals

Under classical OLS assumptions, the error term ε is normally distributed with mean zero and constant variance. In many real-world scenarios, especially if the sample size is large, the Central Limit Theorem helps approximate normality of errors. However, it’s good practice to formally test this assumption.

• Shapiro–Wilk test: A popular choice for checking normality. A significant p-value from this test suggests the distribution of residuals is likely not normal.
• Q–Q (quantile–quantile) plots: A visual display comparing the quantiles of your residuals to the quantiles of a theoretical normal distribution. If the points mostly lie on a straight diagonal line, that’s good news. If they depart sharply, there may be non-normality.

Practically speaking, mild departures from normality usually do not devastate the reliability of your coefficient estimates (particularly with large samples). However, if your distribution is heavily skewed or exhibits a lot of kurtosis, your standard errors and test statistics can be off, which hinders valid inference.

Implications for CFA Candidates

Residual diagnostics speak directly to the credibility of your regression analysis. If your residual plots show patterns, or if your normality tests strongly reject the presence of normal errors, the inferences you draw from your model might be suspect—this could impact (for example) how you perform fundamental analysis or how you forecast asset returns. As covered in Chapter 10 (Simple Linear Regression), residual analysis is also relevant there, but it becomes more critical as we deal with multiple explanatory variables in advanced models.

Multicollinearity

Now, let’s talk about a phenomenon that can be simultaneously subtle and glaring: multicollinearity. This is when explanatory variables are highly correlated with each other, making it tough to tease apart their individual impacts on the dependent variable. Maybe you have total market capitalization and total assets as separate explanatory variables in the same model—they might highly overlap and hamper your ability to interpret their respective coefficient estimates.

Effects of Multicollinearity

Multicollinearity doesn’t necessarily ruin your overall model fit. Indeed, the regression might have a fine R². Where it causes trouble is in:

• Inflated standard errors: The standard errors of individual coefficients increase, making it harder to detect a statistically significant effect for each predictor.
• Unstable coefficients: Small changes in the data can result in large shifts in coefficient estimates. You might notice that a variable’s coefficient is significantly positive in one sample but flips negative in another.
• Difficulty in interpretation: When two variables basically contain overlapping information, it’s challenging to figure out which is truly driving changes in the outcome.

Sometimes, especially for financial models, it’s nearly impossible to avoid some degree of collinearity. Variables like GDP growth, industrial production, and sentiment indicators might be correlated, and that’s just the nature of economic fundamentals. The goal is to detect when the correlation is so strong that it breaks the ability to interpret the coefficients reliably.

Detecting Multicollinearity

Pairwise Correlations

A quick first check is simply to look at the correlation matrix of your predictors:

• If you notice that two variables are correlated at, say, 0.90 or above, well, that’s a bright red flag.
• Even a correlation in the 0.70–0.80 range might prompt you to examine whether both variables are needed.

Variance Inflation Factors (VIFs)

A more formal test is the Variance Inflation Factor (VIF). For each explanatory variable \( X_j \), the VIF is computed as:

where \( R_j^2 \) is the coefficient of determination when \( X_j \) is regressed on the other explanatory variables. A common rule of thumb is:

• A VIF above 5 is often seen as a sign of problematic collinearity.
• Some analysts use a stricter cutoff of 10.

Values much less than 5 typically do not indicate serious collinearity concerns.

Here’s a short Python code snippet that demonstrates how one might compute VIFs:

 1import pandas as pd
 2import statsmodels.api as sm
 3from statsmodels.stats.outliers_influence import variance_inflation_factor
 4
 5# Add a column of ones for the intercept if not already present
 6X = sm.add_constant(X, has_constant='add')
 7
 8vif_data = pd.DataFrame()
 9vif_data["feature"] = X.columns
10vif_data["VIF"] = [variance_inflation_factor(X.values, i) 
11                   for i in range(X.shape[1])]
12
13print(vif_data)

Remedies for Multicollinearity

1. Drop Redundant Variables: If you can theoretically justify leaving out a variable that is highly correlated with another, consider dropping it.
2. Combine Predictors: Sometimes, two correlated variables measure the same underlying construct. Combining them into an index or factor can reduce duplication.
3. Increase Sample Size: In some (though not all) cases, adding more observations can help the model distinguish among correlated variables.
4. Dimensionality Reduction: Principal Component Analysis (PCA) or other factor analysis methods can transform your correlated predictors into a smaller set of uncorrelated components.

I once worked on a macroeconomic forecasting project where unemployment rate, inflation rate, and consumer sentiment were all heavily correlated, so we rolled them into a single principal component. It was a bit of a juggling act to interpret the resulting factor, but it definitely stabilized the regression coefficients.

Putting It All Together: Why It Matters

Residual diagnostics and multicollinearity analysis share a common goal: ensuring your regression model is valid, stable, and interpretable. In the context of equity valuation, mis-specified error structures might cause you to overstate or understate future earnings. In risk management, failing to detect collinearity among volatility measures and yield spreads could lead you to either over-hedge or under-hedge certain positions.

If you’re studying for the CFA exam, keep in mind that exam questions often revolve around interpreting regression output. They might ask how you would spot a problem in a residual plot or what a half-dozen VIF values mean for your analysis. They may also ask you to identify the best course of action when faced with, say, a perfect correlation (1.0) between two predictor variables. Knowing the best practice—perhaps to remove one variable—could be the difference between picking the correct choice and losing points.

Best Practices and Common Pitfalls

• Don’t ignore your diagnostic plots. They can often be more revealing than a table of regression coefficient estimates alone.
• Remember that even if your residuals pass normality tests, a large sample might mask moderate non-normalities. Keep an eye on outliers and leverage points.
• With multicollinearity, be mindful that high correlation among predictors can happen even when it doesn’t seem obvious from the raw data.
• Resist the temptation to keep throwing more correlated variables into your model unless you have a clear theoretical reason to do so.

Example: Stock Return Prediction

Imagine you’re building a model to predict monthly returns of a particular stock. You use these predictors:

• A popular valuation metric (P/E ratio)
• Market sentiment index
• Dividend payout ratio
• The broad market index returns from the last month

Suppose you find the correlation between your valuation metric and dividend payout ratio is 0.88. You also discover that the VIF for the dividend payout ratio is 6.2, well above the rule-of-thumb threshold of 5. This suggests a strong correlation that’s causing inflation in the standard error for the payout ratio coefficient. If, in your theory, P/E ratio is the more direct measure for how the market values the stock, you might decide to drop or at least transform the dividend payout ratio measure.

On the residual front, you notice that your residual plot against fitted values shows an expanding funnel shape, indicating possible heteroskedasticity—another sign to revisit your model’s assumptions and perhaps consider a robust standard error approach or a log transformation of the dependent variable.

Additional Mermaid Diagram: Impact of Multicollinearity

Here’s a second diagram that highlights how high correlation among variables ultimately leads to inflated standard errors:

    flowchart LR
	    A["Correlated <br/> Predictors"] --> B["Multicollinearity <br/> in the Model"]
	    B --> C["Inflates Standard Errors"]
	    C --> D["Less Precise Coefficient <br/> Estimates / Unstable Inference"]
	    D --> E["Potentially Incorrect <br/> Conclusions"]

Final Exam Tips

• Practice diagnosing regressions quickly. Expect to see at least one question that provides a residual plot and asks you to interpret whether the model is misspecified.
• Know your threshold for VIF. The exam might list VIF values for each independent variable and ask whether there’s a multicollinearity problem.
• Understand how to re-specify the model when issues arise—dropping variables, combining them, or using different transformations could all be tested.
• Emphasize conceptual clarity. Be prepared to explain not just how you diagnose these problems but also why it matters for financial analysis (for instance, in portfolio performance evaluation or equity valuation).

References

• Hair, J.F., Jr., Black, W.C., Babin, B.J., & Anderson, R.E. (2019). Multivariate Data Analysis. Cengage.
• Greene, W.H. (2018). Econometric Analysis. Pearson.
• CFA Institute. (2022). “Quantitative Methods in Investment Management,” CFA Program Curriculum.

Below are some additional noteworthy references:

• Kennedy, P. (2008). A Guide to Econometrics, 6th edition. Blackwell Publishing.
• Verbeek, M. (2017). A Guide to Modern Econometrics. Wiley.

Test Your Knowledge: Residual Diagnostics and Multicollinearity