Linear Regression and Residuals
Linear regression is a statistical method for modeling the relationship between two quantitative variables by fitting a straight line (ŷ = a + bx) to the data. Residuals are the differences between the observed y-values and the predicted ŷ-values — they tell you how well the line fits each data point.
Key Takeaways
- The least-squares regression line ŷ = a + bx minimizes the sum of squared residuals. The slope (b) tells you the predicted change in y for a one-unit increase in x.
- The correlation coefficient (r) measures the strength and direction of the linear relationship. r ranges from −1 to +1. The coefficient of determination (r²) tells you the proportion of variation in y explained by x.
- Residuals = observed y − predicted ŷ. A residual plot should show random scatter with no pattern. Patterns in residuals indicate the linear model is not appropriate.
- Conditions for regression inference: Linear relationship, Independent observations, Normal residuals, Equal (constant) variance of residuals — remembered as LINE.
How Linear Regression Works
Linear regression is one of the most practical tools in statistics. It lets you model the relationship between an explanatory variable (x) and a response variable (y) using a straight line. If the data shows a linear trend, the regression line lets you make predictions and quantify the strength of the relationship.
The regression line ŷ = a + bx is calculated using the least-squares method, which finds the line that minimizes the total squared distance between each data point and the line. The slope (b) has a concrete interpretation: for each one-unit increase in x, the predicted value of y changes by b units. The y-intercept (a) is the predicted value of y when x = 0 (though this may not always be meaningful in context).
The correlation coefficient (r) measures how tightly the data clusters around the line. An r near +1 means a strong positive linear relationship; near −1 means a strong negative linear relationship; near 0 means weak or no linear relationship. The coefficient of determination (r²) is even more useful: it tells you the percentage of variation in y that is explained by the regression on x. An r² of 0.85 means 85% of the variability in y can be accounted for by its linear relationship with x.
Residuals are your diagnostic tool. A residual is simply the actual y-value minus the predicted ŷ-value for each data point. If the linear model fits well, the residual plot (residuals vs x or residuals vs predicted values) should show random scatter — no curves, no fan shapes, no patterns. If you see a curved pattern, a linear model isn't appropriate and you might need a transformation (like logging x or y). If the residuals fan out, you have non-constant variance, which violates an assumption of regression inference.
Regression on the AP Stats Exam
Linear regression appears throughout the AP Statistics exam — in multiple-choice, in FRQs, and often in the investigative task. You need to be able to interpret computer output, write regression equations, and analyze residual plots.
The most common FRQ format gives you computer output from a regression analysis and asks you to: (1) write the equation of the regression line, (2) interpret the slope in context, (3) interpret r² in context, and (4) assess whether the linear model is appropriate using a residual plot.
When interpreting slope, always use context: "For each additional year of education, the predicted salary increases by $3,200." When interpreting r², say: "Approximately 72% of the variability in salary can be explained by the linear relationship with years of education."
For inference about regression (t-test for the slope), the hypotheses are H₀: β = 0 (no linear relationship) vs Hₐ: β ≠ 0. Check the LINE conditions, use the t-statistic from computer output, and report the p-value. If p < α, conclude there is a statistically significant linear relationship.
Remember: correlation does not imply causation. Even if the regression shows a strong relationship (high r²), you can only claim causation if the data came from a randomized experiment.
Common Mistakes Students Make
- Using the regression line to predict outside the range of the data (extrapolation). The linear relationship may not hold beyond the observed x-values. The AP exam specifically tests whether you recognize when a prediction involves extrapolation.
- Confusing r and r². If r = 0.9, then r² = 0.81 — meaning 81% of the variation in y is explained by x. Students often say 90% when they should say 81%.
- Ignoring residual plots. A high r² doesn't mean the linear model is appropriate. Always check the residual plot for patterns. A curved residual plot means a straight line is the wrong model, even if r² seems high.
Related Topics
Frequently Asked Questions
The correlation coefficient (r) measures the strength and direction of the linear relationship between two variables, ranging from −1 to +1. The coefficient of determination (r²) is r squared, and it represents the proportion of variability in the response variable that is explained by the linear regression on the explanatory variable. r tells you direction and strength; r² tells you explanatory power.
A good residual plot shows random scatter with no discernible pattern — points scattered evenly above and below zero. If you see a curved pattern (U-shape or S-shape), a linear model isn't appropriate. If residuals fan out (get wider as x increases), the equal variance condition is violated. Both indicate the linear model needs modification.
Extrapolation is using a regression line to make predictions for x-values outside the range of the data used to build the model. It's risky because the linear relationship may not continue beyond the observed data. For example, a model predicting height from age for children ages 5-15 would give absurd predictions for age 50.