Plotting Regression Line

In statistics, linear regression is any approach to modeling the relationship between a scalar variable y and one or more variables denoted X. In linear regression, models of the unknown parameters are estimated from the data using linear functions. Such models are called “linear models.” Most commonly, linear regression refers to a model in which the conditional mean of y given the value of X is an affine function of X. Less commonly, linear regression could refer to a model in which the median, or some other quantile of the conditional distribution of y given X is expressed as a linear function of X. Like all forms of regression analysis, linear regression focuses on the conditional probability distribution of y given X, rather than on the joint probability distribution of y and X, which is the domain of multivariate analysis.

Linear regression was the first type of regression analysis to be studied rigorously, and to be used extensively in practical applications. This is because models which depend linearly on their unknown parameters are easier to fit than models which are non-linearly related to their parameters and because the statistical properties of the resulting estimators are easier to determine.

Linear regression has many practical uses. Most applications of linear regression fall into one of the following two broad categories:

• If the goal is prediction, or forecasting, linear regression can be used to fit a predictive model to an observed data set of y and X values. After developing such a model, if an additional value of X is then given without its accompanying value of y, the fitted model can be used to make a prediction of the value of y.
• Given a variable y and a number of variables X ₁ , ..., X _p that may be related to y, then linear regression analysis can be applied to quantify the strength of the relationship between y and the X _j , to assess which X _j may have no relationship with y at all, and to identify which subsets of the X _j contain redundant information about y, thus once one of them is known, the others are no longer informative.

Linear regression models are often fitted using the least squares approach, but they may also be fitted in other ways, such as by minimizing the “lack of fit” in some other norm, or by minimizing a penalized version of the least squares loss function as in ridge regression. Conversely, the least squares approach can be used to fit models that are not linear models. Thus, while the terms “least squares” and linear model are closely linked, they are not synonymous.

Introduction to linear regression

Given a data set \{y_i,\, x_{i1}, \ldots, x_{ip}\}_{i=1}^n of n statistical units, a linear regression model assumes that the relationship between the dependent variable y _i and the p-vector of regressors x _i is approximately linear. This approximate relationship is modeled through a so-called “disturbance term” ε _i — an unobserved random variable that adds noise to the linear relationship between the dependent variable and regressors. Thus the model takes form

y_i = \beta_1 x_{i1} \cdots \beta_p x_{ip} \varepsilon_i = x'_i\beta \ ...