Linear model
Encyclopedia
In statistics
, the term linear model is used in different ways according to the context. The most common occurrence is in connection with regression models and the term is often taken as synonymous with linear regression
model. However the term is also used in time series analysis with a different meaning. In each case, the designation "linear" is used to identify a subclass of models for which substantial reduction in the complexity of the related statistical theory
is possible.
is as follows. Given a (random) sample
the relation between the observations Yi and the independent variables Xij is formulated as
where
may be nonlinear functions. In the above, the quantities εi are random variables representing errors in the relationship. The "linear" part of the designation relates to the appearance of the regression coefficients, βj in a linear way in the above relationship. Alternatively, one may say that the predicted values corresponding to the above model, namely
are linear functions of the βj.
Given that estimation is undertaken on the basis of a least squares
analysis, estimates of the unknown parameters βj are determined by minimising a sum of squares function
From this, it can readily be seen that the "linear" aspect of the model means the following:
. Here the model for values {Xt} in a time series can be written in the form
where again the quantities εt are random variables representing innovations
which are new random effects that appear at a certain time but also affect values of X at later times. In this instance the use of the term "linear model" refers to the structure of the above relationship in representing Xt as a linear function of past values of the same time series and of current and past values of the innovations. This particular aspect of the structure means that it is relative simple to derive relations for the mean and covariance
properties of the time series. Note that here the "linear" part of the term "linear model" is not referring to the coefficients φi and θi, as it would be in the case of a regression model, which looks structurally similar.
.
Statistics
Statistics is the study of the collection, organization, analysis, and interpretation of data. It deals with all aspects of this, including the planning of data collection in terms of the design of surveys and experiments....
, the term linear model is used in different ways according to the context. The most common occurrence is in connection with regression models and the term is often taken as synonymous with linear regression
Linear regression
In statistics, linear regression is an approach to modeling the relationship between a scalar variable y and one or more explanatory variables denoted X. The case of one explanatory variable is called simple regression...
model. However the term is also used in time series analysis with a different meaning. In each case, the designation "linear" is used to identify a subclass of models for which substantial reduction in the complexity of the related statistical theory
Statistical theory
The theory of statistics provides a basis for the whole range of techniques, in both study design and data analysis, that are used within applications of statistics. The theory covers approaches to statistical-decision problems and to statistical inference, and the actions and deductions that...
is possible.
Linear regression models
For the regression case, the statistical modelStatistical model
A statistical model is a formalization of relationships between variables in the form of mathematical equations. A statistical model describes how one or more random variables are related to one or more random variables. The model is statistical as the variables are not deterministically but...
is as follows. Given a (random) sample
the relation between the observations Yi and the independent variables Xij is formulated aswhere
may be nonlinear functions. In the above, the quantities εi are random variables representing errors in the relationship. The "linear" part of the designation relates to the appearance of the regression coefficients, βj in a linear way in the above relationship. Alternatively, one may say that the predicted values corresponding to the above model, namelyare linear functions of the βj.
Given that estimation is undertaken on the basis of a least squares
Least squares
The method of least squares is a standard approach to the approximate solution of overdetermined systems, i.e., sets of equations in which there are more equations than unknowns. "Least squares" means that the overall solution minimizes the sum of the squares of the errors made in solving every...
analysis, estimates of the unknown parameters βj are determined by minimising a sum of squares function
From this, it can readily be seen that the "linear" aspect of the model means the following:
- the function to be minimised is a quadratic function of the βj for which minimisation is a relatively simple problem;
- the derivatives of the function are linear functions of the βj making it easy to find the minimising values;
- the minimising values βj are linear functions of the observations Yi;
- the minimising values βj are linear functions of the random errors εi which makes it relatively easy to determine the statistical properties of the estimated values of βj.
Time series models
An example of a linear time series model is an autoregressive moving average modelAutoregressive moving average model
In statistics and signal processing, autoregressive–moving-average models, sometimes called Box–Jenkins models after the iterative Box–Jenkins methodology usually used to estimate them, are typically applied to autocorrelated time series data.Given a time series of data Xt, the ARMA model is a...
. Here the model for values {Xt} in a time series can be written in the form
where again the quantities εt are random variables representing innovations
Innovation (signal processing)
In time series analysis — as conducted in statistics, signal processing, and many other fields — the innovation is the difference between the observed value of a variable at time t and the optimal forecast of that value based on information available prior to time t...
which are new random effects that appear at a certain time but also affect values of X at later times. In this instance the use of the term "linear model" refers to the structure of the above relationship in representing Xt as a linear function of past values of the same time series and of current and past values of the innovations. This particular aspect of the structure means that it is relative simple to derive relations for the mean and covariance
Covariance
In probability theory and statistics, covariance is a measure of how much two variables change together. Variance is a special case of the covariance when the two variables are identical.- Definition :...
properties of the time series. Note that here the "linear" part of the term "linear model" is not referring to the coefficients φi and θi, as it would be in the case of a regression model, which looks structurally similar.
Other uses in statistics
There are some other instances where "nonlinear model" is used to contrast with a linearly structured model, although the term "linear model" is not usually applied. One example of this is nonlinear dimensionality reductionNonlinear dimensionality reduction
High-dimensional data, meaning data that requires more than two or three dimensions to represent, can be difficult to interpret. One approach to simplification is to assume that the data of interest lies on an embedded non-linear manifold within the higher-dimensional space...
.