ideas, programming, research resources

Key success to Generalized Linear Model (GLM)

GLM is actually connected to the Kernel method in machine learning society.

One key success of GLM is that, the “linearity” of the model is:

“linear to model parameters”.

not to confuse with :

“linear to input variable”.


Example 1.

The input variable is in 2D dimensional space, denoted as (x1,x2). We have a data set D size of N, { d1, d2, …, dN}. Now for a simple GLM, we can set up a model like this:

Y = (1 ; X) * w + e;

where (1 ; X) contains all data points with a 1 vector augmented on the left, a trick to simplify the w model parameters : w = [w0; w1, … w2].

Trivially this model can be solve by generalized inverse. Therefore we get the w* optimal model parameters.


Example 2.

Same notation as above, but this time we are playing with more a sophisticated GLM model: with a polynomial fitting in GLM.

You might say, “Hey hold on for a second! Polynomial is no more linear! You are still using G-Linear-M, right?”.

Good catch. Yes we are still using GLM, but using the “kernel trick” !

Say we are using 2nd order polynomial, then basically we are constructing a new “feature space” from the original space (x1, x2). Now it’s (x1^2, x1*x2, x2^2).

Hmm.. technically, now for each data point di=[di.x1, di.x2], you compute its new coordinate in the new feature space, then leading to this:

di_new = [(di.x1)^2 , di.x1*di.x2, (di.x1)^2].

The GLM  is now:

Y = [1; Z]*wz + e;

Z is a 3D space mapping of the original 2D input space; Model parameters wz =  [wz0; wz1; wz2; wz3].

See? the new model is still “linear” to the parameter, but “non-linear” to the input variables!


That’s why GLM is quite successful and powerful!




Leave a Reply

Please log in using one of these methods to post your comment: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s