Wednesday, 4 September 2013

Linear Regression

The linear regression model is a fairly simple model, nevertheless it is used in many applications. 

What follows is a fairly mathematical derivation of the linear regression parameters.

Let us denote the observations as $y[n]$ and $x[n]$ for $n=0,1,..,N-1$ for the model 

$y[n] = ax[n]+b$

We assume that we can measure both $x[n]$ and $y[n]$ - the unknowns are $a$ and $b$, the gradient and intercept respectively. We can use the least squares error criterion to find estimates for $a$ and $b$, from data $x[n]$ and $y[n]$.

Let  $e[n] = y[n] - ax[n]-b$ be the error. We can represent the samples $e[n]$, $x[n]$ and $y[n]$ as length $N$ column vectors $\underline{e}$,$\underline{x}$ and $\underline{y}$ respectively.

Thus the error sum of squares (which can be regarded as a cost function $C$ to minimise over) will be


where $\underline{1}$ is a column vector with all elements set to 1.

Expanding this cost function results in:-


Differentiating $C$ w.r.t. to $a$ and $b$:-

$\frac{\partial C}{\partial a}=2a(\underline{x}^T\underline{x})+2b(\underline{x}^T\underline{1})-2\underline{x}^T\underline{y}$

$\frac{\partial C}{\partial b}=2a(\underline{x}^T\underline{1})+2b(\underline{1}^T\underline{1})-2\underline{y}^T\underline{1}$

In order to estimate $\hat{a}$ and $\hat{b}$ (the least squares estimates of $a$ and $a$ respectively) we need to set the above two partial differential equations to 0, and solve for $a$ and $b$. We need to solve the following matrix equation

$\left[\begin{array}{cc}\underline{x}^T\underline{x}&\underline{x}^T\underline{1}\\ \underline{x}^T\underline{1}&\underline{1}^T\underline{1}\end{array}\right]$$\left[\begin{array}{c}\hat{a}\\ \hat{b}\end{array}\right]$ = $\left[\begin{array}{c}\underline{x}^T\underline{y} \\ \underline{y}^T\underline{1} \end{array}\right]$

Inverting the matrix and pre multiplying both sides of the above matrix equation by the inverse results in:

$\left[\begin{array}{c}\hat{a} \\ \hat{b} \end{array}\right]$ = $\frac{1}{N(\underline{x}^T\underline{x})-(\underline{x}^T\underline{1})^2}$$\left[\begin{array}{cc}\underline{1}^T\underline{1}&-\underline{x}^T\underline{1}\\ -\underline{x}^T\underline{1}&\underline{x}^T\underline{x}\end{array}\right]$$\left[\begin{array}{c}\underline{x}^T\underline{y} \\ \underline{y}^T\underline{1} \end{array}\right]$

If we revert to a more explicit notation in terms of the samples, we have



where the denominator of the fraction in the above two equations is the matrix determinant.

Coefficient of Determination

Once we have calculated $\hat{a}$ and $\hat{b}$, we can see how well the linear model accounts for the data. This is achieved by finding the Coefficient of Determination ($r^2$). This is the ratio of the residual sum of squares ($rss$) to the total sum of squares ($tss$),

$\Large r^2=\frac{rss}{tss}$

and has a maximum value of 1, which is attained when the model is exactly linear. The larger $r^2$ is, the better the linear model fits the data. We can think of $r^2$ as the proportion of variation in data explained by the linear model, or the "goodness of fit" of the linear model.

The total sum of squares is given by



$\bar{y}=\frac{1}{N}\sum_{n=0}^{N-1}y[n]$ is the mean of $y$.

The residual sum of squares ($rss$) is the total sum of squares ($tss$) minus the sum of squared errors ($sse$)



If the data perfectly fits the linear model, $sse$ will equal 0, so that $tss$ will equal $rss$ - the coefficient of determination $r^2$ will be $1$.

Summarising, we have

$\Large r^2=\frac{\sum_{n=0}^{N-1}(y[n]-\bar{y})^2-\sum_{n=0}^{N-1}(y[n]-\hat{a}x[n]-\hat{b})^2}{\sum_{n=0}^{N-1}(y[n]-\bar{y})^2}$

A worked example

Now, let us consider the following example:-

$x=[1,2,3,4]$, $y=[1.5,6,7,8]$

The matrix determinant will be $(4\times (1 + 2^2+3^2+4^2))-(1+2+3+4)^2=20$.

The least squares estimate of $a$ is

$\hat{a}=[4\times((1\times 1.5) + (2\times 6) + (3\times 7) + (4\times 8)) \\ -( (1.5+6+7+8)\times(1+2+3+4))]/20 \\ =2.05$

The least squares estimate of $b$ is

$\hat{b}=[(1+2^2+3^2+4^2)\times(1.5+6+7+8)-((1+2+3+4)\times((1\times 1.5) \\+ (2\times 6) + (3\times 7) +(4\times 8)))]/20 \\ = (675 - 665)/20 \\ =0.5$

The $tss$ is


where the average of $y$ is $5.625$.

The $sse$ is

$(1.5 - (2.05\times 1) - 0.5)^2+(6 - (2.05\times 2) - 0.5)^2\\+(7 - (2.05\times 3) - 0.5)^2+(8 - (2.05\times 4) - 0.5)^2=3.675$

So that the $rss$ is $24.688-3.675=21.013$.

The coefficient of determination is


This result is quite high, and is fairly indicative of a linear model.

There is a Linear Regression Calculator in this blog, which can be found here


  1. Hello Everyone out there,I am here to give my testimony about a Herbalist doctor who helped me . I was infected with HERPES SIMPLEX VIRUS in 2011, i went to many hospitals for cure but there was no solution, so I was thinking how can I get a solution out so that my body can be okay. One day I was in the river side thinking where I can go to get solution. so a lady walked to me telling me why am I so sad and i open up all to her telling her my problem, she told me that she can help me out, she introduce me to a doctor who uses herbal medication to cure HERPES SIMPLEX VIRUS and gave me his email, so i mail him. He told me all the things I need to do and also give me instructions to take, which I followed properly. Before I knew what is happening after two weeks the HERPES SIMPLEX VIRUS that was in my body got vanished . so if you are also heart broken and also need a help, you can also email him at {}
    or whatsapp him number: +2349038382931. or website is
    Contact him today and you will have a testimony...Good luck!

    Dr. OLIHA Also Cures:
    1. HIV/AIDS
    2. HERPES 1/2
    3. CANCER
    4. ALS (Lou Gehrig’s disease)
    5. Hepatitis B
    6. chronic pancreatic
    7. Emphysema
    8. COPD (Chronic Obstructive Pulmonary Disease

  2. All thanks to Dr Raymond for helping me to get my penis longer and bigger with his cream in just two week his cream is very active and also it work fast for everyone who have used it, the best thing about his herbal cream is that it does not have side effects and the results is permanent. Note he can also cure the below sicknesses and disease PREMATURE EJACULATIONLOW SPERM COUNTERECTILE DYSFUNCTIONHIV/AIDS CUREHERPES CUREINFECTION CUREPILESTROKEHEART FAILUREYou can email him if you need his help via: or WhatsApp him via: +2348116744524

  3. I'm here to testify about what DR. OYAGU did for me. I have been suffering from (GENITAL HERPES VIRUS) disease for the past 2 years and had constant pain and inching, especially in my private part. During the first year, I had faith in God that i would be cured someday. This disease started circulating all over my body and I have been taking treatment from my doctor, few weeks ago I came across a testimony of Shayla Bosschart on the internet testifying about a Man called DR. OYAGU on how he cured her from 7 years HSV 2. And she also gave the email address of this man, advise anybody to contact him for help on any kind of diseases that he would be of help, so I emailed him telling him about my (HSV 2) he told me not to worry that I was going to be cured!! Well, I never doubted him I have faith he can cure me too,, DR. OYAGU prepared and sent me Healing Soap, roots and herbs which I took. In the first one week, I started experiencing changes all over me, after two weeks of using his Roots/ Herbs, Oil and Soap, I was totally cured. no more inching , pain on me anymore as DR. OYAGU assured me. After some time I went to my doctor to do another test behold the result came out negative. So friends my advise is if you have such disease or know anyone who suffers from it or any other disease like HPV, HBV, HIV, ALS, HBP, CANCER etc. you can contact DR. OYAGU for help via email} visit his website whatsapp him through his mobile} +2348101755322. , Thanks once again DR. OYAGU for making me a happy woman again.

  4. Am Here To Give My Testimony About A Doctor Who Helped Me In My Life. I Was Infected With HERPES SIMPLEX VIRUS In 2019, I Went To Many Hospitals For Cure But There Was No Solution, So I Was Thinking How Can I Get A Solution Out So That My Body Can Be Okay. One Day I Was In The River Side Thinking Where I Can Go To Get Solution. So A Lady Walked To Me Telling Me Why Am I So Sad And I Open Up All To Her Telling Her My Problem, She Told Me That She Can Help Me Out, She Introduce Me To A Doctor Who Uses Herbal Medication To Cure HERPES SIMPLEX VIRUS And Gave Me His Email, So I Mail Him. He Told Me All The Things I Need To Do And Also Give Me Instructions To Take, Which I Followed Properly. Before I Knew What Is Happening After Two Weeks The HERPES SIMPLEX VIRUS That Was In My Body Got Vanished . So If You Are Also Heart Broken And Also Need A Help, You Can Also contact Him Through Dr. Oyagu reach him visit his website Or whatsapp him via +2348101755322

  5. DR OYAGU herbal medicine is a good remedy for herpes simplex virus cure. My name is MIGNON WRIGHT , am from USA,I am here to give my testimony about DR OYAGU who helped me in my life, i want to inform the public how i was cured from (HERPES SIMPLEX VIRUS) by DR OYAGU, i visited different hospital but they gave me list of drugs like Familiar, Zovirax, and Valtrex which is very expensive to treat the symptoms and never cured me. I was browsing through the Internet searching for remedy on HERPES and i saw comment of people talking about how DR OYAGU cured them, when i contacted him he gave me hope and send a Herbal medicine to me that i took and it seriously worked for me, am a free person now without any problem, my HERPES result came out negative, I pray to God to never let me and my family forget you in our life, am now cured permanently, you can also get your self cured my friends if you really need this doctor help, you can reach him on his website ; Email: or via whatsapp +2348101755322


  6. I'm 40 years old female I tested genital herpes (HSV1-2) positive in 2016. I was having bad outbreaks. EXTREMELY PAINFUL. I have try different kinds of drugs and treatment by the medical doctors all to know was avail. Six months ago I was desperately online searching for a helpful remedies for genital herpes (HSV1-2) cure, which I come across some helpful remedies on how Dr OYAGU have help so many people in curing genital herpes (HSV1-2) with the help of herbal treatment because I too believe there is someone somewhere in the world who can cure herpes completely. At of the past 2 months, however, I've been following his herpes protocol Via or +2348101755322 and it stopped all outbreaks completely! To my greatest surprise I was cured completely by following the protocol of his herbal medicine . Don't be discouraged by the medical doctors. There is a cure for HSV with the help of herbs and roots by a herbalist Dr call Dr OYAGU he is so kind and truthful with his herbal treatment, kindly contact him for more information Via or WhatsApp DR on +2348101755322 or visit his website you will be lucky as I am today