Wednesday, 4 September 2013

Linear Regression

The linear regression model is a fairly simple model, nevertheless it is used in many applications. 

What follows is a fairly mathematical derivation of the linear regression parameters.

Let us denote the observations as $y[n]$ and $x[n]$ for $n=0,1,..,N-1$ for the model 

$y[n] = ax[n]+b$

We assume that we can measure both $x[n]$ and $y[n]$ - the unknowns are $a$ and $b$, the gradient and intercept respectively. We can use the least squares error criterion to find estimates for $a$ and $b$, from data $x[n]$ and $y[n]$.

Let  $e[n] = y[n] - ax[n]-b$ be the error. We can represent the samples $e[n]$, $x[n]$ and $y[n]$ as length $N$ column vectors $\underline{e}$,$\underline{x}$ and $\underline{y}$ respectively.

Thus the error sum of squares (which can be regarded as a cost function $C$ to minimise over) will be

$\underline{e}^T\underline{e}=(\underline{y}-a\underline{x}-b\underline{1})^T(\underline{y}-a\underline{x}-b\underline{1}$).

where $\underline{1}$ is a column vector with all elements set to 1.

Expanding this cost function results in:-

$C=\underline{y}^T\underline{y}-2a\underline{x}^T\underline{y}-2b\underline{y}^T\underline{1}+a^2(\underline{x}^T\underline{x})+2ab(\underline{x}^T\underline{1})+b^2(\underline{1}^T\underline{1})$

Differentiating $C$ w.r.t. to $a$ and $b$:-

$\frac{\partial C}{\partial a}=2a(\underline{x}^T\underline{x})+2b(\underline{x}^T\underline{1})-2\underline{x}^T\underline{y}$

$\frac{\partial C}{\partial b}=2a(\underline{x}^T\underline{1})+2b(\underline{1}^T\underline{1})-2\underline{y}^T\underline{1}$

In order to estimate $\hat{a}$ and $\hat{b}$ (the least squares estimates of $a$ and $a$ respectively) we need to set the above two partial differential equations to 0, and solve for $a$ and $b$. We need to solve the following matrix equation

$\left[\begin{array}{cc}\underline{x}^T\underline{x}&\underline{x}^T\underline{1}\\ \underline{x}^T\underline{1}&\underline{1}^T\underline{1}\end{array}\right]$$\left[\begin{array}{c}\hat{a}\\ \hat{b}\end{array}\right]$ = $\left[\begin{array}{c}\underline{x}^T\underline{y} \\ \underline{y}^T\underline{1} \end{array}\right]$

Inverting the matrix and pre multiplying both sides of the above matrix equation by the inverse results in:

$\left[\begin{array}{c}\hat{a} \\ \hat{b} \end{array}\right]$ = $\frac{1}{N(\underline{x}^T\underline{x})-(\underline{x}^T\underline{1})^2}$$\left[\begin{array}{cc}\underline{1}^T\underline{1}&-\underline{x}^T\underline{1}\\ -\underline{x}^T\underline{1}&\underline{x}^T\underline{x}\end{array}\right]$$\left[\begin{array}{c}\underline{x}^T\underline{y} \\ \underline{y}^T\underline{1} \end{array}\right]$

If we revert to a more explicit notation in terms of the samples, we have

$\hat{a}=\frac{1}{N(\sum_{n=0}^{N-1}(x[n])^2)-(\sum_{n=0}^{N-1}x[n])^2}[N(\sum_{n=0}^{N-1}x[n]y[n])-(\sum_{n=0}^{N-1}y[n])(\sum_{n=0}^{N-1}x[n])]$

$\hat{b}=\frac{1}{N(\sum_{n=0}^{N-1}(x[n])^2)-(\sum_{n=0}^{N-1}x[n])^2}[(\sum_{n=0}^{N-1}(x[n])^2)(\sum_{n=0}^{N-1}y[n])-(\sum_{n=0}^{N-1}x[n])(\sum_{n=0}^{N-1}x[n]y[n])]$

where the denominator of the fraction in the above two equations is the matrix determinant.

Coefficient of Determination

Once we have calculated $\hat{a}$ and $\hat{b}$, we can see how well the linear model accounts for the data. This is achieved by finding the Coefficient of Determination ($r^2$). This is the ratio of the residual sum of squares ($rss$) to the total sum of squares ($tss$),

$\Large r^2=\frac{rss}{tss}$

and has a maximum value of 1, which is attained when the model is exactly linear. The larger $r^2$ is, the better the linear model fits the data. We can think of $r^2$ as the proportion of variation in data explained by the linear model, or the "goodness of fit" of the linear model.

The total sum of squares is given by

$tss=\sum_{n=0}^{N-1}(y[n]-\bar{y})^2$

where

$\bar{y}=\frac{1}{N}\sum_{n=0}^{N-1}y[n]$ is the mean of $y$.

The residual sum of squares ($rss$) is the total sum of squares ($tss$) minus the sum of squared errors ($sse$)

where

$sse=\sum_{n=0}^{N-1}(y[n]-\hat{a}x[n]-\hat{b})^2$

If the data perfectly fits the linear model, $sse$ will equal 0, so that $tss$ will equal $rss$ - the coefficient of determination $r^2$ will be $1$.

Summarising, we have

$\Large r^2=\frac{\sum_{n=0}^{N-1}(y[n]-\bar{y})^2-\sum_{n=0}^{N-1}(y[n]-\hat{a}x[n]-\hat{b})^2}{\sum_{n=0}^{N-1}(y[n]-\bar{y})^2}$

A worked example

Now, let us consider the following example:-

$x=[1,2,3,4]$, $y=[1.5,6,7,8]$

The matrix determinant will be $(4\times (1 + 2^2+3^2+4^2))-(1+2+3+4)^2=20$.

The least squares estimate of $a$ is

$\hat{a}=[4\times((1\times 1.5) + (2\times 6) + (3\times 7) + (4\times 8)) \\ -( (1.5+6+7+8)\times(1+2+3+4))]/20 \\ =2.05$

The least squares estimate of $b$ is

$\hat{b}=[(1+2^2+3^2+4^2)\times(1.5+6+7+8)-((1+2+3+4)\times((1\times 1.5) \\+ (2\times 6) + (3\times 7) +(4\times 8)))]/20 \\ = (675 - 665)/20 \\ =0.5$

The $tss$ is

$tss=(1.5-5.625)^2+(6-5.625)^2+(7-5.625)^2+(8-5.625)^2=24.688$

where the average of $y$ is $5.625$.

The $sse$ is

$(1.5 - (2.05\times 1) - 0.5)^2+(6 - (2.05\times 2) - 0.5)^2\\+(7 - (2.05\times 3) - 0.5)^2+(8 - (2.05\times 4) - 0.5)^2=3.675$

So that the $rss$ is $24.688-3.675=21.013$.

The coefficient of determination is

$r^2=21.013/24.688=0.8511$

This result is quite high, and is fairly indicative of a linear model.

There is a Linear Regression Calculator in this blog, which can be found here

16 comments:

  1. Hello Everyone out there,I am here to give my testimony about a Herbalist doctor who helped me . I was infected with HERPES SIMPLEX VIRUS in 2011, i went to many hospitals for cure but there was no solution, so I was thinking how can I get a solution out so that my body can be okay. One day I was in the river side thinking where I can go to get solution. so a lady walked to me telling me why am I so sad and i open up all to her telling her my problem, she told me that she can help me out, she introduce me to a doctor who uses herbal medication to cure HERPES SIMPLEX VIRUS and gave me his email, so i mail him. He told me all the things I need to do and also give me instructions to take, which I followed properly. Before I knew what is happening after two weeks the HERPES SIMPLEX VIRUS that was in my body got vanished . so if you are also heart broken and also need a help, you can also email him at {oliha.miraclemedicine@gmail.com}
    or whatsapp him number: +2349038382931. or website is https://olihamiraclemedicine.webs.com/
    Contact him today and you will have a testimony...Good luck!

    Dr. OLIHA Also Cures:
    1. HIV/AIDS
    2. HERPES 1/2
    3. CANCER
    4. ALS (Lou Gehrig’s disease)
    5. Hepatitis B
    6. chronic pancreatic
    7. Emphysema
    8. COPD (Chronic Obstructive Pulmonary Disease

    ReplyDelete
    Replies
    1. INDEED DR SAYO HERBAL HEALER HAS THE CURE FOR FOLLOWING HERPES VIRUS, HIV/AIDS, CANCER ALL TYPES, DIABETES, HEPATITIS A/B, PAD, ETC......FOR MORE INFORMATION VISIT HIS WEBSITE: http://sayoherbalhome.com/ I was once a victim of the herpes virus with many symptoms on my body, I was cured of the virus thanks to the help of Dr Sayo herbal medicine. I saw most of his patient testimonials on blogs and social media pages, cured of various disease/virus and illnesses. I contacted him on his email address to which he replied and asked me a few questions and the preparation method for which I paid, he sent me the herbal medicines and instructions on how to take it I apply the herbal medicine for 4 weeks and the symptoms on my body were healed after taking the herbal medicine treatment, I went for test and my result was negative with know trace of virus in my blood and am completely cured. You can also be cured, contact him on his email:  sayoherbalhealer@gmail.com  or WhatsApp him on +2349012175679 or visit his facebook page: https://www.facebook.com/SayoHerbalHealer

      Delete
  2. All thanks to Dr Raymond for helping me to get my penis longer and bigger with his cream in just two week his cream is very active and also it work fast for everyone who have used it, the best thing about his herbal cream is that it does not have side effects and the results is permanent. Note he can also cure the below sicknesses and disease PREMATURE EJACULATIONLOW SPERM COUNTERECTILE DYSFUNCTIONHIV/AIDS CUREHERPES CUREINFECTION CUREPILESTROKEHEART FAILUREYou can email him if you need his help via: drraymondherbalcenter@gmail.com or WhatsApp him via: +2348116744524

    ReplyDelete
  3. I'm here to testify about what DR. OYAGU did for me. I have been suffering from (GENITAL HERPES VIRUS) disease for the past 2 years and had constant pain and inching, especially in my private part. During the first year, I had faith in God that i would be cured someday. This disease started circulating all over my body and I have been taking treatment from my doctor, few weeks ago I came across a testimony of Shayla Bosschart on the internet testifying about a Man called DR. OYAGU on how he cured her from 7 years HSV 2. And she also gave the email address of this man, advise anybody to contact him for help on any kind of diseases that he would be of help, so I emailed him telling him about my (HSV 2) he told me not to worry that I was going to be cured!! Well, I never doubted him I have faith he can cure me too,, DR. OYAGU prepared and sent me Healing Soap, roots and herbs which I took. In the first one week, I started experiencing changes all over me, after two weeks of using his Roots/ Herbs, Oil and Soap, I was totally cured. no more inching , pain on me anymore as DR. OYAGU assured me. After some time I went to my doctor to do another test behold the result came out negative. So friends my advise is if you have such disease or know anyone who suffers from it or any other disease like HPV, HBV, HIV, ALS, HBP, CANCER etc. you can contact DR. OYAGU for help via email} oyaguherbalhome@gmail.com visit his website https://oyaguspellcaster.wixsite.com/oyaguherbalhome whatsapp him through his mobile} +2348101755322. , Thanks once again DR. OYAGU for making me a happy woman again.

    ReplyDelete
  4. Am Here To Give My Testimony About A Doctor Who Helped Me In My Life. I Was Infected With HERPES SIMPLEX VIRUS In 2019, I Went To Many Hospitals For Cure But There Was No Solution, So I Was Thinking How Can I Get A Solution Out So That My Body Can Be Okay. One Day I Was In The River Side Thinking Where I Can Go To Get Solution. So A Lady Walked To Me Telling Me Why Am I So Sad And I Open Up All To Her Telling Her My Problem, She Told Me That She Can Help Me Out, She Introduce Me To A Doctor Who Uses Herbal Medication To Cure HERPES SIMPLEX VIRUS And Gave Me His Email, So I Mail Him. He Told Me All The Things I Need To Do And Also Give Me Instructions To Take, Which I Followed Properly. Before I Knew What Is Happening After Two Weeks The HERPES SIMPLEX VIRUS That Was In My Body Got Vanished . So If You Are Also Heart Broken And Also Need A Help, You Can Also contact Him Through Dr. Oyagu reach him oyaguherbalhome@gmail.com visit his website https://oyaguspellcaster.wixsite.com/oyaguherbalhome Or whatsapp him via +2348101755322

    ReplyDelete
  5. DR OYAGU herbal medicine is a good remedy for herpes simplex virus cure. My name is MIGNON WRIGHT , am from USA,I am here to give my testimony about DR OYAGU who helped me in my life, i want to inform the public how i was cured from (HERPES SIMPLEX VIRUS) by DR OYAGU, i visited different hospital but they gave me list of drugs like Familiar, Zovirax, and Valtrex which is very expensive to treat the symptoms and never cured me. I was browsing through the Internet searching for remedy on HERPES and i saw comment of people talking about how DR OYAGU cured them, when i contacted him he gave me hope and send a Herbal medicine to me that i took and it seriously worked for me, am a free person now without any problem, my HERPES result came out negative, I pray to God to never let me and my family forget you in our life, am now cured permanently, you can also get your self cured my friends if you really need this doctor help, you can reach him on his website ;https://oyaguspellcaster.wixsite.com/oyaguherbalhome Email: oyaguherbalhome@gmail.com or via whatsapp +2348101755322

    ReplyDelete

  6. I'm 40 years old female I tested genital herpes (HSV1-2) positive in 2016. I was having bad outbreaks. EXTREMELY PAINFUL. I have try different kinds of drugs and treatment by the medical doctors all to know was avail. Six months ago I was desperately online searching for a helpful remedies for genital herpes (HSV1-2) cure, which I come across some helpful remedies on how Dr OYAGU have help so many people in curing genital herpes (HSV1-2) with the help of herbal treatment because I too believe there is someone somewhere in the world who can cure herpes completely. At of the past 2 months, however, I've been following his herpes protocol Via oyaguherbalhome@gmail.com or +2348101755322 and it stopped all outbreaks completely! To my greatest surprise I was cured completely by following the protocol of his herbal medicine . Don't be discouraged by the medical doctors. There is a cure for HSV with the help of herbs and roots by a herbalist Dr call Dr OYAGU he is so kind and truthful with his herbal treatment, kindly contact him for more information Via oyaguherbalhome@gmail.com or WhatsApp DR on +2348101755322 or visit his website https://oyaguspellcaster.wixsite.com/oyaguherbalhome you will be lucky as I am today

    ReplyDelete
  7. It's a pleasure for me to write this testimony about this wonderful thing that happened to me last week on how i got my HIV aids cured, i have been reading so-many post of some people who were cured of HIV, but i never believed them, I was hurt and depressed so I was too curious and wanted to try dr.odoma then i contacted him on his email on drodomasolutionhome20@gmail.com when i contact him, he assured me 100% that he will heal me, i pleaded with him to help me out, it's a great success that he healed just as he promised, he told me that in three days time that i should go and check on my HIV status, I was floored that when i went to the hospital to check of my status that i was HIV negative, i never thought possible that dr.odoma can do miracles, i never really believed in magic but I played along with a little hope and faith and after everything but dr odoma changed my life and made me a true believer you can contact him drodomasolutionhome20@gmail.com or whatsapp Number: +2348100649947..visit his website:
    https://drodomasolutionhome.simdif.com/
    (facebook page https://www.facebook.com/Drodomaherbalmedicine/) he said that he is also specialize on the following diseases: HERPES, HIV, ALS, HPV, DIABETES, HEPATITIS B, CANCER,SICKLE CELL, VIRGINAL DRYNESS, And Bring Back Your Ex back.

    ReplyDelete
  8. God is Good! I promised God that I would share my testimony on this blog. I had all the signs of STD Virus but I was not too sure as to which one. I did a lot of online research and scared myself straight for a whole week before going to see the nurse. She took one look at my genital part and first said that it could just be the anatomy of my body, then she said it looked like genital warts and that I may have herpes. I was devastated. She gave me some medicine for the herpes and some cream for the warts. I was also tested for every single STD including herpes. I went home and cried searching the web for all sorts of cures for herpes and awaiting my results. I saw a post whereby Dr. Oyagu cured Herpes and other diseases, I copied his contacts out and added him on whats app via (+2348101755322). The next day my test result was ready and i confirmed Herpes positive. I told Dr.Oyagu about my health problems and he assured me of cure. He prepared his herbal medicine and sent it to me. I took it for 14 days (2 weeks). Before the completion of the 14 days in which I completed the dose, the Blisters and Warts that were on my body was cleared. I went back for check-up and I was told I'm free from the virus. Dr. Oyagu cures all types of diseases and viruses with the help of his herbal medicine. You can reach Dr. Oyagu via his email address on (oyahuherbalhome@gmail.com) or WhatsApp him on (+2348101755322) Visit His website on https://oyaguspellcaster.wixsite.com/oyaguherbalhome

    ReplyDelete
  9. I have been suffering from a deadly disease (Hsv) for the past 2 years now, I had spent a lot of money going from one places to another, from churches to churches, hospitals have been my home every day residence. Constant checks up have been my hobby not until this faithful day, I was searching through the internet, I saw a testimony on how Dr Oyagu helped someone in curing his (Hsv) herpes disease, quickly I copied his email which is oyaguherbalhome@gmail.com just to give him a test I spoke to him, he asked me to do some certain things which I did, he told me that he is going to provide the herbal cure to me, which he did, then he asked me to go for medical checkup after some days after using the herbal cure, behold I was free from the deadly disease, he only asked me to post the testimony through the whole world, faithfully am doing it now, please brothers and sisters, he is great, I owe him in return. if you are having a similar problem just email him on oyaguherbalhome@gmail.com visit his website https://oyaguspellcaster.wixsite.com/oyaguherbalhome Or whatsapp him via +2348101755322

    ReplyDelete
  10. INDEED DR SAYO HERBAL HEALER HAS THE CURE FOR FOLLOWING HERPES VIRUS, HIV/AIDS, CANCER ALL TYPES, DIABETES, HEPATITIS A/B, PAD, ETC......FOR MORE INFORMATION VISIT HIS WEBSITE: http://sayoherbalhome.com/ I was once a victim of the herpes virus with many symptoms on my body, I was cured of the virus thanks to the help of Dr Sayo herbal medicine. I saw most of his patient testimonials on blogs and social media pages, cured of various disease/virus and illnesses. I contacted him on his email address to which he replied and asked me a few questions and the preparation method for which I paid, he sent me the herbal medicines and instructions on how to take it I apply the herbal medicine for 4 weeks and the symptoms on my body were healed after taking the herbal medicine treatment, I went for test and my result was negative with know trace of virus in my blood and am completely cured. You can also be cured, contact him on his email:  sayoherbalhealer@gmail.com  or WhatsApp him on +2349012175679 or visit his facebook page: https://www.facebook.com/SayoHerbalHealer

    ReplyDelete
  11. Are you battling with any health related issues or infection is not the end of your life, why not contact a herbal practitioner for cure Dr Oyagu has herbal remedies and permanent solution for all your health related problems. recommend him to you today only because i have also used his herbal medicine and it worked perfectly on my health problem, you can actually save yourself from giving your hard earned money to Scammers if only you can contact Dr Oyagu today for genuine and Permanent cure herbal products . Pls help share this post to save yourself and your love ones. Dr Oyagu is specialize in these areas, Permanent herbal supplements for anxiety and depression, HIV/AIDS, Diabetes, Epilepsy, Blood, Cancer, HPV, ALS, herpes etc. Email: oyaguherbalhome@gmail.com or visit his website https://oyaguspellcaster.wixsite.com/oyaguherbalhome Or Cell Whatsapp Number +2348101755322 thank you

    ReplyDelete
  12. Greatest thanks to Dr Oyagu for his herbal medicine that he prepared for me and when I start using it in just 2weeks I was completely cured and that ended my HERPES SIMPLEX 1&2 DISEASE i am so happy and grateful to Dr Oyagu . after reading about him on a testimony of Jason Cash on a blogger. I knew suddenly Dr Oyagu was the right Doctor to cure my HERPES SIMPLEX 1&2 DISEASE. I discuss with Dr Oyagu and he prepared a herbal medicine for me and when it got sent to me in south Korean . I used the herbal medicine in 2weeks and i went for check up again. after 15years of suffering from HERPES SIMPLEX 1&2 at last i am smiling once again. Dr Oyagu also has remedy to others disease like COLD SORES,HIV/AIDS,DIABETES.CANCER,HIGH BLOOD PRESSURE AND MANY MORE. I oblige everyone to contact this powerful herbalist Dr Oyagu and be free from your suffering. contact his WhatsApp line: +2348101755322 or visit his Website https://oyaguspellcaster.wixsite.com/oyaguherbalhome Email:Oyaguherbalhome@gmail.com

    ReplyDelete
  13. I got diagnose of herpes virus last year, and i was taking some drugs prescribed for me by my family doc the drugs could not work and the herpes in my system was very terrifying that i was so depressed the good news is that i never gave up in searching for natural cure for herpes virus ,cause i believe so much in herbal medication.. One faithful morning i read a comment from a lady called Destiny Hudson on how she was cured with natural herbs made by DR.OYAGU from somewhere is West Africa, i immediately copied out his contact email via oyaguherbalhome@gmail.com and explain all my herpes problem to him via his email. the big truth is that it took just two week for his herbal medication to cure me completely without side affect. Till date am herpes virus free and all thanks to Dr OYAGU for his good deeds for me, Once again am very happy to share this great testimony of DR OYAGU cure hurry up now and contact him via his email address (oyaguherbalhome@gmail.com Call & WhatsApp him on (+2348101755322 ) and see what he can do

    ReplyDelete
  14. Dear friends online My name is Destiny Hudson And I live in the USA, Dominica I am here today to appreciate the good work of Dr Oyagu, for helping me curing my HSV 1&2 Disease, I contacted the virus from my Partner, i found out he was cheating on me, so i broke up with him, 2 month later i went to test myself in the hospital when i was seeing sore on my mouth and Private areas, i was diagnosed of HSV 1&2. i tried using the acyclovir medicine but it didn't help in curing it, i went on a search and found out Dr Oyagu curing people illness, and so i got his email and i was told how i can get his herbal cure, 3 days later i received his medicine after using it for 2 weeks i was tested again and was confirm negative, today i am totally free. If you are over there looking for solution email: oyaguherbalhome@gmail.com or contact him on whatsapp +2348101755322 or visit his website https://oyaguspellcaster.wixsite.com/oyaguherbalhome today and share your own testimony. And He also has herbal cure for the Following DISEASES, eczema, urethra wart, chronic problems. Herpes, Cancer, Als, Hepatitis, Diabetes, HPV ,Infections, ulcer ETC

    ReplyDelete
  15. I want to use this opportunity to thank Dr Oyagu for the great work he has done for me and my cousin sister who was cured from Cancer with his herbal supplement, I was cured from herpes virus through his herbal remedy and my cousin sisters was also cured from Cancer. This is a great testimony for what Dr Oyagu Has done for me and my cousin sister . I’m giving this testimony so that those of you out there can contact him while he keeps saving my relatives. Send him a Dm and get a better health totally. Contact him via WhatsApp +2348101755322 or Email address:oyaguherbalhome@gmail.com you can also reach him on his website https://oyaguspellcaster.wixsite.com/oyaguherbalhome

    ReplyDelete