(upbeat music)
- Hello.
I am Professor Rick Jerz, and I welcome you
to this session titled Multiple Regression Analysis.
Multiple regression analysis is an extension
on what we learned in our previous session
about correlation and simple regression analysis.
The difference is that in our previous session,
we were only answering the question
of whether or not one set of data
is driving the values in another set of data.
In this session, we extend this question a little bit
and ask whether there might be multiple sets
of variables, meaning multiple independent variables
that are driving the results of another set of data.
As you can guess, the mathematics
get much more complex.
Don't worry.
Once again, we are going to use an Excel model
to ease the calculation burden.
One thing I want to mention to you about this topic
is that it is really a very complex topic.
One of the most complex in the study of statistics.
Some universities offer multiple courses on this topic.
We are just going to spend this one session on this topic,
but my Excel model will allow you to explore
the many issues that are commonly addressed
in advanced courses on this topic.
Let's get started.
In my previous session on correlation and regression
analysis, we explored the relationship
between two variables, an independent variable
and a dependent variable.
A logical question to ask is,
might there be other variables, meaning more than one,
that can help us predict the dependent variable?
Think of modeling the stock market
in order to predict a stock's price.
Might stock price be related to unemployment rates,
to general business conditions, to general economic
conditions, maybe the prime rate, maybe housing starts,
maybe the price of gas, et cetera.
Using more than one variable to predict another
gets us into multiple regression analysis.
Its concept is very similar to what we learned
in simple linear regression analysis.
We seek to develop an equation that involves
an intercept and parameters that represent
the slopes of each variable.
For multiple regression,
we prefer to call the slopes coefficients.
One nice thing about simple linear regression
analysis is that we can plot the variables
and their relationship in a simple 2D plot,
meaning a plot of X versus Y on a piece of paper.
This is what we did when we created a scatter diagram
for simple regression analysis.
If we were to have two predictors, such as X1 and X2,
we would need 3D space.
We can still plot 3D space on a piece of paper
by representing it as a cube.
If we want to use, oh, let's say seven values
to predict something, well, you just can't
represent this in physical space,
but you can represent it mathematically.
Here's an equation representing
general multiple regression with k independent variables
given by this long equation.
Remember that the three dots implies
that there can be many more factors in this equation.
The least squares criteria is used
to develop this equation.
Because determining b1, b2, and so forth
is very tedious, many statisticians and textbook authors
suggest resorting to statistical software packages
such as MINITAB or if you use Excel,
then use Excel's Data Analysis ToolPak.
Let's take a look at an example to appreciate
multiple regression analysis.
Salsberry Realty sells homes
along the East Coast of the United States.
One of the questions most frequently asked
by prospective buyers is, if we purchase this home,
how much can we expect to pay
to heat it during the winter?
The research department at Salsberry has been asked
to develop some guidelines regarding heating costs
for single-family homes.
Three variables are thought to relate
to the heating costs.
The first one is mean daily outside temperature.
The second is the number of inches
of insulation in the attic.
And the third is the age in years of the furnace.
To investigate, Salesberry's research department
selected a random sample of 20 recently sold homes.
It also determined the cost to heat each home
last January.
Here's the data that was collected.
Let's look at this data so that we can understand
what we are after.
In this situation we are trying to predict heating cost.
Heating cost is the dependent or Y value,
and the prediction is represented by Y-hat.
In simple linear regression, we also have
one dependent variable, so whether we are doing
simple or multiple regression,
we always are trying to predict one thing.
But now we have three independent variables,
outside temperature, attic insulation,
and age of furnace.
You might be thinking, with three independent variables,
let's just do simple regression three times.
However, it is more complex than that
because these three independent variables
interact with each other.
Conceptually, each independent variable has a slope,
or what we call a coefficient in multiple regression,
and together they have an intercept.
Some textbook authors like ours
don't even attempt to provide you equations
for finding the coefficients because most students
get overwhelmed by them.
Our author shows how to read the output
of Excel's Data Analysis ToolPak
to find the coefficients and intercept.
I will be providing you my own Excel model
that produces these same results.
Here is the regressed equation and its interpretation.
The positive coefficients mean a positive
or direct relationship, and the negative coefficient
means a negative or inverse relationship.
You have to remember that X1
was for the outside temperature,
X2 for the attic insulation,
and X3 for furnace age.
Once we have this multiple regress line,
we can use it to calculate an estimated heating cost
for any home.
For example, we might ask, what is the estimated
heating cost for a home if the mean outside temperature
is 30 degrees, there are five inches of insulation
in the attic, and the furnace is 10 years old?
When we put these values into our multiple regressed
equation, we calculate that the estimated heating cost
is $276.56.
Realize that mathematically using
the least squares technique, we can always calculate
the multiple regressed straight line.
The real question is, how good is it?
We asked this question back in simple regression analysis,
and we learned that we could calculate
a correlation coefficient
and a coefficient of determination.
Well guess what.
We can calculate these same statistics
for our multiple regression analysis
and they have the same meaning and value ranges.
This equation shows the calculation
for the coefficient of multiple determination.
Recall that the coefficient of correlation
is the square root of the coefficient of determination.
Although this form of the equation for R squared
appears simple, the SSR and SS total values
are not easy to calculate.
Recall that the coefficient of determination
is the proportion of the total variation
in the dependent variable Y that is explained
or accounted for by the variation
in the independent variables.
For multiple regression analysis we have a problem.
It was discovered that when you have more than one
independent variable, that the standard equation
for calculating the coefficient of determination
is flawed and needs to be adjusted.
So another equation was developed,
and it is referred to as the adjusted
coefficient of determination.
Recall that in simple regression analysis,
we tested that the coefficient of correlation
was not zero.
We can do the same thing for multiple regression.
We can test that all coefficients
are not equal to zero.
This is sometimes referred to as the global test.
If we reject the hypothesis that all of the coefficients
are zero, this implies that there is a relationship.
We will use p-values for this test.
In addition to the global test for multiple regression,
we can also test that each
independent variable's coefficient is not zero.
Once again, we will do this using p-values.
An important characteristic about these tests
is that when an independent variable
is not significant, meaning it has a p-value
greater than our level of significance,
this implies that this variable
should not be included in our model.
This becomes a very important test
with very important implications, so let me clarify.
When developing a model for prediction,
you can have an infinite number
of independent variables to include.
However, some of these variables may not help
our model predict.
When this happens, remove them from your model.
Set a little differently.
Include only independent variables
that have small p-values for their coefficients.
I should note that Excel's Data Analysis ToolPak
for regression analysis does calculate
these coefficient p-values.
What is interesting or some would say more complex
is that these p-value calculations
depend on how you add independent variables
to your model.
A technique called stepwise regression
attempts to address how to add and remove
independent variables.
The challenge with stepwise regression
is doing the calculations.
Every time we add another independent variable,
we have to go through all the mathematics
to decide whether or not it should be included or removed.
Even using Excel's Data Analysis ToolPaks
requires re-running this tool.
Sometimes stepwise regression is reserved
for more advanced courses
in multiple regression analysis.
Let me introduce another concept,
a concept called dummy variables.
I don't particularly like the word dummy
because there's nothing dumb about this concept.
There may be times where you want
to introduce an independent variable
that does not have a value.
In our example of estimating home heating costs,
we might want to introduce the variable
for whether or not a home
has or does not have a swimming pool.
Yes it does, no it doesn't.
How do we bring this into our model?
This is the role of dummy variables.
We cannot use the words yes or no within our model,
but we can represent a yes with a one and a no with a zero.
What this implies is that we could have
a multiple regression equation that includes
qualitative or categorical values.
This is good news.
It makes multiple regression analysis
even more flexible and more powerful.
So, there we have it.
Multiple regression analysis, a pretty powerful
statistical analytical technique.
Yet so mathematically intense
that our author doesn't give us all the equations.
Unfortunately, I have to mention one more big issue.
Under some conditions we cannot use this technique.
This technique has some assumptions.
I will take a moment and review
some of these assumptions and the test
for these assumptions.
The first assumption is that there is a linear relationship
for each independent variable.
If the relationship is not linear,
such as a curve, then that independent variable
should not be included.
We can test this assumption
by using a scatter diagram to plot
the dependent variable against the independent variable.
We might approach this test by running
simple regression analysis for each independent variable.
Graphically, the variables might look like this.
The scatter plot of cost versus temperature on the left
seems to have a linear relationship.
The scatter plot of cost versus insulation
on the right does not.
This suggests that insulation should be removed
from our model.
The second assumption is that the variation
of residuals are the same.
This assumption is sometimes referred to
as homoscedasticity.
Wow, what a word, and you'll never use this word
at the dinner table.
And what might we ask is a residual?
Very simply, a residual is the error
between what our regressed equation predicts
at one of the values for the independent variable
and what the actual value is.
Let me say this again.
The difference between the predicted
and the actual observed value.
That's a residual.
We can test this assumption again
with the scatter diagram by plotting
the predicted value on the horizontal axis
versus the residuals on the vertical axis.
There should be no trend or correlation.
This left graph represents a condition
where we have not violated this assumption.
There is no apparent trend between the plotted residuals
and the predicted values.
However, in the graph shown on the right,
there does seem to be a trend.
This suggests a violation of the second assumption.
The third assumption is that the residuals
are normally distributed.
We can use a frequency diagram to plot the residuals.
These residuals should be normally distributed.
We can also use a statistical test
called the Chi-squared test for the actual distribution
compared to a perfect normal distribution.
Here is an example of what a frequency diagram
might look like when we plot the residuals.
Does it appear to be a normal distribution?
The fourth assumption is that the independent variables
should not be correlated.
This is called multicollinearity.
Once again, a big fancy word.
Let me give you an example.
Suppose we want to predict the weight
of a person based upon their height.
Sure, this might make sense.
A taller person might weigh more.
But let's assume somebody says,
let's add another variable,
the distance from the floor to that person's navel,
sometimes referred to as your belly button.
And then somebody says, how about adding
the distance from your navel to the top of your head?
Okay, we have three length or distance variables,
but think about these three.
Isn't the height of a person also somewhat
directly related to the distance
from the floor to that person's navel,
and also from the navel to the top of the head?
These three variables are somewhat related to each other,
and they are not independent.
And if we were to include these three variables
in our model, we would be making
the simple measurement of height become too powerful.
To test for multicollinearity, we can calculate
another statistics called
the variance inflation factor, or VIF.
The VIF should be less than 10
in order to not violate this assumption.
Well, no problem.
Another statistic to calculate.
We have already been going crazy
with all these other calculations,
so why not throw in another one?
Typically, multicollinearity is reserved
for more advanced courses
in multiple regression analysis.
Excel's Data Analysis ToolPak doesn't do this calculation.
However, I have included the calculation
for multicollinearity in my Excel model.
The last assumption is that successive residuals
should be independent.
This is sometimes called autocorrelation.
To test this assumption, we can use
a scatter diagram just like we did for the first assumption.
There should not be any pattern of negative
or positive trend, meaning the slope
should be equal to zero.
Here is a graph showing an example
of autocorrelation being violated.
Notice that there is some trend
to the slope of this data.
The last item that I have included
on some of the assumptions and tests
is to be careful predicting outside the data range.
The further we get from the data
that allowed us to produce our model,
the less precise it becomes in making predictions.
Excel's Data Analysis ToolPak does not provide
any features for making predictions
or calculating confidence intervals
or prediction intervals for these predictions.
Well, there we have it.
Multiple regression analysis.
In my next session I will show you
how to use my Excel model to do all these calculations.
(upbeat music)
Thanks for taking time to spend some moments with me.
I hope that you have enjoyed what I have shared with you.
If you ever have any questions or need my help,
please let me know.