{"id":118,"date":"2014-10-16T21:26:25","date_gmt":"2014-10-16T21:26:25","guid":{"rendered":"http:\/\/opentextbc.ca\/introductorybusinessstatistics\/?post_type=chapter&p=118"},"modified":"2019-06-06T20:32:59","modified_gmt":"2019-06-06T20:32:59","slug":"regression-basics-2","status":"publish","type":"chapter","link":"https:\/\/opentextbc.ca\/introductorybusinessstatistics\/chapter\/regression-basics-2\/","title":{"raw":"Chapter 8. Regression Basics","rendered":"Chapter 8. Regression Basics"},"content":{"raw":"Regression analysis, like most multivariate statistics, allows you to infer that there is a relationship between two or more variables. These relationships are seldom exact because there is variation caused by many variables, not just the variables being studied.\r\n\r\nIf you say that students who study more make better grades, you are really hypothesizing that there is a positive relationship between one variable, studying, and another variable, grades. You could then complete your inference and test your hypothesis by gathering a sample of (amount studied, grades) data from some students and use regression to see if the relationship in the sample is strong enough to safely infer that there is a relationship in the population. Notice that even if students who study more make better grades, the relationship in the population would not be perfect; the same amount of studying will not result in the same grades for every student (or for one student every time). Some students are taking harder courses, like chemistry or statistics; some are smarter; some study effectively; and some get lucky and find that the professor has asked them exactly what they understood best. For each level of amount studied, there will be a distribution of grades. If there is a relationship between studying and grades, the location of that distribution of grades will change in an orderly manner as you move from lower to higher levels of studying.\r\n\r\nRegression analysis is one of the most used and most powerful multivariate statistical techniques for it infers the existence and form of a functional relationship in a population. Once you learn how to use regression, you will be able to estimate the parameters \u2014 the slope and intercept \u2014 of the function that\u00a0links two or more variables. With that estimated function, you will be able to infer or forecast things like unit costs, interest rates, or sales over a wide range of conditions. Though the simplest regression techniques seem limited in their applications, statisticians have developed a number of variations on regression that\u00a0greatly expand the usefulness of the technique. In this chapter, the basics will be discussed. Once again, the t-distribution and F-distribution will be used to test hypotheses.\r\n

What is regression?<\/h1>\r\nBefore starting to learn about regression, go back to algebra and review what a function is. The definition of a function can be formal, like the one in my freshman calculus text: \"A function is a set of ordered pairs of numbers (x<\/em>,y<\/em>) such that to each value of the first variable (x<\/em>) there corresponds a unique value of the second variable (y<\/em>)\" (Thomas, 1960).[footnote]Thomas, G.B. (1960). Calculus and analytical geometry<\/em> (3rd ed.). Boston, MA: Addison-Wesley.[\/footnote].\u00a0More intuitively, if there is a regular relationship between two variables, there is usually a function that describes the relationship. Functions are written in a number of forms. The most general is y<\/em> = f(x<\/em>)<\/strong>, which simply says that the value of y depends on the value of x in some regular fashion, though the form of the relationship is not specified. The simplest functional form is the linear function where:\r\n\r\n\"Linear<\/a>\r\n\r\n\u03b1<\/em> and \u03b2<\/em> are parameters, remaining constant as x<\/em> and y<\/em> change. \u03b1<\/em> is the intercept and \u03b2<\/em> is the slope. If the values of\u00a0\u03b1<\/em> and \u03b2<\/em> are known, you can find the y<\/em> that goes with any x<\/em> by putting the x<\/em> into the equation and solving. There can be functions where one variable depends on the values values of two or more other variables where\u00a0x1<\/sub><\/em>\u00a0and\u00a0x2<\/sub><\/em>\u00a0together determine the value of y<\/em>. There can also be non-linear functions, where the value of the dependent variable (y<\/strong><\/em> in all of the examples we have used so far) depends on the values of one or more other variables, but the values of the other variables are squared, or taken to some other power or root or multiplied together, before the value of the dependent variable is determined. Regression allows you to estimate directly the parameters in linear functions only, though there are tricks that\u00a0allow many non-linear functional forms to be estimated indirectly. Regression also allows you to test to see if there is a functional relationship between the variables, by testing the hypothesis that each of the slopes has a value of zero.\r\n\r\nFirst, let us consider the simple case of a two-variable function. You believe that y<\/em>, the dependent variable, is a linear function of x<\/em>, the independent variable \u2014 y<\/em> depends on x<\/em>. Collect a sample of (x<\/em>, y<\/em>) pairs, and plot them on a set of x<\/em>, y<\/em> axes. The basic idea behind regression is to find the equation of the straight line that comes as close as possible to as many of the points as possible. The parameters of the line drawn through the sample are unbiased estimators of the parameters of the line that would come as close as possible to as many of the points as possible in the population, if the population had been gathered and plotted. In keeping with the convention of using Greek letters for population values and Roman letters for sample values, the line drawn through a population is:\r\n\r\n\"Linear<\/a>\r\n\r\nwhile the line drawn through a sample is:\r\n\r\ny<\/em> = a<\/em> + bx<\/em>\r\n\r\nIn most cases, even if the whole population had been gathered, the regression line would not go through every point. Most of the phenomena that business researchers deal with are not perfectly deterministic, so no function will perfectly predict or explain every observation.\r\n\r\nImagine that you wanted to study the estimated price for a one-bedroom apartment in Nelson, BC. You decide to estimate the price\u00a0as a function of its location in relation to downtown. If you collected 12 sample pairs, you would find different apartments located within the same distance from downtown. In other words, you might draw a distribution of prices for apartments located at the same distance from downtown or away from downtown. When you use regression to estimate the parameters of price = f(distance), you are estimating the parameters of the line that connects the mean price\u00a0at each location. Because the best that can be expected is to predict the mean price\u00a0for a certain location, researchers often write their regression models with an extra term, the error term<\/strong>, which notes that many of the members of the population of (location, price of apartment) pairs will not have exactly the predicted price\u00a0because many of the points do not lie directly on the regression line. The error term is usually denoted as \u03b5<\/em><\/strong>, or epsilon<\/strong>, and you often see regression equations written:\r\n\r\n\"Regression<\/a>\r\n\r\nStrictly, the distribution of \u03b5<\/em> at each location\u00a0must be normal, and the distributions of \u03b5<\/em> for all the locations\u00a0must have the same variance (this is known as homoscedasticity to statisticians).\r\n

Simple regression and least squares method<\/h1>\r\nIn estimating the unknown parameters of the population for the regression line, we need to apply a method by which the vertical distances between the yet-to-be estimated regression line and the observed values in our sample are minimized. This minimized distance is called sample error,<\/em> though it is more commonly referred to as residual<\/em> and denoted by e.\u00a0<\/em>In more mathematical form, the difference between the y<\/em> and its predicted value\u00a0<\/em>is the residual in each pair of observations for x<\/em> and y<\/em>. Obviously, some of these residuals will be positive (above the estimated line) and others will be negative\u00a0(below the line). If we add all these residuals over the sample size and raise them to the power 2\u00a0in order to prevent the chance those positive and negative signs are cancelling each other out, we can write the following criterion for our minimization problem:\r\n\r\n\"image7\"<\/a>\r\n\r\nS<\/em> is the sum of squares of the residuals. By minimizing S<\/em> over any given set of observations for x<\/em>\u00a0and y<\/em>, we will get the following useful formula:\r\n\r\n$latex b=\\frac{\\sum{(x-\\bar{x})(y-\\bar{y})}}{\\sum{(x-\\bar{x})^2}}$\r\n\r\nAfter computing the value of b<\/em> from the above formula out of our sample data, and the means of the two series of data on x\u00a0<\/em>and y<\/em>, one can simply recover the intercept of the estimated line using the following equation:\r\n\r\n\"image9\"<\/a>\r\n\r\nFor the sample data, and given the estimated intercept and slope, for each observation we can define a residual as:\r\n\r\n\"image11\"<\/a>\r\n\r\nDepending on the estimated values for intercept and slope, we can draw the estimated line along with all sample data in a y<\/em>-x<\/em> panel. Such graphs are known as scatter diagrams. Consider our analysis of the price of one-bedroom apartments in Nelson, BC. We would collect data for y=<\/em>price of one bedroom apartment, x1<\/sub><\/em>=its associated distance from downtown, and x2<\/sub><\/em>=the size of the apartment, as shown in Table 8.1.\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n
Table 8.1 Data for Price, Size, and Distance of Apartments in Nelson, BC<\/caption>\r\n
y<\/em> = price of apartments in $1000\r\nx1<\/sub><\/em> = distance of each apartment from downtown in kilometres\r\nx2<\/sub><\/em> = size of the apartment in square feet<\/td>\r\n<\/tr>\r\n
y<\/strong><\/td>\r\nx1<\/sub><\/strong><\/td>\r\nx2<\/sub><\/strong><\/td>\r\n<\/tr>\r\n
55<\/td>\r\n1.5<\/td>\r\n350<\/td>\r\n<\/tr>\r\n
51<\/td>\r\n3<\/td>\r\n450<\/td>\r\n<\/tr>\r\n
60<\/td>\r\n1.75<\/td>\r\n300<\/td>\r\n<\/tr>\r\n
75<\/td>\r\n1<\/td>\r\n450<\/td>\r\n<\/tr>\r\n
55.5<\/td>\r\n3.1<\/td>\r\n385<\/td>\r\n<\/tr>\r\n
49<\/td>\r\n1.6<\/td>\r\n210<\/td>\r\n<\/tr>\r\n
65<\/td>\r\n2.3<\/td>\r\n380<\/td>\r\n<\/tr>\r\n
61.5<\/td>\r\n2<\/td>\r\n600<\/td>\r\n<\/tr>\r\n
55<\/td>\r\n4<\/td>\r\n450<\/td>\r\n<\/tr>\r\n
45<\/td>\r\n5<\/td>\r\n325<\/td>\r\n<\/tr>\r\n
75<\/td>\r\n0.65<\/td>\r\n424<\/td>\r\n<\/tr>\r\n
65<\/td>\r\n2<\/td>\r\n285<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\nThe graph (shown in Figure 8.1) is a scatter plot of the prices of the apartments and their distances from downtown, along with a proposed regression line.\r\n\r\n[caption id=\"attachment_1235\" align=\"aligncenter\" width=\"730\"]\"Figure8-1\"<\/a> Figure 8.1 Scatter Plot of Price, Distance from Downtown, along with a Proposed Regression Line[\/caption]\r\n\r\nIn order to plot such a scatter diagram, you can use many available statistical software packages including Excel, SAS, and Minitab.\u00a0In this scatter diagram, a negative simple regression line has been shown. The estimated equation for this scatter diagram from Excel is:\r\n\r\n\"image13\"<\/a>\r\n\r\nWhere a<\/em>=71.84 and\u00a0b<\/em>=-5.38. In other words, for every additional kilometre from downtown an apartment is located, the price of the apartment is estimated to be $5380 cheaper, i.e.\u00a05.38*$1000=$5380. One might also be curious about the fitted values out of this estimated model. You can simply plug the actual value for x<\/em> into the estimated line, and find the fitted values for the prices of the apartments. The residuals for all 12 observations are shown in Figure 8.2.\r\n\r\n[caption id=\"attachment_845\" align=\"aligncenter\" width=\"107\"]\"Residuals_Simple<\/a> Figure 8.2[\/caption]\r\n\r\nYou should also notice\u00a0that by minimizing errors, you have not eliminated them; rather, this method of least squares only guarantees the best fitted<\/em> estimated regression line out of the sample data.\r\n\r\nIn the presence of the remaining errors, one should be aware of the fact that there are still other factors that might not have been included in our regression model and\u00a0are responsible for the fluctuations in the remaining errors. By adding these excluded but relevant factors to the model, we probably expect the remaining error will show less meaningful fluctuations. In determining the price of these apartments, the missing factors may include age of the apartment, size, etc. Because this type of regression model does not include many relevant factors and assumes only a linear relationship, it is known as a simple linear regression model.\r\n

Testing your regression: does y<\/em> really depend on x<\/em>?<\/h2>\r\nUnderstanding that there is a distribution of y<\/em> (apartment price) values at each x<\/em> (distance) is the key for understanding how regression results from a sample can be used to test the hypothesis that there is (or is not) a relationship between x<\/em> and y<\/em>. When you hypothesize that y<\/em> = f(x<\/em>), you hypothesize that the slope of the line (\u03b2<\/em> in y<\/em> = \u03b1<\/em> + \u03b2x<\/em> + \u03b5<\/em>) is not equal to zero. If \u03b2<\/em> was equal to zero, changes in x<\/em> would not cause any change in y<\/em>. Choosing a sample of apartments, and finding each apartment\u2019s distance to downtown, gives you a sample of (x<\/em>, y<\/em>). Finding the equation of the line that best fits the sample will give you a sample intercept, \u03b1<\/em>, and a sample slope, \u03b2<\/em>. These sample statistics are unbiased estimators of the population intercept, \u03b1<\/em>, and slope, \u03b2<\/em>. If another sample of the same size is taken, another sample equation could be generated. If many samples are taken, a sampling distribution of sample \u03b2<\/em>\u2019s, the slopes of the sample lines, will be generated. Statisticians know that this sampling distribution of b<\/em>\u2019s will be normal with a mean equal to \u03b2<\/em>, the population slope. Because the standard deviation of this sampling distribution is seldom known, statisticians developed a method to estimate it from a single sample. With this estimated sb<\/sub><\/em>, a t-statistic for each sample can be computed:\r\n\r\n\"T-statistic\"<\/a>\r\n\r\nwhere n<\/em> = sample size\r\n\r\nm<\/em> = number of explanatory (x<\/em>) variables\r\n\r\nb<\/em> = sample slope\r\n\r\n\u03b2<\/em>= population slope\r\n\r\nsb<\/sub><\/em> = estimated standard deviation of b\u2019s, often called the standard error<\/strong>\r\n\r\nThese t<\/em>\u2019s follow the t-distribution in the tables with n<\/em>-m<\/em>-1 df.\r\n\r\nComputing sb<\/sub><\/em> is tedious, and is almost always left to a computer, especially when there is more than one explanatory variable. The estimate is based on how much the sample points vary from the regression line. If the points in the sample are not very close to the sample regression line, it seems reasonable that the population points are also widely scattered around the population regression line and different samples could easily produce lines with quite varied slopes. Though there are other factors involved, in general when the points in the sample are farther from the regression line, sb<\/sub><\/em> is greater. Rather than learn how to compute sb<\/sub><\/em>, it is more useful for you to learn how to find it on the regression results that you get from statistical software. It is often called the standard error and there is one for each independent variable. The printout in Figure 8.3 is typical.\r\n\r\n[caption id=\"attachment_350\" align=\"aligncenter\" width=\"608\"]\"Simple_Regression\"<\/a> Figure 8.3 Typical Statistical Package Output for Linear Simple Regression Model[\/caption]\r\n\r\nYou will need these standard errors in order to test to see if y<\/em> depends on x<\/em> or not. You want to test to see if the slope of the line in the population, \u03b2<\/em>, is equal to zero or not. If the slope equals zero, then changes in x<\/em> do not result in any change in y<\/em>. Formally, for each independent variable, you will have a test of the hypotheses:\r\n\r\n$latex H_o: \\beta = 0 $\r\n\r\n$latex H_a: \\beta \\neq 0 $\r\n\r\nIf the t-score is large (either negative or positive), then the sample b<\/em> is far from zero (the hypothesized \u03b2<\/em>), and Ha<\/sub><\/em> should be accepted. Substitute zero for b into the t-score equation, and if the t-score is small, b<\/em> is close enough to zero to accept Ha<\/sub><\/em>. To find out what t-value separates \"close to zero\" from \"far from zero\", choose an alpha, find the degrees of freedom, and use a t-table from any textbook, or simply use the interactive Excel template from Chapter 3<\/a>, which is shown again in Figure 8.4.\r\n\r\n