Linear and Logistic Regression

 While correlation is used to describe the strength and direction of an association between two random, uncontrolled continuous variables, regression analysis goes a step further by establishing a functional relationship that allows researchers to predict the value of a dependent (response) variable based on the value of at least one independent (predictor) variable. In the broader context of testing for relationships, regression provides a specific mathematical model to describe the exact nature of how variables interact.

Simple Linear Regression Simple linear regression evaluates the relationship between exactly one independent variable and one continuous dependent variable. In this model, researchers assume that a straight line provides the best description of the relationship, which can be expressed mathematically as $y = \beta_0 + \beta_1x + \epsilon$ (or $y = a + bx$), where $\beta_0$ is the y-intercept, $\beta_1$ is the slope, and $\epsilon$ represents random error.

To determine the most accurate straight line through a scatter plot of data, statisticians use the method of least squares. This objective mathematical procedure calculates the "best-fitting" line by minimizing the sum of the squared vertical deviations (residuals) between the actual observed data points and the predicted values on the line.

To evaluate how well this linear model actually fits the data, researchers calculate the coefficient of determination ($r^2$ or $R^2$). This value represents the proportion or percentage of the total variation in the dependent variable that is directly explained by the linear relationship with the independent variable. For example, if $r^2 = 0.95$, it means 95% of the variability in the outcome is explained by the regression line.

For simple linear regression to be statistically valid, several strict assumptions must be met:

• The independent variable ($x$) is fixed and controlled by the researcher without error.
• For any given value of $x$, the corresponding $y$ values are normally distributed.
• The variance of $y$ remains constant across all values of $x$ (homogeneity of variance).

Multiple Linear Regression Because a single predictor variable is rarely sufficient to accurately forecast an outcome in the real world, researchers use multiple linear regression to evaluate the effects of two or more independent variables simultaneously on a single continuous dependent variable.

Instead of a simple straight line, multiple regression creates a "plane" (for two independent variables) or a "hyperplane" (for three or more) in multidimensional space to best fit the data. This model calculates specific beta coefficients (beta weights) for each independent variable. These coefficients are highly valuable because they indicate the relative importance of each predictor variable and describe the change in the dependent variable when that specific predictor is altered while all other predictors are held constant. Because evaluating many variables can become incredibly complex, researchers often use automated stepwise regression to systematically add or subtract variables from the equation to identify the most statistically useful subset of predictors.

Logistic Regression While linear regression requires a continuous dependent variable, logistic regression is the appropriate predictive model when the dependent variable is a dichotomous or binary discrete outcome (e.g., success/failure, live/die, presence/absence of a disease).

Attempting to use standard linear regression for binary outcomes creates several mathematical problems: the errors cannot be normally distributed, the variance is not constant, and the linear equation could predict impossible probabilities greater than 1.0 or less than 0.0.

To solve this, logistic regression utilizes a logarithmic transformation called the logit function or log-odds. Instead of predicting the exact value of a continuous variable, logistic regression models the natural logarithm of the odds that the specific binary event will occur. This transformation successfully forces the predicted probabilities to always fall strictly between 0 and 1. Like multiple regression, logistic regression can evaluate multiple independent variables at the same time—and these predictor variables can be either continuous or discrete. It is heavily used in clinical and epidemiological studies, such as using factors like a mother's age, weight, and smoking history to predict the binary probability of a premature birth.