Regression Analysis and Linear Models (PDF) von Richard B. Darlington

List of Symbols and Abbreviations  1. Statistical Control and Linear Models 1.1 Statistical Control 1.1.1 The Need for Control 1.1.2 Five Methods of Control 1.1.3 Examples of Statistical Control 1.2 An Overview of Linear Models 1.2.1 What You Should Know Already 1.2.2 Statistical Software for Linear Modeling and Statistical Control 1.2.3 About Formulas 1.2.4 On Symbolic Representations 1.3 Chapter Summary 2. The Simple Regression Model 2.1 Scatterplots and Conditional Distributions 2.1.1 Scatterplots 2.1.2 A Line through Conditional Means 2.1.3 Errors of Estimate 2.2 The Simple Regression Model 2.2.1 The Regression Line 2.2.2 Variance, Covariance, and Correlation 2.2.3 Finding the Regression Line 2.2.4 Example Computations 2.2.5 Linear Regression Analysis by Computer 2.3 The Regression Coefficient versus the Correlation Coefficient 2.3.1 Properties of the Regression and Correlation Coefficients 2.3.2 Uses of the Regression and Correlation Coefficients 2.4 Residuals 2.4.1 The Three Components of Y 2.4.2 Algebraic Properties of Residuals 2.4.3 Residuals as Y Adjusted for Differences in X 2.4.4 Residual Analysis 2.5 Chapter Summary 3. Partial Relationship and the Multiple Regression Model 3.1 Regression Analysis with More Than One Predictor Variable 3.1.1 An Example 3.1.2 Regressors 3.1.3 Models 3.1.4 Representing a Model Geometrically 3.1.5 Model Errors 3.1.6 An Alternative View of the Model 3.2 The Best-Fitting Model 3.2.1 Model Estimation with Computer Software 3.2.2 Partial Regression Coefficients 3.2.3 The Regression Constant 3.2.4 Problems with Three or More Regressors 3.2.5 The Multiple Correlation R 3.3 Scale-Free Measures of Partial Association 3.3.1 Semipartial Correlation 3.3.2 Partial Correlation 3.3.3 The Standardized Regression Coefficient 3.4 Some Relations among Statistics 3.4.1 Relations among Simple, Multiple, Partial, and Semipartial Correlations 3.4.2 Venn Diagrams 3.4.3 Partial Relationships and Simple Relationships May Have Different Signs 3.4.4 How Covariates Affect Regression Coefficients 3.4.5 Formulas for bj, prj, srj, and R 3.5 Chapter Summary 4. Statistical Inference in Regression 4.1 Concepts in Statistical Inference 4.1.1 Statistics and Parameters 4.1.2 Assumptions for Proper Inference 4.1.3 Expected Values and Unbiased Estimation 4.2 The ANOVA Summary Table 4.2.1 Data = Model + Error 4.2.2 Total and Regression Sums of Squares 4.2.3 Degrees of Freedom 4.2.4 Mean Squares 4.3 Inference about the Multiple Correlation 4.3.1 Biased and Less Biased Estimation of TR2 4.3.2 Testing a Hypothesis about TR 4.4 The Distribution of and Inference about a Partial Regression Coefficient 4.4.1 Testing a Null Hypothesis about Tbj 4.4.2 Interval Estimates for Tbj 4.4.3 Factors Affecting the Standard Error of bj 4.4.4 Tolerance 4.5 Inferences about Partial Correlations 4.5.1 Testing a Null Hypothesis about Tprj and Tsrj 4.5.2 Other Inferences about Partial Correlations 4.6 Inferences about Conditional Means 4.7 Miscellaneous Issues in Inference 4.7.1 How Great a Drawback Is Collinearity? 4.7.2 Contradicting Inferences 4.7.3 Sample Size and Nonsignificant Covariates 4.7.4 Inference in Simple Regression (When k = 1) 4.8 Chapter Summary 5. Extending Regression Analysis Principles 5.1 Dichotomous Regressors 5.1.1 Indicator or Dummy Variables 5.1.2 Y Is a Group Mean 5.1.3 The Regression Coefficient for an Indicator Is a Difference 5.1.4 A Graphic Representation 5.1.5 A Caution about Standardized Regression Coefficients for Dichotomous Regressors 5.1.6 Artificial Categorization of Numerical Variables 5.2 Regression to the Mean 5.2.1 How Regression Got Its Name 5.2.2 The Phenomenon 5.2.3 Versions of the Phenomenon 5.2.4 Misconceptions and Mistakes Fostered by Regression to the Mean 5.2.5 Accounting for Regression to the Mean Using Linear Models 5.3 Multidimensional Sets 5.3.1 The Partial and Semipartial Multiple Correlation 5.3.2 What It Means If PR = 0 or SR = 0 5.3.3 Inference Concerning Sets of Variables 5.4 A Glance at the Big Picture 5.4.1 Further Extensions of Regression 5.4.2 Some Difficulties and Limitations 5.5 Chapter Summary 6. Statistical versus Experimental Control 6.1 Why Random Assignment? 6.1.1 Limitations of Statistical Control 6.1.2 The Advantage of Random Assignment 6.1.3 The Meaning of Random Assignment 6.2 Limitations of Random Assignment 6.2.1 Limitations Common to Statistical Control and Random Assignment 6.2.2 Limitations Specific to Random Assignment 6.2.3 Correlation and Causation 6.3 Supplementing Random Assignment with Statistical Control 6.3.1 Increased Precision and Power 6.3.2 Invulnerability to Chance Differences between Groups 6.3.3 Quantifying and Assessing Indirect Effects 6.4 Chapter Summary 7. Regression for Prediction 7.1 Mechanical Prediction and Regression 7.1.1 The Advantages of Mechanical Prediction 7.1.2 Regression as a Mechanical Prediction Method 7.1.3 A Focus on R Rather Than the Regression Weights 7.2 Estimating True Validity 7.2.1 Shrunken versus Adjusted R 7.2.2 Estimating TRS 7.2.3 Shrunken R Using Statistical Software 7.3 Selecting Predictor Variables 7.3.1 Stepwise Regression 7.3.2 All Subsets Regression 7.3.3 How Do Variable Selection Methods Perform? 7.4 Predictor Variable Configurations 7.4.1 Partial Redundancy (the Standard Configuration) 7.4.2 Complete Redundancy 7.4.3 Independence 7.4.4 Complementarity 7.4.5 Suppression 7.4.6 How These Configurations Relate to the Correlation between Predictors 7.4.7 Configurations of Three or More Predictors 7.5 Revisiting the Value of Human Judgment 7.6 Chapter Summary 8. Assessing the Importance of Regressors 8.1 What Does It Mean for a Variable to Be Important? 8.1.1 Variable Importance in Substantive or Applied Terms 8.1.2 Variable Importance in Statistical Terms 8.2 Should Correlations Be Squared? 8.2.1 Decision Theory 8.2.2 Small Squared Correlations Can Reflect Noteworthy Effects 8.2.3 Pearson’s r as the Ratio of a Regression Coefficient to Its Maximum Possible Value 8.2.4 Proportional Reduction in Estimation Error 8.2.5 When the Standard Is Perfection 8.2.6 Summary 8.3 Determining the Relative Importance of Regressors in a Single Regression Model 8.3.1 The Limitations of the Standardized Regression Coefficient 8.3.2 The Advantage of the Semipartial Correlation 8.3.3 Some Equivalences among Measures 8.3.4 Cohen’s f 2 8.3.5 Comparing Two Regression Coefficients in the Same Model 8.4 Dominance Analysis 8.4.1 Complete and Partial Dominance 8.4.2 Example Computations 8.4.3 Dominance Analysis Using a Regression Program 8.5 Chapter Summary 9. Multicategorical Regressors 9.1 Multicategorical Variables as Sets 9.1.1 Indicator (Dummy) Coding 9.1.2 Constructing Indicator Variables 9.1.3 The Reference Category 9.1.4 Testing the Equality of Several Means 9.1.5 Parallels with Analysis of Variance 9.1.6 Interpreting Estimated Y and the Regression Coefficients 9.2 Multicategorical Regressors as or with Covariates 9.2.1 Multicategorical Variables as Covariates 9.2.2 Comparing Groups and Statistical Control 9.2.3 Interpretation of Regression Coefficients 9.2.4 Adjusted Means 9.2.5 Parallels with ANCOVA 9.2.6 More Than One Covariate 9.3 Chapter Summary 10. More on Multicategorical Regressors 10.1 Alternative Coding Systems 10.1.1 Sequential (Adjacent or Repeated Categories) Coding 10.1.2 Helmert Coding 10.1.3 Effect Coding 10.2 Comparisons and Contrasts 10.2.1 Contrasts 10.2.2 Computing the Standard Error of a Contrast 10.2.3 Contrasts Using Statistical Software 10.2.4 Covariates and the Comparison of Adjusted Means 10.3 Weighted Group Coding and Contrasts 10.3.1 Weighted Effect Coding 10.3.2 Weighted Helmert Coding 10.3.3 Weighted Contrasts 10.3.4 Application to Adjusted Means 10.4 Chapter Summary 11. Multiple Tests 11.1 The Multiple-Test Problem 11.1.1 An Illustration through Simulation 11.1.2 The Problem Defined 11.1.3 The Role of Sample Size 11.1.4 The Generality of the Problem 11.1.5 Do Omnibus Tests Offer “Protection”? 11.1.6 Should You Be Concerned about the Multiple-Test Problem? 11.2 The Bonferroni Method 11.2.1 Independent Tests 11.2.2 The Bonferroni Method for Nonindependent Tests 11.2.3 Revisiting the Illustration 11.2.4 Bonferroni Layering 11.2.5 Finding an “Exact” p-Value 11.2.6 Nonsense Values 11.2.7 Flexibility of the Bonferroni Method 11.2.8 Power of the Bonferroni Method 11.3 Some Basic Issues Surrounding Multiple Tests 11.3.1 Why Correct for Multiple Tests at All? 11.3.2 Why Not Correct for the Whole History of Science? 11.3.3 Plausibility and Logical Independence of Hypotheses 11.3.4 Planned versus Unplanned Tests 11.4 Summary 11.5 Chapter Summary 12. Nonlinear Relationships 12.1 Linear Regression Can Model Nonlinear Relationships 12.1.1 When Must Curves Be Fitted? 12.1.2 The Graphical Display of Curvilinearity 12.2 Polynomial Regression 12.2.1 Basic Principles 12.2.2 An Example 12.2.3 The Meaning of the Regression Coefficients for Lower-Order Regressors 12.2.4 Centering Variables in Polynomial Regression 12.2.5 Finding a Parabola’s Maximum or Minimum 12.3 Spline Regression 12.3.1 Linear Spline Regression 12.3.2 Implementation in Statistical Software 12.3.3 Polynomial Spline Regression 12.3.4 Covariates, Weak Curvilinearity, and Choosing Joints 12.4 Transformations of Dependent Variables or Regressors 12.4.1 Logarithmic Transformation 12.4.2 The Box–Cox Transformation 12.5 Chapter Summary 13. Linear Interaction 13.1 Interaction Fundamentals 13.1.1 Interaction as a Difference in Slope 13.1.2 Interaction between Two Numerical Regressors 13.1.3 Interaction versus Intercorrelation 13.1.4 Simple Linear Interaction 13.1.5 Representing Simple Linear Interaction with a Cross-product 13.1.6 The Symmetry of Interaction 13.1.7 Interaction as a Warped Surface 13.1.8 Covariates in a Regression Model with an Interaction 13.1.9 The Meaning of the Regression Coefficients 13.1.10 An Example with Estimation Using Statistical Software 13.2 Interaction Involving a Categorical Regresson 13.2.1 Interaction between a Dichotomous and a Numerical Regressor 13.2.2 The Meaning of the Regression Coefficients 13.2.3 Interaction Involving a Multicategorical and a Numerical Regressor 13.2.4 Inference When Interaction Requires More Than One Regression Coefficient 13.2.5 A Substantive Example 13.2.6 Interpretation of the Regression Coefficients 13.3 Interaction between Two Categorical Regressors 13.3.1 The 2 × 2 Design 13.3.2 Interaction between a Dichotomous and a Multicategorical Regressor 13.3.3 Interaction between Two Multicategorical Regressors 13.4 Chapter Summary 14. Probing Interactions and Various Complexities 14.1 Conditional Effects as Functions 14.1.1 When the Interaction Involves Dichotomous or Numerical Variables 14.1.2 When the Interaction Involves a Multicategorical Variable 14.2 Inference about a Conditional Effect 14.2.1 When the Focal Predictor and Moderator Are Numerical or Dichotomous 14.2.2 When the Focal Predictor or Moderator Is Multicategorical 14.3 Probing an Interaction 14.3.1 Examining Conditional Effects at Various Values of the Moderator 14.3.2 The Johnson–Neyman Technique 14.3.3 Testing versus Probing an Interaction 14.3.4 Comparing Conditional Effects 14.4 Complications and Confusions in the Study of Interactions 14.4.1 The Difficulty of Detecting Interactions 14.4.2 Confusing Interaction with Curvilinearity 14.4.3 How the Scaling of Y Affects Interaction 14.4.4 The Interpretation of Lower-Order Regression Coefficients When a Cross-Product Is Present 14.4.5 Some Myths about Testing Interaction 14.4.6 Interaction and Nonsignificant Linear Terms 14.4.7 Homogeneity of Regression in ANCOVA 14.4.8 Multiple, Higher-Order, and Curvilinear Interactions 14.4.9 Artificial Categorization of Continua 14.5 Organizing Tests on Interaction 14.5.1 Three Approaches to Managing Complications 14.5.2 Broad versus Narrow Tests 14.6 Chapter Summary 15. Mediation and Path Analysis 15.1 Path Analysis and Linear Regression 15.1.1 Direct, Indirect, and Total Effects 15.1.2 The Regression Algebra of Path Analysis 15.1.3 Covariates 15.1.4 Inference about the Total and Direct Effects 15.1.5 Inference about the Indirect Effect 15.1.6 Implementation in Statistical Software 15.2 Multiple Mediator Models 15.2.1 Path Analysis for a Parallel Mediation Model 15.2.2 Path Analysis for a Serial Mediation Model 15.3 Extensions, Complications, and Miscellaneous Issues 15.3.1 Causality and Causal Order 15.3.2 The Causal Steps Approach 15.3.3 Mediation of a Nonsignificant Total Effect 15.3.4 Multicategorical Independent Variables 15.3.5 Fixing Direct Effects to Zero 15.3.6 Nonlinear Effects 15.3.7 Moderated Mediation 15.4 Chapter Summary 16. Detecting and Managing Irregularities 16.1 Regression Diagnostics 16.1.1 Shortcomings of Eyeballing the Data 16.1.2 Types of Extreme Cases 16.1.3 Quantifying Leverage, Distance, and Influence 16.1.4 Using Diagnostic Statistics 16.1.5 Generating Regression Diagnostics with Computer Software 16.2 Detecting Assumption Violations 16.2.1 Detecting Nonlinearity 16.2.2 Detecting Non-Normality 16.2.3 Detecting Heteroscedasticity 16.2.4 Testing Assumptions as a Set 16.2.5 What about Nonindependence? 16.3 Dealing with Irregularities 16.3.1 Heteroscedasticity-Consistent Standard Errors 16.3.2 The Jackknife 16.3.3 Bootstrapping 16.3.4 Permutation Tests 16.4 Inference without Random Sampling 16.5 Keeping the Diagnostic Analysis Manageable 16.6 Chapter Summary 17. Power, Measurement Error, and Various Miscellaneous Topics 17.1 Power and Precision of Estimation 17.1.1 Factors Determining Desirable Sample Size 17.1.2 Revisiting the Standard Error of a Regression Coefficient 17.1.3 On the Effect of Unnecessary Covariates 17.2 Measurement Error 17.2.1 What Is Measurement Error? 17.2.2 Measurement Error in Y 17.2.3 Measurement Error in Independent Variables 17.2.4 The Biggest Weakness of Regression: Measurement Error in Covariates 17.2.5 Summary: The Effects of Measurement Error 17.2.6 Managing Measurement Error 17.3 An Assortment of Problems 17.3.1 Violations of the Basic Assumptions 17.3.2 Collinearity 17.3.3 Singularity 17.3.4 Specification Error and Overcontrol 17.3.5 Noninterval Scaling 17.3.6 Missing Data 17.3.7 Rounding Error 17.4 Chapter Summary 18. Logistic Regression and Other Linear Models 18.1 Logistic Regression 18.1.1 Measuring a Model’s Fit to Data 18.1.2 Odds and Logits 18.1.3 The Logistic Regression Equation 18.1.4 An Example with a Single Regressor 18.1.5 Interpretation of and Inference about the Regression Coefficients 18.1.6 Multiple Logistic Regression and Implementation in Computing Software 18.1.7 Measuring and Testing the Fit of the Model 18.1.8 Further Extensions 18.1.9 Discriminant Function Analysis 18.1.10 Using OLS Regression with a Dichotomous Y 18.2 Other Linear Modeling Methods 18.2.1 Ordered Logistic and Probit Regression 18.2.2 Poisson Regression and Related Models of Count Outcomes 18.2.3 Time Series Analysis 18.2.4 Survival Analysis 18.2.5 Structural Equation Modeling 18.2.6 Multilevel Modeling 18.2.7 Other Resources 18.3 Chapter Summary Appendices A. The RLM Macro for SPSS and SAS B. Linear Regression Analysis Using R C. Statistical Tables D. The Matrix Algebra of Linear Regression Analysis Author Index Subject Index References About the Authors