Which test / procedures? How do we decide?

Which stat testWith so many statistical procedures available, how do we decide which tests are best to address our research objectives? (several posts deal with this topic).
First and foremost, the decision as to which statistical procedures to apply to the data should be made BEFORE the design of the data collection instrument (e.g. the questionnaire), and not AFTER data has been collected. Plan ahead so that your analysis are entirely focused on addressing your research objectives and NOT to address your data. 

Too many researchers remain guilty of waiting to see the data so they can decide what to do with it, and then realise what they can’t do. While your selection of statistical procedures should be reviewed once you have seen the data, you should not wait for the data and then stare at it with the question: “now how shall I analyse it?”
Deciding on what analysis will be required and which statistical procedures to apply will have a profound impact on how we design the instrument to collect the data. These decisions hinge around three key issues: 1) our research objectives, 2) the data characteristics, and 3) the analytical process.
The only reason why we collect data is to address our research objectives / questions (likely aimed to solve our client’s marketing challenges, or for whatever reason the research was commissioned). 
Statistical procedures are grouped into seven categories based on what needs to be achieved:
  1. Describe a phenomena with descriptive statisticsExamples: “measures of central tendency” (e.g. mean scores), and “measures of dispersion and variability” (e.g. standard deviation, variance, coefficient of variation).
  2. Investigate the magnitude of differences that exists between certain response levels or respondent groups.
    Examples: t-test, ANOVA, MANOVA, etc.
  3. Investigate the extent of association, dependence, or relationship between variables
    Examples: Pearson’s product moment (r) correlation, chi-square, canonical correlation.
  4. Prediction of outcome values by predictor variable(s)
    Examples: Different types of regression analysis
  5. Prediction of group membership by predictor variable(s)
    Examples: discriminant analysis, logistic regression
  6. Analyse the existence of structure 
    1. Structure among variables (EFA, PCA, CFA, SEM)
    2. Structure among cases or respondents (cluster analysis, CHAID, latent class analysis)
    3. Structure among objects e.g. brands (perceptual mapping: MDS, correspondence analysis)
  7. A focus on time course of events.
    Examples: Survival / Failure analysis, Time series analysis
Once we have decided which statistical procedures best serve our research objectives, we can determine what the data collection instrument should look like as not all tests have the same data requirements. Alternatively, if we neglected our duty to decide on the specific statistical procedures before collecting data, then we need to review our data and decide which procedures are fit for the data we have. Either way, we need to consider each of the following:
  1. Scale of measurement: Mainly whether the data is categorical (dichotomous, nominal, or ordinal), or continuous (interval or ratio)
  2. Parametric or non-parametric data (e.g. parametric does not violate the normal distribution assumption, typically interval or ratio scales, etc)
  3. Number of dependent and independent variables (generally we differentiate between 1, or more than 1 on each side of the equation)
  4. Number of levels, groups, factors in either the categorical dependent and independent variables (generally we differentiate between 2, or more than 2)
  5. Research design of respondent groups: “between-subjects” (compare independent groups) OR “within-subjects” (compare dependent / paired groups / repeated measure e.g. before and after tests)
  6. Are some variables conceptualised as covariates (continuous scale) or “blocking factors” (categorical scale) so that the effects of some IV’s are assessed after the effects of other extraneous IV’s are statistically removed.
  7. Specific assumptions of each statistical procedure.
Carefully review each of the above and match that with the requirement of the available statistical procedures. 
Reminder: Avoid discretising continuous data just to fit the data assumption of your chosen statistical procedure, such as breaking up a 10-point scale into two dichotomous groups or three ordered groups as you lose statistical power when moving from a continuous variable to a categorical variable. 
Decisions on the priority of introducing variables into the analysis, such as which predictor in multiple regression go in first, and last (e.g. stepwise regression) or other analytical processes such as first conducting cluster analysis followed by a discriminant analysis, will affect our decision of which statistical procedures to select.