1、Practical Meta-Analysis - D. B. Wilson,1,Practical Meta-Analysis,David B. Wilson Evaluators Institute July 16-17, 2010,Practical Meta-Analysis - D. B. Wilson,2,Overview of the Workshop,Topics covered will include Review of the basic methods Problem definition Document Retrieval Coding Effect sizes a
2、nd computation Analysis of effect sizes Publication Bias Cutting edge issues Interpretation of results Evaluating the quality of a meta-analysis,Practical Meta-Analysis - D. B. Wilson,3,Forest Plot from a Meta-Analysis of Correctional Boot-Camps,Practical Meta-Analysis - D. B. Wilson,4,The Great Deb
3、ate,1952: Hans J. Eysenck concluded that there were no favorable effects of psychotherapy, starting a raging debate 20 years of evaluation research and hundreds of studies failed to resolve the debate 1978: To proved Eysenck wrong, Gene V. Glass statistically aggregate the findings of 375 psychother
4、apy outcome studies Glass (and colleague Smith) concluded that psychotherapy did indeed work Glass called his method “meta-analysis”,Practical Meta-Analysis - D. B. Wilson,5,The Emergence of Meta-analysis,Ideas behind meta-analysis predate Glass work by several decades Karl Pearson (1904) averaged c
5、orrelations for studies of the effectiveness of inoculation for typhoid fever R. A. Fisher (1944) “When a number of quite independent tests of significance have been made, it sometimes happens that although few or none can be claimed individually as significant, yet the aggregate gives an impression
6、 that the probabilities are on the whole lower than would often have been obtained by chance” (p. 99). Source of the idea of cumulating probability values,Practical Meta-Analysis - D. B. Wilson,6,The Emergence of Meta-analysis,Ideas behind meta-analysis predate Glass work by several decades W. G. Co
7、chran (1953) Discusses a method of averaging means across independent studies Laid-out much of the statistical foundation that modern meta-analysis is built upon (e.g., Inverse variance weighting and homogeneity testing),Practical Meta-Analysis - D. B. Wilson,7,The Logic of Meta-analysis,Traditional
8、 methods of review focus on statistical significance testing Significance testing is not well suited to this task Highly dependent on sample size Null finding does not carry the same “weight” as a significant finding significant effect is a strong conclusion nonsignificant effect is a weak conclusio
9、n Meta-analysis focuses on the direction and magnitude of the effects across studies, not statistical significance Isnt this what we are interested in anyway? Direction and magnitude are represented by the effect size,Practical Meta-Analysis - D. B. Wilson,8,Illustration,Simulated data from 21 valid
10、ity studies. Taken from: Schimdt, F. L. (1996). Statistical significance testing and cumulative knowledge in psychology: implications for training of researchers. Psychological Methods, 1, 115-129.,Practical Meta-Analysis - D. B. Wilson,9,Illustration (Continued),Practical Meta-Analysis - D. B. Wils
11、on,10,When Can You Do Meta-analysis?,Meta-analysis is applicable to collections of research that Are empirical, rather than theoretical Produce quantitative results, rather than qualitative findings Examine the same constructs and relationships Have findings that can be configured in a comparable st
12、atistical form (e.g., as effect sizes, correlation coefficients, odds-ratios, proportions) Are “comparable” given the question at hand,Practical Meta-Analysis - D. B. Wilson,11,Forms of Research Findings Suitable to Meta-analysis,Central tendency research Prevalence rates Pre-post contrasts Growth r
13、ates Group contrasts Experimentally created groups Comparison of outcomes between treatment and comparison groups Naturally occurring groups Comparison of spatial abilities between boys and girls Rates of morbidity among high and low risk groups,Practical Meta-Analysis - D. B. Wilson,12,Forms of Res
14、earch Findings Suitable to Meta-analysis,Association between variables Measurement research Validity generalization Individual differences research Correlation between personality constructs,Practical Meta-Analysis - D. B. Wilson,13,Effect Size: The Key to Meta-analysis,The effect size makes meta-an
15、alysis possible It is the “dependent variable” It standardizes findings across studies such that they can be directly compared,Practical Meta-Analysis - D. B. Wilson,14,Effect Size: The Key to Meta-analysis,Any standardized index can be an “effect size” (e.g., standardized mean difference, correlati
16、on coefficient, odds-ratio) as long as it meets the following Is comparable across studies (generally requires standardization) Represents the magnitude and direction of the relationship of interest Is independent of sample size Different meta-analyses may use different effect size indices,Practical
17、 Meta-Analysis - D. B. Wilson,15,The Replication Continuum,Pure Replications,Conceptual Replications,You must be able to argue that the collection of studies you are meta-analyzing examine the same relationship. This may be at a broad level of abstraction, such as the relationship between criminal j
18、ustice interventions and recidivism or between school-based prevention programs and problem behavior. Alternatively it may be at a narrow level of abstraction and represent pure replications.The closer to pure replications your collection of studies, the easier it is to argue comparability.,Practica
19、l Meta-Analysis - D. B. Wilson,16,Which Studies to Include?,It is critical to have an explicit inclusion and exclusion criteria (see pages 20-21) The broader the research domain, the more detailed they tend to become Refine criteria as you interact with the literature Components of a detailed criter
20、ia distinguishing features research respondents key variables research methods cultural and linguistic range time frame publication types,Practical Meta-Analysis - D. B. Wilson,17,Methodological Quality Dilemma,Include or exclude low quality studies? The findings of all studies are potentially in er
21、ror (methodological quality is a continuum, not a dichotomy) Being too restrictive may restrict ability to generalize Being too inclusive may weaken the confidence that can be placed in the findings Methodological quality is often in the “eye-of-the-beholder” You must strike a balance that is approp
22、riate to your research question,Practical Meta-Analysis - D. B. Wilson,18,Searching Far and Wide,The “we only included published studies because they have been peer-reviewed” argument Significant findings are more likely to be published than nonsignificant findings Critical to try to identify and re
23、trieve all studies that meet your eligibility criteria,Practical Meta-Analysis - D. B. Wilson,19,Searching Far and Wide (continued),Potential sources for identification of documents Computerized bibliographic databases “Google” internet search engine Authors working in the research domain (email a r
24、elevant Listserv?) Conference programs Dissertations Review articles Hand searching relevant journal Government reports, bibliographies, clearinghouses,Practical Meta-Analysis - D. B. Wilson,20,A Note About Computerized Bibliographies,Rapidly changing area Get to know your local librarian! Searching
25、 one or two databases is generally inadequate Use “wild cards” (e.g., random? will find random, randomization, and randomize) Throw a wide net; filter down with a manual reading of the abstracts,Practical Meta-Analysis - D. B. Wilson,21,Strengths of Meta-analysis,Imposes a discipline on the process
26、of summing up research findings Represents findings in a more differentiated and sophisticated manner than conventional reviews Capable of finding relationships across studies that are obscured in other approaches Protects against over-interpreting differences across studies Can handle a large numbe
27、rs of studies (this would overwhelm traditional approaches to review),Practical Meta-Analysis - D. B. Wilson,22,Weaknesses of Meta-analysis,Requires a good deal of effort Mechanical aspects dont lend themselves to capturing more qualitative distinctions between studies “Apples and oranges” criticism
28、 Most meta-analyses include “blemished” studies to one degree or another (e.g., a randomized design with attrition) Selection bias posses a continual threat Negative and null finding studies that you were unable to find Outcomes for which there were negative or null findings that were not reported Analysis of between study differences is fundamentally correlational,
