EBDM Starter Kit
4c: Becoming a Better Consumer of Research
Navigating the Roadmap
Activity 4: Understand and have the capacity to implement evidence-based practices.
The EBDM Initiative seeks to help local policy teams find and understand evidence-based knowledge about effective justice practices and to design more effective responses to defendants and offenders.1 Many stakeholders already know how to find and use research; others will appreciate these tips regarding how to quickly access reliable research and how to review and understand the findings and their applications. The evidence or empirical studies will be drawn from many fields: evidence-based practices in criminal justice, behavioral health interventions, organizational development, leadership and management, effective collaboration processes, and cost–benefit analyses.
Broadly speaking, the goal of this document is to increase policy officials’ and practitioners’ skills in finding the research that matters and in understanding and translating empirical findings for their use in improving policy and practice. Specifically, this document offers
“Evidence-based practice is the use of direct, current empirical evidence to guide and inform effective and efficient decision making and supervision practices.”
- tips for finding research relevant to critical questions about evidence-based practice;
- a list of searchable databases on criminal justice topics; and
- advice on how to review and assess the quality of the findings in academic articles and the research literature.
This document was developed for EBDM policy teams, their work groups, and agency practitioners to enhance their ability to find and understand the best available research that may be applied to criminal justice problems and proposed solutions.
Step 1: Look in the Right Places to Find the Evidence that Matters
Where should the discerning consumer begin the search for evidence-based policies and programs and answers to specific research questions? The answer is three-fold: the Web, written literature, and experienced colleagues from your local and state criminal justice systems and from national networks of professionals.
Websites that Filter the Information for You: Evidence-Based Program Databases
Websites designed specifically to summarize research in one or more criminal justice practice areas are an excellent place to begin the search for information on effective programs and policies. A growing number of government agencies, academic institutions, and professional groups maintain these databases as a service to criminal justice professionals and the public. These organizations
- formulate evaluation criteria for assessing the strength of research findings;
- employ experts to review multiple studies of research on programs in a single area; and
- indicate which programs are shown to be effective (and at what level of rigor or confidence).
Some of these websites specialize in “systematic reviews” (also called meta-analytic reviews) of the literature regarding specific research questions and program areas. As the Center for Evidence-Based Crime Policy at George Mason University explains, systematic reviews “summarize the best available evidence on a specific topic using transparent, comprehensive search strategies to find a broad range of published and unpublished research, explicit criteria for including comparable studies, systematic coding and analysis, and often quantitative methods for producing an overall indicator of effectiveness.”2
A partial list of evidence-based program databases in criminal justice follows:1
A Website Caution
The consumer of website research summaries should be careful to not take the information at face value. Definitions as to what constitutes evidence, methodological soundness, and robust findings can vary significantly.
Furthermore, researchers do not always agree on what can be concluded from a research study. While some website authors make transparent attempts to give the user an accurate description of research findings, it is up to the user to exercise judgment. It is recommended that the user seek corroborating information to increase confidence in the relative strength of the research and its implications.
- The Campbell Collaboration, The Crime and Justice Coordinating Group (CCJG) is an international network of researchers that prepares and disseminates systematic reviews of high-quality research on methods to reduce crime and delinquency and to improve the quality of justice. www.campbellcollaboration.org/crime_and_justice/
- The Center for the Study of the Prevention of Violence, University of Colorado, maintains a website, Blueprints for Violence Prevention, on evaluated programs to prevent adolescent violence, aggression, and delinquency. www.colorado.edu/cspv/blueprints/
- George Mason University’s Center for Evidence-based Crime Policy offers a number of services, including systematic reviews, research on crime and place, and a summary (matrix) of evidence-based policing practices. http://gunston.gmu.edu/cebcp/
- Substance Abuse and Metal Health Services Administration’s (SAMSHA) National Registry of Evidence-based Programs and Practices (NREPP) provides a database of more than 190 interventions supporting mental health promotion, substance abuse prevention, and mental health and substance abuse treatment. www.nrepp.samhsa.gov
- U.S. Department of Justice, Office of Justice Programs’ Crime Solutions’ website provides research on program effectiveness; easily understandable ratings (effective, promising, and no effects) that indicate whether a program achieves its goal; and key program information and research findings. www.crimesolutions.gov
Websites that Provide Bibliographic Databases
These websites, which provide a listing of hundreds of studies, are often maintained by government agencies and universities. Prominent among these in the criminal justice field are the following:
- The National Criminal Justice Reference Service (NCJRS), supported by the U.S. Department of Justice, Office of Justice Programs. https://www.ncjrs.gov/
- The National Institute of Corrections Information Center. www.nicic.org
- Correctional Services of Canada. http://www.csc-scc.gc.ca/text/rsrch-eng.shtml
Websites that Provide Summaries of Research and Practical Guidance
Some universities, state criminal justice agencies, and professional organizations also run websites that summarize the research on effective criminal justice practice and/or provide guidance to users. While not as extensive as bibliographic databases, these websites focus their publications on the critical issues of most concern to policymakers and practitioners. A partial list follows:
- Center for Evidence-Based Crime Policy. http://gemini.gmu.edu/cebcp
- Correctional Treatment Evaluations, Texas Christian University, Institute for Behavioral Research. This national research center for addiction treatment studies in community and correctional settings provides access to over 700 resources on its website. http://www.ibr.tcu.edu/
- National Implementation Research Network. This website contains research on the successful implementation of new processes within organizations and systems. http://nirn.fpg.unc.edu/
- Stanford University, Evidence-Based Management. This website specializes in evidence directly related to the management of agencies. http://www.evidence-basedmanagement.com/
- University of Cincinnati School of Criminal Justice. This university-based site contains a number of research studies regarding the use of evidence in correctional interventions. http://www.uc.edu/ccjr/reports.html
- Washington State Institute for Public Policy. This website contains a number of helpful studies on what is or is not an effective intervention for reducing recidivism and costs. It is perhaps best known for its cost–benefit studies. http://www.wsipp.wa.gov/
Often an efficient way to check out the results of web-based and library searches is to ask experienced colleagues in your state and local jurisdiction and in national networks for recommendations regarding the latest and most reliable research. This strategy helps triangulate or hone in on the best studies.
Further, when identifying a journal article that appears useful but for which a subscription is required, contact colleagues at nearby colleges and universities and inquire about their ability to access the article from their library and provide a single copy for your review. (Be careful to not copy, distribute, or otherwise violate copyright laws.)
An increasing number of states support websites that summarize evidence-based research and practical guidance that is directly relevant to their criminal justice constituents and agencies. The websites may be hosted by a state criminal justice agency or university. Your colleagues will know how to access these sites.
Step 2: Evaluate Research Quality
What criteria should be used to decide if program evidence has been collected and analyzed according to high quality research standards? As Hess and Savadsky (2009) emphasize in their article “Evaluation Research for Policy Development,” all evidence is not created equally. Familiarity with a few key concepts can help policymakers wade through the growing body of information and make better-informed decisions about what is reliable. Following are a few tips about how to read the research literature and evaluate its quality:4
The Implementation Challenge
Replication of a well-researched, effective intervention in your jurisdiction depends on prescriptions being followed precisely and implemented correctly when repeated at scale. This is not the time to add your own “unique stamp” to the approach. If you do, it should be evaluated to determine if the change improved results.
- Understand the target population of the study and consider its relationship to the target population under consideration in your jurisdiction. Pay attention to sample size and sample selection. In general, larger samples provide more reliable data; however, there is no one hard and fast rule about sample size. The sample size may vary according to the purpose of the study, overall population, sampling error, and so forth.
- Consider the context. What works in one place or for one population may not work for another (e.g., a study completed in a small, rural state with unique characteristics may not be applicable to a large, densely populated state with a different offender profile and justice system challenges). In addition, the context of one study cannot necessarily be transferred to other settings. An often-quoted study examined successful program results and found that 15% of the outcome was derived from the intervention itself (e.g., cognitive program, didactic intervention, or therapeutic community) and 30% from the working alliance with the individual providing the service.5 However, the study was not carried out with correctional clients. The results could be valid across populations but until that hypothesis is tested, caution must be exercised about its applicability to the correctional population.
- Be cautious about assertions of causality. Correlation does not mean causation; an intervention may be related to a certain outcome but may not be responsible for that outcome. For example, a significant portion of many communities’ offender population includes individuals with mental illness. A common assumption is that mental health treatment will reduce the likelihood of reoffense among this population. However, while a mental health condition should be treated, studies have shown that mental health treatment alone is unlikely to reduce recidivism.
- Recognize that changes in implementation can change the outcomes of an intervention. For instance, an effective probation intervention that relies on officers proficient in motivational interviewing, case planning, and problem solving with clients may not work as well if delivered by staff who do not possess these skills.
- Be sure the conclusions follow logically from the reported findings. The summaries or conclusions of some studies can be deceptive or take license in explaining the implications of findings. Consumers should look for research that “measures the impact of particular interventions on identifiable populations under controlled circumstances.”6 These studies offer prescriptive guidance about actions that can be consistently replicated elsewhere.
All Research is Not Created Equal
“The golden rule here is to recognize that everything promoted as ‘research’ is not equally reliable or useful.”
– Hess & Savadsky, 2009
- The issue of confidence in results is important. The research consumer needs to know if the results of the intervention are “statistically significant.” This refers to the likelihood that a result is caused by something other than mere chance. In general, a 5% or power p-value is considered statistically significant. While policymakers may not want to dig through the statistical results’ section in great detail, it is useful to check whether the article mentions that the findings are statistically significant. Other issues such as whether the person(s) conducting the research study has a vested interest in the outcome of the study and whether the study was replicated elsewhere should also be considered.7
1 In Appendix 3 of the Framework for Evidence-Based Decision Making in Local Criminal Justice Systems, the Initiative provides a matrix of research findings on reducing pretrial misbehavior and offender recidivism. EBDM policy teams are encouraged to review this resource; however, the EBDM Research Matrix can only provide a snapshot of the research at one point in time, as new research is continually conducted. Therefore, this Starter Kit document is intended to provide EBDM policy teams with additional guidance on how to keep current with the research on EBDM.
3 Adapted from Fink, 2008.
4 Adapted from Hess & Zavadsky, 2009.
5 Wampold, 2001.
6Hess & Zavadsky, 2009.
7 See Hess & Zavadsky (2009) for more information on how to be a good consumer of research.
Additional Resources and Readings
Hess, F. M. & Savadsky, H. (2009). Evaluating research for policy development. Institute for Public School Initiatives. The University of Texas at Austin.
Fink, A. (2008). The research consumer as detective: Investigating program and bibliographic databases. Practicing Research: Discovering Evidence that Matters (pp. 33–64). Retrieved from http://www.sagepub.com/upm-data/19270_Chapter_2.pdf
Wampold, B. E. (2001). The great psychotherapy debate: Models, methods, and findings. Mahwah, NJ: Lawrence Erlbaum Associates.