EBDM Starter Kit

3b: Conducting a Policy and Practice Analysis

Navigating the Roadmap

Activity 3: Understand current practice within each agency and across the system.

Introduction

The development of a “system map” is a first step in developing a common understanding of how the criminal justice process “works” in a jurisdiction. When completed, the system map reflects the flow of cases through the justice process, key decision points, decision makers, and the volume of cases flowing through the system during a particular period of time. And while it can surface system strengths and challenges, it is the first step to—rather than a substitute for —a more in-depth examination of the policies and practices that underlie the workings of the system.

Following the completion of the system map, a process to “dig deeper” into these decision points should be undertaken. This more in-depth analysis includes an examination of the following as they relate to each decision point:

  • written policies;
  • the application of those policies to practice, as well as other operational practices that are not formally articulated in policy;
  • the various types of data and information collected at each decision point;
  • the ways in which this data and information inform decisions;
  • the ways in which information is stored and shared;
  • the extent to which evidence-based information and research is available and used to make decisions;
  • gaps and barriers that impede the use of evidence-based knowledge to inform these decisions; and
  • other factors related to using data, information, and evidence in the most efficacious ways.

Purpose

Most matters of public policy are guided in part by an array of formal policies contained in federal or state law, county code, case law, individual agency policy, or memoranda of understanding among multiple organizations. An assessment of current policy begins with the identification of these operating guidelines. Formal policy only begins to define the manner in which decisions are made and processes are carried out; in most communities, written policy guides only a small portion of activity. When formal policy leaves activities undefined, informal practices emerge to fill the gaps. Informal practices are sometimes developed with clear purpose and great care; in other instances, they simply evolve over time. Developing an understanding of informal practice is as critical as understanding current policy.

A policy and practice analysis serves to objectively assess the extent to which

  • clear, written policies dictate key decisions and practices;
  • policies are consistent with evidence-based knowledge;
  • policies are consistently carried out, consistent with one another, and support their stated purpose(s), agency missions, and jurisdiction vision;
  • gaps in policy result in discretionary practices, and the degree to which that discretion results in practice that is evidence-based, consistent, and in support of vision and mission; and
  • the collection, storage, sharing, and application of data and information support effective decisions.

Participants

A policy and practice analysis can be conducted by staff internal to the agency(ies) under examination, by staff from colleague agencies within the jurisdiction, and/or by external assessors. The latter approach offers the opportunity to bring to the jurisdiction the perspective of objective subject matter experts. While this might be the ideal approach, it may not always be possible.1

Establishing a “Data Committee”

Many EBDM policy teams elect to form a “data committee” in addition to work groups tasked with examining decision points. These committees provide information and data support to the work groups throughout the mapping and policy and practice analysis processes.

Although in some instances existing data systems may be limited in the amount and type of data that can be collected, the data committees seek to collect, at least, the following baseline information:

  • number of cases following each decision path on the system map;
  • average case processing time between steps;
  • average length of time/stay in pretrial and post-conviction disposition options;
  • characteristics/profiles of cases in disposition options (offense and offender characteristics, including available information on risk and needs);
  • success/completion rates for disposition options; and
  • failures to appear (FTAs), new arrests, and recidivism outcomes by disposition option.

For more information, see 3d: Gathering Baseline Data.

Instructions

Using Outside Experts to Conduct an Analysis

  1. Discuss the purposes of the policy and practice analysis. Make sure the team understands that the analysis will be conducted by individuals with expertise in specific areas who will assess the extent to which current policies and practices regarding key decision points are supported by research. Against that backdrop, engage the team in a discussion about their priorities for the analysis (i.e., What would you like to understand about these decision points? What would you like to have more information about? What (if any) change targets can you envision regarding the decision points? What information can be derived from the analysis to better inform the team about those potential targets?).
  2. Discuss with the policy team the process and timing of the analysis: who will conduct it, the expertise of the individuals involved, the anticipated duration, schedule of interviews and observations, and the method of recording and reporting the findings.
  3. Agree on an approach to informing affected agency staff of the purposes and timing of the assessment.
  4. Ask the outside experts to provide a report following the assessment that outlines key findings and specific recommendations for changes that the site can make to further align itself with the most current research. Ideally, a debrief session with the policy team will be conducted to enable the team to ask questions, deliberate over findings with the aide of subject matter experts, and develop strategies for action.

Process for Conducting a Policy and Practice Analysis through Self-Assessment

  1. Form subcommittees (or work groups) to conduct assessments of discreet areas (e.g., agencies, decision points, a combination of these).
  2. Across all subcommittees, develop a shared sense of purpose for this process—what information is to be collected and why, an agreed-upon set of “ground rules” for carrying out the process (e.g., ethical and professional behavior during and following the analytic process, confidentiality limits and expectations, etc.), the timeline, the final product subcommittees are to produce, and how and with whom these final products will be shared.
  3. Identify a “chair” for each subcommittee. This person should be responsible for calling together their subcommittee, facilitating their meetings, keeping the subcommittee on task, and sharing information with the policy team to ensure good communication, coordination, and information flow.
  4. Subcommittees should become familiar with the relevant research in their area(s) as well as emerging practices that have yet to be demonstrated empirically (if they do not already have familiarity in these areas). This knowledge will be the “backdrop” against which the analysis will be conducted.
  5. Be clear with subcommittees that part of their role is to educate the policy team members on their topic. The goal, in the end, is to have a common, core base of knowledge across all policy team members.
  6. Create mixed-discipline subcommittees. For example, do not assign only those team members who are involved in law enforcement to the cite/detain decision point. Doing otherwise will provide a unique opportunity for cross-disciplinary learning and for an “outsider” to ask questions that those entrenched in current processes may not consider.
  7. Early subcommittee meetings should be focused on developing a detailed work plan and timeline for conducting the analysis.
  8. Subcommittees should continue to meet through the information collection and analysis process to ensure that the work is being carried out as planned, data is recorded in a useful manner, and difficulties with the information collection and analysis process are addressed early.
  9. At the conclusion of the subcommittee’s work, it should be prepared to present its findings and recommendations to the entire team.2

Tips

Consider the following tips for collecting information effectively:

  • The quality of the information collected through this process will have tremendous bearing on the action planning decisions of the team. Seek quality information over quantity if time constraints are a factor. To that end, if it is not possible to thoroughly assess all decision points, agree with the policy team on the initial priorities for assessment.
  • The information gathering process should be used as an opportunity to educate those outside of the formal team about the vision, mission, and goals of the policy team, and to learn their perspectives on the strengths and gaps in the jurisdiction relative to evidence-based decision making. Doing so is likely to both elicit cooperation and engender long-term support for the team’s work.
  • If a team chooses to work together on all of the component sections rather than working in subcommittees, a work plan and timeline for each section should be developed. Dividing the work among individual group members and encouraging them to answer the questions independently—without at least processing them with the team—is not recommended. Doing so will provide only one person’s perspective, and is unlikely to give an accurate or comprehensive picture of policy and practice across the jurisdiction.
  • Do not rely solely on the expertise of policy team members and/or subcommittee members to conduct the analysis. Instead, collect multiple data sources to ascertain current policies and practices. Ideally, interview decision makers regarding current policies and practices; review documents (i.e., written policies, procedure manuals, programmatic materials, forms, and data screens); conduct individual interviews or focus groups with supervisors and staff; and observe the activities being assessed. This approach will provide the broadest view and most reliable information. (For instance, in assessing how pre-trial release recommendations are made, review written statutes, bail schedules, and agency policies; interview the pre-trial administrator, chief prosecutor, and bail commissioner; interview pre-trial release officers; and observe pre-trial interviews, verifications, the development of reports, and bail review hearings).
  • Expect variability in practice. It is not unusual to find that there is a lack of uniformity within a jurisdiction, with some practitioners or agencies doing things one way and others doing it completely differently.
  • In those cases where numerous individuals are involved in carrying out a particular function, committees might consider convening a focus group to understand the variety of approaches in use. For example, it might be helpful to identify those prosecutors handling violations of parole/probation cases and to interview them together in a focus group format to understand their management of these cases.

1 Jurisdictions might explore whether technical assistance is available for this purpose.

2 See 3e: Prioritizing Your Team’s Targets for Change.


Example

EBDM Work Group Charter for Assessing Evidence-Based Policy, Practice, and Decision Making

PDF/Printer Friendly Version of Section