EBDM Starter Kit

4a: Understanding Your Agency: Conducting an EBP Knowledge Survey

Navigating the Roadmap

Activity 4: Understand and have the capacity to implement evidence-based practices.

Introduction

For a jurisdiction to be truly evidence-based, the professionals managing its justice system agencies1—and those working directly with the defendants and offenders in that system—should all share a common base of knowledge about key findings in the correctional literature. A knowledge survey can serve as a diagnostic tool to determine the level of current knowledge among these parties and potentially identify needs for knowledge enhancement. If it is determined that further knowledge development is needed, steps should be included in the jurisdiction’s strategic plan to enhance knowledge among key stakeholders and agency staff. This document provides a process for administering a knowledge survey and considering the implications of its results for the team’s future work.

Purpose

  • To assess the level of evidence-based knowledge among policy team members and of select groups of staff within their agencies.
  • To identify action steps to address gaps in knowledge as identified through the survey.

Participants

While the policy team will ultimately decide who will take the survey, it is recommended that knowledge survey participants include a sample of staff from all agencies represented on the team.

Instructions

  1. During a policy team meeting, discuss the reason(s) why a knowledge survey is important to the team’s work and how it will aid in the development of the team’s strategic plan.
    1. Determine what kind of knowledge (i.e., information that has been demonstrated to rise to the level of “evidence” and therefore be used to inform decision making) the team wants its members to know. From this feedback, either develop a knowledge survey or use the one in Appendix 1.
    2. If the policy team decides to develop its own knowledge survey, it is recommended that the survey be reviewed by a respected researcher and pilot tested with practitioners who are knowledgeable about the research before it is administered.
    3. Distribute the knowledge survey to policy team members.
    4. Survey respondents should not be coached in advance as to the specifics of the survey or possible answers. Results will only reflect accurate indicators of existing agency knowledge if the survey is administered without foreknowledge of content.
    5. Explain that not all agency staff will be surveyed. Instead, a selected group of policy and line level staff will be anonymously surveyed.
Measuring the Impact of an Event

In Yamhill County, Oregon, the policy team used the EBP knowledge survey to measure changes in knowledge regarding EBP after a day-long awareness-building event for system stakeholders.

A greater percentage of respondents answered 17 out of 18 questions correctly following the event than prior to the event. These findings suggest that participation in this event had a positive impact on participants’ level of knowledge about various evidence-based practices.

  1. Secure a commitment from each policy team member to fully participate in the survey process.
  1. Following the team meeting, discuss with each agency’s lead policymaker2 which of their staff members should participate in the survey. It is recommended that only those who are in charge of policy decisions (i.e., directors/ managers/ supervisors) and those with direct contact with defendants, offenders, and victims be included in the survey.
  1. Calculate the number of staff persons who will participate in the survey, based on the criteria agreed to in #3. This will be the total survey “pool” for each agency. If possible, the survey should be sent to all individuals in this survey pool.
  1. The survey may be distributed in hard copy or through an automated online survey mechanism such as Survey Monkey.
  1. Communicate with agency staff in advance about the purpose of the survey. Some recommended email scripts are provided in Appendix 2.
  1. Once the survey results are compiled, review them during a policy team meeting.
    1. Determine how confident you are that the survey results reflect the knowledge of the pool of staff in each agency. Keep this in mind as you discuss the results. (For instructions on how to do this, see Appendix 3.
    2. Determine as a team the threshold for a satisfactory and unsatisfactory knowledge level.
    3. Determine whether the survey results were satisfactory (as determined by the team). If not, for which agencies were the results unsatisfactory?
    4. For those agencies with unsatisfactory knowledge survey scores, determine what action steps the team wants to take.
    5. If one or more action items are identified to enhance knowledge, consider, as part of your action plan, re-administering the knowledge survey after these efforts are completed to determine whether the knowledge gaps have narrowed. In this case, use the same process to determine the survey sample for re-administration as you used for the initial survey. The same individuals do not need to be surveyed; however, the same protocol for identifying the survey group must be followed.
Using an EBP Survey with Agency Leaders

In Charlottesville-Albemarle County, Virginia, the director of the local community corrections agency used the knowledge survey with her statewide agency heads to test their knowledge of EBP principles and practices, and gave out prizes for the highest scores.

Tips

  • Make sure that survey participants understand the purpose of the survey. (See the email template below.)
  • Allow, at a minimum, two weeks (10 business days) for participants to complete the survey.
  • Send an email to the survey participants at least two days before the beginning of the two-week survey period to let them know that they are being asked to participate in the survey. (Make sure the survey is up and running early, in case participants want to fill out the survey immediately!) It is preferable that this email be sent by the agency policymaker or by another person in a leadership position. See Appendix 2 for ideas about what this email should contain.
  • Send out one reminder during the two-week survey period.
  • If, after ten business days, all of the surveys are not returned, send out another reminder indicating that, although the deadline has passed, it is important that each person participate and, therefore, the deadline has been extended for three more business days.

1 Throughout this document, the term “agency” is used to represent independent entities such as a police department, court, organization of public defenders, community corrections agency, etc.

2 For the purposes of this document, the “lead policymaker” is the person on the policy team representing his/her agency (e.g., chief of police, chief public defender, chief judge).


Additional Resources and Readings

The knowledge survey contains information collected from numerous research studies. See the NIC EBDM Framework for research studies that include some of the key findings.

Appendix 1

EBP Survey with Answer Key

Appendix 2

Email Templates

Appendix 3

Determining the Minimum Number of Responses Desired for Accurate Inferences

PDF/Printer Friendly Version of Section