To help policymakers, providers, and the public better understand the impact of innovative opioid use disorder education, prevention, treatment, or recovery services, Minnesota Management and Budget’s Results Management Team is conducting impact evaluations of a subset of services that will be offered through grants related to the state's response to the opioid epidemic.
MMB research staff will partner with grantees to conduct these evaluations at no- or low-cost to the grantees. Although programs that are Proven Effective may be awarded a grant, the impact evaluations will only focus on services that have been rated as Theory Based or Promising by Minnesota’s Results First Team (see here for an inventory of all services and their ratings). Services that are not included in the Minnesota inventory and do not yet have high-quality impact evaluations will also be considered for evaluation.
Rigorous evaluations entail considerable planning. To support evaluation, MMB research scientists compiled a brief guide on what makes a viable and valuable impact evaluation. The team will also offer free technical assistance to potential grant applicants who may be interested in being part of an evaluation. Participating in evaluation technical assistance is voluntary.
Any organization considering applying for an Opioid Epidemic Response or State Opioid Response grant to fund opioid education, prevention, treatment, or recovery services is eligible for 30-minutes of technical assistance from MMB research scientists. This assistance will help applicants plan, as possible, an impact evaluation of the program or service they intend to submit as a grant proposal.
Technical assistance will primarily occur over a video conference call. MMB staff may also field a limited number of emailed questions from organizations. To ensure fairness, all evaluation-relevant questions answered during the calls and summarized email communication will be shared on this website. Public postings will not include any identifying information about the organizations receiving technical assistance or on their proposed projects. This 1-on-1 assistance will be open before the submission deadline to all applicants. Additional technical assistance will be available after the deadline for those projects considered for an evaluation.
To sign up for this evaluation technical assistance, please email [email protected]. MMB staff will reach out to schedule a time for the technical assistance call.
Q1: What kind of data are needed?
A1: In order to do impact evaluations properly we need to have data on both the group of individuals that receive the service and another group of individuals that did not receive the service.
Q2: What do you mean by "outcome measures"?
A2: Grant applicants are asked to submit intended goals and outcome measures. Different types of outcome measures that might be collected will broadly fall into one of three categories: short-, medium-, and long-term. For example, a program might consider knowledge gained from training a short-term outcome, incidences of administered naloxone a medium-term outcome, and mortality a long-term outcome. See here for additional resources.
To support the evaluation, MMB research scientists compiled this guide on what an impact evaluation is, why they can be valuable, and what makes a viable impact evaluation. Grant applicants are encouraged to consider the information in the guide as they write their proposals and engage in evaluation technical assistance.
If an organization's application is funded and selected for an evaluation, they will work closely with MMB to do formal planning for the evaluation. This planning period will further explore the questions in the evaluation guide (link in above section). MMB research scientists will partner with the grantee and other stakeholders to complete the evaluation. Prior to the evaluation beginning, it is MMB’s policy to both pre-register analysis plans and publish all studies, regardless of the results.
The evaluations will be conducted in partnership with MMB’s Impact Evaluation Unit, a part of the Results Management Team. Since 2015, the Results Management Team has worked with agencies, counties, and providers to understand the effectiveness of state investments. Through this work, we have identified nearly 500 government funded services, programs, or practices offered across Minnesota. Of those services, however, only around half have high-quality studies that indicate they are achieving the intended impact on participant outcomes. To improve our understanding of the impact of state investments, the legislature instructed our team to conduct impact evaluations of theory-based human services programs. The team is staffed by research and data scientists with expertise in evaluation design, research operations, data analysis and visualization, and using research findings to help inform policy and decision-making.
MMB’s mission is to serve the people of Minnesota by providing the state with leadership and guidance to support efficient and effective government. The importance of these goals demands that we support continual innovation and improvement of state-funded activities, as well as those of our partners. Through research and evaluation, we can systematically learn to make our services as effective as possible. We are committed to promoting rigor, relevance, transparency, independence, and ethics in the conduct of evaluations. These principles are further detailed in our evaluation policy.
