Fidelity Monitoring

Fidelity is the extent to which the delivery of an intervention is consistent with the original design and intent of service delivery to achieve outcomes for the population the intervention was designed to serve. The following section specifies how the OhioKAN program model conceptualizes fidelity to the context, content, and competence needed to produce positive outcomes for kinship and adoptive caregivers and the children in their care. Attention to fidelity is also critical to promote consistent implementation of the OhioKAN model across a state or jurisdiction. The OhioKAN CQI approach includes intentional use of data to ensure fidelity to the model at all levels of the intervention.

The CQI teams in OhioKAN leverage data to measure the progress towards fidelity, identify areas for improvement, and leverage improvement cycles to enhance consistent service delivery of the program model with a focus on quality of practice. Such information structures feedback for staff on what areas of practice they excel in and where improvement is needed. For instance, in OhioKAN, Coaches base their feedback and coaching sessions on information learned through exploration of the CQI dashboard and through observations of practice. Other opportunities for navigators to reflect on and enhance practice include trainings and learning collaboratives.

The OhioKAN program model conceptualizes fidelity in three domains:

• Context: the structure, framework and environment in which service delivery occurs. This includes characteristics of the physical environment, organizational and administrative requirements. An example of context fidelity in OhioKAN are the Regional Advisory Councils. The RACs track service gaps and barriers in each region to identify what community resources are needed and how to fill those needs, for the purpose of achieving the intended outcomes outlined in the community path of the theory of change.
• Content: the adherence to defined activities or practice behaviors and the dose of intervention such as the frequency and duration of service. An example of content fidelity in OhioKAN is sending referrals within two business days of completing the BASICS with a caregiver.
• Competence: the skills of practitioners and the quality of service delivery encompass the competence fidelity domain. Within OhioKAN, competence fidelity is represented by elements such as the use of the core practice skills defined in the Navigator Practice Profile, which are observed and recorded by Coaches in the Observation and Debrief Checklist, the observational fidelity monitoring tool.

OhioKAN Program Fidelity and Quality Indicators
The OhioKAN core components are the backbone for operationalizing fidelity and identifying indicators to measure fidelity to the program model. The OhioKAN program, in coordination with implementation and evaluation experts, identified a set of indicators to measure the adherence to fidelity and quality. These indicators are outlined below.

• % observations where navigator uses OhioKAN core practice skills, by practice skill
• % families with BASICS completed
• % families sent referral(s) to resources to meet their needs, by BASICS need domain
• % families that receive follow-up after referral
• % BASICS assessments completed within 2 days of initial intake
• % referral packets sent within 2 days of BASICS completion
• % families that receive follow-up within 30 business days of referral
• Average number of contacts in Collaborate and Connect levels of service

Fidelity Monitoring through Observations of Practice
Coaches monitor and observe practice of OhioKAN program procedures and confidentiality standards using indirect and direct practice observation methods. Coaches observe their navigators’ core practice skills at four points of observation each month:

• Point 1: Navigator and family engagement (direct observation)
• Point 2: Debrief of observed navigator and family engagement with the navigator during coaching
• Point 3: Document review (indirect observation)
• Point 4: Debrief of reviewed document with the navigator during coaching

These four observation points allow coaches the opportunity to observe the following navigator core practice skills (further specified in the Navigator Practice Profile): Engagement, Inclusion, Assessment, Intervention, Active Learner, and Reflection. Coaches use the fidelity monitoring tool, the Observation and Debrief Checklist (see Appendix), to document their observations during each of the four observation points. Completed checklists help inform practice areas to explore during regular coaching sessions. In addition, aggregated data from completed checklists are used to observe larger trends for the program’s fidelity and quality.

Direct observation of family engagement
Coaches observe OhioKAN’s staff direct practice through monthly observation of their engagement with OhioKAN families. These observations can occur during screening and intake, BASICS completion, follow-up, and support plan development. Coaches use the Observation and Debrief Checklist, included in the Appendix, to observe the degree to which OhioKAN staff demonstrate core practice skills and deliver services in alignment with the model’s program and confidentiality procedures. In the Observation and Debrief Checklist practice skills are observed during direct observation points to determine learning opportunities to further unpack, debrief, and explore with navigators during coaching. These observations are then debriefed with navigators during coaching, where the coach may also observe the core practice skills of Reflection and Active learner. Direct observation of practice allows coaches to ensure fidelity to the model and monitor for implicit bias to ensure that all families receive high-quality services and navigators demonstrate consistent practice.

Indirect observation through service episode document and data review
Coaches indirectly observe OhioKAN staff core practice skills by reviewing service episode documentation (e.g., BASICS with referral binder or BASICS and referral binder with support plan) each month. This indirect observation method helps coaches understand the degree to which OhioKAN staff demonstrate core practice skills in their service episode documentation. Coaches use the Observation and Debrief Checklist to note which core practice skills they’ve observed in reviewed documentation and whether there are learning opportunities to further unpack, debrief, and explore with navigators during coaching. These observations are then debriefed with navigators during coaching, where the coach may also observe navigator core practice skills of Reflection and Active learner. The review of service episode documentation allows the coach to fully understand practice skill level, biases, and quality of navigation services to identify strengths and opportunities for improvement. In addition, coaches complete a timely review of data quality to observe staff data entry practices. Coaches follow up with navigators as needed on data entry improvement needs and guidance.