Introduction
Child welfare systems face an array of challenges in meeting the needs of children and families. Effective use of quality data provides agencies with an optimized Continuous Quality Improvement structure where root causes can be identified and strategies implemented and adjusted over time to effect sustainable change.
Child welfare systems, regardless of size, are first and foremost accountable to the children, families, and communities they serve. Child welfare systems, however, are also increasingly being required to respond to changing federal oversight, state, and local initiatives, and monitoring related to class action litigation.1 As such, these systems must be dynamic and capable of adapting to significant and sometimes competing demands in an often-fluid environment where quality data can help provide meaningful insight, as well as guide practice and performance improvements.
The Critical Role of Using Quality Data to Improve Outcomes
The Children’s Bureau of the Administration for Children and Families (ACF) issued an Information Memorandum in 2012 that emphasized the importance for state child welfare agencies to have strong CQI systems in place in order to achieve and to maintain positive outcomes for children and families. The use of quality data to identify areas of strengths and concerns, establish targeted strategies for improvement, and track progress toward desired outcomes was highlighted as the foundation of a high functioning CQI system.
The Center for States, as part of the Child Welfare Capacity Building Collaborative funded by the Children’s Bureau, further highlights the essential role that data plays across several key areas in child welfare, including quality data collection, infrastructure, extraction, analysis, and dissemination.
To be successful, child welfare systems need to be more than merely data-oriented – they need to be data-resolute in their ongoing commitment to gathering and producing only quality data, continually sharing, and promoting the effective use of quality data, and ensuring that every step in the CQI problem-solving process is informed by data. It is only when data can be readily accessed, placed into context, shared, understood, and used throughout a child welfare system that it can be considered valuable and, therefore, have a genuine impact on decision-making and outcomes.
Public Knowledge® believes that to ensure the continuous quality improvement of a child welfare system, whether at a broad system level or as part of a more targeted initiative, it is critical to have ongoing access to accurate information in the form of quality data about how that system is operating. The type of information needed is driven by the system’s goals, objectives, and desired outcomes, with the overarching questions being: 1) is the system achieving its goals, objectives, and desired outcomes; and 2) if not, what must be done to correct the system so that it does achieve its goals, objectives, and desired outcomes?
This document highlights some of the key challenges to producing and using quality data that child welfare systems may encounter and offers possible solutions to consider as a means to effect real and sustainable performance improvements.
Solution-Focused Use of Quality Data
Public Knowledge® All Rights Reserved
Defining Data Elements and Outcomes
A common challenge that child welfare systems face is being able to define accurately those data elements and outcomes that are tied to their primary organizational goals; Public Knowledge® has found that data used for performance indicators and outcomes often strays from the practice or activity they are designed to measure. This is particularly true of data from management information systems, where the information maintained and generated is often voluminous and complex. Likewise, data obtained through qualitative methods, including case review and survey instruments, while providing useful information, may not be informing the key practices they are intended to measure without ensuring proper definitions exist. While jurisdictions have lots of reporting requirements, including federal, legislature, or even settlement agreements or consent decrees (which have their own data definitions), it is also imperative that quality data be defined to be used to help improve practice. This is an area where Public Knowledge® can provide aid. Public Knowledge® helps jurisdictions define data elements in three categories: fidelity, key performance indicators, and child and family outcomes.
Defining the quality data to be used as fidelity measures, performance indicators, or outcomes requires determining that the data collected correlate to the identified population of interest and in the appropriate timeframe, capture the precise activities and events of interest, and are selected from the appropriate fields in the appropriate information systems. The same concerns apply to defining the content of qualitative data elements, such as indicators from case review processes. Wording questions and observations for instruments requires careful definition of the behaviors, activities, and events to be captured. It also requires providing clear instructions on gathering and coding the information and establishing parameters for when certain data elements are applicable to a situation. It is critical that measures reflecting desired practices are developed in collaboration with agency staff as part of an implementation strategy that is continuously monitored.
A key objective in Public Knowledge®’s solution-focused approach to using quality data is to work collaboratively with child welfare systems to determine the primary information that is needed by asking the right questions to be answered, the most appropriate sources of that information, and the corresponding performance indicators and outcomes that need to be captured to inform agency practices and improve outcomes. Public Knowledge® works closely with the agency’s technical and programmatic staff to ensure that the collection and use of data are properly aligned with the agency’s intended interventions or mandates.
Producing and Reporting Quality Data
Child welfare systems often face challenges in their ability to produce and report quality data. Agencies may lack the financial resources or necessary technical capacity to produce and/or report quantitative data effectively from their management information systems. Agencies often face similar challenges in their ability to produce and/or report qualitative data, which often stems from not having sufficient quality assurance mechanisms in place that connect to an overarching CQI process.
Quantitative Data
Because changing management information systems to be able to meet data reporting needs often requires that agencies make long-term and potentially financially prohibitive investments, an alternative approach may provide a solution. Using data extracts from existing information systems, whereby data are extracted to a more malleable database from which data reports can be produced using various statistical programs, can offer that alternative.
Regardless of where quantitative data is sourced, Public Knowledge® works closely with agency staff to develop and promote capacity to produce quality data, including developing appropriate business requirements, specifications, and data definitions on their own that will lead to successful production of useful quantitative data reports.
Qualitative Data
A benefit to using qualitative data, in addition to providing context and a deeper understanding of trends identified in aggregate quantitative data, is the ability to understand issues more clearly from the perspective of workers, families, and other sources of the information. For the production and reporting of qualitative data, linking information obtained via methodologies such as case record reviews and surveys to an agency’s desired key performance measurements is vital to understanding what is behind an agency’s quantitative data. For instance, the connections among an agency’s casework practices, the quality of services being provided, and the outcomes an agency wants to achieve can often be understood better with the use of qualitative data.
Public Knowledge® works collaboratively with agencies to design and implement qualitative data collection processes, whether as part of a one-time targeted assessment, or as part of strengthening an agency’s ongoing CQI process that would include, for instance:
- Developing customized case review instruments,
- Conducting focus groups,
- Conducting stakeholder interviews, and
- Developing and administering surveys.
The Role of Data Validation in Ensuring Data Quality
For child welfare systems to be solution-focused on their use of data, it is important to have systems and methodologies in place that ensure that data collected, extracted, and reported by the agency are reliable and valid. The Children’s Bureau highlights the importance of investing in data quality improvement activities that are collaborative and include the active involvement of a broad range of staff and also seek to bridge gaps that may exist between information technology staff and program staff when it comes to underlying assumptions regarding who is responsible for entering data, ensuring data accuracy, and correcting data errors.4
Part of ensuring the quality of data used by child welfare agencies includes being clear on report structure and specifications, specific data elements used to compile reports, and the ongoing review of the reports’ accuracy in providing the intended information. For quantitative data, when the end-users are clear on how the data, they may be entering into the information system are captured and used in reports, it will help to avoid incorrect assumptions and misinterpretations about the information source or calculations and will assist in efforts to improve data quality.
In addition, achieving full integration of quantitative data with child welfare practice requires that data reports clearly reflect the practices they are intended to track. One approach to improving data quality is for an agency to undergo an analysis of its existing data report structure to determine whether the information produced is, in fact, an accurate reflection of desired indicators or outcomes and, if gaps do exist, to determine the root causes of the gaps. Using this problem-solving approach, an agency will be able to ascertain if the issue lies, for instance, in the inaccurate mapping of data elements in the reports because of duplication in the system, errors in the report code, or inaccurate or untimely data entry.
Regarding the quality of qualitative data, having controls in place such as inter-rater reliability checks in case reviews will help to avoid inconsistencies in the data and reduce subjectivity. Developing and using standard and uniform protocols is critical to ensuring consistency of data collected. This includes relying on training protocols, interview guides, and quality assurance reviews of completed case reviews.
Whatever the data quality challenge, validation efforts should be designed to identify the sources of errors when reports are developed, and these should be revisited on an ongoing basis. Public Knowledge® uses these types of methodologies to help child welfare agencies develop data quality approaches that will work best for their unique organizational needs.
Analyzing and Interpreting Quality Data
Having the capacity to analyze quality data fully over time is what allows a child welfare agency to transform its data successfully into knowledge and to guide the agency toward achievement of its goals. Data in its many forms can feel overwhelming and confusing to agencies and it is in the process of analyzing data that missteps can sometimes occur. Misunderstandings or incorrect translations of available data outputs can unwittingly send child welfare systems down the wrong path and inhibit their ability to make effective, timely changes that positively influence outcomes for children and families.
Understanding the strengths and abilities of staff at all levels, including agency leadership, to use data effectively is critical to ensuring the accurate translation of data into information. A common challenge facing child welfare agencies lies in not having the internal analytic capacity to support its CQI needs fully. At times, agencies may resolve this capacity concern by establishing relationships with qualified external entities such as universities or consultants to provide the necessary support.
In other situations, developing the internal agency capacity to analyze and interpret data is also a way to create an organizational environment where everyone can connect data and outcomes to case practices and service provision. Analysis of the data can be as simple as counting frequencies, sorting data by location, or a more complex task, such as identifying associations among variables to help determine root causes or underlying conditions to a problem.
Having a Data Plan
Particularly when large-scale improvement efforts are underway within an agency, Public Knowledge® encourages and assists agencies to develop a comprehensive data plan, a key component of its approach to CQI. Public Knowledge® sees data plans as critical for an agency to have ongoing accurate information about how its system is operating, to understand what data are available and accessible for the success of the improvement efforts, and to understand the gaps and development needs for additional data. Such a plan ideally includes formulating data questions around identified areas of agency inquiry; determining which data will be collected and analyzed, as well as potential data resources; preparing reports that provide analysis and insight into whether the agency is achieving its goals or desired outcomes; and ongoing monitoring and testing of program improvement efforts. Having a data plan in advance also guides collection of data by establishing early on the specific questions the data will address and ensuring we collect the correct data to answer these questions.
Beyond analyzing the data, accurately interpreting the data allows an agency to move from answering questions that are mostly concerned with “what” or “how many,” to answering questions that are concerned with “why” or “how well.” Child welfare agencies also need to understand the barriers to achieving desired outcomes, and interpreting the data is one way of helping with these tasks. Analyzed, quality data is only valuable if it is understood and used by the child welfare agency and its stakeholders.
Accurate interpretation of data can be promoted through deliberate communication with key collaborators who can shed light on what data may or may not mean, as well as by using multiple sources of data, e.g., qualitative and quantitative that may point agencies in the right direction for understanding the data. Public Knowledge® helps child welfare systems interpret the data by ensuring that the data are carefully integrated into intervention/practice activities and that data are routinely examined to monitor and adjust improvement planning activities if needed.
Using Quality Data to Inform, Educate and Create Dialog
A primary consideration for child welfare agencies in making the commitment to produce and report quality data lies in their capacity to use it in a meaningful way; specifically, as part of a structured and continual problem-solving approach that leads to improved service delivery and outcomes. Quality data can only facilitate change if it is routinely shared and used by agency staff and stakeholders who are provided with the necessary supports to interpret what the data mean regarding the practices and services they provide to the children and families they serve.
Some common barriers to the effective production and use of quality data within child welfare agencies include lack of routine access to data and reports by staff, lack of staff understanding of their role in achieving data quality, lack of support in analyzing and interpreting data, and failure to understand how to use the information and knowledge obtained from the data to create needed changes.
The production and use of quality data is critical to creating an environment and agency culture in which staff at all levels, as well as internal and external stakeholders, value and rely on data and evidence to understand and improve their performance and outcomes. In other words, it is a beginning step to ensuring that agencies routinely examine their practice and outcomes, understand why they are one way or another, and rely on data and evidence to make informed decisions that affect the lives of children and families.