Introduction
Child welfare system assessments follow distinct protocols, but the approach can be customizable. A scope of work can be designed to help the agency understand the causes and results that led to an assessment, along with recommendations using implementation-science principles, which can provide guidance on program improvements.
The Child Welfare Information Gateway describes the purpose of a child welfare assessment of a child welfare agency as having a way to examine how well a program or initiative is being implemented and to determine whether that program is achieving desired results. The American Evaluation Association states that this approach involves assessing the strengths and weaknesses of programs, policies, personnel, products, and organizations to improve their effectiveness. Public Knowledge® conducts assessments of child welfare systems utilizing a strengths-based and research approach. These assessments are designed to help agencies better understand their practices and capacity in protecting the health, safety, and lives of children who are under their care and/or responsibility.
Jurisdictions often seek solutions to presenting problems or have concerns resulting from other types of reviews, like the Child and Family Services Review (CFSR, a federal review of title IV-E and IV-B state plan requirements that all states must undergo periodically) and use child welfare assessments to identify the source of the problem(s). At other times, an agency might use an assessment to better understand the strengths within the system that it wishes to expand or build upon in other areas of the work. An accurate child welfare assessment of a program or agency can help leadership, policymakers, and funders determine where to allocate and use resources best, to support agency staff, and to improve services and outcomes for the families they serve.
Whatever the reasons leading to an child welfare assessment, Public Knowledge® works with agencies to design and carry out the assessment in ways that are the most likely to address identified concerns. This includes determining the scope and focus of the assessment, crafting research questions that will lead to understanding underlying causes, ensuring that the data used to inform the assessment are as current and accurate as possible, engaging stakeholders inside and outside the agency throughout the child welfare assessment process, and providing useful and informed recommendations.
Understanding the Problem and Underlying Cause(s)
Within a problem-solving framework, it is important to recognize that the presenting issues the agency wishes to address in an assessment may not reflect the underlying issues that will need to be tackled, in order to realize improvements in outcomes or indicators. Therefore, an assessment should be designed broadly enough to shed light on how children and families experience the child welfare system but focused enough to provide answers or insights to the specific issue(s) that led to the assessment request, as well as to identify areas of practice and organizational support that affect current results families are experiencing.
As an example, an agency may request an child welfare assessment because of the death or serious injury of a child known to the system, or multiple similar incidents. In such situations, public attention to the incidents may focus on wanting a quick solution or answer that can be acted upon immediately, such as perceived leadership or staffing inadequacies. While such factors may, in fact, contribute to undesirable outcomes, an assessment should be designed to identify and explore other possible contributors. This requires working with the child welfare system to ask a broader question of how children and families are protected by the child welfare system, both through agency practices and through organizational support. A well-designed assessment will examine relevant practice issues, for example, the effectiveness of safety and risk assessment procedures, and the agency’s capacity to engage in effective practices such as having adequate staff or comprehensive training for staff.
If the child welfare assessment indicates that staff have an inadequate knowledge of safety and risk assessment to properly identify and act on factors that place children in harm’s way, a ‘quick solution’ might be to strengthen training to increase staff knowledge. However, an understanding of implementation science principles, particularly regarding how adults learn and begin to use new information, might lead to recommendations that focus less on training and more on practicing new skills and receiving feedback in order to strengthen performance and capacity. NODGE® White Paper Series
In Public Knowledge® child welfare assessments, we collaborate with agencies to determine the appropriate research questions that, ideally, will identify the key factors driving the challenges or presenting issues; thus, directly addressing the areas of need. By beginning with a clear sense of what the assessment is designed to explore, and linking specific review questions, data, and other inquiries to the research questions, we increase the likelihood that we can identify and fully assess underlying factors that, if addressed through the implementation of appropriate recommendations, will lead to better outcomes for children and families.
Multiple Sources of Data/Methodology for Child Welfare Assessments
When conducting a system-wide child welfare assessment, it is important to gather information from multiple sources as opposed to relying on a single source. Similar to investigating alleged child maltreatment, a social worker would likely not interview only one person and decide but would, rather, speak to all relevant people involved to gain a broad understanding of what has likely happened. In assessments, both quantitative and qualitative data from different and complementary sources are key to understanding the complex issues that often exist within child welfare operations. Quantitative data, primarily from information systems, will provide bigger picture information on the extent of the issue being explored. Qualitative data, which can include a range of sources including case reviews and interviews, will provide context or a story behind the quantitative data, and help the agency delve deeper into understanding the root causes of the questions being asked.
There are multiple sources of information to consider, whether conducting a child welfare system-wide assessment or a more targeted inquiry into a specific issue or concern. These sources of information generally fall in four broad categories: data analysis, case review, stakeholder engagement and input, and systemic review.
Data Analysis of quantitative data generally comes from information systems within the child welfare system, including the agency’s management information system, court system, or other systems. Child welfare assessments should carefully consider, based on the intended purpose of the assessment, whether longitudinal data or point-in-time data, or some combination of the two, will best address the research questions. The data elements used in the analysis should be tied to relevant or key indicators and outcomes.
Case Review is a critical source of qualitative information that will provide an assessment with rich and tailored information to help identify underlying causes of the issue(s) being assessed. Several off-the-shelf instruments are available to provide qualitative information about key indicators and outcomes tied to child welfare best practices. However, Public Knowledge® most often develops a customized case review instrument tailored to address an agency’s concerns. In collaboration with the child welfare agency, Public Knowledge® develops and administers tools that are customized to the agency’s desired outcomes and designed to provide the level of information needed to answer research questions and solve problems.
- Conducting focus groups in which there are multiple representatives from a group with similar interests or positions within the child welfare system.
- Conducting individual interviews in which key individuals within the child welfare system provide input from their experiences or vantage point; and
- Developing and administering surveys in which broader numbers and types of representatives can be reached for input than might be possible with in-person interviews and focus groups.
Stakeholder Engagement and Input also assist in gaining an understanding of the problem or issue and the underlying causes. Without stakeholder engagement, recommendations or changes made as a result of a child welfare assessment may not have the commitment and backing needed to ensure success. Stakeholder input includes both internal representatives, such as social workers, supervisors, and administrators, as well as external representatives, such as consumers, courts, legislatures, etc. Public Knowledge®® includes stakeholder input in three primary ways:
Review of Systemic Factors permits inquiry into the capacity of the child welfare agency and/or other affiliated organizations to address the issue(s) at hand and identify their contributions to the area being assessed and possible solutions. This may include a review of policies, training/coaching resources, service array, staffing and caseloads, and other factors.
Crafting Recommendations and a Theory of Change for Child Welfare Assessment
When a child welfare assessment has defined problem areas and underlying or contributing causes, a theory of change should clearly link findings to recommendations that are expected to lead to improvements. In other words, based on the findings of the assessment, a theory of change suggests that if certain activities are undertaken by an agency, those activities can be expected to lead to desired outcomes or results. This is because recommendations should be directed toward the underlying causes of the area needing improvement.
In Public Knowledge® work, recommendations for making needed improvements are at the heart of conducting an assessment. Our approach includes making recommendations that are relevant to the underlying and presenting issues, that are able to be implemented, and that is focused on the agencies’ priorities and greatest needs. Public Knowledge® seeks to link programmatic recommendations to principles of implementation science to increase the chances that recommendations will be implemented as intended and lead to the desired outcomes. Recommendations should have the following basic characteristics:
- They should be straightforward and specific in their intent, their ability to be understood and implemented, and their link to child welfare assessment findings.
- They should be prioritized in terms of the implementation sequence and their relative contribution to the overall theory of change; and
- They should be sustainable. For example, if an agency needs to implement a practice model or fundamentally change its approach to working with children and families yet does not have the training or coaching capacity to support such a change, recommendations should be framed in ways that encourage building the foundation first.
Empowering Agencies through Engagement and Collaboration
Establishing and maintaining a strong partnership with agencies is key to the success of any assessment process. A core value in Public Knowledge®’s child welfare assessment approach lies in the belief that our work is not carried out for agencies, but with agencies, which requires engaging staff and stakeholders and working jointly to assess, design, implement and monitor solutions. Child welfare agencies have varying levels of capacity and resources that affect their ability to make improvements. When Public Knowledge® works with agencies throughout the child welfare assessment process, the agencies are better positioned to understand and commit to implementing recommendations, conducting ongoing self-evaluation, and monitoring the change process.
Agency staff and community stakeholders play a critical role in the assessment process and can contribute significantly to the credibility of child welfare assessment findings and recommendations, particularly if they are actively engaged throughout. Stakeholders, who can often contribute substantially to an assessment, include those who are:
- Responsible for carrying out day-to-day implementation of the activities that are part of the program areas being assessed; this frequently includes casework and supervisory level staff.
- In leadership or other decision-making capacities and who can effectively advocate for or authorize changes to programs and policies based on assessment results and recommendations; and
- Consumers of, or otherwise affected by, the program areas, services provided, or assessment results. For child welfare systems, this frequently includes key stakeholders such as children, families, or community stakeholders.
Specific areas where agency staff and stakeholders contribute to the assessment process include:
- Understanding and defining the problem, desired outcome, and underlying causes: Collaboration includes developing the appropriate research questions to be addressed by the assessment.
- Determining the methodology and scope of the assessment: Collaboration includes providing expertise in customizing the child welfare assessment methodology, including determining sample sizes and populations for case record or other document reviews, developing data collection/review instruments, and identifying optimal data sources.
- Participating throughout the assessment process: Agency staff and stakeholders play a vital role during the assessment process by sharing information and feedback as survey respondents or through participation in individual interviews or focus groups. Also, while preserving the independence and objectivity of an assessment, agency staff and stakeholders can often provide meaningful clarification of findings and identify needs and strategies for recommending improvements.
Together with the child welfare agency, Public Knowledge® can deliver a thoughtful, grounded assessment that will support an agency’s desire to make meaningful changes to address their concerns.
1 Assessment. Child Welfare Information Gateway. https://www.childwelfare.gov/topics/systemwide/assessment/. February 12, 2017.
2 American Evaluation Association. http://www.eval.org/p/cm/ld/fid=4. February 13, 2017
3 Child Welfare Privatization Initiatives Assessing Their Implications for the Child Welfare Field and for Federal Child Welfare Programs
Evaluating child welfare programs: a guide for program managers. Topic paper#4 (August 2008). U.S. Department of Health and Human Services
(HHS) Office of the Assistant Secretary for Planning and Evaluation (ASPE).
NOTE: Public Knowledge® was formerly the Center for the Support of Families.