Social Work Research Center

(1)

Social Work Research Center | Colorado State University 1 SafeCare Colorado

Colorado Department of Human Services – Office of Early Childhood

Program Evaluation Report 2014-2017

(2)

SafeCare Colorado Program Evaluation Report 2014-2017

Prepared by:

Kristy Beachy-Quick Chris Lee Luke McConnell

Rebecca Orsi Zachary Timpe Marc Winokur

Social Work Research Center School of Social Work

Prepared for:

Office of Early Childhood

Colorado Department of Human Services

(3)

Social Work Research Center | Colorado State University

SafeCare Colorado Program Evaluation Report 2014-2017

1. Introduction

In Colorado, SafeCare® began in 2009 with a federal grant awarded to the Colorado Judicial Department and the Kempe Center for the Prevention and Treatment of Child Abuse and Neglect (Kempe). In 2013, SafeCare® was selected as part of a group of cornerstone prevention programs formed or expanded under Governor Hickenlooper’s master child welfare plan,

“Keeping Kids Safe and Families Healthy 2.0”. Kempe was selected by the Office of Children Youth and Families at the Colorado Department of Human Services (CDHS) to support the implementation of SafeCare Colorado (SCC). In May 2015, SCC state funding and statewide program oversight moved to the Division of Community and Family Support in the Office of Early Childhood (OEC) at CDHS.

The Social Work Research Center (SWRC) in the School of Social Work at Colorado State University (CSU) serves as the independent evaluator of the SCC program for CDHS. The evaluation is designed to measure the implementation process, program outcomes, and costs.

As such, this evaluation report is organized to align with the methods and results from the process, outcome, and cost evaluations.

1.1. Description of SafeCare^®

SafeCare® is an evidence-based in-home parent education program designed for at-risk families with children ages 0-5. It relies on behavior reinforcement, modeling, and skill practice, through staff observations and parent training developed from social learning theory. In Colorado, SafeCare is being implemented as a home-based, voluntary, and preventive program. The program is structured into three distinct topic areas described below:

1. Home Safety – Safety goals are to identify and reduce the number of hazards in the home environment that are accessible to children. Safety hazard categories include poisons, choking, firearms, and sharp objects. Safety latches are provided to families and a validated and reliable assessment checklist helps parent support providers identify hazards and provide training to parents.

(6)

Social Work Research Center | Colorado State University 2 2. Child Health – Health goals are to teach parents and/or caregivers to recognize and

assess when children are sick or injured and when to provide or seek appropriate treatment. Parents and/or caregivers are provided with a health manual that includes a guide to symptoms and instructions on when to call a nurse or doctor. They are also provided with recording charts and basic health supplies such as a thermometer.

3. Parent-Infant Interaction (PII) or Parent-Child Interaction (PCI) – PII goals (for newborns to about one year) are to promote positive interactions; promote age appropriate and stimulating activities; and promote bonding and attachment. Bonding skills training include: looking, talking, touching, smiling, holding, imitating, and rocking.

PCI goals (for toddlers and preschoolers) are to increase positive interactions; engage children; and prevent challenging child behavior. Positive behaviors are reinforced and problem behaviors are addressed during the sessions.

Each of the SafeCare® topics is conducted over six 1-1.5 hour sessions that typically occur weekly. All topics use a similar teaching model (an assessment session, followed by four sessions of training, and a final re-assessment session). The program is delivered by parent support providers who receive intensive coaching using a coaching model created by the SafeCare program developer and overseen by the National SafeCare® Training and Research Center (NSTRC), established in 2007 at the Georgia State University School of Public Health.

1.2. Overview of the SafeCare Colorado Implementation

The implementation of the SCC program was designed as a gradual rollout, with sites phasing in each year. This report presents data from all sites. Thirteen diverse organizations have been selected to implement SafeCare Colorado, serving 41 counties and two tribal nations across Colorado. Selected through a competitive process, these organizations were required to demonstrate the need for the program in their communities as well as their ability to implement effectively. Table 1 displays the year SCC began, region served, agency type, and primary services provided by the different implementing agencies.

(7)

Social Work Research Center | Colorado State University 3 Table 1: Implementing Agency Characteristics

Site Year Region Served Agency Type Primary Services

Baby Bear Hugs 2014 Northeast CO Community Based Organization

Prevention services, parenting classes, and parent support

Savio House 2014 Central CO Community Based Organization

Intensive services (child placement, residential, family counseling, adolescent services)

for families in child welfare Mesa County Health

Department 2014 Western CO Public Health Public Health (WIC, family planning, immunization clinic) Montezuma County

Public Health 2014 Southwestern CO Public Health Public Health (health inspections, immunization clinic, WIC, dental) Family Tree 2015 Central CO Community Based

Organization

Prevention services and family support, domestic violence,

homelessness Arapahoe County

Early Childhood Council

2015 Central CO Early Childhood Council

Early Childhood Council (services for child care providers, parenting support, advocacy) Catholic Charities

Diocese of Pueblo 2015 Southern CO Community/Faith Based Organization

Housing assistance, immigration family law, parent support High Plains

Community Health Center

2015 Southeastern CO

Federally Qualified Community Health

Center

Primary medicine

Lutheran Family Services Rocky Mountain

2016 Central CO Community/Faith Based Organization

Refugee and asylee programs, pregnancy counseling and adoption services, disaster response, foster care, older adult

and caregiver services, prevention services North Range

Behavioral Health 2016 Northern CO Behavioral Health Center

Crisis support, therapy, addiction treatment, integrated services,

medical services La Llave Family

Resource Center 2016 Southern CO Community Based Organization

Prevention services and family support, parenting, resource and

referral Northwest Colorado

Health Department 2017 Northwest CO Public Health

Public Health (WIC, family planning, immunization clinic,

mental/behavioral health services)

Ute Mountain Ute

Tribe 2017 Southwest CO Tribal Government Employment: Farm and ranch, travel center, casino, pottery

(8)

Social Work Research Center | Colorado State University 4 Figure 1 shows a map of the counties in which SafeCare Colorado currently operates and where the sites are located.

Figure 1: Map of SafeCare Colorado Sites Fiscal Year (FY) 2016

2. Process Evaluation

The process evaluation section features analyses of referrals to SCC, characteristics of program participants, participation outcomes and perspectives from program site staff.

2.1. SafeCare Colorado Referrals

Families are eligible to participate in SCC if they have at least one child under the age of six and if they: 1) had past child welfare involvement; or 2) met a minimum of three risk factors, such as “young caregiver under 20” or “single parent”, as outlined on the referral form developed by program administrators. If a referring party did not know this information, the SCC site would contact the family for a phone screening or to set a visit to further determine eligibility.

Additional eligibility criteria for child welfare involved families was added in 2014. Initially, a

(9)

Social Work Research Center | Colorado State University 5 family had to be screened out, have a closed case, or a closed assessment. Expansion criteria included open family assessment response (FAR) assessments (only available in select counties), non-court involved open cases, and open assessments.

2.1.1. Referral Pathways

As displayed in Table 2, 13,048 referrals were made to SCC sites from January 2014 through March 2017.¹ The majority of these referrals came from child welfare departments at 67 percent. The next most frequent referral categories were self-referrals at 11 percent and medical providers at six percent. All other referral source types combined comprised approximately 15 percent of total referrals.

Table 2: Frequency of Referrals by Referral Source (N = 13,048)

Referral Source Frequency Percent

Department of Human Services (DHS) - Child Welfare 8,798 67.4

Self-Referral 1,456 11.2

Medical Provider 797 6.1

Other Home Visiting Programs 640 4.9

Family Resource Center 270 2.1

DHS - Other 279 2.1

Early Childhood Education/Child Care 258 2.0

Public Health - Women Infants Children (WIC) 212 1.6

Early Intervention 119 0.9

Other 120 0.9

Mental Health 71 0.5

Community Centers 24 0.2

Substance Treatment 4 0.0

On the following page, Table 3 shows that the majority of child welfare referralswere closed high risk assessments (HRA) at 46 percent, followed by screened out referrals at 28 percent, open or closed FAR assessments at 17 percent, open assessments at three percent, and closed cases and non-court involved open cases each at two percent.

1 Referrals with a referral date recorded between January 2014 and March 2017 were included.

(10)

Social Work Research Center | Colorado State University 6 Table 3: Child Welfare Referrals by Referral Type (N = 8,798)

Child Welfare Referral Type Frequency Percent

Closed Assessment (HRA) 3,999 45.5

Screen Out 2,458 27.9

FAR (open or closed) 1,491 16.9

DHS – Pathway Missing 275 3.1

Open Assessment (HRA) 264 3.0

Closed Case 161 1.8

Non-Court Involved Open Case 150 1.7

Table 4 shows the percentage of referrals that resulted in an intake by referral source.² Of the 13,048 unique referrals made to SCC, 2,567 resulted in an intake, which is an acceptance rate of 20 percent. There were 11,855 unique families who were referred to SafeCare and were

recorded as eligible for services. The referral source with the highest percentage of accepted referrals was community centers at 75 percent. Although child welfare referrals from DHS represent the highest proportion of overall referrals, they had the lowest rate of referrals leading to an intake at 10 percent across all sites.

Table 4: Percent of Referrals that Resulted in an Intake by Referral Source (N = 13,048)

Referral Source Number

of Referrals

Number of Intakes

Percent of Accepted Referrals Department of Human Services (DHS) - Child Welfare 8,798 849 9.7

Self-Referral 1,456 628 43.1

Medical Provider 797 242 30.4

Other Home Visiting Programs 640 283 44.2

DHS - Other 279 103 36.9

Family Resource Center 270 112 41.5

Early Childhood Education/Child Care 258 93 36.1

Public Health - Women Infants Children (WIC) 212 92 43.4

Other 120 62 51.7

Early Intervention 119 55 46.2

Mental Health 71 29 40.9

Community Centers 24 18 75.0

Substance Treatment 4 1 25.0

Total 13,048 2,567 19.7

2 The 33 intakes missing referral source data are not included in the table.

(11)

Social Work Research Center | Colorado State University 7 As displayed in Table 5, the most fequent referral source for SCC enrolled families was DHS – Child Welfare at 33 percent, followed by self-referral at 24 percent, other home visiting

programs at 11 percent, and medical provider at nine percent. For the 849 referrals from DHS – Child Welfare, 40 percent were from closed high risk assessments, 22 percent were from screen-outs, 15 percent were open or closed FAR assessments, eight percent were

uncategorized, six percent each were open high risk assessments and non-court involved open cases, and four percent were closed cases.

Table 5: Referral Source and Child Welfare Referral Pathways for Enrolled Families (N = 2,600)

Referral Source Frequency Percent

Department of Human Services (DHS) - Child Welfare* 849 32.7

Self-Referral 628 24.2

Other Home Visiting Programs 283 10.9

Medical Provider 242 9.3

Family Resource Center 112 4.3

DHS - Other 103 4.0

Early Childhood Education/Child Care 93 3.6

Public Health - Women Infants Children (WIC) 92 3.5

Other 62 2.4

Early Intervention 55 2.1

Unknown 33 1.3

Mental Health 29 1.1

Community Centers 18 0.7

Substance Treatment 1 0.0

*Child Welfare Referral Pathway (N = 849)

Closed Assessment (HRA) 338 39.8

Screen Out 189 22.3

FAR (open or closed) 127 15.0

DHS – Pathway Missing 68 8.0

Open Assessment (HRA) 48 5.7

Non-Court Involved Open Case 48 5.7

Closed Case 31 3.7

2.2. SafeCare Colorado Participation

A total of 2,600 unique families were served (i.e., had an intake or a SafeCare® session

recorded) by the SafeCare Colorado program from January 2014 through March 31, 2017. This section presents descriptive results for case characteristics, number of topic sessions, topic

(12)

Social Work Research Center | Colorado State University 8 completion, program engagement, and program completion for parents and/or caregivers from all cohorts. Although households may have several parents and/or caregivers and multiple children in the home, they are asked to select one primary parent and/or caregiver to work with the parent support provider each week (SCC parent and/or caregiver) as well as one primary child under age six (SCC child).

2.2.1. Case Characteristics

As displayed in Table 6, there were 1,070 parent and/or caregivers in Cohort one, 1,183 in Cohort two, 310 in Cohort three, and 11 in Cohort four. The Kempe Center also served 26 families during the trainer/coach certification process. The sites with the most families served were Family Tree at 18 percent, Savio House at 16 percent, Mesa County Health Dept. at 12 percent, and Arapahoe County ECC at 12 percent. It should be noted that the capacity to serve families varied by site based on the number of parent support providers hired and the number of months the programs were operating.

Table 6: Cohort and Site Frequencies

Characteristic Frequency Percent

Cohort (N = 2,600)*

Cohort 1 1,070 42.2

Cohort 2 1,183 45.5

Cohort 3 310 11.9

Cohort 4 11 0.4

Kempe 26 1.0

SafeCare Colorado Sites (N = 2,600)

Family Tree 468 18.0

Savio House 415 16.0

Mesa County Health Dept. 316 12.2

Arapahoe County ECC 304 11.7

Catholic Charities Diocese of Pueblo 211 8.1

High Plains Community Health Center 200 7.7

Baby Bear Hugs 187 7.2

Montezuma Public Health 152 5.8

Lutheran Family Services 117 4.5

North Range Behavioral Health Center 109 4.2

La Llave 84 3.2

Kempe 26 1.0

Northwest Colorado Health 11 0.4

*Note: those served by Kempe were not part of a cohort.

(13)

Social Work Research Center | Colorado State University 9 2.2.2. Family Engagement

As displayed in Table 7, 71 percent of parents and/or caregivers were engaged in the SafeCare Colorado program, which means that they participated in at least three SCC sessions.

Table 7: Family Engagement Outcome for Closed and Open Cases (N = 2,600)

Outcome Frequency Percent

Family Engaged

Yes 1,833 71.0

No 748 29.0

Missing* 19 -

*Note: 19 cases were missing the family engagement outcome.

2.2.3. Topic Sessions

Parents and/or caregivers are given the option of which topic to begin with. When they do not choose a topic, the PSP chooses, based on the families' needs. As displayed in Table 8, of the 1,077 closed cases with health topic sessions, the average number of sessions completed was five. Of the 1,178 closed cases with safety topic sessions, the average number of sessions completed was five. Of the 1,332 closed cases with interaction topic sessions, the average number of sessions completed was six. For the 1,821 closed cases with recorded session data, the average number of total sessions completed across the three topics was ten.

Table 8: Number of Sessions by Topic for Closed Cases

Topic Min Max Mean

Health (N = 1,077) 1 14 5.1

Safety (N = 1,178) 1 16 4.7

Interaction (N = 1,332) 1 22 5.7

Total sessions (N = 1,821) 1 44 10.3

2.2.4. Topic Completion

As displayed in Table 9 on the following page, 81 percent of parents and/or caregivers with a closed case completed the health topic, 78 percent with a closed case completed the safety topic, and 67 percent with a closed case completed the interaction topic.

(14)

Social Work Research Center | Colorado State University 10 Table 9: Topic Completion Outcomes for Closed Cases*

Topic Frequency Percent

Health (N = 1,077)

Did not complete 210 19.5

Completed 867 80.5

Safety (N = 1,178)

Completed 914 77.6

Interaction (N = 1,332)

Completed 894 67.1

*This outcome is based on families that participated in at least one session of the respective topics.

2.2.5. Program Completion

As displayed in Table 10, 41 percent of parents and/or caregivers did not complete any topics, while 20 percent completed one, 12 percent completed two and 27 percent completed all three. Thus 27 percent of parents and/or caregivers with a closed case completed the SafeCare® program, which means they successfully completed all three topics. Interestingly, this is the exact percent completion published in an evaluation of a SafeCare® program implemented in the UK in 2015.³

Table 10: Program Outcomes for Closed Cases (N = 2,107)

Outcome Frequency Percent

Topics completed

None 860 40.8

One 429 20.4

Two 255 12.1

All three (program completed) 563 26.7

Program completed

Yes 563 26.7

No 1,544 73.3

As noted, 27 percent of program participants choose to complete all three SafeCare® topics.

3 https://www.nspcc.org.uk/globalassets/documents/research-reports/safecare-evidence-evaluation-report.pdf

(15)

Social Work Research Center | Colorado State University 11 As shown in Figure 2, 59 percent of parents/ caregivers who initially enroll in SafeCare®

complete one topic; 66 percent of parents/caregivers who complete their first topic go on to complete a second topic; and 69 percent of those who complete two topics go on to finish the

entire program. It should be noted that attrition is a well acknowledged challenge for

prevention programs. A better understanding of the timing of attrition can lead to strategies to increase engagement across topics, resulting in better overall outcomes and cost effectiveness.

Figure 2: Family Engagement across SafeCare Topics (Closed Cases N = 2,107)

2.2.6. Client Support

In addition to the SafeCare®curriculum, parents and/or caregivers may also receive

“client support” visits, in which parent support providers provide resources and referrals to other organizations or offer problem solving support outside of the SafeCare® topic areas. Client support may occur in a stand-alone visit or during a SCCsession visit. Of all families served who also had visit data, 55 percent (N = 1,227) had at least one visit that included client support. Of these, there was a maximum of 31 visits including client support and an average of 4.0 client support visits per family.

0 500 1,000 1,500 2,000

Served (1 topic) Served (2 topics) Completed (3 topics)

Target Engagement Actual Engagement

66% still engaged

59% still engaged from intake

69% still engaged

(16)

Social Work Research Center | Colorado State University 12 As displayed in Table 11 on the following page, the most commonly selected service

provided during a client support visit was “other client services” at 40 percent. The write-in text for this category included such things as: custody issues, school related issues, providing diapers and other concrete supports, referrals to other parenting classes, support groups, car seat safety, auto repairs, financial management, and help for the holidays.

Table 11: Resource and Referrals Provided during Client Support Visits

Client support Frequency Percent

Other client services (N = 926) 2,845 40.4

Other referrals (N = 406) 1,116 15.8

Mental health (N = 359) 589 8.4

Housing (N = 319) 537 7.6

Benefits (N = 309) 454 6.4

Food (N = 266) 347 4.9

Daycare (N = 226) 318 4.5

Extended phone session (N = 179) 290 4.1

Medical (N = 191) 257 3.6

Domestic violence (N = 144) 193 2.7

Substance abuse (N = 89) 101 1.4

2.3. Parent and/or Caregiver, Children, and Household Characteristics This section provides demographics and characteristics of parents and/or caregivers and children served by the SCC program. Parents and/or caregivers were asked to complete

demographic forms during the intake session which were then entered into an Access database maintained by Kempe. The total N varies in each table based on the amount of complete data for each self-reported category.

As displayed in Table 12 on the following page, 92 percent of SCC parents and/or caregivers are female and 47 percent are Hispanic. The most common race is Hispanic at 47 percent, followed by Caucasian at 42 percent, and African American at six percent. For marital status of the SCC parents and/or caregivers, 34 percent are single, 19 percent are living together, 34 percent are married, seven percent are separated, six percent are divorced, and one percent are widowed.

For age group of the SCC parents and/or caregivers, seven percent are 14-19 years old, 29 percent are 20-25 years old, 21 percent are 26-29 years old, 33 percent are 30-39 years old, and 10 percent are 40 years or older.

(17)

Social Work Research Center | Colorado State University 13 Table 12: SCC Parent and/or Caregiver Demographics

Caregiver gender (N = 2,128)

Female 1,958 92.0

Male 170 8.0

Caregiver Hispanic (N = 2,123)

Yes 991 46.7

No 1,132 53.3

Caregiver race (N = 2,104)

Hispanic 991 47.1

Caucasian 888 42.2

African American 125 5.9

Bi-racial⁴ 51 2.4

Native American 33 1.6

Asian 12 0.6

Hawaiian 4 0.2

Caregiver marital status (N = 1,933)

Single 650 33.6

Living together 369 19.1

Married 657 34.0

Separated 128 6.6

Divorced 117 6.1

Widowed 12 0.6

Caregiver age (N = 1,519)

14-19 years old 99 6.5

20-25 years old 436 28.7

26-29 years old 324 21.3

30-39 years old 507 33.4

40+ years old 153 10.1

As displayed in Table 13 on the following page, 36 percent of SCC parents and/or caregivers reported that a language other than English was spoken in their household. For educational level of the SCC parents and/or caregivers, 25 percent have less than a high school diploma, 10 percent have a GED, 26 percent have a high school diploma, 22 percent have one to two years

4 Note that the bi-racial category includes the selection of two or more non-Hispanic race categories.

(18)

Social Work Research Center | Colorado State University 14 of college (no degree), seven percent have an associate's degree, four percent have three to four years of college (no degree), and five percent have a bachelor's degree. For employment status of the SCC parents and/or caregivers, 31 percent are homemakers, 19 percent are full- time, 14 percent are part-time, 13 percent are out of work/looking for work, seven percent are unable to work, six percent are out of work/not currently looking for work, six percent are students, three percent are self-employed, and one percent are retired. For housing situation, 64 percent of SCC parents and/or caregivers rent, 17 percent own, 14 percent share with relatives/friends, four percent are in temporary housing, and one percent are homeless.

Table 13: SCC Parent and/or Caregiver Characteristics

Language other than English (N = 2,009)

Yes 720 35.8

No 1,289 64.2

Education (N = 2,072)

Less than high school 524 25.3

GED 202 9.7

High school diploma 529 25.5

1-2 years of college (no degree) 456 22.0

Associate's degree 140 6.8

3-4 years of college (no degree) 76 3.7

Bachelor's degree 107 5.2

Graduate study (no degree) 8 0.4

Master's degree 23 1.1

Doctoral or professional degree 7 0.3

Employment status (N = 2,085)

Homemaker 642 30.8

Full-time 401 19.2

Part-time 300 14.4

Out of work/looking for work 265 12.7

Unable to work 148 7.1

Out of work/not currently looking for work 131 6.3

Student 128 6.1

Self-employed 59 2.8

Retired 11 0.5

Housing situation (N = 2,077)

Rent 1,334 64.2

Own 347 16.7

Sharing with relatives/friends 290 14.0

(19)

Temporary 91 4.4

Homeless 15 0.7

Table 14 shows that 49 percent of SCC parents and/or caregivers reported receiving SNAP (Supplemental Nutrition Assistance Program), 66 percent received Medicaid, eight percent received earned income tax credit, 16 percent received TANF/Colorado works, and 11 percent received Head Start services. Overall, 70 percent of SCC parents and/or caregivers reported receiving at least one form of public assistance.⁵

Table 14: SCC Parents and/or Caregiver Public Assistance (N = 2,556)

SNAP

Yes 1,242 48.6

No/no answer 1,314 51.4

Medicaid

Yes 1,679 65.7

No/no answer 877 34.3

Earned income tax credit

Yes 208 8.1

TANF/Colorado Works

Yes 420 16.4

Head Start

Yes 280 11.0

Overall public assistance

Yes 1,798 70.3

No/no answer 758 29.7

As displayed in Table 15 on the following page, 42 percent of SCC parents and/or caregivers make $0-$10,000, 24 percent make $10,001-$20,000, 14 percent make $20,001-$30,000, 8 percent make $30,001-$40,000, five percent make $40,001-$50,000, and seven percent make more than $50,001. For difficulty paying bills in the last year, 16 percent of SCC parents and/or

5Note that these are likely conservative estimates as "yes" is indicated by a check mark in the data entry field. If there was no check mark, this indicated either a "no" or the respondent chose to skip the question.

(20)

Social Work Research Center | Colorado State University 16 caregivers reported no difficulty, 19 percent had a little difficulty, 29 percent had some

difficulty, 17 percent had quite a bit of difficulty, and 19 percent had a great deal of difficulty.

For annual savings in the past year, three percent of SCC parents and/or caregivers reported having more than enough, 19 percent had some money left over, 43 percent had just enough, and 35.4 percent had not enough.

Table 15: SCC Parent and/or Caregiver Socioeconomic Status

Annual household income (N = 2,033)

$0-$10,000 849 41.8

$10,001-$20,000 492 24.2

$20,001-$30,000 276 13.6

$30,001-$40,000 166 8.2

$40,001-$50,000 105 5.2

More than $50,001 145 7.1

Difficulty paying bills (N = 2,142)

No difficulty 352 16.4

A little difficulty 413 19.3

Some difficulty 616 28.8

Quite a bit of difficulty 357 16.7

A great deal of difficulty 404 18.9

Annual savings (N = 2,139)

More than enough 56 2.6

Some money left over 411 19.2

Just enough 914 42.7

Not enough 758 35.4

Responses to four health and mental health screening questions are displayed in Table 16 on the following page. These questions are asked of SCC parents and/or caregivers to help the parent support provider identify if further assessment and referral are needed. For

nervousness/anxiety, 37 percent responded they were currently experiencing it, 21 percent responded they had experienced it in the past year, nine percent responded they had not experienced it in the past year, and 33 percent responded they had never experienced it.

Regarding experiences with sadness/depression, 29 percent responded they were currently experiencing it, 23 percent responded they had experienced it in the past year, 15 percent responded they had not experienced it in the past year, and 33 percent responded they had never experienced it. Regarding experiences with sleep difficulties, 33 percent responded they were currently experiencing it, 16 percent responded they had experienced it in the past year,

(21)

Social Work Research Center | Colorado State University 17 10 percent responded they had not experienced it in the past year, and 42 percent responded they had never experienced it. Regarding experiences with physical challenges/disabilities, 20 percent responded they were currently experiencing it, eight percent responded they had experienced it in the past year, 12 percent responded they had not experienced it in the past year, and 60 percent responded they had never experienced it. Overall, 24 percent of SCC parents and/or caregivers reported having none of the challenges, 19 percent reported having one, 19 percent reported having two, 24 percent reported having three, and 15 percent reported having all four challenges, currently or in the past year.⁶

Table 16: SCC Parent and/or Caregiver Health and Mental Health Screening Questions

Challenges/Issue Frequency Percent

Nervousness/anxiety (N = 2,007)

Currently 754 37.6

In the past year 415 20.7

Not in the past year 186 9.3

Never 652 32.5

Sadness/depression (N = 2,005)

Currently 585 29.2

Never 657 32.8

Sleep difficulties (N = 2,008)

Currently 657 32.7

Never 844 42.0

Physical challenges/disabilities (N = 1,995)

Currently 401 20.1

Never 1,190 59.6

Number of challenges (N = 2,024)

None 494 24.4

One 375 18.5

Two 378 18.7

Three 480 23.7

6 It should be noted that responses from 2014 were not included because they had a different response scale.

However, data patterns from 2014 are consistent with the data presented here.

(22)

All four 297 14.7

As displayed in Table 17, 75 percent of SCC parents and/or caregivers reported having a primary care provider, and 58 percent reported having a regular dentist.

Table 17: SCC Parents and/or Caregiver Health Indicators

Primary care provider (N = 1,939)

Yes 1,462 75.4

No 477 24.6

Dentist (N = 1,930)

Yes 1,117 57.9

No 813 42.1

As displayed in Table 18 on the following page, the SCC child was male in 56 percent of the cases and Hispanic in 53 percent of the cases. The most common SCC child race is Hispanic at 50 percent, followed by Caucasian at 36 percent, bi-racial at seven percent and African American at six percent.⁷For SCC child age group, 45 percent were under two years old at referral, with 17 percent under six months, 11 percent between 6-12 months, 10 percent between 12-18 months, and eight percent between 18-24 months. Additionally, 55 percent were two or older at referral, with eight percent at 18-24 months, 16 percent at two years old, 15 percent at three years old, 14 percent at four years old, and nine percent at five years old.

7 Note that the bi-racial category includes the selection of two or more non-Hispanic race categories.

(23)

Social Work Research Center | Colorado State University 19 Table 18: SCC Child Demographics

Child gender (N = 2,430)

Male 1,358 55.9

Female 1,072 44.1

Child Hispanic (N = 2,417)

Yes 1,277 52.8

No 1,140 47.2

Child race (N = 2,421)

Hispanic 1,211 50.0

Caucasian 865 35.7

Bi-racial 162 6.7

African American 132 5.5

Native American 36 1.5

Asian 12 0.5

Hawaiian 2 0.1

Alaskan 1 0.0

Child age group (N = 2,473)*

Under 6 months 417 16.9

6-12 months 262 10.6

12-18 months 237 9.6

18-24 months 198 8.0

2 years old 392 15.9

5 years old 233 9.4

*Age range lower bound is inclusive, upper bound is exclusive.

As displayed in Table 19 on the following page, 20 percent of children were reported by parents and/or caregivers to have challenges with nervousness/anxiety, 15 percent to have challenges with sadness/depression, 26 percent to have challenges with sleep difficulties, 16 percent to have physical challenges/disabilities, 42 percent to have challenges with attention, 47 percent to have challenges with being defiant, 27 percent to have challenges with developmental delay, 24 percent to have challenges with eating/feeding, and 24 percent to have challenges with learning disabilities. Overall, 30 percent of children were reported to have no challenges, 69

(24)

Social Work Research Center | Colorado State University 20 percent reported to have at least one challenge, and one percent reported to have all nine challenges.

Table 19: Health and Mental Health Screening Questions for Any Child in the Home

Nervousness/anxiety (N = 1,896)

Yes 369 19.5

No 1,527 80.5

Sadness/depression (N = 1,891)

Yes 287 15.2

No 1,604 84.8

Sleep difficulties (N = 1,885)

Yes 495 26.3

No 1,390 73.7

Physical challenges/disabilities (N = 1,868)

Yes 305 16.3

No 1,563 83.7

Attention deficit (N = 1,915)

Yes 810 42.3

No 1,105 57.7

Child defiant (N = 1,903)

Yes 884 46.5

No 1,019 53.5

Developmental delay (N = 1,899)

Yes 513 27.0

No 1,386 73.0

Eating/feeding (N = 1,899)

Yes 451 23.7

No 1,448 76.3

Learning disabilities (N = 1,894)

Yes 456 24.1

No 1,438 75.9

Number of challenges (N = 1,937)

None 587 30.3

Between one and eight 1,326 68.5

All nine 24 1.2

(25)

Social Work Research Center | Colorado State University 21 As displayed in Table 20 on the following page, 95 percent of SCC parents and/or caregivers reported that all children in the home had a primary care provider, and 69 percent of SCC parents and/or caregivers reported that all children in the home had a regular dentist.

Table 20: SCC Child Health Indicators

Primary care provider (N = 1,902)

Yes 1,806 95.0

No 96 5.0

Dentist (N = 1,888)

Yes 1,309 69.3

No 579 30.7

As displayed in Table 21, the average SCC parent and/or caregiver age was 29.4 years, the average SCC child age was 2.3 years, the average number of people in the household was 3.8, the average number of children in the household was 1.9, and the average number of children under six in the household was 1.4.

Table 21: SCC Household Characteristics

Characteristic Min Max Mean

Caregiver age (N = 1,519) 14.0 70.0 29.4

Child age (N = 2,473) 0.0 5.9 2.3

Number of people (N = 2,498) 2.0 6.0 3.8

Number of children (N = 2,498) 1.0 5.0 1.9

Number of children under six (N = 2,498) 1.0 4.0 1.4

2.4. Parent Support Providers

This section describes the workforce for SafeCare Colorado, including training and certification, turnover, and perspectives from parent support providers with SCC.

2.4.1. Parent Support Providers: Workforce

Data presented here were collected by the NSTRC and Kempe between January 2014 and mid- April 2017, encompassing all parent support providers hired during this period, including those who have since left SCC. As displayed in Figure 3 on the following page, a total of 120 personnel

(26)

Social Work Research Center | Colorado State University 22 have been employed to facilitate implementation of SafeCare Colorado since January 2014, with 77 active direct service and administrative staff across the 13 SCC sites as of April 2017. In addition to parent support providers, sites reported 23 SCC personnel in administrative and leadership roles (e.g., site coordinators, agency directors) not involved with direct service provision.

Figure 3: Active SafeCare Colorado Staff - April 2017 (N = 77)

Among the 54 currently active PSPs, six are also certified coaches (a designation that requires training, certification, and onoing fidelity monitoring). Training site level coaches has been an important priority for SCC implementation due to their location and ability to offer expertise on-site as well as promote long term sustainability of the program.

Once trained, PSPs attain certification status after completing a series of training and fidelity monitoring requirements. Recordings of home visits are assessed weekly by certified coaches who work one-on-one with parent support providers to ensure the SafeCare® curriculum is being followed to fidelity. After parent support providers are certified and consistently meeting expectations, the coaching calls decrease to monthly but continue throughout the program.

Among the active PSPs, 44 (81 percent) were certified as of mid-April 2017, with all 44 certified in English and six certified in both English and Spanish. Ten PSPs were not yet certified by the end of the reporting period. One-third (n = 18) of active PSPs were also bilingual, able to provide services to families in both Spanish and English. Almost all sites (n = 11), in both urban and rural areas, reported having at least one Spanish-speaking PSP located onsite and able to provide services to families in both English and Spanish.

Examining the entire SCC workforce over time, a total of 96 participants have completed PSP training since 2014. Forty-one PSPs and two administrative staff have ended their employment with SCC since implementation first began, for an overall PSP turnover rate of 43 percent.

23

48

6 Admin / Non-case carrying

PSPs

Coaches / PSPs

(27)

Social Work Research Center | Colorado State University 23 Among the PSPs who have left SCC, 16 left before achieving certification and 25 PSPs departed after becoming certified.

On the following page, Table 22 presents further information on the total number of PSPs who have been hired since 2014. The first three columns on the left indicate the total number of PSPs each site has hired to date, followed by numbers of PSPs who have left SCC and their

certification status at the time of their departure. The last three columns present data on the 54 currently active PSPs and their certification status in English and/or Spanish.

Table 22: PSPs by Site: Total Hired, Departed, Currently Active and Currently Certified Total Left Before

Certification

Left After Certification

Active PSPs

Certified English

Certified Spanish^a

ACECC 9 2 1 6 5 2

BBH 9 0 4 5 5 0

Catholic Charities 8 1 2 5 4 1

Family Tree 12 0 3 9 9 2

High Plains 6 1 1 4 4 0

La Llave 3 0 1 2 2 0

Lutheran FS 6 2 1 3 2 0

Mesa 9 2 2 5 4 0

Montezuma 9 4 3 2 1 0

North Range BH^b 9 1 2 5 3 1

NW CO Health 2 0 0 2 0 0

Savio 13 3 5 5 5 0

Ute Mtn Tribe 1 0 0 1 0 0

Sum 96 16 25 54 44 6

aAll currently active Spanish certified PSPs are certified in both English and Spanish

b One person hired as a PSP at North Range BH is still with SCC, serving in an inactive PSP role

Time to achieving PSP certification can range greatly due to variation in the number of families available to serve, which is tied to a site’s stage of implementation, flow of referrals received, and engagement and attrition of families in the program. On average, PSPs have been achieving certification in about 147 days, or just under five months, with the shortest time to certification completed in 27 days and the longest spanning over a 15 month (476 days) period. As indicated in Figure 4 on the following page, PSPs who ended their employment with SCC stayed close to

(28)

Social Work Research Center | Colorado State University 24 one year (349 days), on average, before departing. The shortest PSP tenure recorded is five days and the longest a little over three years, at 1,183 days.

Figure 4: Minimum, Maximum and Average Times to PSP Certification and PSP Departure (in Days)

2.4.2. Parent Support Providers: Survey Findings

Parent support providers were surveyed in April 2017 and asked a series of questions related to professional development, technical assistance and coaching, data collection and performance targets; job satisfaction and job stress. A summary of findings are presented in the following section (full results are provided in Appendix A).

Surveys were sent via a weblink and PSPs were asked to complete the survey within two weeks. A total of 43 surveys were returned out of 58 sent for a response rate of 86 percent. Of these, 32 respondents reported spending most of their time as PSPs, while nine respondents spent most of their time as supervisors, and two respondents spent most of their time as coaches. The majority (77 percent) work full-time in the SafeCare Colorado program. Seventy nine percent of respondents have had prior experience with home visiting before beginning in their current position and the majority reported having a Bachelor’s degree (70 percent) or Master’s degree (12 percent).

27 101 147 180

476 5

124

349

517

1183

Minimum 1st Quartile Average 3rd Quartile Maximum Certification Departure

(29)

Professional Development

Survey respondents were asked a series of questions related to their professional development. Forty seven percent reported that they are required to do professional development provided by the SCC program, while 48 percent reported they receive other professional development opportunities that are required by their agency in addition to SCC specific professional development. Only five percent reported not being required to do any professional development.

PSPs were asked “In the next 12 months, what professional development activities would you like to attend to support your work in implementing SafeCare?” As displayed in Table 23, child growth and development was listed most frequently at 72 percent, followed by parenting at 60 percent. Motivational interviewing and counseling skills were also selected by a majority of respondents. Prenatal care, college or graduate level courses, and injury prevention were the least selected professional development

activities.

Table 23: Professional Development Activities Desired by PSPs

Topic for Professional Development Percent Frequency

Child growth and development 72.1 31

Parenting 60.5 26

Motivational interviewing 55.8 24

Counseling skills 53.5 23

Domestic violence 48.8 21

Mental health - Other 46.5 20

Team building 46.5 20

Leadership, supervision, or management skills 46.5 20

Managing job stress (ex: general work demands) 44.2 19

Mental health - Pregnancy-related depression 41.9 18

Substance abuse 41.9 18

Cultural competence 39.5 17

Trauma-informed care 37.2 16

Stress from working with traumatized or distressed clients (e.g., secondary trauma)

30.2 13

Training specific to the SafeCare model 30.2 13

Healthy eating and active living 27.9 12

Suicide prevention 27.9 12

Maintaining professional boundaries 27.9 12

(30)

Family planning 23.7 10

Prenatal care 18.6 8

College or graduate level courses 18.6 8

Injury prevention 16.3 7

Other 2.3% 1

Technical Assistance

PSPs were asked a series of questions related to the training and technical assistance they have received in the past 12 months. As displayed in Figure 5, respondents were most satisfied with the technical assistance they received from their peers with 82 percent reporting being very satisfied or satisfied.

Figure 5: PSP Satisfaction with Technical Assistance Provided

Coaching

The majority of respondents received coaching from site coaches located at SCC sites (65 percent), while 56 percent received coaching from the Kempe Center and 23 percent received

7%

9%

5%

2%

19%

28%

21%

14%

12%

63%

33%

35%

28%

21%

26%

33%

12%

33%

28%

47%

53%

58%

56%

0% 20% 40% 60% 80% 100%

Peers (including other sites) Site Managers (Kempe) Trainers (Kempe) Data / CQI / Research (Kempe) Outreach and Marketing (Kempe) Administration / Leadership (Kempe) NSTRC (National SafeCare Training and

Research Center)

Very dissatisfied Dissatisfied Satisfied Very Satisfied N/A

(31)

Social Work Research Center | Colorado State University 27 coaching from NSTRC. PSPs were asked to rate different aspects of the coaching that they received in the past 12 months. As displayed in Figure 6 on the following page, 65 percent of PSPs were satisfied or very satisfied with the timeliness of coaching, 76 percent were satisfied or very satisfied with the access to coaching, 78 percent were satisfied or very satisfied with the amount of coaching, and 82 percent of PSPs were satisfied or very satisfied with the quality of coaching.

Figure 6: Percent of PSP Satisfaction with Coaching Experience

PSPs also were asked about challenges and successes with coaching. Respondents noted that coaching has been helpful, with statements such as “I am able to feel consistently supported in my job because of the availability of coaching.” The ease and availability of coaching was noted as a success for some while others reported that their coach was not as accessible or available as they would have liked.

0%

5%

2%

5%

7%

5%

0%

12%

21%

31%

35%

40%

33%

29%

47%

38%

43%

36%

Quality of coaching Amount of coaching Access to coaching Timeliness of coaching

Very satisfied Satisfied Neutral Dissatisfied Very dissatisfied

(32)

Social Work Research Center | Colorado State University 28 Data Collection and Performance Targets

PSPs were asked to rate their level of agreement with statements related to data collection and performance targets, such as required number of face-to-face visits each week and maintaining a full caseload. On the following page, Figures 7 and 8 present the ratings from respondents. As shown, 70 percent of PSPs agreed or strongly agreed that the data collection requirements are clear and easy to follow, 65 percent agreed or strongly agreed that the data reports are useful and relevant, 63 percent agreed or strongly agreed that performance targets are clear and easy to understand, while only 34 percent of PSPs agreed or strongly agreed that the performance targets are

reasonable and achievable.

Figure 7: PSP Percent Agreement with Data Collection and Report Statements

Figure 8: PSP Percent Agreement with Performance Target Statements 0%

10%

20%

30%

40%

50%

60%

Data collection requirements are clear and easy to follow

Data reports (from Access) are useful and relevant to the work

Strongly disagree Disagree Neutal Agree Strongly Agree

(33)

Social Work Research Center | Colorado State University 29 Related to performance targets, many respondents discussed the challenge of families

cancelleling visits or discontinuing services. One respondent commented, “I struggled with hitting the performance targets due to client cancellations and recently also intake cancellations, which are necessary to get to the expected number of active clients.”

Another survey respondent noted, “I feel that the performance target of 12 face-to-face (FTF) visits a week is reasonable. However, the reality is that I have a big rate of

inconsistency with actual follow through of the home visits on the part of my clients. I feel like that is a huge barrier to the expected FTF visits of 12 per week. I think as a parent support provider, I feel a lot of pressure to reach those targets.”

Job Satisfaction

PSPs were asked to rate their overall job satisfaction on a scale of 1-5 where 1 is very dissatisfied and 5 is very satisfied. As shown in Figure 9, the majority of respondents (70 percent) reported being satisfied or very satisfied with their job.

Figure 9: Parent Support Providers' Overall Job Satisfaction 0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

Performance targets are clear and easy to understand

Performance targets are reasonable and achieveable

Strongly disagree Disagree Neutal Agree Strongly Agree

(34)

Social Work Research Center | Colorado State University 30 PSPs were asked to rate their level of job stress, from 1-5, where 1 is no job stress and 5 is a signficant amount of stress. The mean stress level reported was 3.37. They were also asked to rate their level of burnout, defined as “being constantly overwhelmed,

exhausted, or pessimistic.” The mean burnout level reported was 2.37 on a scale of 1-5, where 1 is no burnout and 5 is a significant amount of burnout. Figure 10 on the

following page presents responses to the question “What caused you the most job stress?” As shown, clients dropping out of services was the most often cited reason for job stress, followed by caring for clients facing substance abuse or mental health issues, and concerns about agency leadership or supervision.

Figure 10: PSP Reported Reasons for Job Stress 2%

0%

28%

47%

23%

Very dissastisfied 2 3 4 Very satisfied

(35)

Social Work Research Center | Colorado State University 31 Lastly, the commitment that many PSPs have for the program, is exemplified by the

following quote: “I look forward to my journey with Safecare and the ability to positively impact families in our community.”

3. Outcome Evaluation

70%

44%

40%

26%

23%

21%

19%

14%

12%

14%

Client drop out

Caring for clients facing mental health or substance abuse issues

Concerns about agency leadership of supervision

Balancing work and personal life

Administrative requirements (ex: data reporting, paperwork)

Salary

Managing sensitive nature of the work

Personal challenges like poor health or family stress

Travel requirements for home visits

Case load is too heavy

Staff turnover and position vacancies

Other:

(36)

Social Work Research Center | Colorado State University 32 The outcome evaluation section includes the following three sections: short term outcomes from SafeCare® surveys assessing skill uptake within each topic; longer term descriptive outcomes without a comparison group from the Trails database for parent and/or caregivers with differing levels of SafeCare Colorado engagement; and child welfare re-involvement outcomes for SCC completers and a matched comparison group.

3.1. Changes in Parent and/or Caregiver’s Skill Level

Whether or not a family passes a topic of SafeCare® is determined by their scores on

assessments designed for each SafeCare® topic: Safety, Health and Parent-Child and Parent- Infant Interaction (Interaction). The SafeCare® program establishes the criteria for a passing score on a topic. Assessments were completed by families at the beginning and end of each SafeCare® topic (see Appendix B for assessments). The following assessment results provide an overall picture of families’ progress when compared to the required passing scores.

Success is defined as follows for each of the three SafeCare® topics. For Safety, success includes a plan for removal of all hazards or a plan for increased supervision, with a note that it may be difficult to remove all hazards and one to three remaining is acceptable. For Health, success is defined by SafeCare® as a score of 100 percent for emergency room scenarios; a score of 80 percent for doctor’s appointment scenarios, and a score of 80 percent for care at home scenarios. Finally, for Interaction, success is defined as demonstrating at least one positive behavior in each of the observed categories being assessed. For example, before the observed activity begins (e.g., getting dressed), the parent is taught to explain the activity to the child, which includes “gets the child’s attention” and “explains the activity.” If the parent does just one of these, they would meet success for that observed area.

Assessments were only analyzed for families recorded as completing the specified topic (for example, home safety assessments were only included for families who completed the Safety topic). Assessments occurring in session one or two were counted as “pre-tests” and only assessments occurring in session five or six were included as “post-tests.” A family was only included in this analysis if they had completed both a pre- and post-test.

3.1.1. Safety Topic

(37)

Social Work Research Center | Colorado State University 33 A sample of 787 families had home safety assessment data which could be analyzed for changes in parent and/or caregiver skill level.⁸ As shown in Table 24, the average number of hazards in the home for this group of families at pre-test was 54; the post-test average number of hazards was six, for an average decrease of 48 hazards per household or 89 percent.

Although in principle all hazards should be removed from a home, this goal is not always realistic. Based on the training criterion selected by Gershater-Molko, Lutzker, and Wesch (2003), families who complete the Safety topic should achieve a reduction of 85 percent or more in the number of hazards. As shown in the last row of Table 24, 68 percent of families with home safety assessment data met this criterion. However, recent published studies of SafeCare® do not report on family assessments, making it difficult to place these results in the context of other implementations.

Table 24: Change in SCC Home Safety Skills (N = 787)

Pre-test hazards Post-test hazards Percent drop in

hazards

Average hazard count 54 6 89%

Number of families meeting 85% reduction in hazards 551 (70%) 3.1.2. Health Topic

Only families with all three child health areas completed at pre-test and post-test were included in the child health assessment analyses. This resulted in a sample of 688 families with child health data which could be analyzed. To complete the Health topic, families had to achieve a 100 percent score for the emergency room scenario and at least an 80 percent score for the doctor appointment and care at home scenarios. As shown in Table 25 on the following page, the average post-test scores for child health skills were: 99 percent for emergency room scenarios⁹, 97 percent for doctor appointment scenarios and 96 percent for care at home scenarios. These results are comparable to the baseline and “posttraining” scores as reported by Gershater-Molko et al. (2003).

8 There was inconsistency in the text data entry for data recording hazards in three rooms of the home across the two time points. For example, “living room” may have been defined as room one at the pre-test and room 3 at the post-test. Therefore assessments were only included in this analysis if all three rooms were checked at both time points. Furthermore, the data collection interface was designed to default to “0” for the number of hazards in a room instead of a blank field. Therefore, a “0” was only considered to be assessment data if that score had a corresponding and valid room name.

9 The underlying data showed that many families scored 100 percent on the pre-test for the emergency room scenario, suggesting that this scenario was not one where parents needed as much education as for the others.

(38)

Social Work Research Center | Colorado State University 34 Table 25: Change in SafeCare Child Health Skills (N = 688)

Emergency Room Baseline Post-test

Score Score

Doctor Appointment Baseline Post-test

Score Score

Care at Home Baseline Post-test

Score Score Average Score 85% 99% 60% 97% 68% 96%

Percent Change 14% 37% 28%

3.1.3. Interaction Topic

Similar to the Health topic, interaction assessments were only included in this analysis if all three activities were observed and scored at both time points.¹⁰ This resulted in a sample of 200 families with PCI data and 39 families with PII data which could be analyzed. In order to pass these modules, parents had to demonstrate at least one positive behavior in each of ten observed categories being assessed.

As shown in Table 26, the average percentage point change for PCI skills was 124 percent with a final mean score of 47 points (out of 60 possible). The average percentage point change for PII skills was 38 percent, with a final average post-test score of 22 points (out of 24 possible). Note that baseline scores were closer to the maximum for the PII data (16 out of 24). Parents

assessed using the PCI data started lower, relative to the maximum score, but achieved substantial gains.

Table 26: Change in Parent-Child Interaction and Parent-Infant Interaction Skill Scores Baseline

Score

Post-test Score

Percentage Change

Parent-Child Average Score (N = 200) 21 47 124%

Parent-Infant Average Score (N = 39) 16 22 38%

3.1.4. Unsuccessful Completion

10 There was inconsistency in the text data entry for activities observed by the parent support provider in the Interaction topic. For example, “snack time” may have been defined as the first activity at the pre-test but as the last activity at post-test.

Social Work Research Center

SafeCare Colorado Program Evaluation Report 2014-2017

Prepared by:

Kristy Beachy-Quick Chris Lee Luke McConnell

Rebecca Orsi Zachary Timpe Marc Winokur

Social Work Research Center School of Social Work

Prepared for:

Office of Early Childhood

Colorado Department of Human Services

Table of Contents

SafeCare Colorado Program Evaluation Report 2014-2017

1. Introduction

2. Process Evaluation

3. Outcome Evaluation