Toolkits | November 5, 2010

Files: PDF | 109 KB | 5 pages

Columbus's continuum operates through an outcomes-based funding model that uses measurable performance standards to monitor agencies and systems progress toward achieving community goals. The community evaluates 15 system-level performance measures and more than 30 client- and program- level measures. These performance standards reinforce an overall vision and strategy for improving the homeless services system and working toward the eventual elimination of homelessness.

A central component of Columbus evaluation process involves scoring individual providers as High, Medium or Low performers. Scores are based on achievement of overall program performance goals and outcomes that are developed by CSB in collaboration with the service provider and outlined in the agency's contract. In addition, Columbus evaluates each performance goal individually as Achieved (Yes/Y), Not Achieved (No/N), or Not Applicable (N/A). An Achieved Goal is defined as 90 percent or better of a numerical goal or within 5 percentage points of a percentage goal. Not Applicable is assigned when a performance goal was not assigned. A program that experiences repeated difficulty meeting performance goals is considered a program of concern. To learn more about Columbus's performance measurement and evaluation processes, click here.

In order to drive overall system improvement and reductions in homelessness, Columbus uses a combination of approaches. CSB incentivizes a high level of performance by using performance-based contracts. In addition, CSB encourages agencies to perform well by making funding decisions and merit awards based on agencies’ annual evaluation and performance ratings. To view sample performance-based contracts, click here.

One of the most remarkable approaches used in Columbus to help agencies strengthen their performance is CSB’s Quality Improvement Intervention Program (QII). Programs rated as “Low,” identified as a “program of concern,” or that fail to comply with CSB’s Administrative and Program Standards go through the QII, which covers the following core areas:

• Problem identification
• Collaborative goal setting and improvement planning
• Responsibility assignment
• Progress updates

Low performers are required to participate in the QII; however, if the low performing program and CSB cannot reach agreement on these core areas, the agency may appeal to the CoC Steering Committee for review and final recommendation.

Throughout the QII process, CSB conducts at least quarterly one-on-one dialogues with the troubled agency. Agencies may also submit monthly indicator reports to CSB and check-in regularly regarding specific action steps and timelines. The process may include organizational development opportunities such as training, peer-to-peer learning opportunities, and outside consulting or facilitation support. The QII may be as short a month or last up to 24 months and is tailored to each program’s needs. If a program is unable to deliver acceptable performance outcomes, it is defunded.

Columbus has a 100 percent success rate for improving organizations outcomes and helping them achieve system standards.

The following scenarios are based on case studies from organizations in multiple communities, not solely Columbus, OH. The information from these cases was altered to maintain the anonymity and privacy of the subject agencies.

Intervention:
To improve service delivery at this organization, the lead CoC agency and the subject agency completed the following steps for the QII:

1. Discuss performance outcomes for the last 12 months.
2. Observe, detail, and chart the agency’s process from intake to housing placement including length of time (in days) it takes to complete each step.
3. Identify specific areas where processes could be further streamlined.
4. Review case files of clients who exited the program without housing and/or exited and returned to homelessness.
5. Conduct client interviews.
6. Plan and implement changes to agency’s process.

The QII revealed that the organization’s service process is overly linear and lengthens clients’ stays in homelessness. Clients’ length of homelessness significantly impacts their housing outcomes. For example, the organization’s intake and assessment processes take only a couple of hours. However, this information is not reviewed or used to start addressing housing barriers (e.g. linking clients with benefits) until the next case team meeting. Team meetings only occur one time per week, and all the clients that entered the agency that week are discussed. The timing of the case meeting relative to a client’s intake and assessment accounts for approximately one week in length of stay variance.

Housing location for clients does not begin until nearly all the housing barriers (e.g., approval of benefits income, approval of relocation assistance) have been addressed, which takes 30 to 40 days. The housing location process takes an additional 3 to 4 weeks to complete. Many clients voluntarily leave the agency during this time but show up in another shelter program only a short time thereafter. Client interviews revealed that clients leave the program prematurely because they find shelter life stressful and they hear that other programs place clients in housing faster. Additionally, some clients are frustrated because they observe clients entering the program as much as 3 weeks later and placed into housing more quickly.

Outcome:
The QII resulted in the following outcomes and changes for this agency:

• Clients complete a shelter exit/housing plan within 48 hours of intake and assessment.
• Team meetings occur on a rolling basis, up to 3 times per week, to facilitate faster response and assistance times from staff in helping clients complete their shelter exit/housing plan.
• Housing location begins within one week of intake.
• One case manager’s role within the organization shifted to focus on helping clients access mainstream benefits, including assisting clients in obtaining the appropriate documentation to move forward.
• Clients are connected with the benefits case worker within 24 hours of intake and assessment.
• Clients are required to schedule and complete meetings with all relevant benefits programs within one week of intake.
• Intermediate benchmarks and objectives were identified, and follow-up meetings were scheduled to monitor progress. The benchmarks/objectives identified were:
• Reduce median length of stay by 3 to 5 days each month for the next 9 months;
• Improve percentage of successful housing outcomes by 5 percentage points, on average, each month for next 9 months;
• Within three months of implementing streamlined service delivery process, complete tasks related to benefits-income and complete first team case meeting within one week of intake for 100 percent of clients.
• Participate in progress update meetings, quarterly, for 12 months.

Intervention:
To improve service delivery at this organization, the lead CoC agency and the subject agency completed the following steps for the QII:

1. Discuss performance outcomes for the last 12 months with executive leadership.
2. Conference with senior case manager(s) to learn more about the approach to supportive services.
3. Comparatively review this organization’s client case files and rapidly re-housed client case files.
4. Plan and implement changes to agency’s process.

The QII found several areas where this agency could adjust its approach and become more cost-effective. These key areas include targeting practices, client programs, and client support services. Regarding targeting, this transitional program only accepts families with limited housing barriers and a strong likelihood of success. Often, the head of household has some employment history, limited history with substance use, at least six months of “clean time” where substance use has been an issue, and minor criminal history if any. Clients served in the program are required to participate in several classes and programs on topics such as parenting, health and nutrition, and financial management. Clients are also required to participate in a vocational, employment, or other education or training program, and they must demonstrate the development of specific life skills such as cleaning, improved social networks and skills (i.e., better friend choices, limited visitors, etc.), and cooking.

Clients in this transitional program demonstrate strong outcomes. However, review of client files shows that clients with similar profiles achieve the same outcomes when rapidly re-housed directly from shelter and provided with short-term or no home-based case management. In fact, families with this client-profile obtain most of their income and social service resources while they are in shelter and can maintain housing with limited financial assistance in the form of a security deposit, back rent payment, and/or first month’s rent. Case management for these clients in a rapid re-housing program builds on client progress in shelter and focuses on helping clients:

• Access resources,
• Quickly access permanent housing,
• Build a landlord-tenant relationship,
• Increase benefits and employment income,
• Develop a housing-focused budget, and
• Maintain or access new social service resources in their transition to permanent housing.

The transitional housing agency’s required programs and case management account for most of the difference in cost per household because they are financed only by the agency’s budget and do not leverage community resources. Further, the rapid re-housing programs serve a larger number of households in a fiscal term than the transitional program, which further decreases their cost per household. Finally, the majority of clients in the transitional housing program remain in the program until they obtain a permanent housing voucher. This makes the permanent housing voucher the primary driver of housing stability for this program’s clients and suggests that the programs and services are unnecessary to help the family access and maintain permanent housing.

Outcome:
At the conclusion of the QII, the lead agency and subject agency jointly determined several options to improve the cost-efficiency of the transitional housing program. The agency could target its programs and services to families with significantly higher barriers. Examples include families involved with child welfare or in need of family reunification and families in which the head of household has active and severe substance use issues. Alternatively, the agency could shift its service model to a rapid re-housing model that is comparable to that of its peers, or become a short-term (i.e., 3 to 6 months) transitional program that serves households with slightly higher barriers than its current population. Examples of the latter include households in which the head of household has a criminal background that significantly impacts his/her ability to obtain housing and employment, citizenship issues, long-term unemployment, and/or extremely poor housing history.

The agency chose to become a short-term transitional housing program that takes clients with slightly higher housing barriers, making the resources expended more cost-appropriate for the client population served. The agency’s transition took 12 months and resulted in the following outcomes and changes:

• Target households with the following housing barriers:
• Long-term unemployment,
• Very poor housing histories.
• Significant criminal backgrounds,
• Citizenship issues, and/or
• Inability to be rapidly re-housed (based on a referral from rapid re-housing case manager or coordinator).
• Discontinue required programming and shift to a limited number of required home-based case visits
• Shift case managers’ roles to emphasize connection to mainstream benefits and services and service “brokering”
• Expand case managers’ responsibilities to include housing location and working with landlords (this was previously a marginal responsibility)
• Participate in training sponsored by the lead agency, including:
• Peer-to-peer learning opportunities and site visits,
• Housing location and working with landlords training, and
• Training on coordinating with mainstream benefits and services programs.
• Participate in quarterly progress update meetings for 18 months.

Intervention:
To improve service delivery at this organization, the lead CoC agency and the subject agency completed the following steps for the QII:

1. Identify the data quality violations.
2. Review client files for the last 12 months of operation.
3. Require data quality issues to be addressed within one month.
4. If issues are unresolved within one month, place the agency in contractual breach and stop funding until the problem is resolved.

Outcome:

1. Follow up with agency after one month.
2. Regular monitoring through monthly indicator reports.
3. Conclude the QII once the intervention requirements are met.

The Columbus Model: Becoming a Data Driven System
The Columbus Model: System Overview
The Columbus Model: Performance Measurement & Evaluation

Downloadable Tools

1. Columbus Program Performance & Evaluation Measures List
2. Columbus Program Performance Measures Definitions and Measurement Methodology
3. Columbus System Performance Measures Definitions and Measurement Methodology
4. Columbus Sample System & Program Level Indicator Report
5. Columbus Performance Outcome Plan (POP) Overview
6. Columbus Performance Outcome Plan (POP) – Blank Sample
7. Columbus Performance-based Contracts Samples
8. Columbus & Community Shelter Board Administrative and Program Standards