The Columbus Model: Becoming a Data Driven System


Toolkits | October 14, 2010

Files: PDF | 123 KB | 4 pages

The HEARTH Act requires communities to implement strategies to prevent the loss of housing, help people quickly move out of homelessness and into housing, and ensure housing stability. At the same time, communities must track and report their homeless assistance system’s progress toward these outcomes. Performing well on these outcomes requires communities to realign their systems to focus on housing-based solutions and strengthen their capacity to collect data and information across programs and improve overall system performance.

This document examines performance measurement and improvement tools and strategies that have helped Columbus, Ohio perform well on HEARTH outcomes.

Columbus, OH has consistently performed well on outcomes in the HEARTH Act, including reducing length of homelessness, new, repeat, and overall homelessness.

In 1995, 1,168 families entered Columbus’s homeless system. This figure peaked at 1,217 in 1997 before declining to 746 in 2009. From 2007 to 2009, the community continued to reduce homelessness in the midst of a nationwide recession with 7 percent, 6 percent, and 4 percent reductions in overall, family, and single adult homelessness respectively during that two-year time frame. In 2009, the majority of homeless families exited the “front door” emergency shelter for housing within 21 days, and more than 60 percent of homeless families exited homelessness for housing where they were the leaseholder or homeowner. Only one percent of single adults and no families returned to homelessness in 2009.

Columbus has achieved these outcomes through a homeless assistance system that quickly connects homeless people to housing and provides the appropriate case management and mainstream service connections to help them achieve stability. Further, Columbus’ continuum operates through an outcomes-based funding model that uses measurable performance standards to monitor agencies’ and systems’ progress toward achieving community goals. As such, these performance standards reinforce an overall vision and strategy for improving the homeless services system and working toward the eventual elimination of homelessness.

Practices such as these make Columbus’ CoC an effective data-driven system. The following describes the approach and tools Columbus uses to track and evaluate system activities and performance as well as the community’s performance improvement strategies. Additional resource documents and tools related to Columbus’ performance measurement and evaluation strategies are also available for download.

Columbus’ approach to data tracking and management is characterized by consistency, transparency, and a high-level of participation. One hundred percent of CoC service providers participate in the community’s HMIS, and Columbus has HMIS coverage of 98 percent of shelters, 91 percent of transitional housing, and 95 percent of permanent supportive housing providers. The HMIS operates as an open system, which makes most of the client information available to all providers in the system including emergency shelter history and receipt of financial assistance. HIPPA-protected health information and domestic violence related information cannot be viewed. The lead agency for Columbus’ CoC, the Community Shelter Board (CSB), meets regularly with HMIS administrators to review system operations and get feedback on needed improvements.

Columbus’ CoC providers track more than 30 client- and program-level outcome measures related but not limited to:

  • Length of client stay and/or participation in a program,
  • Cost per household served, per unit operated, and cost-efficiency,
  • Employment and benefits status changes and income changes,
  • Housing affordability and stability, and
  • Recidivism for prevention, diversion, shelter and permanent housing.

These and other measures are folded into 15 system-wide measures; progress on program- and system- level measures is reviewed quarterly. Each program and system type has its own measures tailored to their specifics (i.e. shelter outcomes are different than supportive housing outcomes). All programs participating in the HMIS are held to contractual data quality and performance standards, established by both HUD and local guidelines, which are checked at least on a quarterly basis. In addition, on-site data quality verifications are performed at each program on an annual basis.

As the lead agency, CSB shepherds the CoC HUD application process and, therefore, performs the tasks that Collaborative Applicants will be expected to perform under the HEARTH Act. As the Collaborative Applicant, CSB will be responsible for ensuring that all CoC programs participate in the HMIS. Managing this responsibility will be easier if CSB is designated as the Unified Funding Agency for Columbus’ CoC and manages funding for the continuum. Currently, CSB disperses nearly all local, state, and federal funds as well as private funds for the CoC. Only those programs that participate in HMIS receive funding from CSB.

For a full list of the data metrics and performance measures tracked in Columbus, click here.

As mentioned, progress on program- and system- level measures is reviewed quarterly and published in the System & Program Level Indicator Report (available on CSB’s website). The report evaluates each system component (i.e., single adult system, family system, permanent supportive housing system) and program (e.g., the YWCA “front door” emergency shelter) based on the 15 system-wide performance measures mentioned above, program goals, variances, and outcome achievements. “Outcome achievement” is defined as 90 percent or better of a numerical target or within five percentage points of a percentage goal. If a system or program meets less than one-half of the outcome goal(s) or encounters repeated difficulties in meeting specific performance goals, it is considered a “low performer” or “program/system of concern.”

Performance benchmarks are established and reviewed annually by the CoC Steering Committee; CSB’s Board of Trustees also reviews performance benchmarks regularly. In order to set performance targets, annual performance is compared with previously established benchmarks and historical performance information. The CSB, its board, the CoC Steering Committee, and other stakeholders examine all performance targets – both unmet and achieved – each year and adjust them to reflect more aggressive targets if benchmarks are achieved or exceeded. Benchmarks are set in the context of Columbus’ Ten Year Plan to End Homelessness, and programs as well as system components are evaluated on “success in ending homelessness.” When new metrics are introduced, a performance benchmarking period is set in order to allow time to appropriately set performance targets.

CSB meets regularly with program administrators and community leaders to review program and system performance and obtain feedback on how to address gaps or improvement needs.

To access a sample System & Program Level Indicator Report, click here. For a full summary of Columbus’ Program Evaluation Standards, click here.

Columbus uses several tools and strategies to maintain performance accountability and encourage performance improvement.

The CoC Steering Committee sets and reviews program performance standards annually, based on HUD requirements and local standards. CSB incorporates these standards into annual program agreements with each provider agency. The agreement includes an annual Program Outcome Plan (POP), which sets each program’s quarterly, semi-annual, and annual goals for all performance indicators for the upcoming year. The POP becomes the basis for performance-based contracts and program evaluation. (To access a sample POP, click here.)

Columbus uses performance-based contracts to hold system agencies accountable, and the community also uses them to improve system performance or address system gaps. For example, if the intent for a performance-based contract is to increase successful housing outcomes, CSB will “purchase” a given number of outcomes from the relevant program(s) or agency(-ies). Performance-based contracts are primarily used for family emergency shelter providers while pro-rated payments and traditional contracts are used for single adult emergency shelter providers. Reimbursement contracts are used for the supportive housing programs. To view samples of performance-based contracts from Columbus, click here. All programs are required to comply with CSB’s Administrative and Program Standards.

If a program encounters repeated difficulties meeting any specific performance goal(s) or fails to achieve its overall performance targets, it is considered a “program of concern” or a “low” performer. Poorly performing programs must participate in the Quality Improvement Intervention Program (QII). In the QII, CSB works one-on-one with the program to support performance improvement. Through quarterly dialogues, technical assistance, and continuous monitoring, CSB and the provider agency consider agency plans and progress on addressing their program issues. The provider agency is required to submit an Improvement Plan to CSB that is monitored as it is implemented. If the agency and/or CSB find that the QII process is not working, or the process otherwise fails, either agency may refer the concerns or issues to the HUD Technical Review Committee.4 The program may also be defunded if the QII does not work. To learn more about Columbus’ QII, click here.

The Columbus Model: Performance Measurement and Evaluation
The Columbus Model: Quality Improvement
The Columbus Model: System Overview

Downloadable Tools

  1. Columbus Program Performance & Evaluation Measures List
  2. Columbus Program Performance Measures Definitions and Measurement Methodology
  3. Columbus System Performance Measures Definitions and Measurement Methodology
  4. Columbus Sample System & Program Level Indicator Report
  5. Columbus Performance Outcome Plan (POP) Overview
  6. Columbus Performance Outcome Plan (POP) – Blank Sample
  7. Columbus Performance-based Contracts Samples
  8. Columbus & Community Shelter Board Administrative and Program Standards

Additional Resources

Community Shelter Board Website