THE CALIFORNIA WELLNESS FOUNDATION

Health Improvement Initiative

FINAL EVALUATION REPORT

Other availability: PDF

Published: June 2003

Author(s): Group Health Cooperative of Puget Sound

---

Table of Contents

1. The HII’s Logic: A Theory of Action
2. The HII Evaluation Design: A Focus on Intermediate Outcomes
3. Partnership Achievements
   • Governance
   • Systems Change
   • Direct Services
   • Population Health Measurement
4. Effects Within Partnership Communities
5. Factors Associated With Success
6. Overall Impact
7. The Theory of Action
8. Implications for Future Efforts

Improving the health of entire populations is a compelling goal for anyone involved in health promotion, whether as an individual researcher or practitioner or as a major funder or government agency. In fact, affecting the health of populations may well be the gold standard by which health promotion interventions are judged.

Why is this goal so persistently alluring, even though it is clearly difficult to attain?

First, it appeals to our sense of fairness and basic equity. If improved health is attainable, we reason, then entire populations – not just fortunate individuals – should share the potential for improved health and quality of life.

Second, it is possible. The Finnish successes of the 1970s, reducing several chronic disease risk factors simultaneously with sustained community health interventions, have inspired both large- and small-scale replications all over the world.

Finally, it makes sense – both intuitively and economically. In the United States, we spend 50 percent more of our Gross Domestic Product (GDP) on health than the rest of the developed world, but we have little to show for it in terms of health outcomes. We are as healthy as other developing nations in only one health indicator: life expectancy over the age of 80. In overall life expectancy, infant mortality, and a host of other indicators, our investments in technology and treatment do not buy us greater health and quality of life. In fact, as Robert Evans and Gregory Stoddard have written, “A society that spends so much on health care that it cannot or will not spend adequately on other health-enhancing activities may actually be reducing the health of its population.” ¹

Improving the health of populations by investing in systems change is promising, potentially viable, and equitable. So why is it so difficult? What factors can increase the chances of success?

In 1996, The California Wellness Foundation (TCWF) launched the Health Improvement Initiative (HII) to answer these questions by identifying the critical factors needed to bring about health systems change at the community level. Learning from previous attempts that lacked sufficient resources, technical support, and realistic time frames, TCWF’s initiative was ambitious and well planned. TCWF invested $20 million in the effort and incorporated a strong support and evaluation structure into the Initiative’s design. The HII spanned five years – a planning year from 1996 to 1997, followed by four years of implementation between 1997 and 2001.

• This report represents the final evaluation of the HII’s first cohort of nine Health Partnership Programs. As described below, the Partnerships anchored a range of planning, intervention, evaluation, and policy activities that contributed to the HII’s outcomes in individual communities as well as to collective outcomes for the Initiative as a whole.

The HII’s Logic: A Theory of Action

The HII consisted of three integrated components:

The Health Partnership Program funded nine California communities to plan and implement population health improvements in four ways: through governance structures, direct services, health systems changes, and population health measurement. Three of the Partnerships included entire counties (Mendocino Community Health Partnership, Solano Health Improvement Initiative, Tehama County Health Partnership), one included a trio of cities (Western Coachella Valley Health Partnership), and others included different configurations of census tracts or zip codes (Oceanside Partners for Healthy Communities, Pasadena/Altadena Health Partnership, North Sacramento/Del Paso Community Alliance, Contra Costa County Partners for Health, and Sonoma County Community Health Partnership).

The Initiative Support Program funded additional Grantees to provide direct technical assistance to the Partnerships (Center for Civic Partnerships), to help them measure population health (The Field Institute/Louis Harris and Associates), to evaluate their efforts (Group Health Community Foundation, and to provide support in Initiative organization/meetings/logistics (Education Programs Associates).

The Public Education/Policy Program created the California Center for Health Improvement (CCHI), the first statewide, independent organization focused on health policy. CCHI in conjunction with The Field Institute and Louis Harris and Associates conducted surveys of the opinions of Californians, and provided public education and information about population health to policy makers and opinion leaders. Grantees under this program also provided policy development support to the Partnerships and the Initiative as a whole.

The conceptual model underlying the HII was the Theory of Action. At the model’s center were the nine Partnerships. The HII began with a planning year, during which Partnerships created a governance structure (or, in some cases, updated an existing one) and reached consensus on detailed Health Improvement Plans that would guide the following four years of implementation. (These were updated annually.)

The governing structure and planning efforts led to goals and activities in three areas: identifying and creating systems change, providing direct preventive health services, and measuring population health. These activities would be supported by technical assistance in data collection, evaluation, and policy development that would not only guide specific efforts, but also create a “learning community” where innovation and problem-solving could be shared across the Partnerships. Collectively, the combination of planning, systems changes, service delivery changes, and technical support would lead to improved population health outcomes, which in turn would be detected because of advances in the breadth and sophistication of population health measures.

The HII Evaluation Design: A Focus on Intermediate Outcomes

Although the HII was designed to contribute to detectable changes in population health outcomes, it was clear at the outset that the HII’s five-year time frame would yield few of these. Instead, the evaluation was designed to capture intermediate outcomes, concentrating on systems changes at the community level and in the provision of direct services.

The evaluation design relied on a participatory approach. The evaluation team – Group Health Community Foundation– designed data collection instruments and analyzed data, in collaboration with local evaluators working directly with the Partnerships. Although this caused some duplication of effort and inconsistencies in data collection, it did allow individual Partnerships to participate in a broader evaluation effort and to share knowledge not only with the evaluation team but with one another as well.

As with any large-scale evaluation of a complex, multi-site effort, this evaluation had to balance the need for detailed information with the burden its collection might impose on the Partnerships. Sources of both qualitative and quantitative data included key informant interviews, governance surveys to assess internal Partnership and management issues, program logs documenting key activities, semi-annual progress reports, site visits, technical support surveys, and polling data.

Using the Theory of Action as its guide, the evaluation sought answers to the following questions:

• What did each Partnership achieve in the HII’s four areas of emphasis: governance, systems change, direct services, and population health measurement?
• What factors appeared to lead to greater success?
• What was the impact of each Partnership in its community? What efforts are likely to be sustained?
• How did the Support Grantees contribute to the success of the individual Partnerships and to the success of the HII overall?
• What was the HII’s impact beyond the nine funded Partnership communities?
• To what extent was the HII Theory of Action validated as an approach to community health improvement?
• What are the implications of the HII for future efforts?

Highlights of the evaluation’s insights into these questions are provided below.

Partnership Achievements

Governance
No single governance structure, frequency of meetings, or level of community resident involvement was associated with Partnership achievements, suggesting that a variety of configurations can support population health improvements. Several of the Partnerships predated the HII, in whole or in part. The organization’s longevity did not appear to be a factor in Partnership success, but pre-existing relationships among key stakeholders were. Whether the Partnerships built on coalitions that predated the HII or were created in response, they reported heightened levels of information sharing, cross-agency referrals, joint projects, and overall coordination as a result of the HII activities.

Not surprisingly, strong project staff and good relationships within the Partnerships (including relationships between members and staff) appeared to contribute to higher levels of accomplishment.

Systems Change
The broad objective of “systems change” was refined early in the HII to include four specific types of changes:

• Service Integration – providing comprehensive, integrated services responsive to the needs of community residents.
• Results-based Budgeting – changing the process by which local funding decisions are made and more closely linking budgets to outcomes.
• Data Integration – increasing the extent to which data are organized and shared across agencies.
• Policy Development – developing and implementing new policies that promote population health.

Of the 24 systems change activities identified through the evaluation, 13 fell into the category of services integration. These changes – which represented eight of the nine Partnerships – included resource centers and physical co-location of formerly fragmented services, as well as greater inter-agency cooperation.

Two Partnerships addressed systems changes in results-based budgeting, three in data integration, and five in policy development.

In assessing the success of efforts, the evaluation team applied two criteria:

• Is the change likely to be sustained?
• Is its potential health status impact significant, either in terms of the number of people affected or the degree of impact on individuals?

For activities that met these criteria, a third was applied:

• How critical was the HII’s role in achieving sustainable, significant systems change?

Applying the first two criteria winnowed the list of 24 systems change activities to 15 – i.e., 15 that would be sustained beyond the HII funding period and appeared to be significant in terms of their health impact (either in terms of broad scope or individual impact).

Next, the evaluation team used key informant interviews, case studies, and site visit and progress reports to rate the HII’s specific contribution as critical (i.e., the systems change almost certainly would not have happened without the HII’s impetus), key (i.e., the Partnership was a major factor, but the change may have occurred on its own), and minor (i.e., the Partnership was involved, but was not a major factor). Applying this screen, the HII Partnerships were critical in seven of the 15 systems change activities and played a key role in another four activities. In the remaining four, the Partnership played a minor role.

Like the overall pool from which they were drawn, most of the seven systems change activities in which the HII Partnerships played a critical role fell into the service integration category (n=5). The other two systems changes involved policy development and data integration.

Direct Services
Direct services included health promotion and/or disease prevention services provided directly to individuals or groups. Examples of direct services launched with HII funding included case management and referral services, mentoring for at-risk youth, HIV testing, immunizations, preventive dental care, a “patient navigator” system for cancer patients, literacy training, and support for enrolling in health insurance. Most of these services will be sustained beyond the HII’s June 2001 end date.

The direct services were self-reported by Partnerships; each Partnership differed not only in the types of services provided but also in the degree to which these were captured for evaluation purposes. In addition, the data were reported as duplicated counts, making it difficult to determine the number of individuals served. Similarly, the mingling of HII and other funds complicated the task of attributing changes to the Initiative alone.

The evaluation team assessed the intensity of direct services. High-intensity services involved one-on-one contact with providers. (For example, case management prevention counseling, TB screening, and well-child visits would be considered high-intensity.) Medium-intensity services included classes, support groups, and other programs that were delivered to groups of people, but usually on a regular (as opposed to one-time) basis. Finally, low-intensity services included presentations, health fairs and other events, and community outreach.

Using these three levels, the evaluation team assessed service delivery activities over the four HII implementation years. As might be expected, the less resource-intensive medium- and low-intensity services accounted for most of the reported total: 37 percent and 42 percent, respectively. (High-intensity services accounted for 21 percent of the reported total.)

Both high- and low-intensity services were concentrated among a few of the Partnerships. Eighty-seven percent of the high-intensity services were provided by three of the nine Partnerships. Similarly, two of the Partnerships accounted for 72% of the low-intensity services. The number of services declined sharply during Year 05 of the Initiative. Suggesting that some services will continue at significantly lower levels than occurred during the Initiative.

Population Health Measurement
The HII Theory of Action envisioned increases in the use of data, both for internal planning or monitoring and for communicating with the public and policy makers. Partnerships did use existing data (mostly from health departments) and collected some of their own to create community report cards and inform strategic decisions, but overall, the use of data fell short of expectations. In part, this was due to factors that are common across community groups and have been reported elsewhere as well: lack of familiarity with population health data, reliance on informal and anecdotal information, a poor fit between available data (such as polling data) and specific Partnership needs, and a shortage of staff with skills in interpreting data.

Effects Within Partnership Communities

The HII’s effects within communities were substantial. Across the nine communities, 24,450 individual high-intensity services were delivered. These included one-on-one encounters such as case management, health risk screening, and mentoring. (Because of the duplicated count, this total represents the number of services, not the number of individuals served. The number of individuals served is lower, since at least a portion of them are likely to have received services, such as case management, on multiple occasions.) Another 42,949 medium-intensive services – such as health education classes and support groups – were reported during the four-year implementation phase. It is expected that most of these services will continue beyond the HII funding period

Seven of the nine Partnerships plan to continue functioning as collaboratives. One of the two Partnerships that will not continue planned to transfer the work that will be sustained to other groups or councils. In the other that will not continue, having a formal Partnership structure was not considered to be a major factor in accomplishing the objectives. Over
90 percent of Partnership survey respondents reported that they would continue to work with member organizations on similar problems in the future, even if the Partnership per se no longer existed.

In each of the communities, to different degrees, the HII brought a track record of successful cooperation in pursuit of common populations health objectives. These successes – and the trust that is an important byproduct of them – are indispensable assets for future population health initiatives.

Factors Associated With Success

The most critical factors in the Partnerships’ ultimate achievements were community-level factors such as community readiness, organizational commitment, and staff/leadership.
Specific factors associated with higher levels of success included:

• Community Context. In the case of the HII, smaller was better – or at least more manageable. Significant changes in policy, budgets, and data sharing occurred in two smaller counties that attempted county-level changes; these would have been much more difficult to achieve in counties with larger populations.
• Partnership Characteristics. Pre-existing relationships among key stakeholders streamlined the process of negotiating data sharing and other arrangements. Regardless of pre-existing relationships, the inclusion of key stakeholders in the Partnerships was associated with success, as was an ability to work together effectively. The latter usually surfaced as an absence of destructive conflicts among participating agencies.
• Staffing and Leadership. The role of Project Coordinator was critical; Coordinators served as the primary means of communication between the HII and the communities and had direct responsibility for implementing the workplans.

Support Grantee Contributions

In addition to TCWF, which provided overall guidance and funded the HII, five organizations provided support to the Partnerships and the Initiative as a whole:

• California Center for Health Improvement (CCHI) – the creation of this new Center filled the need for an independent health policy organization in the state of California, and its establishment represented a major accomplishment of the HII. Via TCWF funding, CCHI organized and disseminated the findings from its statewide surveys and analyses, promoted policy development in support of community-based health improvement efforts, and provided a credible voice in Sacramento to advocate for policies to promote population health.
• The Field Institute/Louis Harris and Associates –conducted statewide and community-level health surveys.
• Center for Civic Partnerships– built technical capacity, supported a “learning community” atmosphere, and coordinated overall technical assistance.
• Education Programs Associates– provided management support, especially by arranging and conducting Advisory Committee and Management meetings.
• Group Health Community Foundation– developed the overall Initiative evaluation design and provided evaluation support in coordination with local evaluators connected to each Partnership.

Two additional organizations received funding to support the goals of the Initiative and the efforts of the Partnerships.

• RAND Corporation – to develop a prototype composite index for measuring population health in California and determine the content for a new periodic report on this topic.
• The Healthcare Forum – to distill and catalog the lessons learned nationwide from productive efforts to develop community health partnerships.

These two tools, the California Health Index and the catalog of Best Practices in Community Health Partnerships were developed as part of the Initiative but were not utilized.

The Partnerships reported satisfaction with the services of the Support Grantees and indicated they appreciated the support and collegiality that this structure generated. In particular, the efforts of the Center for Civic Partnerships created an environment for mutual learning, provided valuable support to Partnership staff, and helped facilitate the Partnerships’ work by providing “roadmaps” for planning, implementing and sustaining their efforts.


Overall Impact

The HII’s lessons learned were extended to both state and national forums. This occurred not by accident but by design. The focus on policy education as well as the “learning community” philosophy and the commitment of resources to a technical assistance structure that targeted the Initiative as a whole, not just individual Partnerships, made this possible.

The entry of CCHI into the Sacramento policy community constituted a natural experiment for the deliberate use of polls and policy analysis to influence policy. During the HII, CCHI moved in the direction of becoming a permanent contributor to health-related policy-making on the state level by branching into new areas and diversifying its funding base. CCHI became a California non-profit corporation on July 1, 1997 and, reflecting its increasing independence and national focus, CCHI changed its name to the Center for Health Improvement (CHI) in 2001. The establishment of this Center will be an important part of the legacy of the HII.

Also at the state level, HII Partnership efforts helped shape and support legislation strengthening local health-related initiatives. Confidential forms for tracking mental health services and needs assessment procedures for SB 697-related work also were results of Partnership innovations with statewide applicability. Many other Partnership activities have potential to cross over into statewide adoption:

• The Tehama County Health Partnership’s community needs assessment plan was recognized by the California Office of Statewide Planning and Development as a model for their required assessment.
• The Solano Health Improvement Initiative’s Integrated Services Act (AB 866) facilitated the integration of services for children and families.
• The Western Coachella Valley Health Partnership shared successful procedures for enrolling eligible families in the Healthy Families program.
• The Pasadena/Altadena Health Partnership’s Quality of Life Index has been used across the state as a model community indicators project.

Nationally, the population health debate has been (and will continue to be) influenced by HII publications and conferences.

The Theory of Action

Is the Theory of Action a viable model for achieving population health improvements at the community level?

The collaborative governance structure envisioned in the model offered mixed results. Certainly, a broad-based coalition appears to be particularly important in the earlier stages, as groups struggle to reach consensus. However, as the work moves from planning to implementation, smaller subsets may be more efficient.

The latitude offered to Partnerships in choosing interventions was an ingredient in their success and sustainability -- especially in comparison to comparable community health projects that dictate areas of emphasis. However, a corollary would be that the reliance on data envisioned in the model did not come to pass. Although there were many reasons for this – none unique to the Partnerships or the HII – this remains a disappointment, given the investment in this area and the support available through the Grantees. In the final analysis, when data skills were not “native” to the Partnerships, it was difficult to superimpose this important aspect of planning from the outside. In its absence, Partnerships tended to revert to more familiar modes of decision-making. The choice of interventions enjoyed support and in most cases led to improved intermediate outcomes, but the fact remains that these choices might have been different had they been more data driven.


Implications for Future Efforts

Community health initiatives are the ultimate iterative efforts, borrowing – with varying degrees of intent – from the successes of the past, and hoping to avoid its failures.

What can future population health efforts learn from the HII?

Match coalitions to issues. Working effectively is more important than including every possible player – especially after the initial phase. Building a coalition for its own sake may lead to an overemphasis on community process, at the expense of outcomes. Among other things, this means some flexibility is not only to be expected, but also desired.

Take community “readiness” into account. Like individuals, communities vary in their stages of change; some are more ready than others. What does being ready mean? Pre-existing relationships among key stakeholders help, especially if they are committed to a collaborative process and have some consensus about their objectives. Communities that lack these features shouldn’t be left to their own devices, however. Instead of implementation funding, they need support with community development, planning, relationship building, and capacity building – so that they will join the ranks of the ready.

Allow wide latitude in systems change strategies. The HII experience showed that giving communities wide latitude led to greater energy and commitment, compared to other, more restricted initiatives.

Emphasize policy change early and separately from other types of systems change. Policy change is difficult and may be even more challenging at the local level. Local efforts in policy change warrant approaches that are different than those for other types of systems change, and substantial resources are needed to assess policy change skills and provide for technical support to build the capacities needed.

Logic models are not optional. Articulating a common purpose and the theoretical basis for choosing strategies to achieve it is important. It need not be set in stone; as the HII (and many other efforts) demonstrated, no one can predict exactly how events will unfold. Still, the discipline of stating goals and selecting means to these ends is critical.

Be realistic. The prize of improved population health is so appealing – and seems to make so much sense – that it is tempting for both funders and Grantees to promise one another more than can be delivered. This is frustrating for both parties, and discourages future efforts. Small wins count, as the HII demonstrated; an accumulation of modest achievements, over time, may be significant.

Use community participation wisely. It sounds good on paper, but engaging community residents in a meaningful way is much more difficult in practice. Be clear about expectations, prepare to be flexible, and revisit the arrangement frequently to make sure everyone’s needs are being met.

It’s not volunteer work. Community-level systems change can and does draw upon the volunteered time of dedicated people who attend meetings and serve on committees, but at its core, the initiative needs paid technical, managerial, and clerical staff.

Expect three phases: planning, implementation, and planning for sustainability. Funders should consider three distinct phases of funding, each with its own set of expectations. Too often, implementation dominates center stage, leaving planning and sustainability in the wings. But without adequate planning, implementation is less likely to succeed; and part of its success is its longevity. Both deserve resources to bracket the important work of implementation.

Support a large, multi-site initiative with a strong center. Multi-site initiatives like the HII tend to operate with centrifugal force, dispersing each site into its own concerns and agendas. A central, coordinating core – like the Support Grantees in the HII – can rein in these forces and stimulate cross-site learning, communication, and standard expectations.

Offer a menu of flexible technical support. Partnerships vary in their initial needs for technical assistance, and in their needs over time. Flexibility in form, focus, and intensity will enhance the usefulness of technical support to Grantees. To be useful from the start, the technical assistance team should be prepared and operational before the Grantees come on board.

Accompany plans to collect and provide local data with strategies to support its utilization. Emphasis should be placed on assessing both the technical capacities and social readiness to gather and apply local area data. It is recommended that funders work with community Grantees before initiatives or programs begin in order to explore how their various views of data use can be combined to help achieve agreed upon outcomes. In addition, resources for technical assistance and support for capacity building related to use of data should be available.

Balance the evaluation trade-offs. These include balancing the rich, deep understanding of communities that comes from a participatory approach with a more quantitative certainty of measurable outcomes. Another trade-off exists between decentralized efforts that build local capacity and centralized ones that maintain comparable data collection and standards.

The HII affected the nine communities in which it took place, not only during the HII’s five-year time frame, but also into the future as the Partnerships sustain their HII activities or transfer them to others. The HII’s impact was felt across the state of California and indeed around the country, as its innovations, lessons, and limitations are discussed. Its conceptual model – the Theory of Action – moved from theory to action, with many lessons learned along the way. Because of the HII’s deliberate emphasis on creating a “learning community,” these lessons will not be lost.

It is our hope that the evaluation will be part of that contribution, documenting and communicating the HII’s unique topography for those who seek to follow in its footsteps, or perhaps develop their own pathways.

 1 Evans RG and Stoddard GL. Producing Health, Consuming Healthcare. Soc Sci Med, 31:1347-1363, 1990.