Skip to Main Content
NY.gov Portal State Agency Listing Search all of NY.gov
Ann Marie T. Sullivan, M.D., Acting Commissioner
Governor Andrew M. Cuomo

New York State Assisted Outpatient Treatment Program Evaluation

June 30, 2009

View Adobe Acrobat Version | Download Adobe Acrobat Reader

Table of Contents

Suggested citation:

Swartz, MS, Swanson, JW, Steadman, HJ, Robbins, PC and Monahan J. New York State Assisted Outpatient Treatment Program Evaluation. Duke University School of Medicine, Durham, NC, June, 2009

Foreword (Acknowledgements)

We gratefully acknowledge the contribution of numerous individuals in collecting, synthesizing, and reporting the data for this effort.

At the Services Effectiveness Research Program in the Department of Psychiatry and Behavioral Sciences at the Duke University School of Medicine, we acknowledge the considerable efforts of: Richard Van Dorn, Ph.D.; Christine Wilder, M.D.; Lorna Moser, Ph.D.; and Allison Gilbert, Ph.D., M.P.H.

At Policy Research Associates, we wish to express our gratitude to: Karli Keator; Wendy Vogel, M.P.A.; Roumen Vesselinov, Ph.D.; Jody Zabel; Steven Hornsby, L.M.S.W; and Amy Thompson, M.S.W.

We also wish to acknowledge the extensive reviews and critical feedback of the MacArthur Research Network on Mandated Community Treatment, who served as an internal advisory group to the study.

Finally, although the findings of the study are solely our responsibility, we gratefully acknowledge the valuable involvement of our project liaison Steven Huz, Ph.D. and the support and assistance of the New York State Office of Mental Health in completing the report including: Michael F. Hogan, Ph.D., Bruce E. Feig; Susan Shilling, Esq., L.C.S.W.; Lloyd I. Sederer, M.D., Peter Lannon; and Qingxian Chen.

Executive Summary

Introduction

In 1999, New York State created a program authorizing court-ordered treatment in the community for people with severe mental illness at risk of relapse or deterioration absent voluntarily compliance with prescribed treatment. To be eligible for this Assisted Outpatient Treatment (AOT) Program–popularly known as "Kendra’s Law," named after Kendra Webdale, a young woman who was killed by a person with untreated mental illness–individuals must be at least 18 years of age, diagnosed with mental illness and assessed to be unlikely to live safely in the community without supervision. In addition, recipients must have a history of treatment noncompliance that has resulted in (1) psychiatric hospitalization or incarceration at least twice in the past 36 months, or (2) committing serious acts or threats of violence to self or others in the past 48 months. Finally, these individuals must be found, as a result of their mental illness, to be unlikely to voluntarily participate in treatment and to be in need of AOT to prevent deterioration that would likely result in harm to themselves or others. Once an AOT order is finalized by a court, recipients are engaged in a comprehensive community-based treatment plan and extensively monitored for adherence to the plan.

The 2005 reauthorization of the AOT Program required an independent evaluation of its implementation and effectiveness, specifically addressing several areas of investigation. The New York State Office of Mental Health issued a competitive Request for Proposals, and the contract for the evaluation was awarded to the Services Effectiveness Research Program in the Department of Psychiatry and Behavioral Sciences at Duke University Medical Center with a subcontract to the Policy Research Associates, Inc. of Delmar, NY, and with additional support from the John D. and Catherine T. MacArthur Foundation Research Network on Mandated Community Treatment.

The contract requires study and reporting on the following areas:

Description of AOT Program and regional variations: Are there regional and cultural differences across the state in AOT programs and their implementation?

Service engagement: What is the level of service engagement of recipients of mental health services during AOT?

Recipient outcomes: What are the outcomes for people with mental illness who are mandated into AOT versus those who receive voluntary enhanced outpatient services?

Recipient perceptions of AOT: What are the opinions of a representative sample of AOT recipients regarding their experiences with AOT?

Service engagement and outcomes after AOT ends: What is the level of service engagement of recipients of mental health services post-AOT?

The impact of AOT on New York’s public mental health system: What is the impact of AOT programs on the availability of resources for individuals with mental illness and perceived barriers to care?

To address these areas of investigation we studied existing records from several extensive data sources, described in the appendices, including: AOT Program administrative data, New York State Office of Mental Health client profile surveys, hospital admissions records, case manager reports on AOT recipients and Assertive Community Treatment recipients, Medicaid claims, arrest records, U.S. Census, and Mental Health Needs Estimation Project data. In addition, we conducted statewide in-person interviews among key stakeholders to gain insight into the operation of the AOT Program and interviewed service recipients in six counties to assess attitudes about treatment, treatment experiences, and treatment outcomes.

Findings

Description of the New York AOT Program and Its Regional Variations

The introduction of New York’s AOT Program was accompanied by a significant infusion of new service dollars and currently features more comprehensive implementation, infrastructure and oversight of the AOT process than any other comparable program in the United States. It is, therefore, a critical test of how a comprehensively implemented and well-funded program of assisted outpatient treatment can perform. However, because New York’s program design is unique, these evaluation findings may not generalize to other states, especially where new service dollars are not available. This report addresses whether AOT can be effective and under what circumstances, not whether it will always be effective wherever or however implemented.

As designed, AOT can be used to prevent relapse or deterioration before hospitalization is needed. However, in nearly three-quarters of all cases, it is actually used as a discharge planning tool for hospitalized patients. Thus, AOT is largely used as a transition plan to improve the effectiveness of treatment following a hospitalization and as a method to reduce hospital recidivism.

Most of New York State’s experience with AOT originates in the New York City region where approximately 70% of all AOT cases are found. AOT was systematically implemented citywide in New York City with well-delineated city-wide policies and procedures. In the remainder of the state, AOT was implemented and utilized at the discretion of counties. In some counties AOT has been rarely used; in several it has not been used at all.

Based on key stakeholder and recipient interviews and on AOT Program data, we found considerable variability in how AOT is implemented across the state, but strong uniformity in how it is implemented in New York City. One important difference among regions was the use of enhanced voluntary service (EVS) agreements (sometimes referred to as "enhanced services") in lieu of a formal AOT court order. (Note that the term "enhanced voluntary services" or ‘EVS’ was developed to describe these agreements and is not an official designation.) Under a voluntary agreement, the recipient signs a statement that he or she will adhere to a prescribed community treatment plan. In the New York City region, voluntary agreements are usually implemented following a period of AOT when a recipient is judged to be ready to transition from an AOT order to voluntary treatment; we refer to this as the "AOT First" model. In other counties, largely outside of New York City, voluntary agreements are more frequently used as trial periods before initiating a formal AOT order; we refer to this pattern as the "EVS First" model. If the trial period proves unsuccessful, an AOT proceeding is then initiated. Across the state, AOT First is used far more frequently than EVS First since the majority of AOT orders occur in New York City.

Racial Disparities in AOT: Are They Real?

An April 2005 report on statewide demographic data from the New York Lawyers for the Public Interest found that African Americans were over represented in the AOT Program. Whether this over-representation is discriminatory rests, in part, on whether AOT is generally seen as beneficial or detrimental to recipients and whether AOT is viewed as a positive mechanism to reduce involuntary hospitalization and improve access to community treatment for an under-served population, or as a program that merely subjects an already-disadvantaged group to a further loss of civil liberties.

We find that the overrepresentation of African Americans in the AOT Program is a function of African Americans’ higher likelihood of being poor, higher likelihood of being uninsured, higher likelihood of being treated by the public mental health system (rather than by private mental health professionals), and higher likelihood of having a history of psychiatric hospitalization. The underlying reasons for these differences in the status of African Americans are beyond the scope of this report. We find no evidence that the AOT Program is disproportionately selecting African Americans for court orders, nor is there evidence of a disproportionate effect on other minority populations. Our interviews with key stakeholders across the state corroborate these findings.

Service Engagement

A key goal of the AOT Program is to motivate consumers to actively engage in treatment during and after their involvement with the program. We find that during the first six months on AOT, service engagement was comparable to service engagement of voluntary patients not on AOT. After 12 months or more on AOT, service engagement increased such that AOT recipients were judged to be more engaged than voluntary patients. This suggests that after 12 months or more, when combined with intensive services, AOT increases service engagement compared to voluntary treatment alone.

Recipient Outcomes

We find consistent evidence that during AOT there is a substantial reduction in the number of psychiatric hospitalizations and in days in the hospital if a person is hospitalized. We also find moderately strong evidence from lifetime arrest records of AOT and EVS recipients from the NYS Division of Criminal Justice Services that AOT reduces the likelihood of being arrested. We find substantial increases in receipt of intensive case management services during AOT. We also find that AOT recipients are far more likely to consistently receive psychotropic medications appropriate to their psychiatric conditions. Case managers of AOT recipients also report subjective improvements in many areas of personal functioning, such as managing appointments, medications, and self-care tasks.

Recipient Perceptions of AOT

Participants were assessed on scales measuring a wide range of AOT-related attitudes and treatment experiences, including: their understanding of AOT; whether they believe it beneficial or harmful; whether they find it stigmatizing; whether it affects their sense of autonomy or empowerment; satisfaction with treatment, perceived coercion related to treatment; perceived pressures to engage in treatment; whether it increases perceived barriers to treatment; and how it affects their sense of being treated fairly.

On the whole, AOT recipients and non-AOT recipients report remarkably similar attitudes and treatment experiences. That is, despite being under a court order to participate in treatment, current AOT recipients feel neither more positive nor more negative about their mental health treatment experiences than comparable individuals who are not under AOT. This suggests that positive and negative attitudes about treatment during AOT are more strongly influenced by other experiences with mental illness and treatment than by recent experiences with AOT itself.

Service Utilization and Outcomes After AOT Ends

We examined whether selected gains made during AOT are sustained over time by examining two key outcomes that improved during AOT: reduced rates of hospitalization and increased receipt of psychotropic medications appropriate to the individual’s diagnosis. We find that sustained improvement after AOT ends varies according to the length of time the recipient spends under the AOT order. If AOT is discontinued after six months, these decreased rates of hospitalization and improved receipt of psychotropic medications are sustained only if recipients continue to receive intensive case management services. However, if AOT continues for longer than 6 months, reduced rates of hospitalization and improved receipt of medications are sustained whether or not intensive case management services are continued after AOT is discontinued. Thus, it appears that improvements are more likely to be sustained if AOT continues for longer than 6 months.

Impact of AOT on New York’s Public Mental Health System

It is unclear whether resources have been diverted away from other adults with severe mental illness as a consequence of AOT implementation. We examined the impact of AOT Programs on the availability of resources for all adult individuals with severe mental illness by focusing on access to high intensity case management services.

The implementation of AOT was accompanied by a large increase in funding for mental health services, which over time increased the availability of intensive services for all service recipients, even those who never got AOT. In the first several years of the AOT Program, between 1999 and 2003, preference for intensive case management services was given to AOT cases, a finding corroborated by our key stakeholder interviews. This meant that in the first several years of the AOT Program, non-AOT recipients were less likely to receive intensive case management services than their AOT counterparts, especially outside of New York City. This may have been because the treatment capacity was greater in New York City, and thus it was able to absorb a greater volume of new AOT cases with less impact on other service recipients with severe mental illness.

After 2003 new AOT orders leveled off in the state and then declined. The new treatment capacity that accompanied the implementation of AOT was apparently then available to other individuals who needed these services, irrespective of AOT status. Thus, following the initial ramp-up of the AOT Programs throughout the state, intensive community-based services increased for individuals on AOT and those not on AOT alike. Because the new service capacity created during the implementation of the AOT Program is now fully utilized, competition for services in the near future may intensify, with unknown effects on AOT relative to non-AOT recipients. Because the implementation of the AOT Program in New York was accompanied by an infusion of new services, it is impossible to generalize these findings to states where services do not simultaneously increase.

Summary

We find that New York State’s AOT Program improves a range of important outcomes for its recipients, apparently without feared negative consequences to recipients. The increased services available under AOT clearly improve recipient outcomes, however, the AOT court order, itself, and its monitoring do appear to offer additional benefits in improving outcomes. It is also important to recognize that the AOT order exerts a critical effect on service providers stimulating their efforts to prioritize care for AOT recipients.

Available data allow only a limited assessment of whether voluntary agreements are effective alternatives to initiating or continuing AOT. There are relatively few voluntary agreements and they typically occur in counties that use the "EVS First" model. However, we found some evidence that AOT recipients are at lower risk of arrest than their counterparts in enhanced voluntary services. We also found evidence in the case manager data that receiving AOT combined with ACT services substantially lowers risk of hospitalization compared to receiving ACT alone.

Recipients appear to fare better during and after AOT if the AOT order lasts for six months or more. Once AOT recipients leave the program, improvements are more likely sustained among those who continue to receive intensive treatment services or received longer periods of AOT.

Perceptions of the AOT Program, experiences of stigma, coercion, and treatment satisfaction appear to be largely unaffected by participation in the program and are likely more strongly shaped by other experiences with mental illness and treatment.

In its early years, the AOT Program did appear to reduce access to services for non-AOT recipients. However, in recent years the reduction in new AOT cases has attenuated this effect. Lack of continued growth of new service dollars will likely increase competition for access to services once again.

Introduction

The New York State Legislature in 1999 enacted the state’s involuntary outpatient commitment statute, named "Kendra’s Law" in memory of a young woman killed by a man with untreated mental illness.1 Beyond passing a new law to address a perceived public safety need, the Legislature funded a new statewide program – Assisted Outpatient Treatment (AOT) – designed to ensure that people with severe mental illness receive the array of services they need in the community.

Kendra’s Law was seen as a legislative model for involuntary outpatient commitment in the United States. The intent of the statute was not simply to authorize court-ordered community treatment but to also provide the resources and oversight necessary for a viable, less restrictive alternative to involuntary hospitalization. The goal was to provide a definitive remedy for the costly "revolving door syndrome."

Whether Kendra’s Law and the AOT Program have succeeded in these terms – and at what cost to the liberty of AOT recipients and the public resources they consume – is a matter of ongoing debate.

Kendra’s Law was initially authorized for a period of five years with continuation made contingent on an internal evaluation of the AOT Program. The Mental Health Commissioner and New York State Office of Mental Health (OMH) submitted reports on the implementation and status of the AOT Program. The Interim Report (2003) and Final Report (2005) highlighted encouraging evidence of AOT’s effectiveness,2 and the Legislature re authorized AOT for a second five-year period.

The 2005 reauthorization of Kendra’s Law required an independent evaluation of the implementation and effectiveness of the AOT Program, specifically addressing several areas of investigation. Beginning in 2006, the Commissioner was also to issue both an annual fiscal and descriptive AOT Program report.

OMH issued a competitive Request for Proposal, and the contract was awarded to the Services Effectiveness Research Program (SERP) in the Department of Psychiatry and Behavioral Sciences at Duke University Medical Center (DUMC) with a subcontract to Policy Research Associates, Inc. (PRA) of Delmar, NY. This project also received funding from the John D. and Catherine T. MacArthur Foundation Research Network on Mandated Community Treatment. This evaluation team is led by Principal Investigators Marvin Swartz, M.D., and Jeffrey Swanson, Ph.D., of DUMC and Henry Steadman, Ph.D., and Pamela Clark Robbins of PRA.

The evaluation team analyzed AOT administrative records and clinical services data spanning nearly a decade (1999 – 2007). The team also collected and analyzed new data from key informant interviews throughout the state, and from structured interviews with a new sample of AOT and voluntary service recipients in six selected counties. This report presents the findings from these analyses, following an overview of the AOT Program and study methods.

Overview of Evaluation Report

The New York State Legislature’s authorization of Kendra’s Law and the accompanying AOT Program is contingent on the evaluation of its effectiveness. The following specific research questions were posed in the evaluation solicitation:

  1. Description of AOT Program. The process by which AOT is implemented across the state is characterized and described, and fundamental questions of the fairness of the AOT program are investigated. Specifically, are there regional and cultural differences across the state in AOT programs and their implementation?
  2. Service engagement. One of the primary aims of AOT is to encourage recipients to engage in community treatment so as to avoid treatment in the more restrictive inpatient setting. What is the level of service engagement of consumers in mental health services during AOT, and does the duration of AOT influence engagement?
  3. Recipient outcomes. The effectiveness of AOT is examined. What are the outcomes for people with mental illness who receive enhanced outpatient services versus those who are in AOT?
  4. Recipient perceptions of AOT. What are the opinions of a representative sample of AOT recipients regarding their experiences with AOT?
  5. Post-AOT service utilization and outcomes. The purpose of AOT, and the optimal duration of AOT to achieve a given purpose, likely varies across recipients. AOT may be used to link some people into treatment that they will, once stabilized, accept voluntarily. For others, AOT is an ongoing tool of leverage intended to maintain treatment adherence in persons who, due to the nature of their illness, are otherwise unwilling or unable to participate consistently in mental health services. What is the level of service utilization of AOT recipients following the termination of the court order, and how does utilization vary as a function of AOT duration and recipient characteristics?
  6. Impact of AOT on service system. An impressive amount of resources have been allocated to support the AOT Program, and individuals under AOT are assured access to services. It is uncertain whether, as a consequence of AOT implementation, resources have been diverted from other adults with severe mental illness in need of treatment. What is the impact of AOT Programs on the availability of resources for individuals with mental illness, and what are the perceived barriers to care?

Summary

An independent evaluation of New York State’s AOT Program was a stipulation of the 2005 reauthorization of Kendra’s Law. This report presents the findings of the evaluation, which examined nearly a decade of administrative and service data as well as newly collected interview data. The report is organized around the six areas of investigation that were outlined in the evaluation request.

Chapter 1: What Does the Implementation of the Assisted Outpatient Treatment (AOT) Program Look Like, and Are There Regional and Cultural Differences Across the State in AOT Programs and Their Implementation?

In this chapter we describe the volume and distribution of AOT orders across New York State and discuss the variations in AOT across local programs, regions, and target populations. In addition to quantitative analysis of administrative data collected by OMH, we present qualitative data gathered in our interviews with key stakeholders. A major finding of our research is that implementation and operations of the AOT Program are not uniform across the state; two discrete programs emerge in different parts of the state. In addition, some counties do not have an AOT Program at all.

Implementation of the AOT Program

A total of 8,752 initial AOT orders and 5,684 renewals were granted from the inception of the AOT program in 1999 through midyear 2007.3 AOT recipients represent a small proportion of the total OMH adult service population. For example, in 2005, of the 138,602 OMH adult service recipients with severe mental illness, only 2,420 (1.7 %) were AOT recipients. And yet, despite their small numbers, persons under AOT receive disproportionate attention, given their serious needs, high cost to the service system, and the public's concern about the target population for Kendra's Law.

Under Kendra's Law, a local AOT Program was to be created to monitor and oversee AOT implementation for each county and New York City. In our interviews with key stakeholders, we found that some counties have an AOT Program but never use it. Instead, these counties use their local Single Point of Access (SPOA) program to coordinate services for high need populations. We also found that some counties have no AOT Program at all. Among the common reasons cited for not utilizing AOT are the lack of infrastructure to support court orders in smaller counties; the belief that mental health problems should not be dealt with by the legal system; and the position that court orders are for the benefit of providers rather than clients. These quotes from key informant administrators in counties that do not routinely use AOT demonstrate various local views on the use of AOT:

AOT is a reactionary approach to a high publicity incident.

The law is implemented–we just don't let it get to court.

I don't think anybody would benefit from AOT. If I thought it would make a difference, I would do it. I've gotten close.

All counties receive AOT Program funding and service dollars, regardless of whether they actually have an AOT Program in place. Although some counties do not use the money for AOT recipients, they may use these funds to serve high-risk clients in other ways. Our interviews of county officials indicate that AOT Program funding is very important to the local service system but yield no specific information about how county-level AOT dollars are spent.

Exhibit 1.1 displays the distribution of 2005 AOT orders by county, with counties shaded according to the size of the population of individuals with serious mental illness (SMI). It is evident that a higher proportion of individuals with SMI are located in New York City and the surrounding counties, along with the upstate metropolitan areas. Likewise, there is a higher density of AOT recipients in these areas. However, there are some areas where the number of AOT orders is inconsistent with the SMI population density, which suggests other sources of variation. Regional differences emerged when we examined how AOT is applied, the duration of the court order, and the origination of petitions.

Exhibit 1.1 AOT Order Density and Estimated Serious Mental Illness Population
Order Density and Estimated Serious Mental Illness Population
Source: Combined analysis of data from OMH, the U.S. Census, and estimates from epidemiological surveys.

Regional variation in AOT

Which Comes First, Court-Mandated AOT or Enhanced Voluntary Services (EVS)?

For AOT to be ordered, individual candidates who are petitioned for AOT must meet legal eligibility criteria, and AOT must be deemed the least restrictive alternative. However, for some petitioned individuals, an alternative plan may be drafted in which the individual agrees to receive Enhanced Voluntary Services (EVS); in most cases, this plan includes being assigned to intensive case management (ICM) or assertive community treatment (ACT). Although voluntary, the agreement may have conditions of treatment participation designed to avoid a court order for AOT.

The process for initiating voluntary agreements and drafting enhanced service plans are not statutory elements of Kendra's Law. However, they are used by many county AOT Programs either prior to initiating AOT or after a period of AOT. Some counties instituted formal procedures for voluntary agreements (i.e., legal documents), and other counties use less formal written or verbal agreements. While counties do not report to OMH the individual or identifying data on persons served under these voluntary agreements, the number of voluntary agreements has been acknowledged and reported in earlier program reports.

EVS First is primarily used in upstate counties and is thought to satisfy the least restrictive alternative requirement.

The extent to which EVS agreements were used varied across counties, and varied in the timing of the order. Some county AOT Programs seek the court mandate before moving towards a voluntary agreement after some success under the mandate (i.e., AOT First); these programs use voluntary agreements as a path out of AOT. Other county AOT Programs first elicit voluntary participation in enhanced services before resorting to the court mandate. They seek a court order only if the individual does not comply with the treatment specified in the voluntary agreement (EVS First). A psychiatrist from an upstate county discussed this approach to providing EVS First in the following way:

We don't do it like downstate or like OMH wants. We use the voluntary order first. We don't approach it in an adversarial way.

But a psychiatrist working with a downstate AOT program took a much different view.

If you meet criteria it would be foolish to do less [than a court order].

We found notable regional differences in the use of these two distinct models of AOT. EVS First is primarily used in upstate counties and is thought to satisfy the least restrictive alternative requirement (inpatient psychiatric hospitalization being the most restrictive). AOT First is the predominant model downstate where court orders are usually given prior to discharge from an inpatient setting. Liability concerns are generally cited as the rationale for this model; that is, the hospital is more comfortable discharging the patient knowing that an AOT court order is in place. However, limited service slots and housing availability are also influential in the decision to use this model as court-ordered individuals are given priority for these scarce resources.

AOT First is the predominant model downstate where court orders are usually given prior to discharge from an inpatient setting.

We examined patterns of AOT among service recipients interviewed in six selected counties. As seen in Exhibit 1.2, the EVS First pattern was extremely rare in the downstate counties of New York, Queens, and Nassau. In contrast, this pattern was quite frequent in two of the upstate counties (Erie and Monroe). The majority of current AOT recipients started the program under a court order (AOT First). However, while 40% of recipients in the upstate AOT Programs began with a voluntary agreement (EVS First), only one individual in the downstate programs followed this pattern. Similarly, of those interviewed who were currently under an EVS agreement, 89% located upstate had started out on the voluntary agreement, whereas only 7% of downstate EVS participants had started on a voluntary agreement.

Exhibit 1.2. Program type by Assisted Outpatient Treatment (AOT) and Enhanced Voluntary Services (EVS) in upstate and downstate counties
Program type by Assisted Outpatient Treatment and Enhanced Voluntary Services in upstate and downstate counties

How Long Does An Order Last? Statewide, 32% of AOT orders last 6 months or less, while 68% last longer than 6 months. A small minority of orders–14%–are kept in place for longer than 30 months. Regions vary to some extent in the duration of AOT recipients' orders, for example, with the Central Region tending to terminate orders sooner (48% end at 6 months), and Long Island extending orders longer (81% last longer than 6 months).

Where Do AOT Petitions Originate?

The vast majority (84%) of petitions are filed while the subject of the petition (the AOT respondent) is an inpatient at a psychiatric hospital. A small proportion of petitions (13%) are filed while the respondent is in the community, and an even smaller proportion (3%) are filed by correctional facilities. The proportion of AOT orders originating in hospitals, prisons/jails, or communities has varied over time, with a slight increase in the proportion of orders originating in jail or prison since the beginning of the program. As shown in Exhibit 1.3, there is notable regional variation in the source of petitions. The Central Region had proportionately more AOT orders originating in jails or prisons (21%), while the Western Region had proportionately more orders originating in the community (43%).

Interviews with key informants provided several reasons for variation in the source of AOT petitions–several driving factors may vary by region and from case to case. One important factor is regional differences in rates of involuntary hospitalization and incarceration due to bed capacity and location of facilities in large metropolitan areas. In some regions, a greater proportion of the AOT target population may be hospitalized, and thus, more accessible to the AOT initiation process. Second, the cost of petitioning for AOT can be prohibitive for smaller hospitals and individuals (i.e., family members). Third, inpatient doctors in some facilities tend to use AOT more routinely as part of a discharge plan and as a form of risk management, although several key informants doubted AOT was an effective risk management strategy. For example:

AOT has some carrot–not the teeth.

The fact is this statute has no teeth and adversarial situations don't work with no teeth. 9.60 (AOT) is gums but no teeth.

Fourth, family members may not wish to initiate the petition, fearing that it will disrupt their relationship with their loved one, and prefer instead to wait until a petition follows from a hospitalization. Fifth, petitions from jail can be problematic given the uncertainty of inmate release times.

Exhibit 1.3. Origin of AOT orders, 2002-2007, for various regions of New York.
Origin of Assisted Outpatient Treatment orders, 2002-2007

Other Regional Variations in AOT Implementation

We found many other differences in how the AOT Program is implemented. Some counties integrated AOT with Single Point of Access (SPOA) to facilitate the process and provide highly coordinated services. One key informant noted:

Central point of access for care co-ordination is key.

Other counties established formal administrative procedures, including standardized forms and reporting policies to expedite administration of the law. In the New York City boroughs, there is a formal pick-up procedure when individuals are noncompliant with the court order, making it easier to execute a Removal Order for someone under AOT who becomes noncompliant with services. Other variations in AOT Program implementation stem from the court proceedings themselves such as the continuity and interest of the presiding judge and the attitudes of the Mental Hygiene Legal Service (MHLS) attorneys.

First, counties vary in whether they have a single judge appointed to preside over all AOT hearings or whether this position is rotated among several judges. Judges' interest in mental health law also varies. Second, MHLS attorneys' attitudes regarding the AOT Program varied across regions.

In some MHLS departments, lawyers tended to view AOT as the least restrictive alternative to hospitalization and as a gateway to receiving needed community services. These lawyers were likely to foster a collaborative, and whenever possible, a non-adversarial relationship with the hospitals' clinical staff – working together in using AOT as a tool to obtain early discharge for patients who had been involuntarily committed to longer stays in psychiatric facilities.

In other MHLS departments, lawyers were likely to view their legal role as appropriately adversarial with respect to the AOT petitioners, representing their client's own wishes rather than “best interest” per se defined by clinicians' or family members.

These divergent views from MHLS attorneys are illustrative:

We see AOT as a way for some clients to get what they need. They are severely mentally ill and need good follow-up treatment in the community. This is a way for them to get out of the hospital much sooner.

The job of MHLS is to just give clients the facts–their rights. We are not best interest advocates. If they want a hearing, so be it.

A judge in one area where MHLS attorneys were known to hold the latter view noted how their advocacy was a major strength of the program.

Trends in the Use of AOT

At the beginning of the AOT Program in 1999, the number of AOT investigations accelerated rapidly and remained much higher than the number of resulting petitions and court orders (see Exhibit 1.4).4 However, the number of investigations began declining after 2001 as more eligible referrals were made to the AOT Program. Over time, program administrators and service providers developed a keener ability to identify candidates likely to meet the legal eligibility requirements for AOT.

Exhibit 1.4. AOT investigations, petitions, hearings, and orders during the initial years of the program
investigations, petitions, hearings, and orders during the initial years of the program

If an investigation does reach court, it is very likely to result in an AOT order; out of 9,307 AOT hearings held, 8,752 (94%) resulted in a court order. Thus, it appears that ineligible AOT requests are largely screened out before investigations reach court. Exhibit 1.5 shows a steep start-up curve in AOT hearings, orders, and renewals from 1999 to 2002. After that, however, the volume of AOT cases leveled off as counties refined policy and procedures. Also, as more people were placed on AOT over time, and as more AOT orders were renewed, the remaining pool of AOT-eligible individuals shrank. Meanwhile, AOT renewal orders increased and reached a peak in 2005 with 1,236 renewal orders.

Exhibit 1.5 AOT hearings, orders, and renewals, 1999 to 2006
Assisted Outpatient Treatment hearings, orders, and renewals, 1999 to 2006

The volume of AOT orders and renewals is inevitably affected by local service capacity, which is more limited in rural areas. For example, many AOT treatment plans specify ACT as the appropriate service modality for the AOT recipient. However, smaller programs may not have ACT teams or may have no available caseload openings on existing teams; ACT caseloads are capped at 48 or 68 clients. Because of limited service capacity, scarce resources are selectively used for individuals most in need, often defined as those at highest risk for violent behavior. Most of the county program personnel we interviewed indicated that their AOT Programs had reached capacity. They were now faced with deciding which individuals would be most appropriately served in the limited number of slots made available through attrition or “graduation” from AOT. Comments from an AOT Director and ICM provider illustrate the problem:

There have been (AOT) capacity issues for some time now.

AOT really does work–but sometimes people get stuck in AOT longer than need to be–almost punishing them.

Some programs indicated that when individuals meet AOT criteria but are not deemed “high risk,” they do not receive AOT until another recipient graduates from AOT.

According to aggregate data provided by the counties, the rate per 100,000 of EVS agreements initially was much higher than the rate of AOT orders, as shown in Exhibit 1.6. However, AOT orders eventually outpaced EVS orders –especially in downstate counties–so that by 2006 the rate of AOT orders was much higher than that for EVS orders.

One important difference between AOT and EVS that emerged in our key informant interviews is that EVS recipients are very rarely renewed in their voluntary agreements. It is highly unlikely, particularly downstate, that a client would remain on a voluntary agreement for longer than six months. In the AOT First model, AOT court-ordered individuals are moved to a voluntary agreement when they transition from AOT, maintaining the same level of enhanced service for 6 months, and then are entirely moved off the formal AOT Program (though they may continue to receive case management and other services as needed). This then makes some room for new AOT court orders.

Exhibit 1.6 Trends in rates of Assisted Outpatient Treatment (AOT) orders and Enhanced Voluntary Services (EVS) agreements across New York counties
Trends in rates of Assisted Outpatient Treatment  orders and Enhanced Voluntary Services agreements across New York counties

Enhanced Services: ACT and ICM

Kendra's Law requires that the written treatment plan include case management services, typically ICM or ACT services, while the individual is under the court order. These enhanced services are also provided to individuals on EVS agreements. ACT teams and intensive case management services are the cornerstones of the AOT treatment plan and have received the majority of the AOT ancillary funding statewide. A look at the distribution of treatment type among all individuals who received AOT orders between 1999 and mid-year 2007 (N=14,127) shows 20% of cases received ACT and 74% received ICM. The remaining 6% may have also received ICM within the context of a blended case management team5 or may have received the less intensive supportive case management.

Demographic Profile of AOT Recipients

A demographic description of all AOT recipients6 is provided in Exhibit 1.7.7 The most typical AOT recipient is a 38-year-old single male in New York City who is diagnosed with schizophrenia and living with others or in a supervised residence.

Nearly half (47%) of AOT recipients also have a co-occurring substance use disorder. Equal proportions (34%) of AOT recipients are white and black, meaning that African Americans are over represented in AOT from a population perspective. This racial disproportionately is investigated below.

Exhibit 1.7. AOT recipient characteristics

Age in years All AOT orders from 1999 to January 22, 2009 (n=7,368)
Mean 38 years
Gender All AOT orders from 1999 to January 22, 2009 (n=7,368)
Male 67%
Race/ethnicity All AOT orders from 1999 to January 22, 2009 (n=7,368)
White 34%
Black 34%
Hispanic 30%
Other 2%
Marital status All AOT orders from 1999 to January 22, 2009 (n=7,368)
Single 76%
Divorced 8%
Married 16%
Living situation All AOT orders from 1999 to January 22, 2009 (n=7,368)
Alone 14%
With others 38%
Supervised setting 36%
Psychiatric diagnosis All AOT orders from 1999 to January 22, 2009 (n=7,368)
Schizophrenia 73%
Bipolar 18%
Co-occurring
substance use
47%
Region All AOT orders from 1999 to January 22, 2009 (n=7,368)
Central 2%
New York City 71%
Hudson 10%
Long Island 11%
Western 4%

Source: AOT Evaluation database*

Racial Disparities in AOT: Are They Real?

Since 1999 about 34% of AOT recipients have been African Americans who make up only 17% of the state's population, while 34% of the people on AOT have been whites, who make up 61% of the population.8 Thus, overall, African Americans are more likely than whites to receive AOT. However, candidates for AOT are largely drawn from a population where blacks are over represented: psychiatric patients with multiple involuntary hospitalizations in public facilities. The answer to the question of whether AOT is being applied fairly must take into account all of the available data.9

To answer this question, we estimated and compared rates of AOT for black and white individuals using several alternative denominators. These denominators can be thought of as a series of concentric circles encompassing relevant target populations, from the broadest to the narrowest definitions of who is “at risk” for receiving AOT.10 We then conducted a multivariable, longitudinal analysis of the association between race and AOT at the county level to see whether the relationship may be accounted for by other underlying factors that co-vary with race and AOT. Details regarding methodology and statistical analysis can be found in Appendix B.

Exhibit 1.8 displays the results graphically for six counties and the state total.11 This analysis shows that in the total population, AOT affects African Americans 3 to 8 times more frequently than whites – about 5 times more frequently on average statewide.

Exhibit 1.8 AOT racial parity indices in 6 representative New York counties and statewide: Black to white ratios of AOT case rates (2003 period-prevalence) using selected alternative denominators
racial parity indices in 6 representative New York counties and statewide
* Source: AOT Evaluation database, Tracking for AOT Cases and Treatments, US Census, Mental Health Needs Assessment Project, OMH Patient Characteristics Survey, OMH hospital admissions data.

However, the analysis also shows that these differences are dependent on context. When the most relevant target populations for AOT are considered, this ratio moves closer to 1 leaving no appreciable racial disparities in selection to AOT. The ratio is reduced substantially when the denominators used are the numbers of black and white individuals who are estimated to have SMI. These county SMI estimates incorporate poverty status, which is statistically associated both with SMI and with African American racial background.12

This ratio declines even further when public-sector service recipients are considered as the denominator. Finally, there is no difference in black and white rates of AOT among those who have been involuntarily hospitalized at least twice. Parallel analyses for Hispanics and other minority populations show this same pattern and no appreciable racial disparities are evident in selection of these groups for AOT.

We find no evidence suggesting racial bias in the application of AOT to individuals.

This analysis implies that the AOT rate is influenced by a number of “upstream” social and systemic variables such as poverty that may correlate with race. However, we find no evidence suggesting racial bias in the application of AOT to individuals. Defining the target population as public mental health system clients with multiple hospitalizations, the rate of application of AOT to white, black and other minority recipients approaches parity.

Summary

AOT court orders rapidly increased since the program's inception but appear to be leveling off in recent years. This trend may be due to filled capacity in AOT Programs and lack of new program funding. Across the state, the highest number of AOT orders tends to be found in areas with a greater concentration of adults with severe mental illness; New York City and surrounding areas represent the majority of AOT orders in the state. We found regional differences across several elements of AOT implementation and administration, including upstate New York's more prominent use of EVS First model before resorting to court-mandated AOT. Downstate New York programs use AOT First almost exclusively, only rarely using voluntary agreements as a transition from AOT. Although a large proportion of AOT recipients are black, there is no apparent racial bias in the program when target population factors are taken into account.

Chapter 2. Engagement in the Assisted Outpatient Treatment Program

Introduction

A key goal of the AOT Program is to motivate service recipients to actively engage in their treatment during and after their involvement with AOT. “Engagement” here means motivation to actively participate in regular community-based treatment and services. As part of their assessment of AOT recipients every six months, case managers are asked to rate the recipients’ level of engagement in services on a scale ranging from “not at all engaged in services” to “independently and appropriately uses services.” For the purposes of our data analysis, recipients were considered to be positively engaged in services if rated either “good–able to partner and can use resources independently” or “excellent–independently and appropriately uses services.”

Findings

Case Manager Ratings of Service Engagement

At entry into the AOT Program, case managers rated 33% of AOT recipients to have positive service engagement, as defined above. Rates of engagement modestly improved over time on AOT: by six months in the AOT Program, 45% were rated as having positive engagement. Similarly, among recipients with 12 months of AOT or more, 46% were rated as positively engaged. However, unless we compare AOT recipients to similarly situated individuals who did not receive AOT, it is difficult to assess whether the court order was a key ingredient in promoting engagement or whether comparable gains in engagement would have occurred over time with voluntary treatment alone.

Comparing Case Manager Ratings of Service Engagement for AOT and Assertive Community Treatment (ACT) Recipients

Most consumers in the AOT Program receive one of two forms of case management; ICM (74% of AOT recipients) or ACT (20% of recipients). ACT is an evidence-based treatment delivery model designed to provide intensive community-based services to persons with severe mental illness who are difficult to serve in conventional outpatient mental health programs. While ACT is regarded by many experts as an appropriate treatment alternative to the use of AOT, it is also used in conjunction with AOT for some recipients. Hence, in New York the OMH case managers systematically collect comparable outcome data for all AOT and ACT recipients but not for voluntary ICM recipients.

It is thus possible to compare levels of engagement among consumers who receive ACT alone, AOT plus ACT, or AOT plus ICM. This comparison allows us to examine whether AOT adds any benefit in engaging recipients in services when compared to voluntary treatment with ACT alone. Unfortunately, there is no voluntary ICM group for comparison in this data source and thus, no rigorous method to compare voluntary versus court-ordered ICM. In addition, recipients who receive ACT versus ICM and the case managers who assess them may not be comparable, even when statistical adjustments to improve their comparability are attempted. Comparisons should not be made between ACT and ICM outcomes in these analyses.

We compared levels of engagement across these groups using multivariable analyses with statistical controls for potential underlying differences that might have been independently associated with higher engagement. Exhibit 2.1 displays the results of this analysis. For recipients receiving six months or more of treatment, AOT with ACT offers no additional benefit in service engagement compared to ACT alone (37% versus 32% respectively). Recipients receiving AOT with ICM demonstrate higher levels of engagement (49%) compared to ACT alone.

Exhibit 2.1. Adjusted* percent with "good" or "excellent" service engagement by treatment and legal status. Results contain all observations for 6 or more months of treatment
observations for 6 or more months of treatment
*Adjusted probability estimates were generated from repeated measures regression models controlled for time, region, race, age, sex, diagnosis, baseline hospitalizations, baseline service engagement, education level, marital status, substance use, medication adherence, and GAF. Statistical models used multiple imputation of missing data.
** Odds are less than 1 in 1000 that the difference between AOT + ICM and the other groups would occur by chance. Sources = AOT Evaluation database and Child and Adolescent Integrated Reporting System

However, AOT of longer duration is associated with modestly higher rates of engagement. Exhibit 2.2 shows these results for the subgroup of recipients with AOT lasting at least 12 months. Over this longer time period, a higher proportion of recipients were rated with positive engagement in the AOT with ACT group, and in the AOT with ICM group, than in the ACT-alone group (55%, 56% and 43% respectively).

Exhibit 2.2. Adjusted* percent with "good" or "excellent" service engagement by treatment and legal status. Results contain observations for 12 or more months of treatment only
observations for 12 or more months of treatment only
*Adjusted probability estimates were generated from repeated measures regression models controlled for time, region, race, age, sex, diagnosis, baseline hospitalizations, baseline service engagement, education level, marital status, substance use, medication adherence, and GAF. Statistical models used multiple imputation of missing data.
** Odds are less than 1 in 1000 that the difference between ACT and the other groups would occur by chance. Data Source = AOT Evaluation database and Child and Adolescent Integrated Reporting System

Summary

Over all, when short-term AOT was included in the analysis, we find that service engagement was comparable for AOT and non-AOT recipients on ACT teams. However, after 12 months or more on AOT, a higher proportion of AOT recipients in an ACT program were judged to be positively engaged than voluntary recipients of ACT services. This suggests that longer-term AOT combined with intensive treatment increases service engagement compared to voluntary treatment alone.

Chapter 3. Recipient Outcomes

Introduction

Comparing AOT to EVS

An important question in evaluating AOT is whether court-ordered treatment with enhanced services is more effective than EVS. Some candidates for AOT are given the opportunity to avoid a court order by signing a voluntary agreement to participate in EVS. Do AOT recipients experience better outcomes than their counterparts being served under EVS? As detailed in Chapter 1, we found that EVS agreements were relatively uncommon. Nonetheless, where possible, we have drawn comparisons between outcomes under EVS versus AOT court orders. Arrest is one important outcome where a direct comparison was possible, due to the availability of lifetime arrest records for individuals enrolled in the study from six counties.

As noted previously, most recipients in the AOT Program receive one of two forms of case management: Intensive Case Management (ICM) or Assertive Community Treatment (ACT). A natural comparison can be made between AOT and ACT, in particular, because ACT is considered a less coercive alternative to AOT for persons with severe mental illness who need intensive outpatient services. Because case managers collect data using a common form for AOT and ACT recipients (but not for voluntary ICM recipients), it is possible to compare case manager assessments of outcomes among three groups of service recipients: (1) ACT alone, (2) AOT plus ACT, and (3) AOT plus ICM. These comparisons allow us to address whether AOT is more effective than a voluntary intensive treatment program (ACT) for similarly situated individuals with severe mental illness. Moreover, these data allow comparison of AOT implementation in conjunction between two different models of intensive case coordination: AOT with ACT, compared to AOT with ICM. Unfortunately there is no voluntary ICM group for comparison in this data source and thus no rigorous method to compare voluntary versus court-ordered ICM. In addition, recipients who receive ACT versus ICM and the case managers who assess them may not be comparable, even when statistical adjustments to improve their comparability are attempted. Comparisons should not be made between ACT and ICM outcomes in these analyses.

We first present an analysis of selected recipient outcomes using data from 211 interviews with individuals in six selected counties. For self-reported outcomes such as violence, suicidality, and homelessness, we compare individuals currently receiving AOT to those who never had AOT or had it longer ago than six months. For arrest, we use official records of lifetime arrests to conduct a more powerful longitudinal analysis, comparing AOT recipients to EVS recipients in the six counties. For these arrest analysis, we used data from 181 individuals and 9,229 person-month observations.

Second, using case manager ratings for 5,634 AOT recipients, we also present descriptive data comparing pre-AOT and during-AOT periods regarding services utilization, functioning, and selected negative outcome events.

Using case manager ratings for AOT and ACT recipients, we conducted a more detailed statistical analysis of selected outcomes in which we compared three service conditions: voluntary ACT alone, ACT plus AOT, and AOT plus ICM. Analyses of the effects after 6 months used data from 3,073 individuals with 7,611 person-period observations. Analyses of effects after 12 months used data from 2,325 individuals with 5,581 person-period observations. As explained in Appendix B, we also used multiple imputation techniques to handle missing data.

Third, using Medicaid claims data and other OMH records, we assess important outcomes including hospitalization, receipt of medications, and receipt of case management. These analyses examine outcomes for AOT recipients prior to and during their AOT experience. The final Medicaid analyses were conducted on a sample of 2,839 AOT recipients with 84,089 person-months of data. (The method of analysis is explained in Appendix B.)

Findings

Client Outcomes From Direct Recipient Interviews in Six Counties

Structured interviews regarding AOT and related treatment experiences, attitudes, and outcomes were conducted in a sample of 211 persons with severe mental illness in six counties: Albany, Erie, Monroe, Nassau, New York, and Queens. A total of 277 interviews were conducted with three groups of these service recipients: 115 individuals currently on AOT; 134 comparable individuals who had never received AOT or received it more than six months ago; and 28 individuals who had completed a period of AOT six months ago. About one third of the sample entered the study in one group, thus later became eligible for the second group; therefore, they were interviewed more than once.

The main descriptive results of the six-county interview study are presented in Chapter 4. Here we examine some selected outcomes for the 115 recipients who were currently on AOT, compared to the 134 recipients with no recent AOT.

Exhibit 3.1 shows that current AOT recipients and those with no recent AOT report comparable rates of violence, suicidality, homelessness, involuntary commitment, and being picked up by police for transport to mental health treatment. However, a slightly lower percentage of current AOT recipients report these negative outcome events.

Exhibit 3.1. Six-county study sample recipient characteristics

  No current or recent AOT (n=134) Current AOT (n=115)
Outcome events (past six months) N % N %
Violent behavior 21 (15.7) 12 (10.4)
Suicidal thoughts or attempts 22 (16.4) 17 (14.8)
Homelessness 13 (9.7) 6 (5.2)
Involuntary commitment 54 (43.2) 46 (41.4)
Mental health pick-up/removal 25 (18.7) 16 (13.9)

* As defined by the MacArthur Community Violence Interview. See Appendix B for description of instruments.
Source: 6-county interviews.

Arrest Outcomes Comparing Current AOT and Current EVS Recipients In Six Counties

Lifetime arrest records were obtained for 181 individuals who either received AOT or EVS in six counties. The AOT Programs identified the individuals receiving EVS through their programs and specified the periods during which they were receiving EVS. Recipients of EVS are persons who would have qualified for AOT orders but signed voluntary agreements to receive intensive services as an alternative to a court order.

Using EVS and AOT tracking information combined with arrest records, we examined longitudinally whether people had been arrested in a given month, by period: pre AOT/pre-EVS, current AOT, and current EVS (results for post-AOT and post-EVS are presented in Chapter 5--Exhibit 5.5). A total of 9,225 person-month observations were available for the multivariable time-series analysis.

Exhibit 3.2 summarizes the results for current AOT and current EVS compared to pre-AOT/EVS. Moving into the current AOT period from the pre-AOT/pre-EVS period, the likelihood of arrest in any given month is reduced from 3.7 to 1.9 percent per month. This result is statistically significant; it would have occurred less than 5 times in 100 by chance alone. The effect for current EVS was not statistically significant, although there was a clear trend toward reduction of arrests during EVS.

Exhibit 3.2. Adjusted* percent arrested in month by current receipt of AOT and EVS
percent arrested in month by current receipt of Assisted Outpatient Treatment and Enhanced Voluntary Services
*Adjusted arrest rate estimates were produced using multivariable time-series regression analysis, controlling for time, region, age, sex, race, education, and diagnosis. Months spent in hospital are excluded from analysis.
Source: 6-county interviews and Division of Criminal Justice Services.

Client Outcomes from Case Manager Reports

Treatment Planning: Care Coordination, Medication Management, Substance Abuse Services, and Housing Support Services

Case managers were asked to report all services that were explicitly identified in AOT recipients’ treatment plans before and after initiation of AOT orders. Because AOT often started a new case management relationship, many case managers had limited knowledge of recipients’ status and service history before AOT began; thus, some of the information in the case managers’ baseline reports should be qualified as uncertain. With that caveat, however, services included in recipients’ treatment plans appear to have increased after initiation of AOT (Exhibit 3.3). In particular, care coordination and psychiatric medication management were included in treatment plans for virtually all AOT recipients after six months (99% and 96%, respectively.) About half of AOT treatment plans addressed substance abuse and housing support services.

Exhibit 3.3. Service components of treatment plan prior to and during AOT order, 1999 – 2007, n = 5634 recipients
Service components of treatment plan prior to and during Assisted Outpatient Treatment order, 1999 – 2007
Percents are unadjusted and based on case manager report at 6-month intervals.
Source: Child and Adolescent Integrated Reporting System and AOT Evaluation database

Recipient Functioning: Adverse Events and Behaviors

Case managers reported a slight decrease in adverse events after six or more months of AOT (Exhibit 3.4). Self-harm decreased from 9 % to 4 % and harm to others decreased from 7 to 4%. The proportion of individuals reporting at least one night of homelessness also decreased from 12% to 7-8%. Because of the low incidence of these events in general and the difficulty case managers may have had in estimating the frequency of events prior to AOT, conclusions are limited from these data regarding the effect of AOT in reducing adverse events.

There was a more notable decrease in case managers’ appraisal of recipients’ non-adherence to medications – from 47% to 33% after six months of AOT. However, non-adherence increased again to 43% in the group of individuals who had been on AOT 12 months or more. This could reflect a retention bias in which individuals who were less adherent were more likely to have their AOT order renewed. Indeed, in the Medicaid data analysis, we found that individuals with lower medication possession rates and higher hospitalization rates during their initial AOT period were significantly more likely to have their AOT orders renewed.

Exhibit 3.4. Adverse events or behaviors prior to and during AOT order, 1999 – 2007, n = 5634 recipients
Adverse events or behaviors prior to and during Assisted Outpatient Treatment order, 1999 – 2007
Percents are unadjusted and based on case manager report at 6 month intervals. Non-adherence was defined as a score of 1 or 2 on a 4 point scale, where 1=rarely or never takes medication as prescribed and 4=takes medication exactly as prescribed. Experiencing homelessness was defined as being homeless for at least one night in the past 6 months. Source: Child and Adolescent Integrated Reporting System and AOT Evaluation database.

Recipient Functioning: Life Management Skills

Recipients showed modest improvements in a variety of life skills after at least six months of AOT, as assessed by case managers (Exhibit 3.5). Case managers reported as much as a 10% increase in the proportion of individuals able to manage their medications and personal finances without substantial help. Case managers likewise reported as much as a 10% increase in individuals who typically engaged in pro-social behaviors such as effectively handling conflict, engaging in social activities, and asking for help when needed.

Exhibit 3.5. Recipient functioning prior to and during AOT order, 1999 – 2007, n = 5634 recipients
Recipient functioning prior to and during Assisted Outpatient Treatment order
Percents are unadjusted and based on case manager report at 6 month intervals. All ratings are based on a 5 point scale. Source: Child and Adolescent Integrated Reporting System and AOT Evaluation database.

Comparing AOT to Voluntary Intensive Treatment Alternatives

Psychiatric Hospitalization Rates from Case Manager Reports

As mentioned in Chapter 2, individuals under AOT typically receive one of two forms of case management: ICM (74% of AOT recipients) or ACT (20% of recipients). Case managers systematically collect comparable outcome data for all AOT and ACT recipients, but not for voluntary ICM recipients. Thus, it is possible to compare reported outcomes for recipients of ACT alone, AOT plus ACT, or AOT plus ICM.

To examine hospitalization outcomes with these case manager data, we conducted a multivariable repeated-measures analysis, controlling for a range of underlying variables that could have affected hospitalization independently of AOT. As shown in Exhibit 3.6 below, hospitalizations were reduced by about one half among individuals who received 12 months or more of AOT (combined with either ACT or ICM), compared to their baseline hospitalization rate. Also, the chance of hospital admission was substantially reduced – from about 58% to 36% – among these AOT recipients with either ACT or ICM, compared to those receiving only ACT without AOT.

Thus, whether we compare AOT recipients at two points in time (baseline and 12 months), or compare AOT-plus-ACT recipients to those receiving ACT alone,

we find highly statistically significant differences in the likelihood of hospital readmission; the odds are less than 1 in 1000 that these results would occur by chance. A limitation of these results is that hospitalizations here are reported by case managers and are not independently verified. (Results using hospital records will be reported later in this chapter.)

Exhibit 3.6. Adjusted* percent hospitalized per 6 months, by AOT status at baseline and after 12 months of treatment
 percent hospitalized per 6 months
*Adjusted probability estimates were generated from repeated measures regression models controlled for time, region, race, age, sex, diagnosis, baseline hospitalizations, baseline arrests, living situation, education level, presence of dependent children, marital status, substance use, service engagement, medication adherence, and Global Assessment of Functioning (GAF). Statistical models used multiple imputation of missing data. Source: Child and Adolescent Integrated Reporting System and AOT Evaluation database.

Substance Abuse Rates From Case Manager Reports

For individuals receiving six months or more of treatment, substance use rates were much lower for individuals receiving AOT plus ICM (29%) compared to those receiving AOT plus ACT (59%) or voluntary ACT alone (61%). These results should be viewed with caution; differences in identifying and treating substance use in ACT and ICM programs could account for these differences.

Exhibit 3.7. Adjusted* percent using substances in last 6 months, by treatment and legal status. Results contain all observations for 6 or more months of treatment.
percent using substances in last 6 months
*Adjusted probability estimates were generated from repeated measures regression models controlled for time, region, race, age, sex, diagnosis, baseline hospitalizations, baseline service engagement, education level, marital status, substance use, medication adherence, and GAF. Statistical models used multiple imputation of missing data.
**Odds are less than 1 in 100 that the difference between the AOT + ICM and the AOT + ACT groups, or between the AOT + ICM and the ACT groups, is due to chance.
Source: Child and Adolescent Integrated Reporting System and AOT Evaluation database.

For recipients who underwent 12 months or more of AOT, substance use outcomes were very similar to those reported for six months or more of treatment.

Outcomes Analysis Using Medicaid and OMH Records

Using Medicaid and OMH records, we are able to assess three important outcomes for AOT recipients: psychiatric hospital admissions, receipt of psychotropic medications, and receipt of case management services. These analyses compared outcomes for AOT recipients before and during their AOT experience, and for short-term AOT (one to six months) and longer-term AOT (12 months or more.) Findings are based on repeated measures multivariable analyses, with statistical controls for potential underlying differences between individuals in these different groups. The Medicaid analyses were conducted on a sample of 2,839 AOT recipients with 84,089 person-months of data. (The method of analysis is explained in the Appendix B.)

Psychiatric Hospitalization Rates and Days in Hospital from Medicaid and OMH Records

Compared to the pre-AOT monthly hospitalization rate of 14%, the probability of hospital admission was reduced to 11% per month during the first six months of AOT and to 9% during the 7-12 month period of AOT. Exhibit 3.8 displays these results. While this decrease in hospital utilization

might appear modest, it would represent substantial reductions in hospitalizations for AOT recipients statewide. Regarding statistical significance, these differences between the pre-AOT state and each of the other two periods of AOT experience would have occurred less than one time in 1000 by chance alone.

Exhibit 3.8. Adjusted* percent with psychiatric inpatient treatment in month, by AOT status
percent with psychiatric inpatient treatment in month
*Adjusted probability estimates were generated from repeated measures regression models controlled for time, region, race, age, sex, diagnosis, and co-insurance status. Models were also weighted for propensity to initially receive AOT and to receive more than 6 months of AOT.
Source: Medicaid claims and OMH admissions database

Exhibit 3.9 presents comparable results for the average number of days hospitalized per six month period. During the period prior to AOT, recipients on average experienced 18 days of hospitalization over the course of six months, excluding the hospitalization when AOT was initiated. In contrast, during AOT, recipients spent 11 days in the hospital during the first six months of AOT and 10 days during the 7-12 month period of AOT. While this decrease in hospital days per six month period might appear modest, it would again represent substantial reductions in hospital days statewide.

Exhibit 3.9. Adjusted* average inpatient days during any 6 month period, by AOT status
average inpatient days during any 6 month period
*Adjusted mean estimates were generated from repeated measures regression models controlled for time, region, race, age, sex, diagnosis, and co-insurance status. Models were also weighted for propensity to initially receive AOT and to receive more than 6 months of AOT.
Source: Medicaid claims and AOT Evaluation database.

Receipt of Psychotropic Medications from Medicaid and OMH Records

We also used Medicaid claims data to examine changes in receipt of psychotropic medication under AOT. Medication receipt was defined as having filled a prescription for a medication appropriate to the diagnosed psychiatric condition and having a sufficient supply during 80% or more of the days in a given month. As shown in Exhibit 3.10, medication receipt (so defined) increased from 35% per month prior to AOT, to 44% during the first six months of AOT, to 50% during the 7-12 month period.

Receipt of Case Management from Medicaid and OMH Records

Exhibit 3.10. Adjusted percent* with at least 80% medication possession in month by AOT status
percent* with at least 80% medication possession in month
*Adjusted probability estimates were generated from repeated measures regression models controlled for time, region, race, age, sex, diagnosis, and co-insurance status. Models were also weighted for propensity to initially receive AOT and to receive more than 6 months of AOT.
Source: Medicaid claims and AOT Evaluation database.

Finally, we used Medicaid data to examine monthly receipt of intensive case management services (ACT and ICM) during AOT. These results are shown in Exhibit 3.11. Monthly receipt of ACT services increased from 1% in the pre-AOT period to 8% in the first six months of AOT and 10% during the 7-12 month period. Receipt of ACT or ICM services increased from 11% in the pre-AOT period to 28% in the first six months and 33% during the 7-12 month period. Receipt of any case management services increased from 18% in the pre-AOT period to 44% in the first six months and 53% during the 7-12 month period.

Exhibit 3.11 Adjusted* percent receiving case management services in month by AOT status
percent receiving case management services in month
*Adjusted probability estimates were generated from repeated measures regression models controlled for time, region, race, age, sex, diagnosis, and co-insurance status. Models were also weighted for propensity to initially receive AOT and to receive more than 6 months of AOT.
Source: Medicaid claims and AOT Evaluation database.

Summary

The outcomes detailed heretofore should be understood and interpreted in the context of other findings in the remaining chapters. For example, elsewhere we examine how recipients fare when AOT ends and how AOT affects the treatment experiences of recipients. These and other results are synthesized in the Executive Summary and the final chapter of the report.

During AOT there is a substantial reduction in the number of psychiatric hospitalizations and in days spent in the hospital if a person is hospitalized. We also find moderately strong evidence from lifetime arrest records of AOT and Enhanced Voluntary Services recipients that AOT reduces the likelihood of being arrested. In addition, we find substantial increases in receipt of intensive case management services during AOT. Compared to their experiences prior to AOT recipients are far more likely to receive intensive forms of case management under AOT. We also find that AOT recipients are far more likely to consistently receive psychotropic medications appropriate to their psychiatric conditions compared to their experiences pre-AOT. Case Managers of AOT recipients also report subjective improvements in many areas of personal functioning, such as managing appointments medications and self-care tasks.

Chapter 4. Recipient Perceptions of the Assisted Outpatient Treatment Program

This chapter presents survey results regarding recipients’ perceptions of AOT, treatment experiences, and related attitudes in a sample of 211 persons with severe mental illness in six counties including Albany, Erie, Monroe, Nassau, New York, and Queens. Structured interviews were conducted with three groups of service recipients:

Sample selection in the six counties was designed to obtain information about the AOT experience throughout the state. However, the sample does not reflect the actual regional distribution of AOT cases. Exhibit 4.1 presents descriptive characteristics of these sample groups.

Exhibit 4.1. Six-county interviews sample characteristics

  No current or recent past AOT
(n=134)
Current AOT
(n=115)
Recent past AOT
(n=28)
Age
Age in years at index hospitalization mean: 38.8
n
(sd: 11.3)
%
mean: 39.4
N
(sd: 11.6)
%
mean: 41.6
n
(sd: 13.2)
%
Sex
Male 80 50.7 76 66.1 18 64.3
Female 54 49.3 39 33.9 10 35.7
Race/Ethnicity
White 67 50.0 52 45.2 16 57.1
Black 59 44.0 50 43.5 11 39.3
Hispanic 17 12.7 10 8.7 0 0.0
Asian 6 4.5 2 1.74 1 3.6
Other 9 6.7 18 15.6 2 7.1
Education
Completed high school 100 74.6 90 78.9 23 82.1
Employment
Worked for pay 25 18.7 20 17.4 6 21.4
Clinical characteristics
Schizophrenia 103 77.4 84 73.0 21 75.0
Affective disorder 31 22.6 31 27.0 7 25.0
Any substance abuse 36 26.9 32 27.8 5 17.9
County
Albany 13 9.7 13 11.3 4 14.3
New York 22 16.4 31 27.0 5 17.9
Nassau 19 14.2 32 27.8 11 39.3
Queens 11 8.2 12 10.4 1 3.6
Erie 43 32.1 8 7.0 1 3.6
Monroe 26 19.4 19 16.5 6 21.4

Source: Interviews with AOT program recipients in six selected counties

The three groups were similar in their demographic and clinical characteristics. The majority were male; about half were white;13 about three quarters had completed high school; and less than one in five was employed even part time (any paid work). About three-quarters of the sample had a diagnosis of schizophrenia or other psychotic disorder; about one quarter had a co-occurring substance abuse problem, i.e., reported symptoms of alcohol use disorder and/or were using illicit drugs. (Case manager assessments of substance abuse comorbidity in this population are higher – around 40%; see Chapter 3.)

Exhibit 4.2 displays mean item scores across a range of attitudinal scales for the three subsamples. These scales are meant to capture subjective perceptions of AOT and comparable treatment experiences for people with SMI in New York. The scales are grouped into areas: (1) AOT understanding, perceived benefits, and stigma; (2) personal autonomy, treatment relationships, and satisfaction; and (3) coercion, pressures, barriers to treatment, and procedural justice.14

Exhibit 4.2. AOT recipients’ attitudes and experience by AOT period

  No current or recent AOT (n=134)1 Current AOT (n=115) Recent past AOT (n=28)
  Item Mean St. Dev. Item Mean St. Dev. Item Mean St. Dev.
AOT beliefs and attitudes
AOT understanding 0.89 0.29 0.97 0.05 0.97 0.05
AOT stigma 0.19 0.40 0.26 0.44 0.07 0.26
AOT perceived effectiveness 0.60 0.46 0.81 0.31 0.90 0.25
Treatment autonomy, relationships and satisfaction
Empowerment 3.70 0.43 3.66 0.43 3.70 0.31
Working alliance 4.06 0.79 3.99 0.79 4.16 0.55
Treatment satisfaction 3.90 0.74 3.74 0.84 4.17 0.47
Attitudes about taking medication 0.74 0.21 0.72 0.20 0.78 0.17
Life satisfaction 4.81 1.51 4.84 1.71 4.57 1.32
Coercion, pressures, and barriers
Coercion 2.76 0.96 2.97 1.15 2.76 0.90
General pressures to adhere to treatment 0.30 0.15 0.31 0.16 0.22 0.12
Pressures-warnings 0.35 0.18 0.36 0.20 0.28 0.17
Pressures-sanctions 0.15 0.19 0.16 0.18 0.05 0.09
Pressures-med oversight 0.48 0.39 0.48 0.38 0.36 0.36
Pressures-commitment 0.27 0.33 0.28 0.33 0.09 0.24
Perceived effectiveness/fairness of pressures 2.68 0.56 2.79 0.57 2.40 0.34
Procedural justice 1.80 0.65 1.96 0.32 2.02 0.29
Mandate-related treatment barriers 0.28 0.31 0.28 0.33 0.17 0.27
Nonmandate-related treatment barriers 0.42 0.34 0.33 0.32 0.30 0.33
Fear of commitment 0.37 0.48 0.37 0.49 0.36 0.49

1 group n=110 for AOT understanding and procedural justice

Source: Interviews with AOT program recipients in six selected counties

AOT Beliefs and Attitudes

Knowledge and understanding of AOT provisions–what AOT legally requires–was measured with 12 true/false items.  Mean item scores ranged from 0 (none correct) to 1 (all correct), with higher scores indicating more accurate understanding of legal requirements under AOT.  A sample item included: "When they have an AOT order, people are required to go to mental health treatment appointments that are part of the treatment plan [true]." Among current and recent past AOT participants, mean item scores approached 1 for this 12-item scale, indicating a high percentage of correct answers and a good understanding of AOT across groups.  The mean score was also high (0.89) for participants with no recent AOT experience. 

Perceived AOT stigma was measured with a single yes/no question:  "When people are under AOT, do you think that most other people think less of them?" Fewer than 1 in 4 participants answered this question affirmatively.  However, endorsement was slightly higher among current AOT participants than non-AOT or post-AOT participants. 

AOT perceived effectiveness was measured with three yes/no items.  Item means ranged from 0 (all no) to 1 (all yes), with higher scores indicating greater agreement that AOT was effective in helping people keep scheduled outpatient treatment appointments, take prescribed medication, and remain in the community without being hospitalized.  Findings for this scale showed a mixed response for non-AOT respondents and a higher, more positive response among current AOT recipients compared to those with no recent AOT (mean=0.60 vs. 0.81.).  The highest mean item score was found for the recent AOT graduates (mean=0.90).

Personal Empowerment, Treatment Relationships, and Satisfaction

Empowerment was measured with a 15-item standardized scale, with mean item scores ranging from 1 to 5, from "strongly disagree" to "strongly agree."  A sample item included:  "When I make plans, I am almost certain to make them work."  All three sample subgroups scored in the positive middle range–above neutral but not agreeing strongly with these items.  There were no differences between samples (mean=3.7 in each group).

Working alliance is a construct that captures the quality and strength of the therapeutic relationship between the service recipient and case manager.  The Working Alliance Inventory (short version) was administered as an 8-item standardized instrument, with responses ranging from 1 to 5, from "strongly disagree" to "strongly agree."  A sample item included:  "[Case manager] and I are working toward mutually agreed upon goals."      All three groups scored on average about 4 out of a possible 5 on these items, indicating positive perceived working alliance, with little or no differences between groups. 

Treatment satisfaction was measured using a 9-item standardized scale, with mean item scores ranging from 1 to 5 from "strongly disagree" to "strongly agree."  A sample item included: "I liked the services that I have received in the past six months."  Findings on this scale indicate similarly positive levels of satisfaction with treatment across all three subsamples–close to 4 out of 5 on average. 

Attitudes and experience with taking medication for mental health problems was captured with the Drug Attitudes Inventory (DAI-modified), a standardized 18-item scale (the original had 10 items), with mean item scores ranging from 0 (no items endorsed) to 1 (all items endorsed).  Higher scores indicated more positive attitudes towards medication, higher perceived effectiveness of medication, and fewer problems with side effects.  A sample item included: "By staying on medications, I can prevent getting sick." Scores on the DAI across the three subsamples averaged about 0.75 on the 0 to 1 mean item scale, indicating similar and mostly positive attitudes about taking medication for mental health problems.  

Life satisfaction or subjective quality of life was measured with a single item–"How do you feel about your life as a whole?"–on a 7-point scale from "terrible=1" to "delighted=7."  Mean scores on this item varied little across all three subsamples–from 4.6 to 4.8–indicating moderately positive perceived quality of life in each group.

Coercion, Pressures, Barriers to Treatment, and Procedural Justice

Perceived coercion was measured with a 5-item standardized scale, with mean item scores ranging on a scale of 1 to 5 from "strongly agree" to "strongly disagree." A sample item included:  "I felt free to do what I wanted about getting treatment."  Strong disagreement with this and similar items indicated greater perceived coercion.  Findings on this scale showed moderate levels of coercion–just under 4 out of a possible 5–across all three subsamples.

General pressures to adhere to treatment were measured with a 33-item standardized scale of yes/no questions.  Mean item scores ranged from 0 (all no) to 1 (all yes), with 1 indicating more pressure being exerted.  The scale also included 4 subscales: warnings, sanctions, medication oversight, and commitment pressure.  Sample items included:  "Did anyone tell you that you may lose your housing if you don’t follow your treatment plan"? [warnings]; "Did anyone report on your behavior to a pro­bation/parole officer?" [sanctions]; "Did anyone watch you take your medication to make sure you took it regularly?" [oversight]; and "Did anyone try to commit you to a hospital against your will?" [commitment pressure].  The pattern of responses on these subscales again showed no differences between subsamples.  Across all groups, similarly low levels of pressures were reported, with the highest being for "oversight" pressure.

Current AOT participants apparently did not experience more adverse subjective conditions around mental health treatment than comparable individuals who were not under AOT.

Perceived effectiveness and fairness of pressures to adhere to treatment was measured with a 9-item standardized scale, with mean item scores ranging on a scale of 1 to 5 from "strongly agree" to "strongly disagree," lower scores indicating greater perceived effectiveness and fairness regarding pressures to adhere to treatment.  A sample items included:  "Overall, the pressures or things people have done were for my own good."  Findings on this scale across all three groups were in the positive-neutral range, between 2.4 and 2.8 on the 1 to 5 mean item scale, indicating similarly mixed views within each group about whether pressures to adhere to treatment were effective and fair.  

Procedural justice was measured on a six-item standardized scale.  Mean item scores varied on a scale from 1 to 3, from "not at all" to "somewhat" to "definitely."  A sample item included:  "When you received the AOT court order, did they treat you respectfully?"  Results for this scale showed some difference between groups, with current AOT recipients reporting higher perceived procedural justice than their counterparts who had not recently experienced AOT (1.96 vs. 1.80).

Barriers to treatment were measured with six true/false items.  Item mean scores could vary from 0 (all false–no barriers reported) to 1 (all true–all barriers reported); higher scores, closer to 1, indicated more barriers.   The items were divided into mandate-related and non-mandate related barriers.  Sample items included:  "Did you delay getting help because you think that if you went for treatment you might be forced to take some medicine or treatment that you don’t want?" [mandate-related barrier]; and "Did you delay getting help because you… [non mandate-related barrier such as believing not in need].  Current AOT recipients were less likely to report non-mandate related barriers than those with no recent AOT (mean=0.33 vs. 0.42). 

Participants were also asked a single yes/no question about fear of involuntary commitment and treatment seeking.  "Has fear of being involuntarily committed ever caused you to avoid treatment for mental health?" The mean score could vary from 0 (if everyone had answered "no") to 1 (if everyone had answered "yes").  The mean score for this variable in Exhibit 4.2 is the same as the percentage of the sample who answered "yes" – approximately one third of participants in each subsample. 

Summary

Face-to-face, structured interviews were conducted with 277 persons with severe mental illness in six counties in New York:  115 current AOT recipients, 134 persons with no recent AOT, and 28 persons with recent AOT experience.  Participants were assessed on standardized scales measuring a wide range of AOT-related attitudes and treatment experiences in three general areas:  (1) AOT understanding, perceived effectiveness, and stigma; (2) personal empowerment, treatment relationships, and satisfaction; and (3) coercion, pressures, barriers to treatment, and procedural justice.For the most part, no differences were found among groups; findings were remarkably similar irrespective of AOT status.  However, current AOT participants reported lower levels of non-mandate related treatment barriers and greater perceived effectiveness of AOT, as summarized in Exhibit 4.3.

Previous studies of coercion have found that when recipients have had an opportunity to voice their concerns about involuntary treatment and have their concerns heard, it attenuates their feeling of being coerced and this may bear on these findings wherein recipients have their "day in court." The overall importance of the six-county survey findings is that, despite being under a court order for treatment, current AOT participants apparently did not experience more adverse subjective conditions around mental health treatment than comparable individuals who were not under AOT.

Exhibit 4.3. Perceived AOT effectiveness and treatment barriers by AOT Status
effectiveness and treatment barriers by Assisted Outpatient Treatment Status
Source: 6-county interviews.

Chapter 5. Service Utilization and Outcomes After Assisted Outpatient Treatment Ends

Introduction

An additional important area of investigation is what happens to AOT recipients after they leave the program. Past research provides little information about post-AOT outcomes. If consumers do make gains during AOT, are these gains sustained over time?

To address this question we used two sources of data: direct interviews with post-AOT recipients and Medicaid claims data. Our sample of post-AOT recipients is quite small (N=28), and therefore, these results are descriptive in nature. In contrast, results reported from Medicaid data contain a large number of recipients and allow a rigorous evaluation of key post-AOT outcomes.

Findings

Post-AOT Outcomes: Direct Recipient Interviews

As presented in Chapter 4, direct in-person interviews regarding AOT and related treatment experiences and attitudes were conducted with 115 individuals currently on AOT; 134 comparable individuals who had never received AOT, or had it longer ago than 12 months; and 28 individuals who had completed a period of AOT within the previous twelve months. Here we compare the interview results for the 28 recipients who had recently completed a period of AOT to the other two sample groups.

We find that recipients report few differences in treatment experiences and attitudes in the post-AOT period. Compared to recipients currently on AOT (see Exhibit 4.2), there were few differences in a wide range of AOT-related attitudes and treatment experiences after AOT ended. These post-AOT recipients resembled those on AOT in their understanding of AOT, their perceptions of AOT effectiveness, and their awareness of social stigma associated with AOT. Post-AOT recipients also did not differ in their sense of personal empowerment, satisfaction with treatment, perceived coercion related to treatment, or perceived informal pressures to engage in treatment. Finally, they did not differ in their reported barriers to treatment and their sense of being fairly treated.

In the post-AOT period recipients reported appointment and medication adherence comparable to adherence during AOT. Community functioning and level of symptomatology were relatively similar as well. Other areas unchanged in the post-AOT period were self-report of substance abuse, violence, suicidality, homelessness, and arrests.

Post-AOT Psychiatric Hospitalization Rates and Days in Hospital from Medicaid and OMH Records

Using Medicaid and OMH records, we assessed recipient outcomes such as hospitalization, receipt of medications, and utilization of case management. These analyses compare outcomes for AOT recipients before, during, and after their AOT experience. We also compare post-AOT outcomes for recipients who had short-term AOT (one to six months) versus longer-term AOT (7 to 12 months), and between those who continued to receive Assertive Community Treatment (ACT) or intensive case management (ICM) services after their AOT order ended and with those who did not continue to receive these services.

Findings in Chapter 3 detailed reductions in rates of psychiatric hospitalization during AOT. Here we explore whether these reductions in rates of psychiatric hospitalization persist once AOT is terminated. Findings are based on multivariable analyses with statistical controls for potential underlying differences among recipients across these different groups.

For individuals who receive AOT for a period of six months or less, we find that the likelihood of subsequent hospitalization depends on whether intensive outpatient services utilization continues after the AOT order ends. If the recipient continues to receive post-AOT intensive case coordination services in the form of ACT or ICM, the predicted probability of hospitalization within any post-AOT month is substantially reduced relative to the pre-AOT period (7% vs. 11% per month). However, if ACT or ICM is also discontinued when AOT ends, the predicted probability of post-AOT hospitalization rises to 10% per month, which is comparable to the pre-AOT hospitalization rate. These results are displayed in Exhibit 5.1.

Exhibit 5.1. Adjusted* percent with psychiatric inpatient treatment in any given month over short-term** AOT course
 percent with psychiatric inpatient treatment in any given month
*Adjusted probability estimates were generated from repeated measures regression models controlled for time, region, race, age, sex, diagnosis, and co-insurance status.
** Short-term AOT defined as receiving a total of 6 months or less.
Source: Medicaid claims and AOT Evaluation database.

In contrast, we find (as seen in Exhibit 5.2 below) that if the initial period of AOT is longer than six months, reduction in hospitalization in the post-AOT period is sustained whether or not the recipient continues to receive intensive treatment in the form of ACT or ICM. For these longer-term AOT recipients, the predicted probability of post-AOT hospitalization remains at substantially reduced level relative to the pre-AOT period, even without continued ACT or ICM services utilization (7% compared to 11%).

Exhibit 5.2. Adjusted* percent with psychiatric inpatient treatment in month over long-term* AOT course
percent with psychiatric inpatient treatment in month
*Adjusted probability estimates were generated from repeated measures regression models controlled for time, region, race, age, sex, diagnosis, and co-insurance status.
** Long-term AOT defined as receiving AOT for > 6 months. Source: Medicaid claims and AOT Evaluation database.

Post-AOT Receipt of Case Management Services from Medicaid and OMH Records

Receipt of intensive case management in the form of ACT or ICM rose from 5% prior to AOT to 45% during AOT. These rates declined modestly when AOT was discontinued: after six months or more of AOT, 28% of recipients continued to receive ACT or ICM. However, after 12 months or more of AOT, 35% of recipients continued to receive ACT or ICM services.

Post-AOT Receipt of Psychotropic Medications from Medicaid and OMH Records

Findings in Chapter 3 also detail improvement in rates of receipt of appropriate psychotropic medications during AOT. Do these improved rates of receipt of psychiatric medications persist once AOT is terminated?

For individuals who receive AOT for a period of six months or less, we find that the likelihood of receiving medications consistent with their diagnosis depends on whether intensive outpatient services utilization is also continued after the AOT order ends. If the recipient continues to receive post-AOT intensive case coordination services in the form of ACT or ICM, the predicted probability of appropriate medication possession within any post-AOT month remains improved relative to the pre-AOT period (45% vs. 37%). However, if ACT or ICM is also discontinued when AOT ends, the predicted probability of post-AOT medication possession declines to 33%, which is comparable to the pre-AOT level. These results are displayed in Exhibit 5.3.

Exhibit 5.3. Adjusted* percent with at least 80% medication possession in any given month over short-term AOT course
percent with at least 80% medication possession in any given month
*Adjusted probability estimates were generated from repeated measures regression models controlled for time, region, race, age, sex, diagnosis, and co-insurance status. Source: Medicaid claims and AOT Evaluation database.

In contrast, we find (as seen in Exhibit 5.4 below) that if the initial period of AOT is longer than six months, improvement in rates of receipt of appropriate psychotropic medications in the post-AOT period are sustained whether or not the recipient continues to receive intensive treatment in the form of ACT or ICM. For these longer-term AOT recipients, the predicted probability of post-AOT medication possession remains at a substantially improved level relative to the pre-AOT period, even without continued ACT or ICM services utilization (50% compared to 37%.) If ACT or ICM is also discontinued when long-term AOT ends, the predicted probability of receiving appropriate psychotropic medications declines to 43%, which is still an improvement over the pre-AOT rate.

Exhibit 5.4. Adjusted* percent with at least 80% medication possession in any given month over long-term AOT course
 percent with at least 80% medication possession in any given month
*Adjusted probability estimates were generated from repeated measures regression models controlled for time, region, race, age, sex, diagnosis, and co-insurance status.
Source: Medicaid claims and AOT Evaluation database.

Arrest Outcomes Comparing Post AOT and Post EVS Recipients In Six Counties

Findings in Chapter 3 reported significant reductions in arrest rates while recipients were on AOT compared to their pre-AOT/EVS period of observation. (Current EVS status was not significantly associated with reduced rates of arrest although there was a clear trend toward reduction of arrests during this period.) Exhibit 5.5 shows that the post-AOT and post-EVS conditions were not significantly associated with lowered probabilities of arrest compared to the pre-AOT/EVS period (current AOT and current EVS periods are included for reference).

Exhibit 5.5 Adjusted percent arrested in month by receipt of AOT and Enhanced Voluntary Services
percent arrested in month by receipt of Assisted Outpatient Treatment Enhanced Voluntary Services
Number of individuals=181; Number of person-month observations=9,229. Adjusted arrest rate estimates were produced using multivariable time-series regression analysis, controlling for time, region, age, sex, race, education, and diagnosis. Months spent in hospital are excluded from analysis.
Source: 6-county interviews and Division of Criminal Justice Services

Summary
We examined whether selected gains made during AOT are sustained over time, continuing into the post-AOT period. We examined three key outcomes that improved during AOT: reduced rates of hospitalization, increased receipt of psychotropic medications appropriate to the individual’s diagnosis, and reduced likelihood of arrest. For the hospitalization and medication outcomes, which were assessed via the Statewide Medicaid data, we find that sustained improvement after AOT ends varies according to the length of time the recipient spends under the AOT order. If AOT is discontinued after six months, these decreased rates of hospitalization and improved receipt of psychotropic medications are only sustained if recipients also continue to receive intensive services after AOT is discontinued. However, if AOT continues for 12 months or longer, reduced rates of hospitalization and improved receipt of medications are sustained whether or not intensive services are continued after AOT is discontinued. Thus, it appears that improvements in hospitalization and medication outcomes are more likely to be sustained if AOT continues for longer than 12 months. However, the post-AOT group did not maintain their reduced rate of arrest that was evident during AOT.

Chapter 6. Impact of Assisted Outpatient Treatment on the New York Service System

This chapter describes the impact of the AOT Program on the public mental health system in New York State. Clearly, AOT had some direct effects on the several thousand individuals who received court orders, as discussed in previous chapters. However, key features of the system of care in which AOT has been implemented–its capacity, resource allocation, and patterns of service utilization–were altered by AOT in ways that may have indirectly affected other persons with severe mental illness (SMI) who were not candidates for AOT.

We examine potential system effects of AOT by addressing three questions:

  1. Did AOT increase service capacity for all recipients?
  2. Did AOT offset, or divert, intensive services from other SMI individuals who would not qualify for AOT?
  3. How did the impact of AOT on system resources vary over time and by region during the years since AOT was initiated?

We examine these questions by focusing on trends in Medicaid claims for Intensive Case Management (ICM) and Assertive Community Treatment (ACT). These two modes of service operationally define a key requirement for intensive case coordination that underlies AOT court-ordered treatment plans; they also function as indicators of met need for service among comparable individuals with SMI who do not receive AOT orders.

To put AOT in a system perspective, recipients of AOT include only a small proportion–about 2%–of the service population with severe mental illness in New York. However, AOT recipients account for about one quarter of those receiving ACT or ICM services as shown in Exhibit 6.1.

Exhibit 6.1 Putting AOT in Population Perspective: 2005 snapshot
Population Perspective: 2005 snapshot

New York State provided substantial new funding to implement AOT Programs statewide and to build service capacity in the public mental health system over the past decade. An annual total of $32 million was appropriated for direct support of AOT Programs. This appropriation included $9.55 million per year to fund new case management slots anticipated for by AOT recipients. The new funding also included $15 million for a medication grant program; $4.4 million for prison and jail discharge managers; $2.4 million for oversight programs; and $0.65 million for drug monitoring.

Exhibit 6.2 Number of clients being served on ACT and ICM by year: New York State Office of Mental Health Patient Characteristics Surveys
Number of clients being served
Source: Medicaid claims and OMH administrative data.

In addition, and in tandem with the AOT Program, the state allocated $125 million yearly for enhanced community services; these funds were used to develop a Single Point of Access Program (SPOA) and to increase ACT and ICM capacity. Exhibit 6.2 illustrates the increase in volume of ACT and ICM service delivery during the years following the start of AOT in 1999. Clearly, ACT teams have been on the increase, effectively replacing ICM for many recipients after 2001. For purposes of this chapter, we combined ACT and ICM into one category of intensive case coordination services.

Exhibit 6.3 illustrates the impact of these new resources on service utilization for AOT recipients and a comparable population of non-AOT service recipients.15 The overall increase in services is seen in the upward trend in the total number of monthly paid Medicaid claims for ACT or ICM: the volume of these services increased 400% between 2000 and 2007. However, important differences in the trend emerge in the comparison of time periods for AOT recipients (before, during, and after AOT), and in the quite different pattern for ACT-ICM recipients who did not receive AOT.

Exhibit 6.3 Distribution of ACT/ICM services by month and AOT status
services by month

ACT-ICM services doubled in the first three years after AOT began (2000-2003), but all of that increase went to AOT recipients. There was no increase in ACT-ICM services to non-AOT recipients as a group during the first three years of AOT implementation. However, between 2003 and 2007, the trend shifted as non-AOT recipients saw an increase in ACT-ICM services that paralleled that for AOT recipients.

After the first three years of AOT implementation, ACT-ICM services increased both for post-AOT recipients and those who never received AOT. As a result, by 2007 ACT-ICM monthly claims were almost evenly distributed between post-AOT and non-AOT participants with a small and diminishing share of the services going to current AOT participants.

Exhibit 6.4 displays the effect of this pattern in terms of the “non-AOT share” of ACT-ICM claims. Specifically, this is the trend in the proportion of monthly claims for services that went to individuals who were not on AOT orders.

Exhibit 6.4. Non-AOT share of ACT-ICM services by month
services by month

Between 2000 and 2003, the non-AOT share of monthly Medicaid claims for ACT-ICM services was cut in half–from 100% to less than 50%. After that, however, the decline in non-AOT share stabilized and gradually reversed. By 2007, three-quarters of ACT-ICM services were going to non-AOT recipients–the same proportion as in 2001. The increase in ACT-ICM services to non-AOT recipients coincided with the decrease in new AOT orders (as described in Chapter 1.)

To examine at the individual level whether the AOT Program in effect diverted services from non-AOT recipients, we conducted multivariable time-series analyses of factors affecting receipt of ACT-ICM in any given month for non-AOT recipients. The sample for the analyses consisted of 3,170 persons with SMI who never received AOT, but received ACT-ICM during some period between 2000 and 2007 as indicated by paid Medicaid claims. The time series analysis included 66,833 person-month observations for these individuals.

By 2007 ACT-ICM monthly claims were almost evenly distributed between post-AOT and non-AOT participants with a small and diminishing share of the services going to current AOT participants.

The first outcome in the analysis was whether, in any given month, the non-AOT individual initiated ACT-ICM, not having received these services previously. The second outcome was whether a non-AOT individual discontinued ACT-ICM–or was no longer receiving these services in a given month–after having received them previously. Control variables included time (year), region, age, sex, race, diagnosis, and co-insurance status.

In the first analysis, we found that increasing the number of AOT orders in the system was significantly associated with a decreasing chance that non-AOT individuals would initiate receipt of ACT or ICM. Specifically, during months when the number of AOT orders exceeded 200, the odds were approximately cut in half that a non-AOT individual would initiate ACT-ICM services.16

In the second analysis, we found that increasing AOT orders in the system significantly increased the odds of non-AOT recipients discontinuing of ACT-ICM.

Specifically, during months when the number of AOT exceeded 200, the odds of non-AOT individuals discontinuing ACT-ICM were increased by about 50% compared to months with fewer AOT orders in the system.17 When the number of AOT orders in the system exceeded 400 in a given month, the odds of discontinued ACT-ICM for non-AOT recipients doubled.18

Finally, we examined whether the impact of AOT varied by region, comparing the New York City region to other regions in the state. We found a similar pattern in New York City and in other regions with two exceptions. First, the effect of discontinuing ACT-ICM for non-AOT recipients occurred more slowly in New York City. Specifically, outside of New York City a significant increase in ACT-ICM discontinuation was seen among non-AOT recipients in the third year after AOT started. In New York City a significant increase in discontinuation for ICM-ACT among non-AOT recipients was not seen until the sixth year following AOT. Second, increasing AOT had a sharper impact on discontinuing ACT-ICM for non-AOT individuals in the non-NYC regions compared to the New York City Region.19 Exhibit 6.5 displays adjusted percents from these analyses.

Exhibit 6.5. Adjusted* percent of non-AOT recipients discontinuing ACT-ICM in any given month, by system AOT volume and region
percent of non-Assisted Outpatient Treatment recipients
*Adjusted probability estimates were generated from repeated measures regression models controlled for time, race, age, sex, diagnosis, and co-insurance status.
Source: Patient Characteristics Survey, US Census, Mental Health Needs Assessment Project, AOT Evaluation database.

Summary

The implementation of AOT coincided with a large increase in mental health services through OMH, which eventually increased the availability of ACT teams and ICM for all service recipients with SMI–even those who never received AOT. In the process of implementing the AOT Program, preference was initially given to new AOT cases in allocation of ACT and ICM. This meant that, even accounting for overall time-trend, region, patient demographics and diagnosis, the increasing number of AOT cases in the system significantly affected ACT-ICM service delivery to non-AOT recipients. Specifically, when AOT cases increased, non-AOT recipients had a significantly lower chance of initiating ACT-ICM services and a significantly higher chance of discontinuing these services if they were previously receiving them. These indirect consequences of the AOT Program occurred more slowly and were not as pronounced in the New York City region compared to other regions of the state; perhaps because the service volume and system capacity was greater in the New York City Region, and thus, it was able to absorb a greater volume of new AOT cases with less impact on other service recipients with SMI. Also, the apparent impact of AOT in diverting services from non-AOT service recipients was concentrated mostly in the first three years of AOT implementation, between 2000 and 2003. During those years, there was essentially no growth in ACT-ICM services to non-AOT individuals; however, following 2003, as the number of new AOT orders stabilized and then declined, the new service capacity that accompanied the implementation of AOT was apparently available to other individuals who needed these services, irrespective of AOT status. Thus, following the initial ramp-up of the AOT Programs throughout the state, intensive community-based services increased for SMI individuals on AOT and non-AOT individuals alike.

Summary and Conclusions

To address the six areas of investigation requested,, we studied existing records from several extensive data sources described in Appendix B including: AOT Program, New York State Office of Mental Health hospitalization, Medicaid claims, U.S. Census, and Mental Health Needs Estimation Project data. In addition, we conducted statewide in-person interviews with key stakeholders to gain insight into the operation of the AOT Program and interviewed service recipients to assess attitudes about treatment, treatment experiences, and treatment outcomes.

Limitations

While this evaluation approach has substantial strengths because of its reliance on multiple sources of data, each data source also has limitations. Case managers provide extensive data about recipient functioning captured in the Child and Adult Integrated Reporting System (CAIRS). However, given heavy clinical and administrative demands on case managers and limited time for training on completing the CAIRS, reporting on this instrument may be inconsistent. Because CAIRS has variable amounts of missing data, we only utilized CAIRS when the level of missing data was acceptable. Case managers may also have unknown biases in reporting of outcomes of recipients in their respective programs. Reliance on Medicaid claims data also has limitations in that Medicaid eligibility may fluctuate, claims may be inconsistently submitted, and Medicaid-ineligible recipients may be different in ways we can not measure. Our analysis approach limits analyses to periods of Medicaid eligibility and may fail to detect differences in outcomes for recipients who are Medicaid ineligible. Wherever possible we have carefully drawn matched comparison groups to examine whether AOT differentially affects outcomes when compared to recipients receiving voluntary treatment. Given a number of alternative data sources, the large volume of data, and careful use of statistical approaches, these analytic approaches have substantial strengths, but these analytic approaches are not as definitive as a rigorously conducted randomized controlled trial.

New York’s AOT Program features more comprehensive implementation, infrastructure and oversight of the AOT process than any other comparable program in the United States.

We summarize findings and conclusions from each area or investigation below. In addition, we provide a summary table indicating the sources and strengths of findings in each area of investigation.

Summary and Conclusions

Description of the New York AOT Program and Regional Variations

The introduction of New York’s AOT Program was accompanied by a significant infusion of new service dollars and currently features more comprehensive implementation, infrastructure and oversight of the AOT process than any other comparable program in the United States. It is, therefore, a critical test of how a comprehensively implemented and well-funded program of assisted outpatient treatment can perform. However, because New York’s program design is unique, these evaluation findings may not generalize to other states, especially where new service dollars are not available. This report addresses whether AOT can be effective and under what circumstances, not whether it will always be effective.

One important difference among regions was the use of voluntary agreements in lieu of a formal AOT court order.

As designed, the AOT statute can be used to prevent relapse or deterioration before hospitalization is needed. However, in nearly three-quarters of all cases, it is actually used as a discharge planning tool for hospitalized patients. Thus, AOT is largely used as a transition plan to improve the effectiveness of treatment following a hospitalization and as a method to reduce hospital recidivism. To quote one Mental Health Legal Service (MHLS) attorney on his view of AOT:

We see AOT as a way for some clients to get what they need. They are severely mentally ill and need good follow-up treatment in the community. This is a way for them to get out of the hospital much sooner.

AOT is largely used as a transition plan to improve the effectiveness of treatment following a hospitalization and as a method to reduce hospital recidivism.

Most of New York State’s experience with AOT originates in the New York City region where approximately 70% of all AOT cases are found. AOT was systematically implemented citywide in New York City with well-delineated citywide policies and procedures. In the remainder of the state, AOT was implemented and utilized at the discretion of counties. In some counties AOT has been used rarely; in several it has not been used at all.

Based on our key stakeholder and recipient interviews and on AOT Program data, we found considerable variability in how AOT is implemented across the state but strong uniformity in how it is implemented in New York City. One important difference among regions was the use of voluntary agreements (sometimes referred to as EVS) in lieu of a formal AOT court order. Under a voluntary agreement, the recipient signs a statement that he or she will adhere to a prescribed community treatment plan. In the New York City Region, an AOT court order almost always precedes an agreement for EVS. Voluntary agreements are usually implemented following a period of AOT as a “step-down” arrangement when a recipient is judged to be ready to transition from an AOT order to voluntary treatment, usually with the same enhanced service package. In one of the key informant interviews, an AOT Program staff remarked:

Voluntary agreements are used (in New York City) as part of the clinical "step-down" process.

In other counties, largely outside of New York City, voluntary agreements are more frequently used before an AOT court order as trial periods before initiating a formal AOT order. If the trial period proves unsuccessful, an AOT proceeding is then initiated. A psychiatrist from an upstate county discussed this approach to providing EVS in the following way:

We don't do it like downstate or like OMH wants. We use the voluntary order first. We don't approach it in any way.

Statewide, use of EVS First is far less common because the majority of AOT orders occur in New York City where voluntary agreements typically come as a trial AOT termination. Because the regions in which these two very different approaches to voluntary agreements occur differ so much in population characteristics and in the availability of treatment services, it is not possible to directly compare their relative effectiveness.

The other major difference across the state lies in the consistency of the AOT court process. There was widespread agreement that judges hearing AOT cases could benefit from additional mental health and AOT training, especially in counties where many judges rotate in these courts. In some counties hearings may be waived, or the client may waive his or her appearance. In uncontested hearings there may be no perceived need to have a doctor present at the hearing because the facts are stipulated and the outcome agreed upon. This results in some significant procedural variations across courts. To quote several judges:

In a situation where the patient agrees with the plan, no doctor is needed. If the plan is contested – that's different. You can always waive a hearing.

Even in the counties that mandated appearances by the physician, almost all agreed that not having a physician appear would reduce costs and scheduling difficulties, particularly for the smaller counties that contract for physician services. Most counties were in favor of increasing the availability of stipulations in the AOT process, especially for renewals. This would reduce the court burden and costs and would reduce some of the hearing logistics and transportation burdens.

Not having a physician appear would reduce costs and scheduling difficulties, particularly for the smaller counties.

In some counties the programs were small enough such that the level of service coordination was maximized with everyone "at the table" working on the AOT treatment plan. Some of the county differences we observed may be due to the fact that some counties are structured differently in their service delivery approach. To quote one Director of County Services:

A big piece of how it works or does not work across New York State is the county structure – are they a service providing department or contracting agency?

Racial Disparities in AOT: Are They Real?

An April 2005 report on statewide demographic data from the New York Lawyers for the Public Interest found that African Americans were over represented in the AOT Program. Whether this over-representation is discriminatory rests, in part, on whether AOT is generally seen as beneficial or detrimental to recipients and whether AOT is viewed as a positive mechanism to reduce involuntary hospitalization and improve access to community treatment for an under-served population, or as a program that merely subjects an already-disadvantaged group to a further loss of civil liberties.

We find no evidence that the AOT Program is disproportionately selecting African Americans for court orders.

We find that the overrepresentation of African Americans in the AOT Program is a function of African Americans’ higher likelihood of being poor, uninsured, higher likelihood of being treated by the public mental health system (rather than by private mental health professionals), and higher likelihood of having a history of psychiatric hospitalization. The underlying reasons for these differences in the status of African Americans are beyond the scope of this report. We find no evidence that the AOT Program is disproportionately selecting African Americans for court orders, nor is there evidence of a disproportionate effect on other minority populations. Our interviews with key stakeholders across the state corroborate these findings.

Service Engagement

A key goal of the AOT Program is to motivate consumers to actively engage in treatment during and after their involvement with the program. We find that during the first six months on AOT, service engagement was comparable to service engagement of voluntary patients not on AOT. After 12 months or more on AOT, service engagement increased such that AOT recipients were judged to be more engaged than voluntary patients. This suggests that after 12 months or more, when combined with intensive services, AOT increases service engagement compared to voluntary treatment alone.

During AOT there is a substantial reduction in the number of psychiatric hospitalizations and in days in the hospital if a person is hospitalized.

Clearly one of the reported strengths of the AOT Program, evidenced in our key informant interviews, was improved recipient access to needed services. Even key informants who had been initially opposed to AOT have come to realize that the additional service dollars associated with the AOT Program provide needed services, although respondents report that the paucity of integrated co-occurring substance abuse services is still problematic. Nonetheless, some respondents still feel that if adequate consumer-driven services were available, there would no need to engage recipients through the mechanism of the AOT process. Comment by a psychiatrist and peer advocate illustrate this:

Kendra's law commits the individual to treatment and commits treatment providers to treating the individual.

AOT would not be needed if services were compassionate and coordinated. Consumers would come.

Recipient Outcomes

We find consistent evidence that during AOT there is a substantial reduction in the number of psychiatric hospitalizations and in days in the hospital if a person is hospitalized. We also find moderately strong evidence from lifetime arrest records of AOT and EVS recipients from the Division of Criminal Justice Services that AOT reduces the likelihood of being arrested. We find substantial increases in receipt of intensive case management services during AOT. We also find that, under AOT, recipients are far more likely to consistently receive psychotropic medications appropriate to their psychiatric conditions. Case managers of AOT recipients also report subjective improvements in many areas of personal functioning such as managing appointments, medications, and self-care tasks.

AOT reduces the likelihood of being arrested.

Selection of recipients for the AOT Program was a source of considerable discussion among key informants who suggested that particular kinds of recipients may or may not benefit from the AOT Program. Most key informants felt that the majority of AOT recipients were appropriate for the program, but they agreed that many who might benefit were never referred. They felt that the recipients’ deference to the authority of the judge might significantly affect the success of the order. Many respondents believe that recipients with substance abuse, personality disorders, or extensive criminal histories were the least likely to be successful in the program. They suggested this might be due to the scarcity of appropriate services for these conditions, or in the case of substance abuse, the perceived lack of enforceability of nonadherence to substance abuse treatment. Comments by a psychiatrist and AOT coordinator illustrate this:

I don't know that AOT doesn't work for substance abusers- it might be more that appropriate services are not always available.

AOT doesn't work well with the seriously drug involved because it's hard to make the case manager connection and with people with antisocial personality, because the court scene doesn't affect them. –

Recipient Perceptions of AOT

Participants were assessed on scales measuring a wide range of AOT-related attitudes and treatment experiences, including their understanding of AOT; whether they believe it beneficial or harmful; whether they find it stigmatizing; whether it affects their sense of autonomy or empowerment; satisfaction with treatment; perceived coercion related to treatment; perceived pressures to engage in treatment; whether it increases perceived barriers to treatment; and how it affects their sense of being fairly treated. On the whole, AOT recipients and non-AOT recipients have remarkably similar attitudes and treatment experiences. That is, despite being under a court order to participate in treatment, current AOT recipients feel neither more positive nor more negative about their mental health treatment experiences than comparable individuals who are not under AOT. This suggests that positive and negative attitudes about treatment during AOT are more strongly influenced by other experiences with mental illness and treatment than by recent experiences with AOT itself.

Positive and negative attitudes about treatment during AOT are more strongly influenced by other experiences with mental illness and treatment than by recent experiences with AOT itself.

Service Utilization and Outcomes After AOT Ends

We examined whether selected gains made during AOT are sustained over time by examining two key outcomes that improved during AOT reduced rates of hospitalization and increased receipt of psychotropic medications appropriate to the individual’s diagnosis. We find that sustained improvement after AOT ends varies according to the length of time the recipient spends under the AOT order. If AOT is discontinued after six months, these decreased rates of hospitalization and improved receipt of psychotropic medications are sustained only if recipients continue to receive intensive case management services. However, if AOT continues for 12 months or longer, reduced rates of hospitalization and improved receipt of medications are sustained whether or not intensive case management services are continued after AOT is discontinued. Thus, it appears that improvements are more likely to be sustained if AOT continues at least 12 months.

Sustained improvement after AOT ends varies according to the length of time the recipient spends under the AOT

Impact of AOT on New York’s Public Mental Health System

It is uncertain whether, as a consequence of AOT implementation, resources have been diverted away from other adults with severe mental illness. We examined the impact of AOT Programs on the availability of resources for all adults with severe mental illness. We focused on access to high intensity case management services.

The introduction of AOT was accompanied by a large increase in funding for mental health services, which, over time, increased the availability of intensive services for all service recipients, even those who never got AOT. In the first several years of the AOT Program, between 1999 and 2003, preference for intensive case management services was given to AOT cases–a finding corroborated by our key stakeholder interviews. In fact, some respondents stated that service and housing providers were more likely to accept clients with an AOT court order and all confirmed that AOT recipients were given priority access. An AOT coordinator made this observation:

AOT doesn't make a big difference in some people’s compliance, but does help with community mental health providers’ willingness to provide services to people.

This meant that in the first several years of the AOT Program, non-AOT recipients were less likely to receive intensive case management services than their AOT counterparts. These indirect consequences of the AOT Program occurred more slowly and were not as pronounced in New York City compared to other regions of the state. This may have been because the treatment capacity was greater in New York City, and thus it was able to absorb a greater volume of new AOT cases with less impact on other service recipients with severe mental illness.

After 2003, new AOT orders leveled off in the state and then declined. The new treatment capacity that accompanied the implementation of AOT was apparently then available to other individuals who needed these services, irrespective of AOT status. Thus, following the initial ramp-up of the AOT Programs throughout the state, intensive community-based services increased for individuals on AOT and those not on AOT alike. However, because the new service capacity created during the introduction of the AOT Program is now fully utilized, competition for services in the near future may intensify, with unknown effects on AOT relative to non-AOT recipients.

Several key informant respondents commented on the lack of new resources for the AOT Program and treatment services. They were concerned about the relatively flat funding for the AOT Program since its inception. And while some service dollars may have increased in categories not directly designated as AOT, the program administration dollars have not changed according to these respondents. These points are emphasized by two AOT personnel:

Money has been consistent over time– to the same counties–even those without AOT.

You can not expect people to be paid the same amount of money seven years later.

Because the implementation of the AOT Program in New York was accompanied by an infusion of new services, it is impossible to generalize the findings of this Report to states where services do not simultaneously increase.

Other Issues to Consider

Key stakeholders we interviewed suggested the state should consider how much consistency it wishes to see in the AOT Program. While the entire state may not opt to adopt policies and procedures used in New York City, common statewide procedures could be more fully developed, while still allowing local flexibility. For example, court procedures across the state could be more uniformly standardized. Some courts allow greater leeway in stipulations, some hearings are waived, and occasionally physicians/examiners are not required to testify in person. These variations in court procedures, if deemed appropriate, could be used to streamline court procedures and reduce court expenses. Standardization should be considered for forms used throughout the AOT process and the issuance of removal orders and the access to certified records could be made more efficient. In addition, there are frequent problems with inter-county transfers and jurisdiction over AOT cases that could be addressed. Key stakeholders also suggested the state consider the status of voluntary agreements, which are not codified in the AOT statute or regulation. They feel that the state should consider whether it wishes to create formal voluntary agreement options.

Overall Summary and Conclusions

We find that New York State’s AOT Program improves a range of important outcomes for its recipients, apparently without feared negative consequences to recipients. The increased services available under AOT clearly improve recipient outcomes, however, the AOT court order and its monitoring do appear to offer additional benefits in improving outcomes. It is also important to recognize that the AOT order exerts a critical effect on service providers, stimulating their efforts to prioritize care for AOT recipients.

Available data allow only a limited assessment of whether voluntary agreements are effective alternatives to initiating or continuing AOT. There are relatively few voluntary agreements and they typically occur in counties that use the "EVS First" model. However, we found some evidence that AOT recipients are at lower risk of arrest than their counterparts in enhanced voluntary services. We also found evidence in the case manager data that receiving AOT combined with ACT services substantially lowers risk of hospitalization compared to receiving ACT alone.

Recipients appear to fare better during and after AOT if the AOT order lasts for six months or more. Once AOT recipients leave the program, improvements are more likely sustained among those who continue to receive intensive treatment services or have longer periods of AOT.

Perceptions of the AOT Program, experiences of stigma, coercion, and treatment satisfaction appear to be largely unaffected by participation in the program and are likely more strongly shaped by other experiences with mental illness and treatment.

In its early years, the AOT Program did appear to reduce access to services for non-AOT recipients. However, in recent years the reduction in new AOT cases has attenuated this effect. Lack of continued growth of new service dollars will likely increase competition for access to services once again.

If New York extends the AOT Program, consideration should be given to further strengthening statewide policies and procedures to achieve a more consistent program.

AOT Report: Summary and Strength of Findings
Data Source & Analysis Approach
  Summary: Combined Measures1 Key Informant Interviews Recipient Interviews, Chart Reviews and Arrest Records Case Manager Reports (CAIRS) Case Manager Reports: Controlled Statistical Analyses Medicaid and OMH Data: Controlled Statistical Analyses
Area and Finding             
Description of AOT Program             
Regional differences in AOT Program +++  +++ +++ n/a n/a +++
Absence of racial disparities in AOT Program administration +++  +++ n/a n/a n/a +++
Positive service engagement of AOT recipients +++  n/a +/- +++ +++ n/a
Recipient Outcomes             
Reduced hospitalization +++  ++ +/- +++ +++ +++
Increased receipt of medication +++  n/a n/a n/a n/a +++
Increased receipt of case management +++  +++ n/a +++ n/a +++
Improved functioning +  n/a +/- + n/a n/a
Decreased arrests ++  n/a ++ +/- +/- n/a
Recipient Perceptions of AOT             
Improved autonomy +/-  n/a +/-  n/a n/a n/a
Improved treatment satisfaction +/-  n/a +/-  n/a n/a n/a
Increased coercion +/-  n/a +/-  n/a n/a n/a
Service Utilization and Outcomes Post-AOT             
Reduced hospitalization +++  +++ +/- +++ n/a +++
Increased receipt of  medication +++  n/a n/a ++ n/a +++
Increased receipt of case management +++  +++ n/a +++ n/a +++
Impact of AOT on Service System             
Increased receipt of case management for AOT recipients +++  +++ n/a +++ n/a +++
Decreased receipt of case management for non-AOT recipients +/-  ++ n/a n/a n/a +/-

1 Key
+++ very strong evidence
++ moderately strong evidence
+ some evidence
+/- equivocal findings
n/a Not applicable, no evidence for or against

 
  1. Appendix A contains an overview of Kendra’s Law and the statute.
  2. OMH’s Interim Report on Kendra’s Law is available on the OMH Web site at http://www.omh.state.ny.us/omhweb/Kendra_web/interimreport/ and the Final Report is at http://www.omh.state.ny.us/omhweb/Kendra_web/finalreport/index.htm
  3. Throughout this report, unless specified otherwise, our data analysis goes through midyear 2007. While OMH continues to collect AOT program data and publishes updated program statistics on its website, our formal evaluation required a cut-off point to standardize the period of observation across datasets and to allow time to complete the extensive analyses presented herein.
  4. Data on the number of investigations was only recorded and reported to OMH until 2003.
  5. Blended case management programs comprise both intensive case managers, who carry a smaller client caseload, and supportive case managers.
  6. Although NY OMH systematically collects data on individuals on a court order, they do not collect data on individuals who receive EVS. Descriptive information for a subsample of individuals receiving EVS is presented in Chapter 4.
  7. The most up-to-date demographic data were derived for the OMH AOT website.
  8. Data updated from M. Cooper, “Racial Disproportion Seen in Applying ‘Kendra’s Law’,” New York Times, 7 April 2005.
  9. Our investigation of this question involved multiple data sets including: OMH administrative and clinical records on persons receiving AOT; OMH data on all service recipients’ characteristics and hospital admissions; U.S. Census estimates of county population by race and poverty status; county estimates of the prevalence of severe mental illness, applying epidemiological survey data to the demographic profile of each county.
  10. Relevant populations used as denominators included: 1) the general population; 2) those with severe mental illness (SMI) in the community; 3) people with SMI receiving mental health services; 4) the public mental health system’s adult services recipient population; 5) people with SMI who have been hospitalized during a given year; 6) those who have been involuntarily committed to inpatient facilities more than once in the previous year.
  11. State totals were weighted by region.
  12. E. Silver et al., “Neighborhood structural characteristics and mental disorder: Faris and Dunham revisited.” Social Science and Medicine. 55 (2002): 1457-1470. C.E. Holzer et al., “Ethnicity, Social Status and Psychiatric Disorder: Evidence from the Epidemiologic Catchment Area Survey.” In R. Price, B. Shea, & H. Mookherjee (Eds.) Social Psychiatry Across Perspectives. (New York: Plenum, 1995): 93-104.
  13. The proportion of whites is higher in this sample than the statewide AOT sample because of the incorporation of several upstate counties.
  14. Specific information about the measures used is provided in Appendix B.
  15. Selection criteria for comparison group: OMH service recipients with history of 2 or more psychiatric admissions in any year since 1999; schizophrenia or affective disorder as billing diagnosis for inpatient admission; total of 14 or more inpatient days in any single year; did not receive AOT but received ACT or ICM services at any time since 1999.
  16. Adjusted Odds Ratio = 0.50; 95% confidence interval = 0.43 - 0.57.
  17. Adjusted Odds Ratio = 1.45; 95% confidence interval = 1.18 – 1.78.
  18. Adjusted Odds Ratio = 2.22; 95% confidence interval = 1.76 – 2.79.
  19. Adjusted odds ratio = 1.67 vs. 1.36 when monthly AOT orders in the system were between 201 and 400, compared to between 0 and 200.