Skip to Main Content

Office of Mental Health

New York State Assisted Outpatient Treatment Program Evaluation

June 30, 2009

Appendix B Methods Overview

View Adobe Acrobat Version | Download Adobe Acrobat Reader

In this section we provide an overview of methodological and design issues relevant to the evaluation, including: (1) multiple sources of data; (2) study samples; (3) recruitment and Institutional Review Board procedures for the 6-county data; (4) measures and instruments; (5) data structure for repeated measures analysis; (6) analytic approaches; (7) sample weighting with propensity scores to adjust for comparison group differences; and (8) multiple imputation of missing data.

1. Multiple Sources of Data

The Assisted Outpatient Treatment (AOT) evaluation research project combined primary data collection with secondary analysis of several existing data sources. Specifically, we interviewed key informants throughout the state and conducted structured interviews with AOT recipients and Enhanced Voluntary Services recipients in six New York counties: Albany, Erie, Monroe, Nassau, New York, and Queens. We obtained lifetime arrest records for sample members in these six counties. We conducted secondary analyses of AOT program administrative, tracking, and evaluation data. We used Medicaid claims and OMH psychiatric facilities’ admissions data to capture hospitalizations and mental health services encounters. We utilized data from the US Census, the Mental Health Needs Assessment Project, and the New York Office of Mental Health (OMH) Patient Characteristics Survey. These various sources of data are represented schematically in Exhibit B.1 and are briefly described next.

Child and Adult Integrated Reporting System (CAIRS). The CAIRS data contain information on Assertive Community Treatment (ACT) and AOT service recipients. ACT team members and AOT case managers complete a standardized assessment for each recipient at the onset of ACT or AOT services and every six months thereafter for the duration of the ACT and AOT services. Data are collected on: (1) demographic characteristics; (2) living situation; (3) services received; (4) engagement in services; (5) adherence to prescribed medications; (6) self-care and social skills; and (7) the occurrence of significant events, including hospitalization, homelessness, arrest, incarceration, and harmful behaviors. The CAIRS data were used to examine a variety of outcomes, including those presented in Chapters 2 and 3.

Tracking for AOT Cases and Treatment (TACT). The TACT database contains information on each AOT court order, including dates of initiation, expiration, and renewal. The TACT data were used to create periods of AOT exposure and were then merged with the Medicaid data.

Medicaid. Medicaid claims and eligibility data were used to describe patterns of inpatient and outpatient services utilization between 1999 and 2007. Only Medicaid-eligible person-months were included in our multivariable analyses (a more detailed discussion of this can be found in the Data Structure and Analytic Approaches sections below). Medicaid data were also used to identify an "intensive treatment" comparison group of non-AOT recipients who had experienced: (1) two or more hospitalizations; (2) 14 inpatient days in any single year; and (3) had received ACT or intensive case management (ICM) services in any year since 1999; this comparison group was used in the analysis of AOT’s impact on the service system (Chapter 6). The same strategy was used to identify a "usual care" group for the 6-county interview study; however, there was no ACT/ICM criterion for the 6-county "usual care" group (see Exhibit B.2 and the Study Samples section below for more information on these two groups). Medicaid analyses are presented in Chapters 3, 5, and 6.

Exhibit B.1. AOT study: Multiple data sources
Multiple data sources

OMH Hospitalization Data. OMH inpatient hospitalization data were merged with Medicaid inpatient service claims data to create an overall summary of inpatient days. All inpatient analyses include both Medicaid and OMH psychiatric facility stays.

County Population Characteristics, AOT Rates, Mental Health Needs, and Services Utilization Data. We combined multiple sources of data for the "racial disparity" analyses presented in Chapter 1. Analyses are based on data from: (1) the OMH Patient Characteristics Survey, a biannual survey that collects information on the population served in the State’s mental health system and types of services received; (2) OMH hospitalization data; (3) the US Census, which we used to obtain estimates of county population by race and poverty status; and (4) synthetic county estimates of the prevalence of severe mental illness (SMI). Estimates were obtained from Professor Charles E. Holzer III at UTMB Galveston. Holzer’s estimates are derived from statistical models which apply epidemiological survey data to the demographic profile of each county (http://psy.utmb.edu/estimation/estimation.htm Leaving OMH site). These estimates were obtained for the total number of African Americans and Whites with SMI in each county, whether or not they were in treatment.

Key Informant Interview Data. Primary data collection included key informant interviews throughout the State with AOT program directors, service providers, Mental Hygiene Legal Service attorneys, and others involved with the AOT program. These data were collected through open-ended qualitative interviews. Data from key informant interviews are interspersed throughout the Report in the form of direct quotes.

6-County Interview Data. Primary data collection was conducted in six counties: Albany, Erie, Monroe, Nassau, New York, and Queens. Data were collected through structured client interviews. Results from this sample are presented in Chapters 3 and 4.

Department of Criminal Justice Services Arrest Records. We obtained Division of Criminal Justice Services (DCJS) lifetime arrest records for AOT and Enhanced Voluntary Service recipients for those in the 6-county sample.

2. Study Samples

Except for the key informant interviews, all participants in the study were mental health service recipients who had been diagnosed with schizophrenia spectrum or affective disorder. All participants were aged 18 years or older. There were four main categories of study samples: AOT, Enhanced Voluntary Service, Intensive Treatment, and Usual Care. Exhibit B.2 summarizes key features of these samples, comparison groups, and methods of analysis. An overview of primary data collection activities in the 6 counties is presented next.

6-County Primary Data Collection

Interviews were conducted with individuals who had been on AOT or who were receiving Enhanced Voluntary Service. (There were a total of 211 unique individuals who completed a total of 277 interviews. Chapter 4 describes how we allocated the 277 interviews across non-duplicative sample groups and the characteristics of those samples.) Our original sampling plan included a usual care group in addition to the AOT and Enhanced Voluntary Service groups. Participants from the usual care group consisted of individuals matched to the AOT and Enhanced Voluntary Service groups on a number of characteristics, including diagnosis, hospitalization history, and region. However, given difficulties in obtaining appropriate matched samples from the Medicaid data, sampling for the usual care group was discontinued after 12 subjects had been interviewed. All subjects were re-classified into three groups: (1) no current or recent AOT, which included individuals who never had an AOT order and those who had not had an AOT order for at least the past 7 months; (2) current AOT, which included individuals on an AOT order at the time of the interview; and (3) recent AOT, which included individuals who had an AOT order in the past 6 months but who were not on an AOT order at the time of the interview.

Exhibit B.2: AOT evaluation project summary of data sources and comparison group operational definitions

  Medicaid data CAIRS data 6-county data2 6-county arrest data2 County aggregate data
AOT sample Medicaid claims 1999-2007 for mental health services for all OMH service recipients with AOT orders since 1999 (n= 2.7M claims); AOT order dates merged from TACT data All AOT Evaluation1 Baseline Assessment Forms and Follow-up Assessments Forms for AOT periods, filled out by case managers every 6 months (n=5,025; usable analytic sample n=3,692 ) AOT clients sampled from county AOT program rosters. Sample stratified into 3 cohorts: AOT current, AOT recent past, no current or recent AOT (see below for description of final samples) AOT clients sampled from county AOT program rosters (arrest data are then avaliable for 100 months). Sample stratified into 3 cohorts: Pre-AOT, Current AOT, and Post AOT (see below for description of final samples) Counts of AOT orders recorded in OMH TACT data 2000 to 2006 (n=372 county-years) and AOT investigations (duplicated annual counts 2000 to 2002; unduplicated counts after 2003)
Enhanced Voluntary Services Sample None (no enhanced voluntary services sample for this analysis). None (no enhanced voluntary services sample for this analysis). Enhanced Voluntary Services participants were sampled from county active AOT program rosters. Sample stratified into 3 cohorts: Pre-EVS, Current EVS, and Post EVS (see below for description of final samples) Enhanced Voluntary Services participants were sampled from county active AOT program rosters. Sample stratified into 3 cohorts: Pre-EVS, Current EVS, and Post EVS (see below for description of final samples) Counts of voluntary service enhancements
Intensive treatment sample3 Medicaid claims 1999-2007 for mental health services for all OMH service recipients meeting the following criteria (n=4.0M claims)
  1. Current service user: OMH certified outpatient service with a date of service of July 1, 2006 to present
  2. Diagnosis: schizophrenia or affective disorder as billing diagnosis for inpatient admission.
  3. Hospital recidivism: 2 or more psychiatric admissions in any year since 1999
  4. Intensive inpatient treatment: total of 14 or more inpatient days in a year
  5. Intensive outpatient services: OMH recipient of ACT or ICM services at any time since 1999
All CAIRS Baseline Assessment Forms and Follow-up Assessments Forms for ACT (non-AOT) periods; filled out by case managers every 6 months; matched to AOT sample on hospitalization history, diagnosis, gender, race (usable analytic matched sample n=744 unique invididuals who never had AOT) None (no intensive treatment sample for this analysis). None (no intensive treatment sample for this analysis). definitionsCounts of voluntarily and involuntarily hospitalized patients admitted to OMH psychiatric centers by county 2000 - 2006

sample 3 Counts of ACT and ICM recipients by county from the Patient Characteristics Surveys4 1999, 2001, 2003, 2005; data for intervening (non-surveyed) years are interpolated.

Usual care sample None (no usual care sample for this analysis). None (no usual care sample for this analysis). Sample from Medicaid usual-care group matched to AOT sample distributions on race, gender, and diagnosis. Sample stratified into 3 cohorts by time from hosp d/c: recent hosp; 6 mos; 12 mos after hosp (see below for description of final samples)
  1. Current service user: OMH certified outpatient service with a date of service of July 1, 2006 to present
  2. Diagnosis: schizophrenia or affective disorder as billing diagnosis for inpatient admission.
  3. Hospital recidivism: 2 or more psychiatric admissions in any year since 1999
  4. Intensive inpatient treatment: total of 14 or more inpatient days in a year
  5. Intensive outpatient services: No criterion for inclusion
None (no usual care sample for this analysis). Counts of OMH service recipients with SMI NOT receiving inpatient or Assertive Community Treatment (ACT) and Intensive Case Management (ICM) by county from the Patient Characteristics Surveys 1999, 2001, 2003, 2005; data for intervening (non-surveyed) years are interpolated.
Comparison groups for analysis The analytic samples for the AOT Medicaid analyses vary by outcome. However, as an example, the number of person-periods available for the various AOT conditions (and subsequent predicted probabilities) when evaluating admission to a hospital were: pre-AOT (n=117,889); AOT 1-6 months (n=26,949); AOT 7-12 months (n=14,916) Analyses included those with 6 or more months of treatment and those with 12 or more months of treatment. Person-periods: 6 month observations: ACT alone (n=1493); AOT+ICM (n=3518); AOT+ACT (n=2600). 12 month observations: ACT alone (n=952); AOT+ICM (n=1734); AOT+ACT (n=852) AOT current (n=115), AOT recent past (n=28), no current or recent AOT (n=134)5 Individuals: AOT group (n=144); EVS group (n=42). Person-periods: Pre AOT/EVS (n=16,709); Current AOT (n=2,083); Post AOT (n=838); Current EVS (n=952); Post EVS (n=518) NY counties aggregated by region: NY City, Central, Hudson, Long Island, Western
Methods of statistical analysis Repeated measures, multivariable logistic regression; statistical controls for underlying differences across individuals within different comparison groups Repeated measures, multivariable logistic regression; statistical controls for underlying differences across individuals within different comparison groups Mean values calculated for outcomes of interest across comparison groups Repeated measures, multivariable logistic regression; statistical controls for underlying differences across individuals within different comparison groups Repeated measures, multivariable logistic regression analyses by region; statistical controls for underlying differences across individuals within different comparison groups
  1. AOT Evaluation Database is a combination of data from the CAIRS and TACT databses.
  2. Albany, Erie, Monroe, New York, Nassau, Queens. Other data collection in these counties includes case manager informant interviews (1 time at AOT exit) and key informant interviews.
  3. The Medicaid intensive treatment sample was used for the analyses that examined the "Impact of Assisted Outpatient Treatment on the New York Service System", which was reviewed in Chapter 6.
  4. The Patient Characteristics Survey (PCS) collects demographic, clinical, and service-related information for each person who receives a mental health service during a specified one-week period. All programs licensed or funded (directly or indirectly) by the NYS Office of Mental Health are required to complete the survey. The PCS is conducted biennially and receives data from over 4,000 programs serving approximately 170,000 people during the week.
  5. For the 6-county sample there were 12 "intensive treatment" (i.e., ACT or ICM)/"usual care" (i.e., no ACT or ICM) participants. Nine of those 12 had received ACT or ICM and the remaining 3 had received "other case management" in the 6 months prior to their interview. These 12 individuals were folded into the "no/never AOT" group for the analyses described in Chapter 4 ("Participants’ perceptions of Assisted Outpatient Treatment (AOT) and related treatment experience and attitudes"). The specific breakdown for all 9 original "groups" was: New AOT (n=39); 6 month AOT (n=41); 12 month AOT (n=34); New EVS (n=22); 6 month EVS (n=45); 12 month EVS (n=15); recent post-AOT (n=42); 6 months post-AOT (n=27); Intensive treatment/Usual care (n=12). These 9 groups were reconstituted as AOT current (n=115); AOT recent past (n=28); and no current or recent AOT (n=134).

3. Recruitment and IRB Procedures for the 6-County Data

Recruitment

Eligible clients from the 6 counties were drawn from a list of current AOT and Enhanced Voluntary Service clients provided by the regional AOT Coordinator. The AOT Coordinator or program staff used a standard script to introduce the research study to the client and obtain permission for our research staff to directly contact the client. Clients interested in learning more about the research signed a screening form and provided contact information. Screening forms were then faxed or mailed to the research coordinator at PRA, who then confirmed eligibility and contacted clients to explain the research project and to schedule a meeting with a research interviewer.

The informed consent document was read aloud to each client by the research interviewer. Interviewers administered a brief assessment as part of the consent process to determine if the client understood the basic elements of the research (e.g., that they were free to refuse to participate or to stop at any time) and was able to communicate clearly. In the event the client failed the assessment (i.e., was not competent to complete the interview), the interviewer informed the client that he/she was not eligible to participate and provided them with a copy of the informed consent plus contact information for the research team. Eligible and competent clients signed, and were provided a copy of, the consent form prior to beginning the interview. The interview took approximately 90 minutes and participants were paid $25.

Institutional Review Board Procedures          

This research was reviewed annually by several Institutional Review Boards (IRBs), including those at Duke University Medical Center, Policy Research Associates, Inc., New York’s OMH, and Biomedical Review Association of New York (BRANY) which served as the IRB of record for the NYC Health and Hospitals Corporation and the NYC Department of Health and Human Services. This research was also subjected to individual facility reviews at Bellevue Hospital, Elmhurst Hospital, and Queens Hospital.

4. Measures and Instruments

We compared recipient groups on multiple outcomes. Descriptions of outcome variables and other variables of interest are listed below, along with their respective measures and data sources (Exhibit B.3).

Exhibit B.3. New York AOT Evaluation Instrument Table

Construct Measure Scale Data Source
Service engagement NY OMH ACT/AOT/CM Assessment[i] One item rated on a 5-point scale (not engaged [no contact with providers, does not participate in services at all] to excellent [independently and appropriately uses services]).

Individuals were rated as "high engagement=1" if they were rated as 4 or 5.

AOT Evaluation Dataset[ii]

Appointment adherence or receipt of services

AOT Evaluation Client Interview[iii] One item rated on a 5-point scale (never missed an appointment to avoided keeping appointments altogether).

Individuals were rated as "appointment adherent=1" is they were rated as 1 or 2.

Interview
Service billing A paid case management service claim reported for a given month is rated as "1" for that person-month observation. Medicaid and OMH records
Medication adherence or receipt NY OMH ACT/AOT/CM Assessment[i] One item rated on a 5-point scale (rarely or never takes medication as prescribed to takes medication exactly as prescribed). "Medication not prescribed" coded as "missing."

Individuals were rated as "high medication adherence=1" if they were rated as 4 or 5.

AOT Evaluation Datasetii
AOT Evaluation Client Interview[iii] One item rated on a 6-point scale (never missed taking medication to never took medication).

Individuals were rated as "medication adherent=1" is they were rated as 1 or 2.

Interview
Prescription filled Medication prescription fills were assessed by examining Medicaid claims and OMH service data. Only medications appropriate for the individual’s psychiatric condition, which had to be diagnosed by a psychiatrist or while in an inpatient hospital stay, were counted.

Individuals were rated as "positive for medication possession=1" if they have a sufficient medication supply for that month, as indicated by duration of prescriptions and defined as > 80% days of a given month.

 
Medicaid and OMH records
Attitudes about medications Modified Drug Attitude Inventory (DAI)[iv] Eight true/false statements. Item response reflects positive attitude toward medication, than=1. Example item: "Medications make me feel more relaxed."

Mean scores calculated, with higher scores reflecting more positive medication attitudes.

Interview

Hospitalization

NY OMH ACT/AOT/CM Assessment[i] Total number of psychiatric hospitalizations in the previous 6 months.

Individuals were rated as "positive for hospitalization=1" if they had > 1 hospitalizations in previous 6 months.

AOT Evaluation Datasetii
Monthly psychiatric hospital admissions Monthly psychiatric hospital admissions were assessed by examining Medicaid claims and OMH service data. An individual admitted in a given month was rated as "positive for hospitalization admission=1" for that person-month observation (an individual with multiple hospital admissions in a month is rated as "1"). Medicaid and OMH records

Harm to others

NY OMH ACT/AOT/CM Assessment[i] One item assessed the recentness of harm to others (never to this week).

Individual rated as "positive for harm to others=1" if an incident was reported within the past 6 months.

AOT Evaluation Datasetii
Modified MacArthur Community Violence Interview[v] A semi-structured interview was used to gather information from the 6-county service recipient sample on whether they engaged in 12 violent/aggressive behaviors, varying in degree of seriousness, in the previous 6 months. Each act is coded as 1=endorsed by recipient and 1=someone, other than interviewee, was physically harmed as a result of incident. Example: "In those six months, did you hit anyone with a fist or beat anyone up? Where did this happen and who else was involved? Was anyone physically hurt (besides you)? [If no, probe--] Not even bruises or cuts?"

The instrument yields a measure of any violence (yes/no).

Interview
Harm to self NY OMH ACT/AOT/CM Assessment[i] One item assessed the recency of an incident of self-harm (never to this week).

Individual rated as "positive for self-harm=1" if an incident was reported within the past 6 months.

AOT Evaluation Datasetii
Arrests Arrest records Arrest records of service recipients who participated in the 6-county interview (n=211) were obtained. Individual was rated as positive or negative for arrest (arrest=1, no arrest=0) for a given month, and the data were structured as person-month observations). Arrests included both misdemeanors and felonies. DCJS records[vi]
Illness Characteristics Modified Colorado Symptom Inventory[vii] Fifteen items assessing psychiatric symptoms experienced in the past month, endorsed on 5-point scale (at least every day to not at all).

Mean scores were calculated, with lower scores reflecting more prominent psychiatric symptoms.

Interview
Alcohol and drug use NY OMH ACT/AOT/CM Assessment[i] Recent alcohol and drug use was assessed across 12 substances.

Individual was rated as a "substance user=1" if reported used any substance in past 6 months (nicotine is not included in scale).

AOT Evaluation Datasetii

Functional impairment

Global Assessment of Functioning[viii] Ranked on a scale of 1 – 100, with high scores representing high functioning. Ten behavioral or symptom descriptors are used to guide the ranking. Example descriptor: 41 - 50 Serious symptoms (e.g., suicidal ideation, severe obsessional rituals, frequent shoplifting) OR any serious impairment in social, occupational, or school functioning (e.g., no friends, unable to keep a job).

Individual was rated as having a "significant functional impairment=1" if rated < 50.

Interview
NY OMH ACT/AOT/CM Assessment[i]:

Self-care and community living

Thirteen items rated on 5-point scale (acts independently to totally dependent). Example: "How much support does the consumer typically need to make and keep necessary appointments?"

Individual was rated as being "impaired in self-care and community living=1" if rated as a 4 or 5 on any of the 11 items.

AOT Evaluation Datasetii
NY OMH ACT/AOT/CM Assessment[i]: Social, interpersonal, and family functioning Nine items rated on 5-point scale (highly typical to highly atypical). Example: "How typical is it for the consumer to effectively handle conflict?"

Individual was rated as being "impaired in social functioning=1" if rated as a 4 or 5 on any of the 9 items.

AOT Evaluation Datasetii
Empowerment CMHEI Empowerment Scale[ix] Sixteen items rated on a 5-point agreement scale (strongly agree to strongly disagree). Example: "People have a right to make their own decisions, even if they are bad ones."

Mean scores were calculated, with higher scores reflecting greater empowerment.

Interview
Life satisfaction AOT Evaluation Client Interview[iii] One item rated on a 7-point scale (terrible to delighted).

Mean score calculated with high scores reflecting greater satisfaction.

Interview
Treatment satisfaction Modified MHSIP Consumer Survey[x] Nine items rated on a 5-point scale (strongly agree to strongly disagree). Example: "I was able to get all of the services I thought I needed."

Mean scores were calculated, with higher scores reflecting greater satisfaction.

Interview
Working alliance Working Alliance Inventory (WAI) Short Form[xi] Eight items rated on a 5-point scale (strongly agree to strongly disagree). Example: "The goals of my work with (provider name) are important to me."

Mean scores were calculated; higher scores reflected stronger working alliance.

Interview
AOT understanding AOT Evaluation Client Interview[iii] Twelve true/false statements. Example: "When they have an AOT order, people are required to go to mental health treatment appointments that are part of the treatment plan." (True).

Mean scores were calculated, with higher scores reflecting greater understanding.

Interview
Perceived AOT stigma AOT Evaluation Client Interview[iii] One yes/no item: "When people are under AOT, do you think that most other people think less of them?"

Individual rated as "perceives AOT stigma=1".

Interview
AOT perceived benefits AOT Evaluation Client Interview[iii] Three yes/no items. Item response positive for benefit=1. Example: "When people are under AOT, do you think they are more likely to keep their mental health or substance abuse appointments?"

Mean scores were calculated, with higher scores reflecting greater perceived benefits.

Interview
Barriers to treatment AOT Evaluation Client Interview[iii] Six true/false items reflecting both mandate- and non-mandate-related barriers. Item response positive for barrier=1. Example mandate-related barrier: "Did you delay getting help because you think that going to treatment might get you in trouble with the law?" Example non-mandate related barrier: "Did you delay getting help because you think that going for help probably wouldn’t do any good?"

Mean scores were calculated, with higher scores reflecting more barriers.

Interview
Fear of involuntary commitment AOT Evaluation Client Interview[iii] One yes/no item, "Has fear of being involuntarily committed ever caused you to avoid treatment for mental health?"

Individual rated as "positive for fear=1".

Interview
Perceived coercion Modified MacArthur Admission Experience Scale[xii] Five items rated on a 5-point scale (strongly agree to strongly disagree) assessing experiences in the previous 6 months. Example: "It was my idea to get treatment."

Mean scores were calculated, with higher scores reflecting greater perceived coercion.

Interview
Procedural justice Modified MacArthur Admission Experience Scale[xii] Six items rated on a 3-point scale (not at all, somewhat, or definitely). Example: "When you received your court order did they treat you respectfully?"

Mean scores were calculated, with higher scores reflecting greater procedural justice.

Interview
General pressures to adhere to treatment AOT Evaluation Client Interview[iii] Thirty-three yes/no items constituting 4 subscales (warnings, sanctions, medication oversight, commitment pressure) assessing experiences in the previous 6 months. Item response positive for pressure=1. Example (warnings subscale): "Did anyone tell you that you may lose your housing if you don’t follow your treatment plan?"

Mean scores were calculated for the total scale and each subscale, with higher scores reflecting greater pressure.

Interview
Pressure benefits AOT Evaluation Client Interview[iii] Nine items rated on a 5-point scale (strongly agree to strongly disagree). Lower scores reflect greater perceived benefits from pressures to adhere to treatment. Example: "Overall, the pressures or things people have done were for my own good."

Mean scores were calculated, with higher scores reflecting fewer perceived benefits of pressure to adhere to treatment.

Interview

5. Data Structure

For all analyses that used the Medicaid and AOT Evaluation Data (i.e., CAIRS/TACT) we created analytic files that contained multiple observations per person, or repeated measures over time. The analytic files created from the Medicaid and AOT Evaluation Data were similar in form; the only differences were related to timeframe in the source data (i.e., the Medicaid data could be grouped into one month intervals, while the AOT Evaluation Data were only available in six month intervals). Because of the overall similarity between the two sets of analytic files, we will provide a brief overview of how we constructed the Medicaid data only.

Our study period for the Medicaid data consisted of 88 months (i.e., between November 1999 and February 2007). Utilizing these 88 months, we created a vertical data shell or "long file" where each individual had 88 rows of data. In this data structure, each month became a separate record and each type of service event or status, such as being on AOT or not, became a separate variable. Specifically, each individual’s Medicaid claims history was examined for receipt of any Medicaid-reimbursed mental health services (e.g., ACT, ICM, inpatient days, other case management, pharmacy fills; OMH hospital days were also merged with the Medicaid data) that occurred in a given month between November 1999 and February 2007. Separate variables for each type of service were created and populated with values from the individual’s Medicaid-reimbursed services. Each of these variables spanned 88 months. We then merged in dates of each individual’s AOT order, which allowed us to evaluate the association between receiving AOT and a variety of outcomes, using repeated measures regression techniques.

6. Analytic Approaches

We used multivariable regression techniques (e.g., logistic regression for dichotomous outcomes, Poisson or negative binomial regression for count outcomes depending on the underlying distribution of the data) to estimate the relationship between AOT and various outcomes. We controlled for time and a wide array of available covariates. We also made appropriate statistical adjustments to account for the non-independence of observations that is present when estimating effects in a repeated measures model. In addition to using all available information to control for differences between subjects, we also created propensity scores that were used to weight the Medicaid data.

7. Propensity Scoring

We calculated propensity scores that were used to weight the longitudinal Medicaid data. This approach, inverse probability of treatment weighting[13], "predicts" the propensity of an individual receiving the treatment they actually received; the propensity scores are then used to weight the data. The goal of using propensity scores is to make the sample similar to a randomized experiment. We created two sets of propensity scores. The first set of scores modeled the likelihood of each person receiving an initial AOT order and the second set of scores modeled the likelihood of each person being renewed on AOT. Propensity scores from each of these models were output and used to weight the longitudinal Medicaid data to account for baseline differences in the likelihood of receiving AOT. Our propensity regression models included all available demographic and clinical variables. Additionally, the propensity models included information on medication possession ratio and prior inpatient hospitalization.

8. Multiple Imputation of Missing Data

Because the CAIRS data had a substantial amount of missing data, we used multiple imputation techniques to provide complete data for subjects. Data were imputed using SAS PROC MIwith imputations set to the default of 5. Imputing data in this manner provides less biased parameter estimates than other missing data strategies such as listwise or pairwise deletion[14]. Most of our analyses had fewer than 5% missing data.

 
  1. New York State OMH ACT, AOT, and Case Management Assessment Form is completed by case managers or ACT team staff and is collected at baseline and every six months.
  2. AOT Evaluation Dataset consists of the CAIRS database (ACT recipient data), the AOT Evaluation database (AOT recipient data), and select TACT variables.
  3. AOT Evaluation Client Interview Instrument was used by PRA staff to interview a subsample of current and past AOT recipients. This instrument consisted of standardized measures, as well as measures created for the purpose of this study.
  4. Hogan TP, Awad AG, Eastwood R. (1983). A self-report scale predictive of drug compliance in schizophrenics: reliability and discriminative ability. Psychological Medicine,13, 177-183.
  5. Steadman, H., Mulvey, E., Monahan, J., Robbins, P., Appelbaum, P., Grisso, T., Roth, L., & Silver, E. (1998). Violence by people discharged from acute psychiatric inpatient facilities and by others in the same neighborhoods. Archives of General Psychiatry, 55, 393-401.
  6. Division of Criminal Justice Services records included lifetime arrest data (i.e., both pending and disposed charges) for the 6-county AOT and EVS groups
  7. Ciarlo JA, Edwards DW, Kiresuk TJ, et al (1981). The Assessment of Client/Patient Outcome Techniques for Use in Mental Health. Contract 278-80-0005. Washington, DC, National Institute of Mental Health; Conrad KJ, Matters MD, Yagelka J, et al (2001). Reliability and validity of a Modified Colorado Symptom Index in a national homeless sample. Mental Health Services Research, 3, 141-153.
  8. Endicott, J, Spitzer, RL, Fleiss, JL, & Cohen, J (1976). The Global Assessment Scale: A procedure for measuring overall severity of psychiatric disturbances. Archives of General Psychiatry, 33, 766-771.
  9. Rogers, ES, Chamberlain, J, Ellison, ML, & Crean, T (1997). A consumer-constructed scale to measure empowerment among users of mental health services. Psychiatric Services, 48, 1042-1047.
  10. Mental Health Statistics Improvement Program (MHSIP) Consumer Survey. Ganju, V (1999). The Mental Health Statistics Improvement Program Consumer Survey. Austin, TX:  Department of Mental Health and Mental Retardation.
  11. Neale, MS, & Rosenheck, RA (1995). Therapeutic alliance and outcome in a VA intensive case management program. Psychiatric Services, 46, 719-721.
  12. Gardner, W, Hoge, SK, Bennett, N, Roth, LH, Lidz, C, Monahan, J, & Mulvey, E (1993). Two scales for measuring patients’ perceptions of coercion during mental hospital admission. Behavioral Sciences and the Law, 11, 307-322; Swartz, MS, Hiday, VA, Wagner, HR, Swanson, JW, Borum, WR, & Burns, BJ (1999). Measuring coercion under involuntary outpatient commitment: Initial findings from a randomized clinical trial. Research in Community and Mental Health, 10, 57-77.
  13. Robins, J.M., Hernan, M.A., & Brumback, B. (2000). Marginal structural models and causal inference in epidemiology. Epidemiology, 11, 550 -560.
  14. Schafer, J.L. & Olsen, M.K. (1999). Multiple imputation for multivariate missing-data problems: A data analyst's perspective. Multivariate Behavioral Research, 33, 545-571.