Summary of the Trainings Outcomes Interview Study

Summary of the Training Outcomes Interview Study^¹

Overview

This training outcomes interview project emerged from discussions of the Training Project Work Group and the Infrastructure Advocacy Work Group of the HRSA SPNS Cooperative Agreement at the May 1998 Steering Committee meeting. The groups were especially interested in understanding the specific ways that trainees use the information from trainings, up to a year or more following the training experience. Thus, the goals were to identify concrete examples to describe the long-term effects of the training experiences, particularly in the areas of patient/client care and procedural or system changes.

Drafts of the interview instrument and design specifications for sampling and other procedures were developed and prepared by The Measurement Group and were distributed for comment to participating projects in early July 1998. Nine SPNS program grantees providing HIV training to service providers participated in this study.

Telephone interviews were conducted with individuals who had participated in the training sessions held by one of the nine projects. Each project provided The Measurement Group with a list of recent trainees, along with their contact information and other relevant details about the specific trainings they attended. Six trained staff members from The Measurement Group conducted all interviews, and each interview lasted from 25 minutes to 45 minutes.

As part of the telephone interview, respondents provided some basic background information, rated the training they attended, and provided specific, concrete examples about how the training affected them in a number of domains concerning patient/client care and system change. These latter questions each had two parts – in one part participants provided examples and in the second part they rated the effect that the training had on a particular domain. Trainees were asked a general question about how the training affected how they think about HIV. Next, they were asked three questions focusing on how the training experience affected how they deal with patients/clients – specifically, in terms of how they provide services, how they communicate with their patients/clients or their families, and how they make referrals. Finally, they were asked four questions about how (if at all) the training affected how the service system in their respective area functions in terms of procedural changes, collaboration with other agencies, and large-scale system change.

Method

Sampling Design. Each project provided The Measurement Group with a list of approximately 75 names of individuals who had participated in training session(s) held by the project. Names were chosen by projects according to a common set of core criteria. Specifically, recent trainees who had participated in high intensity trainings were chosen. Details about the selection criteria that each project used to assemble respondent lists are provided in Table 1, as well as the titles of the training sessions.

In general, trainee lists were drawn from the prior year and were selected because these individuals experienced what was considered to be a modal training session from the perspective of the project.

Table 1
Selection Criteria for Initial Respondent Lists for Each Project

Project Site	When Trainees Participated	General Criteria	Training Completed	Individuals on Initial Sampling List
Cook County MCH/HIV Integration Project	Within the last year (July 1997 to July 1998)	Selected from names given by the contact person for the training Selected from individuals who were likely to be available and willing to be interviewed Mix of individuals from prenatal, labor and delivery, physicians and nurses	One or more of the following trainings: Counseling and Testing AIDS Clinical Trial Group (ACTG O76) Risk Reduction Women and HIV Peers Patient Simulation Goals Implementation CAN (Community Advocacy Network) Training Issues	81
Center for Women Policy Studies	Within the last year (July 1997 to July 1998)	Participation in major project activities, for example, Steering Committee meetings Participated in Cadre Participated in trainings since July 1997	One or more of the following trainings: HIV and Women’s Testing Women and HIV 101 HIV/AIDS 101 Training of Service Providers on Women and HIV/AIDS Case Management Training HIV 101 and STD’s Violence and Victimization in Women and HIV/AIDS	59
University of Colorado	Within the past two years (October 1996 and February 1997)	All individuals documented as having completed the Self-Study Module Training	Self-Study Module: HIV/AIDS Prevention, Early Intervention and Health Promotion: A Self-Study Module for Rural Health Providers	50
Emory University	Within the last year (March 1998 to July 1998)	All corrections health care providers who participated in HIV trainings Every third individual was chosen from attendance rosters To reduce the list further, efforts were taken to ensure that each prison site represented had more than one provider on the list	One or more of the following trainings: Orientation to HIV: Knowledge and Care Issues Pre- and Post-test Counseling for HIV Medical Management of Patients Across the HIV Disease Spectrum Diagnosis and Management of Opportunistic Infections Early HIV Symptom Recognition	75
Health Initiatives for Youth	Within the last year (August 1997 to July 1998)	Attended one of two HIFY trainings that were at least 14 hours in length Individuals who were unable to be reached were eliminated from a larger list	One of two trainings: Youth and the HIV Antibody Test HIV Prevention: From Framework to Field	75
University of Mississippi	Within the last three years (July 1995 to July 1998)	Individuals completed the required amount of trainings required for CME credit They completed 12 distance learning sessions over six months	HIV Primary Care and Computer-Based Distance Learning for Community Health Centers in Rural Mississippi	74
State University of New York	Within the last year (October 1997 to July 1998)	Obstetric and pediatric providers were selected None of the individuals received more than one training Venues were typically either grand rounds, with an audience of over 20 people, or small groups of attendings and residents	Quickie HIV Counseling for the Busy Clinician	75
University of Texas	Within the last year (June to August 1997)	Four sessions were delivered Every other individual from sign-in sheets from these four sessions was selected	Children and Their Families: The Effects of HIV Disease in Their Lives	78
University of Washington	Within the last year (March 1997 to July 1998)	Providers were from two community-based residential care facilities Attended two or more weekly site rounds Those excluded were individuals who attended only one round or who attended rounds at a Seattle-based hospital that saw very few HIV/AIDS patients	Psychiatric Management of HIV/AIDS Patients with Delirium	31

Note. Individual sites were asked to provide The Measurement Group with lists comprised of approximately 75 individuals from recent trainings, with "recent" being defined in terms of the specific characteristics of the training project. Typically, individuals were drawn from trainings held in the prior year.

A letter describing the training outcomes interview study was distributed to each site for project-specific editing. Letters were printed on project letterhead and sent to each trainee on the initial contacts list. The trainee was informed in the letter that an interviewer from The Measurement Group might be telephoning them in the next couple of weeks to discuss the impact of the training sessions on their daily practice and care for patients/clients. Interviewers did not begin calling trainees until ample time elapsed for them to have received the letter describing the study from the project director.

The potential interviewees’ names for each project list were placed in random order, and interviewers attempted to reach trainees in that order.^² Six trained staff members from The Measurement Group conducted the interviews. These interviewers completed an average of 25.0 interviews (sd = 5.1 interviews per interviewer; range 6 interviews to 70 interviews). Every attempt was made to ensure that interviewers conducted approximately equal numbers of interviews across projects.

Participants. A total of 218 individuals (24.3 percent men; 75.7 percent women) were interviewed for this study during a one-month period (August 1998).^³ Thirty-two individuals (or 12.8 percent of the individuals contacted) were excluded from all analyses because once contacted, they either declined to be interviewed and/or did not remember the training experience.^⁴

Training participants were interviewed from nine project sites: Cook County, Center for Women Policy Studies, University of Colorado, Emory University, Health Initiatives for Youth, University of Mississippi, State University of New York, University of Texas, and University of Washington. As seen in Table 2, at least 15 trainees were interviewed from each site with a maximum of 30 trainees interviewed. Each site accounted for from approximately 10.0 to 15.0 percent of the interviews conducted in the entire study. ^⁵

Table 2
Number of Trainees Interviewed by Project Site

Project Site	Number of Individuals on Initial List (N = 599)	Number (Percent) of Individuals Who Declined the Interview and/or Did Not Remember the Training (N = 32)	Number (Percent) of Individuals Interviewed (N = 218)
Cook County MCH/HIV Integration Project	81	6 (18.8)	23 (10.6)
Center for Women Policy Studies	59	6 (18.8)	21 (9.6)
University of Colorado	50	2 (6.3)	20 (9.2)
Emory University	75	3 (9.4)	28 (12.8)
Health Initiatives for Youth	75	0 (0.0)	30 (13.8)
University of Mississippi	74	1 (3.1)	29 (13.3)
State University of New York	75	7 (21.9)	15 (6.9)
University of Texas	78	5 (15.6)	27 (12.4)
University of Washington	31	2 (6.3)	25 (11.5)

Respondents were a mean age of 40.1 years old (sd = 10.3 years; range = 18 years to 78 years). They were predominately Caucasian (54.6 percent). There were slightly higher proportions of African American respondents interviewed in the sample (20.2 percent) compared to Latinos (12.8 percent). Small percentages of respondents identified themselves as either Asian American (6.4 percent), Native American (1.8 percent), or of another race or ethnicity (2.8 percent).

The educational level of the individuals interviewed for this study was generally very high. One fifth of the trainees who were interviewed indicated they had a doctorate or a professional degree (18.3 percent), and approximately one quarter of those interviewed reported that they had a master’s degree (25.2 percent). One third of the sample (33.0 percent) had at least a bachelor’s degree, whereas 23.3 percent of the sample had an associate’s degree or lower.

A number of different types of occupations were represented in this sample. Participants were predominately nurses (27.1 percent), with some program directors/coordinators (15.1 percent), medical doctors and specialists (10.6 percent), and case managers and social workers (10.6 percent). There were smaller proportions of individuals who identified themselves as administrators/managers/supervisors (7.8 percent), nurse practitioners (5.0 percent), support staff (4.1 percent), educators/trainers (3.2 percent), counselors (4.1 percent), outreach workers (1.8 percent), physician assistants (0.5 percent), dentists (0.5 percent), and other specialities (2.8 percent).

Participants in this study had professional degrees in nursing (28.9 percent) and medicine (13.3 percent). A substantial proportion of those interviewed indicated that they had no specific professional credentials (29.8 percent). There were small proportions of advanced nurse practitioners (6.0 percent), dentists (3.7 percent), LCSW/social workers (7.8 percent), individuals with a masters degree in counseling (2.3 percent), physician assistants (0.9 percent), attorneys (0.9 percent), and individuals with other professional credentials such as clergy or educators (3.2 percent).

The type of organization where trainees worked was diverse and included government agencies (18.3 percent), hospitals (14.7 percent), correctional facilities and prisons (11.9 percent), ambulatory/outpatient/free-standing clinics (6.9 percent), community health centers (8.3 percent), long-term care/skilled nurse facilities (9.2 percent), and community-based organizations (9.6 percent). Some trainees were in private practice (5.5 percent), worked in educational settings (2.8 percent), were not involved in health care (3.7 percent), worked at home health/visiting nurse organizations (1.1 percent), or at other types of organizations (2.3 percent).

On average, respondents indicated that they worked in their current position for 4.19 years (sd = 4.82 years; median = 2.50 years) and as a HIV/AIDS service provider or administrator for 5.93 years (sd = 4.98; median = 5.00 years).

Most service providers who were interviewed for this study (82.1 percent) indicated that they provide direct services to patients or clients, and more than half of the sample (56.4 percent) reported that they provide education or training to others on topics related to HIV/AIDS. One third of those interviewed (32.1 percent) reported that they were a program administrator or manager. Out of the program administrators and managers, most reported that they were program administrators for a medical program (57.1 percent) as opposed to a social services program (20.0 percent), an education program (10.0 percent), or another type of program (12.9 percent).

Approximately equal proportions of program administrators and managers were interviewed at each of the project sites. The proportion of respondents who offer HIV education to others differed significantly across sites (c²(8, N = 218) = 24.35, p < .003), with the actual number of educators being greater than expected at one site. The proportions of individuals interviewed who provide direct care to patients/clients also varied by site (c²(8, N = 218) = 29.10, p < .001), with two sites having generally more direct care providers than expected and one other project site having fewer represented in the sample.

The Interview. The interview has three general sections and is provided in Appendix A. In the first section, participants were asked basic background information about themselves, their current position and the number of years they had spent in that position. They were also asked to provide global ratings about the training in which they participated in terms of how valuable they felt the training was and whether they had recommended it to others.

For the final question of the first section, participants were asked to think broadly about how the training session(s) affected how they think about HIV, if at all. They were then asked to rate whether this change was positive or negative and to justify their decision about the impact.

The second section of the interview focused on changes that the training contributed to in patient/client care. Respondents were asked to indicate how the training affected how they provide care to patients/clients, how they inform patients/clients or their families about care options, and how they make referrals. For each question, they were asked to provide specific examples, to rate the overall effect that the training had in this domain (positive or negative), and to justify their rating with specific examples from their experience.

The final interview section probed for whether the training influenced changes in how the system functions or how procedures are put in place at the system or organizational level. That is, trainees were asked whether existing procedures, guidelines, and how care is provided may have changed as a result of the training. The areas of interest included whether the training improved or changed the way care is delivered in general; the way people educate others about care; and the way service providers interact with each other and provide services. They also were asked a more broad question about whether they could provide any examples about how the training improved or contributed to large-scale changes in the way HIV-related care is provided. For each general area, respondents were asked to rate the extent of positive or negative impact the training had and to justify those ratings with specific examples.

Before the close of the interview, respondents were also given the opportunity to provide any additional comments about their experiences during, or as a result of, the training. As a token of thanks, participants were offered to have a summary report of the training interview findings to be sent to them when it is available and/or a copy of The HeART of Training Manual.^⁷

Results

General Assessment of Training Effects

Over half of the respondents (52.1 percent) indicated that they had recommended the training to several people, and the large majority indicated that they had recommended the training to at least one other person (83.4 percent).

On a scale ranging from "not at all valuable" (1) to "extremely valuable" (5), nearly three-quarters of the training participants interviewed (73.4 percent) said that they believed the training they attended was either "very valuable" or "extremely valuable." The mean rating was 3.91 (sd = .81), meaning that, on average, participants tended to think the training was very valuable.

Assessments of Training Effects in Specific Content Areas

Respondents were asked to think about the specific training experience(s) they had with the project, with a specific focus on providing examples about how the experiences may have changed or improved how they think about HIV, deal with patients/clients, and how the system functions. For each of these areas, they rated the extent to which they felt the training had an effect.

Participants were first asked to rate how the training experience(s) affected how they think about HIV. Next, they rated the effects of the training experience(s) in three areas of patient/client care: (1) in providing services, (2) in educating patients/clients and their families about their options for care, and (3) in their referrals or referral patterns for patients/clients. Finally, respondents rated how the training experience(s) affected systemic operations – that is, how procedures, guidelines, and the way services are provided changed as a result of the training. Four areas of possible system change were probed including how the training affected how: (1) the system, in general, offers care; (2) the system can offer educational or training opportunities; (3) the system encourages establishing collaborations among service agencies; and (4) HIV-related care is provided overall.

For each area, participants also rated their perceptions of the effects of the training experience(s) on each of these areas. They used a seven-point Likert-type scale ranging from "Large negative effect; made it much worse" (-3) to "Large positive effect; made it much better" (+3), with a midpoint of "No effect; didn’t make it better or worse" (0). After each rating, participants were asked to state why they decided to make the rating that they did.

Table 3 presents descriptive information about how participants rated the various training effects. The first column of the table shows the means and standard deviations of the ratings in each of the domains. The last two columns of the table provide information about the rating distributions. These last columns are, respectively, the proportion of individuals in the sample who indicated that they thought the training experience(s) had no effect and the proportion of individuals who thought the training experience(s) had some type of positive effect on them (small, medium, or large). Both columns provide valuable information about the perceived effect of the training experience regarding the particular domain in question.

Table 3
Mean Rating of Training Effect and Proportions of Participants Indicating that Training Effects Had No Effect or Had a Positive Effect.

Rating of Training Effect Overall, how would you rate the effect that the training experience(s) had on…	Training Effect Rating mean (sd)	No Training Effect (Percent)	Small, Medium, or Large Positive Training Effect (Percent)
1. How you think about HIV? (n = 218)	1.63 (1.25)	28.4%	70.6%
2. How you provide services to your patients/clients? (n = 195)	1.86 (1.13)	21.0%	78.9%
3. How you educate patient/clients or their families about their options for care? (n = 187)	1.66 (1.21)	30.5%	69.6%
4. How you make referrals for your patients/clients? (n = 180)	1.06 (1.23)	52.2%	47.8%
5. How the system, in general, offers care? (n = 175)	1.53 (1.18)	30.9%	69.1%
6. How the system, in general, can offer other education or training opportunities? (n = 160)	1.59 (1.16)	28.8%	71.3%
7. How the system, in general, encourages establishing collaborations among service agencies? (n = 147)	1.57 (1.19)	28.6%	70.8%
8. The way HIV-related care is provided? (n =148)	1.37 (1.29)	40.5%	58.8%

Note. Means greater than zero denote a positive effect due to the training. Means less than zero denote a negative effect due to the training. Means near zero denote that there was no effect due to the training. The rating scale for each question about the effects of the training ranged from "Large negative effect; Made it much worse" (-3) to "Large positive effect; Made it much better" (+3). The midpoint of the seven-point rating scale was "No effect; Didn’t make it better or worse" (0). Ratings are reported only for those respondents who believed that the question applied to them.

One of the first aspects to note about Table 3 is the relatively high positive means for all domains that were assessed, typically between small and medium positive effects. The highest positive change as a result of the training experience(s) was associated with how the trainees now provide services to patients/clients (mean = 1.86; sd = 1.13). This result is also observed with the relatively low proportion of respondents who indicated that the training had no effect (21.0 percent) and the corresponding high proportion that indicated that the effect was positive (78.9 percent). The means were between 1.37 to 1.66 for all other domains, with the exception of referrals to clients with a mean of 1.06 (sd = 1.23). The referral domain had the highest proportion of individuals who indicated that the training had no effect (52.2 percent). These data suggest that the training experience(s) tended to have slightly less of an impact on how providers make referrals to patients/clients compared to the other areas of patient/client care, general views on HIV, and the assessments regarding how the system changed. Thus, on average, all means shown in Table 3 are relatively high (above the midpoint of the effect continuum).

To examine how ratings of training effect are associated with one another, we examined the interrelations among these domains. Table 4 provides the associations among the specific areas addressed in the interview: General Thoughts about HIV, Patient/Client Care (providing services, informing patients/clients or families about care options, referring clients/patients), and System Functioning (service delivery, educating others, collaborations with other agencies, and large-scale change).

Table 4
Interrelations among the Areas Where Training May Have Had an Effect

		Patient/Client Care				System Functioning
	Thoughts about HIV		Providing Services	Informing Others	Referral Patterns	Service Delivery	Education of Others	Collaborations	Large-Scale
Thoughts about HIV	1.00
Providing Services	.34		1.00
Informing Others	.31		.44	1.00
Referral Patterns	.30		.36	.32	1.00
Service Delivery	.19		.40	.34	.44	1.00
Education of Others	.29		.39	.31	.48	.44	1.00
Collaborations	.25		.36	.30	.37	.51	.49	1.00
Large-Scale Change	.17		.38	.34	.40	.48	.54	.55	1.00

Note. Sample sizes for the pairwise comparisons range from 114 to 195. Correlation coefficients above |.25| in this table are statistically significant at the p < .01 level. The correlation coefficient of .19 is statistically significant at the p < .05 level. Respondents only provided a rating if they felt the main question applied to them. They were asked: Overall, how would you rate the effect that the training had on…? Why is that? Ratings could range from "Large negative effect" (-3) to "No effect" (0) to "Large positive effect" (3). The top panels in this table denote different interview sections.

As might be expected, perceptions about the effects of the training experience(s) across most areas were positively associated with one another. Thus, in general, there tends to be common patterns of ratings across the areas. Higher associations are seen among areas within domains with the same general focus, such as Patient/Client Care areas (providing services, informing others, referral patterns) and System Functioning areas (service delivery, education of others, collaborations, large-scale change). The smallest associations are seen with the first column of correlations with the impact of the training on how one thinks about HIV and specific change in other areas of Patient/Client Care and System Change (range of correlations: r = .17 to .34). The interrelations among the Patient/Client Care areas range from r = .30 to .48. The system impact correlations among ratings are even stronger, ranging from r = .44 to .55.

Do Provider Characteristics Predict Ratings of Training Effectiveness?

To examine to what extent provider background characteristics affected ratings of training impact, a series of univariate and multivariate analyses of variance were performed. Each analysis was developed to deal with specific parts of the interview.^⁸

How One Thinks about HIV. In the first set of analyses, the outcome variable was the training effect rating for how the training affected how providers think about HIV (from large negative effect to large positive effect). Several univariate general linear models were considered to evaluate possible effects of several predictors. These predictors included gender, age, number of years in the current position and in the HIV field, and whether or not the trainee was a program administrator, a provider of HIV education, and/or a direct service provider to patients/clients. Overall, these analyses showed no statistically significant differences as a function of gender, age, being a program administrator, providing HIV education, or providing direct services.

Patient/Client Care. The next set of tested models involved the three ratings of training effectiveness that specifically dealt with how providers offered patient/client care. Respondents were asked to rate the extent to which the training experience(s) affected how services were provided to patients/clients, how they inform patients/clients or their families about HIV-related issues, and how they refer patients/clients to services.

Using these three outcome variables, the effects of gender, age, and job status as a program administrator, an individual who educates others about HIV, and an individual who provides services were examined in separate models. The findings show that in the Patient/Client Care domain, no statistically significant differences emerged as a function of respondent gender, age, number of years at current position or working in the HIV field, being an educator in the HIV field, or providing direct services.

However, whether or not providers indicated that they were a program administrator significantly predicted the rated training impact in the Patient/Client Care domain (F(3, 156) = 3.27, p < .03, eta-squared = .06). Specifically, the effect can be localized to the effect the training experience(s) had on how the respondent makes referrals (F(1, 158) = 7.16, p < .008, eta-squared = .04). In the area of making referrals for patients/clients, program administrators rated the training has having a more positive effect (mean = 1.41) than non-program administrators (mean = .86) on the scale ranging from large negative impact (-3) to large positive impact (+3).

System Functioning. The final set of analyses concern the four rating scales of training effectiveness that focus on how the service system around the respondents function after the training. Trainees rated the extent to which the training experience(s) affected how services are delivered, how others are educated, whether collaborations were fostered, and whether there were any more large-scale changes that resulted.

A series of multivariate models tested how respondent background characteristics predicted these four ratings of system functioning. The findings show that ratings were not affected significantly by respondent gender, age, years at the current position, years working in the HIV field, or whether or not the trainee was a program administrator, an educator in the HIV field, or a direct care service provider.^⁹

However, whether or not providers indicated that they were a program administrator significantly predicted the rated training impact in the Patient/Client Care domain (F(3, 156) = 2.66, p < .05, eta-squared = .05). Specifically, the effect can be localized to the effect the training experience(s) had on how the respondent makes referrals. In the area of making referrals for patients/clients, program administrators rated the training as having a more positive effect (mean = 1.26) than non-program administrators (mean = .73) on the scale ranging from large negative impact (-3) to large positive impact (+3).

Open-Ended Comments from Participants

All participants were asked to comment on a number of different topics and rate regarding how they believed the training experience(s) affected how they think about HIV in general, how it affected how they currently serve patients/clients (three questions in this domain), and how it affected how the overall care system in their area functions (four questions in this domain). General categories have been developed and comments to each question are given by coding category.

Note that The Measurement Group and the Training and Infrastructure Workgroups will further group these open-ended responses based on identified themes. The themes presented here are very preliminary categorizations made on the basis of general content and will need to be subjected to a formal coding by independent raters and psychometric reliability analysis. We have made these categorizations to provide a general sense of the key themes present in these responses.

As seen in the prior section (Table 3), for each general question about the possible changes due to the training, respondents were asked to rate the overall effect of the training in that area. They were also asked to justify their rating with specific comments and examples. These open-ended comments are categorized according to the positive or negative extent of change, as rated.

Thoughts About HIV. Respondents were asked: Overall, how did the training that you attended affect how you think about HIV?

Several themes were identified in the comments made by participants when they indicated how the training affected how they think about HIV. Appendix B-1 provides all open-ended responses (sorted by coding category) for this question.

The major themes in the open-ended responses are as follows:

General Information about HIV. I learned general information about HIV.

"Before, I didn’t know much about HIV, only what I heard in school. After the training I knew tons of information and could answer people’s questions about HIV/AIDS."

Specific Information for Patient/Client Care. I learned specific information that helps me deal with HIV patients/clients better; I have increased knowledge about specific skills and options.

"I learned about the medical management and about striving for simplified medication regimens. The trainer pointed out possible contributor to symptoms, be it medical or psychological of dementia versus delirium."

Heightened Awareness. The training raised my awareness; I see things differently now.

"Now I am more aware and sensitive to issues that women with HIV/AIDS must face, especially with regard to the violence they endure because of their HIV status."

Greater Comfort, Confidence, and Compassion in HIV Care. The training made me more comfortable and gave me confidence with patient/client care.

"It made me more open to HIV – I am no longer fearful, and I can better deal with the issues of HIV/AIDS because of the information I obtained in the training."

Insight and Validation. The training gave me insight and was validating for my own work; it was a refresher for me.

"The rounds were educational. They gave me insight to what I am doing. I see some of the things that I do validated after having attended the rounds."

Changed Attitudes and Beliefs. The training changed my attitudes and opinions about HIV and dispelled myths I had about HIV.

"It gave me a lot of enlightenment about HIV, and it dispelled some of the myths I once believed."

No Effect Because Already Knowledgeable; The "Resistant Trainee." The training did not affect how I think about HIV because I know quite a bit already.

"HIV has never been a problem with me. I have never had a fear of it. I don’t see it as a stigma."

No Effect. I did not learn anything; I don’t know.

All respondents were asked to rate the effect that the training had on how they think about HIV and then justify their rating. Appendix B-2 shows the verbal justifications made for the training of how much the training affected their views (sorted by each rating category). Table 5 provides a summary of representative responses about training effect by rating.

Table 5
Representative Responses for How the Training Affected Respondent Views About HIV
(by Rating Category)

Training Effect Rating	Representative Open-Ended Responses^¹¹
Large negative effect (made it much worse)	No responses given
Medium negative effect (made it worse)	N = 1 The disease is very prevalent. It is a condition you can easily encounter, especially among sexually active youth. 45
Small negative effect (made it slightly worse)	N = 1 The information made me more depressed about what's going on with HIV. 88
No effect (didn’t make it better or worse)	N = 62 It didn't change how I think about HIV because I’ve always had an open mind about it. 5 More awareness is a good thing, although the disease is devastating. I can't rate the effect the training had on how I think about HIV because the positive and negatives balance out. 90 I had previous training. I have been in situations where we discussed the issue of HIV. I have read literature and seen statistics. I typically see a lot of the statistics on the city where I live. I got tidbits here and there. What I learned in the training I had heard before. 193
Small positive effect (made it slightly better)	N = 22 I was already familiar with the disease itself. I know a lot of people who have it. The training taught me more about the medications for HIV/AIDS. 58 I was new when I went to the training, so I was a "fresh sponge getting new information." 148 The things that changed were that I learned the statistics, about the different strains, and the possible treatments, since the disease is ever-changing. The training also taught universal precautions, which I already knew. 191
Medium positive effect (made it better)	N = 60 The questions the trainer asked and the issues about the care she brought up, for example the medications, were all valuable questions, and they make you stop and think about care in a different way. 14 It gave me further insight regarding how to deal with women at-risk, and it taught me how to convince them to get tested, and how best to explain the importance and results of the test. 61 It increased my awareness of the number of youth exposed to HIV/AIDS. It increased my sensitivity to how to do pre- and post-test counseling. 170
Large positive effect (made it much better)	N = 72 I am able to pinpoint certain behaviors, and I have the ability to say that they are part of the dementia-type of behaviors. I am able to identify dementia type behaviors better for their care. I know better what's going on. 10 It gave me a better understanding of transmission and the necessary precautions to prevent transmission. 80 It showed me how to put my own feelings aside when teaching others about HIV/AIDS. It allowed me to reach others with information and give them the opportunity to see how they can educate themselves and others. 143

Note. N = 189. Respondents only provided a rating if they felt the main question applied to them. They were asked: Overall, how would you rate the effect that the training experience(s) had on how you think about HIV? Ratings ranged from "Large negative effect" (-3) to "No effect" (0) to "Large positive effect" (3). The average rating for this sample was 1.63 with a standard deviation of 1.25.

Training Effects on Patient/Client Care

Patient/Client Care: Providing Services. Respondents were asked in the second section of the interview: How has the training session(s) you attended improved or changed the way you provide care to patients/clients? Please tell me a specific example from your own experience about how you changed your provision of care to patients/clients as a result of the trainings. The major categories in the open-ended responses from this question are summarized below, and all open-ended responses are given Appendix C-1.

The major themes in the open-ended responses to the question about how the training affected how care to patients/clients is provided are as follows:

More Ways to Educate Others. I developed new activities to do with patients/clients; I learned new ways to express ideas about HIV to others.

"I learned how to develop activities specifically for the clients I am targeting. I learned how to make programming more accessible and enjoyable for my clients. It taught me what kinds of things to think about when talking to different populations."

Greater Comfort and Confidence. It made me more confident and patient in how I care for my patients/clients.

"It made me more comfortable with talking about testing and bringing the topic up first with youth."

More Knowledge about How to Treat Patients/Client. I changed how I treat patients/clients; I provide more sensitive care; I have more tools and more knowledge.

"Now we are more aware, we push testing more, and have more information available in the office for patients to take with them."

Greater Compassion. I provide more compassionate care.

"It heightened my degree of empathy for the total effect that the disease has on the lives of those affected by it and their families."

Protocols and Procedures. I changed my protocols and procedures.

"Now we have a completed protocol for education, training, treatment, and follow-up."

Universal Precautions. I use universal precautions.

"We don’t have any specific AIDS patients, but it made me realize that I needed to use standard precautions all the time. I can’t pick out who has the virus."

No Effect. I did not learn anything; I don’t know.

Table 6 provides responses from trainees when they were asked to justify why they rated the training effect as they did (as positive, negative, or no effect) in the area of patient/client care. Appendix C-2 provides the actual open-ended responses (sorted by rating).

Table 6
Representative Responses for How the Training Affected How theRespondent Provides Services to Patients/Clients (by Rating Category)

Training Effect Rating	Representative Open-Ended Responses¹²
Large negative effect (made it much worse)	No responses given
Medium negative effect (made it worse)	No responses given
Small negative effect (made it slightly worse)	No responses given
No effect (didn’t make it better or worse)	N = 41 I still treat all my patients the same regardless of their HIV status. This is just how I practice and deal with all patients, always. The training didn't have anything to do with this. 39 I have had no opportunity to utilize it. 120 The care hasn't changed because the clients still get the basic needs. 136
Small positive effect (made it slightly better)	N = 17 I am now more aware that AIDS is of epidemic proportions, and I need to make my patients aware of this as well. 41 I haven't had to use the training very often because I don't see a lot of HIV-positive youth. 101 Things covered in the training presented really good ideas, and I refer back to the manual they gave us every so often when I need information. 168
Medium positive effect (made it better)	N = 65 It really helped create the programming for my clients. I learned concrete ways for doing so, and it helped me create a better "fit" of the program to my clients. 1 It extended the amount of information I give to my patients regarding HIV counseling and testing. I discuss the positives and the negatives associated with HIV counseling and testing and of taking AZT when pregnant (the benefits/drawbacks for both the mother and the fetus). 76 The follow-up session on counseling was a surprise. The role-playing was very good because I saw personally how providers' personal philosophies interplay with taking care of HIV patients. The role-playing gave me deep insight into how to approach patients. The role-playing was so real and gave me practice for real-life situations. 144
Large positive effect (made it much better)	N = 72 I never thought about giving HIV test results before the training. Then, I completed the training and had to start giving results. I am more aware of the process and how to be focused on the client's reactions rather than on what I'm doing. 2 You need guidance, as a doctor. This training was a refresher. It made me more confident to continue treatment with patients. 32 The new medications have allowed for set treatment plans and patients can progress instead of whither. Before the training, you don't know the effects of the medications, but afterwards, you have a good understanding. 165

Note. N = 170. Respondents only provided a rating if they felt the main question applied to them. They were asked: Overall, how would you rate the effect that the training experience(s) had on how you provide services to your patients/clients? Why is that? Ratings ranged from "Large negative effect" (-3) to "No effect" (0) to "Large positive effect" (3). The average rating for this sample was 1.86 with a standard deviation of 1.13.

Patient/Client Care: Informing Patients/Clients or Their Families about Options. In the area of patient/client care, respondents were asked to provide concrete examples (if relevant) about how the training session(s) they attended improved or changed the way they inform patients/clients or family members about HIV/AIDS issues relevant to them. The full set of open-ended responses to this question are provided in Appendix D-1, and the major coding categories identified from the themes in these responses are given below.

The major themes identified in the open-ended responses to the question about how the training affected how patients/clients or their families are informed about options are as follows:

Heightened Awareness. I am better able to inform others because I have a heightened awareness of the issues now, more insight, and perspective.

"It made us more aware that AIDS is everywhere, that no one is exempt from this disease."

More Informed. I am now better able to offer relevant information.

"Now, there are options I can inform patients about, which lets them know they have a new quality of life and it helps the patients cope with their diagnosis."

Changes in Resources, Procedures, or Structure. The training expanded our ability to inform patients/clients or their families by establishing systems to foster discussion.

"It expanded our resource base. Now, we have more items being shared and more places to get updated information. There is more sharing of information and constant updates."

More Confident and Sensitive in Discussions. The training made me more comfortable and confident in how I discuss relevant HIV issues with patients/clients or their families.

"It made me more open and comfortable when discussing the subject as a ‘matter of fact’ issue related to general care. It is part of the medical information exchanged between doctors and patients, so now I don’t ‘tip toe’ up to this topic any longer when talking to my patients."

No effect. The training had no effect.

The effect of the training on how respondents inform patients/clients or their families about options was rated and representative responses from a question about why they made the rating they did is shown in Table 7. Appendix D-2 provides the actual open-ended responses (sorted by rating).

Table 7
Representative Responses for How the Training Effected the Way the Respondent Educates Patients/Clients/Family Members (by Rating Category).

Training Effect Rating	Representative Open-Ended Responses¹³
Large negative effect (made it much worse)	No responses given
Medium negative effect (made it worse)	No responses given
Small negative effect (made it slightly worse)	No responses given
No effect (didn't make it better or worse)	N = 57 I talk to patients to give them a sense of acceptance. I don't inform them of their options for care. 4 I've been doing it from before the training. 30 I only have one case with HIV, so I don't have the role of educating clients. 68
Small positive effect (made it slightly better)	N = 9 The training gave me some ideas about harm reduction and how to approach street youth and inform them about making changes in their lives. 2 Even though you give patients the information, they still deny healthcare and the options for care. 150 Before I went to the training, I didn't know anything. Afterwards, I knew about services available for HIV patients. 187
Medium positive effect (made it better)	N = 62 I am much more aware of side effects and the seriousness of side effects. A provider can give better treatment if she/he is better educated. 178 There is more information available for the patient. A lack of information can compromise a relationship with a patient. 180 I gained better insight of the disease itself and greater information on resources available that I can refer my patients to. 203
Large positive effect (made it much better)	N = 59 Service (HIV counseling and testing) is now available to patients. 113 We have the ability of linking our medical care to social services so that there are more and better options for our patients in terms of care. 114 During the training there was a demonstration (role-playing), which I have applied to the way I talk to my patients. The role-playing allowed for self-critiquing. 127

Note: N =163. Respondents only provided a rating if they felt the main questions applied to them. They were asked: Overall, how would you rate the effect the training had on how you educate patients/clients/family members? Why is that? Ratings ranged form "Large negative effect" (-3) to "No effect" (0) to "Large positive effect" (3). The average rating for this sample was 1.66 with a standard deviation of 1.21.

Patient/Client Care: Referral Patterns. To assess whether respondents perceived change in how they refer patients/clients, they were asked: How has the training session(s) you attended improved or changed the way you refer patients/clients to other services? Please tell me a specific example from your own experience about how the training(s) affected your referrals/referral patterns. All open-ended responses for this question are presented in Appendix E-1.

The major themes from the open-ended responses on changes in referral patterns are as follows:

Discovered New Referral Options. I have learned about the greater treatment options so I can make better referrals.

"Now, we do more referrals because in the training we learned about different types of services, such as housing, the Ryan White CARE Act, and the psychological issues patients may experience. We now know that we can utilize social workers to identify resources in the community."

Heightened Awareness. I am more aware about the importance of the referral process.

"I am more sensitive to how my clients feel about going to a referral. I had a case where a girl had a sexually transmitted disease and I went with the girl to the referral and was supportive of her. In the past, I may not have gone with the girl. I realized that it’s not just about referrals, but being supportive."

Greater Confidence. I am more confident and more efficient with the referrals I make.

"It improved because of the networking with other agencies. I make more referrals now. I feel more comfortable reaching out to different people and places when making referrals."

Do Not Provide Referrals. I do not have an opportunity to provide referrals in my work.

"I don’t think it has changed. The primary care physician does the referrals."

No Effect. The training did not affect how I make referrals.

Table 8 lists representative responses that trainees gave after rating the effect of the training on how the participant makes referrals to patients/clients. These responses are shown for each rating category, and Appendix E-2 provides the actual open-ended responses for each of these categories.

Table 8
Representative Responses for How the Training Affected Respondents’ Referrals/Referral Patterns for Patients/Clients (by Rating Category)

Training Effect Rating	Representative Open-Ended Responses¹⁴
Large negative effect (made it much worse)	No responses given
Medium negative effect (made it worse)	No responses given
Small negative effect (made it slightly worse)	No responses given
No effect (didn’t make it better or worse)	N = 94 I don't make referrals. 4 I had already established a fairly extensive referral network before the training that was more than satisfactory. 48 There are no HIV patients to refer to services. 172
Small positive effect (made it slightly better)	N = 18 Because of our referral services, people can be treated at the university. Now there's people there we can talk to. Before, we were knowledgeable, but now, we know people at these agencies and it makes referrals easier. 24 I had a good network of resources already, but now I have a few more resources where I can refer clients. 90 Now we have a more educated view of the places we can refer clients. 148
Medium positive effect (made it better)	N = 32 There is now more effectiveness in my referrals. I know where to send clients and how to move the process along more quickly now. 49 We were made aware of which programs offer the most appropriate services. We were able to refine our referral pattern. 114 Patients understand more about where they can go to get services. 150
Large positive effect (made it much better)	N = 36 Now we have a referral network; before there was nothing. 17 We are now able to keep our patients in our community longer and can provide the care they need so that our patients don't have to go somewhere else any more to get HIV/AIDS-related care. 65 It opened my eyes to the wealth of resources in my area. 70

Note. N = 157. Respondents only provided a rating if they felt the main question applied to them. They were asked: Overall, how would you rate the effect that the training experience(s) had on your referrals/referral patterns for your patients/clients? Why is that? Ratings ranged from "Large negative effect" (-3) to "No effect" (0) to "Large positive effect" (3). The average rating for this sample was 1.06 with a standard deviation of 1.23.

Training Effects on System Functioning

System Functioning: Service Delivery. For the first question of this section, respondents were asked to provide concrete examples of how the training session improved or changed the way care is delivered in general. They were asked to give examples about how the training experiences affected how care is delivered at the system-level. The full set of open-ended responses to this question is provided in Appendix F-1.

Examination of the open-ended responses about service delivery changes revealed the following major themes:

Heightened Awareness. The training made us more aware about the importance of service delivery.

"I brought up the need to have all staff participate in this training because it would be beneficial to helping our clients."

Greater Confidence and Sensitivity. We are now more confident and sensitive when talking with patients/clients.

"There has been an increase in staff sensitivity toward client needs. The medical information helped us know that we can actually provide the care without having to automatically refer them to the University medical center anymore."

More Resources and Knowledge. From the training we now have more resources, options, and overall knowledge about service delivery.

"It showed us additional tools that we could use to fine tune our existing processes that are already in place for educating service providers."

More Staff Collaboration. We are now more team players when delivering care; we collaborate more.

"The rounds improved the way care is delivered a lot. The staff work together more as a group. The staff looks at things differently and talks about behavioral situations that arise. I learned how to speak to another staff member when a behavioral situation arises."

Change of Procedures and Policies. We have changed our procedures and policies around delivering care as a result of the training.

"My organization didn’t have a program before. It is a great improvement in the quality of care because there was never a program for HIV in place before. In terms of HIV, there was 100 percent no testing of the patients before. Now we screen all of our patients."

More Health Precautions. We take health precautions now.

"Care delivery is better in terms of our workers, line staff and ancillary staff, and accidental exposures to HIV (for example, needle sticks). The training reinforced our safety guidelines and what to do when accidents occur."

No Effect. The training had no effect on how care is delivered in general.

Training effects in the area of service delivery were rated by respondents and justified in verbal responses. Table 9 shows representative responses from trainees about why they rated the training effectiveness as they did. Appendix F-2 provides the actual open-ended responses to the question about the training effectiveness (sorted by rating).

Table 9
Representative Responses for How the Training Affected How the Respondent Delivers Services in General (by Rating Category)

Training Effect Rating	Representative Open-Ended Responses¹⁵
Large negative effect (made it much worse)	No responses given
Medium negative effect (made it worse)	No responses given
Small negative effect (made it slightly worse)	No responses given
No effect (didn’t make it better or worse)	N = 54 What the training basically did was allow me to feel more comfortable with youth. It didn't change how we provide services. It just took out rigidity. Either you qualify for services or you don’t by being HIV positive. 25 No effect because I am not responsible for how the system is organized. 39 We still offer the same high level of care to our patients.76
Small positive effect (made it slightly better)	N = 20 It was the combination of this training and another training that I attended. The system was already good before this training. The trainings helped make the programming a little better by giving concrete examples of ways to make the program a better "fit" for the clients. 1 It helped improve what we were doing. There were no great revelations. It emphasized everything by having people hear it again.130 Staff can provide more counseling and testing because we have more staff with knowledge to do it because of their participation in the training.147
Medium positive effect (made it better)	N = 56 Staff are more loving, caring, and patient with clients now that they know why patients behave as they do when they have delirium and other AIDS-related mental health issues.37 Counseling and testing is now a routine procedure done with all patients. Now our process continues to improve. We have more efficient tools and processes, which equal fewer patients falling through the cracks.113 Because of the training, more employees are capable of counseling than before. We now have more than the original 3-4 staff members who can do counseling since they learned about it in the training.146
Large positive effect (made it much better)	N = 45 The staff works together during rounds, which gets everyone talking. Effective communication didn't happen before the rounds were put in place. There is much more cooperation and better communication among staff now. 3 I learned that HIV/AIDS patients are the same as any other patients. That is, our staff needs to be cautious and provide help to people with HIV/AIDS as they would for anyone else. 62 We were able to develop task forces for HIV and positively impacted our standards for HIV counseling and testing. The training enabled us to develop a standard pediatric plan. 112

Note. N = 153. Respondents only provided a rating if they felt the main question applied to them. They were asked: Overall, how would you rate the effect that the training experience(s) had on how the system, in general, offers care? Why is that? Ratings ranged from "Large negative effect" (-3) to "No effect" (0) to "Large positive effect" (3). The average rating for this sample was 1.53 with a standard deviation of 1.18.

System Functioning: Educating Others about Care. Respondents were next asked: How has the training you attended improved or changed the way people in your area educate others about care. They were asked to provide a specific example from their own experience that shows how the training affected how others in the service system are trained. Appendix G-1 provides the open-ended responses for this question on educational opportunities.

Coding categories for this open-ended question on educational opportunities are as follows:

Heightened Awareness and Improved Communication. The training made us more aware of possible educational opportunities focusing on HIV.

"I have a greater awareness now that staff needs ongoing training because the disease and its treatments change so often."

Increased Education. The training made me able to pass on the information and educate others.

"I teach my staff everything I learn about HIV/AIDS every time we have one of our monthly staff meetings. Students in our clinic have used the notes from the training to educate themselves, so this has brought a lot of studying among staff regarding HIV/AIDS. The staff, including the doctors, now goes on the Internet to learn more about HIV and this is definitely new."

Greater Confidence in Patient/Client Care. The training made us more confident in general.

"It benefited the nursing staff greatly since they are more knowledgeable. There is better compliance with treatment programs, medications, etc. They give more information to patients."

Changed Policies or Procedures. We have modified our procedures because of the training.

"We continue to place emphasis on standards of care and getting everyone to implement the same standard of care. The training gave us information on getting all hospitals on the same page in terms of HIV care."

No Effect. There has been no change in how people in our area educate others.

The effect of the training on how respondents educate others in the area was rated, as well as the representative responses from a follow-up question about why they made the rating they did, are presented in Table 10. Appendix G-2 provides the actual open-ended responses (sorted by rating).

Table 10.
Representative Responses for How the Training Affected How the System, in General, Can Offer Other Educational or Training Opportunities (by Rating Category)

Training Effect Rating	Representative Open-Ended Responses¹⁶
Large negative effect (made it much worse)	No responses given
Medium negative effect (made it worse)	No responses given
Small negative effect (made it slightly worse)	No responses given
No effect (didn’t make it better or worse)	N = 46 We already had in-house trainings on various topics even before the training. 77 The way the administration is set up at this point, there are no guidelines or procedures for system-wide training. Attending trainings are done on an individual basis only. 92 Not everyone at work had the training. 147
Small positive effect (made it slightly better)	N = 16 The system in place is one in which it tends to be conservative and moves forward with little baby steps so that there are gradual changes occurring in the system. 48 It hasn't impacted everyone. Some are less receptive, but those are the ones who didn't attend the training. 53 The further we get from the trainings, the less likely we are to remember what we learned. 120
Medium positive effect (made it better)	N = 56 The training gave me a lot of information to speak comfortably about HIV, which I didn't have before the training. 40 It stressed the importance of continual, ongoing training for staff. 113 The more education we get, the better we can inform people. 199
Large positive effect (made it much better)	N = 42 I am more able to inform patients on various aspects of the disease such as prevention, treatment, referrals, and follow-ups. 45 Our staff now portrays the disease in a positive light. We are not afraid of people with HIV/AIDS nor of contracting the disease through casual contact/interactions. The staff now realizes more that "people with HIV/AIDS are people too." 50 The group is allowing educators to be used in a variety of roles to refine the individual processes of the different types of staff in different programs. 114

Note. N = 141. Respondents only provided a rating if they felt the main question applied to them. They were asked: Overall, how would you rate the effect that the training experience(s) had on how the system, in general, can offer other educational or training opportunities? Why is that? Ratings ranged from "Large negative effect" (-3) to "No effect" (0) to "Large positive effect" (3). The average rating for this sample was 1.59 with a standard deviation of 1.16.

System Functioning: Collaborations among Agencies. To understand whether the training affected how respondents think about their work with other service providers, they were asked: How has the training session(s) you attended improved or chanced the way service providers interact with each other and provide services? They were asked to provide a specific example from their own experience about how the training(s) affected how collaborations among agencies are established. All of the specific responses to this question are listed in Appendix H-1.

Examination of the open-ended responses to the question about how interactions with other service providers were improved or changed revealed the following general categories:

Increased Communication. Among staff and service providers we have improved communication and now have a common language.

"There are now increased case conferences. We have improved how information is transferred from provider to provider and to different types of staff members."

Increased Referral Network. The training helped us to obtain information about other referral agencies and develop networks.

"Collaborations were developed among agencies and the sharing of information was encouraged. Networks were created with other agencies that attended the training. When you know someone at an agency, it makes it easier to make referrals there."

Information Source. We now know who to ask and consult about specific HIV-related issues.

"Case workers now contact other agencies for information and utilize them as useful resources."

More Coordinated Services. The training increased interaction and improved service coordination and procedures.

"The care conferences are multi-disciplinary. The trainer’s participation offers another perspective to share with our departments. Her focus point is to coordinate services and to create more collaboration between departments."

No Effect; Unclassified Responses. The training did not have any effect on how we collaborate with others.

Respondents were also asked to rate the extent to which the training affected how they forge collaborative links with other agencies. Their justifications for their ratings are shown in Table 11, and Appendix H-2 provides the full set of open-ended responses (sorted by rating).

Table 11
Representative Responses for How the Training Affected the Establishment of Collaborations Among Service Providers (by Rating Category)

Training Effect Rating	Representative Open-Ended Responses¹⁷
Large negative effect (made it much worse)	No responses given
Medium negative effect (made it worse)	No responses given
Small negative effect (made it slightly worse)	N = 1 We are not talking to each other.
No effect (didn’t make it better or worse)	N = 42 I didn't meet anyone at the training who I didn't already know. 2 I didn't see any establishment of collaborations among service agencies. 32 We were doing it before. 33
Small positive effect (made it slightly better)	N = 17 It was pretty good how it was. But it's slightly better now. 35 People are a little bit more interested in other services where HIV is being treated. They ask, "what's going on? How do you go about doing…" If we have other problems, we can discuss them with other professionals. We're not the only service. We have other primary doctors. There is a lot more collaboration. 45 There are somewhat more referrals between agencies. 53
Medium positive effect (made it better)	N = 46 We can now help other agencies with their HIV-positive patients. We are more willing to accept referrals from other agencies now. 62 Our agency now has a better understanding and we now feel more comfortable discussing confidential issues. 73 The nursing staff feels more comfortable talking about the delirium of patients, or what they witness as could be delirium, with the providers. 88
Large positive effect (made it much better)	N = 41 It increased the interaction among different staff since we do not have regular staff meetings to discuss cases. 112 Through the educational process, resources and experts were identified. Information packets are available now for practitioners to implement expected standards of care. 114 All different staff works in conjunction with one another; we can't work without each other - without consulting one another. 143

Note. N = 129. Respondents only provided a rating if they felt the main question applied to them. They were asked: Overall, how would you rate the effect that the training had on how the system, in general, encourages establishing collaborations among service agencies? Why is that? Ratings ranged from "Large negative effect" (-3) to "No effect" (0) to "Large positive effect" (3). The average rating for this sample was 1.57 with a standard deviation of 1.19.

System Functioning: Large-Scale Changes. Respondents were also asked to think about how the training session(s) might have improved or contributed to large-scale changes in the way HIV-related care is provided. They were asked to provide specific information about how the training session(s) may have lead to large-scale policy and/or attitude changes. Examination of the responses to this question revealed a number of dominant themes are listed below. The open-ended responses for this question are presented in Appendix I-1.

The dominant themes present in these open-ended responses to the question about how large-scale changes are as follows:

More Knowledgeable and Able to Provide Sensitive Care. We are better informed and that translates into more sensitive care for patients/clients.

"We’ve all become more sensitive to the psychological and mental health issues. We are better able to review a situation, to see the big picture without jumping to conclusions."

Implemented Substantial Policy Change. We have made major changes in how we provide care due to the training.

"With Medicaid reform, we were able to change a lot of the negative issues it tried to inflict upon women. We changed it so that the co-pay for women for medications was reduced from $2 per drug to $0.50."

Implemented (or In the Process of Implementing) Smaller Policy and Procedural Change. We have already started or are in the process of changing how we provide care to patients/clients with HIV as a result of the training.

"We haven’t yet completed any laws, policies, or guidelines. We currently are working on guidelines for treatment by utilizing the project’s procedure as the basis for future guidelines regarding HIV counseling and testing."

Heightened Awareness. We are now more aware about issues relevant to the care of patients/clients with HIV.

"Our agency is very aware of