1	Program Evaluation 101 Andrew Grant, Ph.D., Director, Grants Development Renee Pekmezaris, Ph.D., Vice President, Research The Nerken Center for Research Parker Jewish Institute for Health Care and Rehabilitation New Hyde Park, New York
2	What is Program Evaluation? Program Evaluation is the the collection, analysis and interpretation of information regarding an intervention. The intervention is a program that is conducted for a purpose (e.g., adult day care program).
3	Program Evaluation almost always involves gathering and digesting information about program status and sharing it with the program’s staff, planners or funders.
4	Why Conduct Program Evaluation? Determine the worth of an ongoing program Find ways of improving a program Increase the effectiveness of program management and administration Meet accountability requirements, such as audits Contribute to substantive and methodological social science knowledge
5	Questions to Ask Yourself: 1. What features or aspects of your program are most critical for me to describe? 2. How much and what kind of backup data will be necessary to support the accuracy of the description of each program component?
6	Methods Of Data Collection
7	Examine Records (Medical records, attendance, etc.) Pros Cons Already being collected You don’t have control (Records may be incomplete) Objective; credible Extraction can be time-consuming Prospective-happens during Ethical or legal concerns (HIPPA; treatment or program confidentiality issues)
8	Conduct Observations (Site visit, naturalistic observations, etc.) Pros Cons Highly credible if conducted Hawthorne effect by disinterested outsider Point of view of observer Observation instrument and different from training is time-consuming program employees Usually requires large number of observations
9	Self-Report (Surveys) Pros Cons Structured format allows Interviews more flexible for standard format of many questions/areas of inquiry Can be anonymous Completion Rate is often low Can be administered to Some people express themselves many people at different better orally (without structure sites simultaneously of survey)
10	Self-Report (Structured interviews) Pros Cons Especially effective with Time consuming to conduct people who cannot read or have trouble with written form of English Permit flexibility in Interviewer can influence response of information acquisition interviewee
11	Sampling Sampling is a method which allows you to collect information about the population, based on a smaller group Unless your program is really simple, you probably won’t be able to cost-effectively capture every performance data point on every subject Sample size needs to be large enough to be representative of the groups as a whole If you expect improvement over time, then you have to sample over time
12	Validity Validity: A measure is valid to the extent that it measures what it is intended and presumed to measure. A valid measure must also be reliable. Two types of validity: Internal Validity External Validity
13	Reliability A measure is reliable to the extent that the application of the measure to a given situation produces the same results repeatedly, given that the situation does not change between measurements.
14	Estimating Program Effects Best Randomized experiments Matching designs ( e.g., IQ, SES) Two Group, Quasi-Experimental Designs OOOOOXOOOOOO000000 OOOOOOOOOOOXOOOO0 Single Group, Quasi-Experimental Designs OOOOOXOOOOOO000000 Worst Pre/Post Designs
15	Learn from Others Take the time to learn from the successes and failures of others-you will not be sorry! Conduct a comprehensive literature review There are probably valid, reliable instruments being successfully used with your population Talk to other programs to find out how they approach program evaluation Conduct multi-center sites and benchmark against one another
16	Other Issues to Consider HIPPA Regulations Ethical Issues Role of the IRB (Institutional Review Board) Institutional Policy & Procedure
17	Institutional Review Board (Human Subjects Committee) A committee formulated under Federal regulations to assure the protection of human subjects in research. The IRB reviews, approves the initiation of, and conducts periodic reviews of research involving human subjects. The major mandate of IRBs is to protect the rights and safeguard the welfare of the subjects involved in research.
18	Institutional Review Board A human subject is involved if: The person is alive * Data pertaining to the person will be obtained through: Intervention (e.g., taking a blood sample). Interaction (e.g., taking a medical history). A private/confidential source (e.g., medical records). *Note: NY State law does not include the word “living” in its definition
19
20	Types of IRB Reviews Exempt Expedited Full
21	Types of IRB Reviews: Exempt
22	Types of IRB Review: Expedited
23	Research Eligible for Expedited Review (4) Collection of blood samples by veni-puncture, in amounts not exceeding 450 milliliters in an eight week period and no more often than two times per week. (5) Collection of dental plague provided that the procedure is not more invasive than routine prophylactic scaling of the teeth. (6) Voice recordings made for research purposes. (7) Moderate exercise by healthy volunteers. (8) The study of existing data, documents, records, pathological specimens, or diagnostic specimens.
24	Types of IRB Review: Full
25	Useful Resources
26	Present Your Data Simply “It is sometimes said that the facts speak for themselves. In reality, statistics often stand speechless and silent, tables are sometimes tongue-tied and only the graph cries aloud its message” -Glass G. V., & Stanley, J.C. Statistical methods in education and psychology (1970)
27	Hospitalization/Transfer Percent of Patients Specifying Hospitalization/Transfer Wishes
28	Questions?