From the course: Program Evaluation for Data Science

What is program evaluation?

From the course: Program Evaluation for Data Science

Start my 1-month free trial Buy for my team

What is program evaluation?

“

- This course on going beyond A/B testing is centered on applying program evaluation in data science. To make sure we have no issues about definitions, I want to start by being very clear about what we mean by a program and program evaluation. A typical textbook definition will say a program is an organized response to eliminate or reduce a problem where a response includes either one or more goals, performance of one or more activities, as well as the expenditure of resources like money and staff time. Ugh. I prefer the simple definition that a program is, "Any action taken with the intent to change the status quo." If you're working at a bank trying to reduce charge-offs, then the status quo could refer to the current model or approach to identifying credit-risky customers. If you're working at a health insurance company, perhaps you're building a model to optimize cancer treatment versus the current treatment paradigm. Now that we have a functional definition of a program, we should define program evaluation. I like to call it program evaluation, "A study conducted with the goal of determining how effective a program is at achieving a specific outcome." Coming back to our examples about programs. If your program was implementing a new model to reduce charge-offs, then the program evaluation would measure the change in charge-offs versus the current methods. Similarly, if the program was a modeling effort to improve cancer treatment, then the program evaluation would look at how effective the new model is at increasing life expectancy compared to baseline treatments. When I first started working in banking and private equity, I was impressed by the variety of situations where data science was implemented. There were situations where I had to divide the country to regions and implement models in only one region as a comparison, or when I had monthly time series data that I needed to analyze to see if the model we implemented without a control was making a difference, or when we rolled out a program for customers over a certain threshold of credit, and I was tasked with figuring out the impact. We sure did love to go beyond A/B testing, and you will too. In this course, we're going to work through a number of examples of program evaluations, starting with the A/B testing method, and then moving to other methods where we will offer sample data sets. Now that we have definitions for programs and program evaluation, let's connect this with your own work. Can you write down one or two main programs you've worked on in the last year? Have there been formal evaluations of these programs yet? If not, then maybe your personal goal can be to lead the evaluation of these programs after finishing this course.

Contents