Learning to Improve: * How America\'s Social Institutions Can Get Better at Getting Better. November, 2014
May 22, 2016 | Author: Kelly Phelps | Category: N/A
Short Description
1 Learning to Improve: * How America's Social Institutions Can Get Better at Getting Better November, 2014 * To be publi...
Description
Learning to Improve:* How America's Social Institutions Can Get Better at Getting Better November, 2014 * To
be published by Harvard Education Press, 2/2015
A “friend of evidence” for 40 plus years
2
My Starting Points § We confront a growing chasm between unmet social needs, rising aspirations and what our social institutions can routinely accomplish.
§ Current R&D strategies—too slow, too expensive and
the findings often of limited value to the on the ground problems people are trying to solve.
§ Our goal: Learning faster and better in order
to achieve quality outcomes reliably at scale. 3
Current Paradigm: Evidence-based Movement An academic has an idea
He/she design and fine tunes an intervention
An RCT field trial (5 years later)
Evidence it can work
Reviewed by What Works Clearing House Goes on an “approved list”
Districts required or “incented” to buy only from approved list
Educators “Implement with Fidelity”
Practice Improves! But there are major practical and conceptual problems here
The Power and Limits of Evidence-Based Practice § The genius of randomization – Being able to isolate the effect of one factor, willfully ignoring everything else.
§ Its limitation – ”Everything Else” is what actually produces the wide variability (and often unacceptable variability) in outcomes that we continue to observe.
§ A needed complementarity – Evidence-based practice and practice-based evidence. – This is key to achieving quality outcomes reliably at scale. 5
Core Principles Organizing Learning to Improve (aka developing practice-based evidence) § 1.
Make the work problem-specific and user-centered.
§ 2.
Focus on variation in performance.
§ 3.
See the system that produces the current outcomes.
§ 4. We cannot improve at scale what we cannot measure. § 5.
Use disciplined inquiry to drive improvement.
§ 6.
Accelerate learning through improvement networks. 6
II.Variation in Performance is the Problem to Solve
§ Critical question is not: “What Works?” § It is the Quality Improvement Question: “How to advance effectiveness among diverse practitioners engaging varied populations of children and families and working in different organizational contexts?”
§ Goal: Achieve efficacy with reliability at scale.
An Ideal Case Example § First year results from a
large randomized field trial of Reading Recovery (I3 inititative)
§ Key: a multi-site trial
RCT (average) Treatment Effect: Reading Recovery N=141 schools 16
14
12
It is a success lets spread it!
10
8
6
4
2
0
-‐0.5
-‐0.3
-‐0.1
0.1
0.3
0.5
0.7
0.9
Effect Size
1.1
1.3
1.5
1.7
1.9
DistribuCon of RCT Treatment Effects: Reading Recovery N=141 schools 16
14
12
Count
10
8
6
4
2
0
-‐0.5
-‐0.3
-‐0.1
0.1
0.3
0.5
0.7
0.9
Effect Size
1.1
1.3
1.5
1.7
1.9
DistribuCon of RCT Treatment Effects: Reading Recovery N=141 schools 16
14
12
Count
10
PosiEve Deviants
Undesirable/ Weak Outcomes
8
6
4
2
0
-‐0.5
-‐0.3
-‐0.1
0.1
0.3
0.5
0.7
0.9
Effect Size
1.1
1.3
1.5
1.7
1.9
III. See the system.
It is hard to improve what we do not fully understand.
A Typical Approach to a Coaching IniEaEve District
School
Salary Policies For Coaches
Hiring & Assignment Policies For Coaches
Coaching
Union Contracts
Credit to: A Framework for Effective Management of School System Performance. Lauren Resnick, Mary Besterfield-Sacre, Matthew Mehalik, Jennifer Zoltners Sherer and Erica Halverson.
And then great things are suppose to happen…
Peering inside the “Black Box”: the work processes of coaching
Does Coaching Work?
It is a silly quesEon. Across many different fields of endeavor, we know it can be powerful, but… It is a human and social resource intensive system. Even when well planned out expect to see variable quality in actual performance. What will it take to achieve quality with reliability at scale? This is the quesCon we need to ask and where evidence must be brought to bear.
The Invisible Complexity Schooling The Invisible Complexity of Schooling
17
Achieving Quality Reliably at Scale
How Do We Heal Medicine? Atul Gawande April, 2012
Gawande’s Closing Observation
§ Making systems work is the great task of my generation of physicians and scientists.
But I would go further and say that making systems work — whether in healthcare, education, climate change, making a pathway out of poverty — is the great task of our generation as a whole.
19
An Improvement Research Example: The Presenting Problem
60-‐70%
Students assigned to developmental math course.
80%
Percent of these students that never get past this gate.
500,000 students
in every cohort will never complete college math requirement.
20
Statway
Traditional Sequence
What Is Possible to Achieve 1 Year
2 Years
6%
15%
51%
Triple the success rate in half the time.
It is All About the Details Getting Under the Hood: Root Causes Analysis The OrienEng Problem
Extraordinarily high failure rates among students assigned to developmental math instrucEon
Primary Causes for High Failure Rates
Organizing Improvement event Hypotheses
Lose large # of students at the transiEons
Consolidate the courses into a 1-‐year pathway
material + instrucEon not engaging
Real world problems from staEsEcs as the organizer
Embedded literacy and language barriers Students mindsets undermine success Students “gone” before we know it Faculty pracEces + Beliefs limit success
Faculty development Psycho-‐social intervenEons aimed at “producEve persistence” Rapid analyEcs capacity
Emergence of a Working Theory of Improvement The OrienEng Problem
Primary Causes for High Failure Rates Lose large # of students at the transiEons
Extraordinarily high failure rates among students assigned to developmental math instrucEon
Course material and instrucEon are not engaging Students mindsets undermine success Embedded literacy and language barriers Students “gone” before we know it
Organizing Improvement Hypotheses Consolidate the courses into a 1-‐year pathway Real world college-‐level math problems relevant to students as the organizer Faculty development Psycho-‐social intervenEons aimed at “producEve persistence” A focus on “starEng strong” Rapid analyEcs capacity
1. Assessing Change: Initial Evidence of Efficacy of Starting Strong Package
24
!
II. Predictive Analytics and an Elevated Improvement Priority All students
Black students
Math Class Dropout
80% 70% 60% 50%
71%
“How ogen, if ever, do you wonder: ‘Maybe I don't belong here?’”
50% 40%
40% 28%
30% 20% 10%
12%
7%
13% 11%
14% 14%
Hardly Ever
SomeEmes
0% Never
Frequently
Always
Carnegie FoundaCon naConal survey of N = 714 math students
IIIl PDSA Cycle: Rapid, Small Experimental Trials The Three QuesCons: • What specifically are we trying to accomplish? • What change might we introduce and why? • How will we know that the changes are an improvement?
PLAN
DO
ACT
STUDY
Problem: Declining Alendance ager the Mid-‐term
A Change Idea: Student Group Noticing Routine
Actual observed attendance
Past experience
28
And the network also continue to assess performance context by context… 100%
Positive Deviants
Statway Students
8
Triple success rate line
1 17
50%
6 7 4 5
18 19
14 11 9 3
12
13 2
15 0% 0%
No improvement line
We also have a failure, why? What can we learn?
50% Non-Statway Matched Comparisons
100%
This is the real work of gemng beler at gemng beler.
29
VI. Structured Networks Can Accelerate this Learning to Improve
1. An enormous source of innovation 2. Social connections accelerate testing and diffusion 3. Seeing patterns that otherwise look particular 4. A safe environment for participants to discuss
comparative results • a “learning exchange” • a moral imperative, “if others can, why not us?”
Using Evidence to Get Better at Getting Better “The problem that is managing quality is not just an intellectual endeavor; it is a pragmatic one. The point is not just to know what makes things better or worse; it is to make things actually better.” –Dr. Don Berwick, Founder Institute for Healthcare Improvement Learning Fast to Implement Well to Achieve Quality Outcomes Reliably at Scale. 31
Come Join a Growing Community
View more...
Comments