[ Silence ]

>> Good afternoon.

Welcome to a [noise]CLOSUP
Ford School speaker series

from the Education Policy
Lecture Series we have here.

Today is my privilege and pleasure to introduce
Roland Fryer who is a colleague and a friend,

someone who I knew way back when he was less
famous, but I'm glad he has agreed to come

and talk about some really interesting work
he's doing on the Harlem Children's Zone.

Before I forget, I'd like to thank some
of the folks that helped organize this.

First thanks to the Ford
School for cosponsoring this.

Thanks to Beth Rebar [phonetic], Tom Ivacko,
Jill and Katie in the Communications Office

for helping put all to get-- put together this.

And to Bonnie Roberts as well.

And so, Roland is an economist, the-- as
the Robert Beren, Professor of Economics

and CEO of the Education Innovation
Laboratory at the Harvard University.

He's had original training and work.

For those of you who may know him, was kind
of more an economic theory which has lots

of fun Greek letters and
mathematical expressions.

He has moved to some extent of completely
overtime the Applied Microeconomics side,

looking at a host of interesting
education policy programs.

He-- in prior lives, he served as the Chief
Equality Officer at the New York City Department

of Education in '07-'08 and he has a
number of awards that are listed here.

You can read including the recipient of
Presidential Early Career Award for Scientists

and Engineers, the 2009 Time 100--

Time Magazine's annual list of the world's
most influential people, and most importantly

for our case here, Roland is a incredibly
bright, an articulate and engaging speakers

so I'll forever more enjoy the talk.

The standard is that Roland you can decide
how many questions to accept as you go.

I think we-- it goes to 5:30 so maybe if you'd
leave a few minutes at the end for questions.

And with that, I'll let go and take it away.

>> All right, thank you.

[ Applause ]

Brian is just being modest.

When I first got in education research, I
asked 1 of my advisers I said, "You know,

I think about going to education.

I think it's interesting."

He says, "Yeah, you really ought to be like
Brian Jacob if you want to be good at this."

[Laughter] So it is so good to be here.

Thanks for coming out.

I'm incredibly excited today to talk
about some work we've been doing

with the Harlem Children's Zone and you
guys-- Michigan time is 10 minutes late.

Do I get to go to 5:40?

[Laughter] Or is it--

>> As long as people will stay.

>> Yeah. Oh, good.

I've got some time.

I've got to leave for my flight but other
than that, we can stay in and hang out.

So look, I think the format I like the best is
that you should ask me questions as we go along.

I really don't like it when I'm in the-- your
chair and someone is talking and I'm confused.

And there's a burning question
in my head and I can't ask it

until 45 minutes ago and
then at my age, I forget it.

So, you know, and so like this.

[Laughter] You know-- just,
you know, yell it out whatever.

We'll get to it and-- but feel
free to ask me as we go along.

But let me go ahead and get started.

Alright so here's a quick outline of what
I want to accomplish between now and 5:30.

I'll give you some motivation
behind why I think the 97 blocks

in Harlem are an incredibly rich
laboratory for social science.

Social science is to understand
some deep questions

that we've been thinking about for some time.

I'll briefly and very briefly go
through the related literature.

I'll give you a brief history
of the Harlem Children's Zone.

For some of you who don't know what Geoff Canada
and his staff are doing, then I'll tell you

about all the data that we've collected in
our econometric framework and our results.

And then I'd like to draw a bright
line after the results, okay?

I'm going to give-- I want to be
very clear about a few things,

like I'm going to give you
results that I believe in.

And then we're going to talk about stuff
that I have no clue about, alright?

[Laughter] And that's kind of a fun stuff but I
want to be very crystal-clear about what I know

and what I don't know, the results that come
from our identification strategy I believe in,

what mechanisms are involved,
how you break those down,

how do you move forward, how
do you take these in scale?

I have no clue, alright?

Well, I got a little more than a clue
but not much and so I want to make sure

that you understand that is a very speculate
discussion and I'm happy to have it.

I'm not shying away from it but I believe much
more in my members than I do in what's next.

Alright, so here's the bottom line because I'm
sure at some point, I have this great quality

that tend to piss people off,
that you're going to leave.

The Harlem Children's Zone
is enormously successful

at increasing the academic achievement of
the poorest minority students in Harlem

and I don't really have much
of a clue why, okay?

It feels like in part of the analogy, maybe
it's extreme but I feel like, you know,

we've found a cure for a disease that's been
plaguing us and that the cure is something

like 1 apple slice, 4 nights a week, for 2 and
a half weeks, broccoli every night and man,

that's a really dynamic doctor and nurses
that seem to care about you, alright?

So-- [Inaudible Remark] Yeah, I really
feel like there's something special going

on in the Harlem Children's Zone, but I don't
really know what and the problem with people

like Geoff Canada is he likes to
do a lot of things to help people

and that's great but not for statistics, okay?

[Laughter] And-- [Inaudible Remark] -- so, you
know, I think we have a result here and I want

to show it to you but at the end of the day,

I don't know why they are
having such amazing games.

In the classic, academic way, I kind
of know what it may but it's not,

but I don't know what it is, alright?

So here's the motivation.

The racial achievement gap is
really, really big, alright?

It's about the gaps arise at age 2.

You don't really see them earlier than that.

That could be because they're
not there or it could be

because we're not good at
testing 9-month-old kids.

I think we're pretty good
at testing 9-month-old kids.

I think there are gaps there
but that's a different subject.

I'm happy to talk about that later.

In about 0.64 standard deviations
in math, the gaps are--

and 0.40 standard deviations at school entry.

Now Brian told me this was a mixed audience.

Mixed in the sense that some folks
know what a standard deviation is

and some folks don't know
what a standard deviation is.

No problem.

You should think of it this way.

The average kid gains in the
average American public school

about 0.08 standard deviations per month, okay?

So if I do my math, right.

In math, they are black kids are about 8
months behind in math, about 5 months behind

in English language arts when
they enter school, alright?

You probably know this from the NATE
but the average black 17-year-old reads

at the proficiency level of the average
white 13-year-old in our urban centers--

this is nationally representative data.

In our urban centers, it's much worse.

You know, I do a lot of work in DC and, you
know, if you go to DC if you look at the NATE,

12 percent of the kids are performing
math at grade level, 8 percent are reading

at grade level in Washington DC, okay?

You know, it's-- this business is so hard
trying to, you know, help increase achievement

and every now and then, you get a big bonus
and big, you know, boost and you feel great

about yourself and then you'd
get kicked right down.

It happened to me this fall.

I was in DC with 1 of our schools
because I like to be in our schools

where we're doing this financial
incentive experiments

and I like to be there the first time they
give the checks because we take so much flacks

from all the adults, at least the kids
really like it and so I like to be there

with the kids when they get their first check.

[Laughter] So I was there with the kids

when they got their first check
and the kids were going crazy.

They were so happy about the money and da da da.

And they had earned so much
and I was so excited.

I said, "Man, well I think we're
really doing something here."

And a fight broke out, next to me
while we were given our checks.

[Laughter] "Hey man, you got to separate that."

I said, "What are you guys fighting over?"

He said, "Mister, tell him
that 38 is bigger than 42."

[ Laughter ]

I didn't know what to do.

I don't know.

There have been many attempts
to close the achievement gap.

[Laughter] 'Cause the kid who thought 38
was bigger than 42 was bigger than me.

I didn't know.

[Laughter] There have been many
attempts to close the achievement gap.

There had been early childhood programs, lots
of stuffs, small schools and smaller classrooms.

There had been school choice,
voucher staff you know,

people try to find systematic ways
to make better public schools.

We've had neighborhood scuffled.

We've moved people out of there,
you know, poor neighborhoods

and move them to less poor neighborhoods.

Also it's a des-- you guys, I want to know this.

We've tried a whole lot of things in the past 3
decades and I don't think we've had big results.

In fact, there's a great book called,
So Much Reform, So Little Res--

I think So Little Success, I think it's a typo.

But anyway, it goes through these things.

I think this lack of success was
played into a pretty rancorous debate

about whether schools alone can
actually close the achievement gap.

Okay and when the-- in the roundup
to the presidential election,

there were 2 kind of groups that were--
there wasn't that much distance between them

but they basically were kind of staking
out their claims on this dimension.

Some folks said, "Schools alone are
enough," that was the equal opportunity,

equal-- education equality project.

And others said, "You need
a lot more than schools.

You need a lot of social supports, et cetera."

It turns out Jeff Canon signed both petitions.

I don't think we can get data from that
but it shows that he's a smooth character.

[Laughter] What I'm interested in
is trying to understand whether

or not schools alone can close
the achievement gap or whether

or not we do need these community
investments, okay?

Let me be very clear.

I am not that interested in the
Harlem Children's Zone per se.

I'm not interested in evaluating
a particular charter school

or a set of charter schools, right?

You know, since the results have come out,
I've gotten lots of calls and people saying,

"Can you give me some results like Jeff?"

[Laughter] You think I'm lying?

So I'm not interested in that per se.

What we're interested in is the bigger question
which is, you got community investment,

should you invest in the community
programs that we all know a lot about?

Should you totally invest in schools if you
want to close the achievement gap, right?

And it turns out that these 97 blocks in Harlem
are pretty good laboratory of at least trying

to start to think about whether or not
schools alone are enough or whether

or not you need community investments.

So when I grew up my neighborhood,
the easiest way

to piss someone off is to
step on their new shoes.

In academia, the easiest way to
piss someone off is not have them

on your related literature slide.

[ Laughter ]

It's true and it's likely because all
the academics I know have bad shoe,

so I don't do that.

I don't put anyone up on the
related literature slide.

I know a lot of people in this room have written
incredibly important papers in this literature.

But I don't do that so here is my
very brief related literature, okay?

There is a huge literature on
the racial achievement gap.

Lots of folks have contributed to that.

There early childhood literature, there had been
bunch of stuff on school input charter schools,

class size, teacher quality, et cetera.

There's a bunch of stuff
on neighborhoods, peers.

This really-- the Harlem Children's Zone are
like, putting all of these stuff together

and it's very related to a lot of these
literatures that you have contributed to.

Okay, here's a brief history
of the Harlem Children's Zone.

Just in case you haven't seen
Jeff on Oprah promoted it.

It started in 1970 as New York City's
first truancy-prevention program.

It was called Rheedlen Center
for Children and Families.

Until the late 1990s, basically Rheedlen
was an amalgam of after-school programs

that in just words, helped a handful of kids
escape the neighborhood's cycle of violence

and poverty but allowed many
more to slip to the cracks.

So his exact quote is, "I
was helping them by the tens

and losing them by the thousands,' okay?

So Geoff decided to create
a new organization to focus

on changing the whole neighborhood, alright?

And that's why he dubbed it
the Harlem Children's Zone.

The idea was to address all the problems that
poor kids were facing from bad apartments

to failing schools, violent crime, chronic
health problems, with the kind of cohesive web

of services from birth to college, okay?

HCZ started as a 24-block area.

It expanded to 64 blocks in 2004.

And now, it's a 97-block
area in Central Harlem, okay?

Now I know there're a lot of my colleagues here

who were superstar statisticians
much better than I am.

This is not the good part of Harlem, okay?

Don't worry about that.

This is the Central Harlem, right?

This just Harlem.

Now here are all the programs they have.

They have their charter schools which
admit by lottery which we're going to use.

They have a bunch of early childhood
programs so they got Baby College, right?

It's not for the babies, it's for the parents.

So Baby College is a 9-week parenting program
where the parents come and they learn things

about how to be a parent, like putting up--
whatever, you know-- Brian's got 3 kids.

He's my like idol on this stuff.

I have zero kids that I know about.

[Laughter] So I just like to be honest.

So there's a socket plug thing that
you cover the plugs with, right?

Okay, so they teach you to do that stuff.

My grandmother didn't know anything about that.

She just thought survival of the fittest.

If you're dumb enough to electrocute
yourself-- [laughter] that's just you.

They also teach about discipline, 1 of the
most famous series of classes in the Har--

in the Baby College is about not spanking your
kid and using alternative forms of discipline,

something my grandma also didn't know about.

[Laughter] After you go from
Baby College, at 3 years old--

there's a 3-year-old journey which
is a very similar parenting program

for parents with 3-year-olds.

Then there's the Harlem Gems
which is like Head Start on speed.

It has a 4:1 student and teacher ratio.

Kids learn 3 languages in the Harlem Gems.

It's basically a preschool program.

Just 1 thing of note, Baby College,
your parents can go there as soon as--

you know, when they're expecting parents, okay?

And 1 thing that is pretty unique

about the Harlem Children's Zone is you
don't just go sign up for these programs.

They-- you're actively recruited for it.

So he has a whole list of full-time
employees and volunteers that go door-to-door

in project buildings, in laundry mats, at
the check-cashing place on their corner

and when they see someone with a small child
they say, "You need to be in Baby College.

It's free, da da da," alright?

So in elementary schools, got a
bunch of after-school programs,

same thing with middle school
programs, high school, college.

You see, he's got a bunch of stuff here, okay?

And what's interesting-- what's
nice about it-- oh, that's not good.

Okay, let's hope that didn't happen again.

That was a map for the Harlem Children's Zone.

And what-- there're borders there.

And let's go back to it, you can imagine.

Okay, there are borders here in
this map and what's interesting

about the Harlem Children's Zone is if you live
inside the borders, you're actively recruited

for that list of programs I just showed you.

However, there are also 3 to 4 charter schools
in there and those admit by lottery, right?

So you can see where I'm going with this.

There are some kid-- people signed up for
the lottery, most of them live in the zone.

Some of them get to schools, others don't.

Everyone gets the community programs, okay?

That's the type of variation
I'm going to be using to try

to identify schools versus communities, okay?

In the end, I'll be able to tell
you that it ain't communities alone.

I don't know if it's schools
or school interacting

with the communities that
are the most important.

Although I'm going to make an argument and
it's not just schools but it's an argument.

It's-- I don't have the data
to be able to support this.

>> You have to put the name in the lottery?

>> You do have to put your name in the lottery.

Again, you're actively recruited to do so
and there're no stipulations for the lottery.

It's not like some charter schools
that say, "Everyone's invited as long

as they sign a parent pledge," or, you know,

"You can do it if you satisfy
the following criteria."

And in fact, I'm going to show you data, at
least an observable that these kids look just

like every other kid in Central Harlem.

>> How many kids--

>> Their parents [inaudible]

>> The parents-- absolutely.

I mean-- any lottery-- I'm going to do 2 things.

Let me get to my-- love question.

You're right.

I've thought about it.

I'm not sure I'm that satisfying, okay?

>> What percentage of your kids
usually get in that signed up for it?

>> Half, because about 200 come in.

They only have 100 slots.

Okay, here's the data.

We combined 2 data sets, 1 from the Harlem
Children's Zone Administrative File,

so we went and-- to the zone and helped
them digitize their administrative files

that actually have the lottery
winners and losers, okay?

We also have data from Baby College and Harlem
Gems of all the kids who have actually gone in.

Subsequent to the time I made this slide,

we also have data on every
single community program,

time in and time out for all the
kids who have gone through that.

As you might imagine, it's taken a few
trusty dusty undergrads to get that data

into a working format, but we're working on it.

We merge that data with the Department
of Education and New York data

where we have achievement
data, attendance, et cetera,

all the administrative data
from '03-'04 to '07-'08, okay?

And we have the admin files from the
actual lottery winners and losers

to the achievement data so as long as you're
still inside New York City Public Schools,

that's Manhattan and the other boroughs,
then you're on our data set, okay?

Test scores are only available
for grades 3 through 8.

One second.

Attendance and promotion data
are available for all years.

Yes ma'am.

>> What national data set did you
use from the Department of Education?

>> This has been the Department
Education in New York City.

>> I know, okay.

But what particular demographic did you use?

Did you use that set of data from the Department
of Ed to compare with the Children's Zone?

>> Yeah, so of in--

>> What--

>> In New York City, with the New York City
DOE, they keep data on every single kid and--

>> So it's every child.

>> -- every single child, so we have 10
years where the-- it's 5 years of data.

But we have 10 years of the data now
and all 1.1 million kids for every year.

>> So state thing step rather than a national?

>> It's a city-- New York City Department--

>> Just the city.

>> Yeah. But it's all 5 boroughs.

So if the kids move to Buffalo,
we don't have it.

>> And how many children then?

>> We have about 10 million and-- it's
1.1 million in the New York City schools.

Yeah, it's great for experimentation.

But-- yeah.

You know, it's funny 'cause I was
there as the Chief of Equality Officer.

I hate that tittle so bad,
Chief Equality Officer.

[Laughter] But-- it's a long story
why that that title came out.

But it was not my idea but, you know, New York
City is so big, like I'll remember sitting

in my little cube next to Joe Klein and I'd
say, "Hey, how many schools do you have?"

And they say, "You know that's
interesting question."

I was like, "No, it's not."

[Laughter] No wonder that kid
don't know if 38 bigger than 42.

[Laughter].

Alright, so that-- that's what we have.

We have New York City Department of Ed there
and we have our Harlem Children's Zone stuff

and we're going to merge those stuff.

Alright, so here's how we did that?

We've matched the data used in the file
algorithm so we took your last name,

your first name, and date of birth, okay?

And matched you to the New York City Department
of Education with various abbreviations

and alternatives, spelling, et cetera, okay?

Here's the quality of the matches we got.

For the Harlem Gems, we got 91.2 percent
of the kids who went to Harlem Gems.

We found them in the New
York City data at some point.

In the kindergarten treatment, we got 92.5.

In the control, we got 89.2.

In the middle school treatment, we got 90.6 and
the control we got 85.4 percent of the kids.

Again, if you move outside the city, we don't
have it or if you-- you know, if your name is--

you know, Caitlin with a Q in between and we
didn't get the Q then we don't find you either.

My former colleague, Caroline Hoxby, one
of our students estimate that 1 can expect

about 90 percent based on natural
attrition, we're in that ballpark.

And the nice thing about the results I'm
going to show you is that they're so large

that like even if we took the
control kids and said they--

the ones we can't find were
really terrible, were--

I mean the balance on these
estimates will still--

even in a lower bound estimate,
you'll see big effects.

Okay, so here's the econometric
framework in to your question.

We're going to do 2 statistical strategies,
1 we're going to use the lottery,

the other we're going to use I, okay?

So here're the lotteries.

New York State dictates that oversubscribed
charter schools allocated enrollment offers via

random lottery.

Lottery winners will form a treatment group
and lottery losers will form a control group.

So the ITT, the intent to treat estimate
is the effect of being offered admissions

into the Harlem Children's Schools, okay?

So a lot-- the stuff I'm going to show you is
going to be ITT effects, the intent to treat.

It's kind of like lower bound.

So this means, right, the effect of being
offered admissions, that means you--

Brian wins the lottery, he doesn't
go, I'm still going to count him

as a Harlem Children's Zone kid, okay?

Or if he goes for 2 days and
leaves, I'm still going to count him

as a Harlem Children's Zone kid,
that's the intent to treat, alright?

And we're going to estimate that by
having the outcome on the left hand side

for each individual I, we're going to control
for some basic demographic stuff and we're going

to include a variable for whether or not
you've been treated, that's the ZI, okay?

So gamma is going to estimate
the treatment effect.

Now, the treatment on the treated which
technically not an upper bound but it's--

answered a different question is the effect

of actually attending the Harlem Children's
Zone Charter Schools, this estimate is going

to be obtained by basically
instrumenting for whether

or not you went with your original assignments.

So we need something that's
correlative with whether

or not you go to Harlem Children's Zone, okay?

And that's going to be-- what's correlated

with that your actual original
lottery assignment, okay?

And you want-- you notice 2
things, I don't want to get bugged

down to technical details unless you have
questions but 1 estimate is the effect

of being offered admission, that's
the ITT, the other is the effect

of actually going treatment on the treated.

>> Okay, Roland, my question
was how did the students get

in to the pool that was subject to lottery?

>> So they were actively recruited by
the staff of the Harlem Children's Zone.

>> That's not a random process.

>> No, no.

They-- well, it's not a random process
but it's an exhaustive process.

They go out in these 97 blocks and they knock
on the doors, they go to the laundry mats,

et cetera, and they try to
get people to sign up.

Now, who signs up is not a
random process that's for sure.

>> That's what I'm saying.

>> And okay-- so 2 things.

One, I will show you an observables
that they look very similar.

And 2, we do IVs so that
we don't have to actually--

so this will help us with the
external-- [Simultaneous Talking]

>> Well, what percentage of the illegible
students have really signed up for the lottery.

That would be an interesting--

>> I don't know the denominator because I
don't know how many people they contact.

>>No, just so the students in the region
that are eligible to be in the lottery.

What percentage-- [Simultaneous Talking]

>> I don't know on the top of my head 'cause I
don't know how many kids are in the 97 blocks.

I think-- I don't know.

I don't even want to guess.

If you send me an e-mail when I get home
tonight, I can find the number out for you.

Again, I want to underscore that this isn't
a problem with any lottery-based analysis.

And 2, what I like about the
Harlem Children's Zone is

that they actually actively recruit people
who wouldn't sign upon on their own.

So with-- the 2 important issues which
you just gotten 1 of them with lotteries.

One, the kindergarten lotteries which
were not sufficiently oversubscribed

when we have the data, and 2, lotteries
are not necessarily externally valid, okay?

Who signs up for the lottery could be
very different from the treatment effect

that would happen for the kids
who actually didn't sign up.

So we're going to complement the lotteries
with an instrumental variable specification

so we're going to play very simple
IV strategy using the interaction

between a student's address
and their cohort, okay?

So let me just explain this in other words.

The identification is going to be driven by 2
forces, 1, a comparison of kids within the zone

who were of eligible age in that year
relative to others who were not, okay.

So the Harlem Children's
Zone schools opened in 2004.

They only opened if-- and they were on--

you were only eligible if you were
in kindergarten and in 6th grade.

So 1 of our basic counterfactuals
in our IV estimate is to say, "Okay,

let's compare the achievement of the kids
who were in 6th grade and eligible that year

versus the kids who were in 5th grade
and were ineligible in the zone, just--

I'm sorry, 5th grade is the bad one
'cause they'll be eligible the next year,

7th grade in the zone and who would never be
ineligible for the Harlem Children's Zone, okay?

And the second piece of this is the comparison
of kids who were eligible age who were close

to the zone versus those that are not, okay?

It's the interaction of those 2--
Jesus, are all my slides this way?

This is not-- okay.

Okay, so what are the threats
to this IV strategy?

Where it could go wrong?

Well, if the instruments are correlated
with unobservable determinants

of outcomes, then we're in trouble.

For example, if high achieving people move into
the zone, when they have kids who were eligible

for the lottery, then we're in trouble.

Did that happen?

I don't know but I can tell
you that you're eligible

for the lottery no matter where you live.

So moving would not have helped you in any
way get your kid into the school, okay?

Two, address specific shocks.

Now, so if individuals inside the zone re--

receive some sort of positive shock that is not
received by other kids inside the zone, right--

oh, sorry, other kids outside the zone

or any other cohorts inside the
zone, what do I mean by that?

If I'm given this talk and Brian says,
"Oh Roland, I forgot to tell you.

In 2004, I went to the Harlem
Children's Zone and I gave lectures

about how cool it was to be an economist.

And that if you really study hard in
school, you too can be an economist."

I would be screwed, okay?

Well, I mean, I'm sure that has
no treatment effect but it is.

[Laughter].

So if you imagine-- [laughter].

The biggest assumption I'll make all day is the
fact that Brian has a treatment effect on this.

So-- but if you imagined that Brian had
a program, had a big treatment effect

and he did it in the precise year that we're
measuring and he did it only inside the zone

and kicked out kids who were outside the
zone, then the IV strategy would be invalid.

>> Just to clarify this, what you're
talking about now is just how you're going

to estimate this depth of the
attendance to the charter school itself--

>> No.

>> As opposed to the effect of the
community services and so forth?

>> Yes, this is-- one of them now is how I'm
going to measure the effect of the impact

to the charter schools and I want to compare
these IV estimates with the lottery estimates.

>> Okay. And you're going to--
and the-- and you think that--

I'm asking because the inside the zone
and outside the zone obviously is going

to influence how likely they were to
receive and recruited for services--

>> Exactly.

That's the whole point, right?

And what I was going to show you with this quick
time decomposer thing is that if you look--

if you take a line and it's in the paper if
you have it, if you go towards the border

of the zone and you look at the
probability that you going to sign up,

that probability is pretty flat and
then as you get close to the zone,

it starts to go up, even outside the zone, okay?

So, you know, in same way people
have used distances and instrument,

we're using distance times
cohort as the instrument, okay?

I hope the rest of these slides are here.

Let me show you some results.

This is the problem because I actually
don't have the-- this is a real--

I did this on the Mac and
this is not like working.

>> Yes, that happens.

[Laughter]

>> Thanks for telling me.

So how can-- how should we do this?

I have Mac too.

Can someone have a-- if you have
a card I will PDF it on my Mac

and then we can put it on
these thing as PDF, right?

Sorry for the technical glitch here.

Okay, probably better now, okay cool.

I just want to show you I have a map I
know you think I'm lying but we got a map.

[Laughter] So this is 116th Street,
the 125th Street, and Madison

and so this is were the zone's
middle schools are, alright?

By the 134th Street-- yeah and
131st Street in over here, that--

that's where the elementary schools are.

Te elementary schools is away from the
middle schools, et cetera and I think

that has some interesting hypotheses
about what's going on there but that's

that map, got everything for you now.

Okay, here's the thing I promised before.

So here's-- now I have no pointer, that's okay.

[Laughter] So here is where the
identification is coming from, okay,

so if you are in the previous cohort, so if
you were a year older, if you were in 1st grade

when the lottery was for
kindergarteners, you can't get in so you see

that the black line is being
percent enrolled a hundred meters--

a thousand meters from the zone,
500 meters from the zone, et cetera,

this is going towards the border
which is a little counterintuitive

but going towards the border,
getting closer and closer

to the border of the Harlem Children's Zone.

What we did was we mapped all the kids in a
thousand meter away from the zone using RGIS,

using different points from the zone.

And here the test scores in ELA and math, okay?

So as you can see, as you get closer to the
zone's border actually test course are falling.

Now, here is the 2004 cohort, okay.

So these are kids who actually are eligible.

You see that the percent enrolled is
very close to zero until you get close

to the zone and it actually starts to go up.

>> Pointer.

>> Oh, what does this thing do?

>> It's a little thingy.

Put it the USB and it's--

>> Oh, okay.

It's like you're going to give
me something that's broken.

[Laughter] Alright, let me see if I can do that.

I'm sure-- let's go-- all of us go on break.

[Inaudible Remark] Alright, I'm doing a talk
about whether or not communities are important.

Everyone's having to help me with my own talk.

[Laughter]

>> It's working?

Probably working?

>> Alright, I want to keep talking and
see if we can get this thing to work.

So the identification is coming from this,
the difference between these 2 sides, okay?

Various cohorts, kids who were eligible,
kids who were not, and distance.

Distance interact with cohorts,
just want to show you that, okay?

So here are the results.

>> I try. [Laughter] [Inaudible Remark]

>> She knew that before.

[Laughter] Let's see if I can
spend up some more this time.

Okay, so the-- [laughter] I had to call you.

So here you go.

Here's 5th grade when the kids-- so April 14th

of their 5th grade years
when this lottery happened.

So I'm showing you their test result
in 4th grade and in 5th grade.

No treatment has happened yet, okay.

This is for math scores.

Alright, the dot-- the dash black
line is the average black kid

in New York City public schools.

Alright, the dash gray line is the average
white kid in New York City public schools, okay?

The red line are the lottery losers,
the blue line is the lottery winners.

Okay. So, now to your point, I
still don't have it about whether

or not they have motivated parents.

I don't have that here, point taken.

If you were to do this in KIPP or some
other charter school in New York City,

they'd be a quarter standard deviation above
the city average before they ever start.

Okay? At least that's not the case here.

So, you see what's happening.

Now these are the ITTs.

This is just-- did you actually win the lottery?

Okay. So this-- so for kids who went for 2
days and then left, they're still in here.

This is the ITT, this is
the lower bound, alright?

There are no controls here.

I don't have any free lunch stub, I ain't
control them, or what kind of car they drive.

I have nothing like that.

Okay? So these are just raw treatment effects.

Oops, no way.

Here is if you look at the actual
people who comply with the lottery

versus the control [inaudible] mean.

What that means is these are people
who actually were admitted and went.

So this gets closer to the
treatment on the treated,

the effect of actually going
to the Harlem Children's Zone.

Okay, again I have zero controls here.

So what you see is the same thing,
very similar, increase in 6th grade,

they increase in 7th grade, and a
big bump in 7th-- in 8th grade, okay?

Now we have this data, this
is the-- which cohort is this?

These are 2005 cohort.

We have the data for the other
cohorts and it's very, very similar.

So this is not a one year thing
that's happening here, okay?

Now, I just want to pause and look at this
for a little bit because at least in my work,

I'm always constantly staring
in like 0.2 treatment effects.

Okay, and you saw-- you sit there and you
say it, "This has two stars versus one star,

this is 0.2 and this, you know, this is great."

This is like 1.2, it's very different, okay?

And so I remember when the first time that
like, you know, you have this data program,

this data spits out-- this
[inaudible] the laser thing there.

This laser works.

[ Laughter ]

Next time I will study the racial gap in laser.

[Laughter] [Inaudible Remark] Okay, got it.

Thank you.

Go and hurt me I can say.

So, first of all I saw this, at
first I thought it was a mistake.

But then once we really looked
at the numbers and realized

that there is actually something
here, I had a great time.

Because I have, you know, a few full time
[inaudible] and they're like, you know,

former Harvard undergrad which is I said,--
well, not to him and I said, "You know,

we really got to think about this gap here
because there's like a white-black gap going

on [laughter] and I'm not sure why
you guys can actually achieve."

[Laughter] I would start with your culture.

[ Laughter ]

I said that, they were like,
"Well, we went to Harvard."

"Like just calm down."

[Laughter] Just relax.

Okay, so that's math, and that's great.

On ELA, things are more muted.

>> [Inaudible] please go back one slide there.

>> Yeah sure.

>> So even though there are [inaudible],
these huge treatment effects going on--

>> Yeah.

>> -- grade 7 and 8?

>> Yeah.

>> If the average for all bachelors in
New York City is kind of staying flat.

>> Yes.

>> So is that true?

We cut-- and it's mechanically
true because people largely

and [inaudible] actually doing worst?

>> Yeah, but this is like 200 kids, so-- and
there's like, you know, 500,000 black kids

so even the-- even if they weren't-- even
if they were doing a little bit better,

you wouldn't expect the mean in the line.

>> Even the red line, it's going down.

>> Yeah.

>> So there are actually the ones
who, you know, were complying

and actually didn't get in or doing worse.

>> Yeah. What I should do here, this
is the average in New York City.

What I should do is put the
average in Harlem here.

>> Right.

>> And you know I can do that with-- I don't
know what it looks like but it's not clear

to me that, you know, we know middle schools
are a place where gaps really start over,

but not and it's not clear to me that if you
looked at just the poor place, I mean, did--

you know, did the-- this is all
black kids and the treatment

and the control are just
basically Harlem kids, okay?

That's a fair point, totally fair
point and I should put it here, okay?

Now, ELA is not as dramatic, however, if I'd
shown you this first, you would have left happy.

You see the same thing here, all of them--
they kind of go down and there's a little bit

of a difference here and basically
this difference is about whatever 0.2,

0.3 standard deviation difference
is, you see a little bit more

in the control comply mean, okay?

So the ELA results are not as
impressive at all as the math results.

Why is that true?

I don't know.

One thing I do find interesting though if
you look at the elementary school results

that I'm going to show you in a minute,
those math and ELA are both going

in lock step together, they're both just as big.

So there are obviously these theories
out there that say by the time they're

in 6th grade it's very tough to move
language relative to math, skills, et cetera.

This is consistent with that, whether
or not that's true, I don't know.

Okay, so let me show you the distribution
'cause you might be thinking, well,

Harlem Children's Zone is probably
just good at like taking up the kids

who are already motivated and the
rest of them really didn't happen.

So here you just see the distributions between
the winners and the losers, again this is back

to the ITT which is just a distribution
of 5th grade, it's very similar, okay?

In 7th grade what you see is that, and
this is-- this is math, this is math.

The lower kids you see a push here, okay?

By 7th grade, you see the
distribution starts to separate.

By 8th grade, you just see the--

basically they're pulling apart
of these distributions, okay?

I always like it when you can get a paper and
pictures instead of like saying and, you know,

if you squeeze real hard, so that's great.

In ELA, you basically find
nothing that looks like that.

Very similar in 5th grade, 6th grade very
similar, 7th grade incredibly similar,

and 8th grade you find a little push out, okay?

Why do you find that push out in 8th grade?

I don't know.

But when I was going over the results with
Jeff [inaudible] I said, this is curious, man,

you got nothing here in this little
push out, why amongst these kids?

And if you known New York City public
schools they rate things in scales of 4,

1 and 2 aren't proficient than 3 and 4, right?

And if you look at it like
that, if you discretize it,

you realize all of it is
coming from 2 through 3.

All of it is coming from 2 through 3.

I said, "Yeah.

I don't really-- " I said, "I don't
really know a model to predict that."

He's saying, "I do."

I say "What's your model?"

He said. "I told the kids if you went from
a 2 to a 3, you got a trip to Disneyland."

[Laughter] So, I don't-- I have no idea if
that's what happened but he is unapologetic

with those kids who went to Disneyland.

Okay, so let me show you something a little
bit different which is, you might be thinking,

"Yeah, [inaudible] where those are affects
from the state test scores in New York City

and they concentrate on the test, they
teach to the test and I would argue

that with you till the-- to ignite
with all that but that's fine.

But let me just show you something that's not
even like for public consumption in some sense.

Let me show you something that's just
for HCZ kind of internal purposes.

So we found out that HCZ gives the
Iowa Tests of Basic Skills, okay?

Just for their kind of internal-- one
thing internal test they give kids.

Some people give chapter
test, they do that also,

they also give the Iowa Tests
of Basic Skills, okay.

So here a straight line is normal
growth, so if you're going up,

you're doing more than average growth.

The dotted line is kind of
obviously the 50th percentile, okay?

I don't have treatment effects here for you
because the control kids were not in schools

that gave the Iowa Tests of Basic Skills

so all I can show you is how Harlem
Children's Zone kids are doing by themselves.

You can make up your own mind whether or
not this is big or small or I don't know,

but I think it's something interesting and
I want to show you all the data we got.

So you see in math scores they're
gaining, I mean that they--

on another test not the actual
state test itself.

A very similar, this is the 2005 and 2006 cohort
here, similar thing with a different cohort.

In reading scores, they're going
up in 2006 cohort a little bit,

this is going up and then it's flat.

They're not near the 50th
percentile so that's, you know,

that's something one should consider that.

Whether or not the treatment
effects are huge I don't know again

because I don't have the control
group but I want to show you this.

This is-- some summer statistics for
the lottery, winners versus losers,

the point is there are no
P values that are small.

Here are the middle school lottery results.

Let me just put some numbers into
the pictures that I gave you,

because we all have the kind of, you
know, you reduce class size from 24 to 16,

you get 0.22 standard deviations.

You have your teach for American teachers,

it should gives you 0.15 standard
deviations in math, 0.03 in reading.

You know, we you can go on
about basically 0.2, alright.

So let's just look at what
happens so we take the 2005 cohort.

Okay. You have about 0.2 in the ITT.

It's about 0.279 on the Treatment-on-Treated
okay, for 6th grade math score,

7th grade math score it goes up even more,
by 8th grade math score these are big things

so these are standard deviation units.

I haven't divided by anything,
I didn't pull one on you, okay?

This a 0.733 standard deviations in the ITT,
1.112 in the Treatment-on-Treated, okay?

ELA score is not as big but then if you
get to the end is by 0.239 and 0.363, okay.

You can look at the paper.

I want to get the more things here
but 2006 cohort, this is an old table,

we now have these results but you
can see the pool samples, et cetera.

It's robust across years.

That's what I want to be able to show you.

Now, let me show you some IV results.

Yes?

>> I have a question here.

It looks like that based on that, you know, only
about two-thirds or three quarters of the people

who won the lottery ended up attending,
what's the difference just from going from ITT

to the treated Treatment-on-Treated?

>> Not quite because the
Treatment-on-Treated is like, you know,

yeah, no, actually you're right.-

>> Something like.

>> Yeah, somewhat--

>> Some not completely trivial fraction--

>> Don't even go.

>> So I'm just curious, you have any
sense where they were going and it kind

of get set like the treatment effects.

What is the difference between
the Harlem Children Zone and--

>> Yeah.

>> Is it the neighborhood school--

>> Yeah.

>> -- at some other great
charger outside the zone?

Is it?

>> For the vast majority it's just
the neighborhood school's problem,

the vast majority.

Alright, so let me show the IV results.

Okay, now we have to decide what inside
versus the outside of the zone means

and I don't have any good theory on that, so
I've shown you multiple ways where, you know,

you take 800 meters outside the zone,
you take 1600 meters outside the zone

and 2400 meters outside the
zone, et cetera, okay.

And as you might expect, as you get
2400 meters outside the zone, you know,

you're over in Columbia University and
the more fancy places and so, you know,

the comparison groups are a little different.

>> Yeah, is there kind to teach African and
African-American history in this schools?

>> Yes, but not more so than, you know,

a typical charter does-- I
mean a typical school does.

I think the difference is that-- where
the difference potentially could be is

that there are a tremendous amount of
positive male role models et cetera

who were there teaching the
kids on a day to day basis, so.

>> Were they didn't cooperate again

to the regular curriculum, do
they have separate classes?

>> I don't know the answer to that.

That's a great question,
I don't know the answer.

Okay, so you see the lottery
estimates in general are big,

of the IV estimates in general are bigger, okay?

So now by 8th grade you're
looking up, you know, 1.3, 1.5,

1.3 depending on how you actually define this
and ELA you're still around 0.2, 0.3, okay?

So what's kind-- here's what I take away,

it's a little over a standard deviation
treatment effect in math about 0.3 in ELA, okay.

So we do the results by subsample,
by girls and boys, for example,

and you don't really see big differences except

for the 8th grade math score here
okay, and you see it on boys.

Again, that's something that we haven't-- has
not been consistent in the literature, you know.

My reading of the literature is if you look at
the intervention projects, the ones that are--

the ones that work are more likely
to work for girls than the boys.

Harlem Children's Zone is a
little different in that, okay.

The elementary school IV results, okay,
here I don't have nice pictures to show you

because they were just in 3rd grade.

So it would just be a dot.

Here you see the math scores are about 1.9 to 2.

Now you're saying, "Why the
elementary school things larger?"

Well they had an extra year, okay?

So the average basically is about, you
know, 0.4 a year so and these people--

these kids had an extra year because they
went kindergarten through 4th grade, right?

These other ones are just 6th grades

so they have an extra year,
it's a larger treatment effect.

What's interesting here is that the ELA scores
mimic the math scores in the elementary school.

Yes ma'am.

>> I have a question here.

The reason I have--

>> Oh sorry, sorry.

>> I can't see here because of that.

>> Oh I'm sorry.

No, I would-- I'm sorry I'm
just ignoring it for now.

[Laughter] What do you want?

[Laughter]

>> I actually ignored that question, I'm going
to go on to the different question, okay.

>> Oh [laughs].

>> The-- reading and writing,
English language are it's very--

this is a very different subject, this happen
in 3rd grade than it is in middle school

>> Yes.

>> Do you have any breakdown of ELA scores for
the 3rd graders' reading fluency, comprehension

and vocabulary, anything like that?

>> Great question.

We can, we just got the data.

So we just got about maybe
three weeks to a month ago,

we just got individual question level data
for 10 million children in New York City.

And then-- [Inaudible Remark]

No, undergraduates girl, that's what you do.

[Laughter] So we have that data and
so I'm not sure I'll be any good

in sub grouping the questions but I think the
state test actually puts them in sub-groups.

Yes, I think while I'm talking in and
so-- yes we can do that, we ought to that,

very fantastic question and because of
my age could you e-mail me that question?

[Laughter] Oh good.

Okay, we present lottery results but as I said
the lotteries won't really over subscribed

but I just want to show you everything I know.

These treatment effects are still big
but the standard there is enormous.

Here's the Iowa Test of Basic Skills of the
elementary school 2004 cohort, 2005 cohort.

What's interesting about these is now you're
starting to get above the national median,

you know, and so again I don't have treatment
effects for these but these are some pretty,

well a little wild, but pretty on
average, pretty big gain is coming even

in the Iowa Test of Basic Skills, okay.

Now I want to keep plowing through here.

That's all the data I have that I can
do IV or lottery estimates on them.

Now, I really want to show
some results with Baby College

and with Harlem Gems even
though I can't do it justice.

They don't admit in Baby College in a way that
would allow you-- they don't do it by lottery.

Everyone, they want everyone
to be in Baby College, right?

I know good for people, bad for me or
Harlem Gems, they don't do it either.

So what I'm going to do now is show you some
really, really simple ELAs of just what happens

to kids who come into Baby College
versus those who go to Harlem Gems,

and which way the selection goes?

I don't know.

Okay, I don't know if people
select in who were worst,

I don't know if people selected
who are more motivated.

All I'm going to be able to be able to tell
you is basically there's very low evidence

of a big treatment effect on Baby College, okay?

So here, what I have here is
Harlem Gems and Baby Colleges of.

Now, I'm looking at the math scores in 3rd
grade, so I've traced people who have gone

through Baby College and are now in 3rd
grade, and what I'm doing is looking

at your 3rd grade scores on the left hand side,

controls on the right-hand
side plus an indicator

of whether or not you're in Baby College.

I'm reporting that indicator here, okay?

I'm also doing the same thing for Harlem
Gems, I've also done the same thing

for Promise Academy in an all ELAs ways
you can kind of see, ELA versus ELA, okay?

Promise Academy still looks big, Harlem Gems.

You know, the problem with this is
this is point 2, which is basically

like the effect of head start point 2.

But it's measured with such error,
I can't tell you much, okay?

So it could be, I don't know.

What I'm more confident in is
that Baby College doesn't seem

to have a big effect on test scores.

Kids are coming to school more, they
have less-- sorry, that's the ELA score.

The total absences and so total absences is--

this is point 4, it's incredibly
time and relative to standard error.

So I would say on these dimensions of
achievement I don't see any evidence

that Baby College had any effect, okay?

Now, what I told Jeff, I said,
"Your Baby College ain't nothing."

He says, 'cause that's the way I talk, he
says, "Oh, but you got different outcomes.

You think Baby College is to increase test
scores and total absences," Baby College

of his vision was to get past
stop cursing up the kids.

So he says, "I don't know if
that has treatment effect or not,

we have different views,
different outcomes," okay?

So one of the things I want to do
going forward is to collect more data

on what's actually happening when people
go through Baby College, et cetera.

All I can show you is what I have, alright?

[Inaudible Remark]

>> Yes. There is a long list.

Do you want me-- so you know like,
you know, the stress level is I want

to set the interaction the parents are having
with the kids having come to Baby College.

I would like to figure out a way to
understand even it's just a survey data,

how the discipline is gone, I want to observe
parent-child interactions to figure out if the,

whatever they call it, the
cultured cultivation that theory

that they're teaching Baby
College the action stick in, so.

Bill Wilson, Orlando Patterson, Rob Samson,

myself would put in together a
qualitative team who's actually going

into the Harlem Children's Zone to actually
observe these other outcomes that it's hard

to get it through administrative
data but I don't have that yet.

Okay.

>> These are only students who
ended up in the charter school or--

>> No, these are in New York
City public schools.

>> So who's the, in some of the people,
there's some subset who went to Baby College.

>> Yes.

>> But then there's, you know, a million kids
in New York City who didn't go to Baby College.

>> Yes.

>> But the number of observations here is 109.

Who's the--

>> These are people who signed
up for the lottery.

>> Okay. So it's only who
signed up in the lottery.

>> Yes, I was trying to take out some of the
effects that you're probably thinking about.

Okay?

>> So these are people who
signed up for the lottery,

>>Yes.

>> Didn't get in then went to a public school.

>> Went to a public school, but also--
so these people went to Baby College?

>> Went the Baby College.

>> Yup.

>> But they're not in the charter school?

>> Not all of them, so that's why we
control who went to the charter school.

>> Have you done this for the students
that went in to the chatter schools?

>> Yes.

>> Look at the effect of the Baby College?

>> Yes, yes.

The results, the samples are small, too
small to actually show you anything.

>> Too small.

>> But let me show you the following, okay?

Here's the best evidence I got.

Forget the regressions.

This is just a, did you go to Harlem Gems and
not on your own, your IOWA test of basic skills?

Those lines couldn't be more similar.

Okay. Again this is, I want
to show everything we have.

I'm much less confident obviously
in these results

because there's no identification
strategy than are on the others.

But I will say if someone asks me, is
Baby College knocking it out of the box?

I would say, I doubt, okay?

And so again, you just see that these
things are following a very similar whether

or not you went to Harlem Gems, okay?

>> Okay, this is what I'm thinking.

Going back to your, this is just
the speculative point of it?

>> Oh, yeah, let's call it that.

>> No, this is, I want to honest.

I mean, I don't know.

I mean, I'm confident that
these regressions are correct.

What they measure is a different story.

>> The graph you have up there
on performance in lottery

>> Yes.

>> In school, not in school but in the lottery.

With the performance of the kids who
did not get into the school but who were

in the lottery was flat or declining, but the
performance of the schools of the kids who got

into the school was increasing, you got
that very dramatic results for the map.

That's we're [inaudible] graph right.

>> Yes, that's true.

>> Okay, so all of those kids, those 2 samples,

the ones who've performed
well, the ones who did not.

On average, their parents are the same
because it's just a random draw, right?

In that school, if that school is doing
something for those parents who want

to do something and is working, you
might be able to pick that up with the--

whether or not they actually
went to the Baby College.

Last statement, if I got you right.

>> No, 'cause I think what you want me to
do is you want me to run an interaction term

between Baby College and like the
treatment, whether or not you,

so, Baby College in the treatment.

So what you really want.

Like I have clicker that
works is that, you want this.

You want you want a sub sample to
be Baby College about Baby College?

>> That's right.

>> If I have the data, I can do it in minutes.

I don't know if I had the data but I'll check.

It's great question, great point.

I'll see. You know, the problem with this
obviously is, you know, even we got 486,

so even if we're splitting
by gender, we're on the end,

so we have to be a little careful, okay?

And you know more years, more data.

I mean, I think this is the first
project not the last project

in terms of this treatment process.

Okay. So let me keep me rolling here.

Okay, what have we learned?

And then I want to open it
up obviously for discussion

and you can beat all this up
more than you already have.

I think what we've learned
is the Harlem Children Zone,

at least the Promise Academy Schools or norms
is successful in boosting math achievement,

and ELA achievement is not as good--

ELA and math in elementary school
and math achievement middle school.

Now what else can you squeeze from the data?

Okay. Now, here is where we're speculating.

Is of community versus our schools?

So let me show you what investments you
get if your community versus our schools.

Reiterating what we talked about before.

The community bundle, you get to early childhood
programs, the public elementary school programs,

et cetera, just in the community bundle,
you get the after school stuff, et cetera.

There's also kind of student-family bundle,
maybe it's a bad way of thinking about it

but if you're a sibling of a kid who gets
into the lottery, and you're in that family,

you're going to get some
sort of investment, right?

Because Geoff Canada, he sends
fruits and vegetables home, okay?

He sends pre-made meals home to the kids.

He gives as much materials support, advice
the parents, you also have greater knowledge

about the community programs if you
have a sibling who's coming home saying,

"I'm going to go win the All
Night Basketball games happening

at the Rex center they told
me about today in school."

Alright? That's what I think
of the student-family bundle.

Then there's just a pure school bundle.

Now what is Jeff Kennedy doing?

On several trips to the Harlem Children's Zone,
I tried to figure this out because, you know,

it's like a little mystical
when you first see it.

And so we went to the Harlem Children's
Zone and we just interviewed the principals,

the teachers, the kids, and that kind of staff.

Honestly it was like a bad episode of
Law and Order because the principal--

I would say that the principal,
wow, look at this effects in Math.

How did you-- what things happened?

He said, "Strong principles."

[Laughter].

And then you got all the teachers he said,
"How do you think you can explain this?"

And you'll say, "Just good teaching, boy."

So I am not sure we got much out of that
but what we did get out of it was that,

the kind of list of things that
they're doing in that school.

So they give health and dental services, okay?

So if you're sick, you're not
feeling well, you go right in this--

in a middle school the health and dental
services right inside the building.

They're not if you're in elementary school
and I actually think that's interesting.

They have social workers inside the school, so
I remember when I was a kid and you look out

and see if it's sunny outside, you hit someone
outside in the head, you could go home.

Here you can't go home, they like
talk to you about your feelings.

I mean that's terrible.

[Laughter] So they got like social workers.

So if you get in the fight here, you get
pulled away, you know, why did you do that?

Do you feel regret?

And all that stuff [laughter] and then
you go back into the classroom, okay?

So they also have date oriented instruction.

So, you know, it's not like the term
which is used a lot education reform.

What they do is they test kids every 6 weeks.

They break the tests down by skill so they know
whether or not Mel understands linear equations

with one unknown and then he gets 2
hours after school everyday to practice

on the things that he's the weakest, right?

Okay, you got more time in school, these
kids, they stay in school from 8 to 6

and they only get 27 days out in summer.

In the winter, when they go home
for Christmas break just kind

of it pays 50 dollars a day to come back, okay?

So they actually calculate it.

They spent twice as much,
literally twice as much time

in school relative to kids
in normal public school.

Every time I want to borrow money from Joe, I
tell him, "I tell you to your elementary school,

it's not normal to be in school 'til 6."

He says, "Don't do it, don't do it!"

This may seem silly but they're real.

You know, like I've been in a lot
of schools and the environment

in the school is pretty incredible, like
the teachers seem and they sound silly

but they really care about these kids.

They have skill teachers, they have student
incentives, they have nutritious school lunches.

So it's got like the whole food
basically there like, you know, you go in,

you get like whole grain and like, you know,
broccoli, I've never eat there, it's ridiculous.

[Laughter] They have coordinated after
school programs so if not like your normal

after school program which I participated
as a kid, where you break through school,

you go like 10 minutes then you go after
school people, you just show out, right?

No, these after school people will come
into the school in like the last few periods

and it's a seamless space of the after school.

The other teachers you notice they're
not there, you're working on your stuff.

It's very, very different.

Now, this is cool and all but it
makes statistics impossible, okay?

So communities or schools?

Well, I don't really know, okay?

But let me tell you 4 pieces of
evidence that suggest the change

in communities alone won't do the trick.

Number one, our IV strategy compares kids
relative to other cohorts in the zone

that were not eligible for the lottery.

So in some sense, our IV strategy purges
much of the community effect, okay?

Second, I'm going to show you
these results in a minute I think.

Siblings of the zone kids who had access to
all the community programs but for random luck,

we're not actually in the
schools show no gains, okay?

So if my brother gets in Harlem
Children Zone versus I don't really--

into the actual schools and I don't.

I'm still in the community programs,
I know I had pretty good knowledge

of the community programs 'cause it's
my brother, my test scores don't move.

My access has go down and that's probably
because my brother is in school all day

and my mother is like you
need to go to school too

but my achievement doesn't actually go up, okay?

When we do analysis of sub samples,
like the boys versus the girls,

we notice the kids inside the zone
don't have any difference in effect

of the actual charter school relative to
kids who are actually outside the zone.

Forth, if you look at the MTO experiment?

You know when they move neighborhoods?

I think if you designed an experiment, you
said, "Roland, we should design an experiment

where we're going to change
neighborhoods but not change schools?

MTO would probably be like really, really good.

I mean that's like to a first
order approximation,

that's basically what they did, okay?

I mean not on purpose but folks move
neighborhood but they're going to be very,

very similar schools and what you see is
that the girl's achievement doesn't change,

the boy's achievement actually
goes down a little bit.

And the last thing which is pure speculative and
should be a separate point here is if you talk

to Geoff, you say, why did
you start charter school?"

He say, "because of my community investments.

I didn't see any return in terms of the actual
achievement of the kids going out, okay?

So four pieces of evidence, one anecdote.

Here the sibling results, you
see the math score, it's point 2

but it's measure was set
error, ELA score is this,

the absences were actually statistically
significance or nonetheless absences.

Being on-time, the grade level is nothing
more so these are looking at just the set

of siblings in the Harlem Children Zone.

One kid goes to the charter
school, the other kid does not,

you don't see differences in
achievement during that day.

Now here's a very speculative
discussion and I'll probably end it here

which is what's driving the school bundle, okay.

You got all these stuff, health,
mental health, longer day,

the chef in the school, all this stuff.

What can you parse out from this?

Well, literally impossible to
disentangle what the heck they're doing.

I mean we got like 12 things, right?

And just a few schools was all of
but you're doing all 12 things.

I want to run this experiment where you
test all the different possible combinations

in the bundle but 2 of the 12
experiments and I'm too tired for that.

To tell you, I don't think just
based on other people's research,

not my own research, it's purely speculative.

I don't think it's teachers
[inaudible] value added alone.

Right, 'cause there's other folks who are
doing, teacher's incentive tied to value added,

they're not getting any results
to look anything like this.

I don't think a social worker is alone.

I mean we have programs like
turn around for schools et cetera

that put the social workers inside the schools,

they're not getting any effects
or anything close to this.

I don't think it's due to the incentives alone.

I've been through a lot of those.

I can tell you I don't have any
pretty graphs like this to show.

I got some interesting results but not
of the magnitude of these results are.

And you know, I don't think as longer school day
alone because again, they're been other people

who have done things like
lengthening the school day and--

sorry and not getting really great results.

So what could it be?

I though they were cheating.

I really did, it turns out we got the
questions level stuff that I talked about before

and we looked in like excess variation
then we ran with Brian and Jacobs

and Steve Levitt [phonetic] algorithm
for how to catch cheating teachers

and it turns they're not cheating
or at least cheating very well.

[Laughter] You can't tell the difference
between cheating very well and not cheating.

So it could be amazing teacher though.

They have some very, very, very,
very, very good teachers, okay.

So 8-grade math there's this guy name
Mr. Petite [phonetic], he's an ex marine.

He's a 6-foot-3 black guy who you know
does like math as a combat sport, okay?

And he is unbelievable, right?

And he's a very interesting guy.

What I think it is and what economists
typically have kind of troubles thinking

about is interactions between
elements of the battle.

Okay. And so, you know, at least when I was
in graduate school the cool thing to do is

to estimate a partial derivative.

I held everything else constant
and I increased the school day.

So I'd know exactly the school
day you know, so you know,

'cause it wasn't cool like
have like four things.

All of which could be interacting in ways you
didn't know but the combined effect was large.

That wasn't cool.

What was cool was to estimate
a partial derivative, right?

And my sense is, I'm not
saying anything deep here,

is that the total derivative may
be far more interesting, right?

If you have better teachers with a longer school
day with something else, maybe that total,

right, you remember the chain rule.

Maybe that total derivative
is actually a lot higher.

Okay, so, I think try and understand elements of
the bundle, I mean, interactions of the bundle.

It's really important.

Let me tell you about what we're
doing next and then I'm happy to stay

as long as you want for questions.

We're looking at longer term right
now in non-educational outcomes.

Things like teen pregnancy,
crime and so on, okay.

And before I make a stronger statement
about communities versus schools,

I want to look at other outcomes where you think
the communities will have much more of an effect

than actually just test score achievement exam.

The other thing we're doing is we've been
working with the Obama administration who plans

to roll out 20 of these across the country in a
way that will, you know, putting them together

in a way that will maximize learning,
right so, you know, I'm a nerd

and I called up folks in the administration.

I say, hey, you're rolling out this thing.

How you're rolling it out and the way where
we're going to be able to learn which kind

of is what's driving luck 'cause
you can roll in out in a way

and they said, that's a good question.

No, it's not.

You're spending a billion dollars, you ought
to be able to tell me, you know, how to do that

and I have concrete ideas about how to do
that and we're going to talk about that.

Last thing and then I'll shut up is one of the
things I'm so interested in is, of all these,

is trying to figure out, okay, let me read this.

What the pill looks like.

Okay, I want to figure out what
the magic bullet might look like.

You know, he's doing 12 things.

Could we figure out four or five of those things
to actually try in a random school-based trial

and regular public schools 'cause personally,
I'm not that interested in just, you know,

this set of charters that are all closing
the achievement gap, that's great.

Question is how can we take that stuff and
actually close the gap in public schools

and you know if, you know,
we asked Geoff, so I did.

I said, hey Geoff, I want to do an experiment.

We'll take four things you're doing and
what should those four things do-- be.

He would say human capital, the human
capital piece, you got to have good teachers,

you got to be able to pay them in a way
that's going to attract the right people.

Longer school day, data-oriented
instruction and the culture piece.

You got to make sure that everybody in the
hallway, from the teachers to the janitors,

thinks that 100 percent of
these kids will go to college.

If you don't have that you're going to fail.

So, I would like to figure out, I know
that's complicated how you do an experiment

like that and all that, I get it.

But the goal would be to try to figure out are
there three or four things that we should take

from this or keep or an achievement first.

Put it to a school-based randomized
trial where we get 100 schools to try--

to sign up, 30 of them get this four
element treatment, the others don't,

to see if we can actually get gains
over three years [inaudible], right.

So, last statement there,
I promise I'll shut up.

Here's why I'm so excited about this.

I think before we conducted
this analysis, at least for me,

I was sitting in my computer saying, oh my God,
nothing works, nothing works, nothing works.

Contributing to fatigue,
people saying nothing can work.

This all-- you're just wasting your
money, nothing can work, nothing can work.

Now I feel like we're in a spot
where there is something out there

that is actually has parts of the tide.

It's actually working.

The question is, how can we
boil that down to pill form

so you could transport in other places.

But for me that's a much, much,
much, much better place to be in.

I know there's something.

I just don't what it is versus
maybe [inaudible].

Okay. Yes ma'am.

>> A quick question about
the [inaudible] places.

Obviously, person needs to be part of
the field somehow [inaudible] leadership.

>> I totally disagree with that in
the best way, with all due respect.

I just-- [Inaudible Remark]
Okay, the question is, she says,

and I get this question all the time is that one

of most important things is going
to be to clone Geoff Canada.

[Inaudible Remark] Okay, his type of leadership.

Okay.

>> Okay, that's a broad concept.

>> Okay, that's a broad concept.

Now I'm more on your board-- more on board
such that he is a good manager, okay.

He does, and that's it.

He's a good manager and we-- the supply
of good managers I think is large.

A lot of people take he's a mystical figure

who somehow miraculous can change
schools because he's so smooth.

I say no, he's like my uncle.

He just got a job, that's the only difference.

And look-- and he's-- so, I mean, he's
good and he says he can raise money

but he's a good manager, he has
good performance management.

He sticks to goals, has timelines and
I think we got a lot of folks out there

like that and we certainly got to one.

[ Inaudible Remark ]

Yeah, I can add it to my list
of things I don't know but yeah,

I think that-- I think you may be right.

I mean it's part of what's going on.

I mean, yeah, I think you could be right.

>> Yeah. Are you familiar with that experiment
they did back in '84 out in North Carolina State

where it had college students participate in the
apocentric [phonetic] classes versus the none

and the ones who were in apocentric
classes had GPAs improved by 1.2?

>> No, I haven't seen that.

>>Okay. But I'm just saying that--

>> Well you send it to me.

I would like to read it.

>> Okay.

>> Please do, please do [Inaudible Question]

>> Yes and no.

I have a cost per kid-- I
have cost per kid in the zone.

I don't know what denominator kids take up the
program, okay, so that's 5,000 dollars per kid.

Okay, you raise these 50 million dollars a year.

He's got 10,066 kids in the zone that
he treats for one service or the other.

And you know, so that's the 50 million.

How much of that gets allocated to
the schools versus not, I don't know.

But it's in the 3 to 5,000 dollar over

and beyond what normal New York
City public schools get per kid.

>> I have a question about your
definition of schools and communities.

It's been interesting to hear schools
discussed in contrast to that communities.

>> Yeah.

>> Schools are important part of communities.

>> Yeah.

>> And part of building an effective
school community is making the connections

with that community.

So, I'm just wondering about-- maybe it's
a language [inaudible] I'm just wondering

about how the [inaudible].

>> I will admit I really don't understand your
question but I'm happy to answer it anyway.

[Laughter] Yes, okay.

The slight of hand I did on you today
was I've redefined what a school is.

Okay, that's the slight of hand I did which
is I've got, you know, I've sat at schools

but I've got schools that you know,

he's got a school that's got you know
social workers inside the school,

you know, bababa, right?

The reason I did that is 'cause I think there's
a lot of public schools who are doing that.

Like the number of New York City public
schools I'm gone in and seen a dentist

in the next room from the
ELA teacher is amazing.

And so I think we can get there in
terms of our definition of school.

So I think that-- I'm using the
broadest definition of what school is.

And I just really want to contrast that with
purely community programs that are after school,

et cetera, in terms of driving achievement.

Now whether or not they affect teen pregnancy
and those others, I don't know that yet,

but that's coming, these kids
really aren't old enough, right.

All these kids in the zone
right now are in 10th grade.

So they'll be in 11th grade soon and then they
will start doing the stuff I did 11th grade,

we'll get some good data, okay.

But and-- so that's just the reason,
just a time constraint, we're waiting.

Yes.

[ Inaudible Remark ]

>> Oh, it's in the paper in
the tables, I don't know.

It is lower than-- it's rough--
I think it's roughly the same.

I think it's a little lower than
the public schools in Harlem

but I think we reported in the paper.

It doesn't really matter
for our statistical analysis

because as soon as you are in-- we're fine.

But I don't-- we reported in
the paper just, you know, age.

I don't know but it's in the paper.

Yes.

>> I don't think we need to clone more
Geoffrey Canada but I think we need

to talk a little bit more about what he did.

>> Yeah.

>> 'Cause there's not very [inaudible] people
in our country that are doing what he did

which is he carved out his own community and
he identify what he thought were overtime?

>> Yeah.

>> And he was brave enough to say I
may raise enough to start my own rule.

>> Yeah.

>> So, I'm not going to have to play by
the rules of the game, he has to stay or--

>> Yes.

>> City or district or [inaudible].

>> Yes, yes.

>> Contra dictates and that's a huge thing

>> Yeah.

>> Because as long as you're a leader that has
to lead by the terms of how you get your money.

>> Yeah

>> You can never set it up to [inaudible]
be attracted by what you thing is adoptive.

>> Okay, I was with you until your last
sentence which is his still has to live

by the terms of which he got his money.

There's no such thing as
pre money, I noticed that.

>> Yeah.

>> So, [inaudible]-- I have lots
of fun just doing [inaudible]

like we got ideas, I'm going to be okay.

[Laughter] So, [inaudible]-- but I take
your point, you got a lot more flexibility

than the average person who's over school but
not the average person is over a charter school.

So, two points as in, you know,
the flexibility to be able to--

to develop solutions that you think
are important locally is potentially an

important thing.

Second which I didn't mention
yet in the talk is the--

you know, if you look at Caroline's
work on charter school as you realize

that there's a distribution of treatment
effects in the right tale of that distribution,

others schools they don't have any in this
community programs were getting, you know,

exactly they are the same
results but very similar.

If you look at the work that Josh [inaudible]

and Tom Cane [phonetic] are doing using
charters schools in Boston, you know,

they're finding very similar results again,

they don't have the community element
or the [inaudible] a part of it.

But what they do share is this kind
of, you know, a few elements, you know,

they had longer school days, they basically--
I hate this no excuses thing 'cause-- whatever.

But your [inaudible]-- they basically educate
kids with whatever it takes and so it has a lot

of flexibility that you added
but again I just why it hates it

when I say this I told your men is not
all that, she's like you don't know.

[Laughter] So I [inaudible]--
'cause I hate [inaudible] on that,

I think it's really important because
even people who are his funders think, ow,

'cause just because we got miracle
Jeff and this can never be replicated.

I'm like, yeah, but there's lots of-- there are
other schools that are getting similar results.

And so, I think its too convenient
to say this is a miracle.

I think the best thing we can do
as researchers is collect data

and try to demystify actually what's
going on and make it more formulate.

[Inaudible Remark]

>> So, we're going to-- that's going
to be some of our outcomes that we do.

So let me just say, you know, is-- I don't know
it is a very complicated question because it--

I think I don't actually trust people who go
around and say, "Do you want to go to college?"

"Oh yeah."

You know, I-- I don't know if I trust them.

Right, but I can tell you there is
no expectations they are between--

they are no big group differences
and expectations.

I think some of the group differences come in
and how you actually achieve those expectations.

It's under like the systematic ways,
you know, I want to be a scientist

but I don't like science, that kind thing.

But I don't think if you look at expectations
and at least the date I know about,

look at ECLS-- the GS-- GSS et cetera.

If you look at those you know
and if you ask is your kid going

to be what degree are they going to get, right?

And an enormous fraction of mothers

in the highest poverty places saying their
kids are going to be [inaudible], okay?

So, I-- I mean, again I don't know
what that measures but it is certainly

to [inaudible] measures real expectations
then that seems to be an expectation.

What I don't know is how the treatment effect
of the charter alters those expectations,

if it makes them more real, you know, in
terms of like giving them on the path edge,

you get this, let me get-- [Inaudible
Remark] So, we should talk about this offline

but [inaudible] one of the things that I learn
a lot and in the last couple of years in terms

of trying to get involved in education
realize how hard it was and its like one

of the things I've learned is
like design a survey question.

[Inaudible Remark] Okay, so like even your
question I'm not trying to pick on you

but like there is no way that kids are going to
be like concrete, I don't know, concrete, what?

[Laughter] I really had to designed that, I'm
happy like offline after this I talk to you

about precisely how you work with that but yet
the answer is yes, people try to get it down.

I don't know whether or not the
guy you have do your satisfaction.

>> [Inaudible] with that, I'd like to thank
Roland for joining us and-- [Applause]