Note: This content was originally hosted at nobbs4ccs.com during the 2022 CCS school board election.
Academic Decline
Brake, Brown & Sharp’s claims about CCS’s academics are what initially caused me to start digging into data. When it comes to state proficiency rates, getting to the truth requires data, nuance and common sense. I’ll attempt to supply all three below.
The Claim
Generally, that Carmel schools are in a state of academic decline.
Slides from Dr. Amy Dudley’s 2021 presentation that show 2017 – 2021 rates are often used out of context, including one graphic by the candidates that states ‘Reverse the decline’.
On the campaign’s Facebook page, comments have been made claiming declines over multiple periods (2011 – 2021, 2017 – 2021, 2019 – 2021 and maybe some more that I missed). As I’m currently blocked from the page, I cannot say which were made by the candidates and which were made by their supporters.
The Short Response
Relative to other schools, CCS has performed superbly, taking the #1, #2, #3 or #4 spot in the state every year since 2011. It should be noted that we are the only school system in the state to be in the top 5 for every year in that time period and one of just two school systems to be in the top 10 for each of those years. Over that time, CCS consistently ranked high nationally as well and continues to do so, with niche.com ranking CCS as 112 out of 10,751 and US News & World Report ranking Carmel High School as 394 out of 17,843.
The Long Response
Let’s say someone, we’ll call him Will the weather nut, was tracking whether winters were getting warmer or colder by looking at the average daily low temperature from December to February of each year.
From 2011 – 2015, Will measured and averaged the actual low temperatures.
In 2015, Will decided that wind chill is a better metric than actual temperature, so he switched to tracking and averaging that measurement.
In 2019, Will changed from measuring in Fahrenheit to Celsius.
Then, in 2021, there was a once-in-a-century blizzard that resulted in a full week of new records for coldest wind chills.
Now, let’s say that the actual average low temperature remained exactly the same from 2011 – 2020, and then dropped significantly due to the blizzard in 2021. An accurate graph would show a straight line for a decade, followed by a drop in 2021.
Will’s graph of the data he’s tracked would look very different. It would show the line shifting down moderately in 2015, significantly in 2019 and then with another big drop in 2021.
If Will repeatedly showed you his graph as proof that winters were getting colder, you’d probably assume he was either not very bright or intentionally trying to trick you.
The Brake/Brown/Sharp campaign and its supporters have been essentially using hypothetical Will’s flawed tactics to demonstrate that CCS’s academics are in decline.
Over the past decade, there have been three events that significantly reduced published proficiency rates across the whole of Indiana: 1) the change to iSTEP+ in 2015, 2) the change to iLEARN in 2019 and 3) the pandemic in 2020-2021. Let’s take a brief look at each of them.
2015: The Shift to iSTEP+
In 2015, the state revamped iSTEP to become iSTEP+. The test was intentionally made more rigorous with the new goal of measuring the percentage of students on track to be ‘college and career ready’ upon graduation.
Prior to the new test being rolled out, the state communicated to parents that significant drops in scores were expected and “This does not mean that students or schools are performing worse than in previous years. Instead, it simply means that the test is measuring something different…” This original document is still available on the state website here.
The state was correct:
298 school corporations tested in both 2014 & 2015. Every single one saw scores decline.
The average decline was 22.5%.
Carmel schools declined by 11.3%.
I’m not going to go into more detail on this change, but those curious can Google ‘2015 iSTEP+ scores’ for plenty of reporting on it from the time.
2019: The Shift to iLEARN
The state fully revamped its testing standard again in 2019 with the introduction of iLEARN. The new test was shorter, administered via computer and adaptive, meaning that questions get harder if students are getting them right and easier if they’re missing them.
The effects were similar to what we saw with the previous change:
356 school corporations tested in both 2018 & 2019. All but one saw scores decline. (And the single school that saw an improvement was a small charter school that literally went from 3 students proficient to 5).
The average decline was 13.4%.
Carmel schools declined by 12.9%.
There was collective agreement among the Department of Education and Republican and Democrat politicians that the test had been made more rigorous for students. It was widely reported that the declined scores were due to a combination of the test being more rigorous, teachers having little time to prepare it, it being administered via computer and the adaptive nature of it.
View the Republican statement here
View the Democrat statement here
View a 2019 article with a statement from the Department of Education here
Again, there was lots of reporting on this at the time and Googling ‘2019 iLEARN results’ will give you a lot more to read if you’re curious.
2020-2021: The Pandemic
At this point, I think we’re all exhausted by hearing about the pandemic, but I’m mentioning it because it represents the only actual, real drop in CCS’s proficiency rates for the period we’re looking at. Of course, it also impacted schools across Indiana, the United States and the world as a whole.
That said, while the pandemic in aggregate set learning back two decades (see this article for more detail), there were schools that either improved their proficiency or suffered smaller declines. Generally, in Indiana, these were schools that were either smaller or already had very low proficiency.
Looking at the effects of the pandemic:
Of the 348 school corporations tested in 2019 and 2021, 340 experienced declines and 8 experienced gains.
The average decline was 9.2%
Carmel declined by 10.5%.
It should also be noted that while there’s obviously still work to do to recover from the pandemic, CCS’s scores have started improving. Our proficiency improved by 2.7% in 2022, compared to a statewide improvement of 1.6%. In other words, when people post ‘stop the decline’ and omit 2022 data that’s been available since July, they’re either ignorant of the new data or lying to you. Either option should be troubling coming from school board candidates.
Indiana Schools
I want to briefly touch on Indiana education as a whole. When faced with the fact that Carmel is objectively one of the best school systems in our state, some people will make comments along the lines of being ‘best of the worst.’
First, that’s a nonsensical criticism. Even the actual worst states for education have some schools that achieve tremendous results and rank highly on a national level.
But is Indiana ‘the worst’ or ‘one of the worst’ states for education? No. Absolutely not. US News & World Report currently ranks Indiana 9th for K-12 education, as you can see here. Put another way, CCS is in the top 1% of schools in the state in a state that’s in the top 20% of the nation.
CCS Academic Performance
We know that iSTEP, iSTEP+ and iLEARN scores can’t be directly compared because they’re different standards that provide different scores. And we know the pandemic dealt a one-time blow to the entire education system that we’re all going to have to claw back from. So how do we get to a meaningful evaluation of where Carmel schools are at?
I personally find our statewide rankings to be fairly persuasive.
They become even more compelling if you look at just how much we’re outperforming the state average.
We can also look at how Carmel performed in each year against how the data says it should perform. That is, for any two consecutive years, we can use regression analysis to build a forecasting model that allows us to input a school’s score for the first year and receive a prediction of what its score should be in the second year. That prediction includes a specific expected score, as well as a predictive range of scores. Regarding the range, we can say that any actual score will fall within the predictive range with 95% confidence.
Interestingly, CCS beats the specific prediction for every model from 2012 – 2022. In 2019, we did so well that we were above the predictive range.
Some Closing Thoughts
To me, efforts to paint CCS’s academics as in decline require pretending the school system is responsible for the pandemic and extremely deceptive use of data. While we still have a ways to go to overcome the effects of the pandemic, CCS is performing exceptionally when compared to other schools.
I’d like to note that some people prefer to only compare CCS to similarly sized school districts. I understand that argument and it makes sense, as it’s much more challenging to run a system with thousands and thousands of students than it is to run a small school with fewer than 100 enrolled. That said, the data I used for the above comparisons includes all districts reported on by Indiana, not just ones of similar size to Carmel. Some of the rankings would look even better if we limited comparison to similarly sized districts.
Also, it’s hard for me to overstate how alarming I find Brake, Brown & Sharp’s sustained attacks on our school system’s academics. If I thought they believed what they say, I would be deeply concerned about their qualifications. Unfortunately, I think they’re smart enough to know better, which leads me to suspect they’re intentionally misrepresenting the state of our schools so that they can implement an agenda unrelated to our academics.
I can’t imagine this campaign’s approach working in most aspects of the real world. As an example, let’s say someone is interviewing for CEO of a car manufacturer that has consistently been and remains among the most profitable in the industry. The person goes before the board and insists that the company’s profitability is bad, using charts that incorrectly mix data sources. They demonstrate that they have no understanding of the difference between issues that affect the company vs issues that impact the entire industry. Furthermore, the highest level of detail they offer on what they’re going to do to fix profitability is ‘focus more on profits’. And they have no experience in the automotive industry. That person would be laughed out of the room.
As one final example of my criticism of their tactics, on their website and repeatedly on social media, they have shared slides from Dr. Amy Dudley’s 2021 presentation on test results. These slides show both iSTEP+ and iLEARN data, creating the illusion of a downward trend to those who don’t know any better.
When questioned about mixing iSTEP and iLEARN results, the campaign and/or its supporters have responded with comments along the lines of ‘you’d have to ask Dr. Dudley why she did that.’ Except that’s a lie. The presentation that these slides were pulled from was given at the July 26, 2021 board meeting and Dr. Dudley explains why she’s included the data, explicitly pointing out that it is entirely invalid to compare iSTEP and iLEARN scores over time.
I urge anyone interested to watch Dr. Dudley’s comments here. Then go look at how the Brake, Brown and Sharp campaign is using her slides. Make up your own mind on if they’re lying to you about CCS’s academic performance.
Lastly, for anyone who would like to access most of the data used in the previous analyses, the state makes historic test data available for download here.
Commentaires