Beyond Standardized Tests: Student Survey Data in Matrix

We spend a lot of time in Matrix working on standardized test data. There is, after all, quite a lot of it, and almost all of our Matrix clients have standardized test data in our systems. Even so, I always enjoy getting a chance to work on data the districts are generating themselves. One of our clients periodically takes surveys of their students' experiences and attitudes, and they approached us about bringing this data into Matrix, so it would be available alongside the rest of their data.

There were different sets of questions for elementary, middle, and high school students at each survey administration, and though most of them were of the “Strongly Agree” to “Strongly Disagree” form, there were also questions asking about duration, frequency, or specific elements. We were able to teach Matrix how to distinguish the different answer scales and display the data as a grid or a chart, with all the filtering and drill down capabilities our Matrix clients are familiar with.

It's amazing how displaying data in a more engaging way can change the way you experience it. None of the information in these surveys was new to the client, but the first time they saw it in Matrix as a chart, it prompted a real discussion on policy between the administrators. That's a big goal of any data-driven interface: help users see where there are interesting questions to ask, then help them answer those questions and make decisions.

A Thousand Different Reports. Just Three Clicks.

Powerful, Fast, and Flexible
One of the most iconic reports within Matrix is our lightning fast Drilldown report, which allows school districts to dissect their entire set of data in any way they can imagine using just a few clicks. Our Drilldown report processes huge amounts of information, tens of thousands of data points, in just seconds and groups the data by school, grade, and student sub-groups like gender, race, IEP, and ELL/LEP. Here is a list of typical questions which can be answered using the Drilldown report in just a few movements of the mouse.

• In which school did female Hispanic students score the lowest on the state test? Two clicks.
• Do my 3rd grade free and reduced lunch students score better on the AIMSweb test or Acuity? Two clicks.
• Between all of my elementary schools, which teacher has the highest number of special education students? And how did they perform? Three clicks.

Details are Important
All of the scores in the Drilldown report are color-coded to highlight areas where the district is doing well and areas where the district is struggling. District administrators love to check into the Drilldown whenever new test data is uploaded, so they can get an overall look at how their district performed on the latest assessments. If they see any red-flags they can then instantly dive into the data and determine which student subgroup, teachers, or even classroom course section may have caused a drop in scores. Remember, context is everything, so Matrix will always show the number of students which make up each group and allow you to drilldown to the individual student scores which make up an average.
The Drilldown encompasses all of our main goals with Matrix:

1. Bring all types of data together in powerful, but easy to use reports.
2. See top level data, but be able to get to the details in a couple clicks.
3. Give districts the ability to make decisions by centralizing key performance data.

AP Predictor: Using Data to Drive Decisions

The AP Predictor report in Matrix is designed to identify students who are likely to do well in AP courses, so staff can be sure those students know about their AP course options. The predictive element is taken from research published by The College Board showing a strong correlation between PLAN scores and scores on specific AP tests. They provide a list of AP tests and their associated PLAN subscores and cut scores. For example, according to The College Board, students whose PLAN Math and Science scores average at least 26 have a 75% chance of scoring a 3 or better on the AP Microeconomics test.

Of course you could do this math by hand or in a spreadsheet, but your PLAN scores are already in Matrix, so why not let Matrix do the work for you? The AP Predictor report knows which PLAN subscores go with each AP test, and automatically calculates the relevant average when you select an AP test from the menu. You can filter by student name and minimum AP success category (e.g. only students with a 50% chance of scoring 3+ on the selected AP test), and sort by any of the scores.

Coming up with ways to display historical data that make it easier to absorb is a cornerstone of Matrix, but it's always exciting to build an interface that directly helps educators make decisions about the future.

Everyone has data. Our goal is to have everyone use it.

You have the data. We've spent the last few weeks getting it all from your piles of spreadsheets and student information system into Matrix. Now what do you do?
Luckily, Matrix isn't just a data warehouse. Yes, Matrix stores all kinds of data ranging from Fountas and Pinnell to the ACT to AIMSweb and even attendance, behavior, and assignment grades, but the real power of Matrix is in the dashboarding and reporting tools which allow districts to use all of their data to make game-changing decisions for their schools, teachers, and students.

Using Key Performance Indicators

An Index Measure is a numeric value assigned to a student based on various key performance indicators (KPIs) like attendance percentage or test proficiency. This allows us to normalize different academic areas which could never have been viewed on the same scale before. We can then use these student Index Measures to determine school, grade, teacher, and district performance.

Using Index Measures for College Readiness
An increasingly important and popular Index Measure is college/career readiness. Because every district does college readiness slightly differently, the Index Measure can be customized to include any set of KPIs and performance levels. For this example we will look at a college readiness measure which has two performance levels: “college ready” and “not college ready.”
Just showing up is a huge part of college and career success, so the first thing we are going to include is a student's attendance percentage. Let's say if a student has 90% attendance or better we will give them a point. These points will come into play when we determine how college ready a student is.
Next let's look at classroom and assessment performance. In the classroom, we can expect a student to have earned a C or better in their Algebra class to be at the level they need for college mathematics. We also expect the student to have scored at least a 21 on the ACT exam. Now combining all of these KPIs we can come up with the college/career readiness percentage for a student and aggregate it for a teacher or school.

Making it Easy
Index Measures are adaptive and completely customizable. Add new tests or academic indicators to any Index Measure at any time with no manual calculations. Matrix makes it easy: just pick the students and key performance indicators and let the system do the rest. Spend your time using data not finding it.

Behavior Documentation and Tracking for K-12

Helping districts use data to take more proactive steps towards behavior incidents. 

To effectively analyze and improve a student’s performance, teachers often need access to data beyond a student’s test scores. One example of such data is behavior. As with other types of data analysis, the more detailed and more rich the data is, the better we are at noticing trends and making decisions using it.

The Matrix Behavior Documentation module facilitates this process by providing a quick and easy way for teachers to track behavior incidents at a detailed level.

It gives administrators a comprehensive view of behavior across the entire district, at a building level, and at an individual student level. The district can look at this data in a number of different ways including yearly data, monthly data, referral type (proactive or reactive measure), physical location of the incident, time of day the incident occurred, and more.

 Behavior data is then aggregated at a student level for analysis with the student’s attendance, assessment or test scores, and demographics to give teachers and administrators a comprehensive picture of student progress and performance. This also allows the administration to make policy decisions like having additional adults at lunch or bus time to reduce the number of behavior incidents.

Are your students college ready?

In secondary education, there’s perhaps no bigger buzzword than “college readiness” right now. School districts are under enormous pressure to produce graduates that are prepared for college-level course work or career-ready skills that will give them an effective transition into the work force. It’s a constant battle every year, particularly after a 2013 ACT survey revealed that only 26 percent of college instructors believed high school graduates were ready for post-secondary classes.

We've got the data...

We’ve spoken to a number of district administrators who have emphasized the importance of being able to use data to help address this challenge and to ensure their graduates are ready for college. The good news is most districts already have a plethora of data they are collecting – whether it’s state assessments, common assessments, EXPLORE, PLAN, ACT, PSAT, SAT, AP test data, demographics, behavior, attendance, etc.

But now what?

Well, it’s a matter of getting the data into a usable format for teachers and administrators to use on a daily basis to guide instruction and improve student performance.  In order to effectively use data, districts need to be able to break it down and analyze trends, student growth, instruction effectiveness, etc.

Get it into the hands of teachers....

Better access to data that is in a usable format takes the guesswork (teachers' perceptions or anecdotal evidence) out of analyzing student performance. Teachers can spend less time trying to make sense of data and more time using it to drive instruction...

And improve performance.

Getting Past the Eye-Candy Test with Elementary Assessment Data

You can call it the “eye-candy test” problem.

It may not be as pronounced in education as it is in baseball. However, in almost every conversation I have with school district administrators we work with, they are focused on this core challenge: How can we make sure our data – not through some anecdotal perception or observation -- supports our decisions about how to improve student performance?

Brad Pitt played Oakland Athletics general manager Billy Beane in "Moneyball" a story about how the baseball team began using data to try to level the playing field with more profitable teams in the league.

When the concept comes up, I picture the scene in the movie “Moneyball” that depicts Brad Pitt – who plays general manager Billy Beane -- getting fed up with how his old-school baseball scouts are trying to identify the best players so their cash-strapped major league team can compete with the richer teams like the New York Yankees.

“He passes the eye-candy test. He's got the looks. He's great at playing the part. He just needs to get some playing time,” one scout says, while another questions the same player’s confidence because of how good-looking his girlfriend is.

Those more anecdotal observations could be important as a piece of the puzzle, but if you focus on that, decision-making becomes too arbitrary, which is the point of the book and the movie. Statistics and data add so much more.
Read more »

Timeliness is Key for Teachers When Analyzing Performance Data

It's not difficult to find a quote about the importance of good timing. The dangers of procrastination are easily apparent today when the pace of information available to us comes at a break-neck speed via the Internet and social media.

But my favorite quote on timing, which was uttered long ago, surprisingly applies nicely to our world right now, especially when thinking about data and education:

“It gets late early out there.”
- Yogi Berra

In our frequent conversations with school districts and how they use performance data, timing comes up often. One administrator mentioned it’s tough on their schools and teachers when it takes several weeks for the district to receive its NWEA Map results in a usable format. Because of the time lag, teachers often can’t use the last round of results to prepare for the next assessment, which can undermine their purpose of administering predictor assessments to help with student performance.

Otherwise the data can become obsolete because the teachers are not able to analyze it as they prepare their students for the next assessment. Research has shown a huge value of interim assessments is that they are taken several times throughout the year, giving teachers benchmarks to address with the same group of students:

As with annual assessments, interim assessment results generally have the advantage of being comparable across classrooms, but the frequency of their administration means that teachers can use the data to evaluate their own instructional strategies and to track the progress of their current students in a single school year. For instance, data from a district-wide interim
assessment could help illuminate whether the students who were struggling to convert fractions to decimals improved after receiving targeted small group instruction, or whether students’ expository essays  improved after a unit spent reading and analyzing expository writing.

So for districts it’s not a question of if they have the data to help teachers analyze and improve student performance. It’s a matter of getting the data into a format teachers can use and obviously being able to place it in their hands in enough time before it gets too late to use it.

This issue demonstrates where technology will play a huge role, especially the ability to aggregate different types of data quickly. Otherwise, with the Common Core State Standards shift for most states set to occur in 2014-15, as Yogi Berra said: “It will get late early out there” for districts that aren’t thinking about this.

In Complex Teaching Environment, Access to Data is Key

I read author and English teacher Jim Burke’s excellent list of “10 Elements of Effective Instruction” he developed, and two things struck me.

First, he notes with the complexity of the classroom, managing time is critical for teachers.

“Every year it seems we are asked to do more, though never, of course, given more time in which to accomplish the goals,” he wrote.

In the work we do with districts, we’re constantly finding ways so that technology can help teachers do their jobs the best way possible. I often go back to something an administrator told me once:

 “Put yourself in that mindset, hurried, low frustration level, interrupted continually…” 

I thought of that description again in reading Burke’s list that he modeled after a surgeon’s checklist.

The second part that struck me was No. 5 on his list: “Integrate assessment throughout the instructional process, using the data to establish initial understanding, measure progress, provide feedback, refine instruction, and prepare students for future performances; this includes students reflecting on and assessing their own performance and progress.”

In the work we do with districts, it’s common to discover most schools have more data than they know what to do with it, and often it’s not presented in a format that can useful to them. Burke is right on that integrating assessment and using data to help students improve is critical. The context of his list related to the lack of time that teachers have is even more significant. 

To be able to use data effectively for instruction, it needs to be in format that’s accessible and easy to use for teachers with their busy and complex schedules. It’s also important for data to be timely or in the hands of the teachers as quickly as possible. That’s where technology, if applied correctly, can make a huge difference. Otherwise, “using data” is something that sounds good in a checklist, but it can easily be either scrapped or not used to its fullest potential because, like Burke noted, teachers are never given more time, only more to do.

Daunting task, but great potential for schools in Common Core switch

Remember the line from the 2004 Disney movie “Miracle” when Kurt Russell, playing 1980 U.S. Olympic hockey coach Herb Brooks, gave a pre-game speech before the historic upset of the Soviet team? 

“Great moments, are born from great opportunity.”

Here’s a video that makes me laugh all the time based on that scene.

 

I thought of that quote when I read this article about the recent Kentucky assessment test scores where proficiency levels dropped by a third or more for elementary and middle school students. The results are significant because this is the first set of scores tied to the Common Core Standards. Most states will start taking assessments linked to the new standards by 2014.

This is a one-state sample, but experts expect this to play out similarly in other states. It will be discouraging for districts, educators and students because the bottom line will show a drop in proficiency levels as they apparently move to more rigorous standards.

But if you think about it, this does present a huge opportunity. As districts and states are already preparing for the change, they should be motivated to get the most out of students, especially if the Common Core Standards are tougher than what they are doing right now. Think of it as an “achievement wake-up” call. It’s a major challenge for schools and teachers.

Even if proficiency levels lag or drop initially, if districts are pushing themselves and their students to measure up to tougher standards, the students will benefit in the long run.

In our daily conversations with districts in a number of states, the switch to the Common Core is in the forefront of their minds. It’s one aspect that makes it critical that districts are using as much data as they can on how students are performing to prepare not only for the change in standards but to give teachers the most effective tools for daily instruction.

It’s a huge challenge. It will be difficult. But it is such a great opportunity that can result in so much more than one “great moment.” It will drastically improve schools in nearly every state and ensure that students are better prepared than they ever have been for college and entering the work force.

Matrix

There is a lot of controversy out there about whether or not we should be even using standardized testing as the best means of understanding a student’s academic performance ability. I am not going to make an argument for or against it, but I will say that it doesn’t appear to be going anywhere anytime soon. So I’d like to tell school districts how they can better use the data that they receive from those standardized tests.

The answer is that we have to find a way to make that data become useful information. Often times schools collect this information but find it too overwhelming or costly to make use of. We cannot have administrators and principals making important decisions for our schools based solely on intuition.

Enter “Matrix.”

Matrix is a web-based software program that acts as a data warehouse and student data analysis tool. With Matrix, school districts will be able to have all of a student’s information in one centralized location. Data from the district’s Student Information System (SIS) is integrated into Matrix right alongside the students’ assessment test scores, so that Matrix can make correlations between and generate reports for these two important data sets.

What does that mean?

Well the demographic information, like gender, ethnicity and meal-plan status from the SIS are correlated to assessment test scores for each student and then presented in an easy-to-read and -understand format for school officials to examine. This helps school officials look into factors that occur outside of the classroom, which may be playing a role in how well the student performs. For example, there may be a correlation between the students who receive free or reduced cost meals and their below average test scores. Of course there is a significant difference between correlation and causation, which must be kept in mind. Just because two things are correlated does not mean that one caused the other to occur, but on the other hand it doesn’t necessarily mean that it didn’t either. Matrix at least presents this sort of information to the school districts and gives them the ability to examine and understand teacher and student performance at a much deeper level.

Consider how a teacher may receive a really bad score (which is just the average of all of her students). Does this mean that the teacher should be let go? No, it means that more information needs to be taken into consideration.

Why did he or she get such a low score?

First of all, Matrix allows all aggregated values to be viewed by their raw components. This allows teachers to see which students are doing well and which are not.

Second, Matrix shows not only test scores, but also growth. Perhaps the teacher teaches a large number of academically failing students, but he or she gets a majority of them to improve their scores throughout the year. Even though the ending score is still not proficient, there was great progress made.

Third, Matrix was designed to accumulate data over a long period of time. This means that school officials can examine information for a teacher over the past several years that he or she has taught; the same goes for students. It is never wise to make critical decisions based on a one-time occurrence. Matrix shows when students or teachers consistently perform at a certain level.

Fourth, Matrix allows higher-up school officials to see which cohort of teachers produces the best students or the worst students. This may point out some key areas that are in need of improvement. This also presents school officials with the opportunity to inquire about the teaching methods of those teachers who seem to consistently performing well and then share those methods with the under performing teachers.

Lastly, Matrix presents all of this information in such an easy-to-read and easy-to-understand format that school districts wouldn’t need to waste money on hiring a data maven or training teachers how to use the product. Also reports can be easily generated and printed in order to share the information with other school officials and/or parents.

The bottom line is that if we ever hope to improve the education of our students we have to know what areas need help and what some of the reasons for that may be. Matrix assists in this process in a cost-efficient, easy and fast way. We know that schools are facing increasing budget cuts as well as increased pressure from the State to meet assessment standards and that is why we developed a tool for educators to use to start understanding where improvements need to be made. Matrix gets that conversation going.