What Do Those Star Ratings Mean?
Every year, those of us who cover recruiting hear a common refrain. “Stars don’t mean anything.” These “stars” refer to the one- through five-star ratings assigned to high school football recruits by the three major services. They assign stars based on potential. The scouts at those services take on the unenviable task of rating as many players as they can. For each class, this means they look at film and camp results for hundreds of thousands of high school players. The question becomes, then, what do those star ratings mean?
The Ratings Are Not Perfect
If the ratings were perfect, the team with the highest-rated recruits would beat the team with the lower-rated recruits all the time. Such a world would also produce draft picks according to star ratings meant to measure a player’s future trajectory.
Spoiler alert: nothing is perfect. This means that, even if the rating system does a decent, or even good, job at predicting results, we would never expect perfection. Teams will not draft 100% of 5-star players. Similarly, recruiting ratings of respective teams will not predict the outcome of every game.
To find success, a player must realize his potential, and a team must perform. Even the highest-level performers have to continue to perfect their craft. Those performers must remain relatively injury-free to produce a solid enough body of work to impress NFL scouts. In other words, they must still show up.
Thus, we would never say that recruit ratings guarantee success for players or teams. They do not. Instead, we simply want to know how well recruiting ratings predict future success for both teams and players.
Assessing Star Ratings Objectively
If, as the old refrain goes, stars mean nothing, then we would see very little relationship between the ratings and real-life performance. Therefore, we have to measure performance against prediction. That would help us answer: what do those star ratings mean?
The Plan Of Attack
We decided upon a plan. We would first look at multiple draft classes over the last decade and compare the results to the ratings of drafted players. Next, we would take every game in several college football seasons. For each game, we would predict the outcome based on 247Sports’ team talent composite rankings. In the talent composite, 247Sports measures the total talent each team has on its roster for the first game of the season. It then assigns a score. The higher the score, the more talent on the team (weighed solely by high school recruit rating). Without adjusting for injury, coaching changes, momentum, or AP rank, we assigned a team a win or loss based solely on the teams’ talent composite rankings.
Third, we would look at the final AP rankings each year and compare those rankings to the teams’ talent composite rankings. The more closely they matched up, the more accurate the predictions. Finally, we would look at the playoff teams and the national champion and compare those results to the same composite rankings.
Comparing Star Ratings to NFL Draft Data
We looked at four different draft classes since 2010. While the modern rating system started in 2001, it took a couple of years for the services to rate more than the top thousand or so players reliably. Thus, we started with the 2010 class, given these services the benefit of the doubt for the first five years. We used the most recent two draft classes and picked two other classes at random.
Defining the Data
For each recruiting class during that period, we figured out how many players there were in each rating group, using four groupings. We divided the rated players by four categories. First, we looked at five-star players. Then, we looked at four-star players. Next, we looked at three-star players. Finally, we looked at players below three stars and grouped them together.
For each recruiting class, we found an average of 33 five-star players, 332 four-star players, 2,087 three-star players, and 10,044 players below that. On our last category, we were generous. In reality, there are over 100,000 high school players vying for football scholarships. But, on average, just under 12,500 (12,496 to be exact) incoming players actually find a place on an NCAA roster each year. So we limited our review here to only those who made that particular cut.
For each draft class, we found an average of 255 players drafted by NFL teams. So for each class, we determined how many players from each of our four categories were selected. We also noted which round of the draft each player was drafted in. In general, the average draft selection would have to be in the middle round of the draft (the fourth). Anything above is better than average, and anything below is worse.
Revealing the Data
Five-Star Recruits
In these drafts, the average number of five-star recruits drafted was 20. Since each class has an average of 33 such recruits, we determined that 61% of five-star recruits ultimately get drafted into the NFL. Considering five-star recruits make up just 0.26% of each incoming class, their representation in NFL drafts is strongly predictive of an NFL future. Of the five-star players drafted, we also found that the average player was drafted in the second round, well above the mean draft placement.
Four-Star Recruits
The average number of four-star recruits drafted was 78. Since each class has an average of 332 such recruits, we determined that 23% of four-star recruits ultimately get drafted into the NFL. These recruits make up just 2.7% of each incoming class, their designation is also strongly predictive of an NFL future (though not as predictive as a five-star rating). Of the four-star players drafted, we found that the average player was drafted in the third round, which is still above mean draft placement.
Three-Star Recruits
The average number of three-star recruits drafted was 106. Each class has 2,087 such recruits. As a result, 5.1% of three-star recruits ultimately get drafted into the NFL. This group comprises 17% of each incoming class of NCAA players, so a three-star designation is not fairly predictive of an NFL future (though there are certainly a fair number of these players drafted each year). Of the three-star players drafted, we found that the average player was drafted in the four round, which is the average draft placement.
Two-Star Recruits
Finally, the average number of recruits below the three-star cutoff drafted was 51. Since each class has an average of 10,044 such players, we found that just 0.51% of these players get drafted. This group comprises 80% of each incoming class. Therefore, falling below the three-star cutoff was not predictive at all of an NFL future (even if it didn’t completely foreclose the possibility). Of the players drafted who fell below the three-star level, we found that the average.
Conclusions Drawn
So what do those star ratings mean when it comes to an NFL future? They are not perfect. But they have measurable positive predictive value. A five-star player is more than 260% more likely to be drafted than a four-star player. A four-star player is more than 450% more likely to be drafted than a three-star player. And a three-star player is ten times more likely to be drafted than someone falling below that threshold.
This means that a high recruiting rating does not guarantee success. Of all five-star recruits, 39% of them will not be drafted by the NFL. But considering blue chip players make up less than 4% of all players who play NCAA football, but 38% of all players drafted by the NFL, they definitely mean a lot, especially considering blue chips find their way, on average, to be drafted well above the mean draft placement.
Using Star-Ratings to Predict Wins and Losses
Here, we looked only at regular season team schedules for multiple college football seasons. We limited our analysis here to the Power Five teams plus Notre Dame and BYU. In part, we did so because the sample size of games is much bigger if including Group of Five teams. It would be easier to make mistakes in our data, and it would take far longer to produce and analyze the results.
That said, logic also dictated the result. As seen above, the cut-off point between a three-star player and a player falling below that cutoff results in the most substantial drop-off in production. Because Group of Five schools, on average, bring in just nine three-star players a year, just a couple of injuries at key positions leaves an unrated player starting. In the Power Five world, however, the drop-off in anticipated performance from a starter to a backup is much lower. So we would naturally expect the actual outcomes to vary wildly based on things not necessarily related to talent level.
Our Methodology
Our methodology here is simple. We identified each team’s talent composite ranking and charted each team’s schedule. Based on who they were playing, we determined what they should win or lose that game. We did not factor in home-field advantage when predicting the outcomes. We also did not weigh injuries, coaching staff issues, suspensions, or AP ranking. Instead, we assigned a simple “W” or “L.”
Then, we looked at the outcomes. If the more talented team lost, we got it wrong. Conversely, if a less-talented team won, we got it wrong. Then, after compiling this data for each team, we determined how accurately the talent-based rankings predicted outcomes.
The talent composite rankings, as we noted above, rely solely on recruiting ratings. Simply, 247Sports assigns each player on the roster (whether injured or not) a score based on his recruiting rating. The better total score, the better the talent composite rank. If stars didn’t matter at all, we would simply have a 50/50 split. There would be a winner, and a loser, and this manner of predicting outcomes would bear no consequence.
Revealing the Data
We start by diving into one particular season of data, so that inquiring minds can check our results. We chose the 2019 season because it featured the least amount of predictability from the three years we assessed. In 2019, Power Five teams played a total of 275 conference games. The talent composite ranking accurately predicted the outcome for 65% of those games. Of the incorrectly-predicted games, the home team won 55% of the time.
In that season, Power Five teams played a total of 135 games against non-Power-Five teams. The talent composite ranking accurately predicted 80% of the games. The home team won just 8% of the incorrectly-predicted games.
Power Five teams played a total of 40 out-of-conference games against each other. The talent composite ranking accurately predicted 75% of the outcomes. The home team won 51% of the incorrectly-predicted games.
In total, the talent composite ranking predicted 70% of the regular-season games the Power Five (including Notre Dame and BYU) teams played in 2019.
Over the three years we assessed, the talent composite ranking predicted a total of 68% of conference games. It predicted 82% of the out-of-conference games against non-Power-Five teams. And it predicted 77% of all regular-season out-of-conference games Power Five teams played against each other.
Conclusions Drawn
Again, if stars meant nothing, our predictive analysis would mean nothing. We would fall within a couple percentage points of the actual results every year. Instead, over a three-year sample size of over 1,300 games, the talent composite accurately predicted the outcome 73% of the time. Again, these star ratings do not achieve perfection, but they certainly mean something.
Comparing Recruiting Ratings to Final AP Polls
The next measure we decided upon compared the final AP polls for several years to the talent composite rankings for those same seasons. We looked at all final polls between 2015 and 2019. We chose to exclude 2020 because of the COVID-shortened season and the substantial number of opt outs by players. Again, we focus our analysis on the Power Five teams plus Notre Dame and BYU (for the same reasons we discuss above).
In that five-year period, we see that the Power Five teams had 103 appearances in the final AP poll. Of this sample size, we found that teams with a Top-25 ranking in the talent composite rankings found a place in the final AP poll just under 70% of the time.
Again, while recruiting rankings, and the resulting team talent composite measures, do not paint a perfect picture, they come pretty close. That said, strong concentrations of talent do not guarantee success. Similarly, a lower level of talent falling outside of the top 25 teams does not prohibit success (at least when weighed by this particular measure).
Rising Above Their Station
Some schools do better than others with the talent they have. For example, out of the five seasons we reviewed, Wisconsin found its way in the final AP poll four times without having a team talent ranking inside the Top 25. Three schools did this three of the five years: Iowa, Northwestern, and Oklahoma State. Five teams did it twice: West Virginia, Baylor, TCU, Utah, and Virginia Tech.
Four schools went above and beyond here, outpacing their team talent ranking by more than 35 spots. In 2019, Minnesota finished 36 spots in the final AP poll greater than their talent composite ranking would predict. In 2018, Washington State finished 47 spots greater than predicted, and Syracuse finished +37. Then, in 2016, Colorado finished 45 spots greater than predicted.
Doing Less With More
On the flip side, several schools routinely did less with more in these five seasons than anyone else. Of the teams with a talent composite ranking within the top 25, seven teams failed to finish in the top 25 in the AP poll three or more time. UCLA leads this category with five such seasons. South Carolina, Ole Miss, and Texas A&M earned this dubious distinction four times. And Tennessee, Texas, and Miami earn this “award” three different seasons.
We found that it was pretty rare for a team with top-ten talent levels to avoid the Top 25 in any given season. In fact, that happened less than 20% of the time. But, again, some teams found their way on this list multiple times. Florida State leads the way. Despite having team talent rankings of 4th, 5th (twice), and 6th, the Seminoles failed to make the final Top 25 four different seasons. USC makes this list three different seasons (with a team talent ranking as high as 2nd). Finally, Georgia makes this list twice.
What Do Those Star Ratings Mean at the Highest Level?
Finally, we compared star ratings and the result team composite talent rankings in the context of national champions and playoff contenders. Not surprisingly, the ratings made a great deal of difference here. Again, we focused our effort on the 2015 through 2019 seasons.
In these years, 100% of the teams selected to participate in the College Football Playoffs appeared in the Top 25 team talent rankings. Eighty percent of those teams appeared in the Top 20. And 70% appeared in the Top 10.
Most importantly, the eventual champion appeared in the Top 10 in the team talent rankings in each of those five years.
Clearly, star ratings matter. Like everything else in this world, they are not perfect. Teams like Florida State, USC, and UCLA prove that talent without execution means very little. That said, the road ahead for teams without a high concentration of talent and players without the stars behind the name is far more difficult. And this is why those of us who follow recruiting closely focus so heavily on them.
