To build an MBA ranking, Bloomberg Business Week publishers collect thousands of data points and run a multitude of calculations. The possibility of miscalculating is high, especially if the people overseeing this highly scrutinized annual accounting have little or no experience in statistics. A sophisticated analysis of the latest 2021-2022 MBA ranking by Business week shows the magazine made a big splash on its recently published list.
The analysis, carried out by Yale School of Management Associate Dean Anjani Jain, revealed major errors in the ranking. Using the scores reported by Businessweek for each school, he found that applying the stated weights to the five measures used by the magazine would lead to results “extremely out of step with the published rankings” (see his full analysis here). And he calculated that the only way to reproduce the rankings published by Business week is to apply a very different set of weights to the five metrics.
In fact, Jain’s recalculation would change the positions of 23 of the top 25 business schools, and in many cases the changes are so dramatic they fuel suspicions that Business weekEditors may have felt the need to modify the results to maintain some credibility. Apply the indicated weights to the scores published by Business week unusually dropped MIT’s Sloan School of Management to 21 from 7th place; the Jindal School of Management at the University of Texas at Dallas to soar into the Top Ten, ranking ninth, up 22 places, and the Goizueta Business School at Emory University to achieve its highest ranking in the tenth place, eight places better than the published ranking of 18th.
INSTEAD OF WEIGHING COMPENSATION BY 35.7%, BUSINESSWEEK APPEARS TO GIVE COMPENSATION A WEIGHT OF 58.5%
Jain, an expert in optimization algorithms and combinatorial problem analysis, doesn’t think the dud was intentional, but it could have resulted from serious errors in applying the methodology shown in the ranking. Bloomberg Business Week says its overall ranking is made up of five indices that claim to measure five aspects of an MBA program: compensation, learning, networking, entrepreneurship and diversity. However, in trying to replicate the results published by the magazine with its own reported data, these weights appear to be misapplied.
Instead of weighting the 35.7% offset as Business week argues, the true weight to replicate the 2021 ranking shows that the compensation index was given a weight of 58.5% (see table below). Although Business week maintains that he weighted “learning” by 25.8%, the true weighting of this component turned out to be only 7.7%. The impact on the overall ranking itself is considerable. Yes Business week Weighted the scores correctly, Wharton would have fallen to a rank of 28th, well below his already surprisingly low rank of ninth. Indeed, the “learning” index is largely made up of responses to a survey of students, and many MBA students surveyed by Business week were very disappointed with the school’s response to the pandemic (see Revenge Of The COVID Cohort).
Jain offers a thoughtful explanation for the unusual rankings of Wharton and MIT Sloan, two MBA programs that are widely considered to be among the best in the world. His point of view reinforces the belief that the differences between and among schools are often so small that they do not make sense statistically, unless you add a metric that has greater variation in the results. In this case, the variation appears to come from these student surveys, as expressed in his “learning” index. In the past, Business week would smooth these results by combining them with previous student surveys. It wasn’t this time around, and the impact was severe. However, Businessweek has maintained this approach for its surveys of alumni, with a weight of 50%, 25% and 25% on their responses, from the most recent to the least important.
WHY WHARTON AND MIT SLOAN ARE SO BAD IN THE RANKING
“On the ‘learning’ index of the ranking, Wharton and MIT are both in the bottom quartile of 84 American schools (Wharton at rank 78, MIT at 64 – I note this without editorial comment),” writes Jain. “On the other hand, UT Dallas (Jindal) has the highest ‘learning’ score, ranking # 1 on this index. If “learning” contributed 25.8% of the overall score, Wharton and MIT would need to have dominant scores on the other indices (which they don’t have) to reach their high overall ranks. “
Of course, these counterintuitive results alone would raise serious credibility issues with the ranking. But there is more. Of the 84 U.S. business schools ranked by Business week, 79 of the MBA programs would be ranked differently, 15 of them would experience double-digit changes from their published rankings. Duke University’s Fuqua School of Business would drop 14 places according to the new calculation, from 15th to 29th place. Cornell University’s Johnson Graduate School of Business would drop 12 places, dropping from 20th to 32nd. The Ohio State Fisher College of Business loses 14 places to rank 58th instead of 44th. Baylor University’s Hankamer School of Business would jump 18 places to rank 39th, much better than its published ranking of 57.
In academic research, the ability to replicate the results is essential to the credibility of the authors and the results of their research. When these results cannot be duplicated using the researchers’ own methodology, it raises red flags about the validity of the study. Jain’s scan is a massively large red flag, much like one of those oversized American flags that fly in front of American car dealerships in the Midwest. He notes that “it is problematic (and ironic) that BBW’s published ranking of American schools cannot be replicated by applying their stakeholder-generated weights to the five indices that make up each school’s overall score.”
NOT THE FIRST TIME BUSINESSWEEK RELEASES AN ERROR RANKING
If Jain’s analysis is correct, this wouldn’t be the first time that Business week posted an incorrect classification. Eight years ago, errors in the calculation of what was then a measure of a school’s “intellectual capital” led to major revisions in the ranking. One month after publication, Business week corrected errors online, leading some schools to move up to 10 spots in their intellectual capital rankings. At this moment, Bloomberg BusinessWeek was not so keen on publicizing his own failures. The magazine quietly revised its online rankings with little fanfare or notice (see BusinessWeek’s Big Oops Ranking Moment).
This new challenge of Business weekRanking is the ultimate oops because it impacts not just one data point in the ranking, but all of them. Like any solid researcher, Jain did his homework and then brought his findings directly to Business week for an answer. What came back from Caleb Solomon, the editor in charge of the classification, did not reassure him. In the exchange, dated September 30, Solomon confidently told Jain that his analysis was completely wrong and then got into the weeds of the magazine’s methodology.
Page 1 of 4