PGR: Survey Results Analyzed

By Luis "Suar" Suarez, @PG_Suar

The Numbers Game

Following the release of the PGR Top 50, members of the community are starting to wonder more about the inner workings of the system and its occasionally controversial, though objective, results. With 15 regions represented, 48 panelists consulted, and over 35 tabs in a very packed Excel Sheet, there are many conclusions to be drawn from the data that was researched.

Notably, the survey that was given to over 50 people from around the world yields some pretty eye-opening data.

50 names were drawn out for the survey from high placings at the 22 tournaments that were researched. All the usual suspects that either won or placed top 4 at a tournament were immediately noted and written into the survey -- namely ZeRo, Nairo, and Dabuz.

As old tournament brackets were dusted off and we delved deeper and further into top 8, top 16, and top 32, many names started to come about that we were anticipating.

After pruning the list of inactive members, one-time showings, and other outliers, a preliminary Top 50 list was created and sent off to surveyors around the world.

In order to be as inclusive as possible, we reached out to members of the community in every active Smash community that we could find. Whether they were content creators, TOs, or competitors, we wanted the opinions of anyone we could ask. With a majority of the surveyed residing in North America, we made special efforts to contact those in Japan, Mexico, Europe, and Australia.

The instructions were simple: Give a rating on a scale of 1-10 with 1 being the weakest and 10 being the strongest and 0 being a “skip” option. Those who were surveyed that were also on the survey itself were asked to rate themselves at 0. After a 2 week process of sending out the survey and following up with the panelists through many different mediums and languages, we officially closed the survey and started studying the numbers we received.

The Results

With nearly 2,400 pieces of data, we organized all of the ratings into a spreadsheet in order to run tests.

One of the first things we did was gather all of the averages of the ratings for each name on the list and then we calculated the associated standard deviations. The standard deviation is a measurement of how differently people voted. The higher the SD, the more people voted all over the place. For example, some people voted MVD very high (10) and others very low (3) so MVD’s ratings have a high standard deviation while almost everyone voted ZeRo at 10 so he has a standard deviation close to 0.

We felt that it was imperative to include this measure since it shows by how much the opinions of the panelists varied on any given name. We found that the most controversial players had the highest SDs, while the shoe-ins, like ZeRo, had extremely low SDs.

The SD shows the difference between all the values and the average that was found for each player. The larger the SD, the greater the variability in the opinions. Conversely, the lowest standard deviations were associated with opinions that did not vary much at all.

The Highs

The Mexican Terror, Leo, had the highest SD of any player ranked on the PGR. With a bold 19th finish on the PGR, he also touted a SD of 2.05 and an average rating of 7.9. We figured that such a high variance in his ratings can be attributed to the fact that previous to GOML 2016, Leo had absolutely no North American tournament attendance. However, his rating was so high because many felt that his 1st place dominance of so many tournaments in Mexico for so long had to be reminiscent of a godlike player coupled with the fact that he eliminated the likes of Mr. R and Vinnie.

Following in second, Mr. ConCon had a SD of 1.87 and an average score of 6.67. Given his history-making set win over ZeRo, this SoCal local has garnered much support but also many dissenters. While many pinned it to being a fluke, others recognize that Mr. ConCon is a tournament threat and respect his dedication to Luigi despite the nerfs. Regardless, a lack of a tremendous impact at majors have left many doubting his ability which is reflected in the spread of his ratings.

Static Manny’s results landed him with the 3rd highest SD of 1.74 and an average score of 6.86.  A Tampa native, Static Manny won many regionals in Florida and also made a decent appearance in the beginning of the scene, but trailed off towards the beginning of 2016. This, combined with other Sonic mains such as 6WX and Komorikiri taking the spotlight left many of the surveyors skeptical.

Many other players that had high SDs, such as iStudying and Nick Riddle, had some sort of high octane situation that brought them to the community airwaves, but either did not follow up and deliver a second showing or left many disappointed by not improving in the way they thought they should.

The Lows

As many can imagine, there was only 1 ranking in the entire PGR that was incontestable - number 1. With a SD of .15 and an aggregate score of 9.98, the 48 surveyed overwhelming gave ZeRo a 10/10 with only 1 or 2 opinions dropping down to a 9. Unsurprising, given ZeRo’s historic track record in Smash 4 today.

Second to ZeRo is the Japanese threat Abadango. Settling in with a SD of .70 and an average score of 9.17, Abadango definitely set a name for himself in the Western Hemisphere with a 1st place finish at Pound 2016. Even though Abadango has a wild time in Japan along with the other Japanese smashers that place in and out of top 8 at events, Abadango has consistently seized top 8 finishes in American tournaments. The community will never forget his Mewtwo rolling over everyone in McLean, Virginia at Pound 2016.

After Abadango, Dabuz had the 3rd lowest SD value. At .74, Dabuz placed 3rd on the PGR with a survey rating of 9.64. Coincidentally, Dabuz has always been the third name to be included in sentences that start out with, “ZeRo, Nairo, and ... “. Consistent in Tristate and always in and around top 8 at nationals and majors, Dabuz has unquestionably always been the next best player after ZeRo and Nairo. Getting consistently eliminated by only the Top 5 but eliminating everyone else has the community placing Dabuz at 3rd in their heads and in the survey.

Honorable mentions are Mr. R and ANTi, SDs of .77 and .88 respectively. Surprisingly opinions on ANTi did not vary a considerable amount despite his absence at tournaments where many expect him to be. Mr. R, on the other hand, has been a consistent international threat and his impressive itinerary boasts impressive results worldwide and, thus, high opinions. Also both players being Brawl veterans also draws a level of favorability that others on the list did not always have.

Of course, we did not want this survey to be the only metric by which we rank players on the PGR so, instead, we relied on data derived from placings and set counts from the 22 tournaments we researched since Apex 2015.

For example, Dabuz was 1 place higher on the survey list than on the objective list and Nairo was 1 spot lower on the survey list from his spot on the objective list. While historically Nairo has always been regarded as the next best player after ZeRo with several tournaments wins and even two sets over ZeRo, which Dabuz does not have, a 49th place finish (Pound 2016) can explain a dramatic amount of panelists not favoring Nairo over Dabuz. Nairo’s survey score was 9.50 and Dabuz’s aforementioned 9.64 trumps it by only .14 -- this bias is important to realize when current events can be related to people’s perceptions of the top players.

Thus, the X Factor was perhaps the most important quantitative measure to have come about from the survey since it was able to relate the bias of the panelists to the factual data we found. High X Factors and low X Factors will always tell a different story depending on the player, but it will definitely be something that will stay with the PGR for many iterations to come.