Crowdsourced ratings of football players are underestimated: a study by IDLab researchers

Crowdsourced assessments of the value and ability of football players should be used with caution. As a rule, such estimates are below market value, while the difference depends on the league and the prestige of the players. Dennis Coates and Petr Parshakov, researchers at the International Laboratory for the Economics of Intangible Assets of HSE — Perm, came to such conclusions in the article "The wisdom of crowds and transfer market values", published in the European Journal of Operational Research.

Briefly

In sports, especially in football, crowdsourced evaluation of players is widespread since earnings information is usually unknown, while there is a lot of other data in the public domain (about each match and the actions of the teams). The study uses the market value of the transfer obtained from the website transfermarkt.de. 

What are we talking about?

The crowdsourcing method became popular after James Surowetsky published The Wisdom of Crowds: why the many are smarter than the few and how collective wisdom shapes business, economies, societies, and nations.

Football is one of the most popular and richest sports, especially in Europe, but there is not enough data to study it, for example, about the earnings of players. As a rule, this is confidential information, and contracts are not always indicative, since compensation can be higher than the player's declared salary. The problem of missing data can be solved with the help of crowdsourced player cost estimates for clubs. The website transfermarkt.de (for the German version) publishes such information on a regular basis.

Video game makers are also using crowdsourcing to evaluate the skills of real professional football players to create their virtual likeness. But how accurate is the data? This is what Dennis Coates and Petr Parshakov found out in their study.

How did we study it?

For comparison, 2 types of assessments were used. The first is an estimate of the market value on Transfermarkt, which includes individual performance, age, the future of the player, the current demand for that player and the marketing potential of the player. The second score comes from the FIFA video game simulation developed by EA Sports. Data derived from EA Sports player ratings includes overall rating and ratings across 25 categories.

The study used data on those players who were sold from one club to another at least once and their value was published on Transfermarkt. The work uses data from transfermarkt.de for the period from 1996 to 2016. In total, these are 5860 observations: 598 buying clubs and 955 selling clubs, transfers were made from clubs in 171 leagues to clubs in 96 leagues (these clubs represent the leagues of 83 countries).

What did we get?

The results of the study showed that the crowdsourced valuation is, on average, lower than the market value of the players. In addition, the ratio between the cost of Transfermarkt's crowdsourcing and the actual player transfer fees paid differs between the major and minor leagues. Players from the lower leagues, for whom the actual commission is low, have higher values ​​than players from the higher leagues. Moreover, transfer fees for players from the higher leagues are understated, but not for players from the lower leagues. Actual fees for players with time left on their contract increase by £550,000 to £800,000 the year before the remaining time.

The authors of the article showed that crowdsourced estimates should be treated with caution — they are inaccurate, and a potential source of bias may lie in the mechanism for aggregating estimates.

What is it for?

The findings are useful in addressing the question of whether crowdsourced estimates can be used as a proxy for unknown wages in scientific research.

Also, since Transfermarkt values are often used in negotiations between clubs and players, it is important for both parties to know their accuracy. The results of the study will be of interest to both sports economists who want to understand the football market, and everyone who uses crowdsourcing for forecasts.