Wednesday, June 11, 2014

World Cup Prediction Evaluation Competition

16 June UPDATE: This prediction evaluation exercise is now being updated at Sportingintelligence. See it there!

15 June UPDATED: I have updated the Infostrada predictions with a more recent version, thanks to Simon Gleave (@SimonGleave).

The World Cup starts tomorrow. Prognosticators have been hard at work generating their predictions of who will advance and who will win. But which prediction is the best? Is it the one who picks the winner? Or is it the one which best anticipates the knock-out round seedings? How can we tell?

In an ongoing exercise here at The Least Thing I am going to evaluate 10 different World Cup predictions. To do this I am going to quantify the "skill" of each forecast. It is important to understand that forecast evaluation can be done, literally, in an infinite number of ways. Methodological choices must be made and different approaches may lead to different results. Below I'll spell out the choices that I've made and provide links to all the data.

A first thing to understand is that skill is a technical term which refers to how much a forecast improves upon what is called a "naive baseline," another technical term. (I went into more detail on this at FiveThirtyEight earlier this spring.)  A naive baseline is essentially a simple prediction. For example, in forecast evaluation meteorologists use climatology as a naive baseline and mutual fund managers use the S&P 500 Index. The choice of which naive baseline to use can be the subject of debate, not least because it can set a low or a high bar for showing skill.

The naive baseline I have chosen to use in this excercise is the transfer market value of the 23-man World Cup teams from Transfermarkt.com.  In an ideal world I would use the current club team salaries of each player in the tournament, but these just aren't publicly available. So I'm using the next best thing.

So for example, Lionel Messi, who plays his club team soccer at Barcelona and his national soccer for Argentina, is the world’s most valuable player. His rights have never been sold, as he has been with Barcelona since he was a child, yet he’s estimated to have a transfer market value of more than $200 million. By contrast all 23 men on the USA World Cup squad have a combined estimated value of $100 million. (I have all these data by player and team if you have any questions about them -- they are pretty interesting on their own.)

Here then are the estimated transfer values of each World Cup team:

Team Transfer value
1 Spain  $      1,044,960,000
2 Germany  $         944,160,000
3 Brazil  $         803,040,000
4 France  $         691,740,000
5 Argentina  $         657,720,000
6 Belgium  $         584,640,000
7 England  $         561,120,000
8 Italy  $         542,640,000
9 Portugal  $         517,860,000
10 Uruguay  $         364,476,000
11 Netherlands  $         348,600,000
12 Croatia  $         324,660,000
13 Colombia  $         318,931,200
14 Russia  $         308,784,000
15 Switzerland  $         299,040,000
16 Chile  $         234,864,000
17 Cote D'Ivoire  $         202,389,600
18 Cameroon  $         198,072,000
19 Bosnia and Herzegovina  $         192,780,000
20 Ghana  $         183,708,000
21 Japan  $         164,640,000
22 Mexico  $         152,964,000
23 Nigeria  $         145,908,000
24 Greece  $         134,232,000
25 Ecuador  $         105,588,000
26 United States of America  $           97,104,000
27 Algeria  $           96,096,000
28 Korea Republic  $           88,074,000
29 Costa Rica  $           49,980,000
30 Iran  $           41,076,000
31 Australia  $           36,204,000
32 Honduras  $           35,952,000

In using these numbers, my naive assumption is that the higher valued team will beat a lower valued team. As a method of forecasting that leaves a lot to be desired, obviously, as fans of Moneyball will no doubt understand. There is some evidence to suggest that across sports leagues, soccer has the greatest chance for an underdog to win a match. So in principle, a forecaster using more sophisticated method should be able to beat this naive baseline.

Here is what the naive baseline (based on rosters as of June 5) predicts for the Group Stages of the tournament: The final 4 will see Brazil vs. Germany and Spain vs. Argentina. Spain wins the tournament, beating most everyone’s favorite Brazil. The USA does not get out of the group stage, but England does. All 8 of the top valued teams make it into the final 8.

While this naive baseline is just logic and assumptions, work done by “Soccernomics” authors Stefan Szymanski and Simon Kuper indicates that a football team’s payroll tends to predict where it winds up every year in the league table. Payrolls aren't the same thing as transfer fees, of course, but they are related. Unfortunately, as mentioned above individual player salaries are not available for most soccer leagues around the world (MLS is a notable exception).

I will be evaluating 10 predictions over the course of the World Cup. There are (with links to the data sources that I have used, as of 10 June unless noted otherwise):
The predictions are not all expressed apples to apples. So to place them on a comparable basis I have made the following choices:
  • A team with a higher probability of advancing from the group is assumed to beat a team with lower probability.
  • If no group stage advancement probability is given I use the probability of winning the overall tournament in the same manner.
  • This means that I have converted probabilities into deterministic forecasts. (There are of course far more sophisticated approaches to probabilistic forecast evaluation.) 
  • No draws are predicted, as no teams in the group stages have identical probabilities.
  • The units here, in the group stage at least, will simply be games predicted correctly. No weightings. 
Other choices could of course be made. These are designed to balance simplicity and transparency with a level playing field for the evaluation. Just as is the case with respect to the value of having a diversity of predictions, having a diversity of approaches to forecast evaluation would be instructive. No claim is made here that this is the only or best approach (laying the groundwork here for identifying eventual winners and losers).

With all that as background, below then are the predictions in one table (click on it for a bigger view). The yellow cells indicate the teams that the naive baseline sees advancing to the knockout stages, and the green shows the same for each of the 10 predictions. The numbers show the team rankings according to each prediction.
I will be tracking the performance of the 10 predictions against the naive baseline as the tournament unfolds, scoring them in a league table. I'll also discuss the methods and results as well as the sensitivity of the latter to the former. When the Group Stages wind up I'll reset for a second part of the prediction evaluation.

Finally, for now, I welcome any comments on this exercise. If there are other predictions that you'd like to track in the same manner alongside these, please enter them in the comments.

Let the game within the games begin! 

Sunday, June 1, 2014

FIFA has Bigger Problems than Corruption Alone

With a big investigation on FIFA being reported by The Sunday Times today (£) I thought I'd repost the commentary below, which appeared earlier this month as a guest post at Soccernomics (Thanks Simon Kuper).

FIFA has Bigger Problems than Corruption Alone

For much of the past four years FIFA, the organization which oversees global football and the World Cup, has been dogged by allegations of corruption and poor management.  In 2011, after two members of the FIFA Executive Committee had tried to sell their votes for future World Cup location to undercover reporters[R1], FIFA felt compelled to respond by setting up an internal reform committee to recommend steps to improve the organization’s governance.

That committee, chaired by Professor Mark Pieth of the Basel Institute of Governance, ended its work last year, and issued a hard-hitting, final report last month.

Like many other international sports governance bodies, the Fédération Internationale de Football Association or International Federation of Association Football, is headquartered in Switzerland. Many international organizations have long found the nation to be a favorable location due to its political neutrality, generous tax treatment and hands-off approach to oversight.

For FIFA, the hands-off oversight has arguably allowed mischief to take root. Among the many scandals that have plagued FIFA, the one getting the most attention today is alleged vote-buying and collusion associated with the awarding of the 2018 World Cup to Russia and 2022 World Cup to Qatar. The decision on who gets to host the World Cup is a consequential one. For instance, Qatar has announced $200 billion in new infrastructure projects [R2] associated with the 2022 World Cup, including 8 new stadiums.

The decision as to which country gets to host the World Cup is made by FIFA’s inner circle, its 25-member Executive Committee. In 2010 FIFA held votes on the awarding of the 2018 and 2022 World Cups at the same meeting. Later, Sepp Blatter, FIFA’s long-time president, admitted[R3]  that there was vote trading across the two votes, and that it was a mistake to have made both decisions at the same time.

But vote trading was the least of FIFA’s troubles associated with the World Cup votes. In February it was revealed that the family[R4]  of a member of FIFA’s Executive Committee in 2010 received $2 million from a Qatari company controlled by a former football official in the months following the vote. On the heels of that revelation we learned that the 10-year old daughter of another member of the FIFA Executive Committee[R5]  has $3 million deposited into a bank account in her name. The funds allegedly came from the (at the time) president of FC Barcelona (home to Lionel Messi), which is sponsored by Qatari Airlines and the Qatar Foundation.

These allegations are cited in Pieth’s final report, which says that the future integrity of the organization depends a great deal on how it responds[R6] : “If FIFA is to emerge from the scandals of recent years it must now produce a convincing and transparent answer to any issues relating to hosting decisions… if allegations are confirmed FIFA must ensure that the consequences are meaningful.“

However, FIFA’s ability to address the World Cup vote scandal may be limited by the fact that most of the recommendations of Pieth’s committee have not been implemented. Further, Pieth observes in his report that “some Members of the Executive Committee have not been sufficiently committed to change.”

In a forthcoming paper I compared the actions taken by FIFA to three reports which made recommendations for steps FIFA should take to improve its governance, as well as to a “good governance” scorecard[R7]  developed by several academics in Switzerland for evaluating international sports organizations.

What I found is in agreement with Pieth’s conclusion that “reform is not yet finished.” In fact, it has a very long way to go. Of the 59 total recommendations in the three reports FIFA is judged to have implemented 7, partially implemented 10 and failed to implement 42. With respect to the “good governance” scorecard, FIFA’s reform effort lifted their score from 55.2% of total possible points to 56.3%. Under each of these metrics, FIFA did not implement the vast majority of recommendations offered by experts in good governance.

For its part, FIFA has hired an investigator, Michael Garcia, a former US federal prosecutor, to investigate. In his report, Pieth says that this process needs to go forward, regardless where it leads: “FIFA and all involved individuals must therefore fully and unconditionally cooperate with Mr. Garcia’s investigation.” However, Pieth also refers to recent media reports of a “plot” among some on FIFA’s Executive Committee[R8]  to remove Garcia as an investigator.

For FIFA, its main problem may not lie in the specific details of the 2010 vote, or even its other scandals, but rather in that it has not yet adopted a strong program of governance reform, in the words of Pieth, in order “to truly reach the standards accepted in the corporate world.”

The stakes involved go far beyond corrupt officials who may be trading votes for money and other favors. For example, Qatar’s 2022 World Cup has faced challenges[R9]  with respect to the timing of the event, which is scheduled to take place during the hottest part of the summer. More significantly, Qatar has faced claims that workers[R10]  who are building the infrastructure for the World Cup, are dying by the hundreds, due to unsafe conditions. Similarly, Russia’s incursion into Crimea has led to international sanctions. Two US Senators are among the voices calling for FIFA [R11] to revisit its 2018 hosting decision.

Whatever actions FIFA may or may not choose to take with respect to Qatar and Russia, one thing seems clear: FIFA’s incomplete reform effort stands in the way of a stronger organization, better able to handle challenges and controversies. FIFA’s ability to effectively govern itself goes well beyond issues of corruption.

* Roger Pielke, Jr. is a professor in the environmental studies program at the University of Colorado. He has been researching sports governance since 2011.
————————————————-
[R1] http://www.baselgovernance.org/fileadmin/FIFA/Final_Report_by_IGC_to_FIFA_ExCo_EN.pdf
[R2] http://gulfnews.com/sport/football/qatar-cuts-2022-world-cup-stadia-from-12-to-eight-1.1322179
[R3] http://news.bbc.co.uk/sport2/hi/football/9387891.stm
[R4] http://www.telegraph.co.uk/sport/football/world-cup/10704290/Qatar-World-Cup-2022-investigation-former-Fifa-vice-president-Jack-Warner-and-family-paid-millions.html
[R5] http://www.telegraph.co.uk/sport/football/world-cup/10775490/Daughter-of-World-Cup-official-had-2m-put-into-account.html
[R6] http://www.baselgovernance.org/fileadmin/FIFA/Final_Report_by_IGC_to_FIFA_ExCo_EN.pdf
[R7] http://www.idheap.ch/idheap.nsf/view/D6156F1EF87ACB07C1257B3900538D87/$File/IDHEAP%20Working%20Paper%201-2013.pdf
[R8] http://www.theguardian.com/football/2014/mar/25/fifa-ethics-investigator-michael-garcia-survives-plot-overthrow-him
[R9] http://www.bbc.com/sport/0/football/25653594
[R10] http://www.theguardian.com/world/2014/feb/18/qatar-world-cup-india-migrant-worker-deaths
[R11]http://www.kirk.senate.gov/?p=press_release&id=1018