For more than a decade now, public sector procurement experts have been developing and deploying analytical strategies with the ambition to get meaningful insights from mountains of data, to help decision-making processes and drive value. Big data in public procurement can provide effective policy decisions and implementation compared to past knowledge. For that we need high-quality datasets and easy-to-understand indicators. This seems to be the purpose of the Single Market Scoreboard (SMS) on public procurement, having created indicators that may favour policy options based on them.
Nevertheless, the reality shows that often data is not available, not comparable or simply not adequate. Academia has issued some criticism on the way the data is collected and the indicators valued, in view of the impact of the SMS for the policymakers. The tool is being evaluated at the time of writing this article, with the ambition to develop and upgrade the SMS. It currently covers only four policy areas, one of them being public procurement. The evaluation looks at the use of the scoreboard by the MSs and studies the option of covering other important areas not included so far such as the Digital Single Market (DSM) and consumer policy.
In this blog we briefly analyse the last edition of the European Commission scoreboard on procurement and compare it with previous editions. We look at the compliance of the tool with the ambition of the Commission, to measure the key elements influencing the individual country’s performance. Does it construct a useful tool to provide an overview of the functioning of the internal market on public procurement, while at the same time determining whether contracting authorities are achieving value for money in the respective country?
The comparison of the editions of the scoreboard on public procurement
The European Union’s SMS, first published by the European Commission in 1997, monitors Member States’ performance in implementing the four freedoms of the internal market. Besides the breakdown of information by country and governance instrument, the scoreboard also publishes results by policy area, including public procurement. Using a traffic-light colour system, the scoreboard rates EU Member States in 12 individual indicators [1]. The weight given to the indicators varies, depending on their relevance [2]. The sum of the indicators leads to a final overall performance rating for each country, shown in colours. The colour ratings are based on two factors: qualitative policy assessment of what is good practice, and actual data for individual countries.
By including public procurement in the Internal Market Scoreboard, the Commission states that the scoreboard is based on sufficiently reliable data obtained from the database of mandatory publications of contract notices, TED. With the procurement SMS, the Commission has been trying for years now to measure the key influences on public procurement in a way that is transparent and easy to understand for the public, and allows comparisons to be made with previous years.
Despite the carefully collected data, the Commission itself points out that some aspects of public procurement have been ‘omitted entirely or covered only indirectly, e.g. corruption, the administrative burden and professionalism’ (EU Commission). Further, the Commission recalls that ‘the indicators should be carefully interpreted, ideally in the light of additional quantitative and qualitative information’ (ibid).
Analysing the last five years of the public procurement scoreboard from 2016 to 2020, the number of overall red categorised countries decreased from eleven countries in 2016 to nine countries in 2020. For yellow labelled countries there was an increase of six in 2016 to 13 in 2020. While in 2016 eleven countries achieved an overall green evaluation, in 2020 only eight countries received this evaluation. However, since 2020 the number of evaluated countries decreased by one, as Great Britain is no longer part of the scoreboard, due to the market regulations after Brexit.
What stands out when analysing colour categorisation over the last five years is that colouring remains relatively constant, especially for red countries such as Spain, Italy, Romania and Greece. For the yellow category, Germany, Ireland and the Netherlands remained on this level of evaluation throughout the years, without significant improvements or deteriorations. The only consistently green countries are Norway and Finland, which received a green overall evaluation, as well as Estonia and Sweden, each with a one-year exception of getting a yellow evaluation. While some EU Member States are seeing consistent performance in public procurement, the Internal Market Scoreboard is not the ideal basis for trend analysis. The European Commission’s analysis of 2014 only included six indicators, expanding to nine indicators in 2015 and 12 indicators since 2017. A continuity of analytical tools, such as uniform indicators, compared to previous years would improve the accountability and comparability of the scoreboards and increase their transparency. Member States would thus be able to identify continued success or the need for improvement in certain areas, which is not so obvious with changing indicators. Therefore, conclusions on a trend of individual Member States should be taken with caution until further scoreboards with comparable indicators are published.
The critics and the revision of the scoreboard
Academics reprove that some of the thresholds set by the SMS to determine whether a country has performed well or poorly on a particular indicator are not adequately justified or supported by empirical evidence [3]. The European Commission recognises these limitations of the SMS analysis and therefore treats it with reservation. Indeed, the Commission acknowledges that some aspects of public procurement have been omitted entirely or covered only indirectly, e.g. corruption, the administrative burden and professionalism. So, although the scoreboard provides useful information, it gives only a partial view of EU countries’ public procurement performance. Furthermore, the results of the SMS could become more transparent so that all published results can be easily recalculated and thus verified by outsiders, based on a detailed explanation of the methodology by the Commission. These reviews by the countries would have a positive long-term impact on the countries’ performance in public procurement, as the Commission’s assessment is presented in a comprehensible way for them and is thus a motivation for improvement.
Decisions based on data and not on assumptions is key in public procurement. Therefore, the scores of the SMS, with all its admitted bias, needs to be used with caution and its impact downsized. Extending the SMS to new policy areas while the ones already covered are not yet fully serving their purpose is, in our view, not desirable right now.
Conclusion
The Single Market Scoreboard on public procurement uses performance measures to assess whether contracting authorities are getting good value for their money. In doing so, the countries of the EU Single Market are evaluated on 12 indicators to create a transparent measure that allows comparability with previous and future years of SMS analysis. However, the use of the data in its current form raises a number of issues related to bias, governance, accuracy and reliability of the published results. The EU Commission is currently reviewing the scoreboard’s further development and restructuring (see here). Against the background of the assessment, the restructuring process of the Commission must be critically observed and the changes in the future SMS must be analysed accordingly. Until then, the data and results of the published SMS should be taken with caution, in the spirit of the academic commentaries published [4] and the Commission’s self-explanation on the indicators and used sources of data.
The views expressed in this blog are those of the authors and not necessarily those of EIPA.
This blog is written by Gracia Vara Arribas and Anita Knöchelmann
