Monitoring Compliance with St. Petersburg Summit Commitments

13 may 2007

Marina Larionova

Resume: Over 32 years of its history, the G7/G8 has expanded both its agenda and institutional system, and is now appreciated as an instrument of deliberation, direction-giving and decision-making on global governance issues. It has also become a subject for criticism and reform proposals. The critique mainly focuses on the forum’s representativeness, legitimacy and effectiveness.

There may be three points of departure for reflection on the G8 commitments compliance.

First, when the forum arose in the mid-1970s to respond in a coordinated way to the problems and challenges that the existing international institutions could not cope with, its architects set a very high level of expectations on the meetings’ outcome: they should treat crucial economic, financial and political issues, and they should yield results.

Second, St. Petersburg produced 14 summit documents plus the Chair’s summary totaling 317 specific commitments. Although it has confirmed the tendency for increasing the number of commitments characteristic of the seventh series, this is the highest number of any summit held since 1975. Of these, 216 commitments reflect decisions on the Presidency priority issues: 52 relate to fight against infectious diseases; 114 to global energy security; and 50 to education for innovative society in the 21st century. However impressive this may seem, as Russian Foreign Minister Sergei Lavrov said, “the viability of the decisions hinges on the members’ commitment to their consistent implementation within the systemic strategy of joint actions. Serious and multifaceted work on the St. Petersburg commitments implementation lies ahead, including the period of the German presidency of the G8.” Thus, a weighted assessment of the summit performance and the leaders’ commitment to the decisions made is still to come, inter alia on the basis of compliance study results.

Third, over 32 years of its history, the G7/G8 has expanded both its agenda and institutional system, and is now appreciated as an instrument of deliberation, direction-giving and decision-making on global governance issues. It has also become a subject for criticism and reform proposals. The reform proposals are well known and range from expanding the institution to G10 and G12, restructuring the G20 into L20, restructuring the G8 into G4, abolishing the G8, etc. The critique mainly focuses on the forum’s representativeness, legitimacy and effectiveness.

While it is difficult to argue against proposals to expand the G8 to include China and India, or the rationale for coexistence of the G8 and the L20, it is worthwhile considering what data and instruments of evaluation are available to support, inform or refute the perception of the G8’s shortcomings. It is also useful to analyze what these tools offer for monitoring, comparing and sharing, but, moreover, for communicating the G8 performance results to the wider public.


Scholarly analysis of summit results which has developed over the years includes three different methods of evaluation.

Assessing summit performance. Robert D. Putnam and Nicholas Bayne assess the summits’ achievements on six criteria: leadership, effectiveness, solidarity, durability, acceptability, and consistency.  The assessment is done using a grading system from A to E. According to Bayne, the first summit series (1975-1978) is considered to have been the most productive so far. The first G8 sequence, which coincides with the sixth series summits (1998-2001), has shown consistent B and B+ performance. The seventh series (2002-) is very diverse in achievements, ranging from C+ for Evian (2003) and Sea Island (2004) to A- for Gleneagles.

Assessing behavior of the country holding the G8 presidency.  A ‘scorecard’ approach was developed by the Foreign Policy Center in London (Hugh Barnes and James Owen), which issued the first annual ‘scorecard’ on Russia in 2006. The system aims at monitoring the behavior of the country holding the G8 Presidency on key features relevant for membership in the G8. They include 12 indicators: openness and freedom of speech; political governance; rule of law; civil society; economic weight in the world; inflation; economic stability and solvency; unemployment; volume of trade; protectionism; energy market conditions; and stance on key international issues.

The measure of a country’s compliance with G8 norms is assessed on a five-point scale: (1) broad compliance; (2) moderate compliance; (3) sporadic compliance; (4) lack of compliance; and (5) total failure to comply.  The data for analysis is drawn from the IMF, the WB, national official statistics, the WHO, various other international organizations and think tanks.

Russia’s score according to this first exercise has been far from impressive. On open society the score is (5); on political governance, (4); on the rule of law, (4); on civil society, (4); on economic growth and stability, (3); on inflation, (3); on stable exchange rate and market conditions, (3); on unemployment level, (4); on trade volume, (3); on trade restrictions (protectionism, etc.), (4); on energy market conditions and policies, (4); and on discernable stance on key international issues, (4).

Assessing compliance with the summit commitments. However important to understanding of G8 effectiveness the summits’ performance evaluation or the member states’ compliance to democracy and economic growth – the key characteristics of monitoring – are, it would not be complete without a consistent and quantifiable assessment of the G8 member states’ compliance with the summits commitments.

This assessment has been carried out by the G8 Research Group of the University of Toronto under the leadership of Professor John Kirton and Doctor Ella Kokotsis since the 1996 Lyon summit, and has continued on an annual basis until now.

On February 20, 2007, the G8 St. Petersburg Interim Compliance Report was released by the G8 Research Group of the University of Toronto and the State University–Higher School of Economics G8 Research Group (HSE). The findings for the St. Petersburg summit demonstrate a positive average degree of G8 member states’ compliance performance (33%) and, hence, testify their commitment to a wide range of decisions made at the summit. These findings thereby confirm earlier assessments of the G8 2006 meeting as a successful one.

However, before highlighting results of a new cycle of the study launched this autumn on the St. Petersburg summit commitments and its interim results, it would be useful to remind of the most essential dimensions of the study and some of the key methodology approaches.

First, it should be noted that the main objective of the study is not a cross-country comparison of the member states’ performance on the summit commitments, even though this is probably its most visible and striking result.

More importantly, reflecting on the basis of empirical findings on the factors of “high and low compliance”  the study aims to explore how credible and effective the institution is, namely:

1. To what extent and under what conditions does the G8 live up to the commitments and decisions reached at the summit table?
2. How does the pattern of summit compliance vary by issue area and over time?
3. What factors can enhance or diminish the commitments compliance performance of the member states?

In the course of the study, some of the factors enhancing compliance were identified: the leaders’ personal involvement; the strength of their domestic positions; the presence of domestic institutional structures and an increased number of various-level working and official bodies; the use of existing regimes (such as the IMF and the World Bank) where the G8 are major shareholders and are able to exert their political and financial influence, set the agendas, and secure agreements on the implementation;”  the link of the commitment made with the domestic priorities of the member states; and the degree of consensus on the commitments and the mechanisms of their implementation.
The methodology toolkit includes:

  • the definition of the concept of compliance;
  • the definition of the concept of compliance performance;
  • the methodology of selecting commitments for monitoring;
  • the methodology of assessing the degree of compliance with the commitments.

According to the methodology, commitment is a “distinct, specific, collectively agreed and publicly expressed statements of intent, promise or undertaking by leaders that they will take future action to meet or adjust to an identified target.”
In order to qualify, commitments must satisfy several criteria:

  • Commitments must be distinct, meaning that each goal should represent a separate commitment;
  • Commitments must be specific, identifiable, measurable and contain clear parameters;
  • Commitments must be future-oriented rather than present endorsements of previous actions, that is, they need to represent a pattern for future action.

Verbal instructions to international institutions, issued at the time of the summit, are included as it is assumed that summit members will take action to move toward attaining this result.
Compliance is a conscious new or altered effort by national governments in the post-summit period aimed to implement the provisions contained in summit communiqu?s. In the work of the G8 Research Group, compliance occurs when national governments change their own behavior to fulfill a summit goal or commitment. Leaders legitimize their commitments by either:

  • including them within their national agendas;
  • referring to them in public speeches or press releases;
  • assigning personnel to negotiate the mandates;
  • forming task forces or working groups;
  • launching new diplomatic initiatives; or
  • allocating budgetary resources toward the commitment’s fulfillment.

The measure of compliance is assessed on a three level scale:
1. Full or nearly full compliance with a commitment is assigned a score of +1.
2. Complete or nearly complete failure to implement a commitment is indicated by a score of –1.
3.  An “inability to commit” or “work in progress” is given a score of 0. An “inability to commit” refers to factors outside the executive branch that impede implementation. “Work in progress” refers to an initiative that has been launched by a government but has not yet been completed by the time of the next summit, and whose results therefore cannot be effectively judged.
As only a fraction (not more than 10 percent of commitments made) is selected for monitoring compliance, criteria of selection are relevant for validity of the study results.

Primary selection criteria include:

  • Importance for the summit, the G8 and the world. It was agreed that at least two commitments of each of the priority themes for the summit should be included.
  • Comprehensiveness; the set needs to embrace the economic, global and political-security domains and incorporate at least one from each part of the traditional agenda, i.e., finance, macroeconomics, microeconomics, trade, development, environment/climate change, energy, crime and drugs, terrorism, arms control and proliferation, regional security, and international institutions reform.
  • Balance by document; geographic distribution affecting the G8 members, non-G8 members and the world as a whole; contentiousness in the preparatory process; continuity from previous summits; proportionality among analysis dimensions that are most relevant for current scientific research, such as timetable, international organization, money mobilized, G8 bodies, target, remit mandates, propriety placement, specified agency, etc.

Secondary selection criteria are of practical methodological character. Selected commitments should allow individual and collective compliance monitoring; be feasible to commit fully within the year as the compliance framework is annual; allow monitoring on the basis of sufficient and reliable information; and allow for easy construction of interpretive guidelines.
Tertiary selection criteria include significance to the summit as identified by experts in the host country.


For most part, of the 20 priority commitments selected for the G8 2006 compliance monitoring and assessment, the Russian and Canadian research teams got consistent results. However, for several commitments the teams have not been able to find concerted scores. With regard to Russia, inconsistencies relate to the final scores on three commitments: Renewable Energy, Africa – Security, and Global Partnership – Non-Proliferation. Nevertheless, the average compliance score for Russia is 25 percent according to the officially released version of the St. Petersburg Interim Compliance Report, which reports the assessment drawn by the HSE Team.











For Germany, the discrepancies between the two research teams persist on the final scores for five commitments: Health – Polio Eradication, Education – Qualification Systems and Gender Disparities, Africa – Debt Relief, and Global Partnership – Non-Proliferation. Hence, the average compliance score for Germany team is 45percent according to the officially released version of the St. Petersburg Interim Compliance Report, which reflects assessment of the University of Toronto G8RG, whereas the HSE team score for Germany was 20 percent (Table 1).
Discrepancies between the scores ascribed by the two G8 research teams mostly occur due to:

  • Varying degree of comprehensiveness of the data used;
  • Differences in understanding of the commitment content and interpreting of the data collected;
  • Inconsistencies of interpretation of the commitments in cross-country comparisons.

Two examples will give the readers a taste of the debate.

The St. Petersburg Statement on Non-Proliferation reinforces the commitment made in Kananaskis: “We remain committed to our pledges in Kananaskis to raise up to $20 billion through 2012 for the Global Partnership, initially in Russia, to support projects to address priority areas identified in Kananaskis and to continue to turn these pledges into concrete actions.”
The financial commitments of the G8 member states to the Global Partnership (not including local or associated costs) are as follows: 

Canada                    $743 million
France                     $909 million
Germany                  $1.5 billion
Italy                         $1.21 billion
Japan                       $200 million
Russia                      $2 billion
United Kingdom       $750 million
United States            $10 billion
European Union        $1.21 billion
Non-G8 states          $1.5 billion

Thus, of the total 20 billion USD to be raised over the decade, Russia is to allocate 2 billion.
Assuming the study formula of an equal distribution of funds over the years, Russia is ahead of its obligations, having allocated $1.3 billion out of committed $2 billion (Table 2).

Thus, this country has registered a high level of compliance, meriting +1 in the opinion of the HSE analysts. However, given the methodology requirement that the monitoring relate to the period from one summit to another and the fact that the data available includes the period until June 2006 and there is no evidence that Russia has contributed anything since the St. Petersburg summit, the G8RG of the UoT analysts hold the view that Russia’s compliance score should be 0 for the interim report.

Another example of discrepancy stemming from differences in understanding the content of the commitment and interpreting the data relates to the assessment of Germany’s compliance performance on the G8 St. Petersburg commitment on Education – Qualifications (a commitment to “share information on qualification systems in our countries to increase understanding of national academic practices and traditions.”

The G8RG of the UoT analysts registered full compliance by the German government with this commitment, as indeed Germany has been involved in numerous activities aiming at enhancing transparency and compatibility of qualifications. However, a caveat is due that these activities are part of a different agenda and long-term obligations of Germany as member of the EU and Bologna process, and namely, the European Commission recommendation for the establishment of the European Qualifications Framework (EQF) for lifelong learning, the SOCRATES and LEONARDO exchange programs, as well as Bologna process seminars and research. Thus, the actions represent the country’s compliance with the commitments made within the EU; they have not been launched in response to the St. Petersburg commitment and in fact cannot be considered as compliance performance within the G8 setting. If they are accepted as such, given that Italy, France and the UK are active proponents of the same initiatives, their respective scores (0, –1, and 0) question the consistency of assessment across countries (Table 3).

Table 3.  2006 G8 Compliance scores for Russia and Germany*

Another contentious issue, which needs additional consideration, concerns the case of monitoring activities implemented by the EU G8 member states within the EU programs. If these are regarded as compliance of the EU-25, how valid would be reference to the same actions of each of the four EU G8 member states’ compliance? And a still more difficult question is: How in this case can one differentiate and evaluate individual contribution of these states toward compliance? These questions show the degree of complexity and challenge faced by the researchers undertaking the monitoring.

However, despite all the above discrepancies, the monitoring of commitments compliance performance remains to be a useful tool for assessing and enhancing the effectiveness of the G8 as a global governance institution. It is also extremely useful in evaluating commitment of individual member states to dealing with diverse global issues that demand collective management. Two factors are essential here: ensuring validity, reliability and transparency of the monitoring methodology, on the other hand, and preparedness of the member states’ institutions to use the results of these findings in their work.

To enhance the reliability and validity of the monitoring, the G8 Research Group of the University of Toronto and the State University–Higher School of Economics G8 Research Group  adhere to a combination of several principles.

First, to ensure consistency and integrity of the data analysis, it is necessary to elaborate and agree upon interpretation guidelines which would take account of the commitments’ content.

Second, it is crucial to ensure consistency of assessment across issues and across countries. This can be achieved through interaction in collecting and assessing data on the same commitment for different member states. This procedure puts extra pressure on the team leaders, but seems to be justified by the need for cross-country consistency. This problem has proven to be a challenge so far.

Third, the quality of the expert background materials on the content of the issues monitored is essential to build common understanding of the individual commitments’ specific nature among the analysts. This demand puts extra pressure on the budget of the study.  However, again this would be justified by the need for across issue data coherence and interpretation consistency.

Forth, it is essential that the data should be comprehensive and exhaustive, as these features have a considerable influence on the analysis results.

Finally, to provide for utmost integration of the various data on the commitments compliance it is vital to get a full and timely feedback from the G8 member states (this is far from a smooth and easy process, given the various pressures experienced by the structures involved in the G8 process). The most efficient way to ensure profound data consideration would be through consultations with national expert structures.

The analytical team of the HSE International Organizations Research Institute team and the G8 Research Group of the University of Toronto will continue close cooperation on the G8 compliance monitoring and assessment and persevere in enhancing its validity. Given the early date of the next summit,  the final report will be released by the end of May 2007. (The Heiligendamm summit is scheduled for June 6-8, 2007, indicating as priority issues global economic imbalances, energy and raw materials, world trade, poverty, development assistance, Africa and the Middle East.) Hopefully, the partnership between the two universities’ research groups will contribute to the improving quality of the analysis and assessment, and will help get trustworthy and reliable information on the G8 member states’ commitments to the St. Petersburg summit decisions.

Last updated 13 may 2007, 13:15

} Page 1 of 5