Numerous projects and initiatives to monitor regional integration have accumulated a great deal of organisational experience, both positive and negative. The lessons of this experience allow us to identify a number of issues that will have to be addressed in the course of the development of the monitoring system in order to make it successful, relevant and sustainable. These issues will also have to be addressed in the evaluation of integration processes. These issues are not only methodological and technical, but also political. Our conclusions are organised around five points and can be considered the underlying principles that the EDB team used when developing the SIEI and preparing this report, and which we will draw upon in the future.
1. Monitoring Regional Integration: Technical vs. Political Aspects
The monitoring of regional integration can not be reduced to a technical problem. The starting point for the creation of an indicator system is rather of a political nature. The purpose of monitoring is usually the evaluation of regional integration policies, given the “implementation problem” faced by several regional initiatives, and to test the quality of regional governance.
The political and technical aspects are clearly linked. Several examples of linkages can be mentioned:
• the number of aspects to be considered in an indicator system are a function of the underlying mandate;
• the inclusion of (inter-regional) comparison and benchmarking is a political choice;
• the choice of assigned weights in a monitoring system and in the design of composite indicators cannot be based only on technical (statistical) criteria;
• the choice to include good governance indicators (like transparency, participation and accountability) is also a political decision.
2. The Monitoring Process: By Whom? For Whom? To What End?
Monitoring is not an isolated (academic) activity. It refers to a series of relevant processes of information gathering, processing and dissemination with the aim to influence, scrutinise and/
or evaluate regional integration policies or to secure their implementation. These processes take place in a monitoring system in which different actors participate: regional and national;
public and private. These actors can be internal to the regional integration process (e.g. regional secretariats, regional parliaments, regional development banks) or external to the process (e.g.
academics, nongovernmental organisations (NGOs) (Costea et al., 2008). The role of monitoring can be political, coordinating, academic, technical, and/or financial, and so on. In other words, monitoring can take place close to or far removed from decision-making centres. Monitoring is thus not to be equated with evaluation, and displays both positive and normative aspects.
In some cases, the integration arrangements have built-in monitoring provisions. These are especially effective when, at the same time, the integration agreement itself includes explicit objectives.
The use of extra-regional benchmarks is a particularly sensitive issue and poses a series of methodological problems.
From an academic point of view, indicator-based monitoring is of particular value because it allows the testing of opinions and accepted opinions on the “progress”, “success” or “failure” of particular regional integration processes.
It should therefore not be forgotten that regionalism or regional integration is a “moving target”.
The institutional complexity of regional arrangements tends to increase over time. And shifts have been noticed from unidimensional regional organisations towards multidimensional and hybrid forms of regional cooperation (Hettne and Soderbaum, 2004). A good example of the latter, in the Eurasian context, is the Central Asia Regional Economic Cooperation (CAREC).
Finally, it should be noted that the monitoring actors are not necessarily (intra-) regional actors.
Extra-regional actors (like other regions, international organisations, international NGOs, academics) may also be interested in the monitoring process.
3. Monitoring Experiences: Where Do We Stand?
The lessons from the previously mentioned projects allow us to draw the following conclusions:
• only a few initiatives proved sustainable;
• the political role of monitoring does not seem to be crucial for the regional integration process;
• few actors are usually involved in monitoring;
• different logical components of regional integration are targeted (De Lombaerde and Van Langenhove, 200);
• monitoring in practice seems to have different objectives (including: measurement of the level of regional integration, measurement of preconditions, assessment of the contribution of individual countries to regional integration, evaluation of regional integration policies, comparison, evaluation of donor-financed support programmes, strategic use in the context of interregional negotiation processes);
• monitoring systems are often characterised by underdeveloped conceptual frameworks and poor selection criteria for the indicators (De Lombaerde, Pietrangeli and Weeratunge, 2008);
• the size of the indicator systems varies considerably. A recent review of several systems revealed that indicator systems cover anywhere between less than ten and close to 10 variables (ibidem);
• one third of the included variables do not necessarily provide information about regional integration processes.
Apart from the observations that can be derived from the indicator-based systems, some additional observations can be derived from monitoring experiences more generally;
• the increasing complexity of regional integration makes monitoring more challenging;
• the increasing complexity of regional integration seems to go hand-in-hand with increasingly complex monitoring systems. In the case of EU, for example, the monitoring system consists of a whole array of interconnected processes, both at the level of internal
level (member states and other actors) monitors the regional level, when the regional integration process deepens, regional bodies start to monitor the implementation of regional commitments by member states (De Lombaerde, Estevadeordal and Suominen, 2008);
• monitoring covers the full project cycle. It is not limited to post evaluation; it covers the whole cycle, from the policy preparation phase onwards;
• built-in monitoring agendas are perceived as being functional in nature;
• the role and quality of national institutions is crucial for (good) monitoring, especially in the case of young and shallow forms of regional integration (De Lombaerde, Estevadeordal and Suominen, 2008).
The critical stage is the translation of selected variables into indicators. Although each variable carries specific characteristics, the choice of suitable indicators requires certain general criteria.
Anderson (1991) proposes the following criteria (see Table 2.4):
• An indicator or its underlying information must be readily available and inexpensive.
• An indicator must be easy to understand.
• An indicator must be measurable.
• An indicator must characterise something important in itself, or reflect something more than the subject it measures (e.g., life expectancy data can be used to characterise the general health of the population).
• A short time gap between the described condition/situation and the emergence of the indicator is desirable.
• An indicator must be based on information that can be used to compare different geographic areas, social groups, etc., so as to provide a description of distribution, not total figures or mean values.
• The ability to form international comparisons is desirable.
4. Technical Aspects
The actual design of an indicator-based monitoring system is based on three pillars: the conceptual framework, data and methods.
The conceptual framework should guide the selection of variables and indicators. It can be based on one of the theories from the arsenal of theories available for the purpose, or on a combination of these. One should be aware of the fact that many of the available theoretical frameworks are rooted in European experience, so that transferability should be evaluated. Also, the teleological logic of frameworks like Balassa’s (191) should be critically assessed. Indicator systems should be sufficiently flexible to allow for region-specific variables. When there is an underlying understanding of the/a logic of the integration process, variables and indicators can be classified in categories (institutionalised integration versus “real” integration, positive versus negative integration, by sectors, etc.) (De Lombaerde and Van Langenhove, 200).
For an overview, see for example, Mattli (1999), Rosamond (2000), Wiener and Diez (2003), Laursen (2003), Söderbaum and Shaw (2003), Farrell et al. (200), and Malamud and Schmitter (2007).
In indicator systems set-up to monitor regional integration, the indicators are supposed to reflect an aspect of the process. However, as experience shows us (see above), in practice this is not always the case. At the same time it is true that there will always be a grey area between those variables that should be ‘in’ and those that should be ‘out’. Sometimes a simple transformation of variables can turn irrelevant variables into relevant variables. For example, inflation or growth rates that tell us something about the national economies of the member states can be transformed into (regional) convergence indicators.
Other issues raise themselves during the design of indicator systems. For example, systems can be designed at the regional and/or national level of analysis. An example of the latter is the system proposed by UN ESCWA to assess the participation of each national economy in the regionalisation process in the Middle East (UN ESCWA, 2007). Another issue is related to overlapping memberships and poses serious problems to the design of indicator systems. Yet another issue is related to the question of whether consolidated indices should be constructed.
These consolidated indices might well capture the multi-dimensional nature of the processes and they are easy to read and communicate. However, their interpretation might become rather abstract and the weighting of the different components of the index will always be arbitrary (De Lombaerde, Dorrucci et al., 2008).
When monitoring is based on quantitative data or on a combination of quantitative and qualitative information, one is faced with the problem of data availability. This problem is even more serious at regional level than at national levels. For many variables that are not mere aggregations of national variables (such as the intra-regional flows of people, services and capital, or data on regional budgets and policies), systematic data is often still lacking, even in regions that tend to have relatively good quality data.
5. Comparison and Comparability
The fifth and final point refers to the issues of comparison and comparability. Different contexts, different regional realities and different regional architectures exist. These differences, such as the differences with European institutional architecture, are often confused with differences in effectiveness, but should not be. Comparison should be sensitive to these differences, without adopting the opposite extreme position that specific contexts imply that different processes are incomparable. Different levels of regional interaction and interdependence, and other aspects of regionalisation can be compared.
Comparison can be based on traditional comparative indicators or on relative comparative indicators. The latter compare regional performance first with the region-specific objectives or benchmarks, and then, in a second instance, across regions. Combinations of both approaches are obviously also possible, as the indicator system proposed by UNECA has illustrated (UNECA, 2002). Finally, as UNECA experience also shows, comparison is still a politically sensitive issue at the inter-regional level, although accepted practice at an inter-national level. Before designing an indicator system with a comparative dimension, it is preferable that it is discussed with major stakeholders.
3.1. Composition of SIEI
The EDB’s SIEI consists of three sets of indicators which correspond to the three main aspects of regional cooperation:
(a) analysis of regional integration as integration of markets;
(b) analysis of regional integration as convergence of economic systems;
(c) analysis of regional cooperation.
Due to the non-uniform nature of the data contained in each set, we apply different approaches towards the quantitative assessment of integration and cooperation in the post-Soviet space. This exercise exclusively covers the evaluation of regional integration in the CIS and does not pretend to be suitable for comparative analysis of integration processes elsewhere in the world; therefore, the choice of indicators was determined solely by the availability of data on post-Soviet economies and the importance attached to particular areas of economic cooperation and modernisation of CIS countries.
For the analysis of regional integration, three types of indicators are calculated for: (i) the integration of country pairs; (ii) the integration of a country with a group of countries; and (iii) the integration within a group of countries. Each of these indicators needs to be interpreted separately.
Integration of country pairs means convergence of two particular post-Soviet countries. In other words, for the purposes of the SIEI, we considered all the possible pairs of the twelve CIS countries7 in order to assess the degree to which they are interdependent. These assessments are “symmetric”, i.e. a “high level of integration” means a situation where both countries strongly depend on each other. Accordingly, asymmetric dependence (e.g., one of the two countries is a critical trade partner for the other, but not vice versa) results in a lower level of integration.
Integration of a country and a group of countries characterises the convergence of any of the twelve post-Soviet states and any of the five large regions within the post-Soviet region; these regions may be of particular interest from the point of view of practical integration activity and include several countries each. The history of implementing regional projects in the post-Soviet space (positive or less positive) allows us to define five regions:
1. CIS-12 (all post-Soviet countries);