email icon Email this citation


Differences in Cognitive Complexity Levels among Negotiators and Crisis Outcomes *

Tara E. Santmire, Dept. of Government and Politics, University of Maryland;
Jonathan Wilkenfeld, Dept. of Government and Politics, University of Maryland ** ;
Sarit Kraus, Dept. of Mathematics and Computer Science,
Bar Ilan University, Institute for Advanced Computer Studies, University of Maryland;
Kim M. Holley, Dept. of Government and Politics, University of Maryland;
Toni E. Santmire, Dept. of Educational Psychology, University of Nebraska-Lincoln;
Kristian Gleditsch, Dept. of Political Science, University of Colorado

International Studies Association
Minneapolis MN
March 18-21 1998

Abstract

This paper reports on experiments designed to assess the impact of grouping decision makers by level of cognitive complexity on the outcomes which they attain in crisis negotiations. It attempts to better understand the dynamics which lead certain types of groupings to have greater success in negotiations, and which lead certain groups of adversaries to achieve more mutually beneficial outcomes such as compromise and agreement. Among the unique aspects of these experiments is their focus on crisis decision making, their use of a computer decision support system, and a controlled network environment for communications. The findings point to a positive relationship between the level of homogeneity in cognitive complexity among decision makers and the achievement of positive outcomes in crisis negotiations.

Introduction

This paper reports on a series of experiments designed to assess the impact of the group charcteristic of relative homogeneity of cognitive complexity within the group on their behavior in crisis negotiation situations and on the outcomes which they attain. These experiments are used to study the relationship between cognitive complexity and negotiating behavior, in an effort to better understand the dynamics which lead certain groups to have greater success in negotiations, and which lead certain groups of adversaries to achieve more mutually beneficial outcomes.

Previous work with the decision support system software used in these experiments has examined at how such software can aid in decision making in complex negotiations (Wilkenfeld et al., 1995a and Wilkenfeld, Kraus, and Holley, forthcoming). In addition, previous research has also looked at the relationships between individual levels of cognitive complexity and negotiation behavior and between this behavior and outcomes achieved (Wilkenfeld et al., 1996).

Underlying the present set of experiments are two interrelated expectations. The first is that similarity in cognitive complexity among decision makers (homogeneity) will facilitate the understanding of each others offers and counter offers, and hence such groups will be likely to reach mutually satisfying outcomes. Conversely, groups of decision makers who are cognitively diverse (heterogeneity) will experience difficulty in understanding one another"s positions and hence will have a lower probability of reaching mutually satisfying outcomes. The second expectation is that decision makers with virtually identical levels of cognitive complexity (ultra homogeneity) may experience frustrations in the negotiation process which are likely to lead to a diminution in their ability to achieve mutually satisfying outcomes. These expectations build on previous research results which are summarized later in this paper. Both diverse groups of decision makers and identical groups of decision makers are expected to reach mutually beneficial outcomes - in this case agreements - with lower frequency than groups of moderately dissimilar decision makers.

As the list of authors implies, we have assembled an interdisciplinary group of scholars who cover a wide range of approaches to the study of behavior in negotiation. These areas of expertise range from negotiation and decision theory in social science in general and political science in particular, cognitive schema theory within psychology, and intelligent systems and distributed artificial intelligence in computer science. The overall objective of this team is to develop a better understanding and theoretical model of the dynamics of negotiation, so that we will be in a better position to elaborate the conditions under which agreement and compromise are more likely to emerge than disagreement, conflict, and ultimately, violence. By adopting an experimental approach, we hope to better identify complex interpersonal factors using controlled environments where we will have some confidence in the influence of these factors on negotiation behavior and outcomes.

While this paper will not attempt a comprehensive literature review in any of the fields on which we draw for the design of our experiments, we have been aided in our review of relevant works by several bibliographic articles, including Thompson (1990) on negotiation behavior, Druckman (1994) and Kagel and Roth (1995) on experimental work on negotiation, Holsti (1989) on crisis decision making, Kraus and Wilkenfeld (1993) on strategic models of negotiation, and Ghiaseddin (1987) on the design of effective decision support systems.

The experiments reported on in this paper were characterized by several unique features which tend to set them apart from other experimental work being conducted in foreign policy analysis today (see, for example, Tetlock and Belkin, 1996, Mintz et al., 1997, Geva and Mayhar, 1997). First, the focus is exclusively on behavior in crisis, with a particular emphasis on shortness of time and high level of threat. Second, the subjects have access to an elaborate decision support system which, when properly used, can provide information on the utilities associated with various outcomes and hence enhance their ability to maximize utility, even under conditions of incomplete information. Third, all communications among the subjects in an experiment take place in a controlled network environment, allowing for complete analysis of the types of tools consulted by the subjects, as well as the content and form of the interactions during the negotiation. These features have combined to provide a rich environment in which experiments can be run and their results assessed. Only a portion of the results from the current rounds of experiments will be reported on in this paper.

Crisis Decision Making

It is widely contended that negotiations during crisis differ substantially from more routine negotiations, in that such situations are characterized by decision maker perceptions of high threat to basic values, short time for response, and a heightened probability of involvement in military hostilities (Brecher and Wilkenfeld, 1988, 1997). Under such circumstances, decision makers are often unable to access the information necessary to make utility maximizing decisions, or they are unable to properly evaluate the information in the amount of time available for decisions. Hence, decisions are often made on the basis of previous experience, long-held beliefs, and analogies to seemingly comparable situations, rather than with cold analytic calculation. As a consequence, sub-optimal outcomes are likely to result. While all decision making environments possess the potential for sub-optimality, international crises are particularly dangerous because they can quickly escalate to violence and war. The vast literature of crisis decision making has shown that situations of intense crisis can create a reduced span of attention, cognitive rigidity, and a distorted perspective of time (see Holsti, 1989 and Brecher, 1993 for excellent reviews of the crisis decision making literature).

Theoretical approaches to crisis decision making can be roughly divided into two groups based on the types of explanatory variables employed by the researchers. "The effects of crises may depend on many factors: individual difference variables [italics added] (self-image as effective coper, track record of performance in previous crises) and situational variables [italics added] (the reversibility and severity of existing threats).” (Tetlock & McGuire, 1986). One group of scholars, best exemplified by the work of Margaret Hermann, looks at the personal characteristics of decision makers to explain crisis outcomes. A second group of scholars, exemplified by Daniel Druckman, looks at the situational characteristics of the crisis to explain crisis outcomes.

Hermann believes that one should develop models of decision making that include national attributes, regime factors, decision structures and processes, situational variables, and external relationships (Hermann, 1980). She also concludes that "personal characteristics and orientations to foreign affairs of political leaders are worth including in this integrative effort.” (Hermann, 1980). More recently, Hermann and Kegley (1995) have published research on the effects of the personal characteristics of leaders in crisis situations on the likelihood of war. In addition, Suedfeld and Tetlock (1977) conclude that "information processing complexity, both as a personality characteristic and as an interactive function of personality and environmental variables, is a source of varied and powerful hypotheses and can be investigated by a quantifiable and methodologically rigorous procedure.” Suedfeld has been involved in several research endeavors in the area of crisis, including research on changes in cognitive complexity prior to surprise attacks (Suedfeld and Bluck, 1988), the role of change in conceptual complexity for success in revolutionary leaders (Suedfeld and Rank, 1976), and the role of cognitive complexity in communication during international crises (Suedfeld and Tetlock, 1977). In the work presented here, we attempt to develop the role of personal characteristics in a model of crisis decision making while holding constant the situational variables that also affect crisis decision making. The individual difference variable or personal characteristic variable that we focus on is the level of cognitive complexity exhibited by each individual.

Scholars who study the situational characteristics of crises believe that the environment in which negotiations take place affects the way negotiators behave and thus affects the outcomes of the negotiations. Such environmental variables include the physical location of the negotiations, the context of the negotiation, the structure of the negotiation, the issues involved in the negotiation, the structure of the negotiation teams, the immediate situation, parties to the negotiation, external events, publicity associated with the negotiations, and the structure of the domestic constituency that must agree to the outcome of the negotiations. These variables are thought to affect the flexibility of negotiators, the amount, timing, and kind of concessions negotiators are willing to make, and the negotiating strategies that they employ. See, for example, Druckman (1995). In our experiments, situational variables were kept fixed.

Decision Support Systems

As noted earlier, a key component of the current research program involves an assessment of the impact of the use of sophisticated software tools on the ability of decision makers to make effective decisions. Decision support systems (DSSs) can play a critical role in the crisis decision making process by allowing the decision maker to navigate large amounts of information quickly and to explore interrelationships between factors which may influence the decision. A DSS can also facilitate the simultaneous evaluation of multiple positions in crisis negotiations, thereby allowing the parties to rapidly formulate dynamic strategies and quickly evaluate their opponents' proposals (Wilkenfeld et al., 1995a). Decision support systems have been designed to provide support for the entire negotiation group (Grey, Vogel and Beaulair, 1990, Sycara, 1993), support for a mediator in a negotiation (Jarke, Jelassi and Shakun, 1987, Kersten, 1985), and support for individual negotiators (Bui, 1992, Matwin et al., 1989, Samarasan, 1993). In previous work, our interest has been in how the systematic use of such tools in high pressure decision making contexts can facilitate utility maximization for an individual negotiator (see Wilkenfeld et al., 1995a for a discussion of these results). Use of the DSS allows us to collect data on exactly what type of information each individual chooses to access, and what tools each individual uses to access that information. We also focus on the types of communications exchanged by participants in crisis negotiation. The use of language generators and network-based communications in controlled experimental environments allow us to carefully monitor negotiations as they transpire, and to collect content-analytic data during the negotiation. This unique environment, to be described below (see the GENIE Decision Support System), allows us to test a variety of propositions related to the types of communications exchanged during crisis, and to relate these back to the types of outcomes achieved.

Research Design

Our overall research program has several objectives. One is to identify the types of individuals most likely to achieve utility maximization in crisis negotiation situations. The second is to identify the pairings or groupings of types of individuals most likely to achieve utility maximization in crisis negotiation situations. The third is to identify the types of strategies which are most successfully employed and which individual types in which groupings and pairings employ them. The fourth is to identify the types of tools which are most likely to facilitate such utility maximizing outcomes for which types of individuals. The fifth is to identify the circumstances under which decision support system software can assist decision making in difficult negotiations and how this varies across types of individuals. This paper will report on results for the second objective only, while other papers report on results associated with the other objectives (see Summary of Previous Results below). The current goals of this research program apply only to crisis decision making; however, it is possible to extend them in the future to compare crisis and non-crisis decision making using the same software and approach.

As discussed above, the underlying premise of this research project is that grouping decision makers by level of cognitive complexity impacts on the process of crisis decision making and on crisis outcomes. For purposes of exploring this proposition, we designed experiments which would allow us to measure the level of cognitive complexity for each subject and then to manipulate the variation in cognitive complexity in each group and observe the impact of that manipulation on the outcome of the crisis negotiation situation. In the sections below, we will discuss the four instruments used to conduct these experiments: the Paragraph Completion Measure of cognitive complexity, the Hostage Crisis Simulation Model, the GENIE decision support system, and the language editor. This discussion will be followed by the presentation of the hypotheses tested, the experimental procedures used, and the results of a series of experiments.

The Paragraph Completion Measure (PCM) and Conceptual Level (CL) Score

The Paragraph Completion Measure (PCM) was designed to assess the level of complexity in cognitive structures through which the individual relates to the social world. Based on the theoretical work of Harvey, Hunt, and Schroder (1961) and Schroder, Driver, & Streufert (1967), the instrument usually consists of a set of sentence stems in response to which the individual is asked to generate a short paragraph. The paragraph is then scored according to the structural properties which appear to be required to generate the particular response. Thus, the assessment is of the structural properties of the response rather than its particular content (Schroder, Driver, & Streufert, 1967; Hunt, Butler, Noy & Rosser, 1978).

In this framework, individuals with low levels of cognitive complexity are termed concrete and individuals with high levels of cognitive complexity are termed abstract.

"Concrete" is defined as being equal to minimal differentiation; "abstract" is defined as equal to maximal differentiation and integration. Greater concreteness (as opposed to abstractness) implies (Harvey & Schroder, 1963): (a) fewer differentiations, incomplete integration; (b) a tendency toward bifurcated evaluation (c) dependence on external criteria of validity, e.g., authority, precedent; (d) greater intolerance for ambiguity, e.g. quick judgments in novel situations, susceptibility to salient (and potentially false) cues; (e) inability to change set, stereotyping in attempted solutions of complex problems; (f) greater resistance to change when stress is low, greater likelihood of collapse when stress is high; (g) poor delineation between means and ends, hence a paucity of different routes toward the same goal; (h) poorer capacity to act "as if", empathize, simulate a hypothetical situation; (l) less well-defined self, consequently, less perception of self as a causal agent (external vs. internal control).” (Streufert & Streufert, 1978)

Theoretically, in any situation in which there is conflict between different perspectives on a given issue, the perspectives represent polar opposites on some conceptual dimension. Structural analysis of the relationship between these perspectives in an individual response is used to determine the degree of cognitive complexity of the individual generating the response.

At low levels of cognitive complexity, each position is seen as separate and opposite. Generally, at this level, one position is evaluated as correct or right and the other as wrong. Thus, the two positions are seen as irreconcilable or inconsistent with each other. This is a score one level of cognitive complexity.

As individuals become more cognitively complex, the position of others can increasingly be seen as valid although no way of resolving the contradiction is seen as possible, and the individual pursues his or her own solutions within particular situations. This is a score three level of cognitive complexity.

Increased levels of cognitive complexity lead to more generalized resolutions of the conflict. Medium high levels of cognitive complexity result in a relativistic approach to the conflict suggesting that individuals will have different and justifiable points of view on the issue depending on their experience with the area in question. This is a score five level of cognitive complexity.

The highest level of cognitive complexity with regard to the original area of conflict is reached when different interpretations of the event or issue can be integrated with each other in a manner which allows each its perspective, and finds new ways to find functional relationships between them. This is a score seven level of cognitive complexity.

One of the implications of cognitive complexity scored in this way is that individuals with a low level of cognitive complexity who receive information that is contrary to the schemas or biases that they enter a situation with will be unable to change those schema or biases and that individuals with a high level of cognitive complexity are able to change their schemas or biases in the face of contrary information.

The PCM is generally designed to assess cognitive complexity in the area of social relationships. Thus, the PCM is not a content specific measure. Three types of sentence stems - described below - are generally used: those implying relationships of authority - which imply the conflict between authority and the individual subject to that authority; those implying conflict in relationships among peers - which imply conflict between points of view; and those implying intra-personal conflict in which different alternatives present themselves and a decision among them needs to be made.

In order to assess the level of cognitive complexity, the individual is presented with a set of sentence stems or other stimuli which present an implicit conflict between two perspectives on an issue of relevance to the domain within which cognitive complexity is to be assessed. The individual is asked to write a brief description of his or her approach to or understanding of the issue presented in the stem. Paragraphs for each stem generally require two to three minutes to complete. 1

Each response is then read by a trained rater who ascertains the existence of alternative perspectives of the area tapped by the stem, the decision rules used to decide among the alternatives, and the interrelationships constructed among the alternatives. These are the considerations used to determine the index of cognitive complexity for that response. Typically, five or six sentence stems are utilized and the average of these individual stems constitutes the Conceptual Level (CL) score. The CL score is the score produced by using the PCM. The scores range from one to seven. 2 The CL score of one is the score corresponding to the least cognitively complex level and seven is the score corresponding to the most cognitively complex level. The intermediate scores (2, 4, and 6) are transitional levels of cognitive complexity exhibited by individuals moving from one level of cognitive complexity to another.

Schroder et al. (1967) and Hunt et al. (1978) cite several studies which have used the PCM and which provide reliability data for the instrument over a variety of samples. These studies report inter-rater reliabilities ranging between .74 and .93, with the median reliability being r = .86 (Hunt et al., 1978). This is a satisfactory reliability for statistical analysis using the instrument. 3 Internal estimates of reliability range from .60 to .75 (Gardiner and Schroder, 1972) which is good for an instrument with a small number of items. This level of inter-item reliability suggests that the more items which are used for the total score, the better.

Gardiner and Schroder (1972) did a survey of validity data showing that scores on the PCM tend to correlate positively with greater independence and flexibility and complexity of thought. CL scores correlate negatively with authoritarianism and dogmatism. More recent studies show that the CL score is curvilinearly related to reasoning about capital punishment, with more extreme positions on either side of the issue associated with cognitive simplicity and more moderate positions associated with higher levels of cognitive complexity (DeVries and Walker, 1988).

Vannoy (1965) conducted a factor analytical study of a variety of assessments of cognitive complexity. His analysis showed that the CL scores did not correlate highly with other assessments of complexity and that the CL score appeared on its own factor, suggesting that it was measuring something different than other measures. The other measures he was using tended to be assessments of the differentiation of constructs rather than measures of integration. The suggestion from this study was that integrative complexity as assessed by the PCM is unique to this instrument, but contributes a statistically significant proportion of the communal variance of assessments of cognitive complexity.

One important implication of the theory underlying the PCM is that cognitive complexity in any domain is a progression of understanding in that domain which comes from experience within appropriate training environments. Individuals at lower levels of complexity in a domain either have not yet had the requisite experience or have become arrested in the progression of understanding due to environments which do not facilitate progression beyond a certain level. These individuals are presumed to be unable to understand the reasoning of individuals at levels much higher than their current level. This means that a match between environment and individual is required for the individual to be able to understand how to function in that environment and to grow from interaction with it. In a comprehensive review of literature concerned with the interaction of individual and educational environments, Miller (1981) found that educational interventions specifically designed to adapt the intervention to individual characteristics were only successful in improving educational achievement when the adaptation was done to cognitive complexity characteristics. This suggests that the CL score for an individual can be increased through training or teaching that is specifically targeted to the cognitive complexity level from which the individual is starting. This is as expected to be for both general levels of cognitive complexity and for domain specific levels of cognitive complexity.

An additional implication of the theory underlying the PCM is that individuals may exhibit different levels of cognitive complexity in different domain specific areas. The PCM is a measure of a general level of cognitive complexity likely to be exhibited in all areas. In areas where an individual has more experience or knowledge, the individual is likely to exhibit a higher level of cognitive complexity than their general level. 4 Also, an individual may exhibit a lower level of cognitive complexity in situations or areas that are highly stressful to the individual.

One question asked about the PCM is whether it is a simple duplication of IQ or other general ability measures. Since the test is written, the possibility exists that it is simply another measure of verbal fluency. Theoretically, it should not be duplicative of IQ since it assesses the structure of thought as well as content. IQ, however, is also, to some extent, a measure of progression of understanding in that items are written for a progression of skills over time. Hunt et al. (1978) summarize a number of studies in which correlations between CL scores and IQ or achievement measures were calculated in populations ranging from sixth grade public school students to university students. Correlations ranged from .15 to .43, consistent with the low to moderate correlation predicted by theory.

Cognitive complexity as it is understood here is a trait that can change over time, but one that does not do so rapidly. Cognitive complexity is not a trait that arises from lack of information, but rather is a trait such that those with higher levels of cognitive complexity are better able to deal with lack of information than those with low levels of cognitive complexity. Cognitive complexity is a trait which measures the ability of individuals to integrate disparate or even contradictory information. This type of complexity requires but is not the same as differentiated complexity. The more cognitive complexity an individual exhibits in a differentiated sense, then the more fine differences between different pieces of information they can appreciate. This supports integrative complexity, but is not the same as the ability to integrate all of those pieces of information together into a whole.

CL scores for traditional college students generally range between one and three. The scores for the subjects in this experiment ranged from .60 to 2.34. The mean is 1.62 and the standard deviation is .29. The scores and level of variation in our experiments are similar to what one would expect of a population of college students. The variance in this population is typical for this measure and is not as small as it appears because it is relative to the scale of the measure (Vannoy, 1965, and Gardiner and Schroder, 1972).

Scoring cognitive complexity can be done by either administering a test or scoring archival material. While both Suedfeld and Tetlock have used archival materials instead of administering a PCM to their subjects, the reasons for choosing one over the other are generally practical ones and are not based on any theory that either an administered test or the rating of archival materials is necessarily better than the other. If the subject is a politician or policy maker, then it is not practical to get the access and the time to administer a PCM. On the other hand, if the subjects are paid volunteers or students, then time and access allow for the administration of the PCM, but archival material is generally unavailable.

The Hostage Crisis Model

A hostage crisis situation was chosen as a typical case of multiparty crisis negotiation. The scenario is based on the hypothetical hijacking of a commercial airliner en-route from Europe to India and its forced landing in Pakistan. The passengers are predominantly Indian and the hijackers are known to be Sikhs. India is holding 800 Sikh security prisoners and the Sikh hijackers are demanding their release (see Kraus and Wilkenfeld, 1990a, Kraus et al., 1992). 5 The three parties must consider several possible outcomes: India or Pakistan launch military operations in attempts to free the hostages; the terrorists blow up the plane with the hostages and themselves aboard; India and the Sikhs negotiate a deal involving the release of some number of security prisoners in exchange for the hostages; Pakistan and the Sikhs negotiate a safe passage agreement; and the Sikhs give up.

In the simulation setting, actors negotiate a variety of issues relevant to the crisis until an agreement is reached or a player opts out of the negotiations by launching a military operation or blowing up the plane. The success of such opting out is determined probabilistically.

The values of the different outcomes for each actor depend upon many factors. Each actor has a list of objectives to pursue during the crisis. These objectives are arranged in a preference ordering by assigning utility points to each. Each player can earn up to 1000 utility points. However, as some of the objectives are directly at odds with objectives of the other players and some are even slightly contradictory with each other, in practice, a player can never score a perfect 1000. Table 1 presents a list of objectives for each of the three crisis actors, as well as the utility points associated with each objective. The point scores and objectives were decided upon pursuant to observation of international crises and consultations with experts. All point values, impacts of actions, objectives, etc. are pre-determined in the simulation software and the subjects can determine what these values and impacts are by looking at the information provided in the DSS. 6

Certain actions taken and decisions made during the course of the crisis have an impact upon the number of points awarded for each objective. These include:

Time is incorporated into the model both as a reference point for the calculation of utilities and probabilities and as a differential factor for the three parties. In general, time works in favor of the Sikhs, and against India and Pakistan. Time impacts on the probability of success and failure of an operation to free the hostages (whether it is day or night, whether there is time to train a rescue team, etc.), on publicity for the Sikh's cause (regardless of whether direct press access is granted), and deterioration over time in India and Pakistan's internal and external images (for more detail on the Hostage Crisis Model, see Kraus and Wilkenfeld, 1990, Wilkenfeld et al., 1995a).

TABLE 1 Objectives
INDIAN OBJECTIVES SIKH OBJECTIVES PAKISTANI OBJECTIVES
Safe return of the passengers 180 points Release of prisoners held in India 180 points Pakistani demonstration of control 375 points
Acceptable level of casualties among military 45 points Safe passage for terrorists 100 points
Maintenance of status quo in relations with Pakistan 50 points Message of Sikhs - publicity 320 points Pakistani internal image 100 points
Maintenance of status quo in relations with the major powers 80 points Damage to India deterrence against terrorism 75 points India is not strengthened 50 points
No concessions to terrorists 100 points amage to Indian relations with Pakistan 25 points India is not weakened 25 points
No negative effect on Indian public opinion 100 points Damage to Indian relations with major powers 50 points No strengthening of terrorists 50 points
Restrict publicity for terrorist cause 80 points Damage to Indian external image 100 points Maintenance of status quo with major powers 50 points
No damage to India's external image 50 points Damage to Indian internal image 75 points Maintenance of status quo within region 275 points
Credibility of India's deterrence against terrorism is maintained 140 points Enhanced position of terrorist group in Sikh movement 75 points Maintenance of status quo with India 25 points
India's overall strategic interests are unchanged 140 points Maintenance of status quo with Sikhs 50 points
Experience in counter-terrorism operations augmented 35 points
TOTAL 1000 pts TOTAL 1000 pts TOTAL 1000 pts

The GENIE Decision Support System

Decision support systems (DSSs) offer individuals and/or groups important information management and analytical tools. The GENIE DSS was designed to support subjects in the hostage crisis simulation. It was designed to be used by crisis decision makers who have little or no experience with the use of computers as analytic tools. Therefore, the interface was designed to be menu driven and mouse supported. GENIE's interface presents the subject with a main menu bar with five choices: SCENARIO, OUTCOMES, MESSAGES, ACTIONS, and SYSTEM

The OUTCOMES choice is the central decision-making mechanism of GENIE. It's menu includes the interactive outline, graphs and analytical tools which will be described in detail below.

Interactive Outline (Outcomes)

GENIE is designed to provide the subject with a clear mental picture of the hostage crisis model. GENIE combines its data management and modeling capability in one mousesupported screen (OUTCOMES), which enables a subject to quickly set parameters for the viewing of information. This screen not only provides quick access to information items, but also allows the subject to form a mental picture of the entire simulation. With this outline, the subject can then brainstorm and experiment with different options to form a personalized strategy for utility maximization (see Figure 2).

GENIE's interface allows a subject to define one or more hypothetical state(s) for the world and then to investigate possible future actions based on these states. The subject can explore outcomes resulting from his/her own actions as well as those of his/her opponents. A subject can also switch viewpoints to see things from the point of view of one or more of his/her opponents. This is possible, since the model by design permits full information over the other actors" preferences and the probability of success for attempts to opt out. The system employs a model-specific interactive outline with information categories that a subject can select to see graphic information about the scenario.

The interactive outline screen is organized into three main categories: viewpoints, information items, and world states. All of the model visualization capabilities of the system are contained within these three categories.

The viewpoints section allows the subject to specify one or more points of view for the subsequent queries. Having chosen the viewpoint(s), the subject can select which possible endings to the crisis that they would like to get information about. Once the viewpoint(s) and ending(s) are chosen, the subject can then choose actions that can affect point values. Here, a subject specifies what actions have been taken by which parties at the time that the crisis ending chosen takes place. These parameters are time, whether and when press access has been granted, whether and what level of logistical information has been granted by Pakistan to India, Pakistani behavior in the event of an Indian operation, Indian behavior in the event of a Pakistani operation, and if the Sikhs have killed or released hostages. The behavioral variables constitute aspects of the model which are negotiable among the parties.

During strategy formulation, a decision maker may want to find out whether it is worth it to expend energy to change the behavior of another actor. For example, in the hostage crisis scenario, the Sikhs could decide that India is not willing to deal and Pakistan will not agree to safe passage leaving them with no choice but to blow up the plane. In this case they should try to influence the behavior of the other actors so that at the time of the action, they (or their cause) receive maximum benefit. Influencing the state of the world could mean waiting a certain amount of time or convincing Pakistan to allow press coverage. GENIE lets a subject select multiple values for a given parameter and then simultaneously view the selected outcomes based on the different states. This provides the subject with a powerful tool to evaluate the effect of differing behavior upon the value of the ending of the crisis.

Display Module

GENIE provides the subject with two different graphic output options. A subject can select information about endings for one specified time period. This results in the display of bar graphs representing the utility point totals associated with that ending from the selected viewpoints given the selected world states. If the ending is an opt out ending the probabilities associated with the ending are also shown.

Subjects also have the ability to use expected utility to assess the value of military operations. Expected utility provides a method of evaluating the outcome of risky choices. Subjects may also look at the value of endings across time. This graph is a line graph as opposed to a bar graph.

In addition to these graphic functions, GENIE provides two analytical functions: mutually beneficial resolutions and bargaining minimums. Successful negotiation requires that an agent identify actions that will be most beneficial to him/her while taking into account the possible actions/reactions of his/her opponents. The mutually beneficial resolutions option allows an agent to select outcomes to investigate and then displays information for those outcomes with high ranking payoffs. Despite this capability in the DSS, subjects do not automatically move towards mutually beneficial endings to the crisis. Since we are holding situational variables constant, differences in negotiation outcome would be expected to reflect individual characteristics and their composition in groups.

The bargaining minimums feature helps a negotiating agent to compare the payoff obtained from a negotiated settlement with that obtained from optingout of the negotiation. It automatically calculates a player's reservation price and finds the optimal time period for a player to optout of the negotiations. It also determines a negotiated settlement that has an equivalent or higher payoff than optingout in the optimal time period.

The Language Editor

All communications among the three parties during the negotiation take place through the use of a language editor. This menu driven program allows subjects to chose a recipient, select from different categories of messages, select a message and then if appropriate add threats or promises to the message. A wide variety of negotiating tactics may be employed using the language editor since there are many different messages available to the subject. 7 The message sets are different for each of the different roles (Wilkenfeld, Kraus, and Holley, 1998).

The language editor provides good experimental control by restricting the types of messages which can be sent. As messages are sent, they are recorded in data files which facilitates analysis of the messages at a later date. Each message is a communication similar to an e-mail from one negotiating party to another negotiating party. A copy of a message may be sent to the third negotiating party. The fact that face-to-face communications are missing imparts a sense of reality. It is not typical in this kind of crisis that actors would be conducting negotiations in a face-to-face environment. While it is difficult to estimate the impact of one"s communications in this environment, this is also quite difficult in real life. For more details on the GENIE DSS, see Wilkenfeld et al., 1995a, Kraus and Wilkenfeld, 1990, 1993.

Summary of Previous Results

At the core of our previous experimental work is the development of a strategic model of negotiation, with an accompanying decision support system (see Kraus, Wilkenfeld, and Zlotkin 1995, Wilkenfeld, Kraus, and Holley 1998). Decision support systems (DSSs) can play a crucial role in the crisis decision making process by allowing the decision maker to navigate large amounts of information quickly and to explore interrelationships between factors which may influence the decision. A DSS can also facilitate the simultaneous evaluation of multiple positions in crisis negotiations. This can play a decisive role in real time negotiations by allowing the supported parties to rapidly formulate dynamic strategies and quickly evaluate their adversaries" proposals. Thus, a central theme of this research is that the employment of a DSS by a crisis decision maker can facilitate the identification of utility maximizing strategies on the part of an individual actor. Related to this is the notion that groups of decision makers so supported by a DSS are in a good position to achieve mutually satisfying outcomes to crises.

Previous experimental results address three broad groups of research questions: (1) the impact of the use of decision support systems on the utility maximizing behavior of crisis negotiators; (2) how the dynamics of crisis negotiations impact on their outcomes; and (3) the relationship between the level of cognitive complexity of crisis decision makers and the outcomes of crisis negotiations. All experiments were conducted with University of Maryland undergraduates as subjects, during the period 1991-1996.

(1) The initial set of experiments, reported in Kraus et al. (1992) and Holley and Wilkenfeld (1994), focused on the following two research questions:

(1a) Does use of a sophisticated computer-based decision support system increase the likelihood of higher payoffs to the individual negotiators? Experimental results indicate that regardless of their roles in the simulation - in this case Indian Government, Pakistani Government, Sikh hijackers - the average utility scores achieved at the conclusion of the simulation were higher for decision support system users than for non-users. In addition, the achievement of agreements was a more prevalent outcome for the DSS users than for non-users.

(1b) Do the communications patterns exhibited by participants in a simulation that ended in a negotiated agreement differ from those exhibited by participants in simulations that ended in non-agreement (i.e., violence)? The key to differentiating between crisis negotiations which ended in agreement and those which evolved to violence lies in the communications patterns exhibited by those playing the role of Pakistan in the simulation - the role most capable of mediation. Agreement outcomes were typified by a virtually identical number of messages sent by Pakistan to the other two parties, while non-agreement outcomes showed Pakistan sending three times as many messages to India as it did to the Sikhs. Thus, we concluded that by maintaining open communications channels with each of the other parties, Pakistan was able to play a central role in bringing about peaceful resolution of the crisis.

(2) A second set of experiments, reported in Wilkenfeld et al. (1995a, 1995b), continued the exploration of research questions pertaining to the impact of the use of a decision support system on negotiation processes and outcomes.

(2a) Are DSS users more likely than non-DSS users to identify utility maximization as their primary objective in a crisis negotiation situation? Research findings supported the contention that DSS users were most strongly motivated by utility maximization, while non-DSS users tended to be motivated most strongly by upholding principles. These latter crisis decision makers appear to have been overwhelmed by the vast amount of information available to them and the difficulty of calculating the utility of different actions under the pressures of the negotiation. They were thus led to rely upon deeply held principles. The DSS users were able to calculate utilities quickly and efficiently and were, therefore, able to act as utility maximizers.

(2b) Will DSS users achieve higher utility scores than non-DSS users? Not only does access to the DSS encourage the actors to be motivated by utility maximization as a goal, but they will be more successful than their non-DSS counterparts in actually achieving higher utility scores as outcomes.

(2c) Will the presence of a DSS-supported user among adversaries in a crisis situation produce higher overall utility scores than in groups in which none of the adversaries have such access? In addition to facilitating the achievement of utility maximization for the individual DSS-user, our results also show that the existence of a DSS-supported user among a group of adversaries is likely to result in a higher overall utility score for the group. i.e., a mutually beneficial resolution to the crisis.

(2d) Are negotiations in which a DSS user is present more likely to end in agreement than are negotiations in which no such user is present? Results confirm the greater tendency for crisis situations in which one of the adversaries has access to a DSS to terminate in agreement rather than some type of violent outcome. In these cases, access to this decision support tool by an individual decision maker in the group helps in the identification of a mutually beneficial outcome, which in this case is identified with the achievement of agreement.

(3) The third line of experiments (Wilkenfeld et al., 1996) was designed to assess the impact of cognitive complexity of decision makers on their behavior in crisis negotiation situations and on the outcomes which they attain. These experiments were designed to study the relationship between cognitive complexity and negotiation behavior, in an effort to better understand the dynamics which lead certain persons to have greater success in negotiations, and which lead certain groups of adversaries to achieve more mutually beneficial outcomes. The underlying assumption is that the greater the level of cognitive complexity which the individual brings to the crisis negotiation situation, the more likely it is that his/her process of arriving at decisions will result in utility maximization. Individuals at higher levels of cognitive complexity will be better able to cope with the crisis environment than will those at lower levels. In particular, the perceived shortness of time for response, combined with truncated and restricted communications among the parties, will frustrate negotiators at the lower levels of cognitive complexity and will generate sub-optimal outcomes to negotiations.

The results of this series of experiments are best summarized as follows:

(3a) Subjects at higher levels of cognitive complexity developed greater proficiency with the DSS and were able to master more of the query tools it contains.

(3b) The subjects did not show an overall relationship between cognitive complexity and crisis outcome, either in terms of the achievement of higher scores in the crisis simulation, or in terms of a greater propensity to reach agreement as opposed to violent crisis termination. We speculate about the difficulty which subjects with low cognitive complexity have in taking on the roles of others, with assigning utility to probabilistic events, and that no subjects, no matter how high their level of cognitive complexity, were complete masters of their own destinies in the negotiation. Examination of roles revealed that for those subjects who represented actors with similar characteristics and motivations to their own in such situations, there was a strong positive relationship between level of cognitive complexity and the achievement of high utility scores in the simulation.

(3c) While neither individual subjects nor crisis participants grouped by cognitive complexity level showed any propensity for those at higher levels to be more likely than others to achieve termination of the simulation through agreement, we did find that prior exposure to the principles of international politics and negotiation resulted in a much greater propensity to reach agreements and to solve international crises nonviolently.

Hypotheses

As noted earlier in this paper, the research we are reporting on here constitutes a portion of a large research program. The hypotheses that we present are limited to those which explore the manner in which grouping decision makers by cognitive complexity affects the outcome of the negotiations. 8 Homogeneity, ultra homogeneity, and heterogeneity of crisis decision groups refers to the mix of cognitive development levels of the individual decision makers, and will be operationalized in the section on Experimental Design below.

Large disparities in cognitive complexity levels among individual negotiators will impact negatively on the overall group"s ability to reach an agreement in a crisis situation. This is a direct result of poor comprehension of each others" goals and strategies, in effect an inability to see a situation from the perspectives of the other parties and therefore difficulties in making proposals which are likely to be acceptable to the others. Mutual benefit is made more difficult to reach, while frustration leads to both lengthier periods of negotiation before agreements are reached, and ultimately to a higher probability of opting out, that is, employing violence.

One might expect that the greater the similarity in cognitive complexity among individuals, the easier it would be for them to comprehend each others" goals and strategies, and therefore the more likely they would be to come to mutually beneficial agreements. This is true of homogeneous groupings but not of ultra homogeneous groupings. In the ultra homogeneous groupings the individuals are so similar in the goals they set and the strategies they pursue that they either move relatively swiftly to an agreement, or else they become frustrated when their proposals are not immediately accepted and opt out. Thus, we have the relatively curious possibility that regardless of outcome - agreement or opting out - that outcome will be reached relatively quickly.

To summarize, our hypotheses suggest a positive relationship between the ability of negotiators to understand each others goals and strategies, and their ability to successfully negotiate mutually beneficial agreements. The ability to understand each other is in turn related to the level of heterogeneity of cognitive complexity in the group of negotiators. In the GENIE DSS, for any set of actors, mutually beneficial agreements yield greater utility point totals than the expected utility associated with an individual attempt to opt out of the negotiation. Therefore, we expect that groups will try to achieve such agreements in their attempts to resolve the crisis situation. Our hypotheses suggest that groups that can communicate effectively will be more likely to actually achieve such agreements than groups that cannot communicate effectively. 9

Experimental Design

Experiments based on the Hostage Crisis Simulation and the hypotheses discussed above were run in Spring 1996, Fall 1996, Spring 1997, and Fall 1997. The subjects consisted of University of Maryland juniors and seniors enrolled in either an advanced course in international negotiations or an advanced course in foreign policy analysis. Training and administration procedures for all subjects were identical.

Prior to pre-simulation training, the subjects were asked to complete a Paragraph Completion Measure (PCM) (see above). This is the tool that was used to measure cognitive level (CL). CL scores for traditional college students generally range between 1 and 3. Based on the scoring of these measures, subjects were placed in either ultra homogeneous, homogeneous or heterogeneous groups of three. Ultra homogeneous groups had less than a .1 difference in their CL scores, which is less than the measurement error in the scores. Thus, these groups had for all intents and purposes individuals with identical levels of cognitive complexity. Homogeneous groups had differences in CL scores between .1 and .5, which is greater than the measurement error in the scores. Thus, these individuals were similar but not identical in terms of their levels of cognitive complexity. Heterogeneous groups had a difference in their CL scores greater than .5 or more than twice the measurment error difference in the scores. Thus, these individuals were quite different in their levels of cognitive complexity.

All subjects then attended a three hour training session that consisted of three parts. The first part consisted of a presentation of information on the simulation - how it works, the rules, the scenario, the objectives, and the items that could be negotiated - and information on how to use the computer programs necessary for the simulation - the DSS, the Language Editor, and the network communication package. The second part consisted of 15-20 minutes of time for the subjects to practice on the computer and to ask questions. The third part consisted of taking the quiz that tested DSS proficiency.

One week after the training session subjects returned to participate in the simulation. Prior to starting the simulation all subjects were encouraged to ask any questions they had about the simulation or the computer programs that they would be using. Then the rules governing the simulation were reviewed. The subjects were assigned to the roles of India, Pakistan, and the Sikhs randomly within their groups. A maximum of three hours was allowed for the simulation. At the end of the simulation an evaluation was administered to gather data on motivation, strategy, and source of any frustration experienced.

Students were motivated to do well in the simulation because a portion of their mid-term grade was tied to their sucessful completion of the simulation.

Results

Over a four semester period between 1996 and 1997, data were collected on 52 runs of the Hostage Crisis Simulation. The breakout of those cases for the three types of groupings and by outcome is summarized in Table 1.

Table 1: Group Type and Outcome of Negotiation
Group Type Agreement Opt Out Total
Ultra Homogeneous 6 6 12
Homogeneous 15 5 20
Heterogeneous 7 13 20
Total 28 25

The homogeneous groups clearly achieved the highest percentage of agreements as compared to the other two types of groups. They achieved a mutually beneficial agreement 75% of the time as opposed to 50% of the time for the ultra homogeneous groups and 35% of the time for the heterogeneous groups. As suggested by our hypotheses, both ultra homogeneous groups and heterogeneous groups are less likely to achieve agreements than homogeneous groups. This occurs because both ultra homogeneous groups and heterogeneous groups have communications problems that homogeneous groups do not have.

Statistical analysis of the raw data further supports this claim. A chi square test was used to generate the probabilities that we would observe our results if the null hypothesis that the three types of groups generate the same proportions of outcomes is true. The low probabilites in Table 2 suggest that the null hypotheses are false and that the different groupings do indeed lead to different outcomes. The results are shown in Table 2.

Table 2: Null Hypothesis Test Results
Null Hypothesis Probability of our observing results in Table 1 if the Null Hypothesis were True
Both ultra homogeneous groups and heterogeneous groups achieve outcomes in the same proportion as the homogeneous group. .000
he ultra homogeneous groups achieve outcomes in the same proportion as the homogeneous groups. .045
The heterogeneous groups achieve outcomes in the same proportion as the homogeneous groups .000
The heterogeneous groups achieve outcomes in the same proportion as the ultra homogeneous groups .180

In further support of our results, there is very little correlation either between cognitive complexity and self-rated computer abilities (r = .17, p = .1) or between crisis outcomes and self-rated computer abilities (r = .11, p = .1). So cognitive complexity does not devolve simply to computer skills nor are computer skills an overriding factor in determining crisis outcomes.

The second set of hypotheses made two related arguments concerning the elapsed time to conclusion of the hostage crisis (Hypotheses 1b, 1c, and 2b). The unequal makeup of heterogeneous groups will mean that even when they do manage to reach agreement, such an outcome will take longer to reach than for the homogeneous groups, which presumably have an easier time of communicating their preferences and offers to each other. On the other hand, these same heterogeneous groups will move toward an opt out outcome (military operation or blowing up of the plane) more quickly than their homogeneous counterpart groups, with the latter deliberating at great length until it is absolutely clear that the negotiation has stalled. Finally, the ultra homogeneous groups will move quickly toward an outcome, regardless of whether this is to be an agreement or opting out.

The average times, in terms of mean number of periods to conclusion, for all three types of groups are shown in Tables 3 and 4. These results include a control for the semester in which the results were generated, because the mean time to conclusion varies widely from semester to semester. We believe that this is due to experiments being run at different times of year and in different relationships to other major time committments such as mid-terms.

Table 3: Mean Number of Periods to Conclusion
Group Type Time to Conclusion Time to Agreement Time to Opt Out
Ultra Homogeneous 10.35 10.64 10.07
Homogeneous 11.45 11.08 12.56
Heterogeneous 11.55 12.58 11.00


Table 4: t test one tailed P values that Ultra Homogeneous and Heterogeneous Means are Different from Homogeneous Means
Group Type Time to Conclusion Time to Agreement Time to Opt Out
Ultra Homogeneous na .31 .02
Homogeneous na na na
Heterogeneous na .04 .02

As our hypotheses suggested, the ultra homogeneous groups conclude more quickly than the homogeneous groups regardless of whether they reached agreement or opted out. Furthermore, heterogeneous groups are quicker to opt out than homogeneous groups and slower to reach agreement than the homogeneous groups. These results hold up across all four semesters in which these simulations were run.

Summary of Findings and Conclusions

This examination of the impact of grouping decision makers by their level of cognitive complexity on outcomes has generated several potentially important experimental findings.

Clearly homogeneous groups are more likely to generate mutually beneficial agreements than either ultra homogeneous groups or heterogeneous groups. The importance of this finding is heightened when we note that this is true without having to take account of the characteristics of individuals assigned to different roles which we found necessary in our previous paper (Wilkenfeld et al. 1996). In other words, the level of homogeneity of cognitive complexity within the group appears to be a more important factor in producing mutually beneficial outcomes than the cognitive complexity levels of individual subjects themselves. The abilities to communicate effectively and to clearly comprehend adversaries" goals and objectives are both enhanced by similarity in cognitive levels. However, our results also show that while some ultra homogeneous groups can move even more quickly than homogeneous groups toward crisis conclusion, some of these ultra homogeneous groups are more likely to fail to reach agreements that might have been achieved by the homogeneous groups, albeit achieved more slowly.

Although the relatively small number of deviant cases for each of the three groupings precludes statistical analysis at this point, some interesting observations are possible. In the ultra homogeneous groups, where agreements were unexpected, there was a much greater emphasis on DSS use or on the need for cooperation in the groups reaching agreement than in those opting out. These two emphases may be the result of participants being at particular levels of cognitive complexity. In the homogeneous groups, where opting out was unexpected, the player opting out commented that he felt that at least one of the other parties was unwilling to negotiate. This feeling may be explained by the opting out player's high sensitivity to control issues (as noted on the authority stems of the PCM) than player's in other homogeneous groups. In the heterogeneous groups, where agreements were unexpected, there was an emphasis on cooperation and the need to reach an agreement that was not found in the heterogeneous groups that resulted in opting out. Again this emphasis may be related to the level of cognitive complexity of the individuals involved.

A further set of experiments currently underway will increase the n and may allow statistical analysis of the deviant cases in the future. Among the issues that we would like to explore is the particular makeup of those homogeneous triads that worked against the conclusion of agreements in 25% of the cases, despite expected utility scores favoring such an outcome and a mix of cognitive complexity levels that should have facilitated it. Similarly, what was distinctive about those heterogeneous groups which managed to reach agreement despite a mix of cognitive complexity scores that should have made it difficult for these decision makers to work effectively together? In both instances, is it the mix of scores, the overall level of these scores, the scores on individual components, or some combination of these three which produce these unexpected outcomes? Finally, what are the characteristics of some ultra homogeneous groups that allow them to quickly come to agreement versus others that eventually opt out of the negotiations?

We have also taken some preliminary steps in understanding at least in relative terms how long negotiations may take given different kinds of grouping by cognitive complexity. This may be useful in either providing adequate time for negotiations or in knowing how long one has before negotiations are likely to break down.

Despite the finding of a significant impact on crisis outcome which can be attributed to the particular mix of capabilities of individual negotiators, the practical utility of these findings is nevertheless somewhat limited, since it is not likely that the time constraints usually associated with international crises will allow for careful evaluation of one"s adversaries in a negotiation so that the most appropriate negotiators in terms of cognitive levels can be assembled. Notwithstanding this difficulty, either knowing ahead of time about the cognitive levels of ones adversaries, or being able to estimate them during the course of the negotiation, will allow a skilled negotiator to attempt to adapt his/her approach in both substance and style to that which will most likely lead to a mutually beneficial outcome.

Bibliography

Notes:

Note *: The research reported in this paper was supported by the National Science Foundation under grant #IRI-9423967, and the US Institute of Peace under grant #SG-35-95. Portions of this paper were presented at the 1996 annual meeting of the International Studies Association and at the 1997 annual meeting of the American Political Science Association. The authors wish to acknowledge the helpful comments and suggestions of Daniel Druckman, Peter Suedfeld, Judith Torney-Purta, Margaret Hermann and Keith Shimko. Back.

Note **: Wilkenfeld is also affiliate faculty in the University of Maryland Institute for Advanced Computer Studies. Back.

Note 1: The stems used for the set of experiments reported on here are: I think rules, When I am criticized, When I am uncertain, When I am told what to do, When someone disagrees with me.Additional detail about instructions etc. is available upon request. Back.

Note 2: Several attempts have been made to construct objective or machine scorable assessments of conceptual levels. None of these has proven reliable. The reason for this appears to be that objective instruments are susceptible to the interpretation of the reader and susceptible to faking. They also assess the content of the response, not its structure. Since it is possible to arrive at the same content from different structures, valid assessments of structure are difficult from content oriented methods. (Harvey, (1967), Harvey and Hoffmeister, (1971), Tuckman, (1966)) Back.

Note 3: All scoring for these experiments was done by a trained psychologist who uses the PCM on a regular basis and has an established reliability for scoring as high as that achieved by Harvey. Back.

Note4: Cognitive complexity is specific to content domains. The PCM as developed by Harvey, Hunt, and Schroder (1961) and Schroder, Driver, & Streufert (1967) measures cognitive complexity for the content domain of general social relationships. Since we expect there to be some difference between the domain of general social relationships and the domain of international relations, we decided to attempt to develop a version of the PCM that would measure cognitive complexity in the international relations domain. This international version of the PCM, currently under development, has the potential for being a better predictor of behavior in international negotiations than the more general version. Back.

Note 5: The original specification of the model was based on a Middle East setting involving Israel, Egypt, and Palestinian terrorists. However, this was changed to the IndiaPakistanSikh model in order to minimize subject bias during the course of the simulation. Back.

Note 6: The Hostage Crisis Simulation and the GENIE decision support system (see below) are based on a strategic model of negotiation (Kraus, Wilkenfeld, and Zlotkin, 1995) and a set of utility functions which make the assumption that a negotiated agreement terminating the crisis generates greater mutual benefit than other outcomes of the crisis. That is, while successfully launching a military operation or blowing up the plane (see below) - that is, opting out – may result in greater benefit for a single player (although the success of these events are probabilistic), greatest mutual benefit derives from agreement Back.

Note 7: For example, one possible message that could be sent from the Indians to the Sikhs is "You are not bargaining in good faith, we are withdrawing our promise to persuade Pakistan not to launch an operation." Back.

Note 82: Findings on hypotheses concerning cognitive complexity and the use of the DSS, and on behavior during the negotiation, will be presented in future papers. We will also present our findings on a domain specific PCM for international relations in a future paper. Findings on the effect of the DSS versus not having the DSS on negotiation outcomes have been published previously (see Wilkenfeld et al., 1995a) and are summarized above. Findings on the effects of cognitive complexity on the ability to use the DSS and the outcomes achieved are summarized above and are available in detail in Wilkenfeld et al. (1996). Back.

Note 9: See footnote 8. Mean total points for agreements is 1806 versus 1078 for non-agreements. Mean India points for agreements is 568 versus 491 for non-agreements. Mean Pakistan points for agreements is 711 versus 298 for non-agreements. Mean Sikh points for agreements is 527 versus 288 for non-agreements. Back.