• Home
• Conference Details
• Articles & Guidelines
• Conference Archives

SECTION 5: New Simulations, Tests, and Methods

Simulations such as in-basket exercises, group discussions, management games, and analysis exercises described in Byham’s 1970 Harvard Business Review article are still the bedrock of assessment center methodology. However, they have been supplemented by new types of exercises, most importantly the interaction simulation. In this exercise the assessee is given background information about the need to interact with an individual (subordinate, peer, or customer), and personal information about the individual. After the assessee has had an opportunity to prepare, he or she conducts a simulated interaction with a person trained as a roleplayer. The "interviewee" follows a well-defined role and makes standard responses to all issues that might come up. A trained assessor observes the assessee’s behavior.

Although leaderless group exercises still are used commonly to assess leadership, one-to-one interaction simulations have become more popular. This change reflects a general feeling that individual leadership skills are not necessarily correlated with group leadership skills. Another reason for the switch is that people going through the same group exercise may have quite different experiences. Group interactions depend on the nature of the people involved. Sometimes the group is highly competitive; other times it is quite cooperative. Sometimes several people vie for leadership; other times only one person takes charge. This lack of consistency has caused organizations especially concerned with EEO issues to opt for the more standardized interaction simulations or different forms of group exercises.

A growing number of organizations have adopted a "total simulation" approach to assessment. Instead of having a number of distinct and independent exercises, these organizations have integrated their exercises into a common scenario. Characters introduced in the in-basket exercise are seen in later simulations, and candidates play the same role throughout the assessment process.

Using Videotape to Stimulate Behavior

A development in the last 15 years involves the use of videotape to stimulate assessee behavior. An assessee watches a video of a situation he or she will face on the job (e.g., an interaction with a subordinate). Periodically the tape stops and the assessee is presented with four choices of what to do or say. A score is calculated based on the assessee’s responses to a number of these situations. The scoring system is developed based on a validity study.

Other applications use the video simulation to elicit verbal responses. This, unaided, overcomes any bias caused by the multiple choice format.

Psychological Inventories and Projective Tests

The original AT&T research assessment centers used psychological inventories (e.g., the Edwards Personal Preference Schedule) and projective tests (e.g., the Thematic Apperception Test) to supplement observations of assessees’ behaviors as they progressed through assessment center simulations. AT&T dropped these instruments after they converted their research assessment centers to operational assessment centers run by the Bell operating companies. AT&T dropped the tests for two reasons: (1) the operational assessment centers used managers (rather than psychologists) as assessors, and (2) paper-and-pencil instruments were disputed during this period (1960s) because of possible adverse impact on protected groups. Most organizations that adopted the assessment center methodology followed AT&T’s lead, concentrating on behavioral exercises rather than paper-and-pencil tests. Most even dropped intelligence tests because of the common finding of adverse influence on blacks. However, the Bell companies retainethese tests.

Today intelligence tests (general ability tests) are being used again in conjunction with assessment centers. Research data show that the combination of intelligence data and behavioral observations provides a markedly better means of evaluating people than either used alone (Thornton & Byham, 1982). The problem with using paper-and-pencil intelligence tests and other psychological instruments is that they require very careful validation efforts, and assessors must be specially trained in both data interpretation and how to integrate that data with behavioral data.

Projective tests are frequently used in executive-level assessment centers. Inventories to match job and candidate motivation profiles are used at all levels.

Multiple Perspective (Self-Report, Boss, Subordinate, and Workplace Peer Evaluation Instruments)

An assessment center provides insights into many job dimensions, but usually not all important dimensions. Dimensions such as Work Standards and Energy are not evaluated well in assessment centers. To fill in these gaps and to get additional insights to dimensions that are assessed in assessment centers, many organizations supplement their assessment centers with self-reports and with evaluations by the assessee’s boss and workplace peers or subordinates. In the usual situation, an assessee is given six questionnaires that list the target dimensions with definitions. The assessee completes one and gives the other five to his/her boss, peers, or subordinates. All questions are sent directly to a central location where a computer summarizes the data and prepares a report.

The combination of assessment center, self-, and boss/peer/subordinate evaluations of a common set of dimensions makes a powerful impact on assessees. The feedback counselor and the assessee can compare and contrast each dimension’s ratings from each source (self, others, and the assessment center). Based on these insights, they can define developmental actions more accurately.

New Methodologies

The biggest drawback in the ongoing use of traditional assessment centers is the amount of managerial time required. In a typical assessment center, a manager leaves his/her job for two or three days to observe participants’ performance in simulations and then spends an additional day or two meeting with other observers to make final evaluations. Although managers recognize the importance of selection and promotion decisions, they are often reluctant to devote this much concentrated time to assessment. A related problem is the formality of the traditional assessment center, which tends to make the center an "event." This may build expectations and call attention to who is being assessed and who has not been asked to participate. The traditional assessment center also forces organizations to put people through the process in groups; the method is useless when there are only two candidates for a position.

These constraints have limited assessment center method applications in some organizations to only a few selection or promotion decisions. As a result, many important and effective applications, such as defining training needs, have not been utilized widely. Although organizations recognize the increasing importance of accurately diagnosing training needs before sending people to training programs, the problems associated with staffing developmental assessment centers often make their use prohibitive, even though assessment center methodology is the best available diagnostic instrument for many positions. Managers agree on the importance of thorough and accurate diagnoses, but are reluctant to spend the time needed to produce the excellent diagnoses that the assessment center methodology yields.

Deformalizing the Method

A number of organizations in the United States and overseas have overcome the implementation problems noted earlier by making their assessment centers less formal and rigid while keeping the basic components that provide validity. Organizations do this by incorporating the assessment center method into an organization’s day-to-day activities, rather than by having their managers go off to a designated place, or to a "center."

The individual to be assessed is given a list of managers responsible for filling the position. The assesee then schedules his/her own meetings with these managers over a period of several weeks, according to the schedules of all parties. The managers involved fit the time for the exercises into their usual activities.

During these meetings the managers put the assessee through the same job simulations used in formal assessment centers. For example, one manager might interview the assessee about why he or she took certain actions in the in-basket exercise; another might have the assessee present findings from an analysis and planning exercise; and a third might observe the assessee in a one-to-one interaction with another manager who role-plays a subordinate.

At an appointed time the managers (assessors) meet to hold an assessor discussion that works exactly like such discussions in a traditional assessment center. The assessors give actual examples of the participant’s behavior to back up their ratings on each of the dimensions they evaluated. After sharing all their observations, the assessors reach consensus on the individual’s strengths and weaknesses in each dimension. Then, if the purpose of the assessment center is to provide the basis for selection or promotion decisions, the assessors make an overall evaluation. If the objective of assessment is to diagnose training needs, the assessors’ final step is to develop a profile of the assessee’s strengths and developmental needs.

All key components of the assessment center method are present: multiple job simulations; use of behavior observed in simulations to predict future behavior in the target job; organization of observed behavior around job-related dimensions; and a systematic data integration session involving several assessors who have observed participants independently in the simulations. Only the rigidity is removed. This allows even the smallest organization to apply the assessment center method in making selection/promotion decisions.

Using Videotape To Record Behavior

Another increasingly popular technology is the use of videotape equipment to capture assessee behavior. Rather than having assessors observe individuals in simulations, participants’ behavior is recorded on videotape. The tape and the asseessees’ written output then can be sent virtually anywhere and assessors can view and evaluate the taped and written performance at their convenience. After each assessor has observed and evaluated the assigned simulation, a standard data integration session can be held, or the data can be integrated by a computer using an expert system.

Automated Integration of Data

Software programs can expedite the assessment process, and are therefore advocated by many assessors and administrators. In a common system assessors input their observations directly into computers. The computer organizes behavior by dimension and feeds it back to the assessor in a way that facilitates the rating of each dimension. The computer, using an expert system, then checks the rating and if the computer’s rating differs from that of the assessor, a second assessor reviews the data and shares his or her insights with the assessor. Together, they make a decision on the dimension rating for the exercise.

At the integration meeting, a computer integrates all the behavioral observations across exercises and presents the data in a convenient way for assessor analysis and decision making. In some organizations, an expert system substitutes for the integration meeting. This mathematical data integration is possible because of the high reliability of the assessor exercise dimensional ratings, where reliabilities of .90 and higher are common.

The computer prints out a detailed final report giving dimensional ratings with behavioral examples. The computer system decreases assessor time by more than half and dramatically decreases assessor and administrator training time.

Use of Outside Assessors

Because of downsizing, etc., there are fewer middle managers to act as assessors. Thus, more than half of the organizations operating assessment centers in the United States have turned to trained outsiders to administer and assess candidates.