The Search for Assessment Center Construct Validity Continues: Examining the Relationship Between AC Final Dimension Ratings and External Measures of Cognitive Ability and Personality
For nearly two decades researchers have challenged the construct validity of assessment centers (ACs; Sackett & Dreher, 1982; Howard, 1997). Until recently, most of the evidence casting doubt on this issue was based on internal validation strategies that examined the psychometric soundness of within-exercise dimension ratings. Findings from these studies suggest that while assessor ratings exhibit reasonable levels of convergent validity, they lack discriminant validity. These conclusions also appear to transcend cultural boundaries (Lievens & Klimoski, 2001). Factor analytic examinations have also generally produced disappointing findings (Bycio, Alvares & Hahn, 1987; Schneider & Schmitt, 1992). Taken together, past research using internal validation strategies questions assessors' ability to distinguish among different skill dimensions (Thornton, 1992).
More recently, researchers have begun using external validation strategies to examine AC construct validity. Studies linking assessor ratings with alternate measures of conceptually related measures have produced promising results. For instance, some studies have shown that cognitive ability and personality measures are correlated with comparable overall dimension (Shore, Thornton, & Shore, 1992; Thornton, Tziner, Dahan, Clevenger, & Meir, 1997), and exercise ratings (Spector, Schneider, Vance, & Hezlett, 2000). On the other hand, others have failed to demonstrate such relationships (Chan, 1996; Fleenor, 1996; Goffin, Rothstein, & Johnson, 1996).
Although this line of research has provided alternate construct validity evidence, several of the studies suffer from some limitations (e.g., use of external measures possessing questionable psychometric properties, lack of strong a priori conceptual linkages between personality factors and AC ratings, matching AC dimensions to personality traits that reflect little to none of behaviors associated with the respective AC dimension). The present study extends the search for AC construct validity by investigating the relationship between eight final dimension ratings and cognitive ability (Watson-Glaser Critical Thinking Appraisal) and personality (California Psychological Inventory) correlates. Like past research, it was proposed that final dimension ratings would correlate higher with conceptually related measures than with conceptually unrelated measures.
This session will present the results from a study based on archival data obtained on 71 employees from utility and home improvement retail organizations that participated in a one-day "dual-purpose" AC designed for both selection and development purposes. Theoretical and practical implications (e.g., Can assessors distinguish between meaningful clusters of skill dimensions?), limitations, and future directions will be addressed.
Speaker
Jeffrey Kudisch, Ph.D.
Assistant Professor & Director
Industrial/organizational Psychology Program
University of Southern Mississippi
Dr. Kudisch is an Assistant Professor and Director of the Industrial/Organizational Psychology Doctoral Program at USM. He also serves as the Director of USM's Center for Applied Organizational Studies. Dr. Kudisch is also is a co-founder and Principal Partner of Personnel Assessment Systems, Inc., a human resource consulting firm specializing in management and executive assessment.
Dr. Kudisch received his Ph.D. in I/O Psychology from the University of Tennessee, Knoxville, and his M.S. in I/O Psychology from the University of Central Florida. Dr. Kudisch has provided consulting services in the areas of assessment centers, testing and selection, training and development, organizational change, employee attitudes, customer satisfaction, and performance management (e.g., performance appraisal, multi-source feedback) to both public- and private-sector organizations since 1988. Before venturing into academia, Dr. Kudisch spent several years working in various human resource-related functions at the Tennessee Valley Authority. Dr. Kudisch has written articles in the areas of assessment centers, personnel selection, managerial credibility, charismatic leadership, and multi-source feedback. He is a member of the Society for Industrial/Organizational Psychology, Academy of Management, American Psychological Association, and the South Mississippi Society for Human Resource Management.
