Psychology 708             Psychometric Theory             Fall, 2000

Instructor: Kevin E. O'Grady

Office: Zoology-Psychology 2123H

Office Hours: TT 8:30-12

[A sign-up sheet is posted on my office door]

Office Phone: 301-405-5902

e-mail: ogrady@psyc.umd.edu

WWW: www.kevin-e-ogrady.umd.edu/psy708.htm

Course Objectives:

This course is intended as a general introduction to the construction and validation of measures of individual differences. The primary aim of the course is to provide you with the necessary background in test theory, psychometrics, reliability,validity, and types of tests so that you can begin to be both a critical consumer of research which has employed measures of individual differences, as well as a thoughtful psychologist in the choice, use, and interpretation of tests in applied settings.

Evaluation:

My evaluation of your performance in this course will be based on a paper and a final exam. Each of these will count equally towards your final grade.

The paper should be approximately 15-25 text pages in length (this is an extremely rough guideline), and describe the development and initial validation of a individual difference measure" can be interpreted quite broadly, and can be just about any instrument that could be used in research or practice. Thus,you are free to discuss the development of tests of "psychopathology," "behavioral tests," "attitude tests", etc.,. It should begin with a concise review of the relevant theoretical literature(s) and a survey of similar measures. The remainder of the paper should detail: 1) methods and procedures you would follow to develop your test; and 2) an outline of an initial validation study that you consider the best way to address the value of your efforts.

The final exam will be three hours in length, and will include three questions. The first two questions will be drawn from a list of 10 questions you will be given one week prior to your final. The remaining question will require you to discuss two issues that have been raised in the class and/or readings.

Texts:

**Nunnally, J.C. (1978). Psychometric theory (Second Edition). New York: McGraw-Hill.

Topics to be covered:

A. Introduction to Assessment: Issues and Pseudo-issues; Ethical Issues

B. Measurement and Statistical Foundations

C. Test Theory

D. Generalizability Theory

E. Social Learning Theory

F. Reliability

G. Validity

H. Types of tests

I. Methods of Test Construction

J. Methods of Test Development

K. Methods of Test Validation

L. Prediction

Readings:

A.

Anastasi, A. (1948). The nature of psychological "traits". Psychological Review, 55, 127-138.

**Rotter, J.B. (1954). Social learning and clinical psychology. New York: Prentice-Hall, Chapter 1.

Anastasi, A. (1979). On the formation of psychological traits. American Psychologist, 25, 800-910.

*Fiske, D.W. (1973). Can a personality construct be validated empirically? Psychological Bulletin, 80, 89-92.

*Alker, H.A. (1972). Is personality situationally specific or intrapsychically consistent? Journal of Personality, 40, 1-16.

*Bem, D.J. (1972). Constructing cross-situational consistencies in behavior: Some thoughts of Alker's critique of Mischel. Journal of Personality, 40, 17-26.

*Bowers, K.S. (1973). Situationism in psychology: An analysis and a critique. Psychological Review, 80, 307-336.

**Messick, S. (1981). Constructs and their vicissitudes in educational and psychological measurement. Psychological Bulletin, 89, 575-588.

*Endler, H.S. (1973). The person versus the situation -- a pseudo issue? Journal of Personality, 41, 287-303.

**Sarason, I.G., Smith, R.E., & Diener, E. (1975). Personality research: Components of variance attributable to the person and the situation. Journal of Personality and Social Psychology, 32, 199-204.

**Epstein, S. (1977). Traits are alive and well. In Magnusson, D. and Endler, N.S. (Eds.), Personality at the Crossroads, Hillsdale, NJ: Lawrence Erlbaum Associates. Chapter 4.

**Tryon, W.W. (1979). The test-trait fallacy. American

Psychologist, 34, 402-406.

Korchin, S.J., & Schuldberg, D. (1973). The future of clinical assessment. American Psychologist, 40, 313-321.

Mischel, W., & Peake, P.K. (1982). Beyond deja vu in the search for cross-situational consistency. Psychological Review, 89, 730-755.

Anastasi, A. (1985). Some emerging trends in psychological measurement: A fifty-year perspective. Applied Psychological Measurement, 9, 121-138.

Guilford, J.P. (1985). A sixty-year perspective on psychological assessment. Applied Psychological Measurements, 9, 341-350.

B.

Nunnally: Chapters 1, 2, 4, 5

**Blalock, H.M. (1968). The measurement problem: A gap between

the languages of theory and research. In H.M. Blalock &

B. Blalock (Eds.), Methodology in Social Research, New York: McGraw-Hill.

O'Grady, K.E. (1982). Measures of explained variance: Cautions and limitations. Psychological Bulletin, 902, 766-777.

Block, J. (1964). Recognizing attentuation effects in the strategy of research. Psychological Bulletin, 62, 214-216.

Block, J. (1963). The equivalence of measures and the

correction for attenuation. Psychological Bulletin, 60, 152-156.

**Green, B.F., Jr. (1978). In defense of measurement. American Psychologist, 33, 664-670.

Stevens, S.S. (1946). On the theory of scales of measurement. Science, 103, 667-680

Gaito, J. (1980). Measurement scales and statistics: Resurgence of an old misconception.

Towsend, J.T., & Ashby, F.G. (1984). Measurement scales and statistics: The misconception misconceived. Psychological Bulletin, 96, 394-401.

Mitchell, J. (1986). Measurement scales and statistics: A clash of paradigms. Psychological Bulletin, 100, 398-407.

Maxwell, S.E. & Delaney, H.D. (1985). Measurement scales and statistics: An examination of variable validity. Psychological Bulletin, 97, 85-93.

Davison, M.L. & Sharna, A.R. (1988). Parametric statistics and levels of assessment. Psychological Bulletin, 104, 137-144.

C.

**Green, B.F. (1981). A primer of testing. American Psychologist, 36, 1001-1011.

**Allen, M.J., & Yen, W.M. (1979). Introduction to measurement theory. Monterey, CA: Brooks/Cole, Chapter 3.

Klein, D.V., & Cleary, T.A. (1967). Platonic true scores and errors in psychiatric rating scales. Psychological Bulletin, 68, 77-80.

Hattie, J. (1985). Methodology review: Assessing unidimensionality of tests and trends. Applied Psychological Measurement, 9, 139-164.

D.

**Cronbach, L.J., Rajaratnam, N., & Gleser, G.C. (1963). Theory of generalizability: A liberalization of reliability theory. British Journal of Statistical Psychology, 16, 137-163.

Ward, D.G. (1986). Factor indeterminary in generalizability theory. Applied Psychological Measurement, 10, 159-165.

E.

Rotter, J.B., Chance, J.E., & Phares, E.J. (1972). Applications of a social learning theory of personality. New York: Holt,

Rinehart & Winston, Chapter 1.

**Rotter, J.B. (1960). Some implications of a social learning

theory for the prediction of goal directed behavior from testing procedures. Psychological Review, 67, 301-316.

F.

Nunnally: Chapters 6, 7

**Cronbach, L.J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297-334.

Feldt, L.S., Woodruff, D.J., & Salih, F.A. (1987). Statistical inferences for coefficient alpha. Applied Psychological Measurement, 11, 93-103.

**Shrout, P.E. & Fleiss, J.L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420-428.

*House, A.E., House, B.J., & Campbell, M.B. (1978). Measures of interobserver agreement: Calculation formulas and distribution effects. Journal of Behavioral Assessment, 3, 37-57.

Zwick, R. (1988). Another look at interrater agreement. Psychological Bulletin, 103, 374-378.

*DeMaster, B., Reid, J., & Twentyman, C. (1977). The effects of different amounts of feedback on observer's reliability. Behavior Therapy, 8, 317-329.

Booth, C.L., Mitchell, S.K., & Solin, F.K. (1979). The generalizability study as a method of assessing intra- and interobserver reliability in observational research. Behavioral Research Methods & Instrumentation, 11, 491-494.

**Bejar, I.I. (1980). Biased assessment of program impact due to psychometric artifacts. Psychological Bulletin, 87, 513-524.

James, L.R., Demaree, R.G., & Wolf, G. (1984). Estimating within group interrater reliability with and without response bias. Journal of Applied Psychology, 69, 850-98.

Cicchetti, D.V., Showalter, D., & Tyrer, D.J. (1985). The effect of number of rating scale categories on levels of interrater reliability: A Monte Carlo investigation. Applied Psychological Measurement, 9, 31-36.

G.

Nunnally: Chapter 3

**Cronbach, L.J., & Meehl, P.E. (1955). Construct validity in psychological tests. Psychological Bulletin, 3, 635-694.

**Loevinger, J. (1957). Objective tests as instruments of psychological theory. Psychological Reports, 3, 635-694.

**Campbell, D.T., & Fiske, D.W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105.

**Lesser, G.S. (1959). Population differences in construct validity. Journal of Consulting Psychology, 23, 60-65.

*Campbell, D.T. (1960). Recommendations for APA test standards regarding construct, trait, or discriminative validity. American Psychologist, 15, 546-553.

**Loevinger, J. (1965). Person and population as psychometric concepts. Psychological Review, 72, 143-155.

**MacCroquodale, K., & Meehl, P.E. (1948). On a distinction between hypothetical constructs and intervening variables. Psychological Review, 55, 95-107.

*Messick, S. (1980). Test validity and the ethics of assessment. American Psychologist, 35, 1012-1027.

Rushton, J.P., Brainerd, C.J., & Pressley, M. (1983).

Behavioral development and construct validity: The principle of aggregation. Psychological Bulletin, 94, 18-38.

Linn, R.L. (1978). Single-group validity, differential validity, and differential prediction. Journal of Applied Psychology, 63, 507-512.

Horowitz, L.J., Inouye, D. & Siegelman, E.Y. (1979). On averaging judges' ratings to increase their correlation with an external criterion. Journal of Consulting and Clinical Psychology, 47, 453-458.

Sussmann, M., & Robertson, D.U. (1986). The validity of validity: An analysis of validation study designs. Journal of Applied Psychology, 71, 461-468.

H.

**Rotter: Chapter 8

Frank, L.K. (1939). Projective methods for the study of personality. Journal of Psychology, 8, 389-413.

Cronbach, L.J. (1949). Statistical methods applied to Rorschach scores: A review. Psychological Bulletin, 46, 393-429.

*Henry, E.M., & Rotter, J.B. (1956). Situational influences on Rorschach responses. Journal of Consulting Psychology, 20, 457-462.

*Magnusson, M.G. (1960). Verbal and non-verbal reinforcers in the Rorschach situation. Journal of Clinical Psychology, 16, 167-169.

*Jackson, D.N., & Messick, S.J. (1961). Acquiescence and desirability as response determinants on the MMPI. Educational & Psychological Measurement, 21, 771-790.

*Baxter, J.C., & Morris, K.L. (1968). Item ambiguity and item discrimination in the MMPI. Journal of Consulting and Clinical Psychology, 32, 309-313.

**Scott, W.A., & Johnson, R.C. (1972). Comparative validities of direct and indirect personality tests. Journal of Consulting and Clinical Psychology, 3, 301-318.

**Mischel, W. (1972). Direct versus indirect personality assessment: Evidence and implications. Journal of Consulting and Clinical Psychology, 38, 325-326.

**McClelland, D.C. (1972). Opinions Predict opinions: So what else is new? Journal of Consulting and Clinical Psychology, 38, 325-326.

I.

Nunnally: Chapter 8, 14, 15

*Berg, I.A. (1955). Response bias and personality: The deviation hypothesis. Journal of Psychology, 40, 61-72.

*Couch, A., & Keniston, K. (1960). Yeasayers and naysayers:

Agreeing response set as a personality variable. Journal of Abnormal and Social Psychology, 24, 340-354.

Rorer, L.G. (1965). The great response-style myth. Psychological Bulletin, 63, 129-156.

Reindquist, E.A. (1966). Item and response characteristic in attitude and personality measurement: A reaction to L.G. Rorer's "The great response-style myth." Psychological Bulletin, 66, 166-177.

Winkle, J.D., Kanouse, D.E., & Ware, J.E. (1982). Controlling for acquiescence response set in scale development. Journal of Applied Psychology, 67, 555-561.

Ray, J.J. (1984). A further comment on the Winkler, Kanouse, and Ware method of controlling for acquiescent response bias. Journal of Applied Psychology, 69, 359.

Cowen, E.L., & Tongas, P.N. (1959). The social desirability of trait descriptive terms: Applications to a self-concept inventory. Journal of Consulting Psychology, 23, 361-365.

Thomas, C.W., & Peterson, D.M. (1982). Methodological issues in attitude scale construction. Journal of Social Psychology, 116, 245-253.

**Holden, R.R., & Jackson, D.N. (1979). Item subtlety and face validity in personality assessment. Journal of Consulting and Clinical Psychology, 47, 459-468.

Shapira, Z., & Shirom, A. (1980). New issues in the use of behaviorally anchored rating scales: Level of analysis, the effects of incident frequency, and external validation. Journal of Applied Psychology, 65, 517-523.

Hase, H.D., & Goldberg, L.R. (1967). Comparative validity of different strategies of constructing personality inventory scales. Psychological Bulletin, 67, 321-248.

Zavala, A. (1965). Development of the forced-choice rating scale technique. Psychological Bulletin, 63, 117-124.

J.

**Masling, J. (1960). The influence of situational and interpersonal variables in projective testing. Psychological Bulletin, 56, 65-85.

Murstein, B.I. (1965). Projection of hostility on the TAT as a function of stimulus, background, and personality variables. Journal of Consulting Psychology, 28, 43-48.

Cromwell, R.L., & Lundy, R.M. (1964). Productivity of clinical hypotheses on a sentence completion test. Journal of Consulting Psychology, 18, 421-424.

*Masling, J. (1965). Differential indoctrination of examiners and Rorschach responses. Journal of Consulting Psychology, 29, 198-201.

Mussen, P.H., & Naylor, H.K. (1954). Relationship between overt and fantasy aggression. Journal of Abnormal and Social Psychology, 29, 235-239.

Sacks, J.B. (1949). Effect upon projective responses of stimuli referring to the subject and to others. Journal of Consulting Psychology, 13, 12-21.

**Epstein, S. (1966). Some theoretical considerations on the nature of ambiguity and the use of stimulus dimensions in projective techniques. Journal of Consulting Psychology, 30, 183-192.

*Holmes, D. (1974). The conscious control of thematic projection. Journal of Consulting and Clinical Psychology, 42, 323-329.

**Chapman, L.J., & Chapman, J.P. (1967). Genesis of popular but erroneous psychodiagnostic signs. Journal of Abnormal Psychology, 1967, 72, 193-204.

**Chapman, L.J., & Chapman, J.P. (1969). Illusory correlation as an obstacle to the use of valid diagnostic signs. Journal of Abnormal Psychology, 74, 271-280.

Paunonen, S.V., & Jackson, D.N. (1979). Nonverbal trait

inference. Journal of Personality and Social Psychology, 37, 1645-1659.

**Kane, J.S., & Lawler III, E.E. (1978). Methods of peer assessment. Psychological Bulletin, 85, 555-586.

**Brief, A.P. (1980). Peer assessment revisited: A brief comment on Kane and Lawler. Psychological Bulletin, 88, 78-29.

**Kane, J.S., & Lawler III, E.E. (1980). In defense of peer assessment: A rebuttal to Brief's critique. Psychological Bulletin, 88, 80-81.

**Saal, F.E., Downey, R.G., & Lahey, M.A. (1980). Rating the ratings: Assessing the psychometric quality of rating data. Psychological Bulletin, 88, 413-428.

**Love, K.G. (1981). Comparison of peer assessment methods: Reliability, validity, friendship bias, and user reaction. Journal of Applied Psychology, 66, 451-457.

Landy, F.J., & Farr, J.L. (1980). Performance rating. Psychological Bulletin, 87, 72-107.

K.

Palmer, J.O. (1951). A dual approach to Rorschach validation: A methodological study. Psychological Monographs, 65, No. 8.

Rychlak, J.F., & O'Leary, L.R. (1965). Unhealthy content in the Rorschach responses of children and adolescents. Journal of Projective Techniques, 1965, 29, 354-368.

Murray, J.D., & Rychlak, J.F. (1966). Healthy, neutral, and unhealthy content in the Rorschach responses of schizophrenic and normal adults. Journal of Projective Techniques, 30, 254-260.

Snyder, W.V., & Cohen, L.H. (1944). Validity of imagery testing in schizophrenia. Character and Personality, 9, 35-43.

Fitzgerald, B.J. (1958). The relationship of two projective measures of sociometric measure of dependent behavior. Journal of Abnormal and Social Psychology, 56, 199-204.

*Rotter, J.B., & Wickens, D.D. (1948). Consistency and generality of ratings of social aggressiveness. Journal of Consulting Psychology, 12, 234-239.

*Wanderer, Z.W. (1969). Validity of clinical judgments based on figure drawings. Journal of Consulting and Clinical

Psychology, 33, 143-150.

**Cone, J.D. (1977). The relevance of reliability and validity for behavioral assessment. Behavior Therapy, 8, 411-426.

**Nelson, R.O., Hay, R., & Hay, W.H. (1977). Comments on

Cone's "Relevance of reliability and validity for behavioral assessment". Behavior Therapy, 8, 427-430.

*Cone, J.D. (1978). The behavior assessment grid (BAG): A conceptual framework and a taxonomy. Behavior Therapy 9, 882-898.

Tellegen, A., Kamp, J., & Watson, D. (1982). Recognizing individual differences in predictive structure. Psychological Review, 89, 95-105.

Sawyer, J. (1986). Measurement and prediction, clinical and statistical. Psychological Bulletin, 66, 178-200.

Cone, J. D. (1979). Confounded comparisons in triple response mode assessment research. Behavioral Assessment, 1, 85-95.

Schmitt, N., & Stults, D.M. (1986). Methodology review: Analysis of multitrait-multimethod matrices. Applied Psychological Measurement, 10, 1-22.

L.

**Bem, D.J., & Allen A. (1974). On predicting some of the people some of the time: The search for cross-situational consistencies in behavior. Psychological Review, 81, 506-520.

**Epstein, S. (1979). The stability of behavior: I. On predicting most of the people much of the time. Journal of Personality and Social Psychology, 37, 1097-1128.

**Epstein, S. (1980). The stability of behavior: II. Implications for psychological research. American Psychologist, 35, 790-806.

Allen, B.P., & Potkay, C.R. (1981). On the arbitrary distinction between states and traits. Journal of Personality and Social Psychology, 41, 916-928.

*Uba, L. (1981). Ascribing meaning to single, molecular measures: A question of methodological adequacy. Journal of Personality and Social Psychology, 40, 1090-1094.

Harris, J.B., Jr. (1980). Nomovalidation and idiovalidation: A quest for the true personality profile. American Psychologist, 35, 729-744.

Rotter, J.B. (1981). The psychological situation in social learning theory. In Magnusson, D. (Ed.). Toward a psychology of situations: An interactional perspective. Hillsdale, NJ: Lawrence Erlbaum Associates.

**Goldberg, L.R., & Werts, C.E. (1966). The reliability of clinician's judgments: A Multitrait-multimethod approach. Journal of Consulting Psychology, 30, 19-206.

**Schroeder, H.E. (1970). Use of feedback in clinical prediction. Journal of Consulting and Clinical Psychology, 38, 265-269.

**Rotter, J.B. (1967). Can the clinician learn from experience? Journal of Consulting Psychology, 31, 12-15.

Lautenschlager, G.J., & Mendoza, J.C. (1986). A step-down hierarchical multiple regression analysis for examining hypotheses about test bias in prediction. Applied Psychological Measurement, 10, 133-139.