Cohen on the science of psychological measurement” - Article summary

A utility analysis refers to a family of techniques that entail a cost-benefit analysis designed to yield information relevant to a decision about the usefulness and/or practical value of a tool of assessment. It is done in order to see whether the benefits of using a test outweigh the costs of that test. The objective of a utility analysis determines the required information (1) and the specific methods that have to be used (2).

One method of utility analysis is expectancy data. This is converting the test data to an expectancy table. It can provide a likelihood that a test taker will score within some interval of scores on a criterion measure. Taylor-Russel tables provide an estimate of the extent to which inclusion of a particular test in the selection system will improve selection. It gives an increase in base rate of successful performance that is associated with a particular level of criterion-related validity.

The selection ratio is a numerical value that reflects the relationship between the number of people to be hired and the number of people available to be hired. The base rate refers to the percentage of people hired under the existing system.

Top-down selection is a process of awarding available positions to applicants whereby the highest scorer is awarded the first position. A downside of top-down selection is that this may lead to unintended discriminatory effects.

One limitation of the Taylor-Russell tables is that the relationship between the predictor and the criterion must be linear. Naylor-Shine tables make use of the differences between the means of the selected and unselected groups to derive an index of what the test is adding to the established procedures. It determines the increase in average score on a criterion measure.

The Brogden-Cronbach-Gleser formula calculates the dollar amount of utility gain resulting from using a specific selection instrument. It uses the following formula:

utility gain=NTrxySDyZm-NC

N represents the number of applicants selected per year. T represents the average length of time in the position. Rxy represents the criterion-related validity coefficient for the given predictor and criterion. SDy represents the standard deviation of performance of employees. Zm represents the standardized mean score on the test for selected applicants. C represents the cost of the test for each applicant.

Productivity gain refers to an estimated increase in work output. It helps estimate the percent increase in output expected through the use of a particular test.