Utility Analysis (from: 9th edition of Psychological testing and assessment) - Cohen - Article

2. Utility Analysis (Cohen)

A utility analysis is a family of techniques that consists of cost-benefit analysis designed to yield information that is relevant to a decision about the usefulness and/or practical value of a tool of assessment. For example, a utility analysis may be conducted to evaluate whether the benefits of using a test (or training program or intervention) outweigh the costs.

Depending on the specific objective of a utility analysis, different approaches can be used. Sometimes, a utility analysis requires little more than converting a scatter plot of test data to an expectancy table. Useful tables for a utility analysis are developed by H.C. Taylor and J.T. Russell in 1939. The so-called Taylor-Russell tables provide an estimate of the extent to which inclusion of a particular test in the selection system will improve selection. In other words, the tables offer an estimate of the percentage of employees hired by the use of a particular test who will be successful at their jobs, given different combinations of three variables: (1) the validity of the test; (2) the selection ratio used, and; (3) the base rate. Here, selection ratio refers to a numerical value that reflects the relationship between the number of people being hired and the number of people being available to be hired. Base rate refers to the percentage of people hired under the existing system for a particular position. There are, however, two important limitations of the Taylor-Russell tables. First, these tables assume that the relationship between the predictor (the test) and the criterion (rating of performance on the job) is linear. If this assumption is violated, the Taylor-Russell tables are inappropriate. Second, there is a potential difficulty in identifying a criterion score that separates "successful" from "unsuccessful" employees. This dichotomization if not realistic in most situations. As a side note, it should be said that these tables do not consider the costs of testing in comparison to the benefits.

To overcome the above mentioned limitations, Naylor and Sine (1965) developed an alternative set of tables, called the Naylor-Shine tables. These tables obtain the difference between the means of the selected and unselected groups in order to derive an index of what the test is adding to already existing procedures.

Both sets of tables can be used to aid in the decision of hiring people. It is important, however, to be aware of the fact that such hiring decisions can never be based solely on the basis of variables such as the validity of an employment test and the prevailing selection ratio. Many other kinds of variables are entered into the processes of hiring and personnel selection (for instance: general physical or mental health, drug use, minority status, and so forth).