Critical thinking
Article: Cohen
Utility Analysis
Utility analysis: a family of techniques that entail a family of techniques that entail a cost-benefit analysis designed to yield information relevant to a decision about the usefulness and/or practical value of a tool of assessment.
It is not one specific technique used for one specific objective. It is an umbrella term covering various possible methods, each requiring various kinds of data to be inputted and yielding various kinds of output.
In a most general sense, a utility analysis may be undertaken for the purpose of evaluating whether the benefits of using a test outweigh the costs.
If undertaken to evaluate a test, the utility analysis will help make decisions whether:
- one test is preferable to another test for use for a specific purpose.
- one tool of assessment is preferable to another tool of assessment for a specific purpose
- the addition of one or more tests that are already in use is preferable for a specific purpose.
- no testing or assessment is preferable to any testing or assessment
If undertaken for the purpose of evaluating a training program or intervention, the utility analysis will help make decisions regarding whether:
- one training program is preferable to another training program
- one method of intervention is preferable to another method of intervention
- the addition or subtraction of elements t an existing training program improves the overall training program by making it more effective and efficient
- the addition or subtraction of elements to an existing method of intervention improves the overall intervention by making it more effective and efficient
- no training program is preferable to a given training program
- no intervention is preferable to a given intervention
The endpoint of a utility analysis is typically an educated decision about which of many possible courses of action is optimal.
The specific objective of a utility analysis will dictate what sort of information will be required as well as the specific methods to be used.
Expectancy data
Some utility analyses will require little more than converting a scatterplot of test data to an expectancy table.
An expectancy table can provide an indication of the likelihood that a test-taker will score within some interval of scores on a criterion measure.
Taylor-Russell tables: increase the base rate of successful performance that is associated with a particular level of criterion-related validity.
The value assigned for the test’s validity: the computed validity coefficient.
But, the relationship must be linear.
Naylor-Shine tables: tells us the likely average increase in criterion performance as a result of using a particular test or intervention: also provides selection ratio needed to achieve a particular increase in criterion performance.
The expectancy table or chart: tells us the likelihood that individuals who score within a given range on the predictor will perform successfully on the criterion.
The Brogden-Cronbach-Gleser formula
Used to calculated the utility gain from the use of a particular selection instrument under specified conditions.
Utility gain: an estimate of the benefit of using a particular test or selection method.
Utility grain = (N)(T)(rxy)(SDy)(Zin) – (N)(C)
N =the number of participants
T = tenure
rxy = the criterion-related validity coefficient for the given predictor and criterion
Sdy = the standard deviation of performance
Zin = the mean score on the test for selected participants
C = the cost of the test for each applicant
Utility analysis has its own vocabulary.
Hit: a correct classification
Miss: an incorrect classification
Hit rate: the proportion of people that an assessment tool accurately identifies as possessing or exhibiting a particular trait, ability, behaviour, or attribute
Miss rate: the proportion of people that an assessment tool inaccurately identifies as possessing or exhibiting a particular trait, ability, behaviour, or attribute
False positive: a specific type of miss whereby an assessment tool falsely indicates that the test-taker possesses or exhibits a particular trait, ability, behaviour, or attribute
False negative: a specific type of miss whereby an assessment tool falsely indicates that the test-taker does not possess or exhibit a particular trait, ability, behaviour, or attribute
Base rate: the proportion of people in the population that possesses a particular trait, behaviour, characteristic, or attribute.
Return to investment: the ratio of benefits to costs.
Add new contribution