CWS Frequently Asked Questions

Part 1 | Part 2

Who should use CWS?

Anyone interested in objective assessment of judgmental performance when “correct” answers are unknown.

 

But I’m not studying a judgment task.  Can I still use CWS to assess expertise?

CWS has demonstrated success in assessing performance as well.  If the dependent variable is one that is directly linked to the expertise you are examining, give CWS a try.  It is especially helpful when no gold standard outcome measure is available.  See Thomas and Pounds (2001) in the download area for an example.

 

Is there a CWS computer program?

On the web site, there are individual programs for Schumann-Bradley comparisons and for calculating CWS with nominal responses.  Ordinary statistics programs can be used to calculate the quantities need for the CWS ratio with numerical responses.

 

Is there a CWS book?

The CWS e-Book is being prepared. We will make it available in the downloads section as soon as it is complete.

 

What sort of response instruments yield data that are suitable for a CWS analysis?

The closer to continuous, the better.  Fine gradations avoid artificially deflating the measure of inconsistency that constitutes the denominator of the CWS ratio.  So a twenty-point category scale is likely to be better than a seven-point scale, and a graphic rating (line mark) scale even better.  Try to anchor the instrument with extreme stimuli that you expect to elicit much higher and much lower evaluations than the stimulus set you want the expert to judge.  You may not know enough about the stimuli to choose these end-anchors; perhaps a domain expert can help.

 

How do you calculate CWS when you have nominal data?

The algorithm below was presented in the paper by Weiss and Shanteau (2001).  A computer program implementing the algorithm is available in the software area of this web site.

 

Illustration of CWS Index for Nominal Data (four response alternatives)

For both numerator and denominator, we utilize the reciprocal of the proportion of obtained matches to possible matches.  In measuring discrimination, a match is evidence of failure to discriminate, so the greater the proportion of observed matches, the poorer the discrimination.  In measuring inconsistency, a match means the response was consistent, so the greater the proportion of observed matches, the smaller the inconsistency.  Expert performance is marked by few matches across rows (stimuli), and many matches down columns (replications).  If there are no matches down columns – no consistency at all - the CWS ratio is undefined, but that outcome unambiguously connotes a lack of expertise.

CWS Numerator (Discrimination) =

 

Number of possible matches (per row) = ₅C₂ = 10

 

Numerator =

 

CWS Denominator (Inconsistency) =

 

Number of possible matches (per column) = ₃C₂ = 3

 

Denominator =

 

CWS Index =

 

Part 2 of the CWS FAQ

Site designer:  J. Shawn Farris

Webmaster: C. Vowels cvowels@ksu.edu