
CWS Frequently Asked Questions
Part 1  Part 2

Who should use CWS?

Anyone interested in objective assessment of judgmental performance when “correct” answers are unknown.


But I’m not studying a judgment task. Can I still use CWS to assess expertise?

CWS has demonstrated success in assessing performance as well. If the dependent variable is one that is directly linked to the expertise you are examining, give CWS a try. It is
especially helpful when no gold standard outcome measure is available. See Thomas and Pounds (2001) in the download area for an example.


Is there a CWS computer program?

On the web site, there are individual programs for SchumannBradley comparisons and for calculating CWS with nominal responses. Ordinary
statistics programs can be used to calculate the quantities need for the CWS ratio with numerical responses.


Is there a CWS book?

The CWS eBook is being prepared. We will make it available in the downloads section as soon as it is complete.


What sort of response instruments yield data that are suitable for a CWS analysis?

The closer to continuous, the better. Fine gradations avoid artificially deflating the measure of inconsistency that constitutes the denominator of the CWS ratio. So a
twentypoint category scale is likely to be better than a sevenpoint scale, and a graphic rating (line mark) scale even better. Try to anchor the instrument with extreme stimuli that
you expect to elicit much higher and much lower evaluations than the stimulus set you want the expert to judge. You may not know enough about the stimuli to choose these endanchors;
perhaps a domain expert can help.


How do you calculate CWS when you have nominal data?

The algorithm below was presented in the paper by Weiss and Shanteau (2001). A computer program implementing the algorithm is available in
the software area of this web site.


Illustration of CWS Index for Nominal Data (four response alternatives)

Stimulus 1

Stimulus 2

Stimulus 3

Stimulus 4

Stimulus 5

Matches

Replicate 1

A

D

B

C

C

1

Replicate 2

A

B

B

B

B

6

Replicate 3

A

B

A

B

A

4

Matches

3

1

1

1

0



For both numerator and denominator, we utilize the reciprocal of the proportion of obtained matches to possible matches. In measuring discrimination, a match is evidence of failure to
discriminate, so the greater the proportion of observed matches, the poorer the discrimination. In measuring inconsistency, a match means the response was consistent, so the greater the
proportion of observed matches, the smaller the inconsistency. Expert performance is marked by few matches across rows (stimuli), and many matches down columns (replications). If
there are no matches down columns – no consistency at all  the CWS ratio is undefined, but that outcome unambiguously connotes a lack of expertise.

CWS Numerator (Discrimination) =_{}


Number of possible matches (per row) = _{5}C_{2} = 10


Numerator = _{}


CWS Denominator (Inconsistency) = _{}


Number of possible matches (per column) = _{3}C_{2} = 3


Denominator = _{}


CWS Index = _{}

Part 2 of the CWS FAQ
Site designer: J. Shawn Farris
Webmaster: C. Vowels
cvowels@ksu.edu
