CWS Frequently Asked Questions

Part 1 | Part 2

Can you use CWS when you do not have two or more scores per stimulus?

You do not need multiple responses for all stimuli.  However, a measure of consistency is needed, so there must be some stimuli for which you do have two or more scores. 

 

How do you define “the stimulus” in a dynamic environment?

You must make some possibly arbitrary determinations so that you can distinguish one stimulus one from another.  A possibility is to use a freeze-frame or snapshot approach.  Another option is to control the stimulus at the beginning of a cycle, ignoring variations that occur as time unfolds.  This tricky problem has been addressed in a paper by Thomas and Pounds.

 

Is it possible to have an inconsistency measure of zero?

The denominator of the CWS ratio measures inconsistency.  It will be zero if the judge gives the same response to all instances of each individual stimulus.  One way this can occur is if the judge recognizes each stimulus and remembers the previous response to it.  Another possibility is that the response instrument is coarse (i.e., has few options), so that the judge has no doubt about which response is appropriate.  Both of these possibilities reflect true expertise for the task as presented (although the assessment may not be very meaningful.  Another way to produce zero inconsistency is for the judge to respond identically to all stimuli; of course, this is not expert performance.

 

Why doesn’t CWS include a speed component?  One expects an expert to accomplish a task faster than a non-expert.

Speed-accuracy tradeoffs are a classic finding in experimental psychology.  The issue is that an expert can reduce discrimination and consistency to gain speed, or vice versa.  There seems to be no way to incorporate response time into the CWS index without violating its essential spirit by incorporating expert opinion in determining how to weight the speed-accuracy tradeoff.

 

How can I use CWS to determine whether my (boss, employee, spouse, physician, attorney) is an expert?

You need to be able to elicit repeated judgments of a set of relevant stimuli, so that measures of discrimination and inconsistency can be gathered.  You also need to be able to present the same stimuli to comparable folks, in order to see whether your target person generates higher CWS than the competition.  The necessary presentations may be feasible only in a research setting.

 

The expertise you crave is likely to be multifaceted, so you will need to assess the expertise on a number of tasks.  CWS assesses expertise, exhibited behavior, while you have expressed interest in certification, which is a statement about a person.  A person is “an expert” when he or she demonstrates expertise on the tasks you consider relevant.  To solve the problem formally requires an integration of the demonstrated task expertise, suitably weighted for importance.  The CWS team has not yet worked through the details of this integration. 

 

What is a replicate?

A replicate is simply the set of all of the stimulus conditions that make up a design.  So if we have three replicates, it means we have run through the design 3 times, with everything identical.  It's like we're xeroxing the design.

Confusion sometimes sets in because the term "replicate" is used casually to mean a duplication.  A single duplication (i.e. a second run through the design) means there are two replicates.  We might speak of replicating a specific set of design cells, but that usage of the term is confusing.

 

In the language of fractional factorials, we can speak of a half-replicate of a multi-factor design.  That means a carefully selected (to ensure balanced presentation across levels) half of the stimulus conditions is presented.  Similarly, a quarter-replicate means a carefully selected fourth of the stimulus conditions are presented.  Can you have three replications of a half-replicate?  Sure.

 

Replicates are not required in order to use CWS.  We need an estimate of inconsistency, specifically the extent of discrepancy between the responses to the same stimulus.  Of course, we do not want to look only at discrepancies generated by a single stimulus, because that stimulus might be unique in how it generates variability. 

 

So in order to get a broad representation of discrepancies across the stimulus range, it is convenient to replicate the entire design.  So we generally ask for "replication" in the data sets we analyze, by which we mean two or more complete passes through the design.  That is because we then get what we need in terms of representative discrepancy measures, and the computations are easy with standard Anova programs.

Part 1 of the CWS FAQ

Site designer: J. Shawn Farris

Webmaster: C. Vowels cvowels@ksu.edu