The College of Business Administration at Sam Houston State develops a creative, comprehensive critical thinking measure.

Like many business schools, the College of Business Administration at Sam Houston State University adopted “critical thinking” as a learning goal for its undergraduate and graduate degree programs.  The energy, research and thoroughness that they applied in developing an original measure for this goal, however, is anything but “business as usual.”  Two Business faculty members, Valerie Muehsam and John Miller, who teach statistics worked with a faculty member in philosophy, Frank Fair, and a psychology doctoral student, Wendy McCoy,  to develop a multiple choice test to assess critical thinking skills. The goal was to develop a valid, comprehensive measure that would avoid the costs associated with using commercial tests, and could be easily administered within a 45 minute class period. 

The first essential task was to operationalize the measure, also known as translating the learning goal into measurable student outcomes.  After consulting the literature on critical thinking, the research team considered many definitions of critical thinking including the one put forth by Michael Scriven and Alec Fisher in their book Critical Thinking:  Its Definition and Assessment[1].  The SHSU team decided to adopt the “multiple rating items” methodology promoted by the authors.  The authors argue that this approach can provide a method that is both valid and easily administered.  Dr. Fisher, former Head of the Philosophy Department at the University of East Anglia and Director of the Centre for Research in Critical Thinking was brought in as a consultant. 

Dr. Fisher stated “The SHSU Critical Thinking Assessment Group has worked imaginatively and carefully on constructing this instrument.  They have been particularly imaginative in using multiple-rating items – the first time this has been done in a test which is intended to be publicly available.  They have responded thoughtfully and well to all my comments and have constructed an instrument which has a good chance of working as intended.” 

The Texas Assessment of Critical Thinking Skills (TACTS™ ) instrument is the product of this 2-year, interdisciplinary effort.  The 45 minute, multiple choice tests evaluates 12 thinking skills that the SHSU team identified as important in the business disciplines:

1. Being able to employ basic quantitative reasoning techniques to solve percentage problems, simultaneous equations, etc.
2. Being able to apply the rules of probability calculus.
3. Being able to interpret what inferences can be made from quantitative information presented in a chart or diagram.
4. Being able to distinguish data showing a correlation from information needed to establish a cause and effect relation.
5. Being able to recognize the logical components involved in the process of hypothesis testing.
6. Being able to determine logically possible combinations given a set of constraints.
7. Being able to recognize argument structure and being able to use appropriate concepts such as premise, conclusion, and intermediate conclusion to identify the parts.
8. Being able to distinguish a successful paraphrase of an idea from an unsuccessful paraphrase that does not say the same thing as the original statement.
9. Being able to identify an essential unstated premise or conclusion of an argument
10. Being able to evaluate how strongly a particular set of premises supports a specific conclusion, with special attention to the distinction between deductive validity and weaker connections
11. Being able to evaluate the degree of relevance of particular pieces of evidence to determining the truth or falsity of a conclusion
12. Being able to evaluate the degree of relevance of particular criticisms to the validity or invalidity of an argument

The authors talk about their experience in a working paper submitted for publication in a pedagogy journal:  “There are two points to note about our list.  First, (items) 1 and 2 are examples of the quantitatively oriented items that our (Business School) colleagues were interested in adding… Some of the items, for example those testing quantitative skills and some basics of deductive inferences, were relatively easy to write.  By contrast, as might have been expected, constructing the multiple-rating items consumed most of our time and energy.” 

Multiple rating items provide the test taker with a prompt, followed by a series of statements relative to the prompt that must be labeled by the students.  The labels for each prompt are different but for some we might use labels such as “strengthens the argument,” “weakens the argument,” or “is irrelevant to the argument.”  For example:

At the November city council meeting, the chief of police of Smithville reported a 50% reduction in the number of burglaries committed in the warehouse district of the town over the previous year.  He said this was clearly due to the fact that he had doubled the number of patrols on foot and by car since the previous November.

Statements expressing criticisms of the chief’s report to be evaluated and labeled:

1.                  Mayor Cynthia Johnson claimed that the increased patrols were breaking the city’s budget.  (Answer:  irrelevant.  Even if true, it is not relevant to the chief’s conclusion.)

2.                  Councilman Rogers questioned the report by saying that the largest of the four warehouses had closed.  This meant there was one less warehouse to steal from, so naturally the number of burglaries was down.  (Answer:  Relevant.  If the closed warehouse was responsible for 50% of the burglaries the previous year that would explain the decline all by itself.)

The test was piloted with 95 students in a required course in Spring 2005.  After evaluating the results, and drawing conclusions about student motivation and ability to complete the test, the TACTS test was administered to 300 students the following semester.  In order to motivate the students to do their best, extra credit was awarded based on their performance.

There was a wide variance in the students’ performance on the test items.  On some questions, for example an item about possible versus impossible combinations, 80% of the students had the correct answer, whereas on others, for example a probability question, less than 20% of the students answered correctly.  This provides useful feedback to the faculty in terms of what needs reinforcement in the curriculum.

Plans for the future include subjecting the instrument to rigorous reliability and validity testing; developing an online version of the test; and conducting an item analysis to see whether some items are tapping into the same thinking skills.

The participating faculty are proud of their work.  “We have worked very hard to develop this instrument which we believe accurately measures the critical thinking skills of our students.  We also trust that our students were seriously engaged in the completion of the instrument which gives us confidence in our results.  Changes to the curriculum have already been made to address some of the weaknesses identified in this assessment process.  This cross-discipline collaboration has been one of the most challenging and rewarding of my academic career,” said Valerie Muehsam who is in charge of assessment for the College.  R. Dean Lewis, Dean of the College of Business Administration, states “it is gratifying and rewarding to see this interdisciplinary group of faculty members with the assistance of a doctoral student recognize an almost universal assessment problem and develop an outcomes measure for the complex area of critical thinking.  I applaud and commend Dr. Valerie Muehsam and Dr. John Miller of the College of Business Administration and Dr. Frank Fair and doctoral student Ms. Wendy McCoy of the College of Humanities and Social Sciences for developing a measurement instrument that could assist universities worldwide.”

 The College of Business Administration at Sam Houston State successfully completed their maintenance review in Fall 2005.