Peter Facione – The educational power of measuring Critical Thinking



another thing that people are always concerned about when they ask about critical thinking is whether it is a skill or a set of attitudes or dispositions or habits of mind and my answer is neither critical thinking uses skills but it's a process critical thinking draws on attitudes and those attitudes impel us to apply those skills but critical thinking is the process now a lot of a lot of you are fully aware that when we explain things to each other we human beings we generally fall back on three fairly paradigmatic approaches and this is important to us because if you're going to measure students critical thinking it's important to understand how students do in fact think and how they explain things to one another one of the approaches that we enjoy using is empirical reasoning you're familiar with that you understand how the sciences work we gather data we have formed a model we have a hypothesis we use the data to see if we can ideally disconfirm the hypothesis and grow our understanding by always not only falsifying but expanding our knowledge you're aware that we also enjoy many explanations in our lives that are ideological paradigmatic of ideological reasoning might be our explanations to one another of our political points of view or our religious beliefs where we start from principles and reason downward to their applications and the one that I think that all of us enjoy the most because it's in fact frankly the most interesting and the most fun is when we use comparative reasoning as we do in our literature and our movies and in our explanations to one another when we help a young person understand think something by saying you know it's like this or we help a colleague understand something by saying your experience is so similar to my experiences that I had in this other situation and we use models and metaphors now each of these has pluses and minuses which I've not at the moment going to go into but it's important to know that we use all three of these constantly almost every day every academic discipline uses them but some disciplines stress one more than the other okay so now we're now we're at the of the manor why should we test for critical thinking cousin so much of life it's important that we measure what we value and that to understand that we will get what we measure I want you to take a moment and to reflect on things that we measure that many people thought were not measurable just a hundred years ago hair quality the effectiveness of medications the degree of the capacity of a military group to saturate and convert the minds and hearts of an indigenous population the number of atoms in a molecule these were all important things and what I have always wondered and I've been wondering for more than 40 years is why teeth we teachers don't measure what we value with as much precision and care as our colleagues in other fields why do we as teachers feel reluctant to apply valid objective measurement when it is such an empowering tool I want to share with you some of that power so you can understand what I'm talking about the californian critical thinking skills test as I suggested was made public in 1990 just to give you a sense of how this particular test works it's probably a very familiar kind of test for you the students are given a number of scenarios the scenarios supply the content knowledge and the context and this is the scenario usually ends with a question and then the students are given a number of options to consider and their task is to reason to the best choice from the options offered some of the questions asked the students to interpret some asked them to analyze some asked them to evaluate some asked them to draw inferences so ask them to explain in every case a student will do well if their metacognitive and mindful and that they try to understand what the question is they're being asked rather than jumping in impulsively and just picking one of the answers there are many versions of this test because there are many contexts that people like to use it in especially at the college level and of course there are at levels of this test for elementary school and middle school in high school as well what did the data tell us I want to share with you a research project that we did at a large University and we wanted to find out at that University whether or not courses in critical thinking at the college level produced any measurable gains it certainly was an open question the professors who taught those courses truly believed that they did but so did all the professors who didn't teach those courses believed that their courses also produced measurable gains and students critical thinking we surveyed the faculty we hardly found a single person than the entire University who didn't believe they that they were not also teaching for critical thinking we want to know whether students grew and critical thinking if they didn't take a critical thinking course I mean shouldn't college naturally improve our critical thinking we want to know what demographic factors correlated with gains and critical thinking skills did men do better than women did older students do better than younger students did professors who have more years of experience do better than professors who were newer did full-time faculty do better than part-time faculty we had many many questions like that in our minds and finally we wanted to know if there was a relationship between the critical thinking scores of the students and the grades that professors gave their students we were quite amazed at the findings we used a pretest post-test design we also used the design that had a post-test only we had an experimental group which for the students who took the critical thinking courses a control group students who had not taken any courses and we even had one department volunteer to let us use a second control group which for students who are taking a particular introductory course in this case philosophy we found that the students who took the critical thinking courses gained in critical thinking skills and we're very pleased about that as were the teachers of those courses and you might ask yourself well what subject fields were they well as it turned out there were several different subject fields economics education psychology English philosophy several different disciplines in every case students gained in critical thinking we didn't control for the pedagogy the teachers used we did not control for the textbooks the teachers used we did not control for anything else about the course except how carefully we measured at the beginning at the end whether there were gains and critical thinking skills well that was good because the university had a critical thinking requirement and you wouldn't want students to take a course that was intended to serve that requirement but wouldn't which didn't produce the desired result what about the students who did not take critical thinking courses did they gain in critical thinking no they did not that was a fairly big shock we expected that they were going to going to gain we looked at freshmen and seniors everybody along the way there was absolutely no gain whatsoever by simply going to college I was rather disappointing I hoped that if we replicated that research today 25 years later we'd get a different result we found that men and women gained equally we found that age was not a factor we found that it didn't make a difference if the professor was full-time or part-time or male or female relative to the sex of the student there's when I think a lot of the things that we thought might make a difference actually didn't we did discover that if a professor taught the same critical thinking course too frequently the gains tailed off I think the professor's were getting too tired of repeating the same subject over and over again this was remarkable research and we wondered whether or not we were the only ones who found this and we discovered many years later in a marvelous meta study by Philippa Brahmi and his colleagues this published in the review of educational research that in fact our findings were consistent with the findings of a hundred other studies and it made it very very clear that at the college level and I believe this all levels of education improvements in students critical thinking skills and dispositions cannot be a matter of trusting to mere luck or implicit expectation that educators must take very active steps to make critical thinking reflective problem-solving a clear objective in their courses and in their exams and in their own training you know the act of learning strategies my colleagues here at the conference are talking about these in different vocabularies but but I see a lot of commonality many of my colleagues have mentioned that asking good questions that trigger critical thinking skills as an important pedagogy asking students to analyze to evaluate to infer is a much better approach than simply asking them what do you think or what do you feel assignments that support positive habits of mind which I'll talk about in a moment important part of the strategy may be most important examinations that the teachers develop the genuinely task the students to get engaged in well reasoned and reflective responses an important part we get what we measure and how we as teachers and professors behave if we behave with respect to our students as we behave as if we are mindful and caring to seek the truth and analytical and systematic in our approach if we behaved with a sensitivity toward using our skills to make reflective judgments about our subject matter and about our students they will see that in us and they will model their behavior after us so let's look at some data and I understand that some of you would have no trouble reading a slide like this and others of you might and so I'm just going to try to illustrate some of the key points this is a slide which represents the skills scores on the California critical thinking skills test of 3195 students it turns out these are college students and they're all at the same University on this particular test the National percentile for two-year college students these are students in their first and second years of college the National percentile is a score of 21 and on this particular test these students on average scored a 54% I'll one of the things that educators like about these instruments is they offer choices of national percentiles or international percentiles that educators can use as external benchmarks so this is a typical group of students but what is more interesting than that and then that the group as a group is typical is how different the students are within the group notice that there are some students in the far right who have exceptionally strong critical thinking skills they have scores on this test which I know to be higher than the scores that some of their professors would get and there are students on the far left who have scores which are frankly abysmally poor and you if that student on that day might have had some sort of medical problem or who knows what how did they become a college student this certainly if that's a true score this student is going to struggle in every course they take and of course the vast majority of students are in the middle middle ground there is a percentile on a test like this called a first quartile represents the twenty-fifth percentile 1/4 of the people on any given test are there there's a third percentile and there's a lovely group of the middle the middle 50% and many times I'm asked by teachers which group of students should I address my attention to the top 25% the bottom 25% or the middle 50% of course the answer is all and a good teacher will find ways to develop exercises and assignments it'll reach out to all groups but if you aren't able because of time or other constraints then my suggestion would be to focus on that middle 50% let's look at the habits of mind and look at some data on these again this instrument was developed a number of years ago is translated into a number of languages and it focuses on a set of characteristics or attributes or habits of mind this list of seven came from doing what's called a factor analysis we didn't know when we started if it was going to be seven or six or five or eight it turned out to be seven and we tried to find English names that capture the idea that is the cluster of items that relate to one of these factors each of these factors but I want to tell you more importantly is that each of the factors which is a positive one is paralleled by a factor which is a negative one so for example truth-seeking which is the courageous desire to follow reasons and evidence wherever they might lead it has a negative pole called intellectual dishonesty open-mindedness has a negative pole called intolerance and you probably have met people who are on various sides of this set of polls there are people who are analytical and always trying to be foresight 'full what happens next there are people who are heedless of consequences and do things that you might say to yourself why would they ever choose to do that that's going to cause so much trouble didn't they anticipate that there are people who are systemic and people who are disorganized people who are confident that reason will help them solve problems and people who really distrust reason they actually will tell you I don't want to think about it because that's not the way I want to solve my problems I would rather follow an authority figure or do what I've always done or make a decision even by flipping a coin because I don't trust reasoning as individuals or in groups there are people who are indifferent and people who are curious and there are people who are imprudent and are people who are judicious and wise so how do students fall out on this college students well we'll look only at truth-seeking because we don't have time to look much beyond that truth a truth thinking is the courageous desire to seek the best possible knowledge in any given circumstances and it's the inclination that you've probably seen in yourself and in many others the inclination to ask the hard questions to follow the reasons and the evidence wherever they might lead even if they go against something that might be your cherished belief this is a very difficult attribute for adolescents that's a very difficult attribute period truth seekers would not agree if they were asked do you agree or disagree they would not agree that everyone always argues from self-interest they would not agree that if there are four reasons in favor and one against they would go with the four just because it's four they would certainly not agree that many questions are too frightening to ask but some people certainly would agree with that truth seekers would not be the kinds of folks who would say anything to get someone to agree with them and certainly truth-seekers would not be the kinds of scholars who looked only for facts that already agreed with their preconceptions here's the look at a group of 17 and 18 year olds college freshmen at a university in the United States and this is their scores on that one aspect of the critical thinking habit of mind called truth seeking and as you can see it's a bell-shaped curve and you can see that by the coloring used in the slide on the green side to the right those students had a strong disposition towards truth seeking they were actually quite eager to be in college and they were going to try to follow reasons and evidence as best they could although their their actual skills might need more work they at least had the inclination on the desire and in this group of people there were a number of students college students mind you paying high tuition that were opposed to truth seeking they did not thank you what to ask hard questions they were not psychologically prepared to follow reasons and evidence wherever they would lead these students are going to be a challenge for their professors and there were many students in the middle many students between the two vertical bars in the area that we would characterize as indifferent and maybe a college education would make a difference to them so we looked at these same students four years later each individual captured that individual and matched it up with that individuals post-test four years later we wanted to know if there were changes we knew there were going to be changes the skills unless there was some course in the critical thinking skills but would the experience of being in college changed their truth-seeking which is a different kind of thing it's a habit of mind and we discovered in fact there were positive changes some students move from hostel at least to indifferent some people move maybe many people move from indifferent to positive that was very good news we the faculty at that college were very excited about that this was affirmation they're liberal education undergraduate curriculum was working some people very very few but some actually expand all the way from negative to positive that was great 40% almost of the students moved in a positive direction unfortunately 11% in facts our facts 11% moved the other direction they went from positive to indifferent or worse in the four years of college they went indifference in negative and I would have to say that in those cases perhaps we as a faculty at that college failed those students we didn't give them a value an attitude a desire a habit of mind that we very much as a faculty treasured so now let's think about these two things together along the bottom imagine that we could put all the skills scores poor skills moderate skills strong skills and up to side imagine that we could have a visualization of students who went from being negative toward critical thinking to all the way to students who are positive and strong about it so think about the students who would have strong skills and a strong desire to apply those skills they would be wonderful students you may be one of those kinds of students one of those kinds of teachers and faculty members eager to learn courageous in your questioning systematic organized and very skilled at digging out the knowledge and expanding your repertoire of understandings and applications almost as if for a teacher we could get out of the because those students will learn and all we need to do that is guide them a little bit maybe hire them to assist us then there are students with strong dispositions and moderate skills this is so good we can develop those skills the skills are the are the easiest things to develop the characteristics in the habits of mind are more enduring and more difficult to develop we can teach the skills all kinds of research shows that then there are the students who really are sort of ambivalent hmm perhaps even a little bit on the negative side dispositional II and they have moderate skills these are the challenges these students need to be not only taught the skills and they can be taught the skills they need to be motivated to use those skills encouraged to use those skills shown how critical thinking it can give them an advantage in the marketplace in the educational environment in their practical life that they live and then we find the students that are the most challenging to us the students who do not thank you care to engage problems using thinking they'd much prefer other means and they don't have very good skills either and here we have as teachers our biggest opportunities and also our biggest challenges now tomorrow I can tell you how an employer would look at the same array of data and which one's an employer would hire and which one's an employer would not give an interview to but you can probably figure that out for yourself most recently our measurement has taken us in the direction of numeracy we find that a number of science departments and math departments are interested in critical thinking but of course want contexts that are more numerical and then in our complex global society numeracy skills are incredibly important these days in terms of measuring what matters the company inside assessment to which I am very grateful for helping sponsor my visit here is helping over 40 students worldwide to gather data for their dissertations in all sorts of fields as you can see including business in the military healthcare and stem programs a number of funded researchers using NSF and NIH funding in which is a very competitive funding in the United States are also using these tools as are a number of faculty members and businesses who are using it for internal sorts of purposes everything from admissions to program evaluations well you know I've talked a little bit about what we should do as teachers and I know other colleagues have I thought maybe I would present a list of things we should avoid we could help it and I think you probably could make this list as well as I all the things we could do wrong as teachers and we should try to avoid relying on rote memorization treating opinions as facts you know the list assuming that any group of people is incapable of thinking we could not be more wrong if we make that false assumption or attacking those who disagree if you'd like to see these in action watch the American political parties engage in the next election because I'm sure the candidates will commit every one of these crimes toward each other it's not that much fun for an educator to watch but gives you wonderful examples to use in the classroom but let's get back on the positive side of the equation critical thinking is like a locomotive we have found in research projects other colleagues have done that once students learn the dispositions they endure throughout the period of their time in college and into their professional careers even if they only spent one semester in the very beginning talking about in learning the dispositions we have found in a project with the United States Air Force that once the students are exposed to and learn the skills these endure beyond the training program a year or more in the actual deployment and use of those skills by professionals it is critical thinking is a powerful locomotive you cannot stop critical thinking once you teach someone how to do it the only thing about this analogy that fails is that the locomotive is on a track so you know where the locomotive is going to go and critical thinking is not necessarily on a track you cannot predict where the mind of the student the mind of a group of people will go once you empower them to engage in critical thinking as a process that cannot be stopped no topic is forbidden no belief is beyond question and rightfully although I'm not among them some people are very threatened by students who engage in critical thinking or professors or teachers who teach critical thinking they're very threatened there's a picture of some of them Boko Haram Isis and other groups like that want to put an end to Western education because they know that at the very core of Western education is this fundamental capacity to address problems and solve problems reflectively that once learned cannot be unlearned and cannot be stopped to learn more about critical thinking and and how it is pervasive and how it connects with so much that we do in our civic life I urge you to download for free the English or the Spanish or the Chinese or whatever version of this article critical thinking what it isn't why it counts it's available on many websites including the inside assessment website and I want to reiterate my conviction based on 40 years of research and many many consulting activities with businesses with the military with health care and with education that critical thinking is a first and foremost a key part of sensible democracy for it enables us to make group decisions together well it is also what enables us to engage in deep and disciplined learning it's essentially practical everyday problem-solving but in a reflective and thoughtful way and it is certainly for everything I've ever seen the key predictor and measure a professional success it can be defined we saw the consensus consensus definition it can be learned great deal the research shows that your own experience teaches you that it can be taught effectively and you know the methods and most importantly it can be measured objectively with great precision and validity and reliability allowing you a lot of variance and a lot of data that can be used to help individual students and groups of students there are a number of free resources at the the organization that sponsored my visit er insight assessment urge you to go to their website tap the resources tab at that website there's a about 50 different things there that you might find helpful to you is also a free app that the company makes available and I want to close with both the invitation to ask me questions after my talk by coming up here and also with a little bit of a paraphrase of something that might be mindful of you a little bit of Paul lion scripture critical thinking is skeptical without being cynical there's open-minded without being wishy-washy is analytical without being nitpicky critical thinking can be decisive without being stubborn evaluative without being judgmental and forceful without being opinionated thank you very much again if any of you want to come down and have questions that would be fine Peter thank you very much for the presentation you've been working in this area for a long time and there seems to be a lot of data supporting how good critical thinking is in your opinion why is it so difficult to get critical opinion into the world or the realms of politics and business there are the people who have succeeded and they don't know the source of their expertise and I think this group that in various stages of their careers did use critical thinking and did reflect on their experiences and their expertise is born of pattern recognition and an ever broader sense of explicit of experiences with ever refined nuances of difference between them so let me give you as that example the expert diagnostician the expert business person the leader who can recognize the pattern in a situation and respond to it quickly but it's built on layers and layers and layers of thoughtful and reflective experience so I do I do believe there's critical thinking there and their typical mistake is to not recognize that the given situation is actually slightly different than they think it is they get the diagnosis wrong then there is a group who is afraid of critical thinking who literally cannot get past their ideological mindset this is what I was taught to believe this is what my parents told me my church told me my whatever and I live by that dogma and if you raise a question with me about that I will attack you or try to humiliate you or try to destroy you because what's really happening is you're poking holes in my little mental glass house and I am terrified that all of my beliefs will fall apart and I will put there's one more group and there's the third group is probably a small group but we should mention them just to be fair there's a group of very cynical people and they run political campaigns and they don't believe that voters are thoughtful and so they use wonderful critical thinking to find ways to deceive us I don't admire that group but I have to give them their due because they have a problem and that is how to gain votes thank you for that wonderful question any other questions please yes thank you thank you for your presentation Peter you mentioned in your presentation that there's improvement in critical thinking skills by having a sort of a standalone critical thinking program another speak here at this conference explicitly contradicted that by saying that it didn't work and they had to be integrated and I just wanted maybe you could elaborate on that a little bit for me well my approach would be to use data and the a bra me meta study demonstrates that a critical thinking can be taught in standalone programs so for me to say it can't be done would be to fly in the face of the data but this is for me and both and not an either/or critical thinking gains can also be achieved in courses that are and programs that are part of the regular curriculum of a high school or college or grade school if the instructors engage the critical thinking skills support the critical thinking habits of mind and use assignments and examinations that require students to apply critical thinking in order to do well in those courses and we've seen data of those on that too so as to me it's it's a false dichotomy it mostly becomes a practical question of whether or not a teacher or a college or a school can afford to mount a separate program or wants to integrate it one last cut thought it it doesn't get integrated just because you say you're going to integrate it you really have to do something transformative with the pedagogy and the assessment to make it work and this will be another heresy I'll speak there's no discipline that has the market cornered I've seen success in every academic discipline and I've seen failure in every academic discipline there's nothing automatic about this way or that way it has to be intentional thank you I think a terrific question are you with a college yourself for the university I was last week at Clemson University and when they have a marvelous project and if you already put the university I would suggest you look at look at Clemson just to give you one more model of a highly successful university-wide approach to critical thinking Clemson University it's in South Carolina in the United States um first of all thank you very much for your intervention one of the common themes or current themes here in this conference is how to teach kids how to think how to think better and one of the tools they must obtain is self assessment you've been talking a lot about data you gathered about how critical thinking is infused it is obtained by the students and how it evolves over the years is there a way to teach kids in my case I teach high school kids is there a way to teach them how to self assess yes itical thinking abilities great question thank you we we offer self assessment tools for free at the insight assessment website one of them is the holistic critical thinking scoring rubric it's in several languages and I and others who have used that introduce it to students the very beginning of a course it's a rubric it can be applied easily to written material to group projects to conversations and students acquire the language of critical thinking and the capacity to begin to evaluate it you as a teacher would have to guide them helping them see this is very good this is very bad these are in the middle but once they have a tool like that developmentally it is a phenomenally powerful tool there are also available in various text books my own and others self self tests and things like this that that's you could score immediately and just look at an app that I mentioned from insight assessment but I think you're on the right path developmentally self-assessment from the very beginning is really the key because you're measuring it it's important to you you're helping them understand it they're going to try to give it to you reminds me of a coach teaching soccer or football let's let's do this let's do the drills and I'll let you know how you're doing and forward and have a better and better season from the very first day because I'll help you evaluate yourself and all the drills well thank you very much you've been very kind thank you you

Leave a Reply

Your email address will not be published. Required fields are marked *