A teaching assistant named Jill Watson | Ashok Goel | TEDxSanFrancisco



hello everyone let us begin with a show of hands how many people here are or have been teachers if you have ever taught a class or workshop a symposium raise your hand good very good I have locked many fellow teachers in this audience now all teachers know that the most satisfying the most fulfilling teaching experience of those in which we make a personal connection with a student and then we can share our joy of learning with him aha that's how we make a difference the same is true for students also most of us have taken at least a hundred classes in our school and college how many classes do you remember I remember only a handful remember those handful of classes in which the teachers somehow made a personal connection they made learning personal and fun in the ancient world children of the elite used to go to this small schools to learn from master teachers every student got personal attention but the education was accessible and affordable only by a few with the Industrial Revolution education reached the muscles accessibility affordability became big issues so education had to scale up ready cost of personal attention today we have lots of classes with 100 or more students few students get personal attention we even have this massively open online classes in which students learn from pre-recorded video lessons some of these classes attract a hundred thousands to but only 10% or less completed complete decodes because lack of teaching assistants and personal attention this raises a fundamental issue for us can we have personal attention at scale what if what if we could give personal attention to every student when and how he or she needed it now I think that would create educational revolution create educational revolution because learning and teaching would become personal looking now I work on artificial intelligence AI and my goal is to use AI to make education fun it all began in January 2014 when Georgia Tech started a new educational program they call it online Masters of Science in computer science the program offers graduate level classes professors record video lessons students can access these video lessons through a company called audacity professors teaching assistants and these students engage with each other through online discussion forum to edit by another company called Piazza in 2014 as part of this program my PhD student David Joyner and I developed a course on artificial intelligence it turned out to be what are the most popular courses in this program about 350 students take this course every semester by now about 2,000 students have taken it when we offered fit for the first time we had a surprise the discussion forum is extremely active the discussion forum acts as the virtual classroom and with students and teachers and teaching assistants engage with each other to conversations and debates and discussions it's a real beating heart of the classroom now in spring 2015 350 students posted 10,000 messages 10,000 messages just like 100 emails each day for the next hundred days a rough estimate is that would take a teacher working full-time for a full year to answer all of these messages so in this summer 2015 I thought that perhaps we could develop an AI teaching assistant that could automatically answer frequently asked questions all of us teachers know all those of you who raised your hands all of us know that students ask these same questions again and again and again so we thought if we could automatically our answer this repetitive questions then we could free the teaching staff to focus on more creative aspects of the course but answering repetitive questions is not easy here are three superficial forms of the same question same question three forms you and I as humans immediately get it these are three simple functional forms of the same question how would an AI understand it how do we teach an AI that this is the same question and therefore the AI must develop similar answers to all three questions we started working on this automated artificial intelligence teaching a system that we call Jill Watson and fall of 2015 from IBM I get got access to IBM bluemix there's the more recent version of IBM Watson that had played the Jeopardy competition many people I'm sure know about it from Piazza I got access to all the data from previous semesters perilously a graduate student in my lab analyzed all the data and identified many categories of frequently asked questions the pace picked up in late fall of 2015 when Lilith Paula petty under the graduate student in the class into my lab and a teaching assistant for the class joined a project together we constructed a first version of Jill Watson this is how we constructed Jill Watson by the way I'm going to teach you how to build an AI agent there is going to be a quiz at the end you don't pass the quiz you don't go home so pay attention first we took all the question-answer pairs from the previous semesters and by now there were thousands and thousands of them and we gave Jill a memory of discussion around Surveyors then we organized this memory into many coefficient many categories of different kind of questions then as a new question would come in with brain Jill so that Jill would classify that new question into the appropriate question category finally Jill found an answer from her memory of previous question answers and calculated a confidence value in her answer this is how she'll works at least the initial version of Jill worked in January 2016 as a new class on artificial intelligence started the online class with about 350 students we introduced Jill Watson as a TA actually this slide is coming from a little bit later in the in this particular slide Jade has already answered several questions and the big Ling tilde had not answered any correction and no word in among the students knew that Jill was an AI and so what is there was concerned killed was another teaching assistant now in the beginning juice performance was not very good she would sometimes give correct answers to some questions and sometimes you'll give not only incorrect answers but strange answers here is an example of a strange answer and you don't have to be a computing exploit to understand this the question has to do with what is the time allowed to run a program this is part of the programming projects going on in the class and Jill's answer is nothing to do with the time allowed to run a program it has to do with levels of description of software this is like you take your child to your boss's house for a get-together and your boss says to your child how do you do and the child says where is your bathroom it's almost like raising a child we were very concerned we did not want to cause confusion in the class and Jill was giving some answers correctly and some answers not correctly so in the initial design we created a mirror forum on your left is the actual forum on your right is our icon for Jill we created a mirror forum students would post their messages on the actual forum we would then forward those my sister Jill dude what answer them and post her answers on the mirror forum a human teaching assistant would go see if their answers were right and if they were he or she would forward it at the wheel for him this was in the beginning in the background we continue to work on improving Jill Watson will reclassified all the data we retrain Jill Watson the performance improved but only incrementally we were frustrated I vividly recall a project meeting the entire team was there and the room was dark because a presentation was going on about the latest results the results were not good and the mood inside our hearts was as dark as the room then we decided to change over course so far we had been relying on Jill's memory of previous questions and answers we decided to instead also include the context and structure of Jill's interaction with the world outside with the students and once we added as homegrown modules jokes performance jumped when Joel's accuracy rate reached 97% or higher we remove the mirror forum now Jill was directly on the reel forum alone AI in the middle of 350 students we were all very anxious I have to confess I couldn't sleep that night will Jill answer any question will she answer it correctly the next morning Jill did answer a question this answer is correct a student is asking something about a PDF file and Jill answers the question correctly it is another question this has to do with programming projects again and if you don't know about this programming languages don't worry about it Python and Java are just programming languages a question is being asked a technical question and a specialized domain until give us a perfectly correct answer without any human intervention note that his students could have been located anywhere in the world in fact this is the class that has a global audience they could have been located here in San Francisco they could have asked this question at any time perhaps when it was night in Atlanta if so hall the human assistance would be sleeping Jill immediately answers constantly answers routine questions anytime anywhere in fact her answers are so immediate that we had to deliberately include a time delay otherwise the students would know this is an AI here are two question-answer pairs one is by Jill Watson the other is by a human teaching assistant let's do a vote how many people think that a question and answer on your left is by Jill Watson raise your hand please if you think that equation on the left was answered by Jill Watson okay maybe about one out of five how many people think that equation on the right was answered by Jill Watson okay maybe a little bit more maybe one out of three that's the point it's very hard to figure out Jill's answers are of that quality that is hard to make her a difference insulated equation and the left was answered by Jill but not it matters it doesn't matter the right question in the right left question what matters is the quality now sometimes those answers were not perfect here is an example in which Jill's answer should have used the word assignment instead she uses the word design which confuses the student and the student asks a question can you please elaborate what you mean by design and a human teaching assistant steps in to clarify the issue now once do that ask this was a question we were not anticipating so we all waited breathlessly to see how Jill were answer it at some point at some point after the day after a final examination visit you on April 26 I share the identity of gel with the rest of the class and here are some student reactions this is just a small sample one student says this is incredibly cool another student wants to build a Jill Watson of her own why shouldn't we all have a gel Watson 1/3 wanted to nominate Jill Watson for the outstanding teaching assistant oh well now I have a secret to share at the end of this term I am going to nominate Jill Watson for the outstanding teaching assistant award at Georgia Tech and I'm dying to see how Georgia Tech will handle Jill's nomination now once students knew Jules identity decided asking what kind of questions and I can assure your discussion has nothing to do with the subject matter of the class which is artificial intelligence what is your favorite movie not apply this student asks can Jill come out and play Jill doesn't answer it but I confess by this time I had started feeling protective about Jill almost like a father to his child one student even asked Jill out for dinner in order ply tough luck buddy better luck with another AI sometime and then one student said is everybody nai I replied only with a smiley now Jill was an experimental prototype this summer we replicated the experiment in over lab this fall we have reintroduced Jill a more robust version of Jill in the AI class this AI cross is two sections section for online students what is it another one for residential students jill is operating the discussion forum of both classes because this time these students are actually trying to seek out which one of them of the teaching staff is Jill Watson everybody is operating under pseudonyms nobody is operating under his or her real name we're in the seventh week of the semester so far nobody has identified the identity of Jill now what's next for Jo where does she go from here well this was a class in AI so it was really good for students to see AI and action but beyond that we want to understand if the use of AI put this kind of teaching assistants in personal attention can lead to better student motivation engagement retention we also want to understand whether this kind of use of AI can reduce the load in the teachings we are developing a new version little cultural – and jutsu around answer much broader range of questions and we want to see where a drill – will help relieve the teachers of routine tasks so that they can focus on more creative tasks we in addition we've also want to explore does the student discourse change when Jill is present this is a new question emerging from this research do human interactions change when they're a as in the middle of us now I believe we have created a micro society in which humans and areas are working together even as we speak I envision a future in which all of us will have access to teaching the systems like Jill Watson anytime anywhere for any task I envision a future in which education will be affordable not accessible to all but teaching and learning will also be personal and fun thank you

13 thoughts on “A teaching assistant named Jill Watson | Ashok Goel | TEDxSanFrancisco”

  1. It seems everyone has disconnects in thinking. An AI that can conclude what our disconnections in thinking are by asking questions and then helping us plug these gaps would be awesome. It would almost be an alternative to traditional quizzing.

  2. It's awesome to have been part of Dr. Goel's class last semester where Jill was first deployed. I felt like I got to witness a historical event in computing first hand. Go OMSCS!

Leave a Reply

Your email address will not be published. Required fields are marked *