Adventures with CoPilot

By Tim Marsh

Like most people who teach undergraduates, my first serious encounter with generative AI was an adversarial one. When LLMs became freely available a couple of years ago, they irreversibly disrupted the assessment space, making it cheap and easy to subvert the written tasks we most relied on. I spent a good while in the defensive trenches, trying to AI-proof assessments and writing disclosure rules for the co-intelligent workflows we would permit. But that work brought with it a need to know the enemy, and the more I understood what AI was capable of, I started to see the potential upsides.

Why Copilot, of all things?

Taking stock of what alternatives we could recommend to students with very non-secure usage habits, I found that the most versatile of the AI tools bundled into our university licences is Microsoft’s Copilot Chat. It's included with the Microsoft365 suite every staff member and student already has and sits inside the university’s commercial data-security arrangements. Providing reference documents in your prompts is the key to pedagogically valuable use cases, and it isn’t quietly training itself on that intellectual property. If I’m putting an AI in front of students, we need to role-model responsible use.

The 8,000-character problem

Tucked away in the Copilot menus is the ability to build what Microsoft calls an agent, which is really just Copilot Chat with pre-baked instructions. You can change a few settings behind the scenes (internet searching, starting prompts), but for the instructions you get a tiny budget of exactly 8,000 characters. It’s a real struggle to craft anything useful that succinctly, but a clever user can get a surprising amount done with it, especially if the agent only has to do one or two things well.

My first attempt came early last year, when we overhauled the Final Exam in our BEHV1018 and I wanted students to be able to train themselves on the kinds of questions they would face. So I built a pair of exam tutors, one for the content MCQs and one for creating article summaries with deliberate errors for students to practice identifying. This is where I came to understand the value of specialised training-documents. You cannot cram a semester of content into 8,000 characters, but you can write instructions on how an agent should digest and apply a reference document that does contain all of the core content. When the substance lives in the training-document, the Agent can be tiny and well-specified.

The Parallel Tenants

One architectural wrinkle had to be solved first: our staff and student systems sit on two separate, non-communicating tenants, so any agents I built were invisible to the very students they were made for. ITDS (after much begging on my part) eventually set me up with a dummy student account, whose purpose is to let me clone my agents into the students’ digital ecosystem.

Conceiving Virtual Tim

The biggest breakthrough came when I brought all this into Motivation & Emotion, my second-year subject where I try out my most ambitious ideas. There I built an all-purpose agent to support student understanding of difficult material, through both alternate explanations and Socratic Tutoring: leading a student through escalating questions, probing the edges of what they grasp and explaining where they go wrong. These capacities both supported students struggling with the material, as well as helping more confident students test the limits of their understanding.

Because that subject is a bit of a cult-of-personality, I branded it ‘Virtual Tim’, a stand-in for all the student engagement I couldn’t do because I can’t be everywhere at once, and it was a genuine hit. Virtual Tim handled the Subject Outline I would otherwise endlessly repeat myself on. They fed it each lecture’s training-documents and had the material re-explained in fresh terms, translated into other languages, or Socratically probe their deepest insights. I even gave it role-play instructions so it could walk a student through demonstration scenarios (like an anger-management encounter). Win after win.

Generalising to vUWS-Bot

Colleagues were soon salivating at the possibilities. So this year I filed the serial numbers off Virtual Tim, stripped out the personal flourishes, and rebuilt it as an all-purpose version for any subject. I call it vUWS-Bot and replaced my face with the WSU Logo. It now runs, in one form or another, across several psychology subjects and a few other interested corners of the Faculty of Health.

What the students made of it

I have been polling BEHV1018 students this year, who have had vUWS-Bot scaffolded through the whole semester, and the picture is a clear three-way split. A little over a third engaged consistently with vUWS-Bot, some heavily and with great benefit. The stronger students use the Socratic mode to deepen what they already understand, while the strugglers get the gentler, tailored explanations they need to close the gap. About as good as responsive teaching gets.

A middle group had mixed results, almost always for the same reason: they opened the agent and simply talked to it, skipping the training-document. Without it, the agent falls back on its own general knowledge and gives bland, wishy-wash answers. They were an instructive little experiment in how the thing performs once you take its substance away (the answer being: no better than a generic free LLM).

And then there is a contingent, just shy of 15%, who will not touch AI with a ten-foot pole. I am wary of calling them Luddites, because a principled stance against the excesses of AI is a perfectly reasonable position, and I hope presence of vUWS-Bot in so many sections has not made the subject feel oppressive for them. But it is a group we need to accommodate, those that would rather do everything the hard way than engage an AI for any reason.

Important Takeaways

The lesson, in the end, is that the tools must be finessed into something useful. You can’t just point students at a chatbot, the utility comes from the curation of training-documents and painstakingly short instructions, and the omnipresent struggle of getting students to follow instructions. The agents are only as good as the substance you feed them.

Try it yourself*

*staff login required