Can students do their work with AI and still build the competence their courses are meant to build, in a way we can see? Bastani and colleagues (2025) show why the question is not trivial. In a field experiment with high school students, those with unrestricted GPT-4 access did better on practice problems and worse once the access was removed. A model can produce the work while the competence never forms.
Answering the question meant questioning four assumptions that student work rests on. What if knowing the material is no longer the point? What if one final grade is not the proof? What if feedback happened in the open? And what if students handed in no written report at all?
Each section below drops one of these assumptions, which generates a small list of proposals: some familiar although not widespread, some more outlandish.

1. The competence
When a model can supply the material, knowing the material counts for less. Three other abilities count for more.

1. Judging quality. What if the skill that counts is telling a strong idea from a weak one? Goodwin (1994) called it professional vision: the trained ability to see what matters in a scene. Part of it is knowing a field’s conventions, and models supply conventions well. Make students the editors. They pick from a mass of AI output what deserves the five minutes someone will give it.
2. Judging peers. What if your grade depended on how well you judge someone else’s work? Put students on the panel. They assess a peer’s project, and their grade depends on the quality of their judgement, from the questions they ask to the verdict they reach.
3. Directing AI across media. If working with AI belongs to professional competence, we should teach and certify it like any other part. Hand students the brief and the model. They build an app, a video or a simulation with AI, in whatever form the work needs.
2. The evidence
A grade judges a performance, and we have long treated a good performance as a sign of competence. AI breaks that coupling: a student can hand in a strong piece of work without being able to do it. One final grade on one submission cannot tell the two apart. The alternatives spread the evidence over time and over people, so the same competence has to show up more than once, in settings where the AI cannot simply do the work.

4. Certification in stages. What if no stage counted until the next one confirmed it? Make each stage earn the next. The work is certified in stages, each confirming the one before. A safer version already exists: final-year engineering projects assessed at successive milestones against shared descriptors, so the record of progress informs the final mark (Valderrama et al., 2009). The bolder version hands each stage to a peer who must build on it. If the peer can, the stage holds.
5. A later check. What if the grade stood, and a later course checked it? Let the next course test it. A later course opens with a short task that uses what the earlier course was meant to build, and the result goes into the student’s record as confirmation. A weak result triggers support for the student. It also tells the programme which courses build skills that last. Medical schools run a larger version called progress testing: all students sit a test pitched at graduation level several times a year. Growth and retention then show over time (Wrigley et al., 2012).
6. Witnesses to the work. What if the people who watched the work vouched for it? Collect testimony. Instructors and project partners write down the decisions they saw the student make. Medical education already builds high-stakes decisions from many low-stakes observations of this kind (van der Vleuten & Schuwirth, 2005).
3. The teaching
Feedback on student work usually happens in private, with an instructor commenting on a draft. That assumes the draft shows the student’s thinking. With AI, the draft may show the model’s, and comments on it reach the model’s choices more than the student’s. The AI use itself also happens in private, out of reach of any teaching. The alternatives move both into the open, where the student’s reasoning and their way of working with the model can be seen and corrected.

7. Open AI use. What if AI use stopped being a private matter? Bring the conversation into the room. Students use the model in class and talk about how they use it. Some instructors go further and have each class write its own AI policy together. Judson (2025) did this in sociology courses over three semesters and reports the policies the classes wrote and what enforcing them was like. A step further still, students critique a peer’s conversation with the AI the way they would critique a peer’s draft. Where did the student lead the model, and where did the model lead the student?
8. Live defence in small groups. What if a student had to defend their decisions out loud, every week? Make them argue it live. Each week, four or five students meet, and each defends one decision and the options they rejected. Design education has run on this format for decades as the studio critique, or “crit” (Goldschmidt et al., 2010), and interactive oral assessment, a close relative, has held up where AI could otherwise write the submission (Davey et al., 2025). Now and then, students can also annotate an AI draft of their own work and mark what is wrong, weak or merely conventional. Twice a semester, this trains the eye. Every week, it turns into surveillance.
9. A live system as the patient. What if students treated a live system the way medical students treat a patient? Shulman (2005) called the recurring forms through which a profession teaches judgement signature pedagogies, such as the case dialogue in law or bedside teaching in medicine. Business schools have the case method, and IS courses borrow it, but a case is a written account of the past. Students analyse it, and their decision stays hypothetical. I could not find a signature pedagogy of IS’s own. A candidate would put a live sociotechnical system on the examination table, meaning the people, organisation and technology that Sarker and colleagues (2019) place at the core of the field. Students diagnose it, intervene and see what happens, and the people who run it give feedback in rounds. That puts it closer to a clinical rotation than to a case discussion.
4. The artefact
For a long time, the written report told us what students had learned. Drop it, and three alternatives come up.

10. A working system. What if the final hand-in were a system that has to change in front of you? Change the brief on stage. Students hand in a working system, such as an application or a prototype. During the presentation, the instructor reveals a new requirement, say a changed privacy rule, and the student adapts the system on the spot, with AI if they like. The test is how well they specify the change: which data, roles and process steps it touches. Among 100 students building software by prompting, computer science knowledge predicted success even after controlling for general cognitive skills (Thorgeirsson et al., 2026). Whether that carries over to sociotechnical systems is my extrapolation, and worth testing.
11. Teaching someone else what they learned. What if the grade depended on what someone else can do afterwards? Turn students into teachers. Master’s students run a seminar session for an undergraduate course, or a workshop for the partner organisation’s staff. The student’s grade then depends on what the learners can do afterwards, checked with a short task at the end of the session. Explaining something to another person quickly exposes what you have not understood yourself. In an experiment by Fiorella and Mayer (2013), actually teaching produced better long-term learning than only preparing to teach.
12. Research done live. What if the day’s best result were a better question? Reward the sharper question. Students get a question and a dataset and work through both in a day, with AI in the room and instructors dropping in at checkpoints. The format tests something only if the finding goes beyond what the model would answer. In its strongest version, students are graded on whether they end the day with a better question than they began with. I found nobody assessing that.
What keeps it academic
However, several of these ideas put students on a partner’s live system, with feedback from practice. That setup can drift into an internship, where the student does useful work for the partner and applies what is already known. Academic work has to add something that holds beyond the case.
Hevner’s (2007) three cycles of design science help here. Relevance comes from the application environment, which is the partner. Rigor comes from the knowledge base of theories, methods and prior artefacts, and the university owns that part. The research-teaching nexus literature has long argued that students can take part in research themselves (Elsen et al., 2009). Without a rigor cycle, each of these formats is an internship with a presentation attached.
How I got here
I used C-K design theory (Hatchuel & Weil, 2009), which works with two spaces. Knowledge is what we know. Concepts are ideas that our knowledge cannot yet call true or false. You start from a concept that current knowledge cannot decide, here “students work with AI and their competence still forms”, and split it step by step. Some splits follow known solutions; others go where knowledge does not reach yet. The second kind are the interesting ones, and each ends in a research question. The four sections above are the first splits, and the orange squares mark where knowledge ran out.
The four diagrams show a small part of what came out. The whole tree has 75 ideas across seven branches, including the ones we ruled out or set aside.
I ran the process with AI as a referee. The model checked each idea against the literature and told me when I was restating something already known. Models are good at the known, which is why I gave it that job. Design fixation research adds a warning: examples pull designers toward themselves (Jansson & Smith, 1991), and a model produces examples on demand. While I made the choices, most of the novel options still came from the model’s suggestions. Keeping the new ideas on the human side is harder than it sounds.
A research agenda
Every branch in the tree ends either at something we know or at a question nobody has answered. The dashed boxes in the tree are those questions. Together they make a research agenda for teaching information systems with AI, and most of it can run in ordinary courses.
| Section | Question | Kind | Closest findings |
|---|---|---|---|
| 1. Competence | Can students learn to judge the quality of ideas without years of learning conventions? | Study | First-year students judging by comparison, without criteria, agreed with experts (Jones & Alcock, 2014); comparing examples drew more feedback on content (Bouwer et al., 2018) |
| 2. Evidence | When a grade depends on a peer using your work, does it measure you or the peer? | Study | Milestone assessment builds a record of progress (Valderrama et al., 2009); nothing found on handing work to a peer |
| 3. Teaching | Does a live defence show when a student lacks the skill, even if AI did much of the work? | Study | Interactive orals held up against AI misuse (Davey et al., 2025); orals need several examiners (Wass et al., 2003); no study tests detection directly |
| Does critiquing peers’ AI conversations change how students use AI? | Pilot | Classes writing their own AI policy (Judson, 2025); more confidence in AI went with less critical thinking (Lee et al., 2025) | |
| 4. Artefact | Does domain knowledge predict prompting success in sociotechnical systems, as it does in programming? | Study | CS knowledge predicted vibe-coding performance (Thorgeirsson et al., 2026) |
| Can examiners agree that a reformulated question is better, and not merely different? | Study | None found | |
| Does the model reformulate questions better than students? | Study | None found | |
| Does problem finding mark research competence outside art? | Literature | Problem finding in art students (Getzels & Csikszentmihalyi, 1976) |
The most important question is whether talking through decisions live shows us when a student lacks the skill, even if AI did much of the work. Almost every idea above depends on the answer. To test it, students would complete a piece of work while we record how much they used AI. Examiners who have not seen that record would then question each student about their decisions. Afterwards, we compare the examiners’ verdicts with a separate test of the same skill. If the two agree, live questioning works. Oral exams are only reliable with more than one examiner (Wass et al., 2003), so each student would face at least two.
The second question is whether students can learn to judge the quality of ideas without first spending years on a field’s conventions. Two groups would rank the same set of ideas, one seeing them packaged by AI in the field’s conventional form and one seeing raw notes. Both get short training in comparative judgement with expert feedback, and their rankings are compared with experts’ before and after.
Three questions sit under research done live. Can examiners agree that a reformulated question is better, and not merely different? Several examiners would rate the same reformulations blind. Does the model reformulate questions better than students do? The same briefs would go to both, rated blind; if the model wins, the idea fails. And does problem finding, which Getzels and Csikszentmihalyi (1976) studied in art students, also mark research competence? That one starts in the literature.
A fifth question tests certification in stages. When a grade depends on how well a peer can use your work, does it measure you or the peer? Handing the same work to several receivers would tell: if the grade varies with the receiver, it measures the receiver.
The last is a pilot that any course could run. Does critiquing a peer’s AI conversations change how students use the model themselves? Collect each student’s conversation in week one and week six, and code both for passive versus collaborative use and for how often the student checks the model’s output.
Several ideas in the tree are still open and have not been split further. They include adapting a working system to a requirement revealed live, design science as the field’s signature pedagogy, students as examiners, and assessing only what a student adds beyond what the model produces on the same brief.
References
Bastani, H., Bastani, O., Sungu, A., Ge, H., Kabakcı, Ö., & Mariman, R. (2025). Generative AI without guardrails can harm learning: Evidence from high school mathematics. Proceedings of the National Academy of Sciences, 122(26), e2422633122. https://doi.org/10.1073/pnas.2422633122
Bouwer, R., Lesterhuis, M., Bonne, P., & De Maeyer, S. (2018). Applying criteria to examples or learning by comparison: Effects on students’ evaluative judgment and performance in writing. Frontiers in Education, 3, 86. https://doi.org/10.3389/feduc.2018.00086
Davey, et al. (2025). Utilising one-on-one interactive oral assessments as the major final assessment within a bioscience course. Assessment & Evaluation in Higher Education. https://doi.org/10.1080/02602938.2025.2502577
Elsen, M., Visser-Wijnveen, G. J., van der Rijst, R. M., & van Driel, J. H. (2009). How to strengthen the connection between research and teaching in undergraduate university education. Higher Education Quarterly, 63(1), 64–85. https://doi.org/10.1111/j.1468-2273.2008.00411.x
Fiorella, L., & Mayer, R. E. (2013). The relative benefits of learning by teaching and teaching expectancy. Contemporary Educational Psychology, 38(4), 281–288. https://doi.org/10.1016/j.cedpsych.2013.06.001
Getzels, J. W., & Csikszentmihalyi, M. (1976). The creative vision: A longitudinal study of problem finding in art. Wiley.
Goldschmidt, G., Hochman, H., & Dafni, I. (2010). The design studio “crit”: Teacher-student communication. AI EDAM, 24(3), 285–302.
Goodwin, C. (1994). Professional vision. American Anthropologist, 96(3), 606–633. https://doi.org/10.1525/aa.1994.96.3.02a00100
Hatchuel, A., & Weil, B. (2009). C-K design theory: An advanced formulation. Research in Engineering Design, 19(4), 181–192. https://doi.org/10.1007/s00163-008-0043-4
Hevner, A. R. (2007). A three cycle view of design science research. Scandinavian Journal of Information Systems, 19(2), 87–92. https://aisel.aisnet.org/sjis/vol19/iss2/4
Jansson, D. G., & Smith, S. M. (1991). Design fixation. Design Studies, 12(1), 3–11. https://doi.org/10.1016/0142-694X(91)90003-F
Jones, I., & Alcock, L. (2014). Peer assessment without assessment criteria. Studies in Higher Education, 39(10), 1774–1787. https://doi.org/10.1080/03075079.2013.821974
Judson, H. (2025). “But I didn’t use ChatGPT!”: Democratic course design and generative artificial intelligence in higher education landscape. Teaching Sociology. https://doi.org/10.1177/0092055X251342530
Lee, H.-P., Sarkar, A., Tankelevitch, L., Drosos, I., Rintel, S., Banks, R., & Wilson, N. (2025). The impact of generative AI on critical thinking: Self-reported reductions in cognitive effort and confidence effects from a survey of knowledge workers. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3706598.3713778
Sarker, S., Chatterjee, S., Xiao, X., & Elbanna, A. (2019). The sociotechnical axis of cohesion for the IS discipline: Its historical legacy and its continued relevance. MIS Quarterly, 43(3), 695–719. https://doi.org/10.25300/MISQ/2019/13747
Shulman, L. S. (2005). Signature pedagogies in the professions. Daedalus, 134(3), 52–59. https://doi.org/10.1162/0011526054622015
Thorgeirsson, S., et al. (2026). Computer science achievement and writing skills predict vibe coding proficiency. In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3772318.3791666
Valderrama, E., Rullán, M., Sánchez, F., Pons, J., Mans, C., Giné, F., Jiménez, L., & Peig, E. (2009). Guidelines for the final year project assessment in engineering. In Proceedings of the 39th IEEE Frontiers in Education Conference. https://doi.org/10.1109/FIE.2009.5350767
van der Vleuten, C. P. M., & Schuwirth, L. W. T. (2005). Assessing professional competence: From methods to programmes. Medical Education, 39(3), 309–317. https://doi.org/10.1111/j.1365-2929.2005.02094.x
Wass, V., Wakeford, R., Neighbour, R., & van der Vleuten, C. (2003). Achieving acceptable reliability in oral examinations: An analysis of the Royal College of General Practitioners membership examination’s oral component. Medical Education, 37(2), 126–131. https://doi.org/10.1046/j.1365-2923.2003.01417.x
Wrigley, W., van der Vleuten, C. P. M., Freeman, A., & Muijtjens, A. (2012). A systemic framework for the progress test: Strengths, constraints and issues: AMEE Guide No. 71. Medical Teacher, 34(9), 683–697. https://doi.org/10.3109/0142159X.2012.704437

