In May 2023, I led a research team including about a dozen students, along with several professors and administrators from across campus to launch a survey of all students at the Colorado School of Mines. The goal was to understand how Generative AI had been adopted across engineering education. The survey was taken by over 600 students–a representative sample of the student population. And the results have been very influential!
First of all, what did we learn?
For the full story, take a look at the scientific papers listed below. But in a nutshell, here are the use cases reported by students who were using GenAI in 2023 at Mines:
Yes, there were a few students who appeared to be using Generative AI to cheat (see above, about 2-3% who “generate complete answers”). But the vast majority of students told us a much more nuanced story. Most students say they are using GenAI to support their learning, writing, and coding rather than to replace critical thinking. In particular, one major use case of GenAI is for “conceptual exploration” to dig deeper into concepts only touched upon in class or to learn about core course materials from a different perspective.
Of course, there are also major issues with GenAI. Even if people feel like it’s improving their learning or benefiting their field, that may or may not be true in actuality, esp. given the prevalence of hallucinations in GenAI outputs. (We’d need a different type of study design to generate conclusive evidence of whether GenAI is befitting or harming people and society.)
What the survey in May 2023 gave us was empirical evidence of how students self-report their GenAI usage—which is a better baseline than professors’ and administrators assumptions about what is going on. Ultimately, we did this work in order to ensure that students’ voices and concerns could be understood and made known while the school was in the process of figuring out how to respond to the new public emergence of these tools. (There’s lot more information about the results in the papers listed below.)
So then, how did we actually use the data?
Guidelines at Mines
- Results were presented to a Mines-wide committee (including students, professors, and staff) and discussed extensively in order to influence how we wrote guidelines on GenAI, syllabus language, and educational resources.
- Learn more about how Academic Affairs has formalized Mines’ approach to GenAI here: https://www.mines.edu/academic-affairs/genai/
- Learn more about resources from the Trefny Center here: https://trefnycenter.mines.edu/effective-teaching-and-generative-ai/
Our research team has published two papers on the results
- One paper focused on a subset of the data, only from the Computer Science department. This paper was published at the ITiCSE Conference and presented in Milan, Italy by Kylee Shiekh in July 2024. You can read the paper here: https://dl.acm.org/doi/10.1145/3649217.3653575
- We also published a short workshop paper, using the full set of data from all departments, at the Human-Robot Interaction (HRI) conference in March 2024: https://estellesmithphd.com/wp-content/uploads/2024/03/hri2024-llm-workshop.pdf
Workshops and Guest Lectures
- I have given several workshops and guest lectures in order to share the results back with students and faculty, including for many individual classes at Mines across multiple departments, including Computer Science, Engineering Design & Society, Applied Math & Statistics, and even Petroleum Engineering, as well as events such as the GRADS Symposium, or even ethics-related events nearby at CU Boulder.
So, what are we up to now?
In Fall 2024, we launched a new and improved survey. Similarly to the last survey, a team of students and professors will be analyzing the results and sharing those with the community. We plan to share the results with Academic Affairs, the Trefny Center, and Department Heads across Mines, in order to ensure that engineering educators are sensitized to the needs and values of students. Moreover, a team of faculty and staff (including myself) are participating in the AAC&U Institute on AI, Pedagogy, and the Curriculum (https://www.aacu.org/event/institute-ai-pedagogy-curriculum), with the goal of helping to formulate Mines’ Action Plan moving forward for integrating AI over the next few years. The survey results will be useful for helping to inform and steer our plans.
…And of course, we will also continue publishing the results at top tier academic venues, to share this knowledge with the world!
What’s different about the survey this time?
We’re asking some new questions about students’ motivations for pursuing higher education and for using LLMs specifically. We’re also asking about students’ ethical and pragmatic concerns about the impacts of GenAI on themselves, their academic fields, and society at large. It will be interesting to find out where the opinion needle lands!
Possibility of Broader Deployment
I’m in conversations with other ITiCSE folks about the possibility of applying for a Working Group on this topic, with the goal of multiple institutions and locations deploying their own local version of this survey, in order to gain a more global & uniform “pulse” on the GenAI situation in CS or other disciplines.
Please get in touch if you’re interested in using the survey at your institution, as well! 🙂
~Prof. Estelle Smith
C. Estelle Smith, Ph.D. 