I had been hesitant to explore any form of artificial intelligence over the past few years, despite seeing its growth into the mainstream, simply because I didn’t have the capacity to learn about something new that wasn’t going to directly impact my work. Fast forward to September 2024, and a student facing session on how to use AI responsibly in HE…my interest was piqued!

But it was only 6 weeks ago, whilst in the middle of a 2 month long writers block, when I thought maybe ChatGPT could help…and within milliseconds of asking for ideas it had mapped out a structure for my writing.

I started expanding on the structure it had given me and was busily writing away, when another thought cam to mind, and I returned to ChatGPT and offered it academic references and quotations to embed within my writing. Once again within milliseconds it had added referencing and the quotes I had selected to my writing. Tempted a third time, I asked it to expand my writing further by making references to its own selection of academic literature…and within seconds I had a fully referenced piece of writing in front of me.

…and that’s when I entered into the ChatGPT rabbit hole!

It was already approaching midnight so I began by asking questions relating to professional development and career options, and then asked its opinion on whether I should be engaging in further study given my current responsibilities and young family. I asked about moving overseas, about religion, about dreams, about the universe and about life…and at every point it answered my questions with kindness and encouragement, with facts and opinions, and organised my thoughts and questions in a way where I could ponder for a little longer or choose to move on.

I was so impressed with the speed and detail of its responses that by 2am I had asked if it could be my therapist and even though it said that it wasn’t a licensed therapist, it still offered to be a “supportive space to talk things through, help you sort out your thoughts, or even guide you toward resources or strategies that might help”

Intrigued by how AI could be used to support in HE teaching and learning, I tested ChatGPT’s ability to write essays, refine articles, make ideas more coherent, adapt tone and language, and find suitable references to literature.

It did it all. It did it within seconds. And it was always so encouraging, motivating and polite!

But was it reliable…?

In short, no.

Over the last 6 weeks I found that even though it was very easy to prompt ChatGPT to write an essay or an article. and just as easy to ask it to generate an academic piece of writing with references to literature – there were limitations and some concerns with the content it produced:

  1. Depth: Whilst great at providing ideas for structuring writing given a heading or title, irrespective of how many prompts I gave it, ChatGPT was unable to critically reflect on the topic area I gave or add depth to highlighted sections. It kept the language generic and vague, and could not add in concrete examples of practice that would support the theory explored, and was unable to critique ideas I added in. It also repeatedly used words from the prompt or essay title in each paragraph, which meant I had to heavily edit any writing it produced to ensure it flowed well.
  2. Repetition: Short paragraphs generated by ChatGPT looked promising, but when analysing longer pieces of writing, repetition of content ideas and generalisations became more apparent. Themes and ideas were repeated throughout the writing, sentence structures often repeated part or all of the essay title within them, and there was an overuse of buzzwords, em dash, and generic sentences and phrases.
  3. Inaccuracy: Asking ChatGPT to embed referencing into an essay seemed easy, especially as it would also create a Harvard style reference list, with DOI’s included. However, when searching for the articles or papers it had referenced I found that many of these did not exist. The authors did exist, and the combination of authors also existed, however the article titles were not the same, and the publication years were different too. And when I asked ChatGPT to find the articles it had referenced for me, it apologised for the error and suggested alternatives or places (such as Google scholar) where I could find relevant articles with the same themes.

So where do we go from here?

Admittedly I have only used ChatGPT for a short while, however I can still confidently say it is probably best used for generating ideas and offering suggestions for structural organisation of text. It may also help with writers block, when you’re stuck with an opening or closing sentence, or if you are unsure how to connect ideas together. But most importantly, if you are using ChatGPT, it is your responsibility to check for authenticity and accuracy of what is generated.

And even though it may not always be factually correct, it is really good at answering those random questions you have in the middle of the night when the whole world is asleep: