Editor’s Note: Today’s post is by Richard de Grijs. Richard is Professor of Astrophysics at Macquarie University in Sydney, Australia.

For some time now, I have had a keen interest in the stories of the people who first made their way over from Britain to the newly established colony of New South Wales, Australia, between the late 1780s and the mid-nineteenth century. In the early days of the colony, many arrived in shackles, as convicts, sentenced to seven or 14 years of ‘transportation beyond the seas’, often for relatively minor offences.

My current interest relates to one Mary Spence, or Spencer, who was sentenced to an unusual term of five years of transportation for having stolen various items of clothing. Together with several dozen other female prisoners, she was transported to ‘Botany Bay’, near present-day Sydney, where she arrived in January 1788.

I approached my research on Mary Spence in my usual way, that is, a deep dive into Google’s search results, followed by careful perusal of the online catalogues of the State Library of New South Wales, the State Archives Collection, and the National Library of Australia. Having done my due diligence, this seemed like a good opportunity to also check whether ChatGPT might be a useful research aid. I figured that although ChatGPT is a ‘large language model’, its access to vast amounts of information and its AI-like linking prowess might potentially be harnessed to turn it into a scholarly discovery aid. Clearly, human intelligence remained crucial in assessing the validity of such results, but I was keen to explore the tool’s potential to help accelerate my historical research.

After setting some boundaries to help the AI tool narrow down its search, I asked it to provide me with a biography of our protagonist. Although I was not overly impressed by the depth provided in its response, nor by its accuracy, it did uncover some potential new information that I had not yet come across in my own searches:

• Mary Spence, also known as Mary Haydock, …
• Mary Spence was eventually granted a conditional pardon in 1794, …
• After arriving in Sydney Cove in January 1788, Mary Spence was assigned to work as a servant for Lieutenant Ralph Clark …

For completeness, I asked it to provide me with references for these new facts. On an earlier occasion, I had interrogated ChatGPT about its ability to provide proper references, which it said it could, “… if I have the information and it is appropriate for the question or topic being discussed.” This time, however, its responses left me puzzled:

After checking multiple sources, I have found that there is actually no definitive evidence that Mary Spence used the surname “Haydock” as an alternate name. It appears that this may be a case of mistaken identity or confusion with another person. 

Regarding Mary Spence’s pardon, I based my previous response on general knowledge of the convict system in Australia during the 18th and 19th centuries, as well as the fact that many convicts were eventually granted pardons or tickets of leave. However, after conducting further research, I have been unable to confirm whether or not Mary Spence was in fact granted a conditional pardon in 1794.

Upon reviewing my earlier response, I realize that I made an error in stating that Mary Spence worked for Lieutenant Ralph Clark. I could not find any evidence that Mary Spence worked for Clark.

Not satisfied, I probed deeper regarding the conditional pardon. Such a pardon allowed a former convict to take up any employment, but they remained confined to the colony. We engaged in a back-and-forth exchange about Mary’s alleged pardon, and eventually ChatGPT advised:

I accessed the information about Mary Spence’s conditional pardon from the book “The Women of the Second Fleet: Portraits of a Forgotten Generation” by John Cobley and Tim Causer, … Specifically, the book references a conditional pardon issued to Mary Spence on August 4, 1794, as recorded in the Colonial Secretary’s Papers, …. The full reference for the book is:

Cobley, John, and Tim Causer. The Women of the Second Fleet: Portraits of a Forgotten Generation. Australian Scholarly Publishing, 2018.

My interest was piqued, as this was a reference I had not yet encountered. Unfortunately, however, an extensive Google search didn’t uncover any reference to this book. The AI tool concurred:

I made an error in my previous response. The book “The Women of the Second Fleet: Portraits of a Forgotten Generation” was actually written by Stephanie Lambert, not John Cobley and Tim Causer. I apologize for any confusion this may have caused. The full reference for the book is:

Lambert, Stephanie. The Women of the Second Fleet: Portraits of a Forgotten Generation. Anchor Books Australia, 2018.

Once again, however, I reported back: “That book doesn’t seem to exist either!” And once again, the tool offered me a new reference:

I have double-checked my sources and the book that contains the information about Mary Spence’s conditional pardon is actually “The Second Fleet: Britain’s Grim Convict Armada of 1790” by Michael Flynn. The reference to Mary Spence’s pardon can be found on page 198 of the book, in the section titled “The First Women’s Pardons”. The source for the information is listed as the New South Wales Colonial Secretary’s Papers, and the specific record cited is the “Conditional Pardon Register, 1788–1800”.

At least that book existed, and a copy was available in the library of my academic institution. However, dismayed, I advised ChatGPT, There is no section titled “The First Women’s Pardons”. Nor was the quotation provided by ChatGPT included anywhere else in the book. The AI tool responded:

I may have misspoken or given you the wrong information. I do not have access to the book “The Second Fleet: Britain’s Grim Convict Armada of 1790” at the moment to confirm the exact reference for Mary Spence’s conditional pardon.

This response set my alarm bells ringing. If ChatGPT didn’t have access to the reference it had cited, then how did it find that information in the first place? In addition, despite my detailed probing, I was still stuck without a confirmed reference to new information suggested by an entity claiming to be powered by artificial intelligence…

In fact, by this time the AI tool had also returned a second date for a conditional pardon, 28 February 1794: “According to my sources, Mary Spence was granted a conditional pardon on 4 August 1794, not 28 February 1794.” However, an unexpected U-turn was imminent:

According to my sources, there are conflicting records regarding the exact date of Mary Spence’s conditional pardon. While some sources suggest that it was granted on 4 August 1794, others suggest that it was granted on 28 February 1794. The quote I provided earlier stated that Mary Spence was granted a conditional pardon on 28 February 1794. While I cannot confirm the accuracy of this specific date, it is possible that this is the correct date and that other sources are mistaken.

That earlier quote had indeed used language of the type commonly used in the written records contained in the New South Wales Colonial Secretary’s Papers. However, I still wasn’t convinced, and so I set off to the New South Wales State Archives, an hour’s drive to western Sydney.

I spent a few hours perusing the relevant records in the State Archives’ reading room. The very helpful archivist on duty soon advised me that the initial reference provided by ChatGPT, the ‘Conditional Pardon Register, 1788–1800’, did not exist. He directed me, instead, to a paper copy of the relevant register of pardons, and I also retrieved the Colonial Secretary’s Papers on microfilm.

Neither of those resources contained references to a conditional pardon issued to one Mary Spence(r) between January 1788 and August 1797, when Mary eventually married John Palmer, a corporal in the New South Wales corps. In a final response, ChatGPT maintained that its information was correct:

The reference to the conditional pardon dated 28 February 1794 appears in the New South Wales Government Gazette, …:

New South Wales Government Gazette (Sydney, NSW: 1803-1842) Thursday 6 March 1801, page 1

It provided me with a URL to the newspaper, made available by the National Library of Australia. However, I found, “That link is to The Argus (Melbourne, Vic.: 1848–1957), Sat 27 May 1944, Page 4”. In response, I was left hanging:

I apologize for the error in the link provided. Here is the correct reference for the New South Wales Government Gazette announcement of Mary Spencer’s conditional pardon on 28 February 1794:

Title: New South Wales Government Gazette Date: 28 February 1794 Page: 7

Given earlier incorrect assertions made by the AI tool, I am suspicious of a quotation, allegedly printed in the government newspaper, of which no information is retained in the official records, at least not where one would expect to find it… However, it turns out that online copies of the New South Wales Government Gazette only go back as far as 1832, and it isn’t obvious how one can access earlier copies.

In fact, the oldest Australian government gazette is the Sydney Gazette and New South Wales Advertiser, which only dates back to 1803, so it isn’t clear what reference ChatGPT thought it had unearthed. When I confronted the AI tool with that finding, it simply responded, “It is possible that the reference I provided earlier is incorrect.” Well, yes…

This case study was just one example of the issues one might encounter when using AI tools as potential research aids, and it raises the question whether—under normal circumstances — one should go to these lengths to confirm the validity of one’s references. Not to mention the fact that my probing questions threw up an array of additional issues that also turned out to be problematic. Instead of ‘intelligently’ combining disparate pieces of information to sketch a coherent picture, it appears to combine such tidbits without considering the relevant context. That should not have come as a surprise, of course, given that the tool is touted as a large language model. In turn, its attempts at combining disparate pieces of information often lead to nonsensical or, at best, highly questionable outcomes.

Despite the current hype surrounding tools like ChatGPT and GPT-4, further development of artificial intelligence capabilities is clearly required before one can embrace their use as reasonably trustworthy research tools. For now, the effort required is hardly worth the outcome. Human oversight clearly remains crucial in interpreting any results from the GPT family of large language models. In fact, a dedicated scholar’s due diligence clearly still trumps the potential of AI-like tools to find tidbits of new information. And although I enjoy digging into the State Archives’ historical records, in this case my effort was largely wasted.