The kind of freeform larps that I write can be complicated beasts. Numerous characters, complicated plots, items, abilities, secrets, contingencies. And a timeline that can easily be confused. Even an independent proofreader can miss things.
So when it was suggested that an AI might be able to help check a freeform larp, I thought I’d give it a go.
TL;DR – remarkably effective
The Highgate Club and ChatGPT
I recently re-ran The Highgate Club, a fourteen-player freeform that I co-wrote at Peaky in 2010. This was the third time I had run it, and you would have thought that by now, all the kinks had been ironed out of it.
Certainly, in terms of playing the game, it ran very well (read about it here). There were a few kinks (minor ones) that I needed to address. So I decided to use The Highgate Club as a test case for AI, because I knew the game well and I didn’t think there was much to find. That’s what I thought…
I used ChatGPT. I have a paid account which lets me upload files into a folder so that ChatGPT can analyse them.
So that’s what I did. I uploaded the files as MS Word documents.
(For clarity, at no point in this process did ChatGPT change the game materials – I made the edits myself. And ChatGPT doesn’t use the uploaded files for training purposes; at least, that’s what it says.)
The prompts
My first prompt: Read the entire LARP and review it as a professional editor. Produce a prioritised list of issues but don't rewrite anything yet. Classify each issue as Critical, Major or Minor, explain why it's a problem, and suggest a possible solution.
As you can see, I didn’t even introduce The Highgate Club to ChatGPT… In a later prompt, I remembered to tell it that the larp was written in UK English. I will try to remember that in the initial prompt next time.
(Although this review ended up identifying all sorts of (mostly minor) issues with The Highgate Club, it wasn’t very structured. I plan to develop a more structured approach for future games.)
ChatGPT responded to my post by recommending a full editorial pass rather than a quick proofread, which ended up as follows:
Editorial review
ChatGPT completed an editorial review in four passes:
- Pass 1: Core structure, rules and cross-document consistency
- Pass 2: Character consistency & information flow
- Pass 3: Plot dependency & failure analysis
- Pass 4: Continuity audit
I’m not really sure whether all those passes were necessary, but they brought up lots of points to consider. Obviously, ChatGPT was extremely complimentary about everything else, which was a bit tiring…
Character review
Then we moved on to characters. ChatGPT reviewed every character as a separate pass. I didn’t tell ChatGPT which character to do next – I just let it choose. (In future I will probably tell it which ones to do so that I can track progress. I got a bit lost towards the end – if ChatGPT had omitted a character, I might have missed that.)
Hallucinations
About two-thirds of the way through the character reviews, I realised that ChatGPT was starting to hallucinate when it mentioned a character that didn’t exist.
So I pointed out its mistake, and ChatGPT replied that:
“After reading a large cast with multiple immortals, ancient civilisations, and overlapping mythology, I allowed my internal summary of the characters to drift instead of re-grounding myself in the source documents before writing the review.”
So it then re-read the character and revised its comments. At the end of that, it said.
“From now on, I'll re-read the relevant character sheet immediately before every audit, even if I think I remember it.” (ChatGPT’s emphasis.)
And then it immediately hallucinated again!
So in future, I will remind ChatGPT to re-read the character sheet before every character review.
The output
At the end of the review, I asked ChatGPT to summarise everything in a Word document. (It could also have created an issue log in Excel – I might try that later.)
It created a 15-page report containing 37 different findings which it ranked in order of criticality: Critical, Major, Minor. Each finding has an explanation and a suggested fix.
While I didn’t accept all of the points, I accepted the majority – and every single one has been a valid point worth considering. Examples of the kinds of things it picked up:
- Conflicting instructions
- A character had mistakenly received a handout
- Areas where we had not provided enough information – the sort of thing that we would handwave when running the game, but really should clarify
- Some items had mixed up item numbers
- Character backstory inconsistencies
- Information in the plot summary did not match the characters
- Conflicts in spellings and terminology – Holmes and Homes, for example
So all extremely useful.
One benefit was that ChatGPT doesn’t have my assumptions or biases – so it picks up things that I didn’t even realise were missing.
Other prompts
During the prompting, I also asked it to review the timeline in the game instructions to see if anything was missing. Sure enough, it highlighted a few things. I suspect it could probably assemble a timeline from scratch if I wanted it to.
I also asked ChatGPT: Imagine I've fixed all the consistency errors. Now imagine running this larp with experienced players. Predict the ten most likely problems that will arise during play, and explain why.
It came up with ten examples, all very interesting. I knew about two or three already, and having seen the game in action, I know that others haven’t arisen (and probably wouldn't). But I could see why it had identified each of the ten issues – definitely a good question to ask.
Plus I could ask ChatGPT to suggest other prompts.
Overall
Overall, I found ChatGPT an extremely useful tool in assessing The Highgate Club. You have to treat it with caution and watch for hallucinations, but it produced an extremely useful final report.
I suspect with a little practice (as I improve my prompts and learn how to use it fully), it will become even more effective.
And as for The Highgate Club, the files are now all updated.
