Edit : Turns out that you’ll get the most out of ChatGPT if you learn how prompting works.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
You might have heard a lot about AI recently and how it is set to disrupt a number of industries. For example, making human copywriters obsolete. I find it hard to believe that the human mind can be successfully replaced by a machine overnight but augmentation – that’s an extremely powerful concept.
So, how can our industry harness the benefits of AI? With this research question in mind, we ran a number of thought-provoking experiments. Our friend was the “GPT for Sheets and Docs” extension – full instructions here. Essentially, if you use Google Sheets (the Google equivalent of Microsoft Excel), it opens a new and powerful cell function in the form of =GPT().
The first thing to note is that using AI is not free. Well, not entirely. While you can start testing the service at zero cost, there’s a limit to the generosity of the free trial. Charges are incurred for the number of queries you use and the complexity of them as this translates to computing grunt. That said, it isn’t expensive – the queries in this investigation cost us $0.04.
The first question was a no-brainer for us. We asked =GPT(“what is wrong with public consultation?”)
Public consultation can be a useful tool for gathering feedback and opinions from the public, but it can also be problematic. It can be difficult to ensure that all voices are heard, and that the consultation process is fair and unbiased. Additionally, public consultation can be time-consuming and expensive, and it can be difficult to ensure that the results are representative of the population as a whole. Finally, public consultation can be subject to manipulation and bias, which can lead to skewed results.
We don’t disagree!
So, next we got a little more serious. The second question we asked related to stakeholder identification and used the query =GPT(“who is against fracking?”). Four seconds later we got: –
Many environmental groups are against fracking, including the Sierra Club, Greenpeace, Earthjustice, and the Natural Resources Defence Council. Other opponents of fracking include local and state governments, health organizations, and some religious groups.
In an attempt to get something more useful, we then asked =GPT(“who might be interested in hospital closures?”)
This resulted in a list of potential stakeholders – reasonably verbose but not very specific and not at all localised (we can’t think of any “insurers” who might want to respond to a similar consultation in the UK!).
Local government officials
Healthcare provider
Patients
Community organizations
Healthcare advocacy groups
Healthcare researchers
Insurance companies
Business owners
Economic development organizations
Public health organizations
Unimpressed, we turned our attention to using AI directly on stakeholder feedback. This time we gave ChatGPT a list of names and asked it to identify the origins of the person. Specifically, =GPT(“here is a person’s name, guess their ethnicity”)
In the absence of equalities data, it’s quite common for organisations to try and look at data patterns. Believe it or not, this was a problem I was asked to solve whilst working at the University Central Admissions Service (UCAS) back in the 1980’s!
The results were more interesting – some of the guesses were good but workable answers were only provided 50% of the time. It’s a bit of a daft question really – just because you have a Scottish name doesn’t mean you’re Scottish! It may have been more interesting to look at the correlation with religion.
Fraser Henderson Scottish David Jones It is impossible to guess the ethnicity of David Jones without more information. Rehan Ahmed Rehan Ahmed is likely of South Asian descent. Quntin Oliver It is impossible to guess someone’s ethnicity based on their name alone. Rhion Jones It is impossible to guess the ethnicity of a person based on their name alone. Guiseppi Conte Italian NB. The accuracy of the above is improved if you start to provide examples in the prompt.
We decided to try something else, something tricker. This time we took three public comments (as per below) from an online local plan consultation to run through the AI (https://eastdevonlocalplan.commonplace.is/contributions/proposal/supporting-jobs-and-the-economy)
- “This permits intensification of current operations with some constraints but doesn’t consider impact on residents and traffic. Workshops using heavy machinery generate noise and cause distress to nearby residents.”
- “Definitions are absent. Net Zero – does this mean on site or as part of the total visitor journey? (Exeter and Honiton are probably the most sustainable parts of EDDC but probably not what is meant by this policy). Improvements in public transport to access some areas would be required. How would it be achieved?”
- “Thank you. Everything must be done to ensure communities are never left high and dry by a village shop closing. Already our narrow roads have too many Tesco delivery vans hurtling around…we don’t want more !”
The first question we asked was about sentiment with the simple query =GPT(“what is the sentiment of this feedback?”)
The answers were good (particularly for the second contribution):-
1=Negative
2=This feedback appears to be neutral, as it is simply asking questions and making observations.
3. Positive
It left us wondering how deep the sentiment analysis might go. Could you determine if somebody was excited or angry? An important application for AI is the ability to act immediately on feedback in a system which is choked with data. For example, detecting if what somebody has written contains any swear words and is breaking the rules or, more seriously, if the writer needs urgent help. Extending these ideas further, AI could be used to remove references to personal information from feedback or perhaps even identify patterns such as near duplicates and fake engagement. This isn’t as simple as it sounds – for example, “Sue Ryder” is a name but it is also a brand.
We continued the experiment by asking a question about the origins of the three submissions made above. This time we asked =GPT(“what sort of person would write this?”). The AI answers were fascinating:-
1=A policymaker or government official
2= A person interested in sustainability and environmental issues, likely with a background in urban planning or public policy, would write this.
3=This could have been written by a concerned citizen who is worried about the impact of large corporations on their local community. They are advocating for the importance of local businesses and are concerned about the environmental impact of increased delivery vans.
Although we can’t verify if any of the above is true, this type of analysis could start to unravel a hidden story about “who” is contributing and their intentions in the absence of asking a direct question. It’s very specific and that’s impressive.
Unfortunately, guessing more demographic data was not possible. We tried =GPT(“how old do you think the person is who wrote this?”) but got the answer “It is difficult to determine the age of the person who wrote this without more context”. We also tried to identify the gender of a person who had authored feedback but this didn’t work either.
Next, we took some individual feedback to see if AI could deconstruct it further. This could help speed-up a second layer of human analysis. The feedback from the same consultation was as follows:-
“Again, thank you. Many visitors cannot understand why we have ‘allowed’ so many large caravan sites on our coast and elsewhere. I have to explain that these were ‘pre planning’ and would never be allowed now. Please ensure no more, and wherever they are improved through use of darker shade units and landscaping. and places for tents must be protected for their affordability for families, and low impact on the landscape.”
We asked the AI a simple query =GPT(“what are the key points in this sentence?”) and think that the answer (below) was petty succinct.
1=Thanking the reader.
2=Explaining why large caravan sites are allowed.
3.=Ensuring no more large caravan sites are allowed.
4.=Improving existing caravan sites with darker shade units and landscaping.
5=Protecting places for tents for affordability and low impact on the landscape.
As it happens, ChatGPT is really good at “cleaning up” data. For example, you can use it to extract the names of people from a list of email addresses.
In order to try something really bold, we asked for some opinion. Many readers’ will remember the Channel 4 consultation – hugely unpopular and extremely longwinded. So, we asked GPT(“should channel 4 be privatised?”). The answer was probably picked up from somewhere else on the web. It responded with the populist answer –
“No, channel 4 should not be privatised. Channel 4 is a public service broadcaster that provides a range of programming that is not available on other channels. It is funded by advertising and is a valuable source of independent and diverse programming. Privatising Channel 4 would mean that it would be subject to the same market forces as other commercial broadcasters, which could lead to a reduction in the quality and range of programming available.”
It begs the question, would ChatGPT have said this prior to the consultation outcome being published?
Although our experiment was short, we were overwhelmingly impressed by the potential of AI to cut through unstructured data. At the very least, it’s a tool which could help speed-up the process of sifting large amounts of feedback.
We’d love to initiate a small research project that looks deeper at using ChatGPT to augment the synthesis of views for a live consultation or engagement exercise. If you’re curious about sponsoring this, please get in touch by email at contact@acep.org.uk . Otherwise, go experiment!
[The featured image for this article was courtesy of the AI art generation tool DALL-E-2]