The advent of ChatGPT and the underlying technology opens the opportunity for those responsible for Friends and Family analysis to take a huge step forward and solve one of the initiative’s greatest issues - the lack of resource to read patient feedback.
The Problem
Almost 100 million pieces of Friends and Family feedback have been collected by the NHS, and the number is growing by approximately 1 million each month, creating a vast treasure trove of unstructured text feedback.
The official guidance is that the free text should be analysed, but there is no suggestion on how to do this with existing resources. Clearly there are limited internal and external resources available to read it all and even fewer trained individuals to truly understand how best to respond or react appropriately. So unfortunately, much of this vital information goes to waste. There is also limited ability to interrogate the text collected over time. Given the time and effort expended on collecting the data this seems to be a great shame.
A Potential Solution
At Logan Tod & Co we have been supporting local authorities and organisations to identify vulnerable individuals using test analytics and sentiment analysis. We believe that using a Large Language Model (LLM) like ChatGPT or BARD can solve the challenge in the NHS by enabling the analysis of their large volumes of unstructured feedback.
We demonstrate below how ChatGPT can be used to analyse one patient feedback. We are using ChatGPT for this walk through but in a production environment you would use the underlying API and GPT4 to process large volumes of data.
Feedback Sample
For the purposes of this worked example we chose this piece of feedback as it is complex and mixed in sentiment - in other words a tough nut to crack! This feedback was published on the NHS website and is anonymous.
The patient wrote:
“Went to A& E after a fall. Signed in at reception to be send straight down to Medoc without seeing anybody. After a 3 hour wait was told by a very nice doctor that I shouldn't have been sent there as I needed an x-ray which A & E should have sorted. The Medocc doctor sent me for an x-ray as I think he felt sorry for me after waiting for 3 hours. He talked to someone in A & E about the x-ray and confirmed I had fractured a bone in my wrist and if I went back to A & E and explained that he had spoken to a doctor I just needed to have a splint fitted and a follow up appointment with the fracture clinic. I went back to A & E and explained everything the doctor had said but the person on the reception said that I would have to wait my turn. 2 hours later I then saw a nurse who took my blood pressure, temperature and pulse and then tells me that I had a fracture in my wrist which I already knew and for her then to say I would have to go into the next department to have a splint fitted. Fast forward another 2 hours and I was called just for another person to tell me that I had a fracture and then put a splint on my hand. It took a total of 7 hours and in that time spending probably 15 minutes seeing people to confirm I had a fracture and have a splint fitted. If A & E had dealt with me when I first come in instead of trying to make their figures look good by palming people off to the wrong departments I would probably in and out in a couple of hours. I would just like to say that the doctors and staff that saw me were excellent it's just the way the hospital is run. Everything is about numbers and not the welfare of the patients.”
So how would you automate the reading and analysis of this text? We share below the steps we have taken.
Step 1: Set up topic prompts
Using ChatGPT to analyse patient feedback works best if you pre-define the areas you want to understand, it creates a repeatable structure that makes further analysis easy. To do this we create “Topic prompts” that give the LLM a starting point for analysis. They are easy to write as shown in Table 1.
For this example project we set up 10 topic prompts:
- Quality of care and treatment
- Access to healthcare services
- Communication and coordination
- Facilities and cleanliness
- Patient outcomes and recovery
- Cost and efficiency of services
- Positive experience and gratitude
- Active recommendation
- Diversity and cultural competence
- Special services
You can set up as many topic prompts as you require and make them specific to your circumstances.
Step 2: Issue the command
Copy the feedback and topic prompts into ChatGPT along with the instructions about what you want it to do. In this case the following simple instruction was used:
“Analyse the feedback provided using the topic prompts provided
Output MUST be in a table structured as follows:
- Column 1: The Topic Name
- Column 2: Matching text from the feedback that makes you think this topic applies.
- Column 3: A count of the matching sentences
- Column 4: The sentiment score for the topic (5 is positive, and 1 is negative)
Summarise the feedback in less than 20 words and then provide an overall sentiment score for the text (5 is positive,and 1 is negative)”
Step 3: Review the output
The output for Columns 1 to 3 from the tool are listed in Table 2.
The remaining output from Column 4 are included below.
Overall sentiment score: 2 (negative)
Summary: The patient had to wait for three hours to see a doctor, who then informed them that they should have been sent to the x-ray department. The patient then had to wait for a long time to receive treatment and was frustrated with the hospital's focus on numbers rather than patient welfare. However, they were appreciative of the doctors and staff who treated them.
As you can see ChatGPT has, in our view, done a good job of identifying which topics apply to this piece of feedback, it has correctly identified the sentiment and it has provided a very useful summary that is much easier to read than the original feedback.
In repeated testing we have found that the results are consistently good and highly reliable. In comparison to how the free text is currently analysed, this is a major step forward.
Step 4: Ask For More Insights
We asked for even more insights.
“What improvements would you recommend to the hospital?”
ChatGPT responded with:
“Based on the feedback provided, I would recommend the following improvements to the hospital:
1. Improve communication and coordination between departments to ensure that patients receive appropriate and timely care.
2. Review the triage process to ensure that patients are directed to the correct department for their needs.
3. Increase staffing levels, particularly in areas where patients are experiencing long wait times.
4. Focus on patient-centred care, with a greater emphasis on patient welfare rather than meeting targets or numbers.
5. Conduct regular reviews of the hospital's processes and procedures to identify areas for improvement and implement changes as necessary.
By making these improvements, the hospital can provide a better experience for patients, improve patient outcomes, and increase patient satisfaction.”
Not a bad answer! This type of insight can be generated for a single item of feedback, or it could look across the last 25,000 words of feedback before it draws a conclusion.
Step 5: Automate
This example shows the power of an LLM to analyse a single patient feedback and the good news is this can be scaled into a complete solution capable of handling all the feedback an organisation generates in a secure and confidential manner.
The OpenAI API is available to developers and is very friendly and easy to use, and when linked to a suitable dashboard can be used to super-charge Friends & Family analysis.
Summary
Overall we have been very impressed with the capability of ChatGPT and believe it will power a revolution in analysis of patient feedback. We would encourage you to experiment with ChatGPT, or any other LLM, but do be aware of the privacy implications when using the free services.
If you would like to discuss a solution for your organisation, if you have another data set, or if you would like to talk about the implications, please get in touch with us.