ChatGPT is good at reading customer feedback and telling you what it says. Paste in a few hundred comments and you get a readable summary in seconds. What you don’t get is a dataset. Nothing is stored, nothing is counted the same way twice, and there is nothing to click into or compare next quarter. For a first look at a small batch, use it. For anything you will report, act on or track, structure the feedback first.
Key takeaways
- ChatGPT reads. It does not store, count or remember, and a feedback programme runs on all three
- It was built to write, not to analyse data. It will give you a confident percentage with nothing behind it, and a quote nobody wrote
- Ask it the same question next month and you start from scratch
- A summary with no dataset behind it has nothing to click into, so you can never check it
- Your respondents’ words are customer data. A chat window is not where they belong
Friday afternoon. 500 open-text responses in a spreadsheet. Review on Monday.
You paste them into ChatGPT. ‘What are people complaining about?’ Forty seconds later you have six themes, a sentence on each, and something that looks a lot like a slide.
Nothing wrong with that. The trouble shows up the following Friday, when there are 500 new responses and somebody asks whether delivery got better or worse.
You don’t know. Last week’s six themes were an answer, not a dataset.
What it’s good at
Credit where it’s due. For a first read of a batch you can fit in a prompt, a general model is fast, free and better than skimming.
It summarises. It sorts rough positives from rough negatives. It suggests groupings when you have no idea what people are talking about yet. It answers a question about one export the way a sharp colleague would.
Think of it as a very quick reader who has never seen your business and will have forgotten all of it by tomorrow. Handy. Just be clear about what you’ve got.
It was built to write, not to count
A general model is a text generator. Its whole job is the most plausible next sentence, and it is brilliant at that.
It was never designed to analyse data. You find out the moment you ask it for a number.
Ask how many comments mention delivery. You get a figure that reads well and may have nothing to do with the pile you pasted in. Ask for example quotes. Some are paraphrased. Some are stitched together. Some are invented, in exactly the same confident voice as the real ones. Ask for the split between positive and negative and you get a tidy percentage with no count underneath it.
Any model can hallucinate. A model working on raw text with nothing stored has nothing to check itself against. So the summary is often right and sometimes not, and from inside the chat window you cannot tell which.
A number you can’t verify is a guess with a decimal point.
What’s missing is the dataset
Every one of those uses has the same shape. Text in, text out. That is querying unstructured data, and it caps the result at an opinion about a pile of comments.
A feedback programme runs on something else. Each response broken into the phrases where someone says one thing about one topic. Each phrase filed under one topic and scored for sentiment, using one method for every source. All of it stored with an ID, so it can be counted, sliced and compared.
That is structured data. It is what turns 4,000 comments into a number you can put next to churn.
ChatGPT does none of this. It isn’t meant to. So when the pasted-in analysis calls delivery ‘a major theme’, there is no count behind ‘major’, no way to see which 214 comments it meant, and no March to hold up against June. What that structured layer looks like, and how to tell if a tool has one, is what good AI text analysis looks like.
Where it breaks
Next quarter
The first Friday works. The second Friday you paste in the new responses and get a different set of themes, with different names and different edges, because nothing is holding the topics steady between runs.
You can reconcile them by hand. Congratulations, you are tagging again. That was the job you were trying to get rid of.
A trend needs both ends measured the same way. A general model measures each end fresh, so the trend is whatever two separate answers happen to imply. And proving a fix worked needs something to measure against, which you don’t have.
Scale
Five hundred comments fit in a prompt. Fifty thousand don’t. So the month gets chopped into batches, every batch gets its own slightly different themes, and someone loses an afternoon deciding whether batch three’s ‘slow service’ is batch seven’s ‘long waits’.
The tool that felt free at 500 comments costs a person a day at 50,000.
The click
Someone senior asks how you know delivery is the top complaint.
With a structured dataset, you open the topic and show them the comments, sentiment on every phrase. With a ChatGPT summary, you have a paragraph and your word, and no way of knowing whether the paragraph made anything up.
That is the gap between evidence and a claim. It is why drilling down to the verbatim is the capability that makes AI analysis defensible. You cannot drill into a summary with nothing under it, because there is nothing under it.
The data
Your respondents’ comments are customer data. Names, order numbers, health details, the odd address.
Pasting them into a general chat tool puts them somewhere your data governance never signed off, under terms your DPO probably hasn’t read. For a survey about a new logo, fine. For complaints, support tickets or anything a regulator might one day ask about, no. Feedback belongs in a platform where your data stays contained and separate from other clients, hosted in a region you chose, under certifications your security team recognises.
When ChatGPT is enough
| Your situation | Use ChatGPT | Structure it first |
|---|---|---|
| Volume | A few hundred comments, once | Thousands, every month |
| Cadence | A one-off look | Recurring reporting people rely on |
| Comparison | You only care about now | You need to know if it changed |
| Scrutiny | Nobody will ask how you know | Someone will, and it might be a regulator |
| Data | Anonymous or low-risk | Names, orders, complaints, anything sensitive |
| Action | Curiosity | Fixes with owners and budgets |
ChatGPT is a sketchpad. The right-hand column is a system.
Structure first, then read
None of this is an argument against language models. A structured platform uses them too, for pulling out mentions, scoring sentiment and writing summaries.
The difference is where they sit. In a platform, a model works on a stored, classified dataset that stays consistent across every source and every period, so its summary points at something you can open. In a chat window, the model is the whole process, and when it finishes talking there is nothing left.
So the order matters. Structure the feedback first. One method, every source, every mention filed and scored, all of it kept. Then read, summarise, ask questions and drill into anything you don’t believe. That is CX intelligence, and it is what SmartCX gives every team, one trusted data foundation for every dashboard, decision and action. The assistant already sitting inside your Microsoft stack raises the same question, and it gets its own answer.
That’s how SmartCX is built. Every response structured into stored, scored mentions, summaries that show you where to look, and any number one click from the words behind it. See it on real feedback in our product tour.
Frequently asked questions
Can ChatGPT analyse customer feedback?
Yes, for a first read. It can summarise a batch of comments, apply rough sentiment labels and suggest themes for a single export. It was built to generate text rather than analyse data, so counts and quotes can be invented, it does not store the results or apply the same categories next time, and you cannot click from a theme to the comments behind it. It works poorly as anything you report from or track.
Why can’t you track trends with ChatGPT?
Because each analysis starts from nothing. There is no stored set of topics held steady between runs, so the themes it produces this month won’t match last month’s names or boundaries. A trend needs both periods measured the same way against a stored dataset.
Is it safe to paste customer feedback into ChatGPT?
Treat it with caution. Feedback often contains names, order details and sensitive information, and a general chat tool is not covered by the data agreements your organisation has with its feedback platform. Keep identifiable or regulated feedback inside a platform with regional hosting and recognised security certifications.
When should you move from ChatGPT to a feedback analysis platform?
When feedback starts driving decisions. Recurring reporting, comparison over time, questions about how you know, action with owners and budgets, or any sensitive data all call for the feedback to be structured and stored rather than read on demand.
Does a feedback platform still use large language models?
Yes. Models do the extraction of mentions, the sentiment scoring and the summaries. The difference is that they work on a structured, stored dataset with one consistent set of topics, so every result can be counted, compared over time and traced back to the original comments.
SmartCX is the VoC platform that helps CX teams turn feedback data into CX intelligence the whole business can act on. Watch the product tour or talk to us about what your feedback looks like once it is structured.