August Success Spotlight: Embracing AI for Issue Classification

If you ask your agents what the best part of their job is, they most likely won’t say that it’s manually sorting through and classifying issues.

If you ask your customers what the best part of communicating with support is, they mostly likely won’t say that it’s waiting on hold for a representative.

Automatic issue classification powered by artificial intelligence alleviates these pain points, and Helpshift’s Customer Success Manager Chris Hunt is here to delve into the “how” and “why” in this month’s Customer Success Spotlight!

1. What is the benefit of using AI to classify issues (as opposed to manual/agent or keyword based triage)?

CH: Speed and accuracy — Instant issue classification means your issue-specific workflows are kicked off as soon as the conversation starts. Whether it’s routing, collecting additional information, or bringing an issue-specific bot into the conversation, this has a business impact of reducing both the time to first response and the need for agent capacity. There’s also a soft ROI: agents can focus on the more nuanced and high-touch issues and crafting thoughtful quality responses — whether it’s incorporating game lore into a response to put a smile on a user’s face or sharing enthusiasm about a luxury brand, these small interactions build brand loyalty and delight users.

Accuracy at scale — a healthy model will outperform a team of agents in both accuracy of issue classification and thoroughness. The issue types and labels are managed by an admin so workflows can be automated with more confidence and integrity of report data is significantly improved. With customer support data being relied on more and more by other departments to serve as the voice of the customer, integrity of reporting data is paramount.

2. Is AI/NLP technology ready for this use case? (I’m scared of bad bots!)

CH: Yes. As other companies that have launched large scale AI projects have learned, the sweet spot for leveraging AI starts with identifying a well defined problem and a narrow use case to start. Issue classification meets both these criteria.

At Helpshift, we incorporated these learnings into our product development philosophy. With SensAI Predict, we use AI and NLP to classify the issue. Then we incorporate an IF/THEN rules-based system for building reliable workflows and custom bots. These are core components of our platform with an easy to use UI. This removes the need for costly developer and data science resources while empowering customer support professionals — who often have the most insight — to configure their own AI and bot-powered workflows.

3. How do I set myself up to use AI for issue classification successfully (and how do I prepare for implementation)?

CH: For this application, it’s helpful to think of AI+NLP as working through pattern recognition. You’re feeding in issue patterns to train the Predict algorithm on what each issue type looks like. The more issues that are fed into the model, and the more distinct each issue type is — the better. When identifying what label set to start with, pick the issue types that represent the highest percentage of volume (this should also increases your ROI once successfully implemented), as well as distinct issue types.

Ensuring your issue types are distinct from each other is key to avoid ‘confusing’ the model. For example ‘billing returns’ and ‘billing refunds’ might have very similar message content and would not have unique enough patterns for a model to differentiate between them reliably. In this case you can use a broader label that combines both refunds and returns, then use an automation to have a user select the more granular category.

The data should be structured and verified for accuracy. For Predict, we use a simple CSV with one column for issue label and another for the corresponding first message from the end user. When preparing customers to start using Predict, we ask for minimum of 500 issues per label and total of 15,000 issues. This can come from historical data, (but there should be an audit of the training set data first to ensure the labels match the end user message.) Garbage in equals garbage out.

With Predict, once you’ve uploaded the CSV and trained the model, you can set the confidence threshold. Confidence is the likelihood that a given label may be associated with an issue based on the data available to that model. The confidence threshold is the minimum confidence you want before Predict will apply a label. To start, we recommend a lower threshold of 50 percent. This will label more issues and give your agents and supervisors more chances to provide feedback on accuracy of the predictions.

This also means that training team members on why feedback is important and how to provide it is crucial before deploying. For those worried about agents providing feedback on issues, it’s worth considering the value of crowdsourcing this task. Human feedback is needed to both train and maintain the model. The way your users describe their issues will change over time, and unless you intend to have supervisors or designated agents download and review large data sets periodically, the best way to accomplish this is to incorporate it directly into agent training.

Once your model is deployed and Predict is classifying issues, you can start to think about what workflows and automations you want triggered based on those labels. We’ll dive into that in next month’s post. Please reach out to [email protected] or your account manager with any additional questions in the mean time.