Improving Contributor Satisfaction with AI/ML-powered Customer Support Automation

Manual Ticketing Caused Categorization Errors, Increased Ticket Handle Time and Contributor Churn

Appen was looking for innovative ways to improve their manually-administered ticketing system for the following reasons:

• Contributor-generated tickets had to be categorized accurately to ensure fast and efficient resolution, which was challenging to do given that contributors often failed to self-report issues in the proper category.
• The Contributor Success team of two had to handle — categorize, re-categorize, prioritize, and resolve — over 11K tickets, which left them only seconds to close each issue generated over a month.
• Relying on self-reported categorization resulted in many miscategorized tickets, pushing the Contributor Success team to focus on less important issues, causing delays in ticket response times and resulting in dissatisfied customers.

Provectus proposed to build an automatic ML-powered ticket categorization system and combine it with a routing solution from ZenDesk to handle contributor-generated tickets. Based on Natural Language Processing (NLP), it would process a ticket’s text to automatically categorize, prioritize, and resolve simpler tickets while routing more complex tickets directly to a human team.

The solution as such could shift focus from manual and labor-intensive screening of thousands of tickets, including miscategorized and mishandled ones, to ensure that urgent tickets are resolved first, in order of priority.

As a result, Appen could achieve a better contributor satisfaction rate and encourage their best contributors to remain on the platform, thereby implicitly increasing the quality of the platform’s customer-centric training data.

From Manual System to ML-Powered Ticketing Categorization with Human-In-The-Loop

ML-powered ticket categorization built by Provectus classifies and tags a wide range of incoming tickets, including:

• Account questions
• Job issues
• Suspensions
• Platform errors
• Payment questions

When classified and tagged, tickets with the highest priority that require manual processing are sent to Appen’s Contributor success team. Others are either handled automatically or are reviewed by the staff in order of priority. The classification and tagging are performed for ZenDesk. When generated in ZenDesk, a ticket is sent to Amazon API Gateway, which triggers AWS Lambda to push data to SageMaker’s endpoint where the ML model is hosted.

Once the data has been processed by the ML model, the ticket gets classified and appropriately tagged. Tags are pushed to AWS Lambda and then to ZenDesk via Amazon API Gateway. In ZenDesk, tags get assigned to generated tickets accordingly.

The applied ML model is built using TensorFlow, and it solves a standard text classification problem. Since tickets generated in ZenDesk should include a title, description, and other details, their contents can be combined and pushed to the ML model for analysis — classification and tagging.

Appen’s Contributor Success team creatively suggested using their data labelling platform to generate initial training datasets for the development of ML models. It helped the Provectus team reduce data preparation overhead and focus on experimentation and development.

Every ML model is retrainable and can be enhanced. If it returns low accuracy for a ticket’s tag, it opts to send the ticket for manual review. Once a human support agent properly tags the ticket, this data is collected and can be further utilized for model retraining, which takes place once a month.

The solution automates ticket resolution for a considerable portion of tickets while ensuring prioritization in addressing urgent issues by the staff.

Automated Ticketing System Allows Operations to Scale, Increasing Contributor Satisfaction

Appen received a state-of-the-art ticket categorization powered by Machine Learning and Natural Language Processing, combined with the ZenDesk routing solution.

The solution allowed Appen’s Contributor Success team to handle over 11K tickets generated by contributors over the course of a month. With up to 80% of tickets categorized and resolved automatically, the team was able to focus on complex issues, resolving them more quickly and efficiently.

By reducing ticket resolution time from two weeks to less than 24 hours, Appen’s Contributor Success team ensured the loyalty of their platform’s best contributors. A 10% increase in satisfaction resulted in less churn, which implicitly improved the overall quality of training data delivered by Appen to its wide network of enterprise clients, thereby stimulating growth.

Appen’s Contributor Success team was extremely happy and satisfied with their new ticket categorization solution. Since it automatically handles routine ticket tasks, the team was able to make contributor satisfaction their priority, which not only increased their own performance and productivity, but also produced a positive impact on Appen’s business growth.

Moving Forward

1. Learn more about the Provectus AI Solutions and ML Infrastructure
2. Watch the webinar on MLOps and reproducible ML on AWS
3. Apply for Machine Learning Infrastructure Acceleration Program to get started