How Freshdesk is Turbo-charging Your Social Support with AI
A week ago, we launched Social Signals — Freshdesk’s first AI feature. We built it to help businesses like you engage your customers on social media, and to identify customer needs and resolve issues quickly. The AI in Social Signals is trying to answer a simple question — when a user @mentions a brand on Twitter, is he or she trying to elicit a response from the brand or just ‘mentioning’ it in passing? In other words, is the user @mentioning the brand in the hope that it will respond? Of course, this is an inherently fuzzy concept: different users and brands will have varying notions of whether or not a tweet is relevant – one man’s meat is another man’s poison, as the saying goes.
The question now becomes: across a vast landscape comprising tweets, brands, users, and mentions, can we build an intelligent system that learns how to distinguish real attempts at engagement from passing mentions, and to separate relevance from irrelevance? And once our system has learnt these distinctions in a general fashion, can it tune itself to your specific needs and expectations?
Assembling the Data
First, for an AI to be able to learn, it needs historical data. In our case, historical data is relevant tweets. But, as is often the case when designing systems that cater to human needs, relevance is messy and subjective. What exactly is relevance in this domain, anyway? How do you operationalize this idea and turn it into something tangible?
We decided to use a rather simple rule of thumb (also known as a heuristic) — if you think it’s relevant, we do too: therefore, if I tweet at @amazon asking why my package looks like it was mangled by a bear and they respond in timely fashion, then voilà, we’ve unearthed a relevant tweet. On the other hand, if my tweet mentioning @klm and linking to an article about the 10 best in-flight meals is ignored and not responded to, it will sadly be consigned to irrelevance. Aggregating over hundreds of thousands of tweets, we assembled a corpus of brand-mentioned tweets that have been responded to (or not) in a timely manner.
This raises another question — what exactly does timely mean here? One @amazon tweet might have been responded to in an hour, while another took a few days before being responded to. Does that make the second not worthy of the ‘relevant’ tag? We all intuitively know that brands respond with varying speeds — some with lightning-quick efficiency and others at a more laggard pace. If we were to set a cutoff of 1 day (or a few hours) and only deem those tweets whose responses fell within this (rather strict) deadline relevant, then we might be missing out on a lot of genuine user issues that received responses a little later. In the AI biz, we’d call these missed opportunities ‘false negatives’ — you should have responded to it sooner, but you didn’t.
On the other hand, allowing a more relaxed deadline means that even tweets that aren’t strictly attempting to engage (viz, the @klm tweet I mentioned above) might elicit a (delayed) response from the brand saying ‘Thank you!’. From a support perspective, this isn’t relevant, but it’d be marked as such anyway. In AI-speak we’d refer to these as ‘false positives’ — you probably didn’t need to respond, but you did.
Blink and Miss
One way to resolve this conundrum is to think about the purpose our AI is meant to serve — finding the proverbial needle in a vast Social haystack. Tweets have a short half-life, and are primarily of the blink-and-you’ll-miss-it variety regardless of whether they’re relevant to you or not. Of the 10,000 tweets your brand was mentioned in today, there were probably a handful that were relevant and you may not have been able to respond to them quickly enough to meet a strict cutoff. Thus, it’s better to err on the side of a relaxed response time cut-off rather than a strict one – we consider the distribution of response times for each brand and set a customised cut-off. After all, to paraphrase Blackstone, it’s better to let ten irrelevant tweets go free, than to allow a single relevant one to suffer.It is better to add a few false positives than a single false negative. #machinelearning #ai #customersupport Click To Tweet
On the Incomprehensibility of Words
Here’s a fun digression for those of you who like getting into the details: how do computers know what a tweet means? Tweets are essentially collections of words – how is a computer supposed to understand them? At its heart, a computer is comfortable only with numbers. It understands numbers, and relationships between them. It knows that 5 is larger than 3, and that 4 plus 2 is 6.
So, we’d need a way to convert these tweets into numbers so that the computer has a way to work with them. It just so turns out that the AI community has been hard at work finding ways of representing each word as a series of numbers called vectors. A famous paper (in the AI community) called word2vec (Word to Vector) showed how to find numerical representations of words so that the ‘word vectors’ captured similarity and relationships between words. The most famous example of this is ‘king – man + woman = queen’, where the vector for the word man is subtracted from king, and then the vector for woman is added, and you get the vector for queen. Essentially, the word2vec AI has basically managed to managed to solve an analogy question for you – king:man :: queen:woman – King is to man, what queen is to woman!
So now, we have a solution to the problem – convert (in AI terms, vectorise) the tweet into vectors using word2vec, and now computers have a (relatively) easier time processing them.
Quizzing the Computer
Now that we have the data and we’ve given our computers a way to work with tweets, we need to build an intelligent machine is capable of sifting through your brand’s @mentions and finding the ones you consider relevant. To do this, the AI tries to identify patterns that differentiate tweets that were deemed relevant from ones that weren’t (in technical terms, it’s a binary classifier that uses word2vec representations of tweets as features). Once it has developed a sufficient understanding (after looking at a few hundred thousand), we present it with a set of new tweets, but sneakily hide whether they were relevant (or not).
This is essentially a quiz for our AI to show off how much it’s learnt. It goes over the tweets one at a time and tells us which ones it thinks are relevant (based on what it’s seen before). We compare its responses to the actual relevance (which only we have) and issue a report card of its performance – how many it got correct, how many mistakes, etc.
This tends to be an iterative process where we try to coax the AI to learn in different ways with the aim of improving its quiz score. Once we’re satisfied that it has learnt enough, we make it ready for customers to use on their own tweets.
Tailoring AI to your Needs
Of course, the mock test isn’t exactly the same as the actual exam – we all make mistakes. So, we allow you to teach the AI with feedback. Let’s say it flags a tweet as Relevant (based on what it’s seen before), but you don’t agree – you think it’s Irrelevant. That’s fine – once you let it know, the AI will adjust its learning to incorporate the feedback so that it weeds out similar tweets in the future. Alternatively, let’s say you found a tweet that the AI missed – let it know, and it’ll mark similar ones as Relevant in the future.
Wrapping it up
Building a successful machine learning product or feature is extremely challenging. But what’s more challenging is the kind of effort that is needed to turn the model into an efficient customer-facing tool. However, the advantage of providing the system with your feedback is that over time, your AI and mine will diverge. It will constantly tailor itself to your feedback. It will begin delivering more accurate and customized experience for your customers in the future! So, what are you waiting for? Enable Social Signals in your helpdesk already. If you’ve already started using it, tweet at us and tell us how you like it with the hashtag #SocialSignals. If you haven’t and need help with setting it up, write to us at firstname.lastname@example.org
Before I let you go, here’s something to mull over — We’ve taught the computer how to interpret words, but what about emojis?