Stars in their eyes – why we changed our emoji ratings

We all know the importance of measuring the quality of your interactions with your customers.

Asking for feedback from customers helps managers spot and reward star performers and identify opportunities to improve the quality of support their team provides. There are numerous industry standards for this – Customer Effort Score (CES), Customer Satisfaction (CSAT), Net Promoter Score (NPS), etc. At Intercom, we built what we call “conversation ratings”.

When working on this feature over two years ago, we wanted to design a solution that was in line with our mission, making internet business personal. Thus, while we explored options such as simple thumbs up/thumbs down designs and five-star satisfaction ratings, it was no surprise that we were really drawn to using an emoji scale, with a row of faces – 😠 🙁 😐 😃 😍 – representing a five-point scale from “Terrible” to “Amazing”. The prototype was well received by both our own teammates and end users.

Managers felt it allowed them to better understand how their customers felt, and end users felt it was modern and fun, and offered clarity around what they were being asked. While most feedback was positive, there were some customers who felt it was too expressive or not professional enough for their business.

However, after thoughtful consideration of the pros and cons of each solution, we decided to pursue the emojis as our mode of conversation rating. As we put it at the time, “Using emoji faces felt expressive, modern and in line with the Intercom messenger style.”

Why were people reluctant to send 😍?

Before we shipped the feature, we expected customers to ask to allow them to customize or change the emoji set altogether. Surprisingly, when we carried out the research a couple of months later, this was not a common request. However, we did discover that 30% of the conversations we analyzed included a request to kill 😍. As avid users of this emoji at Intercom, we were a little confused by the request.

As time went by, it became apparent why our customers wanted to change the heart eyes emoji specifically:

https://twitter.com/codykeith/status/1049667345436856321

Not sure this customer support rating system by @intercom with emojis is a good idea. I’m not giving my support rep heart eyes but I would of given him 5 stars. pic.twitter.com/6UTuGFZpBm— MODassic (@modassic) August 1, 2018

Customers were apprehensive to click on 😍 for two main reasons:

• They only used this emoji in communication with a romantic partner.
• Males did not feel comfortable sending it to another male.

The explanation was clear – the heart eye emoji was perceived as being inappropriate or having romantic connotations, which is probably not what you want to suggest when rating a customer support interaction. Indeed, the great thing about emojis is that they are very expressive in a succinct manner.

“This reluctance to use the 😍 emoji resulted in customer support agents and teams getting lower ratings than their performance deserved”

What we did not realize when choosing the scale was that the usage and interpretation of certain emojis would differ so much across cultures and even gender. There are even models to infer gender based on usage of emojis.

This reluctance to use the 😍 emoji resulted in customer support agents and teams getting lower ratings than their performance deserved, largely because of a design decision rather than any fault of their own. This trend was also noticed by our internal customer support team.

From 😍 to 🤩

We knew we had to change this particular emoji and after careful deliberation, we decided 🤩 was the perfect alternative. It was differentiated enough from the other emojis in the scale to suggest very high satisfaction, while communicating less passion and more excitement. Both our own customer support team and customers were unanimous in their appreciation. As one Intercom user put it: “Thumbs up for changing the 😍 to a starry eyed face 👍. Some of our customers have said they would have rated maximum, but the lovely eyes were just too much 😂”.

Beyond the positive reaction, the results we saw were staggering.

Looking at Intercom’s own Customer Support team, we saw an 11% increase in the number of “Amazing” CSAT ratings, almost all of which came from customers shifting from 😃 to 🤩.

Even more impressive was what happened when you looked at just the males on the Customer Support team, as this group saw a 17% increase in the number of “Amazing” ratings when we moved from 😍 to 🤩. This increase was enough to close the small gap between the rates at which male and female customer support representatives received 🤩 and 😃 ratings. This had a real impact on people’s perceived performance.

What we have learned

In text-based communication, emojis can act as substitutes for our body language and facial expressions, and can be used to convey emotion and clarify a message’s meaning. It was fascinating to find such a clear example of how, just like nonverbal cues, the usage and interpretation of emojis changes across cultures and, furthermore, how someone’s choice of emoji can be dependent on the gender of the recipient.

Above all, though, whether it’s expressed with stars or hearts or smiles, we love enabling great customer support and the very real personal interactions that can occur with thoughtful design.