Coronavirus has cut a large and uneven swathe through the world’s businesses. While some fields — hospitality, transportation, restaurants — have been crushed, other fields — grocery delivery, e-commerce — have been crushed in a very different way: with too much demand from sheltering consumers. This is the story of how Tolstoy helped an online garden retailer handle a massive inflow of customer emails as they waged their way through the pandemic.

Pandemic-level demand

While some businesses have seen their demand dry up overnight, other businesses have experienced record inflow. One of the activities consumers stuck at home have turned to is gardening. Garden retailers both in the US and Europe have seen orders skyrocket hundreds, or thousands, in percent growth.

One of these garden retailers approached Tolstoy with a dilemma. The coronavirus spigot had sent them several times more orders than they had capacity to fulfill. Correspondingly, they had received over 50,000 customer emails since the start of March — most of them angry emails inquiring on order status.

The company usually routed customer inquires through Freshdesk, an email platform, that helped categorize inbound mail. But now they were receiving correspondence from many sources, including direct email, which had no categorization applied. They were confronted with the prospect of reading through more than 50,000 emails — with thousands added per week — just to triage high priority emails, such as cancellation requests.

They estimated that if they hired 4–5 staff, it would take about a month to read the emails — during which tens of thousands more emails would arrive.

AI to process emails

Tolstoy took a look at the company’s email data. Email text data can often be messy — everything from the sender fields to legal disclaimers are included, so we cleaned it up with some preprocessing. This meant stripping out all forwarded emails and auto-email generated text, after which we normalized the text by removing punctuation and capitalization.

Next, we considered the training set of about 3,000 categorized emails. These emails were classified through Freshdesk, which meant that customers picked the category. While most of them were useful categories, customers often picked wrong categories — for example, some selected “Cancel” when they actually wanted to inquire about a new product, or “Delivery query” when they meant to cancel. We manually corrected and labelled a set of more than 5,000 emails to assert “ground truth”.

We also balanced the limitations of the training data by hard-coding certain heuristics into the predictions.

After testing different models, we settled on a logistic regression, which is based on the frequency of words found in the email text. To help the model focus on the more important features, we only included words that surpassed a fixed number of appearances across the dataset and incorporated elastic net regularization.

Ultimately, we configured the model to read and categorize emails with ~98% accuracy, and 97–99% precision/recall (a measure of true/false positives captured), based on a set of 1,300 manually corrected emails. This was far more accurate than even the existing categorized emails via Freshdesk.

Record time

The garden retailer sent over their 50,000+ batch of emails on a Friday. We tuned the model on the following Monday, and returned the categorized results back on Tuesday.

This enabled the retailer to immediately start processing and responding to all their customer emails, which they had not been able to do for months. Finally, customers could get their answers and orders fulfilled or refunded.

From Tolstoy to the gardeners of the world, Here’s to hoping you get your gardening in during shelter-in-place!