Taming shoe bots: it’s no small feat
Senior Software Engineer, Developer Relations, Fastly
Automated scripts, or bots, make up a large portion of product purchases on the web today. To some businesses, these bots cause harm, but for others, bots help drive revenue. If you are an online business owner, your stance on how to address them will vary. Whatever your needs, Fastly’s edge cloud scales up to meet the challenge, and helps you to create the right set of policies for your business.
These days, sneakers (or maybe ‘trainers’ depending on where you are from), are no longer simply the footwear of athletes. They have become highly sought-after prizes, fashionable, and iconic. So much so, that some have turned to technology to gain an extra edge in this competition.
Devout sneakerheads and resellers of sneakers are leaning on “bots,” programs designed for the sole purpose of picking up as many pairs as possible of these high-demand items. As brands release their most anticipated products, traffic to purchasing sites spikes — with most of the orders coming from bots. As a result, human buyers can get left out in the cold.
This trend spills over into other industries as well. Hot items like toys and electronics are often snatched up quickly by bots, such as the coveted PlayStation and Xbox video game systems. Bots buying items in high demand during the holiday season are sometimes even referred to as grinch bots.
The robots are coming!
As a consumer, having to compete with bots can make the online shopping experience a frustrating one. Bots are too quick to do their job, and they can buy out the stock before a human shopper can even add the item to their cart. Real shoppers might have to purchase the item from a bot operator reselling at a higher price, or give up on obtaining it entirely.
For the business, the website can receive a lot of traffic from bots. During your product drop, a site that is targeted by a bot can receive millions of views. While it is not the objective for the bot to take your site down – they want to buy your product, after all – this traffic can be expensive to serve and potentially cause outages.
Security problem?
At first glance, it’s easy to think of bots as a security problem.
After all, bots saturate network bandwidth and can cause immense processing load. Making use of bot networks (botnets), they often masquerade their network requests to appear to come from thousands of unique IP addresses in order to circumvent rate limiting. Some bots abuse attack vectors in the shopping cart software or in the network architecture to get that extra advantage. Thus, large purchasing attempts by bots do have similarities to security issues such as distributed denial-of-service (DDoS) attacks.
However, when it comes to bots that want to pay real money to purchase your products, the story isn’t so simple.
Anywhere in the world where stock is limited and customers are ready to spend their cash, the chances are that bots are going to be there to get in on the goods too. But we should keep in mind that there are different types of business models selling these goods. Every store out there, whether it’s Walmart, Best Buy, Amazon, Supreme, Coachella, or Apple, may have their own, unique view on bots.
Ultimately, a sale is a sale. On the one hand, a business that obtains its products at wholesale prices and sells them at a predetermined markup might welcome this as a way to move a bunch of stock. At times, the scarcity created by the bots, however artificial, can make a product feel more exclusive and luxe.
On the other hand, some businesses may still have a priority to sell to human customers. If their online stores turn away too many human shoppers who want their PlayStation 5s and Xboxes, those buyers likely won’t be returning when they need their next TV, either.
There may be other concerns, too. It might be damaging to a brand’s reputation if bots create too large a secondary market for a product. Or, for some businesses, the perspective might be that it’s simply unfair for humans to have to compete with bots.
It turns out the policy needs are absolutely many, based on the site’s business model and priorities. In fact, the solutions used by our customers are so specific that we aren’t able to share details on individual examples here.
How you wish to handle bots that purchase your goods is a business decision, and it’s important to understand this. It means that there is no single magic bullet, and a solution that works for one business may not be the best one for yours.
We believe that it’s important to enable you, the business owner, to decide how much of the bot traffic you want to have, and in what ways you want it.
You’re who?
Websites that want to identify bots use various methods to identify bot traffic. A simple method is to detect when a large number of requests come from the same IP address in a short amount of time, though botnets will be able to circumvent this strategy.
Another technique is the familiar CAPTCHA, where the visitor is required to solve a puzzle involving selecting out photos of fire hydrants or making out the letters in an unclear image. We have an example on the Developer Hub of serving and validating a CAPTCHA at the edge. These days, however, even these are becoming possible to bypass, using artificial intelligence and CAPTCHA farms. While new, privacy-respecting solutions like Private Access Tokens are rolling out, they aren’t broadly available yet.
More complex behavioral detection methods use heuristics based on the premise that humans and bots don’t always behave the same. Sites can look at information such as the following to come to conclusions.
Excessive number of page loads per session
Unnatural page navigation order
Time between loading pages too short
Failure to load dependent resources, such as JavaScript, CSS, and ads
Clicking on links that are not in the visible part of the viewport
Filling out forms with unnatural speed
Browser fingerprinting methods use various techniques to collect information about the visitor, such as the device, browser, and operating system. The collected “fingerprint” is heuristically analyzed to determine if it matches known bots, or if modifications can be seen that attempt to conceal the presence of a bot. Various fingerprinting mechanisms exist, such as the FingerprintJS library, and JA3, a popular method for fingerprinting TLS clients. We have support for JA3 in both our VCL and Compute platforms.
Bots can be computationally expensive to detect. As detection techniques improve, bots also revise their stealth strategies, as quickly as every few weeks, resulting in a never-ending cat-and-mouse game.
You’re a bot! What to do with you?!
Once a visitor has been identified as a bot, it’s now time to decide what to do with them.
If you want to lock them out, then the simplest thing to do is to serve an error code and turn them away on the basis that they are a bot (and likely violating the site’s terms and conditions!). However, we’ve learned that this sort of response motivates the bot authors to come back with an even more aggressive strategy to bypass detection.
Instead, you might opt for a mitigation strategy that does not necessarily let the bot know that they have been detected. For instance, you might use penalty boxes instead, to slow down requests coming from a single IP address.
penaltybox origin_response_pb {}
sub vcl_fetch {
if (beresp.status == 429) {
ratelimit.penaltybox_add(origin_response_pb, client.ip, 1m);
}
}
sub vcl_deliver {
if (ratelimit.penaltybox_has(origin_response_pb, client.ip)) {
resp.tarpit(5, 1000);
}
}The above example shows an example of using penalty boxes, an Edge Rate Limiting feature of Fastly. This code adds an IP address to a penalty box for 1m after the server has rejected the response with a 429 “Too Many Requests” response code. During this time, clients from this IP address are forced to wait 5 seconds before each 1000 bytes of the response are emitted.
You might get even more creative, say serving up fake shopping carts to these bots. Bots could go through a fake checkout, and even see fake success pages — the bot is kept happy without the site actually losing a sale to the bot.
Fastly also provides integrations with partner platforms such as HUMAN Security, which can be used in bot identification and mitigation.
Tools are also available that can be used to put bots on a similar playing field as your human visitors. Solutions such as the waiting room are often used to ensure that humans have a chance at getting in line for purchase during times of peak traffic.
On the other hand, your business might not care about bot traffic, so long as we protect your origin and the shopping cart. Perhaps all you want to do is prevent credential stuffing. Maybe all you need is to be able to differentiate bot traffic from organic in the logs.
Whatever your needs, Fastly’s infrastructure works with you to handle bots:
We work with your business to build the bot mitigation strategies that make sense for your policies.
We provide platforms, such as Compute, and tools, such as Rate Limiting, and solutions, such as Waiting Room, to build your strategy.
We protect your origin, by scaling up and serving your traffic from the edge. The Fastly Next-Gen WAF (powered by Signal Sciences) can also be a powerful tool to help protect you from bots engaging in automated attacks and abuse.
Employ the policies that make sense for your business
Whether we like them or not, bots are here to stay. How does the bots vs humans competition align with your business’s goals? Whether you want to shut them out or create policies so that you are still able to maximize the opportunity for human customers to buy, Fastly is here to work with you and help set up the right set of policies for your business.
