The global COVID-19 pandemic has forced many people to be increasingly online. Social distancing policies have forced many workers to work from home, which rapidly increased the adoption of online video conferencing, online shopping, and other online activities.
As a result of this phenomenon, there has also been a major increase in potential cyber-attack surfaces, and cybercriminals are aware of this situation. Many hackers and cybercriminals are utilizing malicious bots to perform various cyberattacks targeting these new attack surfaces.
There were more than 1.3 billion bot-related cyber attacks in the third quarter of 2020, and in 2021 so far, one-third of global web traffic is made up of malicious bots, and the numbers are still increasing. Also, consider the fact that more cybercriminals are now targeting smaller companies rather than big enterprises.
With these immediate dangers, now bot management is no longer a luxury thing for giant tech companies and famous organizations but is a necessity for anyone with an online presence.
What Is Bot Management
As the name suggests, bot management is an effort of managing bot activities, especially activities from malicious bots but also from good bots (more on this later). The term “bot management” might also apply to a software solution or tool that enables us to manage these bot activities.
The main purpose of bot management is to stop malicious bot traffic. However, when blocking the malicious bot is not possible for one reason or another, but management must also attempt to manage and reduce the potential impact of the malicious traffic.
Two Dilemmas of Bot Management
At first glance, bot management might seem like a pretty simple process: detect the presence of a bot, and stop this bot’s activity.
However, in practice, bot management can be very challenging due to two main dilemmas:
1. The presence of good bots
Bots, by themselves, are neutral tools designed to perform automated tasks, so they are not inherently good or bad. It’s how they are used that will determine whether a bot is malicious or good.
So, there are good bots owned by reputable companies that can be beneficial for your business.
Google’s crawler bot, for example, is responsible for crawling and indexing this web page, allowing it to be ranked by Google.
We wouldn’t want to block these good bots, which will also block their benefits, and yet differentiating these good bots from malicious ones owned by hackers can be very difficult in practice.
2. False positives: accidentally blocking legitimate users
As technology grows, so does the sophistication of malicious bots.
Bot programmers are getting more advanced in inventing new bots while adopting the latest technologies including AI and machine learning. As a result, today’s malicious bots are really sophisticated in masking their identities as human users.
They can, for example, mimic humanlike behaviors like performing non-linear mouse movements, and can rotate between thousands of residential IP addresses per minute.
Accidentally blocking legitimate users can disrupt your performance and revenue, and if you are not careful it can hurt your reputation. Yet, even differentiating between bots and legitimate human users is becoming increasingly challenging.
Bot Management: How It Works
Modern bot management is about tackling the two dilemmas answered above, and so we can divide bot management into two aspects:
- Bot detection: detecting the presence of malicious bots while distinguishing bots from legitimate users and bad bots from good bots
- Bot mitigation: deciding what to do with malicious bot traffic. Blocking is the priority, but other options like a honeypot, rate limiting, and others must also be considered
Each bot management solution might use different bot detection and mitigation methods, and below we will discuss these different methods.
Different Bot Detection Methods
1. Challenge-based detection
The basic idea of this method is fairly simple: we challenge incoming traffic with a test that is designed to be easy for human users but difficult to solve by software/programs.
CAPTCHA is the most common example of challenge-based detection methods that most of us are familiar with, but there are other methods that belong in this category, like anti-spam honeypots.
The main downside of this method is that it will also challenge legitimate users, so it will ruin the user experience.
2. Signature/fingerprinting-based detection
In this type of detection method, the bot management solution compares a user’s activity with a known set of signatures. The most basic method in this category is detection based on IP addresses (IP address here is a signature).
However, the main downside of this method is that it can only detect attacks with known signatures, so it can’t detect brand-new bots without any known fingerprints.
3. Behavior-based detection
Behavior-based detection uses AI technologies and statistical techniques to analyze a huge volume of data to identify behavioral anomalies. So, a bot management solution capable of behavior detection will monitor the network for behaviors that may be linked to malicious bots.
Solutions like DataDome use AI and machine learning technology to monitor and analyze the network traffic and identify the presence of malicious bots in real-time.
Different Bot Mitigation Methods
Once malicious bot traffic is identified, the bot management solution can use several different mitigation approaches:
1. Blocking
Blocking is the most cost-effective mitigation approach since you don’t need to serve any resource at all when you block traffic.
However, blocking is only viable if we are 100% sure that we are not blocking legitimate users and/or good bots. If we are using an advanced, behavior-based bot mitigation solution like DataDome, this won’t be an issue.
2. Challenge
Another approach is to challenge an identified bot with the CAPTCHA or other challenge-based mitigation methods. This way, we are only challenging suspicious traffic that’s most likely malicious bots and not human users.
However, use this method sparingly and carefully not to affect user experience.
3. Feeding fake data
Another effective approach is to reply to the bot’s requests with fake data or content, which will keep the bot active and waste its resources. Doing so will poison the bot’s database and slow down the bot’s operation, and the hope is that by slowing them down enough, the attacker will move on to other targets.
4. Rate limiting
Also known as throttling, it’s an effective mitigation approach for mitigating persistent bots by limiting the bandwidth or data rate we use to serve the bot’s request. Similar to the above, the hope is that the attacker will waste resources and give up.
Closing Thoughts
Effectively managing bot traffic, especially from malicious bots, is crucial if you want to keep your online business protected in 2021 and onwards. By using an advanced bot mitigation solution like DataDome, every request to your websites or mobile apps is analyzed and either blocked or mitigated in real-time.