Website bots: The good, the bad and the ugly
In this help document, you’ll learn about the good and bad of bot traffic and popular methods to keep the ugly bot traffic at bay.
Looking for how to block bots in Lucky Orange? Click here to read more about how you can use Lucky Orange to spot bots and block them from being tracked by Lucky Orange.
What are bots?
By definition, website bots are automated applications/programs that complete tasks online. These can be good tasks, bad tasks and everything in between. Some bots can be legitimate, such as search engine crawlers, while others are malicious, such as spam bots. The number of bots keeps growing, but here’s a quick look at some of the most common ones you may encounter:
| Bot | Type | Description |
| Search engine crawlers | Good | Search engines like Google or Bing use bots to crawl websites and index their content. It’s considered harmless and beneficial for your website’s SEO. In addition to Google or Bing, it can also include Baiduspider and YandexBot. Googlebot is also responsible for |
| Online store speed (Shopify only) | Good | Shopify runs a daily speed report using a Google Developer Tool that tests your homepage, a popular collection page and a popular product page to be compared with similar Shopify sites. These bot visits will come from Cedar Rapids, Iowa (the location of the Google data center) daily. It’s considered harmless and helps you evaluate your Shopify store’s speed. |
| Social network | Good | Social networking bots include the likes of Facebook crawlers and Pinterest crawlers. Once the website has been shared on social networks, these bots crawl the website to ensure it’s not spam content and create better user recommendations. |
| Aggregator | Good | Aggregator bots crawl the RSS/Atom feeds to automatically generate feeds for the preferences of users. An example would be the Feedly Fetcher. |
| Marketing | Good | Marketing bots include SEO and content marketing bots that crawl websites looking for backlinks, keywords (organic and paid) and traffic data. Examples include SEMrush bot and AhrefsBot. |
| Site monitoring | Good | Site monitoring bots pink your website on a regular cadence to evaluate overall performance and downtime. Examples include Uptimebot, WordPress pingbacks and PRTG Network Monitor. |
| Voice engine | Good | The Alexa Crawler and Applebot crawl websites to give accurate answers to questions asked by users to their voice assistant devices like Alexa or Siri. |
| Ad verification & testing | Good | Ad bots, which include verification, testing, and competitive research, crawl your website to ensure that the ads are properly working. |
| Ad performance monitoring | Good | These bots actively monitor the ad performance to ensure that ads and the surrounding content load properly and offer a smooth user experience. |
| Web scrapers | Bad | Web scraping bots extract data from websites. For example, they navigate through web pages, aggregate content and research prices. It can be used for a variety of reasons - even by your competitors - but can put a strain on your website’s server resources. It’s often used without the website’s permissions which violates copyright laws. |
| Spam | Bad | Spam bots flood websites and comment sections with unwanted or irrelevant content. This can include advertisements, pornography and/or malicious links. |
| Click | Bad | Click bots simulate clicks on ads to make it appear that the ad is generating engagement There are “ click farms” with the sole intent to fraudulently generate ad impressions and click-through rates. |
| Credential Stuffing | Bad | Credential stuffing bots use stolen usernames and passwords on websites to gain unauthorized access to user accounts. |
| DDoS | Bad | These bots are part of a larger “botnet” to launch Distributed Denial of Service (DDoS) attacks which overwhelm a website’s servers with massive traffic to make it inaccessible. There was a large DDoS attack on Amazon Web Services in 2020 that impacted thousands of websites. |
| Phishing | Bad | Phishing bots are typically done through chat bots. They will imitate legitimate customer service representatives to trick users into providing sensitive information such as credit card or banking details. |
READ MORE: See how one BigCommerce website spotted and stopped a DDoS to prevent the site from crashing.
How to prevent bot traffic
Good bots
Some bots, such as search engine crawlers and Shopify’s online store speed bots, can’t be prevented. These are both “good” bots that are harmless and can even help your website. If you so desire to block them, there is code you can add to your website.
Remember: good bots do have benefits to your website, such as SEO. Typically websites allow these good bots but can still block them from being tracked within Lucky Orange.
Bad bot
Malicious bots are the torn in your website’s proverbial side, and the defenses against them can range from the simple to the complex. A few of the options you can consider include:
- Implementing CAPTCHA tests: These tests can separate humans from bots without disrupting your user experience.
- Adding a web-based firewall: Firewalls can block knock IP addresses to help prevent attacks
- Blocking known bot IP addresses: Refer to your website developer or platform to learn how to block specific IP addresses that generate a lot of bot traffic. Read on to learn how Lucky Orange can help.
- Paying for bot detection and prevention: This can be a tool such as DataDomeor a WordPress plugin such as SiteGround. Lucky Orange doesn’t vet these companies and recommends that you complete your own research prior to paying for a bot detection/prevention service.
- Using a Content Delivery Network (CDN): CDNs distribute traffic across multiple servers, which make it more difficult for botnets to coordinate an attack on your website. CDNs alone can’t stop bots and need to be properly secured to avoid being hijacked. If using a CDN, make sure that you complete additional research and review into the CDN security history and documentation.
- Increasing sign-in security: This can include:
- Limiting the number of sign-in attempts
- Requiring additional verification the sign-in IP address doesn’t match the last known IP address
- Confirming sign-in with the account’s email address and/or text
- Providing single sign-on options
- Offering 2-step account verification
For more questions related to bots, we highly recommend speaking with your developer or a third party for information specific to your website and needs.