Text only | Text with Images

Randem Systems Support Board

BotBanish => BotBanish - General Questions, Report Bugs, Problems etc... => Topic started by: sonnyh on July 17, 2019, 02:32:21 AM

Title: Blacklist not blocking BOT
Post by: sonnyh on July 17, 2019, 02:32:21 AM
Hi,
I have previously added crawl-66-249-69-212.googlebot.com to the spider blacklist.
I noticed that is was in the "Whos Onlibne" list this morning.

Is there anything else that needs to be done after something is added to the blacklist?
Title: Re: Blacklist not blocking BOT
Post by: Randem on July 17, 2019, 03:25:14 AM
Hi sonnyh,

There are 3 blacklist, Domain, IP and Spiders. crawl-66-249-69-212.googlebot.com would need to be added to the Domain blacklist for it to be blocked. Adding it to any other blacklist would not get the desired effect.
Title: Re: Blacklist not blocking BOT
Post by: sonnyh on July 17, 2019, 08:57:07 AM
Thank you.
I misunderstood, thought that was the Bot name.
Title: Re: Blacklist not blocking BOT
Post by: Randem on August 10, 2019, 01:02:10 PM
sonnyh,

Add gogglebot.com to the Domain Blacklist and googlebot to the Spider Blacklist.
Title: Re: Blacklist not blocking BOT
Post by: sonnyh on August 11, 2019, 03:33:32 AM
Hi,
I added googlebot to the spider list.
The Domain list shows:
crawl-66-249-73-82.googlebot.com   1   
136   crawl-66-249-73-84.googlebot.com   1   
119   crawl-66-249-66-128.googlebot.com   1   
120   crawl-66-249-66-130.googlebot.com   1   
121   crawl-66-249-66-159.googlebot.com   1   
122   crawl-66-249-66-214.googlebot.com   1   
144   crawl-66-249-66-87.googlebot.com   1   
117   crawl-66-249-69-210.googlebot.com   1   
118   crawl-66-249-69-212.googlebot.com   1   
124   crawl-66-249-69-214.googlebot.com   1   
128   crawl-66-249-69-50.googlebot.com   1   
134   crawl-66-249-73-86.googlebot.com   1   
143   crawl-66-249-75-116.googlebot.com   1   
123   crawl-66-249-79-20.googlebot.com   1   
78   googlebot.com
Title: Re: Blacklist not blocking BOT
Post by: Randem on August 11, 2019, 09:43:20 AM
You only need one entry googlebot.com for all those entries... No need for the rest.
Title: Re: Blacklist not blocking BOT
Post by: sonnyh on August 11, 2019, 10:17:02 AM
Thank you
Title: Re: Blacklist not blocking BOT
Post by: sonnyh on August 12, 2019, 02:22:26 AM
Good Morning,
Spiders listed in the "Whos on line"
crawl-66-249-66-159.googlebot.com
crawl-66-249-66-128.googlebot.com

I have:
Blacklist Domain: googlebot.com
Blacklist Spider: googlebot
Title: Re: Blacklist not blocking BOT
Post by: Randem on August 12, 2019, 07:37:42 AM
Have you taken GoggleBot out of the Whitelists? The Whitelists override the Blacklists.
To Block Google:

Remove GoogleBot from ANY Whitelist.
Place GoggleBot, Google in the Spider BlackList.
Place GoogleBot.com, google.com in the Domain Blacklist
Place 66.249. in the IP Blacklist.

You may not need all of the above to accomplish the task.
Title: Re: Blacklist not blocking BOT
Post by: sonnyh on August 12, 2019, 09:59:12 AM
I made the changes.
Title: Re: Blacklist not blocking BOT
Post by: sonnyh on August 14, 2019, 02:25:05 AM
More questions:

We are seeing additional bots and unwanted foreign searches:
crawl11.bl.semrush.com  (semrush.com was in the Domain Blacklist, but not blocked)  are the names case sensitive?

China Telecom Anhui - 113.203.192.0 - 116.201.255.255 (is there an easy to block this IP range?)

Iran - 94.101.140.0 - 94.101.141.255 (Same question as above)
Title: Re: Blacklist not blocking BOT
Post by: Randem on August 14, 2019, 09:36:30 AM
Sometime you need more than one blacklist item for the bots can spoof just about anything. You need to add semrush to the Spider blacklist. The names are not case sensitive. For the Iran blocking of IP's use 94.101. in the IP blacklist. As far as China, that is a large area to block. You would need several IP ranges to block that (113. , 114. 115. 116.) or you may find a domain to block which would be easier (EX. something.ch, where .ch would be the ext from china).
Title: Re: Blacklist not blocking BOT
Post by: sonnyh on August 15, 2019, 02:00:04 AM
Thank you
Title: Re: Blacklist not blocking BOT
Post by: Randem on August 15, 2019, 09:11:15 AM
On second thought on the Iran IP addresses you should only put 94.101.140. and 94.101.141. in the IP blacklist. 94.101. may be too wide a range to block.
Title: Re: Blacklist not blocking BOT
Post by: sonnyh on August 15, 2019, 11:11:44 AM
Thanks, I made the correction
Text only | Text with Images