Sponsored

ht.Acess file on Apache server and blocking back link tools

#1
Hi there ive been on a run around with this , ive been trying to block online backlink crawlers via Ht.access in my apache server for some of my websites... i blocked them with via user agent string and ip ranges.

Thing is it seems while ip ranges work they change all the time and user agent string the bot can call itself anything even google bot if it wants , ive looked into using a whitelist , blocking everything but google itself from my website, but that seems like a foolish idea.

Yet somehow some people manage to do it i think via 301 redirects im not sure how this process works though as bots can detect redirects as i understand and follow on, ( also this seems like alot of work to set up multiple sites ) ive tried looking at some of my competitors ranked above me and they show no links at all :) I don't want their links i just want to hide mine ... am i missing something here?

Any help would be appreciated
Regards
Juan Pierre
 

Mark.J

Administrator
Staff member
ISPreview Team
#2
When you say IP ranges, do you mean the full CIDR? You could also identify the common geolocation (country) and block traffic from the whole country if it's not relevant to your site. Bad traffic also tends to come from certain server farms and if you identify the primary company supporting those then you can find all of their CIDRs and impose a blanket block.

Sadly there's no fool proof way to stop everything without also impacting legitimate traffic and IPv6 creates a new headache on this front. I don't think 301s are a totally effective solution either.
 
#3
Hi mark i seem to have made a duplicate thread , i could not find this one for some reason .... Thanks for your response .
Common geo location is india but surprisingly mostly america , ive have read a long time ago that blocking a country specifically america from your website is not a good thing with regards to google and its crawling ( even though we have google.de .za .whatever else there is.

Sadly there's no fool proof way to stop everything without also impacting legitimate traffic and IPv6 creates a new headache on this front. I don't think 301s are a totally effective solution either.
Yea this is kind of the conclusion ive come to, and having thought it over attaining my backlinks will be no pick nick either , and if the webmaster is foolish enough to try them in bulk google might flag them because ive taken months to build them and they all have unique content. I think while they do that , my time is better spent on bettering my websites position.

Although i have not been affected yet ,content scrapers and bandwidth leechers are also something i will have to learn about, do they operate in similar fashion ?
 

Mark.J

Administrator
Staff member
ISPreview Team
#4
Depends really, a lot of content scrapers use more familiar servers and so can be easily blocked via IP / CIDRs. Bandwidth leeching is more of a problem but there are .htaccess tweaks that can be used to make that more difficult and a quick Google may offer some examples.

However I wouldn't worry too much about that unless it becomes something that's a big problem, as there will always be people remote linking to your content, which to a certain degree is par-for-the-course of running a public website.
 
Top
Promotion
Cheapest Superfast ISPs
  • Hyperoptic £18.00 (*22.00)
    Avg. Speed 30Mbps, Unlimited
    Gift: Code: CHRISTMAS18
  • Onestream £19.95 (*34.99)
    Avg. Speed 35Mbps, Unlimited
    Gift: None
  • TalkTalk £22.50
    Avg. Speed 36Mbps, Unlimited
    Gift: None
  • Direct Save Telecom £22.95 (*29.95)
    Avg. Speed 35Mbps, Unlimited (FUP)
    Gift: None
  • Vodafone £23.00 (*25.00)
    Avg. Speed 35Mbps, Unlimited
    Gift: None
Prices inc. Line Rental | View All
Helpful ISP Guides and Tips
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
»
Promotion
The Top 20 Category Tags
  1. BT (2290)
  2. FTTP (1754)
  3. FTTC (1521)
  4. Broadband Delivery UK (1490)
  5. Openreach (1234)
  6. Politics (1225)
  7. Business (1093)
  8. Statistics (964)
  9. Mobile Broadband (885)
  10. Fibre Optic (881)
  11. FTTH (818)
  12. Ofcom Regulation (811)
  13. Wireless Internet (806)
  14. 4G (768)
  15. Virgin Media (742)
  16. Sky Broadband (546)
  17. TalkTalk (525)
  18. EE (507)
  19. Vodafone (396)
  20. Security (371)
Sponsored

Copyright © 1999 to Present - ISPreview.co.uk - All Rights Reserved - Terms  ,  Privacy and Cookie Policy  ,  Links  ,  Website Rules