Page 1 of 1
Is Google excluded from crawling this board?
Posted: Thu Jul 24, 2025 2:07 pm
by NicTheQuick
Hi Fred,
today ChrigiGee wrote this post in the German forum:
https://www.purebasic.fr/german/viewtopic.php?t=33431
After a few tests I made with Google search it appears that it no longer indexes the board properly. Could this be related to the server upgrade you made recently? Since the built-in board search is pretty poor, it would be a shame if Google no longer functioned as the search engine for the forum.
Re: Is Google excluded from crawling this board?
Posted: Thu Jul 24, 2025 5:07 pm
by Bisonte
confirmed. Nearly no post is found by google anymore.
Re: Is Google excluded from crawling this board?
Posted: Thu Jul 24, 2025 8:30 pm
by ricardo_sdl
I use duckduckgo and can't find any forum posts and links on It either. The Domain is fone from the results.
Re: Is Google excluded from crawling this board?
Posted: Thu Jul 24, 2025 9:52 pm
by Caronte3D
Maybe with the last server change the robots.txt file is blocking all bots?

Re: Is Google excluded from crawling this board?
Posted: Thu Jul 24, 2025 10:37 pm
by Quin
Seems likely yeah, I used to see Bing BOT, google BOT etc as always online users, now I never do.
Re: Is Google excluded from crawling this board?
Posted: Thu Jul 24, 2025 10:57 pm
by Olli
@fred
You should check if a malware has modified the file
robots.txt (looking after the several access dates).
https://support.google.com/webmasters/a ... -robotstxt
https://developers.google.com/search/do ... k-indexing
Remark : check also the ad system, because any
fake news appear on the page queues...

Re: Is Google excluded from crawling this board?
Posted: Thu Jul 24, 2025 11:23 pm
by idle
you can look at the contents of robots.txt
User-agent: CCBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
so maybe there's another robots.txt higher up in the path or it's being blocked by dns
Re: Is Google excluded from crawling this board?
Posted: Mon Jul 28, 2025 2:06 pm
by Fred
I removed the bot check, it should be better
Re: Is Google excluded from crawling this board?
Posted: Mon Jul 28, 2025 3:35 pm
by NicTheQuick
Fred wrote: Mon Jul 28, 2025 2:06 pm
I removed the bot check, it should be better
Thanks, Fred. Let's wait a few days until Google is up-to-date again now.
Re: Is Google excluded from crawling this board?
Posted: Mon Jul 28, 2025 9:33 pm
by Olli
Google help tells us two other ways to block it : the keyword 'noindex' in the page, or in the response (post thing).
Note that maybe you are for nothing, but a malware in the ads, that Google detected, and cut the whole site containing it.
I mail an example... (to alpha send at 22:35) As you can see, I do not think you are agree with that type of ad...
Re: Is Google excluded from crawling this board?
Posted: Tue Jul 29, 2025 5:40 am
by Olli
Purebasic is back !