Page 1 of 1

Is Google excluded from crawling this board?

Posted: Thu Jul 24, 2025 2:07 pm
by NicTheQuick
Hi Fred,

today ChrigiGee wrote this post in the German forum: https://www.purebasic.fr/german/viewtopic.php?t=33431

After a few tests I made with Google search it appears that it no longer indexes the board properly. Could this be related to the server upgrade you made recently? Since the built-in board search is pretty poor, it would be a shame if Google no longer functioned as the search engine for the forum.

Re: Is Google excluded from crawling this board?

Posted: Thu Jul 24, 2025 5:07 pm
by Bisonte
confirmed. Nearly no post is found by google anymore.

Re: Is Google excluded from crawling this board?

Posted: Thu Jul 24, 2025 8:30 pm
by ricardo_sdl
I use duckduckgo and can't find any forum posts and links on It either. The Domain is fone from the results.

Re: Is Google excluded from crawling this board?

Posted: Thu Jul 24, 2025 9:52 pm
by Caronte3D
Maybe with the last server change the robots.txt file is blocking all bots? :?:

Re: Is Google excluded from crawling this board?

Posted: Thu Jul 24, 2025 10:37 pm
by Quin
Seems likely yeah, I used to see Bing BOT, google BOT etc as always online users, now I never do.

Re: Is Google excluded from crawling this board?

Posted: Thu Jul 24, 2025 10:57 pm
by Olli
@fred

You should check if a malware has modified the file robots.txt (looking after the several access dates).

https://support.google.com/webmasters/a ... -robotstxt
https://developers.google.com/search/do ... k-indexing

Remark : check also the ad system, because any fake news appear on the page queues... :?

Re: Is Google excluded from crawling this board?

Posted: Thu Jul 24, 2025 11:23 pm
by idle
you can look at the contents of robots.txt

User-agent: CCBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

so maybe there's another robots.txt higher up in the path or it's being blocked by dns

Re: Is Google excluded from crawling this board?

Posted: Mon Jul 28, 2025 2:06 pm
by Fred
I removed the bot check, it should be better

Re: Is Google excluded from crawling this board?

Posted: Mon Jul 28, 2025 3:35 pm
by NicTheQuick
Fred wrote: Mon Jul 28, 2025 2:06 pm I removed the bot check, it should be better
Thanks, Fred. Let's wait a few days until Google is up-to-date again now.

Re: Is Google excluded from crawling this board?

Posted: Mon Jul 28, 2025 9:33 pm
by Olli
Google help tells us two other ways to block it : the keyword 'noindex' in the page, or in the response (post thing).

Note that maybe you are for nothing, but a malware in the ads, that Google detected, and cut the whole site containing it.

I mail an example... (to alpha send at 22:35) As you can see, I do not think you are agree with that type of ad...

Re: Is Google excluded from crawling this board?

Posted: Tue Jul 29, 2025 5:40 am
by Olli
Purebasic is back !