Is Google excluded from crawling this board?

[NicTheQuick]

Hi Fred,

today ChrigiGee wrote this post in the German forum: https://www.purebasic.fr/german/viewtopic.php?t=33431

After a few tests I made with Google search it appears that it no longer indexes the board properly. Could this be related to the server upgrade you made recently? Since the built-in board search is pretty poor, it would be a shame if Google no longer functioned as the search engine for the forum.

[Bisonte]

confirmed. Nearly no post is found by google anymore.

[ricardo_sdl]

I use duckduckgo and can't find any forum posts and links on It either. The Domain is fone from the results.

[Caronte3D]

Maybe with the last server change the robots.txt file is blocking all bots?

[Quin]

Seems likely yeah, I used to see Bing BOT, google BOT etc as always online users, now I never do.

[Olli]

@fred

You should check if a malware has modified the file robots.txt (looking after the several access dates).

https://support.google.com/webmasters/a ... -robotstxt
https://developers.google.com/search/do ... k-indexing

Remark : check also the ad system, because any fake news appear on the page queues...

[idle]

you can look at the contents of robots.txt

User-agent: CCBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

so maybe there's another robots.txt higher up in the path or it's being blocked by dns

[Fred]

I removed the bot check, it should be better

[NicTheQuick]

I removed the bot check, it should be better

Thanks, Fred. Let's wait a few days until Google is up-to-date again now.

[Olli]

Google help tells us two other ways to block it : the keyword 'noindex' in the page, or in the response (post thing).

Note that maybe you are for nothing, but a malware in the ads, that Google detected, and cut the whole site containing it.

I mail an example... (to alpha send at 22:35) As you can see, I do not think you are agree with that type of ad...

[Olli]

Purebasic is back !