Is Google excluded from crawling this board?

[NicTheQuick]

Hi Fred,

today ChrigiGee wrote this post in the German forum: https://www.purebasic.fr/german/viewtopic.php?t=33431

After a few tests I made with Google search it appears that it no longer indexes the board properly. Could this be related to the server upgrade you made recently? Since the built-in board search is pretty poor, it would be a shame if Google no longer functioned as the search engine for the forum.

[Bisonte]

confirmed. Nearly no post is found by google anymore.

[ricardo_sdl]

I use duckduckgo and can't find any forum posts and links on It either. The Domain is fone from the results.

[Caronte3D]

Maybe with the last server change the robots.txt file is blocking all bots?

[Quin]

Seems likely yeah, I used to see Bing BOT, google BOT etc as always online users, now I never do.

[Olli]

@fred

You should check if a malware has modified the file robots.txt (looking after the several access dates).

https://support.google.com/webmasters/a ... -robotstxt
https://developers.google.com/search/do ... k-indexing

Remark : check also the ad system, because any fake news appear on the page queues...

[idle]

you can look at the contents of robots.txt

User-agent: CCBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

so maybe there's another robots.txt higher up in the path or it's being blocked by dns

[Fred]

I removed the bot check, it should be better

[NicTheQuick]

I removed the bot check, it should be better

Thanks, Fred. Let's wait a few days until Google is up-to-date again now.

[Olli]

Google help tells us two other ways to block it : the keyword 'noindex' in the page, or in the response (post thing).

Note that maybe you are for nothing, but a malware in the ads, that Google detected, and cut the whole site containing it.

I mail an example... (to alpha send at 22:35) As you can see, I do not think you are agree with that type of ad...

[Olli]

Purebasic is back !

[spikey]

This problem seems to be back again for me this afternoon. Can anyone else confirm?

Code: Select all

PureBasic
https://www.purebasic.com
No information is available for this page.
Learn why

PureBasic Forum
PureBasic
https://forums.purebasic.com
No information is available for this page.
Learn why

Duck Duck Go seems unable to crawl it too.

[minimy]

This problem seems to be back again for me this afternoon. Can anyone else confirm?

Code: Select all

PureBasic
https://www.purebasic.com
No information is available for this page.
Learn why

PureBasic Forum
PureBasic
https://forums.purebasic.com
No information is available for this page.
Learn why

Duck Duck Go seems unable to crawl it too.

Confirmed google is .... mmm? google?
Google= No work
DuckDuckGo= Work
Bing= Work
Yahoo= Work
Ecosia= Work
Yandex= Work
Gibiru= Work
Dogpile= Work
Startpage= Work
Qwant= Work

As i said... google is google...

[Olli]

Strange to see such a treatment difference :
I go to this forum, only with Google search. And, actually, I do not see any problem...
I am curious to see anybody making a Google visibility "map" of the forum, with a VPN.

Is this message of minimy a presage of a none status in the next hours ?

[Rinzwind]

I removed the bot check, it should be better

The Robots.txt needs adjustment, that is for sure. ChatGPT can't access it:

I couldn’t fetch purebasic.com directly here (blocked by robots.txt)

Too bad, because it is the main resource of PB syntax rules and solutions. @Fred!