Is Google excluded from crawling this board?

[Fred]

I missed this one, should be good now, thanks !

[Rinzwind]

I missed this one, should be good now, thanks !

Do you still have ip blocks in placd? If so please check that these ranges are allowed:

https://openai.com/searchbot.json
https://openai.com/chatgpt-user.json
https://openai.com/gptbot.json

From https://platform.openai.com/docs/bots/o ... i-crawlers

There's also a more fine tuned third party github page:
https://github.com/FabrizioCafolla/open ... es-all.txt

Because OpenAI still has randomly issues browsing the website. Checked with https://www.purebasic.com/documentation/ and https://www.purebasic.fr/english/

Also have case open with them. But I suspect now one of the browsertool ip's is on your blocklist? Could you check? Thanks.

edit: at the moment it can't access https://www.purebasic.fr/english/, but can access https://www.purebasic.com/documentation

[Rinzwind]

Update
ChatGPT user browsertool can accesss
http://www.purebasic.fr/german
http://www.purebasic.fr/french
https://www.purebasic.com/documentation/

But can NOT access
https://www.purebasic.fr/english/


Claude can btw.

[Fred]

There is nothing in robot.txt anymore for the english board

[Rinzwind]

There is nothing in robot.txt anymore for the english board

I know, but do you have IP ranges blocked or user-agents?

[Fred]

No all is similar between the forums

[Rinzwind]

No all is similar between the forums

OAI-SearchBot
Full user-agent string will contain ; OAI-SearchBot/1.0; +https://openai.com/searchbot

ChatGPT-User
Full user-agent string: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot

GPTBot
Full user-agent string: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.1; +https://openai.com/gptbot

[Olli]

Thank you fred and all the PBTeam which responds quickly.

Using ownly Google search systematically and manually to reach the forum, I observed only one cut off, which has been completely and quickly solved.

I appreciate a lot the human side of the forum.
Thank you.