Page 2 of 2

Re: Is Google excluded from crawling this board?

Posted: Mon Aug 18, 2025 9:17 am
by Fred
I missed this one, should be good now, thanks !

Re: Is Google excluded from crawling this board?

Posted: Tue Aug 19, 2025 2:59 am
by Rinzwind
Fred wrote: Mon Aug 18, 2025 9:17 am I missed this one, should be good now, thanks !
Do you still have ip blocks in placd? If so please check that these ranges are allowed:

https://openai.com/searchbot.json
https://openai.com/chatgpt-user.json
https://openai.com/gptbot.json

From https://platform.openai.com/docs/bots/o ... i-crawlers

There's also a more fine tuned third party github page:
https://github.com/FabrizioCafolla/open ... es-all.txt

Because OpenAI still has randomly issues browsing the website. Checked with https://www.purebasic.com/documentation/ and https://www.purebasic.fr/english/

Also have case open with them. But I suspect now one of the browsertool ip's is on your blocklist? Could you check? Thanks.

edit: at the moment it can't access https://www.purebasic.fr/english/, but can access https://www.purebasic.com/documentation

Re: Is Google excluded from crawling this board?

Posted: Wed Aug 20, 2025 2:31 am
by Rinzwind

Re: Is Google excluded from crawling this board?

Posted: Wed Aug 20, 2025 8:18 am
by Fred
There is nothing in robot.txt anymore for the english board

Re: Is Google excluded from crawling this board?

Posted: Wed Aug 20, 2025 8:28 am
by Rinzwind
Fred wrote: Wed Aug 20, 2025 8:18 am There is nothing in robot.txt anymore for the english board
I know, but do you have IP ranges blocked or user-agents?

Re: Is Google excluded from crawling this board?

Posted: Wed Aug 20, 2025 8:29 am
by Fred
No all is similar between the forums

Re: Is Google excluded from crawling this board?

Posted: Wed Aug 20, 2025 8:31 am
by Rinzwind
Fred wrote: Wed Aug 20, 2025 8:29 am No all is similar between the forums
OAI-SearchBot
Full user-agent string will contain ; OAI-SearchBot/1.0; +https://openai.com/searchbot

ChatGPT-User
Full user-agent string: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot

GPTBot
Full user-agent string: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.1; +https://openai.com/gptbot

Re: Is Google excluded from crawling this board?

Posted: Wed Aug 20, 2025 11:47 am
by Olli
Thank you fred and all the PBTeam which responds quickly.

Using ownly Google search systematically and manually to reach the forum, I observed only one cut off, which has been completely and quickly solved.

I appreciate a lot the human side of the forum.
Thank you.