Page 2 of 2
Re: Is Google excluded from crawling this board?
Posted: Mon Aug 18, 2025 9:17 am
by Fred
I missed this one, should be good now, thanks !
Re: Is Google excluded from crawling this board?
Posted: Tue Aug 19, 2025 2:59 am
by Rinzwind
Fred wrote: Mon Aug 18, 2025 9:17 am
I missed this one, should be good now, thanks !
Do you still have ip blocks in placd? If so please check that these ranges are allowed:
https://openai.com/searchbot.json
https://openai.com/chatgpt-user.json
https://openai.com/gptbot.json
From
https://platform.openai.com/docs/bots/o ... i-crawlers
There's also a more fine tuned third party github page:
https://github.com/FabrizioCafolla/open ... es-all.txt
Because OpenAI still has randomly issues browsing the website. Checked with
https://www.purebasic.com/documentation/ and
https://www.purebasic.fr/english/
Also have case open with them. But I suspect now one of the browsertool ip's is on your blocklist? Could you check? Thanks.
edit: at the moment it can't access
https://www.purebasic.fr/english/, but can access
https://www.purebasic.com/documentation
Re: Is Google excluded from crawling this board?
Posted: Wed Aug 20, 2025 2:31 am
by Rinzwind
Re: Is Google excluded from crawling this board?
Posted: Wed Aug 20, 2025 8:18 am
by Fred
There is nothing in robot.txt anymore for the english board
Re: Is Google excluded from crawling this board?
Posted: Wed Aug 20, 2025 8:28 am
by Rinzwind
Fred wrote: Wed Aug 20, 2025 8:18 am
There is nothing in robot.txt anymore for the english board
I know, but do you have IP ranges blocked or user-agents?
Re: Is Google excluded from crawling this board?
Posted: Wed Aug 20, 2025 8:29 am
by Fred
No all is similar between the forums
Re: Is Google excluded from crawling this board?
Posted: Wed Aug 20, 2025 8:31 am
by Rinzwind
Fred wrote: Wed Aug 20, 2025 8:29 am
No all is similar between the forums
OAI-SearchBot
Full user-agent string will contain ; OAI-SearchBot/1.0; +
https://openai.com/searchbot
ChatGPT-User
Full user-agent string: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +
https://openai.com/bot
GPTBot
Full user-agent string: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.1; +
https://openai.com/gptbot
Re: Is Google excluded from crawling this board?
Posted: Wed Aug 20, 2025 11:47 am
by Olli
Thank you fred and all the PBTeam which responds quickly.
Using ownly Google search systematically and manually to reach the forum, I observed only one cut off, which has been completely and quickly solved.
I appreciate a lot the human side of the forum.
Thank you.