PureBasic Forums - English

http://www.purebasic.com
https://www.purebasic.fr/english/

Website page and button click (with a WebGadget?)

https://www.purebasic.fr/english/viewtopic.php?t=85248

Page 1 of 1

Website page and button click (with a WebGadget?)

Posted: Mon Sep 02, 2024 10:51 pm
by boddhi
Hello,

I apologize in advance if I can't provide any code because I have a specific need for a site containing personal information and I haven't found any other sites with such behavior (of course, that doesn't mean there aren't any, but I don't know of any that don't work this way).
I'll do my best to explain what I want to do.

I'm also completely unfamiliar with the use of WebGadget.

So here are my two needs:
1) In classic use (i.e: with any browser), I search for data on the site according to certain criteria. The first 20 data are displayed, followed by a button offering to display the next 20 data in succession (i.e: always on the same page), and so on.
With PB, I tried with HTTPRequest, modifying the parameters for the following data each time (i.e. having the same link as the one proposed by the button) but it didn't work, the request always returned the first 20 data.
My first need would therefore be to automate the click on the button so as to be able to display all the data. Possibly in a WebGadget if that's the best way to succeed.
2) And secondly, to be able to retrieve the final page's html source code so that I can process and retrieve the information I'm interested in.

How can I do this?
Thanks for your help.

Re: Website page and button click (with a WebGadget?)

Posted: Tue Sep 03, 2024 7:16 am
by infratec
Your first way is right, but .... you do something wrong.

If you can not provide the original side, write your own dem.html which does the same.

Re: Website page and button click (with a WebGadget?)

Posted: Tue Sep 03, 2024 1:04 pm
by boddhi
Hello Infratec,

Thanks for your reply.
infratec wrote: Your first way is right, but ....you do something wrong.
Maybe... certainly :mrgreen:
With a classic browser, when I copy the button link and past it in a new tab, it's the same result.
After inspection of its HTML source code, it is constructed as follows:

Code: Select all

<a onClick="tagDeClick({elem:this, level2:'91', name:'Show_more_offers', chapter1:'Offers_list', type:'navigation'})" 
	data-async-trigger="true" role="button" class="btn btn-primary" 
	href="/offers/search.searchoffer:showmoreresults/20-39/1?location=75000&amp;
	keywords=something%C3%A9&amp;PartnerOffers=true&amp;radius=20&amp;sort=1">
	Show the 20 following offers
</a>
On mouseover, it looks like this:

Code: Select all

https://website.com/offers/search.offersearch:showmoreresults/[color=#FF0000]20-39/[/color]1?locations=75000&keywords=something&PartnerOffers=true&radius=20&sort=1
Since I don't know anything about javascript, could it be that there's an injection (tagDeClick() ?), causing an alteration of the link, between the moment I click on the button and the moment the page is refreshed? Because I've noticed that afterwards, the address in the navigation bar isn't quite the same. In fact, some elements are missing, e.g for the example given, the range '20-39'... corresponding to the following 20 data items requested.

Code: Select all

https://website.com/offers/search?locations=75000&keywords=something&PartnerOffers=true&radius=20&sort=1
:oops:

Re: Website page and button click (with a WebGadget?)

Posted: Tue Sep 03, 2024 11:55 pm
by boddhi
Let's imagine another situation:
Consider a website built like this:

Code: Select all

Global Window_1
Global Frame_0, Frame_1, ScrollArea_0, Frame_3, Frame_4, Frame_5, Frame_6, Frame_7, Frame_8, Frame_9, Frame_10, Frame_11, Frame_12

Window_1 = OpenWindow(#PB_Any, 20, 20, 1072, 456, "", #PB_Window_SystemMenu)
Frame_0 = FrameGadget(#PB_Any, 0, 0, 1072, 104, "Header section")
Frame_1 = FrameGadget(#PB_Any, 0, 104, 272, 352, "Aside section")
ScrollArea_0 = ScrollAreaGadget(#PB_Any, 280, 112, 776, 336, 300, 730, 1)
Frame_3 = FrameGadget(#PB_Any, 0, 0, 760, 72, "Block 1")
Frame_4 = FrameGadget(#PB_Any, 0, 73, 760, 72, "Block 2")
Frame_5 = FrameGadget(#PB_Any, 0, 146, 760, 72, "Block 3")
Frame_6 = FrameGadget(#PB_Any, 0, 219, 760, 72, "Block 4")
Frame_7 = FrameGadget(#PB_Any, 0, 292, 760, 72, "Block 5")
Frame_8 = FrameGadget(#PB_Any, 0, 365, 760, 72, "New block 6 added")
Frame_9 = FrameGadget(#PB_Any, 0, 438, 760, 72, "New block 7 added")
Frame_10 = FrameGadget(#PB_Any, 0, 511, 760, 72, "New block 8 added")
Frame_11 = FrameGadget(#PB_Any, 0, 584, 760, 72, "New block 9 added")
Frame_12 = FrameGadget(#PB_Any, 0, 657, 760, 72, "New block X added")
CloseGadgetList()

Repeat
Until WaitWindowEvent() = #PB_Event_CloseWindow
Independently of the browser's vscrollbar, the page contains a ScrollAreaGadget container. Each time the vscrollbar of this container is scrolled, new data is added to the initial ones.

How can I simulate the scrolling of this vscrollbar to its maximum so that I can retrieve all the data via an HTTPRequest? Is this even possible?

The HTML source code of the container and the first article.

Code: Select all

<div _ngcontent-offer-front-c2775876888="" data-cs-capture="" data-cs-scroll-container="" class="h-100 cc-scrollable-content-y">
  <mat-expansion-panel _ngcontent-offer-front-c2775876888="" id="map-expansion-panel" class="mat-expansion-panel ng-tns-c857250080-0">
    <div role="region" class="mat-expansion-panel-content ng-trigger ng-trigger-bodyExpansion" id="cdk-accordion-child-0" aria-labelledby="mat-expansion-panel-header-0" style="height: 0px; visibility: hidden;">
      <div class="mat-expansion-panel-body">
        <!----><!---->
      </div>
    </div>
  </mat-expansion-panel>
  <!----><!---->
  <div _ngcontent-offer-front-c2775876888="" class="position-relative">
    <app-offer-list _ngcontent-offer-front-c2775876888="" id="app-offers" class="cc-matching-list" _nghost-offer-front-c1241593182="">
      <ul _ngcontent-offer-front-c1241593182="" role="tree" tabindex="0" class="list-group">
        <!---->
        <li _ngcontent-offer-front-c1241593182="" role="treeitem" class="list-group-item cc-list-group-item cc-border-none cc-radius-0 p-4 cc-cursor-pointer already-viewed cc-bg-gray-bright-hover" id="offer-40117079">
          <app-offer-list-item _ngcontent-offer-front-c1241593182="" _nghost-offer-front-c807488702="" class="cc-offer-list-item">
            <article _ngcontent-offer-front-c807488702="" class="d-flex cc-offer">
              <cc-offer-logo _ngcontent-offer-front-c807488702="" class="d-none d-lg-flex align-items-center mr-4 cc-font-size-xlarge cc-offer-logo" _nghost-offer-front-c2192531554="">
                <a _ngcontent-offer-front-c2192531554="" class="logo text-center truncated-text-fix d-flex align-items-center cc-font-weight-headings" title="See all X's offers">
                  <img _ngcontent-offer-front-c2192531554="" data-container=".mj-offer" title="Society : X" alt="X" src="/images/3/0/0/3_IMAGE_SMALL.png?1567783575000" class="ng-star-inserted"><!----><!---->
                </a>
                <!----><!---->
              </cc-offer-logo>
              <div _ngcontent-offer-front-c807488702="" class="truncated-text-fix d-flex cc-flex-1 justify-content-between has-logo">
                <div _ngcontent-offer-front-c807488702="" class="d-flex cc-flex-1 flex-column w-100 flex-lg-row flex-lg-wrap">
                  <div _ngcontent-offer-front-c807488702="" class="truncated-text-fix d-flex flex-column align-self-lg-center cc-offer-main-content">
                    <div _ngcontent-offer-front-c807488702="" class="mb-1">
                      <!---->
                      <div _ngcontent-offer-front-c807488702="" class="d-inline-block mr-3">
                        <span _ngcontent-offer-front-c807488702="" class="cc-font-size-small align-text-top mb-0">
                          <p _ngcontent-offer-front-c807488702="" class="cc-font-size-small align-text-top mb-0">
                            <span _ngcontent-offer-front-c807488702="" class="ft ft-clock mr-1"></span>
                            3 days ago
                             <span _ngcontent-offer-front-c807488702="" class="cc-font-size-small font-weight-bold cc-font-primary align-text-top mb-0 ml-3">New</span>
                             <!---->
                          </p>
                        </span>
                      </div>
                      <!---->
                      <div _ngcontent-offer-front-c807488702="" class="d-inline-block">
                        <p _ngcontent-offer-front-c807488702="" class="cc-font-size-small align-text-top mb-0"><span _ngcontent-offer-front-c807488702="" aria-hidden="true" class="fas fa-bolt cc-font-primary mr-1"></span></p>
                      </div>
                      <!---->
                    </div>
                    <h2 _ngcontent-offer-front-c807488702="" class="cc-cursor-pointer text-truncate cc-offer-title cc-offer-title-hover cc-font-size-large mb-0 mt-1">Merchandise X</h2>
                    <p _ngcontent-offer-front-c807488702="" class="d-inline-block mt-1 mb-0" id="40117079-company-name">Manpower</p>
                    <!----><!---->
                  </div>
                  </div>
                </div>
              </div>
            </article>
            <article>
              <!---->
            </article>
          </app-offer-list-item></app-offer-list-item>
        </li>
      </ul>
    </app-offer-list>
  </div>
</div>
 
Does anyone have any ideas?
Thanks.
All times are UTC+01:00
Page 1 of 1

Powered by phpBB® Forum Software © phpBB Limited