Page 8 of 17

Re: Chromium Browser WebGadget

Posted: Mon Jul 04, 2022 10:11 pm
by skinkairewalker
Hello, I found the project interesting, but I have some doubts...

*can I export as a standalone executable?
*Is it mandatory to install external dependencies in order to work?
*consume a lot of ram?

Re: Chromium Browser WebGadget

Posted: Tue Jul 05, 2022 5:17 pm
by JHPJHP
Hi skinkairewalker,
skinkairewalker wrote:*can I export as a standalone executable?
Free executables (x86/x64) can be downloaded from My PureBasic Stuff; see FREE STUFF.
skinkairewalker wrote:*Is it mandatory to install external dependencies in order to work?
No external dependencies other then a supported browser...
The WebGadget currently works with Edge and Chrome, but other Chromium based browsers could be added.
The Standalone application currently works with Edge, Chrome, Opera, Vivaldi, Brave, Epic, but other Chromium based browsers could be added.
skinkairewalker wrote:*consume a lot of ram?
No, but test the free executables for yourself.

Re: Chromium Browser WebGadget

Posted: Thu Jul 07, 2022 3:19 pm
by JHPJHP
Updated:
- added some error checking
- small improvement to Headless mode

Error Checking
Previously, the scripts where JSON is parsed had the potential to cause a crash, this has now been fixed.

Re: Chromium WebGadget

Posted: Sat Jul 09, 2022 10:57 pm
by JHPJHP
Updated:
- improved CDP Control Panel
- improved CBWG DevTools examples
- fixed a couple things

CDP Control Panel
1. Run the example CDP_Control_Panel.pb.
2. Press the Select Browser button.
3. Select an available browser.
4. Press the Listen Port button.
5. Press the Connect DevTools button.
6. Right-mouse-click the Command List.
7. Select the Load Command List... option.
8. Select the file compile.json.
9. Right-mouse-click the Command List.
10. Select the Send All option.
11. On the webpage enter jhpjhp in the Search field.

- Navigates to the PureBasic forum.
- Compiles some user-defined JavaScript (includes an event listener)
- Includes the compiled JavaScript into the PureBasic webpage.
- When jhpjhp is entered in the search field all the forum titles are changed to JHPJHP.

CBWG DevTools
1. Run the example CB_WebGadget_Detail.pb.
2. Select the Page.navigate ComboBox item.
3. Press the Send Command button.
4. Select the DOM.getNodeForLocation ComboBox item.
5. Manually scroll the webpage to any location.
6. Put the mouse-pointer over any hyperlink.
7. Press the Insert key to update parameters.
8. Press the Send Command button.

- Returns the Layout Metrics from the webpage.
- Finds the X/Y scroll position from the Layout Metrics.
- Returns the X/Y mouse location (Insert key Procedure).
- Calculates the combined scroll position and mouse location.
- Returns the Node ID under the mouse pointer using the calculation.
- Using the returned Node ID extracts the outer HTML associated to the node.

Image

Re: Chromium WebGadget

Posted: Mon Jul 11, 2022 3:12 pm
by JHPJHP
Updated:
- improved CDP Control Panel
-- added DevTools Protocol help / filter (F1, F2, F3, Esc)
- minor improvements to DevTools examples

Early on there was some interest regarding the Chromium WebGadget, but at that time only 20 plus hours were invested, and most of the WebGadget was experimental. Very little of that earlier version can be found in the latest examples.

Suffice to say, if you're doing any programming that requires access to elements of a webpage, you should be familiar with the DevTools Protocol.

Currently, 400 plus hours have been spent writing the Chromium WebGadget, and at least half of that time was dedicated to understanding and incorporating the DevTools Protocol into the CB WebGadget and standalone Control Panel.

NB*: See previous post for some basic instructions.

Re: Chromium WebGadget

Posted: Wed Jul 13, 2022 2:12 am
by JHPJHP
Updated (CDP Contol Panel)
- added support for the Maxthon browser
- various minor improvements

Re: Chromium WebGadget

Posted: Fri Jul 15, 2022 10:39 pm
by JHPJHP
Updated:
- added URL blocking support
- various minor improvements
- squashed a small bug

URL Blocking Support
Created a JSON file to demonstrate how easy it is to block URLs using the DevTools Protocol.
Includes event driven script prompting the user to decline or allow access to the URL.

-------------------------------------------------------

Updated:
- extended URL blocking support to return document body
- added URL blocking support to CB_WebGadget_Detail.pb
- squashed a bug

CB_WebGadget_Detail.pb
The document body is now returned during the Fetch request allowing for editing before being passed to the browser.

1. Execute the Fetch.enable command.
2. Execute the Page.navigate command.
3. If the page is allowed to load, the word Windows is substituted with JHPJHP.

Because the document body was larger then 65536 bytes, this was the first time I was able to test my WebSocket (send frame) fragmentation script; hence the squashed bug.

Re: Chromium WebGadget

Posted: Sun Jul 17, 2022 6:20 pm
by JHPJHP
Updated:
- improved page connections
- various minor improvements
- better error checking

If you're using the framework or the standalone tool this is a must download as it provides better support for page connections.

DevTools Examples
Previously it was the programmers responsibility to track when a new tab or window was opened and handle accordingly.
Event script has been added to stop new tabs and windows from opening, redirecting the current page to the selected URL.

CDP Control Panel (standalone tool)
Added a window listing multiple page connections.
Pressing the Port Listening button will either populate the Page ID field immediately (single connection), or open the Page Data window when multiple connections are detected.

Re: Chromium WebGadget

Posted: Tue Jul 19, 2022 5:28 pm
by JHPJHP
Updated:
- added new DevTools example
- couple minor improvements

CB_WebGadget_ChartJS.pb
Line Chart Stacked was copied from the ChartJS Chromium example, and with a few additional lines of code the DevTools Protocol was added. This provided PureBasic the ability to execute JavaScript and manipulate the line chart; see image below.

I was lazy using only the PureBasic ButtonGadget, but the example does include everything needed to create interactive charts.

Using script from the nine charts included in the original Chromium example, combined with the DevTools script included in the new example, should provide enough information to create any of the ChartJS samples with complete JavaScript support.

Something Fun To try (remote controlling a WebGadget)
Run in parallel the CDP Control Panel application. Ignore the Select Browser button and only press the Listen Port button. After establishing a DevTools WebSocket connection to the ChartJS example, any commands sent from the Control Panel will be executed in the WebGadget.

Image

Re: Chromium WebGadget

Posted: Thu Jul 21, 2022 6:56 pm
by JHPJHP
Updated:
- improved Page handling
- improved DevTools help
- added browser check
- modified readme.txt

Page Handling (DevTools examples)
Noticed that some pages were not being redirected due to an empty returned URL parameter; Target.targetInfoChanged event was added.

DevTools Help (standalone CDP Control Panel)
The Protocol is now filtered on the ComboBoxGadget Changed event.
The filter button now returns the Method, provided both Domain and Command have been selected.

Re: Chromium WebGadget

Posted: Fri Jul 22, 2022 3:29 pm
by skinkairewalker
it would be wonderful, if you could port Chromium WebGadget to Linux & Mac too...

Re: Chromium WebGadget

Posted: Sat Jul 23, 2022 5:51 am
by JHPJHP
Hi skinkairewalker,
skinkairewalker wrote:it would be wonderful, if you could port Chromium WebGadget to Linux & Mac too...
Sorry, it's not something I'm planning, especially with the poor forum response. But, all of the CDP Procedures should be cross-platform...
So, I am considering a limited cross-platform tool with similar functionality to the CDP Control Panel.

----------------------------------

Updated:
- couple minor improvements
- rearranged some of the code

Re: Chromium WebGadget

Posted: Sun Jul 24, 2022 7:54 am
by zikitrake
I currently use PB.Ex.Webgadget (thank you very much, @RSBASIC) to automate data entry and extraction on my work intranet.

I've been following this project for months and I think it's time to convert my small automators and not depend on CEF Sharp / .NET libraries.

Back from holydays I hope to be able to do the conversion without too much trouble :)

In short, thank you very much for your hard work.

Re: Chromium WebGadget

Posted: Mon Jul 25, 2022 3:17 am
by JHPJHP
Hi zikitrake,

I'm glad you're taking a second look at the Chromium WebGadget, you were one of the first people to give support and provide feedback.

Please download the latest version as some Procedure names were recently changed.

---------------------------------------------------

Updated:
- name changes
- small improvements
- removed redundant code

Re: Chromium WebGadget

Posted: Mon Jul 25, 2022 11:29 pm
by zikitrake
Thank you. Done!

I've done a little test and for now I pause with Delay() to make sure that the loading of a page is finished but of course, it doesn't give me much security.

Is there any way to check that a page has actually finished loading?

For example, in PB.Ex Webgadget() there is the option GetWebGadgetExAttribute(ID, #PB_Web_Busy, @ErrorOutput$) to ensure this.

My current procedure:

Code: Select all

Procedure pbCBWEB_PageNavigation(dataUrl.s, waitFullLoad.b = #True)
  Protected SendData.SEND_DATA
  SendData\EventID = 20300
  SendData\Method = "Page.navigate"
  SendData\Params = #DQUOTE$ + "url" + #DQUOTE$ + ":" + #DQUOTE$ + dataUrl + #DQUOTE$
  
  If Not CB_WebSocket_Send(cbWEBConnection, @SendData)
    Debug "Error Sending Frame: " + Str(SendData\EventID) + " in pbCBWEB_PageNavigation(" + dataUrl + ")"
  EndIf
EndProcedure
I have added the waitFullLoad parameter, but I have no idea how to continue :lol: