Driving a Web Browser with PureBasic

Just starting out? Need help? Post your questions and find answers here.
User avatar
Kwai chang caine
Always Here
Always Here
Posts: 5342
Joined: Sun Nov 05, 2006 11:42 pm
Location: Lyon - France

Re: Driving a Web Browser with PureBasic

Post by Kwai chang caine »

Hi, sorry for slow reply
Never mind, i have just thinking we are two to abandon :lol:
finally, I got an example for Firefox Portable :)
Then here !!!! :shock:
Even if you come in two month, i welcome you with the arms great open !!! :D
For now, It only works with Click Elements
You are smart...get around the problem, by using EXECUTE and JS....is really a great TWIRL 8)

Image

Thanks a lot 8)
Thanks to you, i move and have again a little bit of hope now 8) 8)

Me too...i have try to use EXECUTE, without succes :oops:
Now you have show me how use EXECUTE, it's the highway to the succes….because if we can inject JS... we can nearly all do :wink:

Just one thing…in the Webdriver_DesiredCapabilitiesAdd() procedure, i Believe you have missing something…. :idea:
Perhaps And param$ = "default" :lol: :lol:

Code: Select all

If param$ = "default" And param$ = "default"
Again congratulation, you are too strong and the best Webdriver programmer of this forum 8)
ImageThe happiness is a road...
Not a destination
User avatar
Kwai chang caine
Always Here
Always Here
Posts: 5342
Joined: Sun Nov 05, 2006 11:42 pm
Location: Lyon - France

Re: Driving a Web Browser with PureBasic

Post by Kwai chang caine »

ZIKITRAKE !!!!!!
I LOVE YOOOOOOOOOO !!!!!!

Image

I have try to reproduce your EXECUTE code....and

That works !!! i can click on the link :shock:

Never i can do that without you 8)
I'm so happy,... that i jump like a goat, like if i have wings that push me in my back :D

Image

Code: Select all

InitNetwork()

Macro Req(Method, Loc, Cleng)
 
 Method + " " + Loc + " HTTP/1.1" + #CRLF$ +
          "Connection: Keep-Alive" + #CRLF$ +
          "Content-Type: application/json;charset=utf-8" + #CRLF$ +
          "Accept: */*" + #CRLF$ +
          "User-Agent: Mozilla/4.0" + #CRLF$ +
          "Content-Length: " + Cleng + #CRLF$ +
          #CRLF$
 
EndMacro

Procedure.s Webdriver_Req(ConnectionID, url.s, sData.s, JsGetItem.s, JSONType = #PB_JSON_String, ReqType.s = "POST")

 Protected pReq$ = Req(ReqType, url, StringByteLength(sData, #PB_Ascii)) + sData
 SendNetworkString(ConnectionID, pReq$, #PB_Ascii)
 Protected Size = 1024 * 1024 * 10
 Protected *pRepReq = AllocateMemory(Size)
 ReceiveNetworkData(ConnectionID,*pRepReq, Size)
 Protected RepReq.s = StringField(PeekS(*pRepReq, -1, #PB_Ascii), 2, #CRLF$ + #CRLF$)
 FreeMemory(*pRepReq)
 
 Protected Jid = ParseJSON(#PB_Any, RepReq)
 
 If Jid
  
  Protected jsV  = JSONValue(Jid)
  
  If jsV
   
   JSONMember = GetJSONMember(jsV, JsGetItem)
   
   Select JSONType
     
    Case #PB_JSON_Null
    
     Debug "null"
     
    Case #PB_JSON_String
     
     Debug "string"
     
     If JSONMember
      RepReq = GetJSONString(JSONMember)
     EndIf
     
    Case #PB_JSON_Number:  Debug  "number"
     
     If JSONMember
      RepReq = Str(GetJSONInteger(JSONMember))
     EndIf 
     
    Case #PB_JSON_Boolean
     
     Debug "boolean"
     
    Case #PB_JSON_Array
     
     Debug "array"
     
    Case #PB_JSON_Object
    
     Debug "object"
     
   EndSelect
   
  EndIf
  
 Else
  
  Debug "JSON error : " + JSONErrorMessage()
  
 EndIf
 
 ProcedureReturn  RepReq
 
EndProcedure
; 
Procedure.s Webdriver_FF_GetSessionId(JSONValeur.s)

 Protected Jid = ParseJSON(#PB_Any, JSONValeur)
 
 If Jid
 
  Protected jsV  = JSONValue(Jid)
  
  If jsV
      
   JSONMember1 = GetJSONMember(jsV, "value")
   
   If JSONMember1
    
    JSONMember2 = GetJSONMember(JSONMember1, "sessionId")
    
    If JSONMember2
     SessionId$ = GetJSONString(JSONMember2)
    Else
     SessionId$ = "The member ''SessionId'' is Not found."
    EndIf
    
   Else
      
    SessionId$ = "The member ''Value'' is not found."  
  
   EndIf
   
  Else
  
   Debug "JSON error : " + JSONErrorMessage()
   
  EndIf
  
  ProcedureReturn SessionId$
  
 EndIf
 
EndProcedure

default_driver_Port = 4444
driver_location$ = "geckodriver.exe"
Firefox.s = "H:/A/J/C/O/Firefox50.1.0/App/Firefox/Firefox.exe"

Compilateur = RunProgram(driver_location$ ,"-v --binary " + Chr(34) + ReplaceString(Firefox, "/", "\") + Chr(34) ,"",#PB_Program_Open)
Cid = OpenNetworkConnection("127.0.0.1", default_driver_Port)
 
Url$ = "{"
Url$ +   "|capabilities|: {"
Url$ +             "|browserName|: |firefox|,"
Url$ +             "|javascriptEnabled|: true,"
Url$ +             "|acceptInsecureCerts|: true,"
Url$ +             "|cssSelectorsEnabled|: true,"
Url$ +             "|moz:webdriverClick|: true,"
Url$ +             "|moz:firefoxOptions|: {"
Url$ +                 "|binary|: |" + Firefox + "|"
Url$ +             "}"
Url$ +     "}"
Url$ + "}"

Answer$ = Webdriver_Req(Cid, "/session", ReplaceString(Url$, "|", Chr(34)), "")
Debug Answer$
SessionId$ = Webdriver_FF_GetSessionId(Answer$)
Debug SessionId$

; Aller à la page Pure basic
json$ = "{|url|:|https://www.purebasic.com/|}"
json$ = ReplaceString(json$, "|", Chr(34))
Webdriver_Req(Cid, "/session/" + SessionId$ + "/url", json$, "status", #PB_JSON_Number)

;Cliquer sur le lien des Screenshot
xpath.s = "/html/body/table[1]/tbody/tr[2]/td/table/tbody/tr/td[7]/a"
json$ = "{"
json$ + "|using|:|xpath|,"
json$ + "|value|:|" + xpath + "|"
json$ + "}"
json$ = ReplaceString(json$, "|", Chr(34))
res$ = Webdriver_Req(Cid, "/session/" + sessionID$ + "/element", json$, "value", #PB_JSON_Object, "POST")
Jid = ParseJSON(#PB_Any, res$)
jsV  = JSONValue(Jid)
ObjectValue = GetJSONMember(jsV, "value")
MemberKey$ = JSONMemberKey(ObjectValue)
MemberValue = JSONMemberValue(ObjectValue)
MemberValue$ = GetJSONString(MemberValue)

Js$ = "document.evaluate('" + xpath + "', document, null, XPathResult.FIRST_ORDERED_NODE_TYPE, null).singleNodeValue.click();"

json$ = "{"
json$ + "|script|:|" + Js$ + "|,"
json$ + "|args|: [{|ELEMENT|:|" + MemberValue$ + "|}]"
json$ + "}"
json$ = ReplaceString(json$, "|", Chr(34))
   
res$ = Webdriver_Req(Cid, "/session/" + sessionID$ + "/execute/sync" + Async$, json$, "", #PB_JSON_Object)

CloseNetworkConnection(Cid)
CloseProgram(Compilateur)
One thousand of thanks 8)
If you find why the native "/click" function not works....don't hesitate :mrgreen:

Have the best day of the world...
ImageThe happiness is a road...
Not a destination
zikitrake
Addict
Addict
Posts: 834
Joined: Thu Mar 25, 2004 2:15 pm
Location: Spain

Re: Driving a Web Browser with PureBasic

Post by zikitrake »

duplicated
Last edited by zikitrake on Thu Nov 01, 2018 3:33 pm, edited 1 time in total.
zikitrake
Addict
Addict
Posts: 834
Joined: Thu Mar 25, 2004 2:15 pm
Location: Spain

Re: Driving a Web Browser with PureBasic

Post by zikitrake »

:lol: :lol: :lol:
Kwai chang caine wrote:...Just one thing…in the Webdriver_DesiredCapabilitiesAdd() procedure, i Believe you have missing something…. :idea:
Perhaps And param$ = "default" :lol: :lol:

Code: Select all

If param$ = "default" And param$ = "default"
:oops: Fixed, should be If param$ = "default" And value$ = "default"
Kwai chang caine wrote:Again congratulation, you are too strong and the best Webdriver programmer of this forum 8)
:lol: :lol: :lol: Not at all! I am a code mixer, I took advantage of the code posted by CELTIC88 and continued searching in Google for "Webdriver click example", "Webdriver get text example" and so on... and I start reading between the first 10-20 results :D
Kwai chang caine wrote:If you find why the native "/click" function not works....don't hesitate :mrgreen:
of course!!!!

Have a nice day too
User avatar
Kwai chang caine
Always Here
Always Here
Posts: 5342
Joined: Sun Nov 05, 2006 11:42 pm
Location: Lyon - France

Re: Driving a Web Browser with PureBasic

Post by Kwai chang caine »

and I start reading between the first 10-20 results
Me too, i do that...and often i read all the google result without succes :|
But my problem is surely here....

Image

Then the other MASTERS members have rather this :mrgreen:

Image

You have see how the big brains of JS, have decided to manage the XPATH function for only a little click ? :shock:

Code: Select all

Js$ = "document.evaluate('/html/body/table[1]/tbody/tr[2]/td/table/tbody/tr/td[7]/a', document, null, XPathResult.FIRST_ORDERED_NODE_TYPE, null).singleNodeValue.click();"
Seriously.....i always say it's evident that FRED is not in the team :lol: :lol: :lol:

Even under torture,

Image

i can't scream a sentence like this :mrgreen:
ImageThe happiness is a road...
Not a destination
User avatar
Kwai chang caine
Always Here
Always Here
Posts: 5342
Joined: Sun Nov 05, 2006 11:42 pm
Location: Lyon - France

Re: Driving a Web Browser with PureBasic

Post by Kwai chang caine »

zikitrake !!!!!!!!
Never you believe me !!!!


Image

I did not leave the case, and continue to search....
This is the firefoxFox native clic :mrgreen:

Code: Select all

InitNetwork()

Macro Req(Method, Loc, Cleng)
 
 Method + " " + Loc + " HTTP/1.1" + #CRLF$ +
          "Connection: Keep-Alive" + #CRLF$ +
          "Content-Type: application/json;charset=utf-8" + #CRLF$ +
          "Accept: */*" + #CRLF$ +
          "User-Agent: Mozilla/4.0" + #CRLF$ +
          "Content-Length: " + Cleng + #CRLF$ +
          #CRLF$
 
EndMacro

Procedure.s Webdriver_Req(ConnectionID, url.s, sData.s, JsGetItem.s, JSONType = #PB_JSON_String, ReqType.s = "POST")

 Protected pReq$ = Req(ReqType, url, StringByteLength(sData, #PB_Ascii)) + sData
 SendNetworkString(ConnectionID, pReq$, #PB_Ascii)
 Protected Size = 1024 * 1024 * 10
 Protected *pRepReq = AllocateMemory(Size)
 ReceiveNetworkData(ConnectionID,*pRepReq, Size)
 Protected RepReq.s = StringField(PeekS(*pRepReq, -1, #PB_Ascii), 2, #CRLF$ + #CRLF$)
 FreeMemory(*pRepReq)
 
 Protected Jid = ParseJSON(#PB_Any, RepReq)
 
 If Jid
 
  Protected jsV  = JSONValue(Jid)
 
  If jsV
   
   JSONMember = GetJSONMember(jsV, JsGetItem)
   
   Select JSONType
     
    Case #PB_JSON_Null
   
     Debug "null"
     
    Case #PB_JSON_String
     
     Debug "string"
     
     If JSONMember
      RepReq = GetJSONString(JSONMember)
     EndIf
     
    Case #PB_JSON_Number:  Debug  "number"
     
     If JSONMember
      RepReq = Str(GetJSONInteger(JSONMember))
     EndIf
     
    Case #PB_JSON_Boolean
     
     Debug "boolean"
     
    Case #PB_JSON_Array
     
     Debug "array"
     
    Case #PB_JSON_Object
   
     Debug "object"
     
   EndSelect
   
  EndIf
 
 Else
 
  Debug "JSON error : " + JSONErrorMessage()
 
 EndIf
 
 ProcedureReturn  RepReq
 
EndProcedure
;
Procedure.s Webdriver_FF_GetSessionId(JSONValeur.s)

 Protected Jid = ParseJSON(#PB_Any, JSONValeur)
 
 If Jid
 
  Protected jsV  = JSONValue(Jid)
 
  If jsV
     
   JSONMember1 = GetJSONMember(jsV, "value")
   
   If JSONMember1
   
    JSONMember2 = GetJSONMember(JSONMember1, "sessionId")
   
    If JSONMember2
     SessionId$ = GetJSONString(JSONMember2)
    Else
     SessionId$ = "The member ''SessionId'' is Not found."
    EndIf
   
   Else
     
    SessionId$ = "The member ''Value'' is not found." 
 
   EndIf
   
  Else
 
   Debug "JSON error : " + JSONErrorMessage()
   
  EndIf
 
  ProcedureReturn SessionId$
 
 EndIf
 
EndProcedure

default_driver_Port = 4444
driver_location$ = "geckodriver.exe"
Firefox.s = "D:/A/J/C/O/Firefox62.0.3/App/Firefox/Firefox.exe"

Compilateur = RunProgram(driver_location$ ,"-v --binary " + Chr(34) + ReplaceString(Firefox, "/", "\") + Chr(34) ,"",#PB_Program_Open)
Cid = OpenNetworkConnection("127.0.0.1", default_driver_Port)
 
Url$ = "{"
Url$ +   "|capabilities|: {"
Url$ +             "|browserName|: |firefox|,"
Url$ +             "|javascriptEnabled|: true,"
Url$ +             "|acceptInsecureCerts|: true,"
Url$ +             "|cssSelectorsEnabled|: true,"
Url$ +             "|moz:webdriverClick|: true,"
Url$ +             "|moz:firefoxOptions|: {"
Url$ +                 "|binary|: |" + Firefox + "|"
Url$ +             "}"
Url$ +     "}"
Url$ + "}"

Answer$ = Webdriver_Req(Cid, "/session", ReplaceString(Url$, "|", Chr(34)), "")
Debug Answer$
SessionId$ = Webdriver_FF_GetSessionId(Answer$)
Debug SessionId$

; Aller à la page Pure basic
json$ = "{|url|:|https://www.purebasic.com/|}"
json$ = ReplaceString(json$, "|", Chr(34))
Webdriver_Req(Cid, "/session/" + SessionId$ + "/url", json$, "status", #PB_JSON_Number)

;Cliquer sur le lien des Screenshot
xpath.s = "/html/body/table[1]/tbody/tr[2]/td/table/tbody/tr/td[7]/a"
json$ = "{"
json$ + "|using|:|xpath|,"
json$ + "|value|:|" + xpath + "|"
json$ + "}"
json$ = ReplaceString(json$, "|", Chr(34))
res$ = Webdriver_Req(Cid, "/session/" + sessionID$ + "/element", json$, "value", #PB_JSON_Object, "POST")
Jid = ParseJSON(#PB_Any, res$)
jsV  = JSONValue(Jid)
ObjectValue = GetJSONMember(jsV, "value")
MemberKey$ = JSONMemberKey(ObjectValue)
MemberValue = JSONMemberValue(ObjectValue)
MemberValue$ = GetJSONString(MemberValue)

json$ = "{"
json$ + "|id|:|" + xpath + "|"
json$ + "}"
json$ = ReplaceString(json$, "|", Chr(34))
   
Res$ = Webdriver_Req(Cid, "/session/" + SessionId$ + "/element/" + MemberValue$ + "/click", json$, "", #PB_JSON_Object)

CloseNetworkConnection(Cid)
CloseProgram(Compilateur)
Since two days, i search for natively select dropdown element
And here...it's "another pair of sleeve" :| (French expression) :wink:
Surely an history of "Css selector" but again worst, than native click, to found :?
ImageThe happiness is a road...
Not a destination
Marc56us
Addict
Addict
Posts: 1477
Joined: Sat Feb 08, 2014 3:26 pm

Re: Driving a Web Browser with PureBasic

Post by Marc56us »

Kwai chang caine wrote: Since two days, i search for natively select dropdown element
And here...it's "another pair of sleeve" :| (French expression) :wink:
Surely an history of "Css selector" but again worst, than native click, to found :?
CSS or XPath of ressource on a www page (Firefox)
here dropdown element
Image
Je hope que ça te help peut-être...
(screenshot by me)

:wink:
User avatar
Kwai chang caine
Always Here
Always Here
Posts: 5342
Joined: Sun Nov 05, 2006 11:42 pm
Location: Lyon - France

Re: Driving a Web Browser with PureBasic

Post by Kwai chang caine »

Ouaaaaahhh !!!! :shock:
You are definitively someone i put in my friends bag 8)

1/ Thanks to you, i learn FF, have an insert XPath copy :shock:
They are several month where i break my "nenete" :mrgreen: to install XPathFinder or the deprecated Firebug for found XPath :oops:
And you don't imagine how i break my "C ? L" too :mrgreen: :mrgreen: to manage in the same time, several versions of FF because sometime XpathFinder not works, etc .... :?

2/ You learned to me, also there are a "selector CSS " copy too :shock: :shock:
Because i have yesterday a glow of intelligence (Yes it happens to me sometimes :mrgreen: but the less possible, do not worry :oops:) but me, i choose "Path CSS" and see there are nothing to see, too much long expression, comparate of the Geckodriver LOG found on the web :|

I don't explain to you, how i row in this galley :|

Image

The doc is the worst i have never seen before :evil:
The examples in the web are nearly all for SELENIUM, then the code have no example in the HTTP format :?
I search, i search, i search again and sometime i found another function, little bit by little bit, i begin to create my own wrapper, but much path are again to do :|

For the moment, after several days of research, an good idea accross my head, and chance also, i have found how manage the dropdown (ComBox) :D
Put i ensure you...it's relly not simple to thinking at this method, seriously....why there are not a FRED in each team ???
In fact, i don't know how i have do for understand, :shock: but for the dropdown, you must use two steps and not one like the others controls :wink:

After i have found, for the field with Jquery autocompletion
It's a hell this "truc", you can writing easily inside, but the page at the validation, not see the text if you not enter it with the dropdown and the mouse :shock: :?
You type only one character, a dropdown appears, but it's not a real dropdown, then you can't use the basic dropdown procedure

After seraching one day....Kcc = 1 / Geckodriver = 0 :lol:

Image

but it's when even a bad hack method :oops:

"Bref", i continue alone, because on "Developpez.com" nobody is interesting
Fortunately i have you and castanets man :wink: for sometime push me in the right way 8)

Because it's the first time, where i want abandon several time, so much an apparently easy task, are finally so hard
All that not for an history of pointers, API, UNICODE, etc ...where i never understand :oops: and need the kind help of Masters of the forums 8)
But just for an history, of syntax, comma, quote or special logical of the FF team :twisted:
It's really frustrating :|

Now i search another thing.....how extract value of result
I have found cookie, but it's not really simple
Create file, but for security reason that not works always
HttpRequest() thanks to "the.weavster" 8)
But except the "/Screeshot" option who extract a B64 text, apparently Geckodriver not have something to extract "AND RETURN" text of the page or variable value, etc ...., just "source code" (and that not works perfectly for the moment, the source is not full :?) and "screenshot" :|
Surely i'm forcing another time to inject JS, with "innhertext" or "innherHtml"

Everywhere in the LOG i see "marionnet" :shock: and i wondering if it's not an exe...perhaps a new way too ???

"Bref2" it's not terminate, like you see :oops:
I search...i search !!!

Image
screenshot by me
Very very nice !!! 8) the arrows :shock:
Artist perhaps ???? :D

Again thanks my friend, for this another "coup de paluche" always usefull for little KCC lost in this ocean of difficulty 8)

Image
ImageThe happiness is a road...
Not a destination
Marc56us
Addict
Addict
Posts: 1477
Joined: Sat Feb 08, 2014 3:26 pm

Re: Driving a Web Browser with PureBasic

Post by Marc56us »

Kwai chang caine wrote:Everywhere in the LOG i see "marionnet" :shock: and i wondering if it's not an exe...perhaps a new way too ???
You're not far away:

"Marionette is an automation driver for Mozilla’s Gecko engine. It can remotely control either the UI or the internal JavaScript of a Gecko platform, such as Firefox. It can control both the chrome (i.e. menus and functions) or the content (the webpage loaded inside the browsing context), giving a high level of control and ability to replicate user actions. In addition to performing actions on the browser, Marionette can also read the properties and attributes of the DOM.

If this sounds similar to Selenium/WebDriver then you’re correct! Marionette shares much of the same ethos and API as Selenium/WebDriver, with additional commands to interact with Gecko’s chrome interface. Its goal is to replicate what Selenium does for web content: to enable the tester to have the ability to send commands to remotely control a user agent.
"

Source: https://firefox-source-docs.mozilla.org ... Intro.html

:wink:
User avatar
Kwai chang caine
Always Here
Always Here
Posts: 5342
Joined: Sun Nov 05, 2006 11:42 pm
Location: Lyon - France

Re: Driving a Web Browser with PureBasic

Post by Kwai chang caine »

:shock:
But then ???? at what serve Geckodriver ??? :shock:
More i move into this hitory, less i understand :oops:

PYTHON/PHP/C# ==> SELENIUM ==> Geckodriver ==> Marionet ==> FIREFOX
You surprise me..it's a little bit slow :lol: :lol:

Then Geckodriver is just here for sending HTTP to Marionnet ?
Already it's thanks to you, i can bypass SELENIUM and talking directly to Geckodriver 8)
So perhaps, we can again win a level, and talking to Marionet ? :D
I'm affraid this time it's OOP, and not possible in PB...but perhaps also, the langage is more simple, or again better more powerfull ??? :idea:
Because the Geckodriver command are not really numerous :|
We cannot manage the browser, adding tab, reading data, etc ...
ImageThe happiness is a road...
Not a destination
User avatar
RSBasic
Moderator
Moderator
Posts: 1218
Joined: Thu Dec 31, 2009 11:05 pm
Location: Gernsbach (Germany)
Contact:

Re: Driving a Web Browser with PureBasic

Post by RSBasic »

[offtopic]
@Kwai chang caine
How many gifs did you create in total? You are a king of Gif. :D
[/offtopic]
Image
Image
User avatar
Kwai chang caine
Always Here
Always Here
Posts: 5342
Joined: Sun Nov 05, 2006 11:42 pm
Location: Lyon - France

Re: Driving a Web Browser with PureBasic

Post by Kwai chang caine »

:lol: :lol:
I'm very happy you like all my childrens :mrgreen:
In fact, as i did not know myself, i went to the server folder for count it (When we love...we don't count :mrgreen:)
Because i create one, each time a new subject come and i need to illustrate my often too much long text :oops:
So, since more than ten years, i have create 34 animated gif KCC, and 27 statics KCC images :wink:
Some are very olds

Thanks a lot for your kinds words 8)
ImageThe happiness is a road...
Not a destination
Post Reply