HTTPS Login retrieve source problem
-
- New User
- Posts: 8
- Joined: Sun Sep 24, 2017 3:29 am
HTTPS Login retrieve source problem
Hi there , I have been having a dabble in Purebasic lately and am wandering if there is some kind of solution for the following problem.
Iam trying to access data from a HTTPS source which can only be accessed after logging into the website.
Is there some way to Login into the website securely , and then RecieveHttp the Page after login?
Currently I can only access non SSL Pages using the HttpRecieve Method.
Thank you.
The Anchient LibCurl.pbi does not seem to work anymore.
edit: something like this for PB I think is what Iam looking for;
https://stackoverflow.com/questions/128 ... mmatically
Iam trying to access data from a HTTPS source which can only be accessed after logging into the website.
Is there some way to Login into the website securely , and then RecieveHttp the Page after login?
Currently I can only access non SSL Pages using the HttpRecieve Method.
Thank you.
The Anchient LibCurl.pbi does not seem to work anymore.
edit: something like this for PB I think is what Iam looking for;
https://stackoverflow.com/questions/128 ... mmatically
Re: HTTPS Login retrieve source problem
Hi,
libcurl.pbi is still valid and working
What's your exact problem?
Normally you rceive a sessionid or a cookie after your login.
This you have to use at any further access.
That's the reason why you can not use ReceiveHTTP... from PB afterwards.
You have to use the 'hand made' solution with libcurl.pbi to get the pages.
Bernd
libcurl.pbi is still valid and working
What's your exact problem?
Normally you rceive a sessionid or a cookie after your login.
This you have to use at any further access.
That's the reason why you can not use ReceiveHTTP... from PB afterwards.
You have to use the 'hand made' solution with libcurl.pbi to get the pages.
Bernd
-
- New User
- Posts: 8
- Joined: Sun Sep 24, 2017 3:29 am
Re: HTTPS Login retrieve source problem
Appreciate your reply infratec
are there any good examples of how to do what you just stated please?
for example how would I use libcurl.pbi to login to purebasic.fr and then return the source code for a page afterwards?
are there any good examples of how to do what you just stated please?
for example how would I use libcurl.pbi to login to purebasic.fr and then return the source code for a page afterwards?
-
- New User
- Posts: 8
- Joined: Sun Sep 24, 2017 3:29 am
Re: HTTPS Login retrieve source problem
Resolved the not working issueinfratec wrote:Hi,
libcurl.pbi is still valid and working
What's your exact problem?
how would I go about doing the following for example;Normally you rceive a sessionid or a cookie after your login.
This you have to use at any further access.
That's the reason why you can not use ReceiveHTTP... from PB afterwards.
You have to use the 'hand made' solution with libcurl.pbi to get the pages.
Bernd
Login into this website and then open a post and view the source code.
or more so assign the sourcecode to a variable for filtering later using StringFind etc.
Re: HTTPS Login retrieve source problem
Hi,
this site did not use https
And...
It really depends which kind of authorization your specific site needs.
Also it decides if it then uses a cookie or a sessionid.
So more informations are needed, else I write something which is not useful and a waste of time and energy.
Bernd
this site did not use https
And...
It really depends which kind of authorization your specific site needs.
Also it decides if it then uses a cookie or a sessionid.
So more informations are needed, else I write something which is not useful and a waste of time and energy.
Bernd
-
- New User
- Posts: 8
- Joined: Sun Sep 24, 2017 3:29 am
Re: HTTPS Login retrieve source problem
Hello Again infratec the site that Iam currently trying to acquire information from is https://nomansskymods.com/login/
basically in order to acquire the <Mod Download Link> in short, you need to be logged in to reveal and activate the download.
This Site uses Wordpress Type Authentication which is I think Cookie Authentication according to this Codex:
https://codex.wordpress.org/WordPress_Cookies
a Shame that there is no Documentation on how-to use the libcurl.pbi in these cases. :/
basically in order to acquire the <Mod Download Link> in short, you need to be logged in to reveal and activate the download.
This Site uses Wordpress Type Authentication which is I think Cookie Authentication according to this Codex:
https://codex.wordpress.org/WordPress_Cookies
a Shame that there is no Documentation on how-to use the libcurl.pbi in these cases. :/
Last edited by someFellow on Mon Sep 25, 2017 11:28 pm, edited 1 time in total.
Re: HTTPS Login retrieve source problem
libcurl.pbi is from a forum member like you.someFellow wrote:a Shame that there is no Documentation on how-to use the libcurl.pbi in these cases. :/
It is not part of PB.
So it is more a shame that you not provide as here a full working version with complete documentation.
I'll check if I can help you.
-
- New User
- Posts: 8
- Joined: Sun Sep 24, 2017 3:29 am
Re: HTTPS Login retrieve source problem
would be appreciated if you can , understand if you are busy.
Cheers
Cheers
Re: HTTPS Login retrieve source problem
Hi,
since I have no user and password, please test this and post if the login was succesfull.
Bernd
since I have no user and password, please test this and post if the login was succesfull.
Code: Select all
#User$ = "test"
#Passwd$ = "test"
IncludeFile "libcurl.pbi"
InitNetwork()
#LoginURL$ = "https://nomansskymods.com/wp-login.php"
; name:log id:user_login
; name:pwd id:user_pass
curl = curl_easy_init()
If curl
Post$ = "log=" + #User$ + "&pwd=" + #Passwd$ + "&wp_submit=1"
curl_easy_setopt(curl, #CURLOPT_URL, str2curl(#LoginURL$))
curl_easy_setopt(curl, #CURLOPT_POST, 1)
curl_easy_setopt(curl, #CURLOPT_POSTFIELDS, str2curl(Post$))
curl_easy_setopt(curl, #CURLOPT_SSL_VERIFYPEER, 0)
curl_easy_setopt(curl, #CURLOPT_SSL_VERIFYHOST, 0)
curl_easy_setopt(curl, #CURLOPT_HEADER, @"")
curl_easy_setopt(curl, #CURLOPT_WRITEFUNCTION, @curlWriteData())
res = curl_easy_perform(curl)
If res = #CURLE_OK
Result$ = curlGetData()
Debug Result$
EndIf
curl_easy_cleanup(curl)
EndIf
-
- New User
- Posts: 8
- Joined: Sun Sep 24, 2017 3:29 am
Re: HTTPS Login retrieve source problem
Youre a Star there Bernd I think its ready for the next move.
recieved back from server;
*The <encrypted_code> bits I have ofc filtered out.
Iam assuming that any all transactions will need the acquired cookies and sessionid's to be sent along with URL$ from this point forwards?.
Just wandering what the next step is in order to retrieve the source-code preferably into a variable like the RecieveHttpMemory Function in PB & then finally to download a file from the server?
Thank you.
recieved back from server;
Code: Select all
HTTP/1.1 302 Found
Date: Mon, 25 Sep 2017 20:12:24 GMT
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
Set-Cookie: __cfduid=<encrypted_code>; expires=Tue, 25-Sep-18 20:12:23 GMT; path=/; domain=.nomansskymods.com; HttpOnly
X-Powered-By: PHP/7.0.23
Set-Cookie: PHPSESSID=<encrypted_code>; path=/
Pragma: no-cache
Set-Cookie: gdbbx_session_activity=0; expires=Mon, 25-Sep-2017 20:42:25 GMT; Max-Age=1800; path=/
Set-Cookie: gdbbx_tracking_activity=1506370345; expires=Tue, 25-Sep-2018 20:12:25 GMT; Max-Age=31536000; path=/
Expires: Wed, 11 Jan 1984 05:00:00 GMT
Cache-Control: no-cache, must-revalidate, max-age=0
Set-Cookie: wordpress_test_cookie=WP+Cookie+check; path=/; secure
X-Frame-Options: SAMEORIGIN
Set-Cookie: wordpress_sec_<encrypted_code_1>=USERNAME%<encrypted_code_2>%<encrypted_code_3>; path=/wp-content/plugins; secure; HttpOnly
Set-Cookie: wordpress_sec_<encrypted_code_1>=USERNAME%<encrypted_code_2>%<encrypted_code_3>; path=/wp-admin; secure; HttpOnly
Set-Cookie: wordpress_logged_in_<encrypted>=USERNAME%<encrypted_code_2>%<encrypted_code_4>; path=/; HttpOnly
Location: http://nomansskymods.com
Server: cloudflare-nginx
CF-RAY: <encrypted>-LOCATION
Iam assuming that any all transactions will need the acquired cookies and sessionid's to be sent along with URL$ from this point forwards?.
Just wandering what the next step is in order to retrieve the source-code preferably into a variable like the RecieveHttpMemory Function in PB & then finally to download a file from the server?
Thank you.
Re: HTTPS Login retrieve source problem
Hi,
it's very difficult to write something when you can not test it.
Bernd
it's very difficult to write something when you can not test it.
Code: Select all
EnableExplicit
#NoMansSkyModsURL$ = "https://nomansskymods.com"
#User$ = "test"
#Passwd$ = "test"
Global ReceivedData.s
IncludeFile "libcurl.pbi"
Procedure.i NoMansSkyModsLogin(User$, Passwd$, CookieFilename$)
Protected Result.i, curl.i, Post$, Result$
curl = curl_easy_init()
If curl
Post$ = "log=" + #User$ + "&pwd=" + #Passwd$ + "&wp_submit=1"
curl_easy_setopt(curl, #CURLOPT_URL, str2curl(#NoMansSkyModsURL$ + "/wp-login.php"))
curl_easy_setopt(curl, #CURLOPT_POST, 1)
curl_easy_setopt(curl, #CURLOPT_POSTFIELDS, str2curl(Post$))
curl_easy_setopt(curl, #CURLOPT_SSL_VERIFYPEER, 0)
curl_easy_setopt(curl, #CURLOPT_SSL_VERIFYHOST, 0)
curl_easy_setopt(curl, #CURLOPT_HEADER, @"")
curl_easy_setopt(curl, #CURLOPT_WRITEFUNCTION, @curlWriteData())
curl_easy_setopt(curl, #CURLOPT_COOKIEJAR, str2curl(CookieFilename$))
Result = curl_easy_perform(curl)
If Result = #CURLE_OK
Result$ = curlGetData()
If FindString(Result$, "user_login")
Result = #False
Else
Result = #True
EndIf
Else
Result = #False
EndIf
curl_easy_cleanup(curl)
EndIf
ProcedureReturn Result
EndProcedure
Procedure.s GetNoMansSkyModsWebPage(Url$, CookieFilename$)
Protected Result$, curl.i, Result.i
curl = curl_easy_init()
If curl
curl_easy_setopt(curl, #CURLOPT_URL, str2curl(Url$))
curl_easy_setopt(curl, #CURLOPT_SSL_VERIFYPEER, 0)
curl_easy_setopt(curl, #CURLOPT_SSL_VERIFYHOST, 0)
curl_easy_setopt(curl, #CURLOPT_HEADER, @"")
curl_easy_setopt(curl, #CURLOPT_WRITEFUNCTION, @curlWriteData())
curl_easy_setopt(curl, #CURLOPT_COOKIEFILE, str2curl(CookieFilename$))
Result = curl_easy_perform(curl)
If Result = #CURLE_OK
Result$ = curlGetData()
EndIf
curl_easy_cleanup(curl)
EndIf
ProcedureReturn Result$
EndProcedure
InitNetwork()
Define CookieFilename$
CookieFilename$ = GetTemporaryDirectory() + "NoMansSkyModsCookies.txt"
If NoMansSkyModsLogin(#User$, #Passwd$, CookieFilename$)
Debug GetNoMansSkyModsWebPage(#NoMansSkyModsURL$ + "/", CookieFilename$)
EndIf
If FileSize(CookieFilename$) > 0
Debug "Cookies deleted"
DeleteFile(CookieFilename$)
EndIf
-
- New User
- Posts: 8
- Joined: Sun Sep 24, 2017 3:29 am
Re: HTTPS Login retrieve source problem
i'm not having much luck with this one. any help will be appreciated. cheers.
Re: HTTPS Login retrieve source problem
Hi,
works for me:
For an example I leave the debug and dump functions inside this listing.
You can remove them.
Bernd
works for me:
Code: Select all
EnableExplicit
#NoMansSkyModsURL$ = "https://nomansskymods.com"
#User$ = "blabla"
#Passwd$ = "blublu"
Structure DownloadInfoStructure
FileURL$
_wpnonce$
id$
shortcodeId$
backurl$
EndStructure
Global ReceivedData.s
IncludeFile "libcurl.pbi"
Procedure Dump(Text$, *Data, Size.i)
Protected.i File, i, c, Byte
Protected Line$
File = OpenFile(#PB_Any, "c:\tmp\libcurl_debug.txt", #PB_File_Append)
If File
WriteStringN(File, Text$)
WriteStringN(File, "")
For i = 0 To size - 1 Step 16
Line$ = ""
For c = 0 To 15
If i + c < size
Line$ + RSet(Hex(PeekA(*Data + i + c), #PB_Byte), 2, "0") + " "
Else
Line$ + " "
EndIf
Next c
For c = 0 To 15
If i + c < size
Byte = PeekA(*Data + i + c)
If Byte < ' '
Line$ + "."
Else
Line$ + Chr(Byte)
EndIf
Else
Line$ + " "
EndIf
Next c
WriteStringN(File, Line$)
Next i
WriteStringN(File, "")
CloseFile(File)
EndIf
EndProcedure
ProcedureC.l my_trace(*handle, type.l, *Data, size.l, *userp)
Protected Result.l
Select type
Case #CURLINFO_TEXT
Debug "== Info: " + PeekS(*Data, size, #PB_UTF8)
Case #CURLINFO_HEADER_OUT
Dump("Send header", *Data, size)
Case #CURLINFO_DATA_OUT
Dump("Send Data", *Data, size)
Case #CURLINFO_SSL_DATA_OUT
Dump("Send SSL Data", *Data, size)
Case #CURLINFO_HEADER_IN
Dump("Recv header", *Data, size)
Case #CURLINFO_DATA_IN
Dump("Recv data", *Data, size)
Case #CURLINFO_SSL_DATA_IN
Dump("Recv SSL data", *Data, size)
EndSelect
ProcedureReturn Result
EndProcedure
Procedure.i NoMansSkyModsLogin(User$, Passwd$, CookieFilename$)
Protected Result.i, curl.i, Post$, Result$, *Post
curl = curl_easy_init()
If curl
Post$ = "log=" + #User$ + "&"
Post$ + "pwd=" + #Passwd$ + "&"
Post$ + "wp-submit=Log+In&"
Post$ + "redirect_to=" + #NoMansSkyModsURL$ + "/login/&"
Post$ + "wppb_login=true&"
Post$ + "wppb_form_location=page&"
Post$ + "wppb_request_url=" + #NoMansSkyModsURL$ + ":433/login/&"
Post$ + "wppb_lostpassword_url=&"
Post$ + "wppb_redirect_priority=&"
Post$ + "wppb_referer_url="
Debug Post$
Debug ""
curl_easy_setopt(curl, #CURLOPT_URL, str2curl(#NoMansSkyModsURL$ + "/wp-login.php"))
curl_easy_setopt(curl, #CURLOPT_POST, 1)
;curl_easy_setopt(curl, #CURLOPT_POSTFIELDS, str2curl(Post$))
*Post = UTF8(Post$) ; needed because it has to be available over the complete time
curl_easy_setopt(curl, #CURLOPT_POSTFIELDS, *Post)
curl_easy_setopt(curl, #CURLOPT_SSL_VERIFYPEER, 0)
curl_easy_setopt(curl, #CURLOPT_SSL_VERIFYHOST, 0)
curl_easy_setopt(curl, #CURLOPT_HEADER, @"")
curl_easy_setopt(curl, #CURLOPT_WRITEFUNCTION, @curlWriteData())
curl_easy_setopt(curl, #CURLOPT_COOKIEJAR, str2curl(CookieFilename$))
curl_easy_setopt(curl, #CURLOPT_REFERER, str2curl(#NoMansSkyModsURL$ + "/login/"))
; the following 2 lines are needed for debugging
;curl_easy_setopt(curl, #CURLOPT_DEBUGFUNCTION, @my_trace())
;curl_easy_setopt(curl, #CURLOPT_VERBOSE, 1)
Result = curl_easy_perform(curl)
If Result = #CURLE_OK
Result$ = curlGetData()
;Debug Result$
If FindString(Result$, "user_login")
Result = #False
Else
Result = #True
EndIf
Else
Result = #False
EndIf
curl_easy_cleanup(curl)
FreeMemory(*Post)
EndIf
ProcedureReturn Result
EndProcedure
Procedure.s GetNoMansSkyModsDownloadPage(Url$, CookieFilename$, *DownloadInfo.DownloadInfoStructure)
Protected Result$, curl.i, Result.i, Pos1.i, Pos2.i, Pos3.i, Pos4.i
curl = curl_easy_init()
If curl
curl_easy_setopt(curl, #CURLOPT_URL, str2curl(Url$))
curl_easy_setopt(curl, #CURLOPT_SSL_VERIFYPEER, 0)
curl_easy_setopt(curl, #CURLOPT_SSL_VERIFYHOST, 0)
curl_easy_setopt(curl, #CURLOPT_HEADER, @"")
curl_easy_setopt(curl, #CURLOPT_WRITEFUNCTION, @curlWriteData())
curl_easy_setopt(curl, #CURLOPT_COOKIEFILE, str2curl(CookieFilename$))
Result = curl_easy_perform(curl)
If Result = #CURLE_OK
Result$ = curlGetData()
If FindString(Result$, " 200 OK")
Pos2 = FindString(Result$, ".zip")
If Pos2
Pos1 = Pos2
Pos2 + 4
While Mid(Result$, Pos1, 1) <> #DQUOTE$ And Pos1 > 0
Pos1 - 1
Wend
If Pos1
Pos1 + 1
*DownloadInfo\FileURL$ = Mid(Result$, Pos1, Pos2 - Pos1)
Pos3 = FindString(Result$, "_wpnonce", Pos2)
If Pos3
Pos3 + 17
Pos4 = FindString(Result$, #DQUOTE$, Pos3)
*DownloadInfo\_wpnonce$ = Mid(Result$, Pos3, Pos4 - Pos3)
EndIf
Pos3 = FindString(Result$, #DQUOTE$ + "id" + #DQUOTE$, Pos2)
If Pos3
Pos3 + 12
Pos4 = FindString(Result$, #DQUOTE$, Pos3)
*DownloadInfo\id$ = Mid(Result$, Pos3, Pos4 - Pos3)
EndIf
Pos3 = FindString(Result$, "shortcodeId", Pos2)
If Pos3
Pos3 + 20
Pos4 = FindString(Result$, #DQUOTE$, Pos3)
*DownloadInfo\shortcodeId$ = Mid(Result$, Pos3, Pos4 - Pos3)
EndIf
Pos3 = FindString(Result$, "backurl", Pos2)
If Pos3
Pos3 + 16
Pos4 = FindString(Result$, #DQUOTE$, Pos3)
*DownloadInfo\backurl$ = Mid(Result$, Pos3, Pos4 - Pos3)
EndIf
Else
Result$ = ""
EndIf
Else
Result$ = ""
EndIf
Else
Result$ = ""
EndIf
Else
Debug "BKK"
EndIf
curl_easy_cleanup(curl)
EndIf
ProcedureReturn *DownloadInfo\FileURL$
EndProcedure
ProcedureC.l curlWriteFile(*ptr, Size, NMemB, *Stream)
ProcedureReturn WriteData(*Stream, *ptr, Size * NMemB)
EndProcedure
Procedure.i GetNoMansSkyModsFile(URL$, SaveAs$, CookieFilename$, *DownloadInfo.DownloadInfoStructure)
Protected Result.i, Curl.i, File.i, Post$, *Post
File = CreateFile(#PB_Any, SaveAs$)
If File
curl = curl_easy_init()
If curl
curl_easy_setopt(curl, #CURLOPT_URL, str2curl(Url$))
Post$ = "_wpnonce=" + *DownloadInfo\_wpnonce$ + "&"
Post$ + "id=" + *DownloadInfo\id$ + "&"
Post$ + "shortcodeID=" + *DownloadInfo\shortcodeId$ + "&"
Post$ + "backrl=" + *DownloadInfo\backurl$
*Post = UTF8(Post$) ; needed because it has to be available over the complete time
curl_easy_setopt(curl, #CURLOPT_POSTFIELDS, *Post)
curl_easy_setopt(curl, #CURLOPT_SSL_VERIFYPEER, 0)
curl_easy_setopt(curl, #CURLOPT_SSL_VERIFYHOST, 0)
curl_easy_setopt(curl, #CURLOPT_HEADER, @"")
curl_easy_setopt(curl, #CURLOPT_WRITEFUNCTION, @curlWriteFile())
curl_easy_setopt(curl, #CURLOPT_COOKIEFILE, str2curl(CookieFilename$))
curl_easy_setopt(curl, #CURLOPT_WRITEDATA, File)
Result = curl_easy_perform(curl)
If Result = #CURLE_OK
Result = #True
Else
Result = #False
EndIf
curl_easy_cleanup(curl)
FreeMemory(*Post)
EndIf
CloseFile(File)
EndIf
ProcedureReturn Result
EndProcedure
;-Main
Define CookieFilename$, FileURL$
Define DownloadInfo.DownloadInfoStructure
InitNetwork()
CookieFilename$ = GetTemporaryDirectory() + "NoMansSkyModsCookies.txt"
Debug "Cookiefile:" + CookieFilename$
Debug ""
If NoMansSkyModsLogin(#User$, #Passwd$, CookieFilename$)
Debug ""
Debug "Login Ok"
Debug ""
FileURL$ = GetNoMansSkyModsDownloadPage(#NoMansSkyModsURL$ + "/mods/no-mans-sky-save-editor/", CookieFilename$, @DownloadInfo)
If FileURL$ <> ""
; Debug "c:\tmp\" + GetFilePart(FileURL$)
;
; Debug "_wpnonce: " + DownloadInfo\_wpnonce$
; Debug "id: " + DownloadInfo\id$
; Debug "ShortCodeID: " + DownloadInfo\shortcodeId$
; Debug "BackURL: " + DownloadInfo\backurl$
If GetNoMansSkyModsFile(FileURL$, "c:\tmp\" + GetFilePart(FileURL$), CookieFilename$, @DownloadInfo)
Debug "File receive Ok"
EndIf
EndIf
EndIf
If FileSize(CookieFilename$) > 0
If DeleteFile(CookieFilename$)
Debug "Cookies deleted"
EndIf
EndIf
You can remove them.
Bernd
-
- New User
- Posts: 8
- Joined: Sun Sep 24, 2017 3:29 am
Re: HTTPS Login retrieve source problem
ok Bernd. Idk how you done it , but you did!.
Thats a sh*tload of code to wade through my gosh. But for the most post it is legible_ish for someone of my calibre (amature) , you have obvioudly been coding for a long time and seem to have an indepth knowledge of the inner workings of websies etc.
Most appreciated for working this out for me. honestly was no small feat by the looks of it either.
Iam guessing thst you worked part time for NASA or some place similar? *jokes.
wow.
Thank You very much.
Legend!
Thats a sh*tload of code to wade through my gosh. But for the most post it is legible_ish for someone of my calibre (amature) , you have obvioudly been coding for a long time and seem to have an indepth knowledge of the inner workings of websies etc.
Most appreciated for working this out for me. honestly was no small feat by the looks of it either.
Iam guessing thst you worked part time for NASA or some place similar? *jokes.
wow.
Thank You very much.
Legend!