The codes will get stored on your hard disk for faster access.
When compiling it the first time, you will need to download also my IDE Tool Helper:
http://www.purebasic.fr/english/viewtop ... 12&t=62033
Next time the HttpInclude inside the code will work (this is a hen egg problem
)The tool is self installing into your IDE, so I would recommend not to compile it directly from the IDE, because then it will set the tool to "Purebasic_CompilationX.X".
Better first create an executable and then start this.
And don't forget to add HttpInclude to your custom keywords.
Examples in next thread.
Code: Select all
;*******************************************
;*
;* Filename: HTTP_Include.pb
;* Version: V1.0.1
;* Date: 03.05.2021
;* Author: HeX0R
;* http://hex0rs.coderbu.de
;* URL: https://www.purebasic.fr/english/viewtopic.php?f=12&t=62689
;*
;* License: BEER-WARE
;* Thomas 'HeX0R' Milz wrote this file. As long as you retain this notice you
;* can do whatever you want with this stuff. If we meet some day, and you think
;* this stuff is worth it, you can buy me a beer in return.
;* HeX0R@coderbu.de
;*
;* OS: [x] Windows (> Windows XP!!)
;* [x] Linux
;* [?] MacOS (untested)
;*
;* Description: With this IDE Tool you can
;* directly include code from the internet
;*
;*
;* Usage
;* Just use HTTPInclude "URL" instead of IncludeFile "FILENAME"
;* e.g.
;* HTTPInclude "http://www.purebasic.fr/english/viewtopic.php?f=12&t=62033"
;* you can add also special parameters to the end, e.g.:
;* HTTPInclude "http://www.purebasic.fr/english/viewtopic.php?f=12&t=62033" ;reload ;raw
;* Explanation:
;* ;reload -> Will force the source to be reloaded (otherwise it will use the local copy on your PC (if any))
;* ;raw -> Will just use the whole file, otherwise the tool expects the code is part of a forum thread (and will use regular expressions, see ;usesource below)
;* ;load=#num -> load=2 will load the second code it finds in the forum thread
;* ;usebom= -> You can force to read the file with a special BOM, usable: utf8, utf16, ascii
;* ;usesource=name -> If you don't set this, the default for the pureboards will be used (or depending on the URL, please see below)
;* Create more Regular expressions for different boards, please take a look into %APPDATA%\http_include\settings.prefs:
;* [Source_{Name}]
;* Name = Name of the Regex, usually the forum name it will handle.
;* URL = Regex to extract an anchor from the URL
;* CodeReg = Regex to extract the code from the board
;* AnchorStart = This is NO Regex! But just a text to find. Anything will start from that position then
;* RootURL = when set, all URLs starting with this string, will be use this Source (if not forced to use a different one via usesource=xxx)
;* UseRaw = when set to 1, all regular expressions will not be used, and the raw data from the URL is directly picked up.
;*
;*
;*******************************************
Enumeration
#File_Compiled_Source
#File_Downloaded_Source
#File_New_Offline_Source
#File_Error_Log
EndEnumeration
CompilerIf Defined(DEBUG_SOURCE, #PB_Constant) = 0
;only for debugging purposes
#DEBUG_SOURCE = "C:\Temp\PB_EditorOutput.pb"
CompilerEndIf
InitNetwork()
EnableExplicit
;On the first run you will need this Include of course (henn/egg problem)!
;Just download it and use IncludeFile instead.
HTTPInclude "https://www.purebasic.fr/english/viewtopic.php?f=12&t=62033"
Structure _Source_
Name.s
RootURL.s
URLStart.s
URLReg.i
AnchorStart.s
CodeReg.i
UseRaw.b
EndStructure
Global NewList Lines.s()
Global NewList Sources._Source_()
Procedure ErrorLog(Text.s)
Static First
If First = 0
CreateFile(#File_Error_Log, GetUserDirectory(#PB_Directory_ProgramData) + "http_include/error.log")
First = #True
EndIf
If IsFile(#File_Error_Log)
WriteStringN(#File_Error_Log, FormatDate("%hh:%ii:%ss ", Date()) + Text)
EndIf
EndProcedure
#REG_1 = "pureboard"
#REG_2 = "#p\d*"
#REG_3 = "(?<=<code>)[^<]*"
#REG_4 = ~"<div id=\"{ANCHOR}\""
Procedure SelectSource(Name.s, RootURL.s)
Protected Result
ForEach Sources()
If Name
If LCase(Name) = Sources()\Name
Result = #True
Break
EndIf
ElseIf LCase(RootURL) = Sources()\RootURL
Result = #True
Break
ElseIf Sources()\Name = #REG_1
Result = #True
Break
EndIf
Next
ProcedureReturn Result
EndProcedure
Procedure InitSources()
Protected a$, b$
If PreferenceGroup("Source_PureBoard") = 0
WritePreferenceString("RootURL", "https://www.purebasic.fr/")
WritePreferenceString("Name", #REG_1)
WritePreferenceString("URL", #REG_2)
WritePreferenceString("CodeReg", #REG_3)
WritePreferenceString("AnchorStart", #REG_4)
WritePreferenceInteger("UseRaw", 0)
EndIf
With Sources()
If ExaminePreferenceGroups()
While NextPreferenceGroup()
a$ = PreferenceGroupName()
If Left(LCase(a$), 6) = "source_"
AddElement(Sources())
\Name = LCase(Mid(a$, 8))
\UseRaw = ReadPreferenceInteger("UseRaw", 0)
\RootURL = LCase(ReadPreferenceString("RootURL", "https://www.purebasic.fr/"))
a$ = ReadPreferenceString("URL", #REG_2)
If a$
\URLReg = CreateRegularExpression(#PB_Any, a$)
If \URLReg = 0
ErrorLog("There is an error in the URL Regex of '" + \Name + "': " + RegularExpressionError())
EndIf
EndIf
\AnchorStart = ReadPreferenceString("AnchorStart", #REG_4)
a$ = ReadPreferenceString("CodeReg", #REG_3)
If a$
\CodeReg = CreateRegularExpression(#PB_Any, a$, #PB_RegularExpression_DotAll)
If \CodeReg = 0
ErrorLog("There is an error in the Code Regex of '" + \Name + "': " + RegularExpressionError())
EndIf
EndIf
EndIf
Wend
EndIf
If SelectSource("", "") = 0
AddElement(Sources())
\Name = "pureboard"
\URLReg = CreateRegularExpression(#PB_Any, #REG_2)
\AnchorStart = #REG_4
\CodeReg = CreateRegularExpression(#PB_Any, #REG_3, #PB_RegularExpression_DotAll)
\RootURL = "https://www.purebasic.fr/"
\UseRaw = 0
EndIf
EndWith
EndProcedure
Procedure main()
Protected SourceCode.s, BOM, URL.s, i, j, k, Pos, Reload, PathToPrefs.s, PathToIncludes.s
Protected a$, Code.s, Changed, FileName.s, Num, Raw, SourceBOM, SourceBOMText.s, Anchor.s, Name.s
CompilerIf #PB_Compiler_Debugger
SourceCode = #DEBUG_SOURCE
CompilerElse
SourceCode = ProgramParameter(0)
CompilerEndIf
;First create our directorys
PathToPrefs = GetUserDirectory(#PB_Directory_ProgramData) + "http_include/"
If FileSize(PathToPrefs) <> -2
CreateDirectory(PathToPrefs)
EndIf
;then open (or create) preferences
If OpenPreferences(PathToPrefs + "settings.prefs") = 0
CreatePreferences(PathToPrefs + "settings.prefs")
EndIf
Debug SourceCode
InitSources()
PreferenceGroup("main")
;this is the path, were the includes will be stored
PathToIncludes = ReadPreferenceString("PathToIncludes", PathToPrefs + "includes/")
If SourceCode
;we have a program parameter
;let's go
;first open the SourceCode
If OpenFile(#File_Compiled_Source, SourceCode)
BOM = ReadStringFormat(#File_Compiled_Source) ;<- save the BOM
Changed = #False ;<- If we changed something, this will be true
While Eof(#File_Compiled_Source) = 0
;read any line and add it to our list
AddElement(Lines())
Lines() = ReadString(#File_Compiled_Source, BOM)
If FindString(Trim(Trim(Lines()), #TAB$), "httpinclude", 1, #PB_String_NoCase) = 1 And CountString(Lines(), #DQUOTE$) > 1 ;<- is there a httpinclude at the beginning?
Code = ""
Num = 1
SourceBOM = #PB_UTF8 ;<- this is the BOM for the external file
URL = StringField(Lines(), 2, #DQUOTE$) ;<- get the URL of the include
Pos = Len(URL) + 13 ;<- start searching position for optional parameters (URL + 2x DQOUTE + httpinclude)
Reload = FindString(Lines(), ";reload", Pos, #PB_String_NoCase) ;<- if there is a ;reload behind it, this include will be updated
Raw = FindString(Lines(), ";raw", Pos, #PB_String_NoCase) ;<- if ;raw is an optional parameter we expect a raw code file instead of code inside a forum thread
i = FindString(Lines(), ";usesource=", Pos, #PB_String_NoCase)
If i
Name = Trim(LCase(Mid(Lines(), i + 11)))
If SelectSource(Name, "") = 0 And SelectSource("", URL) = 0
SelectSource("", "")
EndIf
ElseIf SelectSource("", URL) = 0
SelectSource("", "")
EndIf
If Raw = 0
Raw = Sources()\UseRaw
EndIf
i = FindString(Lines(), ";load=", Pos, #PB_String_NoCase) ;<- with load=2 e.g. we load the second code in a thread (only useful for code inside a forum thread)
If i
Num = Val(Mid(Lines(), i + 6))
EndIf
i = FindString(Lines(), ";usebom=", Pos, #PB_String_NoCase) ;<- by default we expect the external file in UTF8 format, but maybe the site stores it different?
If i
SourceBOMText = Mid(Lines(), i + 8)
If Left(LCase(SourceBOMText), 4) = "utf8"
SourceBOM = #PB_UTF8
ElseIf Left(LCase(SourceBOMText), 5) = "utf16"
SourceBOM = #PB_UTF16
ElseIf Left(LCase(SourceBOMText), 5) = "ascii"
SourceBOM = #PB_Ascii
EndIf
EndIf
If PreferenceGroup(URL) And ReadPreferenceString("file_" + Str(Num), "") And Reload = 0 ;<- check if we already have stored this include and there is no ;reload
Lines() = "XIncludeFile " + #DQUOTE$ + ReadPreferenceString("file_" + Str(Num), "") + #DQUOTE$ ;<- yes, o.k. just change this line and include our previously loaded (and stored) source
Changed = #True ;<- we changed something
Else
;no old include there, or user wants to update it
;first load the include from the internet to a temporary directory
If ReceiveHTTPFile(URL, GetTemporaryDirectory() + "http_include.tmp")
;o.k., now open the html-file
If ReadFile(#File_Downloaded_Source, GetTemporaryDirectory() + "http_include.tmp") = 0
ErrorLog("Can't open downloaded file '" + GetTemporaryDirectory() + "http_include.tmp'")
Else
i = ReadStringFormat(#File_Downloaded_Source)
If i <> #PB_Ascii ;if there is a BOM inside the file we overwrite the user setting
SourceBOM = i
EndIf
a$ = ReadString(#File_Downloaded_Source, SourceBOM | #PB_File_IgnoreEOL) ;<- read anything in a$
CloseFile(#File_Downloaded_Source) ;<- we are finished with the tmp file, we have anything inside a$, so close it and delete it.
DeleteFile(GetTemporaryDirectory() + "http_include.tmp")
;now find the start of the code
If Raw
;we want to use the whole file as is, not much to do here
Code = a$
Else
;o.k. I guess this code is inside a forum thread, so we need to find the code in it.
;the below will only work inside pureboards, we probably have to change something for other boards.
Dim Result$(0)
If IsRegularExpression(Sources()\URLReg)
k = ExtractRegularExpression(Sources()\URLReg, URL, Result$())
If k
Anchor = Result$(0)
If Left(Anchor, 1) = "#"
Anchor = Mid(Anchor, 2)
EndIf
EndIf
EndIf
If Anchor
k = FindString(a$, ReplaceString(Sources()\AnchorStart, "{ANCHOR}", Anchor, #PB_String_NoCase), 1, #PB_String_NoCase)
If k
a$ = Mid(a$, k)
EndIf
EndIf
If IsRegularExpression(Sources()\CodeReg)
k = ExtractRegularExpression(Sources()\CodeReg, a$, Result$())
If k > 0 And Num <= k
Code = Result$(Num - 1)
;now we have to decode a few html tags
;maybe I didn't catch anything, need to keep an eye on this
Code = ReplaceString(Code, " ", " ")
Code = ReplaceString(ReplaceString(Code, "<br />", #CRLF$), " ", " ")
Code = ReplaceString(ReplaceString(Code, "[", "["), "]", "]")
Code = ReplaceString(ReplaceString(Code, """, #DQUOTE$), "<", "<")
Code = ReplaceString(ReplaceString(Code, ">", ">"), "&", "&")
EndIf
EndIf
EndIf
If Code
If FileSize(PathToIncludes) <> -2
;this seems to be the first file we want to store, so create the directory
CreateDirectory(PathToIncludes)
EndIf
If Reload
;We already had this file before and user wants it to be updated, so just take the original filename
FileName = ReadPreferenceString("file_" + Str(Num), "")
Else
;search for the next unused filename
j = 0
While FileSize(PathToIncludes + RSet(Str(j), 6, "0") + ".pb") <> -1
j + 1
Wend
FileName = PathToIncludes + RSet(Str(j), 6, "0") + ".pb"
EndIf
;Save our new include
If CreateFile(#File_New_Offline_Source, FileName)
WriteStringFormat(#File_New_Offline_Source, #PB_UTF8) ;<- for PB I usually always use UTF8 as file format
WriteString(#File_New_Offline_Source, Code, #PB_UTF8)
CloseFile(#File_New_Offline_Source)
If Reload = 0
;only write the FileName into our preference file, if there were no "reload" parameter.
WritePreferenceString("file_" + Str(Num), FileName)
EndIf
;Change the line now finally
Lines() = "XIncludeFile " + #DQUOTE$ + FileName + #DQUOTE$
Changed = #True ;<- we changed something
EndIf
EndIf
EndIf
ElseIf Reload ;<- seems the URL couldn't get accessed, but this is just a reload, we can still use our old file
FileName = ReadPreferenceString("file_" + Str(Num), "")
Lines() = "XIncludeFile " + #DQUOTE$ + FileName + #DQUOTE$
Changed = #True ;<- we changed something
Else
ErrorLog("Can't reach '" + URL + "'!")
EndIf
EndIf
EndIf
Wend
CloseFile(#File_Compiled_Source)
If Changed
;something changed, so rewrite the file for the compiler
DeleteFile(SourceCode)
If CreateFile(#File_Compiled_Source, SourceCode)
WriteStringFormat(#File_Compiled_Source, BOM)
ForEach Lines()
WriteStringN(#File_Compiled_Source, Lines(), BOM)
Next
CloseFile(#File_Compiled_Source)
EndIf
EndIf
Else
EndIf
ClosePreferences()
Else
;no programparameter, so try to install it
ClosePreferences()
;See my IDE tool here: http://www.purebasic.fr/english/viewtopic.php?f=12&t=62033
If IDETOOL::Install("HTTPIncludeA", #DQUOTE$ + "%COMPILEFILE" + #DQUOTE$, "", IDETOOL::#TRIGGER_BEFORE_COMPILING, IDETOOL::#FLAG_WAIT_FOR_TOOL | IDETOOL::#FLAG_START_HIDDEN, 0, 0, 0, 1)
If IDETOOL::Install("HTTPIncludeB", #DQUOTE$ + "%COMPILEFILE" + #DQUOTE$, "", IDETOOL::#TRIGGER_BEFORE_EXE_CREATION, IDETOOL::#FLAG_WAIT_FOR_TOOL | IDETOOL::#FLAG_START_HIDDEN, 0, 0, 0, 1)
OpenPreferences(PathToPrefs + "settings.prefs") ;<- open is enough, because we already created it on top of main()
PreferenceGroup("main")
a$ = PathRequester("Select Path where online Includes should be stored", PathToIncludes)
If a$
WritePreferenceString("PathToIncludes", a$)
EndIf
ClosePreferences()
MessageRequester("Success!", "Successfully installed tool 'HTTPInclude'!" + #LF$ + "If your IDE is open, please restart it." + #LF$ + #LF$ + "(Remember to add HTTPInclude to your custom keywords!)")
EndIf
EndIf
EndIf
If IsFile(#File_Error_Log)
CloseFile(#File_Error_Log)
EndIf
EndProcedure
main()
