HttpInclude [IDE Tool]
Posted: Fri Jul 31, 2015 7:02 pm
A small IDE Tool to make it possible to use codes directly from the pureboards or purearea.
The codes will get stored on your hard disk for faster access.
When compiling it the first time, you will need to download also my IDE Tool Helper:
http://www.purebasic.fr/english/viewtop ... 12&t=62033
Next time the HttpInclude inside the code will work (this is a hen egg problem )
The tool is self installing into your IDE, so I would recommend not to compile it directly from the IDE, because then it will set the tool to "Purebasic_CompilationX.X".
Better first create an executable and then start this.
And don't forget to add HttpInclude to your custom keywords.
Examples in next thread.
The codes will get stored on your hard disk for faster access.
When compiling it the first time, you will need to download also my IDE Tool Helper:
http://www.purebasic.fr/english/viewtop ... 12&t=62033
Next time the HttpInclude inside the code will work (this is a hen egg problem )
The tool is self installing into your IDE, so I would recommend not to compile it directly from the IDE, because then it will set the tool to "Purebasic_CompilationX.X".
Better first create an executable and then start this.
And don't forget to add HttpInclude to your custom keywords.
Examples in next thread.
Code: Select all
;*******************************************
;*
;* Filename: HTTP_Include.pb
;* Version: V1.0.1
;* Date: 03.05.2021
;* Author: HeX0R
;* http://hex0rs.coderbu.de
;* URL: https://www.purebasic.fr/english/viewtopic.php?f=12&t=62689
;*
;* License: BEER-WARE
;* Thomas 'HeX0R' Milz wrote this file. As long as you retain this notice you
;* can do whatever you want with this stuff. If we meet some day, and you think
;* this stuff is worth it, you can buy me a beer in return.
;* HeX0R@coderbu.de
;*
;* OS: [x] Windows (> Windows XP!!)
;* [x] Linux
;* [?] MacOS (untested)
;*
;* Description: With this IDE Tool you can
;* directly include code from the internet
;*
;*
;* Usage
;* Just use HTTPInclude "URL" instead of IncludeFile "FILENAME"
;* e.g.
;* HTTPInclude "http://www.purebasic.fr/english/viewtopic.php?f=12&t=62033"
;* you can add also special parameters to the end, e.g.:
;* HTTPInclude "http://www.purebasic.fr/english/viewtopic.php?f=12&t=62033" ;reload ;raw
;* Explanation:
;* ;reload -> Will force the source to be reloaded (otherwise it will use the local copy on your PC (if any))
;* ;raw -> Will just use the whole file, otherwise the tool expects the code is part of a forum thread (and will use regular expressions, see ;usesource below)
;* ;load=#num -> load=2 will load the second code it finds in the forum thread
;* ;usebom= -> You can force to read the file with a special BOM, usable: utf8, utf16, ascii
;* ;usesource=name -> If you don't set this, the default for the pureboards will be used (or depending on the URL, please see below)
;* Create more Regular expressions for different boards, please take a look into %APPDATA%\http_include\settings.prefs:
;* [Source_{Name}]
;* Name = Name of the Regex, usually the forum name it will handle.
;* URL = Regex to extract an anchor from the URL
;* CodeReg = Regex to extract the code from the board
;* AnchorStart = This is NO Regex! But just a text to find. Anything will start from that position then
;* RootURL = when set, all URLs starting with this string, will be use this Source (if not forced to use a different one via usesource=xxx)
;* UseRaw = when set to 1, all regular expressions will not be used, and the raw data from the URL is directly picked up.
;*
;*
;*******************************************
Enumeration
#File_Compiled_Source
#File_Downloaded_Source
#File_New_Offline_Source
#File_Error_Log
EndEnumeration
CompilerIf Defined(DEBUG_SOURCE, #PB_Constant) = 0
;only for debugging purposes
#DEBUG_SOURCE = "C:\Temp\PB_EditorOutput.pb"
CompilerEndIf
InitNetwork()
EnableExplicit
;On the first run you will need this Include of course (henn/egg problem)!
;Just download it and use IncludeFile instead.
HTTPInclude "https://www.purebasic.fr/english/viewtopic.php?f=12&t=62033"
Structure _Source_
Name.s
RootURL.s
URLStart.s
URLReg.i
AnchorStart.s
CodeReg.i
UseRaw.b
EndStructure
Global NewList Lines.s()
Global NewList Sources._Source_()
Procedure ErrorLog(Text.s)
Static First
If First = 0
CreateFile(#File_Error_Log, GetUserDirectory(#PB_Directory_ProgramData) + "http_include/error.log")
First = #True
EndIf
If IsFile(#File_Error_Log)
WriteStringN(#File_Error_Log, FormatDate("%hh:%ii:%ss ", Date()) + Text)
EndIf
EndProcedure
#REG_1 = "pureboard"
#REG_2 = "#p\d*"
#REG_3 = "(?<=<code>)[^<]*"
#REG_4 = ~"<div id=\"{ANCHOR}\""
Procedure SelectSource(Name.s, RootURL.s)
Protected Result
ForEach Sources()
If Name
If LCase(Name) = Sources()\Name
Result = #True
Break
EndIf
ElseIf LCase(RootURL) = Sources()\RootURL
Result = #True
Break
ElseIf Sources()\Name = #REG_1
Result = #True
Break
EndIf
Next
ProcedureReturn Result
EndProcedure
Procedure InitSources()
Protected a$, b$
If PreferenceGroup("Source_PureBoard") = 0
WritePreferenceString("RootURL", "https://www.purebasic.fr/")
WritePreferenceString("Name", #REG_1)
WritePreferenceString("URL", #REG_2)
WritePreferenceString("CodeReg", #REG_3)
WritePreferenceString("AnchorStart", #REG_4)
WritePreferenceInteger("UseRaw", 0)
EndIf
With Sources()
If ExaminePreferenceGroups()
While NextPreferenceGroup()
a$ = PreferenceGroupName()
If Left(LCase(a$), 6) = "source_"
AddElement(Sources())
\Name = LCase(Mid(a$, 8))
\UseRaw = ReadPreferenceInteger("UseRaw", 0)
\RootURL = LCase(ReadPreferenceString("RootURL", "https://www.purebasic.fr/"))
a$ = ReadPreferenceString("URL", #REG_2)
If a$
\URLReg = CreateRegularExpression(#PB_Any, a$)
If \URLReg = 0
ErrorLog("There is an error in the URL Regex of '" + \Name + "': " + RegularExpressionError())
EndIf
EndIf
\AnchorStart = ReadPreferenceString("AnchorStart", #REG_4)
a$ = ReadPreferenceString("CodeReg", #REG_3)
If a$
\CodeReg = CreateRegularExpression(#PB_Any, a$, #PB_RegularExpression_DotAll)
If \CodeReg = 0
ErrorLog("There is an error in the Code Regex of '" + \Name + "': " + RegularExpressionError())
EndIf
EndIf
EndIf
Wend
EndIf
If SelectSource("", "") = 0
AddElement(Sources())
\Name = "pureboard"
\URLReg = CreateRegularExpression(#PB_Any, #REG_2)
\AnchorStart = #REG_4
\CodeReg = CreateRegularExpression(#PB_Any, #REG_3, #PB_RegularExpression_DotAll)
\RootURL = "https://www.purebasic.fr/"
\UseRaw = 0
EndIf
EndWith
EndProcedure
Procedure main()
Protected SourceCode.s, BOM, URL.s, i, j, k, Pos, Reload, PathToPrefs.s, PathToIncludes.s
Protected a$, Code.s, Changed, FileName.s, Num, Raw, SourceBOM, SourceBOMText.s, Anchor.s, Name.s
CompilerIf #PB_Compiler_Debugger
SourceCode = #DEBUG_SOURCE
CompilerElse
SourceCode = ProgramParameter(0)
CompilerEndIf
;First create our directorys
PathToPrefs = GetUserDirectory(#PB_Directory_ProgramData) + "http_include/"
If FileSize(PathToPrefs) <> -2
CreateDirectory(PathToPrefs)
EndIf
;then open (or create) preferences
If OpenPreferences(PathToPrefs + "settings.prefs") = 0
CreatePreferences(PathToPrefs + "settings.prefs")
EndIf
Debug SourceCode
InitSources()
PreferenceGroup("main")
;this is the path, were the includes will be stored
PathToIncludes = ReadPreferenceString("PathToIncludes", PathToPrefs + "includes/")
If SourceCode
;we have a program parameter
;let's go
;first open the SourceCode
If OpenFile(#File_Compiled_Source, SourceCode)
BOM = ReadStringFormat(#File_Compiled_Source) ;<- save the BOM
Changed = #False ;<- If we changed something, this will be true
While Eof(#File_Compiled_Source) = 0
;read any line and add it to our list
AddElement(Lines())
Lines() = ReadString(#File_Compiled_Source, BOM)
If FindString(Trim(Trim(Lines()), #TAB$), "httpinclude", 1, #PB_String_NoCase) = 1 And CountString(Lines(), #DQUOTE$) > 1 ;<- is there a httpinclude at the beginning?
Code = ""
Num = 1
SourceBOM = #PB_UTF8 ;<- this is the BOM for the external file
URL = StringField(Lines(), 2, #DQUOTE$) ;<- get the URL of the include
Pos = Len(URL) + 13 ;<- start searching position for optional parameters (URL + 2x DQOUTE + httpinclude)
Reload = FindString(Lines(), ";reload", Pos, #PB_String_NoCase) ;<- if there is a ;reload behind it, this include will be updated
Raw = FindString(Lines(), ";raw", Pos, #PB_String_NoCase) ;<- if ;raw is an optional parameter we expect a raw code file instead of code inside a forum thread
i = FindString(Lines(), ";usesource=", Pos, #PB_String_NoCase)
If i
Name = Trim(LCase(Mid(Lines(), i + 11)))
If SelectSource(Name, "") = 0 And SelectSource("", URL) = 0
SelectSource("", "")
EndIf
ElseIf SelectSource("", URL) = 0
SelectSource("", "")
EndIf
If Raw = 0
Raw = Sources()\UseRaw
EndIf
i = FindString(Lines(), ";load=", Pos, #PB_String_NoCase) ;<- with load=2 e.g. we load the second code in a thread (only useful for code inside a forum thread)
If i
Num = Val(Mid(Lines(), i + 6))
EndIf
i = FindString(Lines(), ";usebom=", Pos, #PB_String_NoCase) ;<- by default we expect the external file in UTF8 format, but maybe the site stores it different?
If i
SourceBOMText = Mid(Lines(), i + 8)
If Left(LCase(SourceBOMText), 4) = "utf8"
SourceBOM = #PB_UTF8
ElseIf Left(LCase(SourceBOMText), 5) = "utf16"
SourceBOM = #PB_UTF16
ElseIf Left(LCase(SourceBOMText), 5) = "ascii"
SourceBOM = #PB_Ascii
EndIf
EndIf
If PreferenceGroup(URL) And ReadPreferenceString("file_" + Str(Num), "") And Reload = 0 ;<- check if we already have stored this include and there is no ;reload
Lines() = "XIncludeFile " + #DQUOTE$ + ReadPreferenceString("file_" + Str(Num), "") + #DQUOTE$ ;<- yes, o.k. just change this line and include our previously loaded (and stored) source
Changed = #True ;<- we changed something
Else
;no old include there, or user wants to update it
;first load the include from the internet to a temporary directory
If ReceiveHTTPFile(URL, GetTemporaryDirectory() + "http_include.tmp")
;o.k., now open the html-file
If ReadFile(#File_Downloaded_Source, GetTemporaryDirectory() + "http_include.tmp") = 0
ErrorLog("Can't open downloaded file '" + GetTemporaryDirectory() + "http_include.tmp'")
Else
i = ReadStringFormat(#File_Downloaded_Source)
If i <> #PB_Ascii ;if there is a BOM inside the file we overwrite the user setting
SourceBOM = i
EndIf
a$ = ReadString(#File_Downloaded_Source, SourceBOM | #PB_File_IgnoreEOL) ;<- read anything in a$
CloseFile(#File_Downloaded_Source) ;<- we are finished with the tmp file, we have anything inside a$, so close it and delete it.
DeleteFile(GetTemporaryDirectory() + "http_include.tmp")
;now find the start of the code
If Raw
;we want to use the whole file as is, not much to do here
Code = a$
Else
;o.k. I guess this code is inside a forum thread, so we need to find the code in it.
;the below will only work inside pureboards, we probably have to change something for other boards.
Dim Result$(0)
If IsRegularExpression(Sources()\URLReg)
k = ExtractRegularExpression(Sources()\URLReg, URL, Result$())
If k
Anchor = Result$(0)
If Left(Anchor, 1) = "#"
Anchor = Mid(Anchor, 2)
EndIf
EndIf
EndIf
If Anchor
k = FindString(a$, ReplaceString(Sources()\AnchorStart, "{ANCHOR}", Anchor, #PB_String_NoCase), 1, #PB_String_NoCase)
If k
a$ = Mid(a$, k)
EndIf
EndIf
If IsRegularExpression(Sources()\CodeReg)
k = ExtractRegularExpression(Sources()\CodeReg, a$, Result$())
If k > 0 And Num <= k
Code = Result$(Num - 1)
;now we have to decode a few html tags
;maybe I didn't catch anything, need to keep an eye on this
Code = ReplaceString(Code, " ", " ")
Code = ReplaceString(ReplaceString(Code, "<br />", #CRLF$), " ", " ")
Code = ReplaceString(ReplaceString(Code, "[", "["), "]", "]")
Code = ReplaceString(ReplaceString(Code, """, #DQUOTE$), "<", "<")
Code = ReplaceString(ReplaceString(Code, ">", ">"), "&", "&")
EndIf
EndIf
EndIf
If Code
If FileSize(PathToIncludes) <> -2
;this seems to be the first file we want to store, so create the directory
CreateDirectory(PathToIncludes)
EndIf
If Reload
;We already had this file before and user wants it to be updated, so just take the original filename
FileName = ReadPreferenceString("file_" + Str(Num), "")
Else
;search for the next unused filename
j = 0
While FileSize(PathToIncludes + RSet(Str(j), 6, "0") + ".pb") <> -1
j + 1
Wend
FileName = PathToIncludes + RSet(Str(j), 6, "0") + ".pb"
EndIf
;Save our new include
If CreateFile(#File_New_Offline_Source, FileName)
WriteStringFormat(#File_New_Offline_Source, #PB_UTF8) ;<- for PB I usually always use UTF8 as file format
WriteString(#File_New_Offline_Source, Code, #PB_UTF8)
CloseFile(#File_New_Offline_Source)
If Reload = 0
;only write the FileName into our preference file, if there were no "reload" parameter.
WritePreferenceString("file_" + Str(Num), FileName)
EndIf
;Change the line now finally
Lines() = "XIncludeFile " + #DQUOTE$ + FileName + #DQUOTE$
Changed = #True ;<- we changed something
EndIf
EndIf
EndIf
ElseIf Reload ;<- seems the URL couldn't get accessed, but this is just a reload, we can still use our old file
FileName = ReadPreferenceString("file_" + Str(Num), "")
Lines() = "XIncludeFile " + #DQUOTE$ + FileName + #DQUOTE$
Changed = #True ;<- we changed something
Else
ErrorLog("Can't reach '" + URL + "'!")
EndIf
EndIf
EndIf
Wend
CloseFile(#File_Compiled_Source)
If Changed
;something changed, so rewrite the file for the compiler
DeleteFile(SourceCode)
If CreateFile(#File_Compiled_Source, SourceCode)
WriteStringFormat(#File_Compiled_Source, BOM)
ForEach Lines()
WriteStringN(#File_Compiled_Source, Lines(), BOM)
Next
CloseFile(#File_Compiled_Source)
EndIf
EndIf
Else
EndIf
ClosePreferences()
Else
;no programparameter, so try to install it
ClosePreferences()
;See my IDE tool here: http://www.purebasic.fr/english/viewtopic.php?f=12&t=62033
If IDETOOL::Install("HTTPIncludeA", #DQUOTE$ + "%COMPILEFILE" + #DQUOTE$, "", IDETOOL::#TRIGGER_BEFORE_COMPILING, IDETOOL::#FLAG_WAIT_FOR_TOOL | IDETOOL::#FLAG_START_HIDDEN, 0, 0, 0, 1)
If IDETOOL::Install("HTTPIncludeB", #DQUOTE$ + "%COMPILEFILE" + #DQUOTE$, "", IDETOOL::#TRIGGER_BEFORE_EXE_CREATION, IDETOOL::#FLAG_WAIT_FOR_TOOL | IDETOOL::#FLAG_START_HIDDEN, 0, 0, 0, 1)
OpenPreferences(PathToPrefs + "settings.prefs") ;<- open is enough, because we already created it on top of main()
PreferenceGroup("main")
a$ = PathRequester("Select Path where online Includes should be stored", PathToIncludes)
If a$
WritePreferenceString("PathToIncludes", a$)
EndIf
ClosePreferences()
MessageRequester("Success!", "Successfully installed tool 'HTTPInclude'!" + #LF$ + "If your IDE is open, please restart it." + #LF$ + #LF$ + "(Remember to add HTTPInclude to your custom keywords!)")
EndIf
EndIf
EndIf
If IsFile(#File_Error_Log)
CloseFile(#File_Error_Log)
EndIf
EndProcedure
main()