Zufälliges Problem mit Multithreading OpenCryptRandom()

Hier werden, insbesondere in den Beta-Phasen, Bugmeldungen gepostet. Das offizielle BugForum ist allerdings hier.
Benutzeravatar
Ground0
Beiträge: 318
Registriert: 05.02.2005 02:09
Wohnort: Stilli, Schweiz
Kontaktdaten:

Zufälliges Problem mit Multithreading OpenCryptRandom()

Beitrag von Ground0 »

Getestet mit: Purebasic 5.44LTS, 5.51 sowie 5.6b7

Code: Alles auswählen

PB-Version: PureBasic 5.44LTS, 5.51 sowie 5.6b7&8 (Windows & Linux - x64)
OS: Windows 10 64 Bit & openSUSE Tumbleweed 64 Bit vom 25.2.2017
Compilereinstellungen:
    [ ] Unicode-Executable erstellen
    [x] Thread-sicheres Executable erstellen
    [ ] OnError-Unterstützung einschalten
    [ ] Moderne Themen-Unterstützung aktivieren (für Windows XP und höher)
    [ ] Administrator-Modus für Windows Vista und höher anfordern
    [ ] User-Modus für Windows Vista und höher anfordern (keine Virtualisierung)
    Library Subsystem:gtk3
Und es ist der Cryptrandom den es verhaut....

Code: Alles auswählen

      If OpenCryptRandom() And Key
      DebugLvl3
      Debug "Randomcrypt."
      DebugEnd
        
        CryptRandomData(Key, 16)
       
Unter Linux gibt es keine Rückmeldung....
Unter Windows motzt der Debugger das zuerst OpenCryptRandom() für CryptRandomData(Key, 16) Aufgerufen werden muss ...

Also ist diese Funktion nicht Threadsafe ... :coderselixir:

Debug Ausgabe:

Code: Alles auswählen

[08:18:18] Warte auf den Start des Executable...
[08:18:18] Executable-Typ: Linux - x64  (64bit, Unicode, Thread)
[08:18:18] Executable gestartet.
[08:18:18] [Debug] Test
[08:18:18] [Debug] Start Allocate
[08:18:18] [Debug] Alocated Address:8049624
[08:18:18] [Debug] Randomcrypt.
[08:18:18] [Debug] Memoryfreigabe
[08:18:18] [Debug] P_Numbers
[08:18:18] [Debug] Umwandeln zu String
[08:18:18] [Debug] Rückgabe:D37A3444-6A90-40F6-8D4C-D3CB4DCB81AF
[08:18:18] [Debug] Start Allocate
[08:18:18] [Debug] Alocated Address:8049720
[08:18:18] [Debug] Randomcrypt.
[08:18:18] [Debug] Memoryfreigabe
[08:18:18] [Debug] P_Numbers
[08:18:18] [Debug] Umwandeln zu String
[08:18:18] [Debug] Rückgabe:833AC84B-CCC1-41AD-8219-E972B79DF4E0
[08:18:18] [Debug] Start Allocate
[08:18:18] [Debug] Alocated Address:7990584
[08:18:18] [Debug] Randomcrypt.
[08:18:18] [Debug] Memoryfreigabe
[08:18:18] [Debug] P_Numbers
[08:18:18] [Debug] Umwandeln zu String
[08:18:18] [Debug] Rückgabe:948A2CE7-55C2-427A-AA33-9CCECF8620A7
[08:18:18] [Debug] Start Allocate
[08:18:18] [Debug] Alocated Address:8049720
[08:18:18] [Debug] Randomcrypt.
[08:18:18] [Debug] Memoryfreigabe
[08:18:18] [Debug] P_Numbers
[08:18:18] [Debug] Umwandeln zu String
[08:18:18] [Debug] Rückgabe:4F2CB4DC-B3C3-456B-8B47-A39138E69E73
[08:18:18] [Debug] Start Allocate
[08:18:18] [Debug] Alocated Address:7990776
[08:18:18] [Debug] Randomcrypt.
[08:18:18] [Debug] Memoryfreigabe
[08:18:18] [Debug] P_Numbers
[08:18:18] [Debug] Umwandeln zu String
[08:18:18] [Debug] Rückgabe:08EED4EF-8FD7-477E-9E2D-EC0E7C0B5C01
[08:18:18] [Debug] D37A3444-6A90-40F6-8D4C-D3CB4DCB81AF
[08:18:18] [Debug] 833AC84B-CCC1-41AD-8219-E972B79DF4E0
[08:18:18] [Debug] 948A2CE7-55C2-427A-AA33-9CCECF8620A7
[08:18:18] [Debug] 4F2CB4DC-B3C3-456B-8B47-A39138E69E73
[08:18:18] [Debug] Threads Starten:
[08:18:18] [Debug] Thread 1 gestartet.
[08:18:18] [Debug] Thread 2 gestartet.
[08:18:18] [Debug] Thread 3 gestartet.
[08:18:18] [Debug] APP Lock UUID erstellen ID Adress:7990768
[08:18:18] [Debug] APP Lock UUID erstellen ID Adress:8049712
[08:18:18] [Debug] Start Allocate
[08:18:18] [Debug] Thread 4 gestartet.
[08:18:18] [Debug] Start Allocate
[08:18:18] [Debug] Alocated Address:8036328
[08:18:18] [Debug] Threads Gestartet:
[08:18:18] [Debug] Alocated Address:8036776
[08:18:18] [Debug] APP Lock UUID erstellen ID Adress:7990864
[08:18:18] [Debug] Randomcrypt.
[08:18:18] [Debug] Start Allocate
[08:18:18] [Debug] Randomcrypt.
[08:18:18] [Debug] Alocated Address:8037848
[08:18:18] [Debug] APP Lock UUID erstellen ID Adress:7990960
[08:18:18] [Debug] Randomcrypt.
[08:18:18] [Debug] Start Allocate
[08:18:18] [Debug] Alocated Address:8038120
[08:18:18] [Debug] Randomcrypt.
[08:18:18] [Debug] Memoryfreigabe
[08:18:18] [Debug] Memoryfreigabe
[08:18:18] [Debug] P_Numbers
[08:18:18] [Debug] Umwandeln zu String
[08:18:18] [Debug] P_Numbers
[08:18:18] [Debug] Umwandeln zu String
[08:18:18] [Debug] Rückgabe:00000000-0000-4000-8000-000000000000
[08:18:18] [Debug] Lock UUID erstellt:00000000-0000-4000-8000-000000000000
[08:18:18] [Debug] APP 4 Stack_UUID:4F2CB4DC-B3C3-456B-8B47-A39138E69E73
[08:18:18] [Debug] APP 4 : Fill Stack With 100000 UUID's
[08:18:18] [Debug] APP 4 : Timed...
[08:18:18] [Debug] APP 4 : Create Element...
[08:18:18] [Debug] Start Allocate
[08:18:18] [Debug] Rückgabe:B3783F9C-ECD5-48D4-A829-8408690A7998
[08:18:18] [Debug] Lock UUID erstellt:B3783F9C-ECD5-48D4-A829-8408690A7998
[08:18:18] [Debug] Alocated Address:8037944
[08:18:18] [Debug] APP 3 Stack_UUID:948A2CE7-55C2-427A-AA33-9CCECF8620A7
[08:18:18] [Debug] Randomcrypt.
[08:18:18] [Debug] APP 3 : Fill Stack With 100000 UUID's
[08:18:18] [Debug] APP 3 : Timed...
[08:18:18] [Debug] APP 3 : Create Element...
[08:18:18] [Debug] Memoryfreigabe
[08:18:18] [Debug] P_Numbers
[08:18:18] [Debug] Umwandeln zu String
[08:18:18] [Debug] Start Allocate
[08:18:18] [Debug] Alocated Address:8037944
[08:18:18] [Debug] Rückgabe:6EFF8BDB-8019-47BD-B5C4-763EC66EE6F1
[08:18:18] [Debug] APP 4 : Created Element : 6EFF8BDB-8019-47BD-B5C4-763EC66EE6F1
[08:18:18] [Debug] Randomcrypt.
[08:18:18] [Debug] Start Allocate
[08:18:18] [Debug] Alocated Address:8163512
[08:18:18] [Debug] Randomcrypt.
[08:18:18] [Debug] Memoryfreigabe
[08:18:18] [Debug] P_Numbers
[08:18:18] [Debug] 
Das PRogramm hängt einfach da fest und macht nichts mehr, im Kompilierten Modus wie auch im Debug modus und kann einfach mit Ctrl-C Abgebrochen werden...
Also läuft noch irgendwie. Starte ich alle Threads nacheinander mit den WaitThreads(PID1) usw. funktioniert es einwandfrei ... Und jetzt der Source:

core.pb:

Code: Alles auswählen

;**************************************
;* 
;* Core Programm LiHaSo Cloud Suite
;*
;* (c) by Linder Hard- und Software
;*
;This work is licensed under the Creative Commons Attribution-ShareAlike 4.0 International License. 
;To view a copy of this license, 
;visit http://creativecommons.org/licenses/by-sa/4.0/ 
;OR send a letter To Creative Commons, PO Box 1866, Mountain View, CA 94042, USA.

EnableExplicit

#WCS_DEBUG = 5        ;(Möchte nur core.pb Debugs sehen :-) = 0)

XIncludeFile "inc/SYS/GEN/GEN.pbi"

XIncludeFile "inc/SYS/GEN/UUID.pbi"
;XIncludeFile "inc/SYS/STK/STK.pbi" ;Den braucht es nur für einen Stack ansonsten direkt nur den STKMGR verwenden der bindet STK.pbi ein.
;XIncludeFile "inc/SYS/STK/STKMGR.pbi"
;XIncludeFile "inc/CRYPT/CRYMGR.pbi"
;XIncludeFile "inc/CRYPT/TECH/AESSTREAM.pbi"
;XIncludeFile "inc/CRYPT/TECH/AESFILE.pbi"
;XIncludeFile "inc/CRYPT/STK/HASH.pbi"
;XIncludeFile "inc/UI/UIMGR.pbi"
;XIncludeFile "inc/UI/SYNC/SRV.pbi"       ;Für Serverseite datensammelstelle die woanders ausgewertet wird.
;XIncludeFile "inc/UI/SYNC/CLI.pbi"       ;Für Clientseite darstellende Applikation kann über STKMGR-IPC - NW Gesynct werden
;XIncludeFile "inc/UI/GUI/GUIMGR.pbi"
;XIncludeFile "inc/UI/GUI/TYPE/GTK.pbi"
;XIncludeFile "inc/UI/GUI/TYPE/TUI.pbi"
;XIncludeFile "inc/UI/GUI/TYPE/WEB.pbi" ;Benötigt das FCGI Modul
;XINcludeFile "inc/UI/GUI/TYPE/QT.pbi"
;XIncludeFile "inc/IO/MEM/TMPFS.pbi" ;Temp FS Filesystem Function OS (Only Functions who access to a Stack)
;XIncludeFile "inc/IO/FILE/FSMGR.pbi"
;XIncludeFile "inc/IO/FILE/FSSYS/EXT4.pbi"
;XIncludeFile "inc/IO/FILE/FSSYS/NTFS.pbi"
;XIncludeFile "inc/IO/APP/APACHE/FCGI.pbi"
;XIncludeFile "inc/IO/DB/DBMGR.pbi"
;XIncludeFile "inc/IO/DB/SQLITE.pbi"
;XIncludeFile "inc/IO/DB/MYSQL.pbi"
;XIncludeFile "inc/IO/DB/PGSQL.pbi"
;XIncludeFile "inc/IO/NW/NIS.pbi"
;XIncludeFile "inc/IO/NW/NSS.pbi"
;XIncludeFile "inc/IO/NW/NSS.pbi" (Includiert NIS & NOS Selbständig)
;XIncludeFile "inc/IO/IO.pbi"
;XIncludeFile "inc/SYS/STK/STKMGR-IPC.pbi"
;XIncludeFile "inc/SYS/PROC/PLAIN.pbi"
;XIncludeFile "inc/SYS/PROC/ANALYSED/OS/CPU.pbi"
;XIncludeFile "inc/SYS/PROC/ANALYSED/OS/RAM.pbi"
;XIncludeFile "inc/SYS/PROC/ANALYSED/OS/STORE.pbi"
;XIncludeFile "inc/SYS/PROC/ANALYSED/OS/NW.pbi"
;XIncludeFile "inc/SYS/PROC/ANALYSED/APP/WEB/APACHE2.pbi"
;XIncludeFile "inc/SYS/PROC/ANALYSED/APP/JAVA/TOMCAT.pbi"
;XIncludeFile "inc/SYS/PROC/ANALYSED/APP/EMU/QEMU.pbi"
;XIncludeFile "inc/SYS/PROC/ANALYSED/APP/EMU/VIRSH.pbi"
;XIncludeFile "inc/SYS/PROC/ANALYSED/APP/DB/MYSQL.pbi"
;XIncludeFile "inc/APP/APPMGR.pbi"
;XIncludeFile "inc/APP/MON/Datensamler.pbi"
;XIncludeFile "inc/APP/MON/Datenaufbereiter.pbi"
;XIncludeFile "inc/APP/MON/Datenanalyse.pbi"
;XIncludeFile "inc/APP/MON/Benachrichtiger.pbi"
;XIncludeFile "inc/APP/MON/Servermonitor.pbi"
;XIncludeFile "inc/APP/MON/PingService.pbi"

;4 Thread Test
Define Stack_UUID1.s
Define Stack_UUID2.s
Define Stack_UUID3.s
Define Stack_UUID4.s
Define Counter.i
Define Lock_UUID.s
Define Current_UUID.s
Define RandomUUID.s
Define SelectedElem.s
Define StartTime.q

OpenConsole()

Structure Test_Struc
  Counter.i
  UUID.s
EndStructure


Procedure TestThread(StringID.i)
  Protected NewMap TestMap.Test_Struc()
  Protected ID.s
  Protected StartTime.q
  Protected Counter.i
  Protected Current_UUID.s
  Protected Stack_UUID.s
  Protected UUID_Tosave.s
  Protected sLock_UUID.s
  Protected UUID.s
  Debug "APP Lock UUID erstellen ID Adress:"+ Str(StringID)
  sLock_UUID = CreateUUID()
  Debug "Lock UUID erstellt:" + sLock_UUID
  UUID = PeekS(StringID)
  ID = Left(UUID, 1)
  UUID = Right(UUID, 36)
  ;PrintN("APP "+ID+" Stack_UUID:"+UUID)
  Debug "APP "+ID+" Stack_UUID:"+UUID
  Stack_UUID = UUID
  ;PrintN("APP "+ID+" : Fill Stack With 1000 UUID's")
  Debug "APP "+ID+" : Fill Stack With 1000 UUID's"
  ;PrintN("APP "+ID+" : Timed...")
  Debug "APP "+ID+" : Timed..."
  StartTime = ElapsedMilliseconds()
  For Counter = 1 To 1000
    Debug "APP "+ID+" : Create Element..."
    Current_UUID = CreateUUID()
    Debug "APP "+ID+" : Created Element : " + Current_UUID
    UUID_Tosave = CreateUUID()
    Debug "APP "+ID+" : Save to Element..."
    TestMap(Current_UUID)\Counter = Counter
    TestMap(Current_UUID)\UUID = UUID_Tosave
    Debug "APP "+ID+" : Count: "+Str(Counter)+" UUID Saved:"+UUID_Tosave
;    Print("APP "+ID+" : Count: "+Str(Counter)+" UUID Saved:"+UUID_Tosave)
  Next Counter
  Debug "APP "+ID+": Endtime:" + Str(ElapsedMilliseconds()-StartTime)
  ;PrintN("APP "+ID+": Endtime:" + Str(ElapsedMilliseconds()-StartTime))
  Debug "APP "+ID+": Finished"
  ;PrintN("APP "+ID+": Finished")
  
EndProcedure


Debug "Test"
Stack_UUID1 = CreateUUID()
Stack_UUID2 = CreateUUID()
Stack_UUID3 = CreateUUID()
Stack_UUID4 = CreateUUID()

Lock_UUID = CreateUUID()

Define Para1.s 
Define Para2.s 
Define Para3.s 
Define Para4.s 
Para1.s = "1"+Stack_UUID1
Para2.s = "2"+Stack_UUID2
Para3.s = "3"+Stack_UUID3
Para4.s = "4"+Stack_UUID4

Debug Stack_UUID1
Debug Stack_UUID2
Debug Stack_UUID3
Debug Stack_UUID4

Define.i PID1, PID2, PID3, PID4
PrintN("Thread 1:")
Debug "Threads Starten:"
PID1 = CreateThread(@TestThread(), @Para1)
Debug "Thread 1 gestartet."
;WaitThread(PID1)
PrintN("Thread 2:")
PID2 = CreateThread(@TestThread(), @Para2)
Debug "Thread 2 gestartet."
;WaitThread(PID2)
PrintN("Thread 3:")
PID3 = CreateThread(@TestThread(), @Para3)
Debug "Thread 3 gestartet."
;WaitThread(PID3)
PrintN("Thread 4:")
PID4 = CreateThread(@TestThread(), @Para4)
Debug "Thread 4 gestartet."
;WaitThread(PID4)
Debug "Threads Gestartet:"


WaitThread(PID1)
WaitThread(PID2)
WaitThread(PID3)
WaitThread(PID4)


;Debug CreateThread(@TestThread(), Para1)

PrintN("Alle Threads beendet")
PrintN("Beenden? [Enter]")
Input()
; If InitTMPFS()
;   Debug "TMPFS: Initialised"
; Else
;   Debug "TMPFS: Failed."
;   End
; EndIf
; 
; ;Beispiel ein UUID Sync Prozess für mehrere APPS.
; 
; If InitStackIPCManager()
;   Debug "STKMGR-IPC: Initialised"
; Else
;   Debug "STKMGR-IPC: Failed"
;   End
; EndIf
; 
;   
;Debug "FileStatus:"+Str(CreateTMPFSFile(Stack_UUID))

;KillStack(Stack_UUID)
End
inc/SYS/GEN/GEB.pbi

Code: Alles auswählen

;**************************************
;* 
;* GEN.pbi
;*
;* Generische Funktionen Makros
;*
;* (c) by Linder Hard- und Software
;*
;This work is licensed under the Creative Commons Attribution-ShareAlike 4.0 International License. 
;To view a copy of this license, 
;visit http://creativecommons.org/licenses/by-sa/4.0/ 
;OR send a letter To Creative Commons, PO Box 1866, Mountain View, CA 94042, USA.

CompilerIf Not Defined(WCS_DEBUG, #PB_Constant)
  CompilerError "#WCS_DEBUG Muss Definiert sein"
CompilerEndIf

Macro DebugLvl0
  If #WCS_DEBUG >= 0
EndMacro

Macro DebugLvl1
  If #WCS_DEBUG >= 1
EndMacro

Macro DebugLvl2
  If #WCS_DEBUG >= 2
EndMacro

Macro DebugLvl3
  If #WCS_DEBUG >= 3
EndMacro
  
Macro DebugEnd
EndIf
EndMacro
inc/SYS/GEN/UUID.pbi

Code: Alles auswählen

;**************************************
;* 
;* UUID.pbi
;*
;* (c) by Linder Hard- und Software
;*
; To the extent possible under law, the person who associated CC0 With
; WCS_UUID has waived all copyright And related Or neighboring rights
; To WCS_UUID.
;
; You should have received a copy of the CC0 legalcode along With this
; work. If Not, see <http://creativecommons.org/publicdomain/zero/1.0/>.

Enumeration WCS_UUID_Version
  #UUID_V1
  #UUID_V2
  #UUID_V3
  #UUID_V4 ; Standard.
  #UUID_V5
EndEnumeration

Procedure.s CreateUUID(Type.i=#UUID_V4, String.s="")
  ;[T] CreateUUID(Type.i, String.s="")
  ;[D] Gibt eine UUID des Entsprechenden Types  als String zurück.
  ;[D] Derzeit nur Version 4 Implementiert. 
  ;[V] 0.0.1
  ;[M] 0.0.1   
  Protected P_Type.i = Type.i
  Protected P_String.s = String.s
  Protected P_UUID.s = ""
  Protected Dim P_Numbers.a(16)
  ;Protected P_Temp.a
  Protected P_Count.a
  Protected Key
  
  Structure Struc_UUID
   
    low.a 
  EndStructure
 
  Select P_Type
;     Case #UUID_V1
;      
;     Case #UUID_V2
;      
;     Case #UUID_V3
;      
;     Case #UUID_V5
    Case 10
      
    Default     ;#UUID_V4 RFC 4122 Konform
                ;xx xx xx xx - xx xx - 4x xx - yx xx - xx xx xx xx xx xx
      DebugLvl3
      Debug "Start Allocate"
      DebugEnd
      Key = AllocateMemory(16)
      ;Key = 0
            DebugLvl3
      Debug "Alocated Address:"+Str(Key)
      DebugEnd
      If OpenCryptRandom() And Key
      DebugLvl3
      Debug "Randomcrypt."
      DebugEnd
        
        CryptRandomData(Key, 16)
       
        For P_Count = 0 To 15
          P_Numbers(P_Count) = PeekB(Key+P_Count)
        Next P_Count     
       
        CloseCryptRandom()
      Else
        ;Not Save
      DebugLvl3
      Debug "Unsave Random"
      DebugEnd
        
        For P_Count = 0 To 15
          P_Numbers(P_Count) = Random(255)
        Next P_Count
      EndIf
      DebugLvl3
      Debug "Memoryfreigabe"
      DebugEnd
     
      If Key
        FreeMemory(Key)
      EndIf
      DebugLvl3
      Debug "P_Numbers"
      DebugEnd
     
      P_Numbers(6) = Val("$4"+Right(RSet(Hex(P_Numbers(6), #PB_Byte), 2, "0"), 1))
      P_Numbers(8) = Val("%10"+Right(RSet(Bin(P_Numbers(8), #PB_Byte), 8, "0"), 6))
      DebugLvl3
      Debug "Umwandeln zu String"
      DebugEnd
      
      P_UUID = ""
      
      For P_Count = 0 To 15
        If P_Count = 4 Or P_Count = 6 Or P_Count = 8 Or P_Count = 10
          P_UUID + "-"
        EndIf
        P_UUID + RSet(Hex(P_Numbers(P_Count), #PB_Byte), 2, "0")
      Next P_Count
     
  EndSelect
      DebugLvl3
      Debug "Rückgabe:"+P_UUID
      DebugEnd

  ProcedureReturn P_UUID
EndProcedure

Procedure IsUUID(UUID.s , Type.i = #UUID_V4)
  ;[T] IsUUID(UUID.s , Type.i = #UUID_V4) 
  ;[D] Prüft ob die UUID entsprechend Version 4 und RFC 4122 Konform ist.
  ;[V] 0.0.1
  ;[M] 0.0.1     
  If Mid(UUID.s, 15,1) = Str(Type+1)
    If Left(Bin(Val("$"+Mid(UUID.s, 15,2)), #PB_Byte),2) = "10"
      ProcedureReturn #True
    EndIf
  EndIf
  
  ProcedureReturn #False
EndProcedure
Zuletzt geändert von Ground0 am 01.03.2017 10:41, insgesamt 2-mal geändert.
lmon Monitoring Service inkl. Clients
lweb Multi-Threaded Webserver
lbup-server Backup Server applikation
lmanager ERP Applikation.
Benutzeravatar
NicTheQuick
Ein Admin
Beiträge: 8675
Registriert: 29.08.2004 20:20
Computerausstattung: Ryzen 7 5800X, 32 GB DDR4-3200
Ubuntu 22.04.3 LTS
GeForce RTX 3080 Ti
Wohnort: Saarbrücken
Kontaktdaten:

Re: Zufälliges Problem mit Multithreading

Beitrag von NicTheQuick »

Könntest du noch etwas mehr Informationen liefern, in etwa wie hier beschrieben: Wie melde ich einen Bug?
Vor allem die Compiler-Einstellungen wären interessant.
Bild
Benutzeravatar
Ground0
Beiträge: 318
Registriert: 05.02.2005 02:09
Wohnort: Stilli, Schweiz
Kontaktdaten:

Re: Zufälliges Problem mit Multithreading

Beitrag von Ground0 »

Danke für den Hinweis ... nun habe ich den "Fehler"/Bug gefunden da ich das ganze noch unter Windows Getestet habe...

Liegt wohl an dem das für alle Globale Initialisierungen ein Init*** in der Hilfe steht und dieser Open heisst. Daher jeweils in der Prozedur Geöffnet und geschlossen und der andere Thread motzt natürlich das er Geschlossen wurde...

Entweder würde ich sagen einen Globalen Init und OpenCryptRandom() in InitCryptRandom() Umwandeln (Analog den anderen Befehlen) oder einen Handle zurückgeben der auch im Zusammenhang genutzt werden kann und natürlich die entsprechenden Befehle so anpassen das diese mit einem Handle umgehen können... :D
lmon Monitoring Service inkl. Clients
lweb Multi-Threaded Webserver
lbup-server Backup Server applikation
lmanager ERP Applikation.
Antworten