Base64 decoding and blobs issue

[Fangbeast]

Without posting the entire 22,000 line program somewhere, could someone tell me how this code below could generate a memory read error at the DatabaseUpdate statement? Because that is what the compiler is telling me.

A good bloke is helping me but he can't do everything and I am stumped. Can send entire project if needed. Or post just the data import procedure that this is part of

Code: Select all

ActualStringLength.i   = StringByteLength(PictureString.s, #PB_Ascii)
Debug "Actual picture length: " + Str(ActualStringLength.i)
*DecodeBuffer = AllocateMemory(ActualStringLength.i + (ActualStringLength.i / 30))
If *DecodeBuffer
  Debug "Decoding buffer: " + Str(*DecodeBuffer)
  *OutputBuffer = AllocateMemory(ActualStringLength.i)
  If *OutputBuffer
    Debug "Output buffer: "   + Str(*OutputBuffer)
    BytesPoked.i = PokeS(*DecodeBuffer, AddressItem\Picture, ActualStringLength.i, #PB_Ascii | #PB_String_NoZero)
    If BytesPoked.i <> #NoData
      Debug "Bytes poked: " + Str(BytesPoked.i)
      ActualBytesDecoded.i = Base64Decoder(*DecodeBuffer, ActualStringLength.i, *OutputBuffer, ActualStringLength.i)
      If ActualBytesDecoded.i
        Debug "Bytes decoded: "   + Str(ActualBytesDecoded.i)
        SetDatabaseBlob(DatabaseHandle.i, 0, *OutputBuffer, ActualBytesDecoded.i)
      Else
        Debug "Could not decode the pifcture string back to picture data"
      EndIf
    Else
      Debug "Could not poke the picture string into the decode buffer"
    EndIf
    FreeMemory(*OutputBuffer)
  EndIf
  FreeMemory(*DecodeBuffer)
EndIf

If DatabaseUpdate(DatabaseHandle.i, DatabaseUpdate.s) <> 0  : ; Invalid memory access (read error at 9946696)
  

[Julian]

insert a line above the databaseupdate to debug print DatabaseUpdate.s to make sure its actually got something in

[sancho2]

Databasehandle is most likely the culprit as databaseupdate problem would return zero from the function.
(edit) Maybe put IsDatabase() before the if statement to see if you really got what you think you got.

But a question: Why after they are declared are you continuing to use the .i and .s on your variables?

[Fangbeast]

Databasehandle is most likely the culprit as databaseupdate problem would return zero from the function.
(edit) Maybe put IsDatabase() before the if statement to see if you really got what you think you got.

DatabaseHandle is okay. A database would not arbitrarily shut itself or disappear in the middle of an operation. All other functions work fine.

The import seems to always fail at the 14'th record being imported unless I remove blobs from the code.

If I delete almost all records except for the last 10, then export and then reimport, everything seems to be okay.

I seem to be able to add, delete and edit records normally, along with their blob data. I can export all records with their blob data. It's the mass import that fails at the 14'th record. Shorter runs seem to work fine.

[TI-994A]

Hi Fangbeast. Although I couldn't say if this is the cause of your woes, a precursory glance at the snippet that you posted exposes this error:

Code: Select all

ActualStringLength.i  = StringByteLength(PictureString.s, #PB_Ascii)
Debug "Actual FILE NAME length: " + Str(ActualStringLength.i)

The StringByteLength() function only returns the length of the file name, and not the file. Accordingly, all subsequent use of the ActualStringLength variable to allocate memory would thus not be sufficient.

Perhaps you could remedy that first, and then let us know how it progresses from there.

[Fangbeast]

Hi Fangbeast. Although I couldn't say if this is the cause of your woes, a precursory glance at the snippet that you posted exposes this error:

Code: Select all

ActualStringLength.i  = StringByteLength(PictureString.s, #PB_Ascii)
Debug "Actual FILE NAME length: " + Str(ActualStringLength.i)

The StringByteLength() function only returns the length of the file name, and not the file. Accordingly, all subsequent use of the ActualStringLength variable to allocate memory would thus not be sufficient.

Perhaps you could remedy that first, and then let us know how it progresses from there.

The PureBasic manual does not say this. It clearly illustrates in the manual that it returns the length of a string and does not mention anywhere in the help file that the string is a file name and not a string of data.

Result = StringByteLength(String$ [, Format])
Description

Returns the number of bytes required to store the string in memory in a given format.

Would you like to show me where in the manual it says this? Certainly not in/on the StringByteLength section.

I'd be happy to send you my entire project to look at:):):):)

[TI-994A]

...It clearly illustrates in the manual that it returns the length of a string and does not mention anywhere in the help file that the string is a file name and not a string of data.

Hello again. You're absolutely correct; an out-of-context presumption.

In the absence of the preceding code, I simply assumed that PictureString was a file name.

Good luck with that, though.

[Fangbeast]

...It clearly illustrates in the manual that it returns the length of a string and does not mention anywhere in the help file that the string is a file name and not a string of data.

Hello again. You're absolutely correct; an out-of-context presumption.

In the absence of the preceding code, I simply assumed that PictureString was a file name.

Good luck with that, though.

I'll need more than luck for this one.

[IdeasVacuum]

Try replacing this

Code: Select all

*DecodeBuffer = AllocateMemory(ActualStringLength.i + (ActualStringLength.i / 30))

With this

Code: Select all

*DecodeBuffer = AllocateMemory(ActualStringLength.i + (ActualStringLength.i / 2)  + SizeOf(character))

[Fangbeast]

Try replacing this

Code: Select all

*DecodeBuffer = AllocateMemory(ActualStringLength.i + (ActualStringLength.i / 30))

With this

Code: Select all

*DecodeBuffer = AllocateMemory(ActualStringLength.i + (ActualStringLength.i / 2)  + SizeOf(character))

I'll try anything, I am desperate (or was that something my wife said to me??)

**EDIT** Ah crap, it's still dying here now, says the memoryid is invalid. Thought the 'if' was supposed to take care of that. The code import 14 out of 32 records in the xml export file before crashing. Both the last record imported and the one after and perfectly ordinary xml elements, nothing special or hidden that I can see.

If *OutputBuffer
FreeMemory(*OutputBuffer)
EndIf

[Fangbeast]

Code: Select all

; Import a previously exported or other addresses list


Procedure ImportAddresses()
  
  Structure Contact
    contacttype.s
    contactdetails.s
  EndStructure
  
  Structure AddressItem
    Titlename.s
    Firstname.s
    Middlename.s
    Lastname.s
    Business.s
    Nickname.s
    Customname.s
    Pobox.s
    Street.s
    Suburb.s
    City.s
    State.s
    Postcode.s
    Country.s
    Displayformat.s
    Displayas.s
    Ownername.s
    Category.s
    Comment.s
    Picture.s
    Label.s
    Favourite.s
    Locked.s
    Deleted.s
    Updated.s
    Recordid.s
    List ContactList.Contact()
  EndStructure
  
  Structure AddressBook
    List AddressList.AddressItem()
  EndStructure
  
  Protected.i XMLImportFileHandle.i, DetectionState.i, ImportCounter.i
  Protected   XMLImportFilename.s,   LineToProcess.s, PictureString.s
  Protected   AddressItem.AddressItem
  
  XMLImportFilename.s = OpenFileRequester("Choose an export file to import", "", "XML|*.xml", 0)
  
  ImportCounter.i = #NoItems
  
  If XMLImportFilename.s
    
    XMLImportFileHandle.i = ReadFile(#PB_Any, XMLImportFilename.s)
    
    If XMLImportFileHandle.i
      While Not Eof(XMLImportFileHandle.i)
        LineToProcess.s = XMLToText(LTrim(ReadString(XMLImportFileHandle.i, #PB_UTF8)))
        Select DetectionState.i
          Case 0      ; Address book item
            Select Left(LineToProcess.s, 5)
              Case "<Titl"  : AddressItem\Titlename   = RepQuote(GetPartBetweenBrackets(LineToProcess.s))
              Case "<Firs"  : AddressItem\Firstname   = RepQuote(GetPartBetweenBrackets(LineToProcess.s))
              Case "<Midd"  : AddressItem\Middlename  = RepQuote(GetPartBetweenBrackets(LineToProcess.s))
              Case "<Last"  : AddressItem\Lastname    = RepQuote(GetPartBetweenBrackets(LineToProcess.s))
              Case "<Busi"  : AddressItem\Business    = RepQuote(GetPartBetweenBrackets(LineToProcess.s))
              Case "<Nick"  : AddressItem\Nickname    = RepQuote(GetPartBetweenBrackets(LineToProcess.s))
              Case "<Cust"  : AddressItem\Customname  = RepQuote(GetPartBetweenBrackets(LineToProcess.s))
              Case "<Pobo"  : AddressItem\Pobox       = RepQuote(GetPartBetweenBrackets(LineToProcess.s))
              Case "<Stre"  : AddressItem\Street      = RepQuote(GetPartBetweenBrackets(LineToProcess.s))
              Case "<Subu"  : AddressItem\Suburb      = RepQuote(GetPartBetweenBrackets(LineToProcess.s))
              Case "<City"  : AddressItem\City        = RepQuote(GetPartBetweenBrackets(LineToProcess.s))
              Case "<Stat"  : AddressItem\State       = RepQuote(GetPartBetweenBrackets(LineToProcess.s))
              Case "<Post"  : AddressItem\Postcode    = RepQuote(GetPartBetweenBrackets(LineToProcess.s))
              Case "<Coun"  : AddressItem\Country     = RepQuote(GetPartBetweenBrackets(LineToProcess.s))
              Case "<Disp"  
                If Left(LineToProcess.s, 9) = "<Displayf"
                  AddressItem\Displayformat = RepQuote(GetPartBetweenBrackets(LineToProcess.s))
                Else
                  AddressItem\Displayas     = RepQuote(GetPartBetweenBrackets(LineToProcess.s))
                EndIf
              Case "<Owne"  : AddressItem\Ownername   = RepQuote(GetPartBetweenBrackets(LineToProcess.s))
              Case "<Cate"  : AddressItem\Category    = RepQuote(GetPartBetweenBrackets(LineToProcess.s))
              Case "<Comm"  : AddressItem\Comment     = RepQuote(GetPartBetweenBrackets(LineToProcess.s))
              Case "<Pict"  : PictureString.s         = ""    : DetectionState.i = 1
              Case "<Labe"  : AddressItem\Label       = RepQuote(GetPartBetweenBrackets(LineToProcess.s))
              Case "<Favo"  : AddressItem\Favourite   = RepQuote(GetPartBetweenBrackets(LineToProcess.s))
              Case "<Lock"  : AddressItem\Locked      = RepQuote(GetPartBetweenBrackets(LineToProcess.s))
              Case "<Dele"  : AddressItem\Deleted     = RepQuote(GetPartBetweenBrackets(LineToProcess.s))
              Case "<Upda"  : AddressItem\Updated     = RepQuote(GetPartBetweenBrackets(LineToProcess.s))
              Case "<Cont"  : ClearList(AddressItem\ContactList())  : DetectionState.i = 2
              Case "</ele"
                
                ; Add to AddressBook at this point
                
                If AddressItem\Picture  <> #EmptyString
                  Debug "Picture data found, Processing"
                  ActualStringLength.i   = StringByteLength(AddressItem\Picture, #PB_Ascii)
                  Debug "Actual picture length: " + Str(ActualStringLength.i)
;                 *DecodeBuffer = AllocateMemory(ActualStringLength.i)
;                 *DecodeBuffer = AllocateMemory(ActualStringLength.i + (ActualStringLength.i / 30))
                  *DecodeBuffer = AllocateMemory(ActualStringLength.i + (ActualStringLength.i / 2)  + SizeOf(character))
                  If MemorySize(*DecodeBuffer)
                    Debug "Decoding buffer: " + Str(*DecodeBuffer)
                    *OutputBuffer = AllocateMemory(ActualStringLength.i)
                    If MemorySize(*OutputBuffer)
                      Debug "Output buffer: "   + Str(*OutputBuffer)
                      BytesPoked.i = PokeS(*DecodeBuffer, AddressItem\Picture, ActualStringLength.i, #PB_Ascii | #PB_String_NoZero)
                      If BytesPoked.i <> #NoData
                        Debug "Bytes poked: " + Str(BytesPoked.i)
                        ActualBytesDecoded.i = Base64Decoder(*DecodeBuffer, ActualStringLength.i, *OutputBuffer, ActualStringLength.i)
                        If ActualBytesDecoded.i
                          Debug "Bytes decoded: "   + Str(ActualBytesDecoded.i)
                          SetDatabaseBlob(Program\DatabaseHandle, 0, *OutputBuffer, ActualBytesDecoded.i)
                        Else
                          SetInfobarArea("Headings", "Error", "Could not decode the picture string back to picture data", "ImportAddresses")
                        EndIf
                      Else
                        SetInfobarArea("Headings", "Error", "Could not poke the picture string into the decode buffer", "ImportAddresses")
                      EndIf
                      ; FreeMemory(*OutputBuffer)
                    EndIf
                    FreeMemory(*DecodeBuffer)
                  EndIf
                  Debug "==============================================================================================================="                  
                  AddressItem\Picture = ""
                  DatabaseUpdate.s = "INSERT INTO Addresses("
                  DatabaseUpdate.s + "Titlename, Firstname, Middlename, Lastname, Business, Nickname, "
                  DatabaseUpdate.s + "Customname, Pobox, Street, Suburb, City, State, Postcode, Country, "
                  DatabaseUpdate.s + "Displayformat, Displayas, Ownername, Category, Comment, Picture, Label, "
                  DatabaseUpdate.s + "Favourite, Locked, Deleted, Updated) "
                  DatabaseUpdate.s + "VALUES('"  + AddressItem\Titlename     + "', '" + AddressItem\Firstname   + "', '" + AddressItem\Middlename
                  DatabaseUpdate.s + "', '"      + AddressItem\Lastname      + "', '" + AddressItem\Business    + "', '" + AddressItem\Nickname
                  DatabaseUpdate.s + "', '"      + AddressItem\Customname    + "', '" + AddressItem\Pobox       + "', '" + AddressItem\Street
                  DatabaseUpdate.s + "', '"      + AddressItem\Suburb        + "', '" + AddressItem\City        + "', '" + AddressItem\State
                  DatabaseUpdate.s + "', '"      + AddressItem\Postcode      + "', '" + AddressItem\Country     + "', '" + AddressItem\Displayformat
                  DatabaseUpdate.s + "', '"      + AddressItem\Displayas     + "', '" + AddressItem\Ownername   + "', '" + AddressItem\Category
                  DatabaseUpdate.s + "', '"      + AddressItem\Comment       + "', "                            + "?, '" + AddressItem\Label
                  DatabaseUpdate.s + "', '"      + AddressItem\Favourite     + "', '" + AddressItem\Locked      + "', '" + AddressItem\Deleted
                  DatabaseUpdate.s + "', '"      + AddressItem\Updated       + "')"
                Else
                  DatabaseUpdate.s = "INSERT INTO Addresses("
                  DatabaseUpdate.s + "Titlename, Firstname, Middlename, Lastname, Business, Nickname, "
                  DatabaseUpdate.s + "Customname, Pobox, Street, Suburb, City, State, Postcode, Country, "
                  DatabaseUpdate.s + "Displayformat, Displayas, Ownername, Category, Comment, Label, "
                  DatabaseUpdate.s + "Favourite, Locked, Deleted, Updated) "
                  DatabaseUpdate.s + "VALUES('"  + AddressItem\Titlename     + "', '" + AddressItem\Firstname   + "', '" + AddressItem\Middlename
                  DatabaseUpdate.s + "', '"      + AddressItem\Lastname      + "', '" + AddressItem\Business    + "', '" + AddressItem\Nickname
                  DatabaseUpdate.s + "', '"      + AddressItem\Customname    + "', '" + AddressItem\Pobox       + "', '" + AddressItem\Street
                  DatabaseUpdate.s + "', '"      + AddressItem\Suburb        + "', '" + AddressItem\City        + "', '" + AddressItem\State
                  DatabaseUpdate.s + "', '"      + AddressItem\Postcode      + "', '" + AddressItem\Country     + "', '" + AddressItem\Displayformat
                  DatabaseUpdate.s + "', '"      + AddressItem\Displayas     + "', '" + AddressItem\Ownername   + "', '" + AddressItem\Category
                  DatabaseUpdate.s + "', '"      + AddressItem\Comment       + "',' "                           +          AddressItem\Label
                  DatabaseUpdate.s + "', '"      + AddressItem\Favourite     + "', '" + AddressItem\Locked      + "', '" + AddressItem\Deleted
                  DatabaseUpdate.s + "', '"      + AddressItem\Updated       + "')"
                EndIf
                
                Debug DatabaseUpdate.s
                
                If DatabaseUpdate(Program\DatabaseHandle, DatabaseUpdate.s) <> #DatabaseUpdateFail
                  AddressItem\RecordId = DatabaseLastInsertRowId()    ; This will be the linked Recordid for the contacts
                  If AddressItem\Recordid <> #EmptyString
                    SetInfobarArea("Headings", "Info", "Inserted record " + AddressItem\RecordId  + " into the database" , "ImportAddresses")
                    ImportCounter.i + 1
                    ForEach AddressItem\ContactList()
                      DatabaseUpdate.s  = "INSERT INTO Contacts(Contacttype, Contactdetails, Recordid) "  ; Contactid is an autoincrement field
                      DatabaseUpdate.s  + "VALUES("
                      DatabaseUpdate.s  + "'"  + AddressItem\ContactList()\contacttype    + "', "
                      DatabaseUpdate.s  + "'"  + AddressItem\ContactList()\contactdetails + "', "
                      DatabaseUpdate.s  + "'"  + AddressItem\RecordId + "')"
                      If DatabaseUpdate(Program\DatabaseHandle, DatabaseUpdate.s) <> #DatabaseUpdateFail
                        SetInfobarArea("Headings", "Info", "Inserted contact for current record into database" , "ImportAddresses")
                      Else
                        SetInfobarArea("Headings", "Error", "Could not insert imported main contacts table: " + DatabaseError(), "ImportAddresses")
                      EndIf
                    Next
                  Else
                    SetInfobarArea("Headings", "Error", "No Recordid found so we cannot write the contact details to the database out", "ImportAddresses")
                  EndIf
                  If *OutputBuffer
                    FreeMemory(*OutputBuffer)
                  EndIf
                Else
                  SetInfobarArea("Headings", "Error", "The database import failed " + DatabaseError(), "ImportAddresses")
                EndIf
                ; No more processing on this line
            EndSelect
            ; Picture item
          Case 1
            If LineToProcess.s = "</Picture>"
              AddressItem\Picture = PictureString.s
              PictureString.s = ""
              DetectionState.i = 0
            Else
              PictureString.s + LineToProcess.s
            EndIf
            ; Contact book item
          Case 2
            If LineToProcess.s = "</ContactList>"
              DetectionState.i = 0
            Else
              If LineToProcess.s = "<element>"
                AddElement(AddressItem\ContactList())
              Else
                Select Mid(LineToProcess.s, 9, 3)
                  Case "typ"
                    AddressItem\ContactList()\contacttype = GetPartBetweenBrackets(LineToProcess.s)
                  Case "det"
                    AddressItem\ContactList()\contactdetails = GetPartBetweenBrackets(LineToProcess.s)
                EndSelect
              EndIf
            EndIf            
            ; Stop processing information in the current line
        EndSelect
        ; Process the next line of the import file
      Wend
      ; 
      CloseFile(XMLImportFileHandle.i)
      XMLImportFilename.s = ""
      PostEvent(#PB_Event_Gadget, #Window_AddressBook, #Gadget_AddressBook_Titles, #PB_EventType_Change)
      ; 
    Else
      SetInfobarArea("Headings", "Error", "Got no handle for the import file name", "ImportAddresses")
    EndIf
  Else
    SetInfobarArea("Headings", "Error", "Filename importing was cancelled by the user", "ImportAddresses")
  EndIf
EndProcedure

[IdeasVacuum]

What memoryid is invalid?

I think this post needs help from somebody that knows what they are talking about, but while you are waiting:

AddressItem\Picture is a Base 64 encoded string?

If so, try changing this

Code: Select all

ActualStringLength.i   = StringByteLength(AddressItem\Picture, #PB_Ascii)

to this

Code: Select all

ActualStringLength.i   = StringByteLength(AddressItem\Picture, #PB_UTF8)

...and change this

Code: Select all

BytesPoked.i = PokeS(*DecodeBuffer, AddressItem\Picture, ActualStringLength.i, #PB_Ascii | #PB_String_NoZero)

to this

Code: Select all

BytesPoked.i = PokeS(*DecodeBuffer, AddressItem\Picture, ActualStringLength.i, #PB_UTF8 | #PB_String_NoZero)

[IdeasVacuum]

Also, if an expert chips in to help, he/she will no doubt need to see the XML file - can you upload it (zipped) somewhere?

[Fangbeast]

Also, if an expert chips in to help, he/she will no doubt need to see the XML file - can you upload it (zipped) somewhere?

Hello IdeasVacuum, thanks for trying to help me. InfraTec has been giving me his valuable time as well with enormous amounts of help.

I can't post the xml file as it has information for security forces that cannot be distributed. And if I changed it to generic data, I might destroy whatever hidden character (I am assuming that is what it is)
that is trashing the code.

This is the error message

[22:28:52] Executable type: Windows - x86 (32bit, Unicode, Purifier)
[22:28:52] Executable started.
[22:29:14] [ERROR] _ImportAddresses.pbi (Line: 336)
[22:29:14] [ERROR] The specified '*MemoryID' is not valid.

This is the section

If *OutputBuffer
FreeMemory(*OutputBuffer)
EndIf

We were changing the buffer around to see where it should be closed and getting invalid memory read errors at a DatabaseUpdate statement????

[IdeasVacuum]

This section

Code: Select all

If *OutputBuffer
FreeMemory(*OutputBuffer)
EndIf

Isn't in the code you posted, but you can simply comment it out to see what happens next.

Edit: yes it is there!

You could change it to this, which is working elsewhere in your code:

Code: Select all

If MemorySize(*OutputBuffer)
   FreeMemory(*OutputBuffer)
EndIf

...so an attempt to free memory will only occur if the allocation was successful

If you think there could be corruption/hidden chars in the XML file, make a copy of the file (backup) and then load it into a text editor (like UltraEdit or NotePad++). With the file now displayed as text rather than XML, it should be easy to spot a fault.