MealMaster: Decoding text file format to database

[Fangbeast]

What: RecipeMuncher recipe manager
Why: Because other wives threatened to bash me if I didn't
Credits: InfraTec, ElectroChrisso, yrreti, RASHAD and others.
License: Free to use and abuse but may not be sold.

Notes:

--Tested on 1.25 gigabytes of uncompressed, raw recipe files.

--Infratec constantly adds and improves all the recipe importing code into a universal importer and puts it into the forum on page 6 of this thread. I could not do it. (Latest code added to the distribution archive)

--I constantly attempt to integrate his changes into RecipeMuncher and upload the latest program here:

https://www.box.com/s/8txhhbzrriqzmvsuf6ua

--I have cleaned up all the non relevant/old update posts to make sure that there is absolutely no confusion over which is the current code and information. All old versions are removed from the link above as the new ones are posted.

--The latest changes to the program will always be written in this first post. The old changes log will be removed.

--I will attempt to update a new version every few days as I make enough changes or fixes.

--Request Would a moderator move all this to Tips n Tricks please and delete all posts that I have marked as 'Outdated' in the body?

**Note** The updated link above is to all the PureBasic projects I mess around with, RecipeMuncher separately and the recipe archives.

======[current changes]========================

!!IMPORTANT NOTE!! I forgot to mention that this version uses constants from PB5.1x.

@Everyone else. v1.41b is at the download link. Updated help file there as well.

25/02/2013

Added Reloadable keyboard shortcuts when they have been edited.

Added Edit form for keyboard shortcuts. Took longer than I thought.

24/02/2013

Added A huge update. The ability to open another database, temporarily attach to your main one, search for and transfer copies of the attached database recipes to your own. Read the Recipe transfer between databases section for the details.

Any feedback folks? Or am I back to shagging srod's hamsters again??? (Right, now I really will go back to study)

[spikey]

Well if the example were typical the algorithm would run something like this:

1. Ignore the first two lines.
2. Line 3 is the title - remove whitespace and the string Title: from the front end. [a job for Trim and ReplaceString]
3. Line 4 is indexing keywords, remove whitespace and the string Categories: from the front end, split the remained at , (possibly / too - you could do this by replacing / with , before splitting) - use the resultant string list to index. [StringField]
4. Line 5 is the portion yield - remove whitespace and the string Yield: from the front end.
5. Line 6 is blank but denotes the start of the ingredients.
6. Multiple ingredients on a single line are separated from each other by more then 5 whitespaces. [CountString, FindString and then Trim]
7. Quantities are separated from units by a single space, units from ingredients by two spaces.
8. Another blank line denotes the end of the ingredients and the start of the method.
9. The flag MMMMM terminates the file.

The problem arises by how many variants on a theme there are and how far they 'deviate' from the 'mean'.

You could be more selective - by say looking for the presence of the words Title, Catagories and Yield to correctly locate and process these lines but I'm guessing that the problems are going to be identifying the start and end of the ingredients list and separating the multiple entry ingredients lines from each other if the 5 whitespace rule doesn't hold true - providing it doesn't go less than 3 whitespaces at any time it should be feasible though.

If there are a few deviations then you could process them all this way and then identify and correct those that bomb out manually - or possible just delete them and reenter manually in their entirety.

If there are a lot then you will need to identify their deviance and process separately.

[infratec]

Hi Fangbeast,

give this a try:

Code: Select all

Enumeration
  #RecipeNone
  #RecipeHeaderFound
  #RecipeHeaderInfo
  #RecipeIncredients
  #RecipeMethod
  #RecipeEnd
EndEnumeration


UseSQLiteDatabase()


Procedure.i ImportRecipe(DB.i, Filename$)
  
  Protected.i Result, File, State, Pos
  Protected Line$, Text$, Title$, Category$, Yield$, Incredients$, Method$
  
  
  File = ReadFile(#PB_Any, Filename$)
  If File
    
    While Not Eof(File)
      Line$ = ReadString(File)
      Select State
        Case #RecipeNone
          If FindString(Line$, "MMMMM-----")
            State = #RecipeHeaderFound
          EndIf
        Case #RecipeHeaderFound
          If Len(Trim(Line$)) = 0
            State = #RecipeHeaderInfo
          EndIf
        Case #RecipeHeaderInfo
          If Len(Trim(Line$)) = 0
            Incredients$ = ""
            State = #RecipeIncredients
          Else
            Pos = FindString(Line$, "Title:")
            If Pos
              Title$ = Trim(Mid(Line$, Pos + 7))
              Debug Title$
            Else
              Pos = FindString(Line$, "Categories:")
              If Pos
                Category$ = Trim(Mid(Line$, Pos + 12))
                Debug Category$
              Else
                Pos = FindString(Line$, "Yield:")
                If Pos
                  Yield$ = Trim(Mid(Line$, Pos + 7))
                  Debug Yield$
                EndIf
              EndIf
            EndIf
          EndIf
        Case #RecipeIncredients
          If Len(Trim(Line$)) = 0
            Debug Incredients$
            Method$ = ""
            State = #RecipeMethod
          Else
            Line$ = Trim(Line$)
            Pos = FindString(Line$, "   ")
            If Pos
              Incredients$ + Trim(Left(Line$, Pos)) + #CRLF$
              Incredients$ + Trim(Mid(Line$, Pos)) + #CRLF$
            Else
              Incredients$ + Trim(Line$) + #CRLF$
            EndIf
          EndIf
        Case #RecipeMethod
          If FindString(Line$, "MMMMM")
            Debug Method$
            
            If IsDatabase(DB)
              SQL$ = "INSERT INTO ... "
              SQL$ + "VALUES ("
              SQL$ + "'" + Category$ + "',"
              SQL$ + "'" + Title$ + "',"
              SQL$ + "'" + Yield$ + "',"
              SQL$ + "'" + Incredients$ + "',"
              SQL$ + "'" + Method$ + "')"
              DatabaseUpdate(DB, SQL$)
            EndIf
            
            State = #RecipeNone
          Else
            Line$ = Trim(Line$)
            If Len(Line$) <> 0
              Method$ + Line$ + #CRLF$
            EndIf
          EndIf
      EndSelect
    Wend
    
    CloseFile(File)
  EndIf
  
  ProcedureReturn Result
  
EndProcedure


ImportRecipe(0, "Recipe01.txt")

Bernd

[Kiffi]

@Bernd:

little typo:

Code: Select all

SQL$ + "'" + Method$ + ")"

should be:

Code: Select all

SQL$ + "'" + Method$ + "')"

Greetings ... Kiffi

[infratec]

Thanks Kiffi

Since I didn't test the database stuff, it was overseen from me
I corrected the listing above.

Bernd

[Fangbeast]

Moderator, Please remove this post, outdated.

[Fangbeast]

Moderator, Please remove this post, outdated.

[IdeasVacuum]

...The MealMaster app is freeware now - pretty comprehensive too. Shame there is a lot of DOS in there though.

[electrochrisso]

Hey! This MealMaster stuff is addictive.
I can get it working, but it is time to build a better user interface.

[Fangbeast]

Hey! This MealMaster stuff is addictive.

What can I say, we love to cook:):)

[electrochrisso]

Fangbeast, I posted some code in the Pure Punch contest for Meal Master.

[Fangbeast]

Fangbeast, I posted some code in the Pure Punch contest for Meal Master.

You did?? Damn, how did I miss that??

Okay, looking now

[electrochrisso]

Fangbeast, I posted some code in the Pure Punch contest for Meal Master.

You did?? Damn, how did I miss that??

Okay, looking now

Well, I only just posted it a little while ago, and I also downloaded at least 10,000 recipes.

[electrochrisso]

Here you go Fangbeast, the code a little easier to read, I also have put a printing routine in too.

Code: Select all

OpenWindow(0,100,0,842,540,"Menu Master Quickview",#PB_Window_SystemMenu)
CreateToolBar(0,WindowID(0)):ToolBarStandardButton(0,#PB_ToolBarIcon_Open)
ToolBarStandardButton(1,#PB_ToolBarIcon_Print)
ListViewGadget(0,0,28,200,512):EditorGadget(1,202,28,640,512,#PB_Editor_ReadOnly)
SetGadgetColor(0,#PB_Gadget_BackColor,$FAFFF5)
SetGadgetColor(1,#PB_Gadget_BackColor,$F0FFFF)
SetGadgetFont(1,LoadFont(0,"Courier",10))

Procedure Editor_Print(Gadget.l,docname.s)
  Protected DC
  lppd.PRINTDLG
  lppd\lStructsize=SizeOf(PRINTDLG)
  lppd\Flags=#PD_ALLPAGES|#PD_HIDEPRINTTOFILE|#PD_NOSELECTION|#PD_RETURNDC
  PrintDlg_(lppd)
  DC=lppd\hDC
  cRect.RECT
  FormatRange.FORMATRANGE
  Docinfo.Docinfo
  Docinfo\cbSize = SizeOf(Docinfo)
  Docinfo\lpszDocName = @docname
  Docinfo\lpszOutput = #Null
  StartDoc_(DC,Docinfo)
  LastChar=0
  MaxLen=Len(GetGadgetText(Gadget))-SendMessage_(GadgetID(Gadget),#EM_GETLINECOUNT,0,0) 
  OldMapMode=GetMapMode_(DC)
  SetMapMode_(DC,#MM_TWIPS)
  OffsetX = GetDeviceCaps_(DC,#PHYSICALOFFSETX)
  OffsetY = -GetDeviceCaps_(DC,#PHYSICALOFFSETY)
  HorzRes = GetDeviceCaps_(DC,#HORZRES)
  VertRes = -GetDeviceCaps_(DC,#VERTRES)
  SetRect_(cRect,OffsetX,OffsetY,HorzRes,VertRes)
  DPtoLP_(DC,cRect,2)
  SetMapMode_(DC,OldMapMode)
  FormatRange\hDC = DC
  FormatRange\hdcTarget = DC
  FormatRange\rc\left = cRect\left
  FormatRange\rc\top = cRect\top
  FormatRange\rc\right = cRect\right
  FormatRange\rc\bottom = cRect\bottom
  FormatRange\rcPage\left = cRect\left
  FormatRange\rcPage\top = cRect\top
  FormatRange\rcPage\right = cRect\right
  FormatRange\rcPage\bottom = cRect\bottom
  a=1
  Repeat
    StartPage_(DC)
    FormatRange\chrg\cpMax = -1 
    LastChar = SendMessage_(GadgetID(Gadget),#EM_FORMATRANGE,#True,@FormatRange)
    FormatRange\chrg\cpMin = LastChar
    SendMessage_(GadgetID(Gadget),#EM_DISPLAYBAND,0,cRect)
    a+1
    EndPage_(DC)
  Until LastChar >= MaxLen Or LastChar = -1
  EndDoc_(DC)
  SendMessage_(GadgetID(Gadget),#EM_FORMATRANGE,0,0)
EndProcedure

Repeat
  Select WaitWindowEvent()
    Case #PB_Event_Menu
      Select EventMenu()
        Case 0
          f$=OpenFileRequester("Choose Menu Master Recipe File",GetCurrentDirectory(),"Text (*.txt)",0)
          If f$
            If ReadFile(0,f$)
              ClearGadgetItems(0):SetGadgetText(1,"")
              Repeat
                a$=ReadString(0)
                If Left(a$,12)="      Title:"
                  AddGadgetItem(0,-1,Mid(a$,13))
                EndIf
              Until Eof(0)
              CloseFile(0)
            EndIf
          EndIf
          
        Case 1
          Editor_Print(1,"Testing")
          
      EndSelect
      
    Case #PB_Event_Gadget
      Select EventGadget()
        Case 0
          If ReadFile(0,f$)
            Repeat
              a$=ReadString(0)
            Until a$=Left(a$,12)+GetGadgetItemText(0,GetGadgetState(0))
            While a$<>"MMMMM"
              If a$<>"MMMMM"
                b$+a$+#CRLF$
                a$=ReadString(0)
              EndIf
            Wend
            CloseFile(0)
          EndIf
          SetGadgetText(1,b$):b$=""
      EndSelect
      
    Case #PB_Event_CloseWindow
      End
      
  EndSelect
ForEver

[Fangbeast]

electrochrisso, that's service for you. I promise to never send srod's diseased sheep your way for the help. And his festering underpants and painfully abused goat collection.

I promise to only do that to Idle from now on!!

Now I have to study your code.

I have it in mind to enable checkboxes in the title list and when I point to a recipe that is in the correct format, it will be added to the database and removed from the originating file.

Probably smarter for me to start with the smaller original recipes and not from the concatenated files that I exported out of the MealMaster DOS program.

I have several hundred thousand and I intend to get more. My dream is to have ALL of them in a database.