Graphical Artificial Neural Net

[pdwyer]

This is something I've wanted to do for a lot of years but whether VB, powerbasic or Purebasic, I could never get working. Recently I had another idea for a use for it so to test it I wanted to do the good old basic character recognition thing for numerals (ANN's 101). I finally got it working and now have a basic ann.pbi built that handles creation of a perceptron with 1 hidden layer and backpropergation.

Next, in order to see it I created this little app with some very ugly hacked code as I was in a hurry to see how it worked, it has all sorts of horrible code like calling the include files globals directly and lots of hard coded positional values . It helped me debug a few things with regards to neurons firing and some issues with the weight state distribution.

You can change the number of hidden neurons or the learning level to see how it performs. Run it at "Fast" to get through a few hundred epochs to get to a point where the task is getting solved then slow it down to actually see it. On high speed, the flickering makes it hard to see how the solution works. The net usually converges on a solution at between 700 and 2000 epochs although it does get stuck sometimes.

First the Include file... "ANN.PBI"

Code: Select all

#HiddenLayer    = 1
#OutputLayer    = 2

;Export Functions
Declare ANNCreate(InputCount.l, HiddenCount.l, OutputCount.l, LearningRate.d) 
Declare ANNTrainSet(Array Inputs.d(1), Array Outputs.d(1), Array AnswerOutputs.d(1)) ;'Back propergates and learns
Declare ANNQuerySet(Array Inputs.d(1), Array Outputs.d(1)) 
Declare BackProp(ANNLayer.l )   
Declare.d MeanSqrErr() 
Declare CopyDoubleArray(Array DestArray.d(1), Array SrcArray.d(1) ) 
Declare CopyDouble2DArray(Array DestArray.d(2), Array SrcArray.d(2))
Declare  LoadWeights(Filename.s)
Declare  SaveWeights(Filename.s)
 
;NonExported Functions
Declare  CalcLayerOutput(ANNLayer.l) 
    ;'if %HiddenLayer, take raw inputs, hidden weights and write to hidden neurons
    ;'if %Outputlayer, Take Hiddenlayer vals as inputs, output weights and write to output neurons
Declare.d ActivateNeuron(InputTotal.d)

;'Global Vars
Global ANNInputCount.l 
Global ANNHiddenCount.l 
Global ANNOutputCount.l 
Global ANNLearningRate.d

Global Dim ANNInputs.d(0)             ;'Raw input values 
Global Dim ANNHiddenNeurons.d(0) ;'output values
Global Dim ANNOutputNeurons.d(0)       ;'output values 
Global Dim ANNHiddenWeights.d(0,0)       ;'weight values
Global Dim ANNOutputWeights.d(0,0)       ;'weight values    
Global Dim ANNAnswerOutputs.d(0)       ;'Answer outputs to test against    
Global Dim ANNOutputErrors.d(0) ;'Error value from the output layer to be remembered for the hidden layer backprop
                        

;'Init Network
Procedure ANNCreate(InputCount.l, HiddenCount.l, OutputCount.l,LearningRate.d)

    ;Randomize Timer
    
    RndLoop1.l
    RndLoop2.l
        
    ANNInputCount = InputCount
    ANNHiddenCount = HiddenCount
    ANNOutputCount = OutputCount
    ANNLearningRate = LearningRate
    
    Dim ANNInputs.d(ANNInputCount) 
    Dim ANNHiddenWeights.d(ANNInputCount, ANNHiddenCount)
    Dim ANNOutputWeights.d(ANNHiddenCount, ANNOutputCount)
    Dim ANNHiddenNeurons.d(ANNHiddenCount)
    Dim ANNOutputNeurons.d(ANNOutputCount)
    Dim ANNAnswerOutputs.d(ANNOutputCount) 
    Dim ANNOutputErrors.d(ANNOutputCount)   

    ;init weights With inital random values
    
    ANNHiddenNeurons(0) = 1
    ANNInputs(0) = 1
    
    RandomSeed(ElapsedMilliseconds()) 
    
    For RndLoop1 = 0 To ANNHiddenCount
        For RndLoop2 = 0 To ANNInputCount
             ANNHiddenWeights(RndLoop2, RndLoop1) = (5 - Random(10)) / 10 ; -0.5 to 0.5 system 
        Next
    Next

    For RndLoop1 = 0 To ANNOutputCount
        For RndLoop2 = 0 To ANNHiddenCount
             ANNOutputWeights(RndLoop2, RndLoop1) = (5 - Random(10)) / 10
             ;Debug ANNOutputWeights(RndLoop2, RndLoop1)
        Next
    Next      

EndProcedure 

;==================================================================================

Procedure ANNTrainSet(Array Inputs.d(1), Array Outputs.d(1), Array AnswerOutputs.d(1))   ;all as double
    
    ;bias
    Inputs(0) = 1
    CopyDoubleArray(ANNAnswerOutputs(),AnswerOutputs())  
    CopyDoubleArray(ANNInputs(),Inputs())  

    CalcLayerOutput(#HiddenLayer)   
    CalcLayerOutput(#OutputLayer)

    BackProp(#OutputLayer)   
    BackProp(#HiddenLayer)     
    
 
    CopyDoubleArray(Outputs(),ANNOutputNeurons())
    
EndProcedure

;==================================================================================

Procedure ANNQuerySet(Array Inputs.d(1), Array Outputs.d(1))    ;all as double        
    
    Inputs(0) = 1
    
    CopyDoubleArray(ANNInputs(),Inputs()) 
            
    CalcLayerOutput(#HiddenLayer)   
    CalcLayerOutput(#OutputLayer)
    
    CopyDoubleArray(Outputs(),ANNOutputNeurons())            
                        
EndProcedure

;==================================================================================

Procedure CalcLayerOutput(ANNLayer.l)

    InputTotal.d  
    InputCount.l  
    OutputCount.l  
    Dim Inputs.d(0)  
    Dim Outputs.d(0)  
    Dim Weights.d(0,0)   
    
    InputNeuronLoop.l  
    HiddenNeuronLoop.l  
    WeightLoop.l 

    ; Prepare DataArrays depending on layer, (makes it easier To add layers in future)    
    If ANNLayer = #HiddenLayer 
        
        InputCount = ANNInputCount 
        OutputCount = ANNHiddenCount
        
        Dim Inputs.d(InputCount)
        Dim Outputs.d(OutputCount)
        Dim Weights.d(InputCount, OutputCount)
        
        ;Copy Arrays
        CopyDoubleArray(Inputs(),ANNInputs())     
        CopyDoubleArray(Outputs(),ANNHiddenNeurons())
        CopyDouble2dArray(Weights(),ANNHiddenWeights())
         
    ElseIf ANNLayer = #OutputLayer 
        
        InputCount = ANNHiddenCount 
        OutputCount = ANNOutputCount
        
        Dim Inputs.d(InputCount)
        Dim Outputs.d(OutputCount)
        Dim Weights.d(InputCount, OutputCount)
        
        ;Copy Arrays
        CopyDoubleArray(Inputs(),ANNHiddenNeurons())
        CopyDoubleArray(Outputs(),ANNOutputNeurons())
        CopyDouble2dArray(Weights(),ANNOutputWeights())
            
    Else
        Debug "Layer Error"
    EndIf
    
    ;Calculate Outputs from inputs And weights For each neuron in layer.
    For HiddenNeuronLoop = 1 To OutputCount 
    
        For WeightLoop = 0 To InputCount
            InputTotal = InputTotal + ( Inputs(WeightLoop) * Weights(WeightLoop, HiddenNeuronLoop) )
        Next 

        Outputs(HiddenNeuronLoop) = ActivateNeuron(InputTotal)  
     
        InputTotal = 0 
    Next
    
    ;copy Data back again
    If ANNLayer = #HiddenLayer          
        CopyDoubleArray(ANNHiddenNeurons(),Outputs())        
    ElseIf ANNLayer = #OutputLayer 
        CopyDoubleArray(ANNOutputNeurons(),Outputs())           
    Else
        Debug "Layer Error"
    EndIf
        
EndProcedure

;'==================================================================================

Procedure.d ActivateNeuron(InputTotal.d)

      ProcedureReturn 1 / (1 + Pow(2.718282,(-1 * InputTotal)))

EndProcedure

;'==================================================================================

Procedure BackProp(ANNLayer.l)
    
    NeuronLoop.l
    InputLoop.l
    HiddenLoop.l
    InputWeightLoop.l
    WeightLoop.l
    DeltaError.d
    ;'Dim MeanErr As double
    
    Ok.d    ;'Output for K
    Ik .d   ;'Output for Input neuron
    Tk.d   ;'Target for K
    Wjk.d   ;'Weight for OutputK from hidden J 
    Wij.d   ;'Weight for HiddenJ from Input i
    J.d     ;'Output of Hidden J 
    Ej.d     ;'Error for hidden layer 
    SigmaTotal.d ;'running total of additions in hidden layer
    
    If ANNLayer = #OutputLayer 
        
        For NeuronLoop = 0 To ANNOutputCount            ;'loop output neurons 
            
            Ok = ANNOutputNeurons(NeuronLoop) 
            Tk = ANNAnswerOutputs(NeuronLoop)

            DeltaError = (Tk - Ok) * Ok * (1 - Ok)     

            ANNOutputErrors(NeuronLoop) = DeltaError

            For WeightLoop = 0 To ANNHiddenCount        ;'adjust weight from each hidden layer neuron
                
                Wjk = ANNOutputWeights(WeightLoop,  NeuronLoop)
                J = ANNHiddenNeurons(WeightLoop)
                Wjk = Wjk + (ANNLearningRate * DeltaError * J)  
                                                                                          
                ANNOutputWeights(WeightLoop,NeuronLoop ) = Wjk

            Next 
        Next   

    ElseIf ANNLayer = #HiddenLayer  
    
        Ej = 0

        For HiddenLoop = 0 To ANNHiddenCount
            
            Ok = ANNHiddenNeurons(HiddenLoop) 

            SigmaTotal = 0
            
            For NeuronLoop = 1 To ANNOutputCount            
                SigmaTotal = SigmaTotal + (ANNOutputErrors(NeuronLoop) * ANNOutputWeights(ANNHiddenCount, ANNOutputCount))       
            Next

            Ej = Ok * (1-Ok) * SigmaTotal ;* 10

            For inputloop = 0 To ANNInputCount
                    
                Wij = ANNHiddenWeights(InputLoop,HiddenLoop)
                Wij = Wij + (ANNLearningRate * Ej * ANNInputs(InputLoop))
                ANNHiddenWeights(InputLoop,HiddenLoop) = Wij 
                
            Next
        Next                               
    Else
        Debug "Layer Error"
    EndIf 
        
        
EndProcedure
        
;'==================================================================================

Procedure.d MeanSqrErr() 

    TotalSqrErr.d
    ErrLoop.l
    
;     If ANNOutputCount = 0;< 0.0000001  ;'bail before div by zero
;         ProcedureReturn -1
;     EndIf
    
    For errloop = 0 To ANNOutputCount
        TotalSqrErr =  TotalSqrErr + Pow((ANNOutputErrors(ErrLoop)),2)       
    Next

;     If TotalSqrErr =0; 1  ;'bail before div by zero
;         ProcedureReturn -1
;     EndIf
        
    ProcedureReturn TotalSqrErr / ANNOutputCount
     
EndProcedure

;'==================================================================================

Procedure CopyDoubleArray(Array DestArray.d(1), Array SrcArray.d(1)) ;all as double

    ArrayLoop.l
    
    VarSize.l = 8
    Elements.l = ArraySize(DestArray())
    CopyMemory(@SrcArray(),@DestArray(),VarSize * (Elements+1))

EndProcedure

;'==================================================================================

Procedure CopyDouble2DArray(Array DestArray.d(2), Array SrcArray.d(2)) ;all as double

    ArrayLoop.l
    
    VarSize.l = 8
    Elements1.l = ArraySize(DestArray(),1)
    Elements2.l = ArraySize(DestArray(),2)
    CopyMemory(@SrcArray(),@DestArray(),VarSize * (Elements1+1) * (Elements2 + 1))

EndProcedure

;'==================================================================================

Procedure SaveWeights(Filename.s)

    hfile.i
    hfile = CreateFile(#PB_Any, Filename) 
    For ic.l = 0 To ANNInputCount 
        For hc.l = 0 To ANNHiddenCount 
            WriteDouble(hfile, ANNHiddenWeights(ic, hc)) 
        Next 
    Next 
    For hc.l = 0 To ANNHiddenCount 
        For oc.l = 0 To ANNOutputCount 
            WriteDouble(hfile, ANNOutputWeights(hc, oc)) 
        Next 
    Next 
    CloseFile(hfile) 
EndProcedure

;'==================================================================================
                
Procedure LoadWeights(Filename.s)

    hfile.i
    hfile = ReadFile(#PB_Any, Filename) 
    For ic.l = 0 To ANNInputCount 
        For hc.l = 0 To ANNHiddenCount 
            ANNHiddenWeights(ic, hc) = ReadDouble(hfile) 
        Next 
    Next 
    For hc.l = 0 To ANNHiddenCount 
        For oc.l = 0 To ANNOutputCount 
            ANNOutputWeights(hc, oc) = ReadDouble(hfile) 
        Next 
    Next 
    CloseFile(hfile) 
EndProcedure

;'==================================================================================

Next the app.

Code: Select all

XIncludeFile "ann.pbi" 

#ScrWidth  = 1200
#ScrHeight = 800
#Exp = 2.7182818284590451 

#CmdSlow = 1
#CmdMed = 2
#CmdFast = 3
#CmdRestart = 4

Declare Main()
Declare MakeArray(Array TheArray.d(1), CSV.s)

main()

Procedure Main()
    If OpenWindow(0, 0, 0, #ScrWidth, #ScrHeight, "Nerual Network", #PB_Window_SystemMenu | #PB_Window_ScreenCentered | #PB_Window_MinimizeGadget) ;And CreateGadgetList(WindowID(0))
     
        If CreateImage(0, #ScrWidth, #ScrHeight-22, 32)
            ImageGadget(0, 0, 0, #ScrWidth, #ScrHeight-22, ImageID(0))                      ; imagegadget standard
        EndIf

        StartDrawing(ImageOutput(0))
            Box(1,1,#ScrWidth, #ScrHeight,RGB(30,30,30))
        StopDrawing()
                
        ButtonGadget(#CmdSlow, 10,780, 50, 20, "Slow")
        ButtonGadget(#CmdMed,  60,780, 50, 20, "Normal")
        ButtonGadget(#CmdFast,  110,780, 50, 20, "Fast")
        ButtonGadget(#CmdRestart,  160,780, 50, 20, "Restart")
        frame.q = 0

        InputCount.l 
        HiddenCount.l
        OutputCount.l
        LearningRate.d
        TrainingLoop.l  
        Dim Outputs.d(10)

        Dim One.d(20)
        Dim Two.d(20)
        Dim Three.d(20)
        Dim Four.d(20)
        Dim Five.d(20)
        Dim Six.d(20)
        Dim Seven.d(20)
        Dim Eight.d(20)
        Dim Nine.d(20)
        Dim Zero.d(20)
        
        Dim Display.d(20) 
        Dim CurrentDispPtr.l(10)
        Dim CurrentAnsPtr.l(10)
        Dim CurrentAnswer.d(10)
        
        Dim Answer1.d(10): Answer1(1) = 1
        Dim Answer2.d(10): Answer2(2) = 1
        Dim Answer3.d(10): Answer3(3) = 1
        Dim Answer4.d(10): Answer4(4) = 1
        Dim Answer5.d(10): Answer5(5) = 1
        Dim Answer6.d(10): Answer6(6) = 1
        Dim Answer7.d(10): Answer7(7) = 1
        Dim Answer8.d(10): Answer8(8) = 1
        Dim Answer9.d(10): Answer9(9) = 1
        Dim Answer0.d(10): Answer0(10) = 1
    
        MakeArray(Zero(),   "0.01,0.99,0.99,0.01,0.99,0.01,0.01,0.99,0.99,0.01,0.01,0.99,0.99,0.01,0.01,0.99,0.01,0.99,0.99,0.01")
        MakeArray(One(),   "0.01,0.99,0.01,0.01,0.99,0.99,0.01,0.01,0.01,0.99,0.01,0.01,0.01,0.99,0.01,0.01,0.99,0.99,0.99,0.01")
        MakeArray(two(), "0.01,0.99,0.99,0.01,0.99,0.01,0.01,0.99,0.01,0.01,0.99,0.01,0.01,0.99,0.01,0.01,0.99,0.99,0.99,0.99")
        MakeArray(three(),  "0.99,0.99,0.99,0.01,0.01,0.01,0.01,0.99,0.01,0.99,0.99,0.01,0.01,0.01,0.01,0.99,0.99,0.99,0.99,0.01")
        MakeArray(Four(),  "0.99,0.01,0.01,0.01,0.99,0.01,0.99,0.01,0.99,0.99,0.99,0.99,0.01,0.01,0.99,0.01,0.01,0.01,0.99,0.01")
        MakeArray(five(),   "0.99,0.99,0.99,0.99,0.99,0.01,0.01,0.01,0.99,0.99,0.99,0.01,0.01,0.01,0.01,0.99,0.99,0.99,0.99,0.01")
        MakeArray(Six(), "0.01,0.99,0.99,0.99,0.99,0.01,0.01,0.01,0.99,0.99,0.99,0.01,0.99,0.01,0.01,0.99,0.01,0.99,0.99,0.01")
        MakeArray(seven(), "0.99,0.99,0.99,0.99,0.01,0.01,0.01,0.99,0.01,0.01,0.99,0.01,0.01,0.99,0.01,0.01,0.01,0.99,0.01,0.01")
        MakeArray(eight(),  "0.01,0.99,0.99,0.01,0.99,0.01,0.01,0.99,0.01,0.99,0.99,0.01,0.99,0.01,0.01,0.99,0.01,0.99,0.99,0.01")
        MakeArray(nine(),  "0.01,0.99,0.99,0.01,0.99,0.01,0.01,0.99,0.01,0.99,0.99,0.99,0.01,0.01,0.01,0.99,0.99,0.99,0.99,0.01")
        
        CurrentDispPtr(1) = @one()    
        CurrentDispPtr(2) = @two() 
        CurrentDispPtr(3) = @three() 
        CurrentDispPtr(4) = @four() 
        CurrentDispPtr(5) = @five() 
        CurrentDispPtr(6) = @six() 
        CurrentDispPtr(7) = @seven() 
        CurrentDispPtr(8) = @eight() 
        CurrentDispPtr(9) = @nine() 
        CurrentDispPtr(10) = @zero() 
        
        CurrentAnsPtr(1) = @Answer1()
        CurrentAnsPtr(2) = @Answer2()
        CurrentAnsPtr(3) = @Answer3()
        CurrentAnsPtr(4) = @Answer4()
        CurrentAnsPtr(5) = @Answer5()
        CurrentAnsPtr(6) = @Answer6()
        CurrentAnsPtr(7) = @Answer7()
        CurrentAnsPtr(8) = @Answer8()
        CurrentAnsPtr(9) = @Answer9()
        CurrentAnsPtr(10) = @Answer0()
        
        For i = 1 To 20
            Display(i) = 1
        Next
        
        For i = 1 To 10
            CurrentAnswer(i) = 1
        Next
        
        Digitloop = 1
        RefreshDelay = 50
        
        ;Create ANN
        InputCount = 20
        HiddenCount = 25
        OutputCount = 10
        LearningRate = 0.55
        
        ANNCreate(InputCount, HiddenCount, OutputCount, LearningRate) 
        FontID1 = FontID(LoadFont(#PB_Any, "Arial"  ,  12, #PB_Font_Bold))
            
        Repeat        
            frame = frame + 1

            ; Init Display Drawing
            StartDrawing(ImageOutput(0))
            Box(1,1,#ScrWidth, #ScrHeight,RGB(30,30,30))
            DigXoffset.l = 20
            DigYOffset.l = 300
            DigPixelSize = 9
            DispY = 100
            
            Digitloop = Digitloop + 1
            If Digitloop > 10
                Digitloop = 1
            EndIf
            
            CopyMemory(CurrentDispPtr(Digitloop),@Display(),8 * 21)
            CopyMemory(CurrentAnsPtr(Digitloop),@CurrentAnswer(),8 * 11)
            
            ANNTrainSet(Display(), Outputs(), CurrentAnswer())
            
            ;Display Layer 1

            
            For i = 1 To 20 Step 4                  
                LineXY(DigXoffset + 14, DigYOffset + (i*3), 250, DispY, RGB(0,Display(i) * 255,0))
                LineXY(DigXoffset + 26, DigYOffset + (i*3), 250, DispY + 30, RGB(0,Display(i+1) * 255,0))
                LineXY(DigXoffset + 38, DigYOffset + (i*3), 250, DispY + 60, RGB(0,Display(i+2) * 255,0))
                LineXY(DigXoffset + 50, DigYOffset + (i*3), 250, DispY + 90, RGB(0,Display(i+3) * 255,0)) 
                
                Circle(250, 70 , DigPixelSize-1, RGB(0,ANNInputs(0) * 127,255))
                Circle(250, DispY, DigPixelSize-1, RGB(0,ANNInputs(i) * 255,0))
                Circle(250, DispY + 30, DigPixelSize-1, RGB(0,ANNInputs(i+1) * 255,0))
                Circle(250, DispY + 60, DigPixelSize-1, RGB(0,ANNInputs(i+2) * 255,0))
                Circle(250, DispY + 90, DigPixelSize-1, RGB(0,ANNInputs(i+3) * 255,0))
                
                DispY = DispY + 120
                
                Box(DigXoffset + 12, DigYOffset + (i*3), DigPixelSize, DigPixelSize,RGB(0,Display(i) * 255,0))
                Box(DigXoffset + 24, DigYOffset + (i*3), DigPixelSize, DigPixelSize,RGB(0,Display(i+1) * 255,0))
                Box(DigXoffset + 36, DigYOffset + (i*3), DigPixelSize, DigPixelSize,RGB(0,Display(i+2) * 255,0))
                Box(DigXoffset + 48, DigYOffset + (i*3), DigPixelSize, DigPixelSize,RGB(0,Display(i+3) * 255,0))             
            Next

            ;Display HiddenLayer

            DispY = 50
            NegLevel.d = 0
            For i = 0 To HiddenCount
            
                If i = 0
                    Circle(600, DispY + (i*25), DigPixelSize-1, RGB(0,ANNHiddenNeurons(i) * 127,255)) 
                Else
                    Circle(600, DispY + (i*25), DigPixelSize-1, RGB(0,ANNHiddenNeurons(i) * 255,0)) 
                EndIf
  
                ;weights from input layer
                For j = 0 To 20
                    If ANNHiddenWeights(j,i) < 0
                        NegLevel.d = 1 /(1 + Pow(#Exp,(-1 * ANNHiddenWeights(j,i))))
                         
                        PosLevel.d = 0
                    Else
                        NegLevel.d = 0
                        PosLevel.d = 1 /(1 + Pow(#Exp,(-1 * ANNHiddenWeights(j,i))))
                        ;Debug StrD(PosLevel) + "   " + StrD(ANNHiddenWeights(j,i))
                    EndIf 
                    
                    If i > 0       
                        LineXY(250, DispY + 20 + (j*30), 600, DispY + (i*25), RGB(NegLevel * 255,PosLevel * 255,0))
                    EndIf
                Next
            
            Next

            ;Display Output Layer

            DispY = 100
            
            DrawingFont(FontID1)
            For i = 1 To 10           
                Circle(905, DispY + 12 + (i*40), DigPixelSize, RGB(0,Outputs(i) * 255,0))    
                DrawText(930, DispY + (i*40), Right(Str(i),1) + "  (" + StrD(Outputs(i),2) + ")", RGB(0,Outputs(i) * 255,0) , RGB(30,30,30))
            
                ;weights from hidden layer
                For j = 0 To HiddenCount
                    If ANNOutputWeights(j,i) < 0
                        NegLevel = (1 / (1 + Pow(#Exp,(-1 * ANNOutputWeights(j,i))))) ;* -1
                        PosLevel = 0
                    Else
                        NegLevel = 0
                        PosLevel = (1 / (1 + Pow(#Exp,(-1 * ANNOutputWeights(j,i))))) 
                    EndIf
                    
                    LineXY(600, DispY - 50 + (j*25), 900, DispY + 10 + (i*40), RGB(NegLevel * 255,PosLevel * 255,0))
                Next               
            Next            
            
            
            ;Update text labels
            DrawText(900, 650, "Epoch: " + Str(frame/10), RGB(0,255,0) , RGB(30,30,30))
            
            TmpAvgErr.d = TmpAvgErr  + MeanSqrErr()
            If Digitloop = 10
                AvgErr.d = TmpAvgErr / 10
                TmpAvgErr = 0
            EndIf
            DrawText(900, 670, "Avg Err: " + StrD(AvgErr,6), RGB(0,255,0) , RGB(30,30,30))
            
            Circle(12, 697, DigPixelSize-1, RGB(0,127,255))
            DrawText(30, 690, "Bias ", RGB(0,255,0) , RGB(30,30,30))
            Circle(12, 717, DigPixelSize-1, RGB(0,255,0))
            DrawText(30, 710, "Neuron (Green level shows activiation intensity)", RGB(0,255,0) , RGB(30,30,30))
            LineXY(5, 757, 20, 757, RGB(255,0,0))
            LineXY(5, 737, 20, 737, RGB(0,255,0))
            DrawText(30, 730, "Weight (Green level shows Positive Intensity)", RGB(0,255,0) , RGB(30,30,30))
            DrawText(30, 750, "Weight (Red level shows negative Intensity)", RGB(255,0,0) , RGB(30,30,30))
            
            Delay(RefreshDelay)
            
            StopDrawing()     
            SetGadgetState(0, ImageID(0))
    
            Repeat 
                event.l = WindowEvent()   
                Select event                  
                    Case #PB_Event_Gadget
                    
                        Select EventMenu()  ; To see which menu has been selected   
                            Case #CmdSlow
                                RefreshDelay = 400            
                            Case #CmdMed
                                RefreshDelay = 50
                            Case #CmdFast     
                                RefreshDelay = 0  
                            Case #CmdRestart
                                ANNCreate(InputCount, HiddenCount, OutputCount, LearningRate)    
                                frame = 1  
                                Digitloop = 1         
                        EndSelect
                                         
                EndSelect               
            Until event = 0 Or event = #PB_Event_CloseWindow 
    
        Until event = #PB_Event_CloseWindow  
                       
    EndIf
EndProcedure


Procedure MakeArray(Array TheArray.d(1), CSV.s)
    
    CSVLoop.l
    FieldCount.l
    
    FieldCount = CountString(CSV, ",") + 1 
    ReDim  TheArray(FieldCount)
    
    For CSVLoop = 1 To FieldCount    
        TheArray(CSVLoop) = ValD(StringField(CSV,CSVLoop ,"," ))      
    Next
    
EndProcedure 

[pdwyer]

This version of the app, same gui, same ann.pbi is modified to show activity though the network rather than the state of the network. ie, the electrical charge rather than the width of the synapse. This one is much more energetic to watch

Code: Select all

XIncludeFile "ann.pbi" 

#ScrWidth  = 1200
#ScrHeight = 800
#Exp = 2.7182818284590451 

#CmdSlow = 1
#CmdMed = 2
#CmdFast = 3
#CmdRestart = 4

Declare Main()
Declare MakeArray(Array TheArray.d(1), CSV.s)

main()

Procedure Main()
    If OpenWindow(0, 0, 0, #ScrWidth, #ScrHeight, "Nerual Network", #PB_Window_SystemMenu | #PB_Window_ScreenCentered | #PB_Window_MinimizeGadget) ;And CreateGadgetList(WindowID(0))
     
        If CreateImage(0, #ScrWidth, #ScrHeight-22, 32)
            ImageGadget(0, 0, 0, #ScrWidth, #ScrHeight-22, ImageID(0))                      ; imagegadget standard
        EndIf

        StartDrawing(ImageOutput(0))
            Box(1,1,#ScrWidth, #ScrHeight,RGB(30,30,30))
        StopDrawing()
                
        ButtonGadget(#CmdSlow, 10,780, 50, 20, "Slow")
        ButtonGadget(#CmdMed,  60,780, 50, 20, "Normal")
        ButtonGadget(#CmdFast,  110,780, 50, 20, "Fast")
        ButtonGadget(#CmdRestart,  160,780, 50, 20, "Restart")
        frame.q = 0

        InputCount.l 
        HiddenCount.l
        OutputCount.l
        LearningRate.d
        TrainingLoop.l  
        Dim Outputs.d(10)

        Dim One.d(20)
        Dim Two.d(20)
        Dim Three.d(20)
        Dim Four.d(20)
        Dim Five.d(20)
        Dim Six.d(20)
        Dim Seven.d(20)
        Dim Eight.d(20)
        Dim Nine.d(20)
        Dim Zero.d(20)
        
        Dim Display.d(20) 
        Dim CurrentDispPtr.l(10)
        Dim CurrentAnsPtr.l(10)
        Dim CurrentAnswer.d(10)
        
        Dim Answer1.d(10): Answer1(1) = 1
        Dim Answer2.d(10): Answer2(2) = 1
        Dim Answer3.d(10): Answer3(3) = 1
        Dim Answer4.d(10): Answer4(4) = 1
        Dim Answer5.d(10): Answer5(5) = 1
        Dim Answer6.d(10): Answer6(6) = 1
        Dim Answer7.d(10): Answer7(7) = 1
        Dim Answer8.d(10): Answer8(8) = 1
        Dim Answer9.d(10): Answer9(9) = 1
        Dim Answer0.d(10): Answer0(10) = 1
    
        MakeArray(Zero(),   "0.01,0.99,0.99,0.01,0.99,0.01,0.01,0.99,0.99,0.01,0.01,0.99,0.99,0.01,0.01,0.99,0.01,0.99,0.99,0.01")
        MakeArray(One(),   "0.01,0.99,0.01,0.01,0.99,0.99,0.01,0.01,0.01,0.99,0.01,0.01,0.01,0.99,0.01,0.01,0.99,0.99,0.99,0.01")
        MakeArray(two(), "0.01,0.99,0.99,0.01,0.99,0.01,0.01,0.99,0.01,0.01,0.99,0.01,0.01,0.99,0.01,0.01,0.99,0.99,0.99,0.99")
        MakeArray(three(),  "0.99,0.99,0.99,0.01,0.01,0.01,0.01,0.99,0.01,0.99,0.99,0.01,0.01,0.01,0.01,0.99,0.99,0.99,0.99,0.01")
        MakeArray(Four(),  "0.99,0.01,0.01,0.01,0.99,0.01,0.99,0.01,0.99,0.99,0.99,0.99,0.01,0.01,0.99,0.01,0.01,0.01,0.99,0.01")
        MakeArray(five(),   "0.99,0.99,0.99,0.99,0.99,0.01,0.01,0.01,0.99,0.99,0.99,0.01,0.01,0.01,0.01,0.99,0.99,0.99,0.99,0.01")
        MakeArray(Six(), "0.01,0.99,0.99,0.99,0.99,0.01,0.01,0.01,0.99,0.99,0.99,0.01,0.99,0.01,0.01,0.99,0.01,0.99,0.99,0.01")
        MakeArray(seven(), "0.99,0.99,0.99,0.99,0.01,0.01,0.01,0.99,0.01,0.01,0.99,0.01,0.01,0.99,0.01,0.01,0.01,0.99,0.01,0.01")
        MakeArray(eight(),  "0.01,0.99,0.99,0.01,0.99,0.01,0.01,0.99,0.01,0.99,0.99,0.01,0.99,0.01,0.01,0.99,0.01,0.99,0.99,0.01")
        MakeArray(nine(),  "0.01,0.99,0.99,0.01,0.99,0.01,0.01,0.99,0.01,0.99,0.99,0.99,0.01,0.01,0.01,0.99,0.99,0.99,0.99,0.01")
        
        CurrentDispPtr(1) = @one()    
        CurrentDispPtr(2) = @two() 
        CurrentDispPtr(3) = @three() 
        CurrentDispPtr(4) = @four() 
        CurrentDispPtr(5) = @five() 
        CurrentDispPtr(6) = @six() 
        CurrentDispPtr(7) = @seven() 
        CurrentDispPtr(8) = @eight() 
        CurrentDispPtr(9) = @nine() 
        CurrentDispPtr(10) = @zero() 
        
        CurrentAnsPtr(1) = @Answer1()
        CurrentAnsPtr(2) = @Answer2()
        CurrentAnsPtr(3) = @Answer3()
        CurrentAnsPtr(4) = @Answer4()
        CurrentAnsPtr(5) = @Answer5()
        CurrentAnsPtr(6) = @Answer6()
        CurrentAnsPtr(7) = @Answer7()
        CurrentAnsPtr(8) = @Answer8()
        CurrentAnsPtr(9) = @Answer9()
        CurrentAnsPtr(10) = @Answer0()
        
        For i = 1 To 20
            Display(i) = 1
        Next
        
        For i = 1 To 10
            CurrentAnswer(i) = 1
        Next
        
        Digitloop = 1
        RefreshDelay = 50
        
        ;Create ANN
        InputCount = 20
        HiddenCount = 25
        OutputCount = 10
        LearningRate = 0.55
        
        ANNCreate(InputCount, HiddenCount, OutputCount, LearningRate) 
        FontID1 = FontID(LoadFont(#PB_Any, "Arial"  ,  12, #PB_Font_Bold))
            
        Repeat        
            frame = frame + 1

            ; Init Display Drawing
            StartDrawing(ImageOutput(0))
            Box(1,1,#ScrWidth, #ScrHeight,RGB(30,30,30))
            DigXoffset.l = 20
            DigYOffset.l = 300
            DigPixelSize = 9
            DispY = 100
            
            Digitloop = Digitloop + 1
            If Digitloop > 10
                Digitloop = 1
            EndIf
            
            CopyMemory(CurrentDispPtr(Digitloop),@Display(),8 * 21)
            CopyMemory(CurrentAnsPtr(Digitloop),@CurrentAnswer(),8 * 11)
            
            ANNTrainSet(Display(), Outputs(), CurrentAnswer())
            
            ;Display Layer 1

            
            For i = 1 To 20 Step 4                  
                LineXY(DigXoffset + 14, DigYOffset + (i*3), 250, DispY, RGB(0,Display(i) * 255,0))
                LineXY(DigXoffset + 26, DigYOffset + (i*3), 250, DispY + 30, RGB(0,Display(i+1) * 255,0))
                LineXY(DigXoffset + 38, DigYOffset + (i*3), 250, DispY + 60, RGB(0,Display(i+2) * 255,0))
                LineXY(DigXoffset + 50, DigYOffset + (i*3), 250, DispY + 90, RGB(0,Display(i+3) * 255,0)) 
                
                Circle(250, 70 , DigPixelSize-1, RGB(0,ANNInputs(0) * 127,255))
                Circle(250, DispY, DigPixelSize-1, RGB(0,ANNInputs(i) * 255,0))
                Circle(250, DispY + 30, DigPixelSize-1, RGB(0,ANNInputs(i+1) * 255,0))
                Circle(250, DispY + 60, DigPixelSize-1, RGB(0,ANNInputs(i+2) * 255,0))
                Circle(250, DispY + 90, DigPixelSize-1, RGB(0,ANNInputs(i+3) * 255,0))
                
                DispY = DispY + 120
                
                Box(DigXoffset + 12, DigYOffset + (i*3), DigPixelSize, DigPixelSize,RGB(0,Display(i) * 255,0))
                Box(DigXoffset + 24, DigYOffset + (i*3), DigPixelSize, DigPixelSize,RGB(0,Display(i+1) * 255,0))
                Box(DigXoffset + 36, DigYOffset + (i*3), DigPixelSize, DigPixelSize,RGB(0,Display(i+2) * 255,0))
                Box(DigXoffset + 48, DigYOffset + (i*3), DigPixelSize, DigPixelSize,RGB(0,Display(i+3) * 255,0))             
            Next

            ;Display HiddenLayer

            DispY = 50
            NegLevel.d = 0
            For i = 0 To HiddenCount
            
                If i = 0
                    Circle(600, DispY + (i*25), DigPixelSize-1, RGB(0,ANNHiddenNeurons(i) * 127,255)) 
                Else
                    Circle(600, DispY + (i*25), DigPixelSize-1, RGB(0,ANNHiddenNeurons(i) * 255,0)) 
                EndIf
  
                ;weights from input layer
                For j = 0 To 20
                    If ANNHiddenWeights(j,i) * ANNHiddenNeurons(i) < 0
                        NegLevel.d = ANNHiddenWeights(j,i) * ANNHiddenNeurons(i) * -1
                         
                        PosLevel.d = 0
                    Else
                        NegLevel.d = 0
                        PosLevel.d = ANNHiddenWeights(j,i) * ANNHiddenNeurons(i)
                        ;Debug StrD(PosLevel) + "   " + StrD(ANNHiddenWeights(j,i))
                    EndIf 
                    
                    If i > 0       
                        LineXY(250, DispY + 20 + (j*30), 600, DispY + (i*25), RGB(NegLevel * 255,PosLevel * 255,0))
                    EndIf
                Next
            
            Next

            ;Display Output Layer

            DispY = 100
            
            DrawingFont(FontID1)
            For i = 1 To 10           
                Circle(905, DispY + 12 + (i*40), DigPixelSize, RGB(0,Outputs(i) * 255,0))    
                DrawText(930, DispY + (i*40), Right(Str(i),1) + "  (" + StrD(Outputs(i),2) + ")", RGB(0,Outputs(i) * 255,0) , RGB(30,30,30))
            
                ;weights from hidden layer
                For j = 0 To HiddenCount
                    If ANNOutputWeights(j,i) * Outputs(i) < 0
                        NegLevel = Outputs(i) * ANNOutputWeights(j,i) * -1
                        PosLevel = 0
                    Else
                        NegLevel = 0
                        PosLevel = Outputs(i) * ANNOutputWeights(j,i)
                    EndIf
                    
                    LineXY(600, DispY - 50 + (j*25), 900, DispY + 10 + (i*40), RGB(NegLevel * 255,PosLevel * 255,0))
                Next               
            Next            
            
            
            ;Update text labels
            DrawText(900, 650, "Epoch: " + Str(frame/10), RGB(0,255,0) , RGB(30,30,30))
            
            TmpAvgErr.d = TmpAvgErr  + MeanSqrErr()
            If Digitloop = 10
                AvgErr.d = TmpAvgErr / 10
                TmpAvgErr = 0
            EndIf
            DrawText(900, 670, "Avg Err: " + StrD(AvgErr,6), RGB(0,255,0) , RGB(30,30,30))
            
            Circle(12, 697, DigPixelSize-1, RGB(0,127,255))
            DrawText(30, 690, "Bias ", RGB(0,255,0) , RGB(30,30,30))
            Circle(12, 717, DigPixelSize-1, RGB(0,255,0))
            DrawText(30, 710, "Neuron (Green level shows activiation intensity)", RGB(0,255,0) , RGB(30,30,30))
            LineXY(5, 757, 20, 757, RGB(255,0,0))
            LineXY(5, 737, 20, 737, RGB(0,255,0))
            DrawText(30, 730, "Weight (Green level shows Positive Intensity)", RGB(0,255,0) , RGB(30,30,30))
            DrawText(30, 750, "Weight (Red level shows negative Intensity)", RGB(255,0,0) , RGB(30,30,30))
            
            Delay(RefreshDelay)
            
            StopDrawing()     
            SetGadgetState(0, ImageID(0))
    
            Repeat 
                event.l = WindowEvent()   
                Select event                  
                    Case #PB_Event_Gadget
                    
                        Select EventMenu()  ; To see which menu has been selected   
                            Case #CmdSlow
                                RefreshDelay = 400            
                            Case #CmdMed
                                RefreshDelay = 50
                            Case #CmdFast     
                                RefreshDelay = 0  
                            Case #CmdRestart
                                ANNCreate(InputCount, HiddenCount, OutputCount, LearningRate)    
                                frame = 1  
                                Digitloop = 1         
                        EndSelect
                                         
                EndSelect               
            Until event = 0 Or event = #PB_Event_CloseWindow 
    
        Until event = #PB_Event_CloseWindow  
                       
    EndIf
EndProcedure


Procedure MakeArray(Array TheArray.d(1), CSV.s)
    
    CSVLoop.l
    FieldCount.l
    
    FieldCount = CountString(CSV, ",") + 1 
    ReDim  TheArray(FieldCount)
    
    For CSVLoop = 1 To FieldCount    
        TheArray(CSVLoop) = ValD(StringField(CSV,CSVLoop ,"," ))      
    Next
    
EndProcedure 

[djes]

Interesting, I have to take a look at that when I'll have time! Seems to take time to find 9 and 6. Could it be faster with more neurons?

[pdwyer]

Thanks for taking a look

Since the starting weight values are random it can get stuck in different places, usually just letting it run will get past them but it might take a while, I've seen it take 50,000 epochs to converge sometimes.

From what I've learned about the hidden neurons, there's a lot of people who talk about ideas for the number to use, but no hard and fast rules. the more you have the more CPU time it takes to calc the weights. The code above uses 25 so 20x25 + 10x25 = 750 weight calcs. Going under about 12 hidden neurons and the network often fails, kicking the learning rate up a bit can help but even then. taking the hidden neurons up to about 35 and lowering the learning rate to about 0.3 is often more reliable and doesn't get stuck as much but it takes more epochs and is a gradual learn. (more than about 30 neurons and they go off the screen but they still work)

I need to work on the gui more so you can put other things in, and maybe switch the display type between the two above and finally have another button for even faster with the gui disabled to push through a quick 1000 epochs or so.

I want to train an ANN to learn opening moves for gomoku for next years programing tournament so I want the gui to be able to help me debug that.

Any ideas are very welcome as I'm a bit new to this

[idle]

wow nice

and something else to distract me from doing work!

[dige]

looks amazing! :D

[ebs]

Paul,

Thank you for the great NN code!
I have been playing with this stuff for years, and attempted (unsuccessfully) to develop something similar.
Your code, especially the graphical representation of the NN, is amazingly concise.

I hope you don't mind, but I couldn't resist adding a pause and save/load functions to your application.
My changes are indicated by comments ending with "(ebs)".
I don't think I messed anything up!

Regards,
Eric

Code: Select all

XIncludeFile "ann.pbi"

#ScrWidth  = 1200
#ScrHeight = 800
#Exp = 2.7182818284590451

#CmdSlow = 1
#CmdMed = 2
#CmdFast = 3
#CmdRestart = 4
; added buttons for pause, save and load (ebs)
#CmdPause = 5
#CmdSave = 6
#CmdLoad = 7

Declare Main()
Declare MakeArray(Array TheArray.d(1), CSV.s)

Main()

Procedure Main()
  If OpenWindow(0, 0, 0, #ScrWidth, #ScrHeight, "Neural Network", #PB_Window_SystemMenu | #PB_Window_ScreenCentered | #PB_Window_MinimizeGadget) ;And CreateGadgetList(WindowID(0))
    
    If CreateImage(0, #ScrWidth, #ScrHeight-22, 32)
      ImageGadget(0, 0, 0, #ScrWidth, #ScrHeight-22, ImageID(0))                      ; imagegadget standard
    EndIf
    
    StartDrawing(ImageOutput(0))
      Box(1,1,#ScrWidth, #ScrHeight,RGB(30,30,30))
    StopDrawing()
    
    ButtonGadget(#CmdSlow, 10,780, 50, 20, "Slow")
    ButtonGadget(#CmdMed,  60,780, 50, 20, "Normal")
    ButtonGadget(#CmdFast,  110,780, 50, 20, "Fast")
    ButtonGadget(#CmdRestart,  160,780, 50, 20, "Restart")
    ; added buttons for pause, save and load (ebs)
    ButtonGadget(#CmdPause,  210,780, 50, 20, "Pause")
    ButtonGadget(#CmdSave,  260,780, 50, 20, "Save")
    ButtonGadget(#CmdLoad,  310,780, 50, 20, "Load")
    frame.q = 0
    
    InputCount.l
    HiddenCount.l
    OutputCount.l
    LearningRate.d
    TrainingLoop.l 
    Dim Outputs.d(10)
    
    Dim One.d(20)
    Dim Two.d(20)
    Dim Three.d(20)
    Dim Four.d(20)
    Dim Five.d(20)
    Dim Six.d(20)
    Dim Seven.d(20)
    Dim Eight.d(20)
    Dim Nine.d(20)
    Dim Zero.d(20)
    
    Dim Display.d(20)
    Dim CurrentDispPtr.l(10)
    Dim CurrentAnsPtr.l(10)
    Dim CurrentAnswer.d(10)
    
    Dim Answer1.d(10): Answer1(1) = 1
    Dim Answer2.d(10): Answer2(2) = 1
    Dim Answer3.d(10): Answer3(3) = 1
    Dim Answer4.d(10): Answer4(4) = 1
    Dim Answer5.d(10): Answer5(5) = 1
    Dim Answer6.d(10): Answer6(6) = 1
    Dim Answer7.d(10): Answer7(7) = 1
    Dim Answer8.d(10): Answer8(8) = 1
    Dim Answer9.d(10): Answer9(9) = 1
    Dim Answer0.d(10): Answer0(10) = 1
    
    ; paused flag (ebs)
    Paused.l = #False
    
    MakeArray(Zero(),   "0.01,0.99,0.99,0.01,0.99,0.01,0.01,0.99,0.99,0.01,0.01,0.99,0.99,0.01,0.01,0.99,0.01,0.99,0.99,0.01")
    MakeArray(One(),   "0.01,0.99,0.01,0.01,0.99,0.99,0.01,0.01,0.01,0.99,0.01,0.01,0.01,0.99,0.01,0.01,0.99,0.99,0.99,0.01")
    MakeArray(Two(), "0.01,0.99,0.99,0.01,0.99,0.01,0.01,0.99,0.01,0.01,0.99,0.01,0.01,0.99,0.01,0.01,0.99,0.99,0.99,0.99")
    MakeArray(Three(),  "0.99,0.99,0.99,0.01,0.01,0.01,0.01,0.99,0.01,0.99,0.99,0.01,0.01,0.01,0.01,0.99,0.99,0.99,0.99,0.01")
    MakeArray(Four(),  "0.99,0.01,0.01,0.01,0.99,0.01,0.99,0.01,0.99,0.99,0.99,0.99,0.01,0.01,0.99,0.01,0.01,0.01,0.99,0.01")
    MakeArray(Five(),   "0.99,0.99,0.99,0.99,0.99,0.01,0.01,0.01,0.99,0.99,0.99,0.01,0.01,0.01,0.01,0.99,0.99,0.99,0.99,0.01")
    MakeArray(Six(), "0.01,0.99,0.99,0.99,0.99,0.01,0.01,0.01,0.99,0.99,0.99,0.01,0.99,0.01,0.01,0.99,0.01,0.99,0.99,0.01")
    MakeArray(Seven(), "0.99,0.99,0.99,0.99,0.01,0.01,0.01,0.99,0.01,0.01,0.99,0.01,0.01,0.99,0.01,0.01,0.01,0.99,0.01,0.01")
    MakeArray(Eight(),  "0.01,0.99,0.99,0.01,0.99,0.01,0.01,0.99,0.01,0.99,0.99,0.01,0.99,0.01,0.01,0.99,0.01,0.99,0.99,0.01")
    MakeArray(Nine(),  "0.01,0.99,0.99,0.01,0.99,0.01,0.01,0.99,0.01,0.99,0.99,0.99,0.01,0.01,0.01,0.99,0.99,0.99,0.99,0.01")
    
    CurrentDispPtr(1) = @One()   
    CurrentDispPtr(2) = @Two()
    CurrentDispPtr(3) = @Three()
    CurrentDispPtr(4) = @Four()
    CurrentDispPtr(5) = @Five()
    CurrentDispPtr(6) = @Six()
    CurrentDispPtr(7) = @Seven()
    CurrentDispPtr(8) = @Eight()
    CurrentDispPtr(9) = @Nine()
    CurrentDispPtr(10) = @Zero()
    
    CurrentAnsPtr(1) = @Answer1()
    CurrentAnsPtr(2) = @Answer2()
    CurrentAnsPtr(3) = @Answer3()
    CurrentAnsPtr(4) = @Answer4()
    CurrentAnsPtr(5) = @Answer5()
    CurrentAnsPtr(6) = @Answer6()
    CurrentAnsPtr(7) = @Answer7()
    CurrentAnsPtr(8) = @Answer8()
    CurrentAnsPtr(9) = @Answer9()
    CurrentAnsPtr(10) = @Answer0()
    
    For i = 1 To 20
      Display(i) = 1
    Next
    
    For i = 1 To 10
      CurrentAnswer(i) = 1
    Next
    
    Digitloop = 1
    RefreshDelay = 50
    
    ;Create ANN
    InputCount = 20
    HiddenCount = 25
    OutputCount = 10
    LearningRate = 0.55
    
    ANNCreate(InputCount, HiddenCount, OutputCount, LearningRate)
    FontID1 = FontID(LoadFont(#PB_Any, "Arial"  ,  12, #PB_Font_Bold))
    
    Repeat       
      ; don't update NN if paused (ebs)
      If Paused = #False
        frame = frame + 1
        
        ; Init Display Drawing
        StartDrawing(ImageOutput(0))
          Box(1,1,#ScrWidth, #ScrHeight,RGB(30,30,30))
          DigXoffset.l = 20
          DigYOffset.l = 300
          DigPixelSize = 9
          DispY = 100
          
          Digitloop = Digitloop + 1
          If Digitloop > 10
            Digitloop = 1
          EndIf
          
          CopyMemory(CurrentDispPtr(Digitloop),@Display(),8 * 21)
          CopyMemory(CurrentAnsPtr(Digitloop),@CurrentAnswer(),8 * 11)
          
          ANNTrainSet(Display(), Outputs(), CurrentAnswer())
          
          ;Display Layer 1
          
          
          For i = 1 To 20 Step 4                 
            LineXY(DigXoffset + 14, DigYOffset + (i*3), 250, DispY, RGB(0,Display(i) * 255,0))
            LineXY(DigXoffset + 26, DigYOffset + (i*3), 250, DispY + 30, RGB(0,Display(i+1) * 255,0))
            LineXY(DigXoffset + 38, DigYOffset + (i*3), 250, DispY + 60, RGB(0,Display(i+2) * 255,0))
            LineXY(DigXoffset + 50, DigYOffset + (i*3), 250, DispY + 90, RGB(0,Display(i+3) * 255,0))
            
            Circle(250, 70 , DigPixelSize-1, RGB(0,ANNInputs(0) * 127,255))
            Circle(250, DispY, DigPixelSize-1, RGB(0,ANNInputs(i) * 255,0))
            Circle(250, DispY + 30, DigPixelSize-1, RGB(0,ANNInputs(i+1) * 255,0))
            Circle(250, DispY + 60, DigPixelSize-1, RGB(0,ANNInputs(i+2) * 255,0))
            Circle(250, DispY + 90, DigPixelSize-1, RGB(0,ANNInputs(i+3) * 255,0))
            
            DispY = DispY + 120
            
            Box(DigXoffset + 12, DigYOffset + (i*3), DigPixelSize, DigPixelSize,RGB(0,Display(i) * 255,0))
            Box(DigXoffset + 24, DigYOffset + (i*3), DigPixelSize, DigPixelSize,RGB(0,Display(i+1) * 255,0))
            Box(DigXoffset + 36, DigYOffset + (i*3), DigPixelSize, DigPixelSize,RGB(0,Display(i+2) * 255,0))
            Box(DigXoffset + 48, DigYOffset + (i*3), DigPixelSize, DigPixelSize,RGB(0,Display(i+3) * 255,0))             
          Next
          
          ;Display HiddenLayer
          
          DispY = 50
          NegLevel.d = 0
          For i = 0 To HiddenCount
            
            If i = 0
              Circle(600, DispY + (i*25), DigPixelSize-1, RGB(0,ANNHiddenNeurons(i) * 127,255))
            Else
              Circle(600, DispY + (i*25), DigPixelSize-1, RGB(0,ANNHiddenNeurons(i) * 255,0))
            EndIf
            
            ;weights from input layer
            For J = 0 To 20
              If ANNHiddenWeights(J,i) * ANNHiddenNeurons(i) < 0
                NegLevel.d = ANNHiddenWeights(J,i) * ANNHiddenNeurons(i) * -1
                
                PosLevel.d = 0
              Else
                NegLevel.d = 0
                PosLevel.d = ANNHiddenWeights(J,i) * ANNHiddenNeurons(i)
                ;Debug StrD(PosLevel) + "   " + StrD(ANNHiddenWeights(j,i))
              EndIf
              
              If i > 0       
                LineXY(250, DispY + 20 + (J*30), 600, DispY + (i*25), RGB(NegLevel * 255,PosLevel * 255,0))
              EndIf
            Next
            
          Next
          
          ;Display Output Layer
          
          DispY = 100
          
          DrawingFont(FontID1)
          For i = 1 To 10           
            Circle(905, DispY + 12 + (i*40), DigPixelSize, RGB(0,Outputs(i) * 255,0))   
            DrawText(930, DispY + (i*40), Right(Str(i),1) + "  (" + StrD(Outputs(i),2) + ")", RGB(0,Outputs(i) * 255,0) , RGB(30,30,30))
            
            ;weights from hidden layer
            For J = 0 To HiddenCount
              If ANNOutputWeights(J,i) * Outputs(i) < 0
                NegLevel = Outputs(i) * ANNOutputWeights(J,i) * -1
                PosLevel = 0
              Else
                NegLevel = 0
                PosLevel = Outputs(i) * ANNOutputWeights(J,i)
              EndIf
              
              LineXY(600, DispY - 50 + (J*25), 900, DispY + 10 + (i*40), RGB(NegLevel * 255,PosLevel * 255,0))
            Next               
          Next           
          
          
          ;Update text labels
          DrawText(900, 650, "Epoch: " + Str(frame/10), RGB(0,255,0) , RGB(30,30,30))
          
          TmpAvgErr.d = TmpAvgErr  + MeanSqrErr()
          If Digitloop = 10
            AvgErr.d = TmpAvgErr / 10
            TmpAvgErr = 0
          EndIf
          DrawText(900, 670, "Avg Err: " + StrD(AvgErr,6), RGB(0,255,0) , RGB(30,30,30))
          
          Circle(12, 697, DigPixelSize-1, RGB(0,127,255))
          DrawText(30, 690, "Bias ", RGB(0,255,0) , RGB(30,30,30))
          Circle(12, 717, DigPixelSize-1, RGB(0,255,0))
          DrawText(30, 710, "Neuron (Green level shows activiation intensity)", RGB(0,255,0) , RGB(30,30,30))
          LineXY(5, 757, 20, 757, RGB(255,0,0))
          LineXY(5, 737, 20, 737, RGB(0,255,0))
          DrawText(30, 730, "Weight (Green level shows Positive Intensity)", RGB(0,255,0) , RGB(30,30,30))
          DrawText(30, 750, "Weight (Red level shows negative Intensity)", RGB(255,0,0) , RGB(30,30,30))
          
          Delay(RefreshDelay)
          
        StopDrawing()     
        SetGadgetState(0, ImageID(0))
      EndIf
      
      Repeat
        event.l = WindowEvent()   
        Select event                 
          Case #PB_Event_Gadget
            
            Select EventMenu()  ; To see which menu has been selected   
              Case #CmdSlow
                RefreshDelay = 400           
              Case #CmdMed
                RefreshDelay = 50
              Case #CmdFast     
                RefreshDelay = 0 
              Case #CmdRestart
                ANNCreate(InputCount, HiddenCount, OutputCount, LearningRate)   
                frame = 1 
                Digitloop = 1         
              ; added buttons for pause, save and load (ebs)
              Case #CmdPause
                If Paused 
                  Paused = #False
                  SetGadgetText(#CmdPause, "Pause")
                Else
                  Paused = #True
                  SetGadgetText(#CmdPause, "Run")
                EndIf
              Case #CmdSave
                ; save weights (ebs)
                CreateFile(0, "weights.txt")
                For ic.l = 0 To ANNInputCount
                  For hc.l = 0 To ANNHiddenCount
                    WriteDouble(0, ANNHiddenWeights(ic, hc))
                  Next
                Next
                For hc.l = 0 To ANNHiddenCount
                  For oc.l = 0 To ANNOutputCount
                    WriteDouble(0, ANNOutputWeights(hc, oc))
                  Next
                Next
                CloseFile(0)
              Case #CmdLoad
                frame = 1 
                Digitloop = 1         
                ; load weights (ebs)
                ReadFile(0, "weights.txt")
                For ic.l = 0 To ANNInputCount
                  For hc.l = 0 To ANNHiddenCount
                    ANNHiddenWeights(ic, hc) = ReadDouble(0)
                  Next
                Next
                For hc.l = 0 To ANNHiddenCount
                  For oc.l = 0 To ANNOutputCount
                    ANNOutputWeights(hc, oc) = ReadDouble(0)
                  Next
                Next
                CloseFile(0)
            
            EndSelect 
        EndSelect               
      Until event = 0 Or event = #PB_Event_CloseWindow
      
    Until event = #PB_Event_CloseWindow 
    
  EndIf
EndProcedure


Procedure MakeArray(Array TheArray.d(1), CSV.s)
  
  CSVLoop.l
  FieldCount.l
  
  FieldCount = CountString(CSV, ",") + 1
  Redim  TheArray(FieldCount)
  
  For CSVLoop = 1 To FieldCount   
    TheArray(CSVLoop) = ValD(StringField(CSV,CSVLoop ,"," ))     
  Next
  
EndProcedure

[pdwyer]

Thanks! I had the save and load on my to do list to put into the ann.pbi so I can adapt your now.

I'm working on the gui part to be more generic so that:

- main code works in a separate thread
- you can load an input and answers file and the net will build based on that (col count will be neuron count, row counts must match)
- learning and hidden neurons can be adjusted from the gui
- dynamic display of nuerons regardless of count
- highlight target output neuron (with a circle around it) so you can see desired output
- save net (mostly done by you )
- can skip ahead say 100+ epochs to speed things up more

Then, hopefully, people cad add their own data (try an alphabet, or pics saved as csv data, game info or whatever!)

I added your code like this to the pbi file...

Code: Select all

;'==================================================================================

Procedure SaveWeights(Filename.s)

    hfile.i
    hfile = CreateFile(#PB_Any, Filename) 
    For ic.l = 0 To ANNInputCount 
        For hc.l = 0 To ANNHiddenCount 
            WriteDouble(hfile, ANNHiddenWeights(ic, hc)) 
        Next 
    Next 
    For hc.l = 0 To ANNHiddenCount 
        For oc.l = 0 To ANNOutputCount 
            WriteDouble(hfile, ANNOutputWeights(hc, oc)) 
        Next 
    Next 
    CloseFile(hfile) 
EndProcedure

;'==================================================================================
                
Procedure LoadWeights(Filename.s)

    hfile.i
    hfile = ReadFile(#PB_Any, Filename) 
    For ic.l = 0 To ANNInputCount 
        For hc.l = 0 To ANNHiddenCount 
            ANNHiddenWeights(ic, hc) = ReadDouble(hfile) 
        Next 
    Next 
    For hc.l = 0 To ANNHiddenCount 
        For oc.l = 0 To ANNOutputCount 
            ANNOutputWeights(hc, oc) = ReadDouble(hfile) 
        Next 
    Next 
    CloseFile(hfile) 
EndProcedure

;'==================================================================================

[pdwyer]

Here is where I'm up to, added ebs's ideas plus did a bit of a clean up, moved the code into a thread and changed the inputs and outputs to be text so I can load them later. still haven't started the dynamic neuron placement yet. Also option for showing activity or weights and switching between them.

ann.pbi above is updated too with the save and load code

Code: Select all

XIncludeFile "ann.pbi" 

#ScrWidth  = 1200
#ScrHeight = 800
#Exp = 2.7182818284590451 

#CmdSlow = 1
#CmdMed = 2
#CmdFast = 3
#CmdRestart = 4
#CmdSave = 5
#CmdLoad = 6
#CmdPause = 7
#OptWeights = 8
#OptAct = 9

#DispWeights = 1
#DispActivity = 2

Global RefreshDelay.l
Global frame.q
Global digiloop.l
Global InputCount.l 
Global HiddenCount.l
Global OutputCount.l
Global LearningRate.d
Global DisplayType.l
Global Paused.l

Declare Main()
Declare MakeArray(Array TheArray.d(1), CSV.s)
Declare ANNThread(Void.i)

main()

Procedure Main()
    If OpenWindow(0, 0, 0, #ScrWidth, #ScrHeight, "Neural Network", #PB_Window_SystemMenu | #PB_Window_ScreenCentered | #PB_Window_MinimizeGadget) ;And CreateGadgetList(WindowID(0))
     
        If CreateImage(0, #ScrWidth, #ScrHeight-22, 32)
            ImageGadget(0, 0, 0, #ScrWidth, #ScrHeight-22, ImageID(0))                      ; imagegadget standard
        EndIf

        DisplayType = #DispActivity
        
        WorkerThread.i
        
        StartDrawing(ImageOutput(0))
            Box(1,1,#ScrWidth, #ScrHeight,RGB(30,30,30))
        StopDrawing()
                
        ButtonGadget(#CmdSlow, 10,780, 50, 20, "Slow")
        ButtonGadget(#CmdMed,  60,780, 50, 20, "Normal")
        ButtonGadget(#CmdFast,  110,780, 50, 20, "Fast")
        ButtonGadget(#CmdRestart,  160,780, 50, 20, "Restart")
        ButtonGadget(#CmdSave,  260,780, 50, 20, "Save") 
        ButtonGadget(#CmdLoad,  310,780, 50, 20, "Load") 
        ButtonGadget(#CmdPause,  210,780, 50, 20, "Pause")
          
        OptionGadget(#OptWeights, 400,780, 50, 20, "Weights")
        OptionGadget(#OptAct, 480,780, 60, 20, "Activity")  
        
        SetGadgetState(#OptAct, 1)

        Repeat 
            event.l = WindowEvent()   
            Select event                  
                Case #PB_Event_Gadget
                
                    Select EventMenu()  ; To see which menu has been selected   
                        Case #CmdSlow
                            RefreshDelay = 400            
                        Case #CmdMed
                            RefreshDelay = 50
                        Case #CmdFast     
                            RefreshDelay = 0
                        Case #CmdSave 
                            saveweights("weights.txt")
                        Case #CmdLoad 
                            frame = 1 
                            Digitloop = 1          
                            LoadWeights("weights.txt")      
                        Case #CmdPause 
                            If Paused 
                                Paused = #False 
                                SetGadgetText(#CmdPause, "Pause") 
                            Else 
                                Paused = #True 
                                SetGadgetText(#CmdPause, "Run") 
                            EndIf
                        Case #OptWeights
                            DisplayType = #DispWeights
                        Case #OptAct    
                            DisplayType = #DispActivity
                            
                        Case #CmdRestart
                            If WorkerThread
                                KillThread(WorkerThread)
                                Delay(50)
                            EndIf
                            WorkerThread = CreateThread(@ANNThread(),0)           
                    EndSelect
                                     
            EndSelect               
        Until event = #PB_Event_CloseWindow 
             
    EndIf
EndProcedure


Procedure ANNThread(Void.i)

    ANNCreate(InputCount, HiddenCount, OutputCount, LearningRate)    
    frame = 1  
    Digitloop = 1  
    
    TrainingLoop.l  
    Dim Outputs.d(10)
    Dim Inputs.s(10)
    Dim Answers.s(10)    
    Dim Display.d(20) 
    Dim CurrentDispPtr.l(10)
    Dim CurrentAnsPtr.l(10)
    Dim CurrentAnswer.d(10)

    
    Inputs(1) = "0.01,0.99,0.01,0.01,0.99,0.99,0.01,0.01,0.01,0.99,0.01,0.01,0.01,0.99,0.01,0.01,0.99,0.99,0.99,0.01"
    Inputs(2) = "0.01,0.99,0.99,0.01,0.99,0.01,0.01,0.99,0.01,0.01,0.99,0.01,0.01,0.99,0.01,0.01,0.99,0.99,0.99,0.99"
    Inputs(3) = "0.99,0.99,0.99,0.01,0.01,0.01,0.01,0.99,0.01,0.99,0.99,0.01,0.01,0.01,0.01,0.99,0.99,0.99,0.99,0.01"
    Inputs(4) = "0.99,0.01,0.01,0.01,0.99,0.01,0.99,0.01,0.99,0.99,0.99,0.99,0.01,0.01,0.99,0.01,0.01,0.01,0.99,0.01"
    Inputs(5) = "0.99,0.99,0.99,0.99,0.99,0.01,0.01,0.01,0.99,0.99,0.99,0.01,0.01,0.01,0.01,0.99,0.99,0.99,0.99,0.01"
    Inputs(6) = "0.01,0.99,0.99,0.99,0.99,0.01,0.01,0.01,0.99,0.99,0.99,0.01,0.99,0.01,0.01,0.99,0.01,0.99,0.99,0.01"
    Inputs(7) = "0.99,0.99,0.99,0.99,0.01,0.01,0.01,0.99,0.01,0.01,0.99,0.01,0.01,0.99,0.01,0.01,0.01,0.99,0.01,0.01"
    Inputs(8) = "0.01,0.99,0.99,0.01,0.99,0.01,0.01,0.99,0.01,0.99,0.99,0.01,0.99,0.01,0.01,0.99,0.01,0.99,0.99,0.01"
    Inputs(9) = "0.01,0.99,0.99,0.01,0.99,0.01,0.01,0.99,0.01,0.99,0.99,0.99,0.01,0.01,0.01,0.99,0.99,0.99,0.99,0.01"
    Inputs(10) = "0.01,0.99,0.99,0.01,0.99,0.01,0.01,0.99,0.99,0.01,0.01,0.99,0.99,0.01,0.01,0.99,0.01,0.99,0.99,0.01"
    
    Answers(1) = "1,0,0,0,0,0,0,0,0,0"
    Answers(2) = "0,1,0,0,0,0,0,0,0,0"
    Answers(3) = "0,0,1,0,0,0,0,0,0,0"
    Answers(4) = "0,0,0,1,0,0,0,0,0,0"
    Answers(5) = "0,0,0,0,1,0,0,0,0,0"
    Answers(6) = "0,0,0,0,0,1,0,0,0,0"
    Answers(7) = "0,0,0,0,0,0,1,0,0,0"
    Answers(8) = "0,0,0,0,0,0,0,1,0,0"
    Answers(9) = "0,0,0,0,0,0,0,0,1,0"
    Answers(10) = "0,0,0,0,0,0,0,0,0,1"
;     
    Digitloop = 1
    RefreshDelay = 50
    
    ;Create ANN
    InputCount = 20
    HiddenCount = 25
    OutputCount = 10
    LearningRate = 0.55
    
    ANNCreate(InputCount, HiddenCount, OutputCount, LearningRate) 
    FontID1 = FontID(LoadFont(#PB_Any, "Arial"  ,  12, #PB_Font_Bold))
    
    Repeat  
        If Paused = #False
              
            frame = frame + 1
            
            ; Init Display Drawing
            StartDrawing(ImageOutput(0))
            Box(1,1,#ScrWidth, #ScrHeight,RGB(30,30,30))
            DigXoffset.l = 20
            DigYOffset.l = 300
            DigPixelSize = 9
            DispY = 100
            
            Digitloop = Digitloop + 1
            If Digitloop > 10
                Digitloop = 1
            EndIf

            
            MakeArray(Display(),inputs(Digitloop))
            MakeArray(CurrentAnswer(),Answers(Digitloop))
            
            ANNTrainSet(Display(), Outputs(), CurrentAnswer())
            
            ;Display Layer 1

            For i = 1 To 20 Step 4                  
                LineXY(DigXoffset + 14, DigYOffset + (i*3), 250, DispY, RGB(0,Display(i) * 255,0))
                LineXY(DigXoffset + 26, DigYOffset + (i*3), 250, DispY + 30, RGB(0,Display(i+1) * 255,0))
                LineXY(DigXoffset + 38, DigYOffset + (i*3), 250, DispY + 60, RGB(0,Display(i+2) * 255,0))
                LineXY(DigXoffset + 50, DigYOffset + (i*3), 250, DispY + 90, RGB(0,Display(i+3) * 255,0)) 
                
                Circle(250, 70 , DigPixelSize-1, RGB(0,ANNInputs(0) * 127,255))
                Circle(250, DispY, DigPixelSize-1, RGB(0,ANNInputs(i) * 255,0))
                Circle(250, DispY + 30, DigPixelSize-1, RGB(0,ANNInputs(i+1) * 255,0))
                Circle(250, DispY + 60, DigPixelSize-1, RGB(0,ANNInputs(i+2) * 255,0))
                Circle(250, DispY + 90, DigPixelSize-1, RGB(0,ANNInputs(i+3) * 255,0))
                
                DispY = DispY + 120
                
                Box(DigXoffset + 12, DigYOffset + (i*3), DigPixelSize, DigPixelSize,RGB(0,Display(i) * 255,0))
                Box(DigXoffset + 24, DigYOffset + (i*3), DigPixelSize, DigPixelSize,RGB(0,Display(i+1) * 255,0))
                Box(DigXoffset + 36, DigYOffset + (i*3), DigPixelSize, DigPixelSize,RGB(0,Display(i+2) * 255,0))
                Box(DigXoffset + 48, DigYOffset + (i*3), DigPixelSize, DigPixelSize,RGB(0,Display(i+3) * 255,0))             
            Next
            
            ;Display HiddenLayer
            
            DispY = 50
            NegLevel.d = 0
            For i = 0 To HiddenCount
            
                If i = 0
                    Circle(600, DispY + (i*25), DigPixelSize-1, RGB(0,ANNHiddenNeurons(i) * 127,255)) 
                Else
                    Circle(600, DispY + (i*25), DigPixelSize-1, RGB(0,ANNHiddenNeurons(i) * 255,0)) 
                EndIf
            
                ;weights from input layer
                
                For j = 0 To 20
                    If DisplayType = #DispActivity
                        If ANNHiddenWeights(j,i) * ANNHiddenNeurons(i) < 0
                            NegLevel.d = ANNHiddenWeights(j,i) * ANNHiddenNeurons(i) * -1                    
                            PosLevel.d = 0
                        Else
                            NegLevel.d = 0
                            PosLevel.d = ANNHiddenWeights(j,i) * ANNHiddenNeurons(i)
                        EndIf 
                    Else
                        If ANNHiddenWeights(j,i) < 0
                            NegLevel.d = 1 /(1 + Pow(#Exp,(-1 * ANNHiddenWeights(j,i))))
                             
                            PosLevel.d = 0
                        Else
                            NegLevel.d = 0
                            PosLevel.d = 1 /(1 + Pow(#Exp,(-1 * ANNHiddenWeights(j,i))))
                        EndIf
                    EndIf 

                    
                    If i > 0       
                        LineXY(250, DispY + 20 + (j*30), 600, DispY + (i*25), RGB(NegLevel * 255,PosLevel * 255,0))
                    EndIf
                Next
                
            
            Next
            
            ;Display Output Layer
            
            DispY = 100
            
            DrawingFont(FontID1)
            For i = 1 To 10           
                Circle(905, DispY + 12 + (i*40), DigPixelSize, RGB(0,Outputs(i) * 255,0))    
                DrawText(930, DispY + (i*40), Right(Str(i),1) + "  (" + StrD(Outputs(i),2) + ")", RGB(0,Outputs(i) * 255,0) , RGB(30,30,30))
            
                ;weights from hidden layer
                For j = 0 To HiddenCount
                    If DisplayType = #DispActivity
                        If ANNOutputWeights(j,i) * Outputs(i) < 0
                            NegLevel = Outputs(i) * ANNOutputWeights(j,i) * -1
                            PosLevel = 0
                        Else
                            NegLevel = 0
                            PosLevel = Outputs(i) * ANNOutputWeights(j,i)
                        EndIf
                    Else
                        If ANNOutputWeights(j,i) < 0
                            NegLevel = (1 / (1 + Pow(#Exp,(-1 * ANNOutputWeights(j,i))))) ;* -1
                            PosLevel = 0
                        Else
                            NegLevel = 0
                            PosLevel = (1 / (1 + Pow(#Exp,(-1 * ANNOutputWeights(j,i))))) 
                        EndIf
                    EndIf
                    LineXY(600, DispY - 50 + (j*25), 900, DispY + 10 + (i*40), RGB(NegLevel * 255,PosLevel * 255,0))
                Next               
            Next            
            
            
            ;Update text labels
            DrawText(900, 650, "Epoch: " + Str(frame/10), RGB(0,255,0) , RGB(30,30,30))
            
            TmpAvgErr.d = TmpAvgErr  + MeanSqrErr()
            If Digitloop = 10
                AvgErr.d = TmpAvgErr / 10
                TmpAvgErr = 0
            EndIf
            DrawText(900, 670, "Avg Err: " + StrD(AvgErr,8), RGB(0,255,0) , RGB(30,30,30))
            
            Circle(12, 697, DigPixelSize-1, RGB(0,127,255))
            DrawText(30, 690, "Bias ", RGB(0,255,0) , RGB(30,30,30))
            Circle(12, 717, DigPixelSize-1, RGB(0,255,0))
            DrawText(30, 710, "Neuron (Green level shows activiation intensity)", RGB(0,255,0) , RGB(30,30,30))
            LineXY(5, 757, 20, 757, RGB(255,0,0))
            LineXY(5, 737, 20, 737, RGB(0,255,0))
            DrawText(30, 730, "Weight (Green level shows Positive Intensity)", RGB(0,255,0) , RGB(30,30,30))
            DrawText(30, 750, "Weight (Red level shows negative Intensity)", RGB(255,0,0) , RGB(30,30,30))
            
            Delay(RefreshDelay)
            
            StopDrawing()     
            SetGadgetState(0, ImageID(0))
        EndIf
    ForEver
    
EndProcedure

Procedure MakeArray(Array TheArray.d(1), CSV.s)
    
    CSVLoop.l
    FieldCount.l
    
    FieldCount = CountString(CSV, ",") + 1 
    ReDim  TheArray(FieldCount)
    
    For CSVLoop = 1 To FieldCount    
        TheArray(CSVLoop) = ValD(StringField(CSV,CSVLoop ,"," ))      
    Next
    
EndProcedure 

[pdwyer]

New version
- neuron placement is dynamic and scalable. inputs and outputs based on data for the most part, one hard coded area left
- can adjust learning and hidden neurons (need to hit restart button to take effect)
- disable net graphics switch to speed it up through epochs
- configured this time with upper case alphabet rather than numeral. takes a few thousand epochs to train this
- target output is shown with a circle
- suppressed weight lines if close to zero activity (getting two crowded to see what's happening)
- ann.pbi same as above still

pretty much ready if you want to put you're own data in to play with or learn. Few things left to do still

Code: Select all

XIncludeFile "ann.pbi" 

#ScrWidth  = 1200
#ScrHeight = 800
#Exp = 2.7182818284590451 

#CmdSlow = 1
#CmdMed = 2
#CmdFast = 3
#CmdRestart = 4
#CmdSave = 5
#CmdLoad = 6
#CmdPause = 7
#OptWeights = 8
#OptAct = 9
#txtLearning = 10
#txtHiddenNeurons = 11
#lblLearning = 12
#lblHiddenNeurons = 13
#ChkShowGraphics = 14

#DispWeights = 1
#DispActivity = 2
#weightDispSensitivity = 0.05

Global RefreshDelay.l
Global frame.q
Global digiloop.l
Global InputCount.l 
Global HiddenCount.l
Global OutputCount.l
Global LearningRate.d
Global DisplayType.l
Global DisplayGraphics.l = #True
Global Paused.l

Declare Main()
Declare MakeArray(Array TheArray.d(1), CSV.s)
Declare ANNThread(Void.i)

;=======================================================================

main()

;=======================================================================

Procedure Main()
    If OpenWindow(0, 0, 0, #ScrWidth, #ScrHeight, "Neural Network", #PB_Window_SystemMenu | #PB_Window_ScreenCentered | #PB_Window_MinimizeGadget) ;And CreateGadgetList(WindowID(0))
     
        If CreateImage(0, #ScrWidth, #ScrHeight-22, 32)
            ImageGadget(0, 0, 0, #ScrWidth, #ScrHeight-22, ImageID(0))                      ; imagegadget standard
        EndIf

        DisplayType = #DispActivity
        
        WorkerThread.i
        
        StartDrawing(ImageOutput(0))
            Box(1,1,#ScrWidth, #ScrHeight,RGB(30,30,30))
        StopDrawing()
                
        ButtonGadget(#CmdSlow, 10,780, 50, 20, "Slow")
        ButtonGadget(#CmdMed,  60,780, 50, 20, "Normal")
        ButtonGadget(#CmdFast,  110,780, 50, 20, "Fast")
        ButtonGadget(#CmdRestart,  160,780, 50, 20, "Restart")
        ButtonGadget(#CmdSave,  260,780, 50, 20, "Save") 
        ButtonGadget(#CmdLoad,  310,780, 50, 20, "Load") 
        ButtonGadget(#CmdPause,  210,780, 50, 20, "Pause")
          
        OptionGadget(#OptWeights, 400,780, 50, 20, "Weights")
        OptionGadget(#OptAct, 480,780, 60, 20, "Activity")         
        SetGadgetState(#OptAct, 1)
        
        TextGadget(#lblLearning, 560,783, 120, 20, "Learning Rate:")
        TextGadget(#lblHiddenNeurons, 700,783, 160, 20, "Hidden Neurons:")
        StringGadget(#txtLearning, 640, 780, 30, 20, "")
        StringGadget(#txtHiddenNeurons, 800, 780, 30, 20, "")
        
        SetGadgetText(#txtLearning,"0.45")
        SetGadgetText(#txtHiddenNeurons,"50")
        
        CheckBoxGadget(#ChkShowGraphics, 850, 780, 160, 20, "Show Graphics")
        SetGadgetState(#ChkShowGraphics, 1)

        Repeat 
            event.l = WindowEvent()   
            Select event                  
                Case #PB_Event_Gadget
                
                    Select EventMenu()  ; To see which menu has been selected   
                        Case #CmdSlow
                            RefreshDelay = 400            
                        Case #CmdMed
                            RefreshDelay = 50
                        Case #CmdFast     
                            RefreshDelay = 0
                        Case #CmdSave 
                            saveweights("weights.txt")
                        Case #CmdLoad 
                            frame = 1 
                            Digitloop = 1          
                            LoadWeights("weights.txt")      
                        Case #CmdPause 
                            If Paused 
                                Paused = #False 
                                SetGadgetText(#CmdPause, "Pause") 
                            Else 
                                Paused = #True 
                                SetGadgetText(#CmdPause, "Run") 
                            EndIf
                        Case #OptWeights
                            DisplayType = #DispWeights
                        Case #OptAct    
                            DisplayType = #DispActivity
                        Case #ChkShowGraphics    
                            DisplayGraphics = GetGadgetState(#ChkShowGraphics)
                            
                        Case #CmdRestart
                            If WorkerThread
                                KillThread(WorkerThread)
                                Delay(50)
                            EndIf
                            LearningRate = ValD(GetGadgetText(#txtLearning))
                            HiddenCount = ValD(GetGadgetText(#txtHiddenNeurons))
                            WorkerThread = CreateThread(@ANNThread(),0)           
                    EndSelect
                                     
            EndSelect               
        Until event = #PB_Event_CloseWindow 
             
    EndIf
EndProcedure

;=======================================================================

Procedure ANNThread(Void.i)

    ANNCreate(InputCount, HiddenCount, OutputCount, LearningRate)    
    frame = 1  
    Digitloop = 1  
    
    TrainingLoop.l  
    
    Dim Inputs.s(26)
    Dim Answers.s(26)    
    
;     Inputs(1) = "0.01,0.99,0.01,0.01,0.99,0.99,0.01,0.01,0.01,0.99,0.01,0.01,0.01,0.99,0.01,0.01,0.99,0.99,0.99,0.01"
;     Inputs(2) = "0.01,0.99,0.99,0.01,0.99,0.01,0.01,0.99,0.01,0.01,0.99,0.01,0.01,0.99,0.01,0.01,0.99,0.99,0.99,0.99"
;     Inputs(3) = "0.99,0.99,0.99,0.01,0.01,0.01,0.01,0.99,0.01,0.99,0.99,0.01,0.01,0.01,0.01,0.99,0.99,0.99,0.99,0.01"
;     Inputs(4) = "0.99,0.01,0.01,0.01,0.99,0.01,0.99,0.01,0.99,0.99,0.99,0.99,0.01,0.01,0.99,0.01,0.01,0.01,0.99,0.01"
;     Inputs(5) = "0.99,0.99,0.99,0.99,0.99,0.01,0.01,0.01,0.99,0.99,0.99,0.01,0.01,0.01,0.01,0.99,0.99,0.99,0.99,0.01"
;     Inputs(6) = "0.01,0.99,0.99,0.99,0.99,0.01,0.01,0.01,0.99,0.99,0.99,0.01,0.99,0.01,0.01,0.99,0.01,0.99,0.99,0.01"
;     Inputs(7) = "0.99,0.99,0.99,0.99,0.01,0.01,0.01,0.99,0.01,0.01,0.99,0.01,0.01,0.99,0.01,0.01,0.01,0.99,0.01,0.01"
;     Inputs(8) = "0.01,0.99,0.99,0.01,0.99,0.01,0.01,0.99,0.01,0.99,0.99,0.01,0.99,0.01,0.01,0.99,0.01,0.99,0.99,0.01"
;     Inputs(9) = "0.01,0.99,0.99,0.01,0.99,0.01,0.01,0.99,0.01,0.99,0.99,0.99,0.01,0.01,0.01,0.99,0.99,0.99,0.99,0.01"
;     Inputs(10) = "0.01,0.99,0.99,0.01,0.99,0.01,0.01,0.99,0.99,0.01,0.01,0.99,0.99,0.01,0.01,0.99,0.01,0.99,0.99,0.01"  
;         
;     Answers(1) = "1,0,0,0,0,0,0,0,0,0"
;     Answers(2) = "0,1,0,0,0,0,0,0,0,0"
;     Answers(3) = "0,0,1,0,0,0,0,0,0,0"
;     Answers(4) = "0,0,0,1,0,0,0,0,0,0"
;     Answers(5) = "0,0,0,0,1,0,0,0,0,0"
;     Answers(6) = "0,0,0,0,0,1,0,0,0,0"
;     Answers(7) = "0,0,0,0,0,0,1,0,0,0"
;     Answers(8) = "0,0,0,0,0,0,0,1,0,0"
;     Answers(9) = "0,0,0,0,0,0,0,0,1,0"
;     Answers(10) = "0,0,0,0,0,0,0,0,0,1"

    Inputs(1)  = "0.1,0.9,0.9,0.1,0.1,0.9,0.1,0.1,0.9,0.1,0.9,0.9,0.9,0.9,0.1,0.9,0.1,0.1,0.9,0.1,0.9,0.1,0.1,0.9,0.1"
    Inputs(2)  = "0.9,0.9,0.9,0.1,0.1,0.9,0.1,0.1,0.9,0.1,0.9,0.9,0.9,0.1,0.1,0.9,0.1,0.1,0.9,0.1,0.9,0.9,0.9,0.1,0.1"
    Inputs(3)  = "0.1,0.9,0.9,0.1,0.1,0.9,0.1,0.1,0.9,0.1,0.9,0.1,0.1,0.1,0.1,0.9,0.1,0.1,0.9,0.1,0.1,0.9,0.9,0.1,0.1"
    Inputs(4)  = "0.9,0.9,0.9,0.1,0.1,0.9,0.1,0.1,0.9,0.1,0.9,0.1,0.1,0.9,0.1,0.9,0.1,0.1,0.9,0.1,0.9,0.9,0.9,0.1,0.1"
    Inputs(5)  = "0.9,0.9,0.9,0.9,0.1,0.9,0.1,0.1,0.1,0.1,0.9,0.9,0.9,0.1,0.1,0.9,0.1,0.1,0.1,0.1,0.9,0.9,0.9,0.9,0.1"
    Inputs(6)  = "0.9,0.9,0.9,0.9,0.1,0.9,0.1,0.1,0.1,0.1,0.9,0.9,0.9,0.1,0.1,0.9,0.1,0.1,0.1,0.1,0.9,0.1,0.1,0.1,0.1"
    Inputs(7)  = "0.1,0.9,0.9,0.1,0.1,0.9,0.1,0.1,0.9,0.1,0.9,0.1,0.1,0.1,0.1,0.9,0.1,0.1,0.9,0.9,0.1,0.9,0.9,0.1,0.1"
    Inputs(8)  = "0.9,0.1,0.1,0.9,0.1,0.9,0.1,0.1,0.9,0.1,0.9,0.9,0.9,0.9,0.1,0.9,0.1,0.1,0.9,0.1,0.9,0.1,0.1,0.9,0.1"
    Inputs(9)  = "0.1,0.9,0.9,0.9,0.1,0.1,0.1,0.9,0.1,0.1,0.1,0.1,0.9,0.1,0.1,0.1,0.1,0.9,0.1,0.1,0.1,0.9,0.9,0.9,0.1"
    Inputs(10)  = "0.1,0.9,0.9,0.9,0.1,0.1,0.1,0.9,0.1,0.1,0.1,0.1,0.9,0.1,0.1,0.9,0.1,0.9,0.1,0.1,0.9,0.9,0.9,0.1,0.1"
    Inputs(11)  = "0.9,0.1,0.1,0.9,0.1,0.9,0.1,0.9,0.1,0.1,0.9,0.9,0.1,0.1,0.1,0.9,0.1,0.9,0.1,0.1,0.9,0.1,0.1,0.9,0.1"
    Inputs(12)  = "0.1,0.1,0.9,0.1,0.1,0.1,0.1,0.9,0.1,0.1,0.1,0.1,0.9,0.1,0.1,0.1,0.1,0.9,0.1,0.1,0.1,0.1,0.9,0.9,0.9"
    Inputs(13)  = "0.9,0.1,0.1,0.1,0.9,0.9,0.9,0.1,0.9,0.9,0.9,0.1,0.9,0.1,0.9,0.9,0.1,0.9,0.1,0.9,0.9,0.1,0.1,0.1,0.9"
    Inputs(14)  = "0.9,0.1,0.1,0.1,0.9,0.9,0.9,0.1,0.1,0.9,0.9,0.1,0.9,0.1,0.9,0.9,0.1,0.1,0.9,0.9,0.9,0.1,0.1,0.1,0.9"
    Inputs(15)  = "0.1,0.9,0.9,0.1,0.1,0.9,0.1,0.1,0.9,0.1,0.9,0.1,0.1,0.9,0.1,0.9,0.1,0.1,0.9,0.1,0.1,0.9,0.9,0.1,0.1"
    Inputs(16)  = "0.9,0.9,0.9,0.1,0.1,0.9,0.1,0.1,0.9,0.1,0.9,0.9,0.9,0.1,0.1,0.9,0.1,0.1,0.1,0.1,0.9,0.1,0.1,0.1,0.1"
    Inputs(17)  = "0.1,0.9,0.9,0.1,0.1,0.9,0.1,0.1,0.9,0.1,0.9,0.1,0.1,0.9,0.1,0.9,0.1,0.9,0.9,0.1,0.1,0.9,0.9,0.9,0.1"
    Inputs(18)  = "0.9,0.9,0.9,0.1,0.1,0.9,0.1,0.1,0.9,0.1,0.9,0.9,0.9,0.1,0.1,0.9,0.1,0.1,0.9,0.1,0.9,0.1,0.1,0.9,0.1"
    Inputs(19)  = "0.1,0.9,0.9,0.9,0.1,0.9,0.1,0.1,0.1,0.1,0.1,0.9,0.9,0.1,0.1,0.1,0.1,0.1,0.9,0.1,0.9,0.9,0.9,0.1,0.1"
    Inputs(20)  = "0.9,0.9,0.9,0.9,0.9,0.1,0.1,0.9,0.1,0.1,0.1,0.1,0.9,0.1,0.1,0.1,0.1,0.9,0.1,0.1,0.1,0.1,0.9,0.1,0.1"
    Inputs(21)  = "0.9,0.1,0.1,0.9,0.1,0.9,0.1,0.1,0.9,0.1,0.9,0.1,0.1,0.9,0.1,0.9,0.1,0.1,0.9,0.1,0.1,0.9,0.9,0.1,0.1"
    Inputs(22)  = "0.9,0.1,0.1,0.1,0.9,0.1,0.9,0.1,0.9,0.1,0.1,0.9,0.1,0.9,0.1,0.1,0.9,0.1,0.9,0.1,0.1,0.1,0.9,0.1,0.1"
    Inputs(23)  = "0.9,0.1,0.1,0.1,0.9,0.9,0.1,0.9,0.1,0.9,0.9,0.1,0.9,0.1,0.9,0.9,0.1,0.9,0.1,0.9,0.9,0.9,0.1,0.9,0.9"
    Inputs(24)  = "0.9,0.1,0.1,0.1,0.9,0.1,0.9,0.1,0.9,0.1,0.1,0.1,0.9,0.1,0.1,0.1,0.9,0.1,0.9,0.1,0.9,0.1,0.1,0.1,0.9"
    Inputs(25)  = "0.9,0.1,0.1,0.1,0.9,0.1,0.9,0.1,0.9,0.1,0.1,0.1,0.9,0.1,0.1,0.1,0.1,0.9,0.1,0.1,0.1,0.1,0.9,0.1,0.1"
    Inputs(26)  = "0.9,0.9,0.9,0.9,0.9,0.1,0.1,0.1,0.9,0.1,0.1,0.1,0.9,0.1,0.1,0.1,0.9,0.1,0.1,0.1,0.9,0.9,0.9,0.9,0.9"
    
    
    Answers(1)  = "1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0"
    Answers(2)  = "0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0"
    Answers(3)  = "0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0"
    Answers(4)  = "0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0"
    Answers(5)  = "0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0"
    Answers(6)  = "0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0"
    Answers(7)  = "0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0"
    Answers(8)  = "0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0"
    Answers(9)  = "0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0"
    Answers(10)  = "0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0"
    Answers(11)  = "0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0"
    Answers(12)  = "0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0"
    Answers(13)  = "0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0"
    Answers(14)  = "0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0"
    Answers(15)  = "0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0"
    Answers(16)  = "0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0"
    Answers(17)  = "0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0"
    Answers(18)  = "0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0"
    Answers(19)  = "0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0"
    Answers(20)  = "0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0"
    Answers(21)  = "0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0"
    Answers(22)  = "0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0"
    Answers(23)  = "0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0"
    Answers(24)  = "0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0"
    Answers(25)  = "0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0"
    Answers(26)  = "0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1"


;     
    Digitloop = 1
    RefreshDelay = 50
    HiddenColHeight = 20
    
    ;Create ANN
    InputCount = CountString(Inputs(1), ",") + 1;20
    ;HiddenCount = 25
    OutputCount = CountString(Answers(1), ",") + 1 ;10
    TrainCount = ArraySize(Inputs())
    ;LearningRate = 0.55
    
    Dim InputNeuronLoc.point(InputCount)
    Dim OutputNeuronLoc.point(OutputCount)
    Dim HiddenNeuronLoc.point(HiddenCount)
    Dim Outputs.d(OutputCount)
    Dim Display.d(InputCount) 
    Dim CurrentAnswer.d(OutputCount)
    
    ANNCreate(InputCount, HiddenCount, OutputCount, LearningRate) 
    FontID1 = FontID(LoadFont(#PB_Any, "Arial"  ,  12, #PB_Font_Bold))
    
    XInOffset = 70
    YInOffset = 300

    XOutOffset = 900
    YOutOffset = 300

    XHiddenOffset = 600
    YHiddenOffset = 400
            
    InputNeuronWidth.l = Round(Sqr(InputCount),0)
    OutputNeuronWidth.l = Round(Sqr(OutputCount),0)
    
    Repeat  
        If Paused = #False
              
            frame = frame + 1
            
            ; Init Display Drawing
            StartDrawing(ImageOutput(0))
            Box(1,1,#ScrWidth, #ScrHeight,RGB(30,30,30))
            NeuronSize = 8
            NeuronGap = 15
            DispY = 100
            
            Trainloop = Trainloop + 1
            If Trainloop > TrainCount
                Trainloop = 1
            EndIf

            
            MakeArray(Display(),inputs(Trainloop))
            MakeArray(CurrentAnswer(),Answers(Trainloop))
            
            ANNTrainSet(Display(), Outputs(), CurrentAnswer())
            
            If DisplayGraphics = #True
                ;Display Layer 1
                
                ;Bias Locations
                InputNeuronLoc(0)\x = XInOffset + (InputNeuronWidth * NeuronGap +  NeuronGap)
                InputNeuronLoc(0)\y = YInOffset - (NeuronGap * 4)
                
                HiddenNeuronLoc(0)\x = XHiddenOffset
                HiddenNeuronLoc(0)\y = YHiddenOffset - (24 * NeuronGap)
                
                For i = 1 To InputCount ; Step 4            
                    InputNeuronLoc(i)\x = XInOffset + ((i-1) % InputNeuronWidth) * (NeuronSize + NeuronGap) 
                    InputNeuronLoc(i)\y = YInOffset + ((i-1)/InputNeuronWidth*(NeuronSize + NeuronGap))
                Next
                
                For i = 1 To OutputCount ; Step 4            
                    OutputNeuronLoc(i)\x = XOutOffset + ((i-1) % OutputNeuronWidth) * (NeuronSize + NeuronGap) 
                    OutputNeuronLoc(i)\y = YOutOffset + ((i-1)/OutputNeuronWidth*(NeuronSize + NeuronGap))
                Next
                
                For i = 1 To HiddenCount / HiddenColHeight + 1 ;do we need to catch extras then or make this "+1"                 
                            ;i=x, j=y         
                    For j = 1 To HiddenColHeight  
                        If ((i-1) * HiddenColHeight + j) > HiddenCount
                            ;skip
                        Else
                            HiddenNeuronLoc((i-1) * HiddenColHeight + j)\x = XHiddenOffset + ((i-1) * 2 * NeuronGap)
                            If j % 2 = 0
                                HiddenNeuronLoc((i-1) * HiddenColHeight + j)\y = YHiddenOffset - (j * NeuronGap)
                            Else
                                HiddenNeuronLoc((i-1) * HiddenColHeight + j)\y = YHiddenOffset + (j * NeuronGap)
                            EndIf
                        EndIf
                    Next
                Next

                                              
                DispY = 50
                NegLevel.d = 0
                For i = 0 To HiddenCount                
                    ;weights from input layer
                    
                    For j = 0 To InputCount
                        If DisplayType = #DispActivity
                            If ANNHiddenWeights(j,i) * ANNHiddenNeurons(i) < 0
                                NegLevel.d = ANNHiddenWeights(j,i) * ANNHiddenNeurons(i) * -1                    
                                PosLevel.d = 0
                            Else
                                NegLevel.d = 0
                                PosLevel.d = ANNHiddenWeights(j,i) * ANNHiddenNeurons(i)
                            EndIf 
                        Else
                            If ANNHiddenWeights(j,i) < 0
                                NegLevel.d = 1 /(1 + Pow(#Exp,(-1 * ANNHiddenWeights(j,i))))
                                 
                                PosLevel.d = 0
                            Else
                                NegLevel.d = 0
                                PosLevel.d = 1 /(1 + Pow(#Exp,(-1 * ANNHiddenWeights(j,i))))
                            EndIf
                        EndIf 
    
                        
                        If i > 0        
                            If NegLevel < #weightDispSensitivity And PosLevel < #weightDispSensitivity
                            Else
                                LineXY(InputNeuronLoc(j)\x, InputNeuronLoc(j)\y, HiddenNeuronLoc(i)\x, HiddenNeuronLoc(i)\y, RGB(NegLevel * 255,PosLevel * 255,0))
                            EndIf
                        EndIf
                    Next
                    
                
                Next
                
                Circle(InputNeuronLoc(0)\x, InputNeuronLoc(0)\y, NeuronSize, RGB(0,127,255))
                For i = 1 To InputCount ; Step 4
                    Circle(InputNeuronLoc(i)\x, InputNeuronLoc(i)\y, NeuronSize, RGB(0,Display(i) * 255,0))  ;need to adj for negatives
                Next


                                    
                ;Display Output Layer
                
                DispY = 100
                
                
                For i = 1 To OutputCount           
                    ;weights from hidden layer
                    For j = 0 To HiddenCount
                        If DisplayType = #DispActivity
                            If ANNOutputWeights(j,i) * Outputs(i) < 0
                                NegLevel = Outputs(i) * ANNOutputWeights(j,i) * -1
                                PosLevel = 0
                            Else
                                NegLevel = 0
                                PosLevel = Outputs(i) * ANNOutputWeights(j,i)
                            EndIf
                        Else
                            If ANNOutputWeights(j,i) < 0
                                NegLevel = (1 / (1 + Pow(#Exp,(-1 * ANNOutputWeights(j,i))))) ;* -1
                                PosLevel = 0
                            Else
                                NegLevel = 0
                                PosLevel = (1 / (1 + Pow(#Exp,(-1 * ANNOutputWeights(j,i))))) 
                            EndIf
                        EndIf
                        If NegLevel < #weightDispSensitivity And PosLevel < #weightDispSensitivity
                        Else
                            LineXY(HiddenNeuronLoc(j)\x, HiddenNeuronLoc(j)\y, OutputNeuronLoc(i)\x, OutputNeuronLoc(i)\y, RGB(NegLevel * 255,PosLevel * 255,0))
                        EndIf
                    Next               
                Next            
    
                For i = 0 To HiddenCount
                
                    If i = 0
                        Circle(HiddenNeuronLoc(0)\x, HiddenNeuronLoc(0)\y, NeuronSize, RGB(0,ANNHiddenNeurons(i) * 127,255)) 
                    Else
                        Circle(HiddenNeuronLoc(i)\x, HiddenNeuronLoc(i)\y, NeuronSize, RGB(0,ANNHiddenNeurons(i) * 255,0)) 
                    EndIf
                Next
                
                For i = 1 To OutputCount           
                    ;Circle(905, DispY + 12 + (i*40), NeuronSize, RGB(0,Outputs(i) * 255,0))
                    Circle(OutputNeuronLoc(i)\x, OutputNeuronLoc(i)\y, NeuronSize, RGB(0,Outputs(i) * 255,0))
                    
                    If CurrentAnswer(i) = 1
                        DrawingMode(#PB_2DDrawing_Outlined)
                            Circle(OutputNeuronLoc(i)\x, OutputNeuronLoc(i)\y, NeuronSize + 1, RGB(0,127,255))            
                        DrawingMode(#PB_2DDrawing_Default)
                    EndIf
                Next                
                ;Update text labels
                
            
            EndIf
            
            TmpAvgErr.d = TmpAvgErr  + MeanSqrErr()
            If Trainloop = TrainCount
                AvgErr.d = TmpAvgErr / 10
                TmpAvgErr = 0
            EndIf
           
            DrawingFont(FontID1)
            DrawText(900, 670, "Avg Err: " + StrD(AvgErr,8), RGB(0,255,0) , RGB(30,30,30))
            DrawText(900, 650, "Epoch: " + Str(frame/10), RGB(0,255,0) , RGB(30,30,30))
            
            DrawingMode(#PB_2DDrawing_Outlined)
                Circle(12, 678, NeuronSize + 1, RGB(0,127,255))            
            DrawingMode(#PB_2DDrawing_Default)
            DrawText(30, 670, "Output Target", RGB(0,127,255) , RGB(30,30,30))
                    
            Circle(12, 698, NeuronSize, RGB(0,127,255))
            DrawText(30, 690, "Bias ", RGB(0,127,255) , RGB(30,30,30))
            Circle(12, 718, NeuronSize, RGB(0,255,0))
            DrawText(30, 710, "Neuron (Green level shows activiation intensity)", RGB(0,255,0) , RGB(30,30,30))
            LineXY(5, 757, 20, 757, RGB(255,0,0))
            LineXY(5, 737, 20, 737, RGB(0,255,0))
            DrawText(30, 730, "Weight (Green level shows Positive Intensity)", RGB(0,255,0) , RGB(30,30,30))
            DrawText(30, 750, "Weight (Red level shows negative Intensity)", RGB(255,0,0) , RGB(30,30,30))
            
            Delay(RefreshDelay)
            
            StopDrawing()     
            SetGadgetState(0, ImageID(0))
        EndIf
    ForEver
    
EndProcedure

;=======================================================================

; Procedure UpdateDisplay()
; 
; 
; EndProcedure

;=======================================================================

Procedure MakeArray(Array TheArray.d(1), CSV.s)
    
    CSVLoop.l
    FieldCount.l
    
    FieldCount = CountString(CSV, ",") + 1 
    ReDim  TheArray(FieldCount)
    
    For CSVLoop = 1 To FieldCount    
        TheArray(CSVLoop) = ValD(StringField(CSV,CSVLoop ,"," ))      
    Next
    
EndProcedure 

;=======================================================================


;=======================================================================

[ebs]

Paul,

Keep the improvements coming - they're great!

I looked at your latest code, and I have a question, a suggestion, and a request.

First, the question - In your neuron display code, there are these lines,
which appear to do the same thing whether J is even or odd. Is this what you intended?

Code: Select all

                If J % 2 = 0
                  HiddenNeuronLoc((i-1) * HiddenColHeight + J)\y = YHiddenOffset - (J * NeuronGap)
                Else
                  HiddenNeuronLoc((i-1) * HiddenColHeight + J)\y = YHiddenOffset + (J * NeuronGap)
                EndIf

Now for the suggestion - In your input and hidden layer loops, you use "ANNOutputWeights(J,i)" many times.
Setting a temporary variable equal to this value at the top of each loop might make the code smaller and/or faster
(I'm not certain, as I haven't done any testing).

Lastly, the request - How about an interface where the user can present a single input character
and see the resulting weights and output value? This would be a way to check the results of the training.

Please feel free to consider or ignore any of this stuff as you see fit!

Thanks,
Eric

[pdwyer]

The code snippet you posted is as intended, not the last sign before the brackets is a + an on and a - in the other. this is so starting from a center position the hidden neurons are stepped away. the output is essentially

1, -2, +3, -4, +5, -6, +7

It's not necessarily the best way to do it but it seems to work.

Temp var would be good but the values of "j" and "i" are always changing as they are in a loop so you'd be reassigning to temp vals all the time.

Yes, this last I'm thinking about. basically, this interface is for training then once the net is trained it can be saved and used somewhere else. Currently I'm only using the "ANNTrainSet" call and not the "ANNQuerySet" which should be added too to test or use the net as it doesn't propergate any changes back.

Currently , I'm working on ways to have the network dynamically set itself up so I can have input files rather than have hard coded inputs in the code. I'm pretty much there so I'll post again soon.

Ideas are very welcome though thanks