Neural Network and pattern recognition

[BalrogSoft]

Hi, i made a little example with one of my sources about Neural Networks, its a pattern recognition program, and learn to detect ellipses, with pixels of noise. To made work the program, you must to teach to the neural network what is a ellipse, the program shows a graphic form by 0 and 1, it says what thinks neural network, a ellipse or a box, and you must to put "y"(es) or "n"(o) if it is a ellipse, not if the neural network gives a correct output, push "y" if its a ellipse or "n" if it is a box, if you see that neural network is getting correct outputs, you can push a "i"(terate), this will work without modify the weights, and test if the weights are correct for this problem(saving this weights (Dim W(MaxNeuralNetNumber, MaxInputNumber)), we can use to solve this problem without teaching the neural network.

Code: Select all

; code edited, see last post

[dell_jockey]

Pedro, is this back-propagation at work here?

[BalrogSoft]

Code updated for 5.20+

Hi, this example use the simple perceptron network, is the most simple to implement and probably is faster than more complex neural networks, i got teaching correctly the network a error between 6% and 8%, and this is a very good percent of correct detection about 94% and 92%. If you dont use neural networks to solve this problems, you will need to write dozens of lines to made the same work with neural networks that need a little number of code lines. Obviously, a back propagation algorithm will detect better this type of graphics, but at the moment i dont implement it. This is a new version with a window interface, and graphics to show information, use "Teach" to teach one time to neural network, or use "Teach X" to teach a defined number by "X:", you can push the big image button to change the information:

Code: Select all

; Neural network Pattern recognition example by Pedro Gil (Balrog Software)
; First we initialize the variables
; Edited to solve some bugs and make better graphics
Enumeration
#Window_0
EndEnumeration

;- Gadget Constants
;
Enumeration
#Pattern
#Frame3D_0
#Teach
#TeachX
#Text_0
#XNb
#Recognise
#Information
EndEnumeration

CreateImage(2,66,66)
CreateImage(3,220,170)

Procedure Open_Window_0()
  If OpenWindow(#Window_0, 0, 0, 315, 205, "Neural Network", #PB_Window_SystemMenu | #PB_Window_TitleBar | #PB_Window_ScreenCentered)
    ImageGadget(#Pattern, 10, 20, 66, 66, ImageID(2))
    FrameGadget(#Frame3D_0, 5, 5, 305, 195, "Neural Network Pattern Recognition")
    ButtonGadget(#Teach, 10, 95, 70, 20, "Teach")
    ButtonGadget(#TeachX, 10, 120, 70, 20, "Teach X")
    TextGadget(#Text_0, 15, 150, 15, 15, "X:")
    StringGadget(#XNb, 35, 145, 45, 20, "", #PB_String_Numeric)
    ButtonGadget(#Recognise, 10, 170, 70, 20, "Recognise")
    ButtonImageGadget(#Information, 85, 20, 220, 175, ImageID(3))
  EndIf
EndProcedure

MaxNeuralNetNumber = 1
MaxInputNumber = 256
InputNumbers = 256
InputComb = Pow(MaxInputNumber, 2)
Dim Image.b(16, 16)

Dim W.f(MaxNeuralNetNumber, MaxInputNumber); this is the weights
Dim Net(MaxNeuralNetNumber); this is the output for the neurals network
Dim Output(MaxNeuralNetNumber); if the Net is greater than 0 the output gets 1
Dim Inputs(MaxNeuralNetNumber, MaxInputNumber); this set the inputs for the neural networks
Dim InputImportance.s(MaxNeuralNetNumber, Inputcomb); input importance is for the supervised learning
Dim Error.f(50000)
Dim Correct.f(50000)

; and say to the neural network that this input
; must gets an output

W(1, 0)=1 ; we set a first values for neural networks
For Weights = 1 To InputNumbers ; exist also W(1,0) but this line is only to make more easy
  W(1, Weights)=Random(5)-3  ; calculations
Next Weights
Open_Window_0()

Gosub MakeInfo

Info=0
Repeat
  WEvent=WindowEvent()
  If WEvent=#PB_Event_CloseWindow
    Quit=1
  ElseIf WEvent=#PB_Event_Gadget
    GEvent=EventGadget()
    If GEvent=#Teach
      TeachRep=1
      Teach=#True
      Gosub Recognise
      Teach=#False
    ElseIf GEvent=#Recognise
      TeachRep=1
      Gosub Recognise
    ElseIf GEvent=#TeachX
      TeachRep=Val(GetGadgetText(#XNb))
      Teach=#True
      Gosub Recognise
      Teach=#False
     
    ElseIf GEvent=#Information
      Info+1
      If Info=3
        Info=0
      EndIf
      Gosub MakeInfo
    EndIf
  EndIf
 
Until Quit=1
End

Recognise:
Ta.f = 0.9 ; Ta.f is the learning constant, and must be between 0 and 1
; you can think that when this value was higher would be
; better, because you can think that it can learn more faster
; but usually not, a high value can make that weights of the
; neural networks change very faster, and dont get the
; optimal value.

For Rep = 1 To TeachRep

  ; Create a 16x16 image with a random figure (box or ellipse), with noise
  CreateImage(1, 16, 16)
  If StartDrawing(ImageOutput(1))
    FrontColor(RGB(255, 255, 255))
    BackColor(RGB(0, 0, 0))
    DrawingMode(4)
    Fig = Random(1)
    If Fig = 1
      Ellipse(7 + Random(2), 7 + Random(2), 5 + Random(2), 5 + Random(2))
    ElseIf Fig=0
      Box(1 + Random(3), 1 + Random(3), 9 + Random(3), 9+ Random(3))
   ; Else
   ;   For d=0 To Random(1)
   ;     LineXY(Random(16),Random(16),Random(16),Random(16))
   ;   Next
    EndIf
    For t = 0 To Random(16*2)
      If Random(1)
        FrontColor(RGB(255, 255, 255))
      Else
        FrontColor(RGB(0, 0, 0))
      EndIf
      Plot(Random(15), Random(15))
    Next t
   
    ; Set inputs of Neural Network with each pixel of image
    cnt = 0
    For y = 0 To 15
      For x = 0 To 15
        c = Point(x, y)
        If c = 0
          Image(x, y)=0
        Else
          Image(x, y)=1
        EndIf
        cnt + 1
        Inputs(1, cnt)=Image(x, y)
      Next x
    Next y
    StopDrawing()
  EndIf
 
  CreateImage(2,66,66)
  If StartDrawing(ImageOutput(2))
    DrawImage(ImageID(1),1,1,64,64)
    StopDrawing()
  EndIf
 
  SetGadgetState(#Pattern, ImageID(2))
 
  Net(1)=W(1, 0) ; and we get an output for the net
  For In = 1 To InputNumbers
    Net(1)+(Inputs(1, In)*W(1, In))
  Next In
 
  ; **
  If Net(1)=>0 ; and if Net its greater than 0 we get an output
    Output(1)=#True
  Else
    Output(1)=#False
  EndIf
 
  If Fig=1;) Or (Output(1)=0 And Fig<>1)
    InputImportance=1
  Else
    InputImportance=0
  EndIf
 
  If Output(1)=1
    Figure.s = "Ellipse"
  Else
    Figure.s = "Non-Ellipse"
  EndIf
 
 
  ; execute to modify weights, if you want to iterate without teach to neural network this code is not executed
  If Teach=#True
    Increment = InputImportance - Output(1) ; get the difference of the output of the net and the correct output
    For In = 1 To InputNumbers
      W(1, In)=W(1, In)+Ta.f * Increment * Inputs(1, In) ; and get the new weights
    Next In
    W(1, 0)=W(1, 0)+Ta.f * Increment
  EndIf
  If (Output(1)=1 And Fig=1) Or (Output(1)=0 And Fig<>1)
    Correct + 1
  Else
    Wrong + 1
  EndIf
  AllPatterns + 1
  Error(AllPatterns)=100*Wrong/AllPatterns
 
  Correct(AllPatterns)=100*Correct/AllPatterns
 
  Gosub MakeInfo
 
Next Rep
Return

MakeInfo:
If Info=0
  CreateImage(3,220,170)
  If StartDrawing(ImageOutput(3))
    a.f=0
    For x=0 To 255
      cl=48*(W(1,x))+127
      If cl>255
        cl=255
      ElseIf cl<0
        cl=0
      EndIf
     
      LineXY(4+(x/1.2),96-(Sin(a.f)*64),110,140, RGB(cl,0,0))
      a.f+(3.141592/256)
    Next x
    Circle(110,140,4, RGB(0,200,0))
    
    DrawingMode(4)
    Circle(110,140,5, RGB(0,100,0))
    DrawingMode(1)
   
    DrawText(110-TextWidth(Figure)/2,150, Figure, RGB(200,200,200))
    DrawText(110-TextWidth("Input weights")/2,10, "Input weights", RGB(0,200,64))
    StopDrawing()
  EndIf

  SetGadgetAttribute(#Information, #PB_Button_Image, ImageID(3))
ElseIf Info=1
  CreateImage(4,220,170)
  If StartDrawing(ImageOutput(4))
    FrontColor(RGB(200,200,200))
   
    Line(9,59,0,102)
    Line(7,59,5,0)
    Line(7,161,5,0)
    For x=1 To 201
      pos=Round(x*AllPatterns/200,1)
      If Error(Pos)>=0
        Plot(9+x,160-(Error(pos)),RGB(255,0,0))
      EndIf
      If Correct(Pos)>=0
        Plot(9+x,160-(Correct(pos)),RGB(0,255,0))
      EndIf
    Next x
   
    DrawingMode(1)
    DrawText(4,44, "100 %")
    FrontColor(RGB(0,255,0))
    DrawText(110-TextWidth("Correct: "+Str(Correct)+" ("+Str(Round(Correct(AllPatterns),0))+"%)")/2,2, "Correct: "+Str(Correct)+" ("+Str(Round(Correct(AllPatterns),0))+"%)")
    
    FrontColor(RGB(255,0,0))
    DrawText(110-TextWidth("Wrong: "+Str(Wrong)+" ("+Str(100-Round(Correct(AllPatterns),0))+"%)")/2,22, "Wrong: "+Str(Wrong)+" ("+Str(100-Round(Correct(AllPatterns),0))+"%)")
    StopDrawing()
  EndIf
  
  SetGadgetAttribute(#Information, #PB_Button_Image, ImageID(4))
ElseIf Info=2
  CreateImage(5,220,170)
  If StartDrawing(ImageOutput(5))
    FrontColor(RGB(200,200,200))
  
    in=0
    For y=0 To 15
      For x=0 To 15
        cl=48*(W(1,in))+127
        in+1
        If cl>255
          cl=255
        ElseIf cl<0
          cl=0
        EndIf
        Box(46+x*8,32+y*8,8,8,RGB(cl,0,0))
      Next x
    Next y
   
    DrawingMode(1)
    FrontColor(RGB(0,255,0))
    DrawText(110-TextWidth("Weights")/2,8, "Weights")
  
    StopDrawing()
  EndIf
  
  SetGadgetAttribute(#Information, #PB_Button_Image, ImageID(5))
EndIf
Return

[dell_jockey]

Pedro, nice example!

FYI: a book I keep referring to rather often is (a German translation of) 'The Essence of Neural Networks', written by Robert Callan, published by Prentice Hall Europe (1999) by an arrangement with Pearson Education Ltd. It doesn't contain code, but describes contemporary developments rather well.

[StarHawk]

That's not a neural network. To have a neural network, you must have a neural microprocessor.

A Neural Network is an interconnected assembly of simple processing elements, units or nodes, whose functionality is loosely based on the animal neuron. The processing ability of the network is stored in the inter-unit connection strengths, or weights, obtained by a process of adaptation to, or learning from, a set of training patterns.

This is very different from the processing done by conventional computers and their chipsets.

In essence what you are doing is trying to mimic the behavior of a neural chip through a conventional computer which amounts to some interesting processes, but in the end is still a conventional computer because of the chips.

What you are creating is classified as an "Artificial Neural Program" aka "Artificial Neural Networks"... the keyword being "Artificial". It could also be classified as "neural-network simulation software running on a conventional single-microprocessor computer"

If you want the real deal, then you need to purchase additional neural chips. Here are some organizations that you may be able to purchase such chips through.

http://www.kip.uni-heidelberg.de/vision ... hagen.html

http://www.cs.ucl.ac.uk/staff/D.Gorse/r ... /pRAM.html

http://dibe.unige.it/department/micro/A ... cessor.htm

http://polimage.polito.it/groups/chip/chip.html

[Blade]

Hi, can you explain how to test your program?
If there is an ellipese and the prog says non-ellipse, what should I do?
I expected to find a "true" and "false" button to teach the program...
Can't understand the "teach" and "teach x" buttons...

tnx...

[dell_jockey]

Starhawk,

can you please explain what is not artificial about the 'neural chips' you're referring to? Don't tell me they come with carbon-hydrate neurons and all the (biological) stuff to interconnect them.
So, what's the diff, except that those purpose built chips are bound to be faster at their specialized job?

[BalrogSoft]

Hi StarHawk, you can call as you want, you probably are right that really are called Artificial Neural Network, but on dozens of websites, this is know as Neural Network, on the magazine about programming where i found information about this, was called also Neural Networks, and i dont want to buy chips, this is only an example, here you will find websites where this type of Artificial Neural Networks are called Neural Networks:

http://uhavax.hartford.edu/compsci/neur ... torial.htm
http://www.pmsi.fr/neurin2a.htm

And here another site that says "Neural networks or artificial neural networks (ANN), are densely interconnected networks of simple computational elements":
http://dms.irb.hr/tutorial/tut_nnets_short.php

Hi Blade, the program automaticly teach to the artificial neural network(ANN), when you push Teach or Teach X, you can view how ANN is learning with stadistics, and use recognise when the error percent is lower than 10%.

You can download a new version totally configurable on the main window
and can detect numbers, this is a screenshot:


You can download here: http://www.balrogsoftware.com/downloads ... attern.zip
I will explain some buttons:

One iteration/X iterations: This is to make one iteration or a defined X number of iterations for teach or recognise.

Load / Save weights: Load or save weights when ANN learn the problem.

Information Frame: Contains a image button that shows differents informations about ANN, Input weights contains a diagram of an ANN, with the output of the current teach or recognise, Weights show the weights orderer like the pattern image given to the ANN and show the importance of every input weight, General stadistics show a general stadistic of wrong and corrects outputs generated by the ANN, and Stadistics by figure, shows the error and corrects inputs generated by ANN.

Reset Stadistics/Weights: Reset stadistics or weights to start a new recognition problem, i recommend reset stadistics when you will get a error percent lower than 10%, and know the real percent on this moment, because when you are teaching to ANN, the first times ANN have a lot of errors that are used to make the stadistics.

Learning constant: between 0 and 1, and represents how fast learn the ANN, see the code for more information.

Output Function: exist two functions to generate an output, i think that sigma funciton works better.

Patterns options Frame:
Type: Recognise numbers or figures.
Figure Number: Total number of figures to be recognised.
Figure to recognise: Figure that you want to make recognise.
Add noise: add noise pixels to pattern image, sometimes works better teach to ANN without noise, and recognise with noise.
Show: Show only the stadistics by figure of the figure selected to be recognised.

[Dare2]

Pedro, props to you. You are too clever by half.

This may be slightly off topic, but just out of curiosity, why would a neural network be more effective in predictive or analyical apps (like determing a good credit risk, for eg) than something more traditional? Predictive seems to be the bias, from the little I've read (and I've understood less!)

I would have thought that their usage was more useful things like security systems with fingerprint or retinal image detection, etc, and the slight variations that can occur there.

[BalrogSoft]

Hi Dare2, i read something before about use ANNs for predictive problems, but this really is a pattern recognition, like this example but using other type of intputs, the ANNs are teached to find patterns of previous behavior on economy markets, and used to know when a determinate market can down or up, this is the only one thing that i can say about use ANNs as predictive systems.

[NoahPhense]

I downloaded a neural network client. I can't move now, blood is dripping
from my eye sockets, and this RS232 cable is really starting to irritate my
sphincter?!

- np

8O

[dell_jockey]

BalrogSoft,

ANN can be used for predictive stuff. It's been done in electronics, the stock market, etc.

Imagine using a couple of time series as an input to train your network. The network should predict stock prices in, say, five days. What you do is, at each learning cycle, to take the input series and take the values that are 5 samples in the past to predict the value to the current output sample.
So, at each sample from the input time series, you train the network to predict the output value five samples into the future. If there's some relation at all between the input series and the output series (five samples into the future) the network will pick it up an be trained accordingly.

This type of ANN is still matching patterns, only that you're matching current set of patterns to what will likely happen sometime in the (very...) near future...

[TronDoc]

A Neural Network is...

...giving me a headache right now

[Dare2]

[StarHawk]

I understand that some websites and a few publications omit the word "Artificial". At MIT in the "Things that Think" lab... the instructor lets the "bottom up", "top down" schools argue it out as part of the learning process. But after that 2 week period, the instructor issues a challenge to those who think that can similate with a conventional computer, any bottom up approach. The test is "Put up or Shutup".

The challenge is simple yet complex. Create a program that similates thinking using NO "IF" statements whatsoever. Of course some sissy cry babies drop the class and setup websites claiming they have created neural networks or neural programs on conventional computers, of course using "IF" statements....nerds.

In any event, the rest of the class moves on into fascinating (and some classified) uses of neural net chips in various devices.

What interests most nowdays is a hybrid approach to the "top down' versus "bottom up" approaches. That is the "learning" takes place on the neural chips and is fed to conventional chips to carry out certain slave "non-learning" tasks. The problem many of us have with this is that the neural chips themselves exhibit strange behaviors that some with degrees in both computers and psychology liken to "dreaming". When the chip is given time to "sleep", the abnormalities vanish. While I'm not certain that neural nets need time to sleep, they clearly need downtime while conventional computers just keep chugging right along. And, if "dreaming" behaviors aren't bad enough, certain chips formed large holes in their lattices where weightings were thrown completely off as all signals vanished into the hole. Again, some have theorized that the chip formed an "addiction". Much like in real life, when someone is addicted a hole is formed in which all areas of their life are impacted as things flow into that hole to keep the addiction going.