Kohonen selforganising maps (Artificial Neural Networks)

[BalrogSoft]

Code updated For 5.20+

Hi, i post some codes of artificial neural networks, but at the moment these examples was based on supervised learning, the ann learn on these examples modifing the weights when the ann have a error, but you need to specify to the ann when its the error produced to learn. But i implemented a non supervised ann called kohonen maps, after see some examples and information about kohonen maps.

A self-organising map is a type of neural network that forms a topographic map of input data where each neuron in the network represents a class of inputs.

On this example exist a space in 2d that can take values between 0.0 and 1.0, the expanding net of points represent the kohonen map, where each node is the 2d position represented for a neuron. The lines between neurons represent the neighbourhood relationships between neurons. The input is represented by a blue cross, and the winner neuron have a red circle.

This map represent a good learning of data:
http://www.balrogsoftware.com/imagenes/kohonen1.png

This map represent a bad learning of data:
http://www.balrogsoftware.com/imagenes/kohonen2.png

And here is the code:

Code: Select all

; Kohonen selforganized map example by Pedro Gil (Balrog Software)

Global num_inputs

num_inputs = 2
width = 10
height = 10

;init_learning_rate.f = 0.1
learning_rate.f = 0.1
learning_rate_decay.f = 0.99

init_update_radius.f = -1
update_radius.f = -1
update_radius_decay.f = 0.999

winner_takes_all.b = #True
wrap_borders.b = #False

Global Dim inputs.f(num_inputs-1)
Global Dim outputs.f(width-1, height-1)
Global Dim weights.f(width-1,height-1,num_inputs-1)
Global Dim w.f(num_inputs-1)

If init_update_radius < 0 : init_update_radius = width / 3 : EndIf
update_radius = init_update_radius

For y=0 To height-1
  For x=0 To width-1
    For w=0 To num_inputs-1
      weights(x, y, w) = 0.25 + ((Random(10)/20) )
    Next w
  Next x
Next y

Procedure.f CalculateEuclideanDistance(x, y)
  diff.f=0
  sum.f=0
  For i = 0 To num_inputs-1
    diff = weights(x,y,i)-inputs(i)
    sum + (diff*diff)
  Next i
  ProcedureReturn Sqr(sum)
EndProcedure


Procedure.f CalculateOutput(x, y)
  ProcedureReturn CalculateEuclideanDistance(x,y)
EndProcedure

Procedure Max(a.f,b.f)
  If a.f>b.f
    ProcedureReturn a.f
  Else
    ProcedureReturn b.f
  EndIf
EndProcedure

Procedure Min(a.f,b.f)
  If a.f>b.f
    ProcedureReturn b.f
  Else
    ProcedureReturn a.f
  EndIf
EndProcedure

CreateImage(0,101, 101)

OpenWindow(0, 100, 100, 121, 121, "Kohonen map", #PB_Window_SystemMenu|#PB_Window_ScreenCentered)

ImageGadget(0, 10, 10, 101, 101, ImageID(0))

Repeat
  WEvent=WindowEvent()
  
  If WEvent=#PB_Event_CloseWindow
    Quit=#True
  EndIf
  
  ; Set input for the artificial neural network
  
  inputs(0)=Random(100)/100
  inputs(1)=Random(100)/100
  
  ; Search the output winner, winner is the most similar to the input
  
  winner_x = 0
  winner_y = 0
  
  outputs(winner_x, winner_y) = calculateOutput(winner_x, winner_y)
  
  best.f = outputs(winner_x, winner_y)
  worst.f = outputs(winner_x, winner_y)
  sum.f = 0.0
  
  For y = 0 To height-1
    For x = 0 To width-1
      outputs(x,y) = calculateOutput(x,y)
      If (outputs(x,y) < best)
        best = outputs(x,y)
        winner_x = x
        winner_y = y
      ElseIf (outputs(x,y) > worst)
        worst = outputs(x,y)
      EndIf
    Next x
  Next y
  
  If winner_takes_all
    For y = 0 To height-1
      For x = 0 To width-1
        outputs(x,y) = 0.0
      Next x
    Next y
    outputs(winner_x,winner_y) = 1.0
  EndIf
  
  ; And update the weights of the ANN
  
  ur = Round(update_radius, 1)
  x1 = winner_x - ur
  y1 = winner_y - ur
  x2 = winner_x + ur
  y2 = winner_y + ur
  
  If wrap_borders=#False
    x1 = max(x1, 0)
    y1 = max(y1, 0)
    x2 = min(x2, width-1)
    y2 = min(y2, height-1)
  EndIf
  
  dx = 0
  dy = 0
  distance.f = 0.0
  
  nx = 0
  ny = 0
  
  For y = y1 To y2
    
    dy = y - winner_y
    ny = (y + height) % height
    
    For x = x1 To x2
      
      dx = Min(Abs(x - winner_x), Abs((x+width) - winner_x))
      nx = (x + width) % width
      
      distance = Sqr(dx*dx + dy*dy)
      
      If distance < update_radius
        For i = 0 To num_inputs-1
          weights(nx,ny,i) + learning_rate * (inputs(i) - weights(nx, ny, i))
        Next i
      EndIf
    Next x
  Next y
  
  learning_rate = 0.1 + ((learning_rate-0.1)*learning_rate_decay)
  update_radius = 1.0 + ((update_radius-1.0)*update_radius_decay)
  
  ; This show kohonen map
  
  CreateImage(0, 101,101)
  StartDrawing(ImageOutput(0))
  
  DrawingMode(0)
  FrontColor(RGB(255,255,255))
  Box(0,0,102,102)
  
  FrontColor(RGB(190,190,190))
  
  For i = 0 To width
    LineXY(i*10, 0, i*10, (height)*10)
  Next i
  
  For j = 0 To height
    LineXY(0, j*10, width*10, j*10)
  Next j
  
  FrontColor(RGB(128,128,128))
  For i = 0 To width-1
    For j = 0 To height-2
      x1 = (weights(i,j,0) * width)*10
      y1 = (weights(i,j,1) * height)*10
      x2 = (weights(i,j+1,0) * width)*10
      y2 = (weights(i,j+1,1) * height)*10
      LineXY(x1, y1, x2, y2)
    Next j
  Next i
  
  For j = 0 To height-1
    For i = 0 To width-2
      x1 = (weights(i,j,0) * width)*10
      y1 = (weights(i,j,1) * height)*10
      x2 = (weights(i+1,j,0) * width)*10
      y2 = (weights(i+1,j,1) * height)*10
      LineXY(x1, y1, x2, y2)
    Next i
  Next j
  
  DrawingMode(4)
  
  FrontColor(RGB(0,0,0))
  For j = 0 To height-1
    For i = 0 To width-1
      x1 = (weights(i,j,0) * width)*10
      y1 = (weights(i,j,1) * height)*10
      Line(x1-1, y1, 3, 0, RGB(32,32,32))
      Line(x1, y1-1, 0, 3, RGB(32,32,32))
      If i=winner_x And j=winner_y
        Circle(x1, y1, 3, RGB(255,0,0))
      EndIf
    Next i
  Next j
  
  FrontColor(RGB(0,0,255))
  x1 = (inputs(0) * width)*10
  y1 = (inputs(1) * height)*10
  LineXY(x1-2, y1-2, x1+2, y1+2)
  LineXY(x1-2, y1+2, x1+2, y1-2)
  
  StopDrawing()
  
  SetGadgetState(0, ImageID(0))
  
  Delay(50)
Until Quit=#True

CloseWindow(0)
End

[Psychophanta]

If you add comments to your code, it should be much more interesting

[dmoc]

...and maybe an example of practical use?

[thefool]

well, dmoc, as he said, the neurons get the data by the BLUE cross. So they learn! If the blue cross hits somewhere, they learn. And bit after bit they learn to make a square by themselves

i dont know any practical examples of how to use it, but it is quite interesting.. Nice example!

[Karbon]

This *type* of logic can be used in any scenario (games) where you want your program to learn to do something on it's own..

[remi_meier]

Really amazing work!
Thanks Balrog! That helps me a lot!

[BalrogSoft]

Hi, here is another example of self color organization, each input is a color component, and it organize the colors on 128 groups, (every group is a neuron output). It have more comments than previous example.

This image is the initial color spread:


This image is the finish color clasification:


And the code:

Code: Select all

; Kohonen selforganized map example by Pedro Gil (Balrog Software)

Global num_inputs

num_inputs = 3
width = 16
height = 16

learning_rate.f = 0.1
learning_rate_decay.f = 0.99

init_update_radius.f = -1
update_radius.f = -1
update_radius_decay.f = 0.999

winner_takes_all.b = #False
wrap_borders.b = #False

Dim inputs.f(num_inputs-1)
Dim outputs.f(width-1, height-1)
Dim weights.f(width-1,height-1,num_inputs-1)

If init_update_radius < 0 : init_update_radius = width / 3 : EndIf
update_radius = init_update_radius

For y=0 To height-1
  For x=0 To width-1
    For w=0 To num_inputs-1
      weights(x, y, w) = 0.25 + ((Random(10)/20) )
    Next w
  Next x
Next y

Procedure.f CalculateEuclideanDistance(x, y)
  diff.f=0
  sum.f=0
  For i = 0 To num_inputs-1
    diff = weights(x,y,i)-inputs(i)
    sum + (diff*diff)
  Next i
  ProcedureReturn Sqr(sum)
EndProcedure


Procedure.f CalculateOutput(x, y)
  ProcedureReturn CalculateEuclideanDistance(x,y)
EndProcedure

Procedure Max(a.f,b.f)
 If a.f>b.f
   ProcedureReturn a.f
 Else
   ProcedureReturn b.f
 EndIf
EndProcedure

Procedure Min(a.f,b.f)
 If a.f>b.f
   ProcedureReturn b.f
 Else
   ProcedureReturn a.f
 EndIf
EndProcedure

CreateImage(0,128, 128)

OpenWindow(0, 100, 100, 148, 148, #PB_Window_SystemMenu|#PB_Window_ScreenCentered, "Kohonen map")
CreateGadgetList(WindowID())
ImageGadget(0, 10, 10, 128, 128, UseImage(0))

Repeat
  WEvent=WindowEvent()
  
  If WEvent=#PB_Event_CloseWindow
    Quit=#True
  EndIf
  
  ; Set input for the artificial neural network, each input is a color component, red, green and blue
  
  inputs(0)=Random(15)/15
  inputs(1)=Random(15)/15
  inputs(2)=Random(15)/15
  
  ; Search the output winner, winner is the most similar to the input
  ; the winner will be the neuron that is most closer to the input
  
  winner_x = 0
  winner_y = 0
  
  outputs(winner_x, winner_y) = calculateOutput(winner_x, winner_y)

  best.f = outputs(winner_x, winner_y)
  worst.f = outputs(winner_x, winner_y)
  sum.f = 0.0

  For y = 0 To height-1
    For x = 0 To width-1
      outputs(x,y) = calculateOutput(x,y)
      If (outputs(x,y) < best)
        best = outputs(x,y)
        winner_x = x
        winner_y = y
      ElseIf (outputs(x,y) > worst)
        worst = outputs(x,y)
      EndIf
    Next x
  Next y
  
  ; Exist two methods to modify weights, a neuron winner take all mode set the winner output
  ; to 1, all other neurons output are set to 0
  
  If winner_takes_all
    For y = 0 To height-1
      For x = 0 To width-1
         outputs(x,y) = 0.0
      Next x
    Next y
    outputs(winner_x,winner_y) = 1.0
  EndIf
  
  ; And update the weights of the ANN
  
  ; Find the coordenates of neightbour neurons
  
  ur = Round(update_radius, 1)
  x1 = winner_x - ur
  y1 = winner_y - ur
  x2 = winner_x + ur
  y2 = winner_y + ur
  
  If wrap_borders=#False
    x1 = max(x1, 0)
    y1 = max(y1, 0)
    x2 = min(x2, width-1)
    y2 = min(y2, height-1)
  EndIf

  dx = 0
  dy = 0
  distance.f = 0.0

  nx = 0
  ny = 0
 
  For y = y1 To y2

    dy = y - winner_y
    ny = (y + height) % height

    For x = x1 To x2

      dx = Min(Abs(x - winner_x), Abs((x+width) - winner_x))
      nx = (x + width) % width
      
      ; Find the distance between the winner neuron, and the actual neighbor neuron 
      
      distance = Sqr(dx*dx + dy*dy)

      If distance < update_radius
        For i = 0 To num_inputs-1
          ; and modify the weight of the neuron
          weights(nx,ny,i) + learning_rate * (inputs(i) - weights(nx, ny, i))
        Next i
      EndIf
    Next x
  Next y
  
  ; Update learning rate and update radius
    
  learning_rate = 0.1 + ((learning_rate-0.1)*learning_rate_decay)
  update_radius = 1.0 + ((update_radius-1.0)*update_radius_decay)

  ; This show kohonen map
  
  CreateImage(0, 128, 128)
  StartDrawing(ImageOutput())
  
  For j = 0 To height-1 
    For i = 0 To width-1

      Box(i*8, j*8, 8, 8, RGB(weights(i,j,0)*255 ,weights(i,j,1)*255 ,weights(i,j,2)*255) )

    Next i
  Next j

  StopDrawing()
  
  SetGadgetState(0, UseImage(0))

  Delay(50)
Until Quit=#True

CloseWindow(0)
End

[thefool]

well it seems that you are pretty much an expert on this, and it seems interesting! Could you explain how your code works? I would be very interested in an explanation!

[kns]

Brief intro,

http://www.cs.bham.ac.uk/resources/cour ... ohonen.htm

Home of Kohonen,

http://www.cis.hut.fi/research/

Software

http://www.cis.hut.fi/research/software.shtml

[thefool]

thx

[kns]

Perhaps stating the obvious...

Neural networks are often used for classification tasks. That is, with data for many things/people in a sample population, the objective is to group those objects. The supervised versions assume that examples exist for which the correct classification is available. The training is so that the network can build an internal representation, in the weights, for later generalization of that knowledge. That is, novel exemplars for which the classification is not known can be identified. Unsupervised learning, on the other hand attempts to group based the proximity of the vectors describing each example without prior knowledge of correct group membership. The added benefit of Kohonen/SOM networks is that it groups and organizes the data in a spatially ordered 2D map. That is, two groups that are close together will be represented in close proximity on the map.

A simple application might be to classify patients suffering from related ailments based on their symptoms plus demographic information. Another medical example would be to use such networks to classify xrays/scans.