Page 1 of 1

Matrix Inversion

Posted: Sat Jun 16, 2018 11:54 am
by doctornash
Needed to invert some large matrices, so rolled me some PB to do so, based on the method presented in this paper:
https://arxiv.org/ftp/arxiv/papers/1304/1304.6893.pdf
I've only verified the results for <= 16x16 matrices so far (and it yielded the right answer in each case).
In the code, I use the same 4x4 source matrix example as in the paper, and although the inverse result is correct here, it is actually NOT correct in the paper :shock:
Anyway, there's not a shred of 'advanced PB' in the code, in fact it is all pretty much 'entry level' - so hopefully very easy to understand, but it sacrifices efficiency and there's plenty of opportunity for optimization...

Code: Select all

Declare UpdateNumbersArray(pVal.l, kVal.l)
Declare DeterminePK()

Global NewList N.s() ;Dictionary List of columns
Global NewList B.s() ;Dictionary List of rows

Global Dim A.d(4,4) ;The 'Numbers Array' ie Source Matrix to be inverted. Change dimensions for different size matrix
Global Dim InvArr.d(4,4) ;Holds inverse of the Source Matrix. Change dimensions to that of Numbers Array, if different size matrix

;Change number of elements in B and N according to size of Numbers Array:
AddElement(B()) : B() = "y;1"
AddElement(B()) : B() = "y;2"
AddElement(B()) : B() = "y;3"
AddElement(B()) : B() = "y;4"

AddElement(N()) : N() = "x;1"
AddElement(N()) : N() = "x;2"
AddElement(N()) : N() = "x;3"
AddElement(N()) : N() = "x;4"

A(1,1) = 2
A(1,2) = 5
A(1,3) = 3
A(1,4) = 2

A(2,1) = 4
A(2,2) = 10
A(2,3) = 1
A(2,4) = 7

A(3,1) = 1
A(3,2) = 5
A(3,3) = 2
A(3,4) = 2

A(4,1) = 2
A(4,2) = 1
A(4,3) = 2
A(4,4) = 1


;Method for inverting a matrix per paper "A space efficient flexible pivot selection approach to evaluate determinant and inverse of a matrix":

;First determine p (row val) and k (column val):
;1. iterate through the B list until you find the first letter a 'y'. Make the position of p the value after the colon
;2. look in matrix of numbers at position (p,p). If it is non zero, then look at the entry in N at position p, and make k the number after the colon.
;If the value is zero, then keep iterating to the right: (p,p+n) until a non zero is found. then, look at the entry in N at position p+n, and make k the number after the colon
;Now that the p and k values are known:
;3. Refresh the values in lists B and N according to following formula: B = B + [xk] - [yp] and N = N - [xk] + [yp]. Add the entry at the same position as the entry removed
;4. For all values in the Numbers matrix other than those in row p and column k, calculate new values as follows:
;mi = a(i,k)/a(p,k) and then: a(i,j) = a(i,j) + a(p,j)*mi


Procedure DeterminePK()
  
p.l ;row position of pivot point
k.l ;column position of pivot point
  
ResetList(B())
ResetList(N())

;Find position of p:
Pos = 0
While NextElement(B())
  Pos = Pos + 1
  If StringField(B(),1,";") <> "x"
    Break
  EndIf  
Wend  
p = Pos

;Find position of k by finding first non-zero value from Numbers Array:
For t = Pos To ArraySize(A(),2)
  If A(Pos,t) <> 0
    Break
  EndIf  
Next 

SelectElement(N(), t-1) ;we do t-1 because for 'SelectElement', element numbering begins at 0. So Pos=1 is actually 0 for SelectElement
k = Val(StringField(N(),2,";"))

;Now we refresh the values in lists B and N, as per rule 3:

ResetList(B())
While NextElement(B())
  If StringField(B(),1,";") = "y" And Val(StringField(B(),2,";")) = p
    B() = "x" + ";" + Str(k)
    Break
  EndIf  
Wend

ResetList(N())
While NextElement(N())
  If StringField(N(),1,";") = "x" And Val(StringField(N(),2,";")) = k
    N() = "y" + ";" + Str(p)
    Break
  EndIf  
Wend

UpdateNumbersArray(p,k)
  
EndProcedure  


Procedure UpdateNumbersArray(pVal.l, kVal.l)
;The pivot point is (p,k) ie (pVal,kVal)
;Divide pivot row by value at pivot point, and pivot column by -ve of value at pivot point. Set the value at pivot point as 1 divided by the value at pivot point: 
  
  For t = 1 To ArraySize(A(),2)
    If t<>kVal 
      A(pVal,t) = A(pVal,t)/A(pVal,kVal)
    EndIf  
  Next
  
  For t = 1 To ArraySize(A(),1)
    If t<>pVal
      A(t,kVal) = -1*A(t,kVal)/A(pVal,kVal)
    EndIf  
  Next
  
  A(pVal,kVal) = 1/A(pVal,kVal)
  
  ;Now calculate new values for the Numbers Array as per rule 4:
  
  For t = 1 To ArraySize(A(),1)
    For v = 1 To ArraySize(A(),2)
      If t<>pVal And v<>kVal
        m.d = A(t,kVal)/A(pVal,kVal)
        A(t,v) = A(t,v) + (A(pVal,v)*m)
      EndIf  
    Next
  Next
   
EndProcedure  

;If the matrix has an inverse, the number of pivot elements would be equal to the order of the matrix. So, we call DeterminePK() as many times as the order of the matrix.
;The order of the matrix is the number of rows or columns of the matrix (doesn't matter which, as it has to be square to have an inverse):
If ArraySize(A(),1) = ArraySize(A(),2)
  For t = 1 To ArraySize(A(),1)
  DeterminePK()
  Next
EndIf

;Re-arrange the values of A() into InvArr according to the row and column sequence indicated by the dictionary lists B() and N():
For t = 1 To ArraySize(A(),1)
  SelectElement(B(), t-1)
  RowVal = Val(StringField(B(),2,";"))
  For v = 1 To ArraySize(A(),2)
  SelectElement(N(), v-1)
  ColVal = Val(StringField(N(),2,";"))
  InvArr(RowVal,ColVal) = A(t,v)  
  Next
Next

;Print out the values in InvArr. This is the inverse of Matrix A():
For t = 1 To ArraySize(InvArr(),1)
  For v = 1 To ArraySize(InvArr(),2)
    Debug StrD(InvArr(t,v),4) ;Change 4 to however many decimal places you require in the result 
  Next
Next

Re: Matrix Inversion

Posted: Sat Jun 16, 2018 7:29 pm
by bbanelli
Is it only me or this seems as an ideal candidate for AVX optimization? Wilbert or other ASM gurus??? :)

Good job doctornash, BTW!

Re: Matrix Inversion

Posted: Sat Jun 16, 2018 9:11 pm
by davido
@doctornash,

Excellent. :D
Thank you for sharing.

Re: Matrix Inversion

Posted: Sun Jun 17, 2018 2:35 pm
by wilbert
bbanelli wrote:Is it only me or this seems as an ideal candidate for AVX optimization?
I dont't see it :shock:

Like doctornash already mentioned, there's already a lot of room for optimization.
I'm thinking about not using strings, using arrays instead of lists and less division operations.

Re: Matrix Inversion

Posted: Sun Jun 17, 2018 5:26 pm
by wilbert
A bit more optimized PB code.
I haven't compared the speed and it may need more testing for bugs :?

Code removed.
See post below for updated code
viewtopic.php?p=523704#p523704


Maybe a stupid question but I'm not very familiar with matrix math.
What is supposed to be the inverse matrix if an entire row or column is 0 ?

Re: Matrix Inversion

Posted: Sun Jun 17, 2018 10:13 pm
by doctornash
What is supposed to be the inverse matrix if an entire row or column is 0 ?
If the entire row or column of a square matrix is 0, then the determinant of the matrix is 0, and if the determinant is 0, the matrix is not invertible. Such a matrix (whose inverse does not exist), is said to be 'singular'.
it may need more testing for bugs
Wow that's some compact code right there! :shock: Am getting IMA at line 10 'determine p' with PB4.60 though.

Re: Matrix Inversion

Posted: Mon Jun 18, 2018 5:43 am
by wilbert
doctornash wrote:Wow that's some compact code right there! :shock: Am getting IMA at line 10 'determine p' with PB4.60 though.
There are some bugs in PB4.60 related to multidimensional arrays and swapping values. :(
Please try again with the updated code.

Re: Matrix Inversion

Posted: Tue Jun 19, 2018 9:52 am
by wilbert
ASM optimized versions

Edit:
Updated code July 6, 2018

Code: Select all

Procedure InverseMatrixD(*Matrix, *InvMatrix, n)                ; n x n matrix inverse
  Protected.l n1=n-1,p,k,t,v
  Protected.d r,m,*s.Double,*d.Double
  Protected Dim Mat.d(n1,n1)
  Protected Dim Piv.u(n1)
  CopyMemory(*Matrix, @Mat(), n*n<<3)                           ; << copy *Matrix to Mat() >>
  For t=0 To n1 : Piv(t)=t : Next                               ; << init Piv() >>
  For p=0 To n1 : m=0
    For t=p To n1                                               ; << find highest absolute value >>
      r=Abs(Mat(p,Piv(t))) : If r>m : m=r : v=t : EndIf
    Next
    If m=0 : ProcedureReturn 0 : EndIf                          ; << exit if all 0 >>
    Swap Piv(p),Piv(v) : k=Piv(p) : r=1/Mat(p,k) : Mat(p,k)=1   ; << get k and reciprocal for Mat(p,k) >>
    *d=@Mat(p,0) : For t=0 To n1 : *d\d*r : *d+8 : Next         ; << multiply pivot row by reciprocal >>
    *d=@Mat() 
    For t=0 To n1
      If t<>p
        ; << calculate new Mat() values >>
        *s=@Mat(p,0) : m=-Mat(t,k) : Mat(t,k)=0
        !mov ecx, [p.v_n]
        CompilerIf #PB_Compiler_Processor = #PB_Processor_x64
          !mov rax, [p.p_s]
          !mov rdx, [p.p_d]
          !movsd xmm5, [p.v_m]
          !movlhps xmm5, xmm5
          !test ecx, 1
          !jz .l0
          !movsd xmm0, [rax]
          !movsd xmm1, [rdx]
          !mulsd xmm0, xmm5
          !addsd xmm0, xmm1
          !movsd [rdx], xmm0
          !add rax, 8
          !add rdx, 8
          !sub ecx, 1
          !jz .l1
          !.l0:
          !movupd xmm0, [rax]
          !movupd xmm1, [rdx]
          !mulpd xmm0, xmm5
          !addpd xmm0, xmm1
          !movupd [rdx], xmm0
          !add rax, 16
          !add rdx, 16
          !sub ecx, 2
          !jnz .l0
          !.l1:
          !mov [p.p_d], rdx
        CompilerElse
          !mov eax, [p.p_s]
          !mov edx, [p.p_d]
          !fld qword [p.v_m]
          !.l0:
          !fld qword [eax]
          !fmul st0, st1
          !fadd qword [edx]
          !fstp qword [edx]
          !add eax, 8
          !add edx, 8
          !sub ecx, 1
          !jnz .l0
          !fstp st0
          !mov [p.p_d], edx
        CompilerEndIf
      Else
        *d+n<<3
      EndIf
    Next
  Next
  For t=0 To n1 : *d=*InvMatrix+Piv(t)*n<<3
    For v=0 To n1 : *d\d=Mat(t,Piv(v)) : *d+8 : Next            ; << set *InvMatrix >> 
  Next
  ProcedureReturn #True
EndProcedure

Procedure InverseMatrixF(*Matrix, *InvMatrix, n)                ; n x n matrix inverse
  Protected.l n1=n-1,p,k,t,v
  Protected.d r,m,*s.Double,*d.Double
  Protected *f.Float=*Matrix
  Protected Dim Mat.d(n1,n1)
  Protected Dim Piv.u(n1)
  *d=@Mat()
  For t=0 To n1
    For v=0 To n1 : *d\d=*f\f : *d+8 : *f+4 : Next              ; << copy *Matrix to Mat() >>
  Next  
  For t=0 To n1 : Piv(t)=t : Next                               ; << init Piv() >>
  For p=0 To n1 : m=0
    For t=p To n1                                               ; << find highest absolute value >>
      r=Abs(Mat(p,Piv(t))) : If r>m : m=r : v=t : EndIf
    Next
    If m=0 : ProcedureReturn 0 : EndIf                          ; << exit if all 0 >>
    Swap Piv(p),Piv(v) : k=Piv(p) : r=1/Mat(p,k) : Mat(p,k)=1   ; << get k and reciprocal for Mat(p,k) >>
    *d=@Mat(p,0) : For t=0 To n1 : *d\d*r : *d+8 : Next         ; << multiply pivot row by reciprocal >>
    *d=@Mat() 
    For t=0 To n1
      If t<>p
        ; << calculate new Mat() values >>
        *s=@Mat(p,0) : m=-Mat(t,k) : Mat(t,k)=0
        !mov ecx, [p.v_n]
        CompilerIf #PB_Compiler_Processor = #PB_Processor_x64
          !mov rax, [p.p_s]
          !mov rdx, [p.p_d]
          !movsd xmm5, [p.v_m]
          !movlhps xmm5, xmm5
          !test ecx, 1
          !jz .l0
          !movsd xmm0, [rax]
          !movsd xmm1, [rdx]
          !mulsd xmm0, xmm5
          !addsd xmm0, xmm1
          !movsd [rdx], xmm0
          !add rax, 8
          !add rdx, 8
          !sub ecx, 1
          !jz .l1
          !.l0:
          !movupd xmm0, [rax]
          !movupd xmm1, [rdx]
          !mulpd xmm0, xmm5
          !addpd xmm0, xmm1
          !movupd [rdx], xmm0
          !add rax, 16
          !add rdx, 16
          !sub ecx, 2
          !jnz .l0
          !.l1:
          !mov [p.p_d], rdx
        CompilerElse
          !mov eax, [p.p_s]
          !mov edx, [p.p_d]
          !fld qword [p.v_m]
          !.l0:
          !fld qword [eax]
          !fmul st0, st1
          !fadd qword [edx]
          !fstp qword [edx]
          !add eax, 8
          !add edx, 8
          !sub ecx, 1
          !jnz .l0
          !fstp st0
          !mov [p.p_d], edx
        CompilerEndIf
      Else
        *d+n<<3
      EndIf
    Next
  Next
  For t=0 To n1 : *f=*InvMatrix+Piv(t)*n<<2
    For v=0 To n1 : *f\f=Mat(t,Piv(v)) : *f+4 : Next            ; << set *InvMatrix >> 
  Next
  ProcedureReturn #True
EndProcedure

Procedure InverseMatrixD_InPlace(*Matrix, n, PivotMethod=1, PivotTolerance.d=1e-9)
  ; +------------------------+
  ; |  n x n matrix inverse  |
  ; |  ====================  |
  ; |  method 0 = non zero   |
  ; |  method 1 = max abs    |
  ; +------------------------+
  Protected.l n1=n-1,rb=n<<3,p,t,v
  Protected.d r,m,*s.Double,*d.Double
  Protected Dim Piv.u(n1)
  For t=0 To n1 : Piv(t)=t : Next                               ; << init Piv() >>
  For p=0 To n1
    *s=*Matrix+p*rb+p<<3 : m=PivotTolerance
    For t=p To n1
      r=Abs(*s\d) : *s+8                                        ; << find pivot >>
      If r>m
        m=r : v=t
        If PivotMethod=0 : Break : EndIf
      EndIf          
    Next
    If m=PivotTolerance : ProcedureReturn 0 : EndIf             ; << exit If all 0 >>
    If p<>v
      *s=*Matrix+p<<3 : *d=*Matrix+v<<3
      For t=0 To n1 : Swap *d\d,*s\d : *s+rb : *d+rb : Next     ; << swap column p and v >>
      Swap Piv(p),Piv(v)
    EndIf
    *s=*Matrix+p*rb+p<<3 : r=1/*s\d : *s\d=1                    ; << get reciprocal for Mat(p,p) >>
    *d=*Matrix+p*rb : For t=0 To n1 : *d\d*r : *d+8 : Next      ; << multiply pivot row by reciprocal >>
    *d=*Matrix
    For t=0 To n1
      If t<>p
        ; << calculate new Mat() values >>
        *s=*Matrix+t*rb+p<<3 : m=-*s\d : *s\d=0
        *s=*Matrix+p*rb
        !mov ecx, [p.v_n]
        CompilerIf #PB_Compiler_Processor = #PB_Processor_x64
          !mov rax, [p.p_s]
          !mov rdx, [p.p_d]
          !movsd xmm5, [p.v_m]
          !movlhps xmm5, xmm5
          !test ecx, 1
          !jz .l0
          !movsd xmm0, [rax]
          !movsd xmm1, [rdx]
          !mulsd xmm0, xmm5
          !addsd xmm0, xmm1
          !movsd [rdx], xmm0
          !add rax, 8
          !add rdx, 8
          !sub ecx, 1
          !jz .l1
          !.l0:
          !movupd xmm0, [rax]
          !movupd xmm1, [rdx]
          !mulpd xmm0, xmm5
          !addpd xmm0, xmm1
          !movupd [rdx], xmm0
          !add rax, 16
          !add rdx, 16
          !sub ecx, 2
          !jnz .l0
          !.l1:
          !mov [p.p_d], rdx
        CompilerElse
          !mov eax, [p.p_s]
          !mov edx, [p.p_d]
          !fld qword [p.v_m]
          !.l0:
          !fld qword [eax]
          !fmul st0, st1
          !fadd qword [edx]
          !fstp qword [edx]
          !add eax, 8
          !add edx, 8
          !sub ecx, 1
          !jnz .l0
          !fstp st0
          !mov [p.p_d], edx
        CompilerEndIf
      Else
        *d+n<<3
      EndIf
    Next
  Next
  For t=0 To n1
    p=Piv(t)
    While t<>p
      *s=*Matrix+t*rb : *d=*Matrix+p*rb
      For v=0 To n1 : Swap *d\d,*s\d : *s+8 : *d+8 : Next       ; << swap row t and p >>
      v=Piv(p) : Piv(p)=p : p=v
    Wend
  Next
  ProcedureReturn #True
EndProcedure

Re: Matrix Inversion

Posted: Sat Jun 23, 2018 7:10 am
by doctornash
Thanks Wilbert for the effort - your ASM optimized code's been running fine and producing the right results :mrgreen:

Re: Matrix Inversion

Posted: Sat Jun 23, 2018 4:27 pm
by wilbert
doctornash wrote:Thanks Wilbert for the effort - your ASM optimized code's been running fine and producing the right results :mrgreen:
I did find a problem with both your original code and my implementation so I updated my code.
When you did an inversion of the inverted matrix you would expect to get the original matrix back.
If you repeated this step multiple times, it got values close to zero instead of zero making the selection of k wrong.

Here's a more accurate version using a different way to find k.
Instead of searching for the first non zero value, it takes the highest absolute value.

Code: Select all

Procedure InverseMatrixD(*Matrix, *InvMatrix, n)                ; n x n matrix inverse
  Protected.l n1=n-1,p,k,t,v
  Protected.d r,m,*s.Double,*d.Double
  Protected Dim Mat.d(n1,n1)
  Protected Dim Piv.u(n1)
  CopyMemory(*Matrix, @Mat(), n*n<<3)                           ; << copy *Matrix to Mat() >>
  For t=0 To n1 : Piv(t)=t : Next                               ; << init Piv() >>
  For p=0 To n1 : m=0
    For t=p To n1                                               ; << find highest absolute value >>
      r=Abs(Mat(p,Piv(t))) : If r>m : m=r : v=t : EndIf
    Next
    If m=0 : ProcedureReturn 0 : EndIf                          ; << exit if all 0 >>
    Swap Piv(p),Piv(v) : k=Piv(p) : r=1/Mat(p,k) : Mat(p,k)=1   ; << get k and reciprocal for Mat(p,k) >>
    *d=@Mat(p,0) : For t=0 To n1 : *d\d*r : *d+8 : Next         ; << multiply pivot row by reciprocal >>
    *d=@Mat() 
    For t=0 To n1
      If t<>p
        *s=@Mat(p,0) : m=-Mat(t,k) : Mat(t,k)=0
        For v=0 To n1 : *d\d+*s\d*m : *s+8 : *d+8 : Next        ; << calculate new Mat() values >>
      Else
        *d+n<<3
      EndIf
    Next
  Next
  For t=0 To n1 : *d=*InvMatrix+Piv(t)*n<<3
    For v=0 To n1 : *d\d=Mat(t,Piv(v)) : *d+8 : Next            ; << set *InvMatrix >> 
  Next
  ProcedureReturn #True
EndProcedure


Procedure InverseMatrixF(*Matrix, *InvMatrix, n)                ; n x n matrix inverse
  Protected.l n1=n-1,p,k,t,v
  Protected.d r,m,*s.Double,*d.Double
  Protected *f.Float=*Matrix
  Protected Dim Mat.d(n1,n1)
  Protected Dim Piv.u(n1)
  *d=@Mat()
  For t=0 To n1
    For v=0 To n1 : *d\d=*f\f : *d+8 : *f+4 : Next              ; << copy *Matrix to Mat() >>
  Next
  For t=0 To n1 : Piv(t)=t : Next                               ; << init Piv() >>
  For p=0 To n1 : m=0
    For t=p To n1                                               ; << find highest absolute value >>
      r=Abs(Mat(p,Piv(t))) : If r>m : m=r : v=t : EndIf
    Next
    If m=0 : ProcedureReturn 0 : EndIf                          ; << exit if all 0 >>
    Swap Piv(p),Piv(v) : k=Piv(p) : r=1/Mat(p,k) : Mat(p,k)=1   ; << get k and reciprocal for Mat(p,k) >>
    *d=@Mat(p,0) : For t=0 To n1 : *d\d*r : *d+8 : Next         ; << multiply pivot row by reciprocal >>
    *d=@Mat() 
    For t=0 To n1
      If t<>p
        *s=@Mat(p,0) : m=-Mat(t,k) : Mat(t,k)=0
        For v=0 To n1 : *d\d+*s\d*m : *s+8 : *d+8 : Next        ; << calculate new Mat() values >>
      Else
        *d+n<<3
      EndIf
    Next
  Next
  For t=0 To n1 : *f=*InvMatrix+Piv(t)*n<<2
    For v=0 To n1 : *f\f=Mat(t,Piv(v)) : *f+4 : Next            ; << set *InvMatrix >> 
  Next
  ProcedureReturn #True
EndProcedure



Dim A.d(3,3)
A(0,0) = 2
A(0,1) = 5
A(0,2) = 3
A(0,3) = 2

A(1,0) = 4
A(1,1) = 10
A(1,2) = 1
A(1,3) = 7

A(2,0) = 1
A(2,1) = 5
A(2,2) = 2
A(2,3) = 2

A(3,0) = 2
A(3,1) = 1
A(3,2) = 2
A(3,3) = 1

InverseMatrixD(@A(), @A(), 4)

For t=0 To 3 : For v=0 To 3 : Debug StrD(A(t,v),5) : Next : Next

Re: Matrix Inversion

Posted: Sun Jul 01, 2018 10:54 am
by Psychophanta
Interesting topic. Thanks!

Re: Matrix Inversion

Posted: Mon Jul 02, 2018 2:27 pm
by wilbert
Psychophanta wrote:Interesting topic.
I agree.

I hadn't looked into matrix inversion before but it's a nice algorithm doctornash came up with.