Page 1 of 1

Dim and out of memory

Posted: Mon Oct 19, 2009 3:30 pm
by luis
What happen when a Dim fails ?

Using memory allocation you can test the returned value for zero, is there a clean way to trap a Dim failure ?


BTW: don't know what happen if a Dim fails, probably a program crash... I didn't look at the generated code but I suppose dim-med vars are allocated on the heap, so probably could be doable something like this: if a Dim test.i(10000) fails, the array could be resized to 0 and one could test with ArraySize to see if the Dim was successful.

Re: Dim and out of memory

Posted: Mon Oct 19, 2009 5:51 pm
by Demivec
luis wrote:BTW: don't know what happen if a Dim fails, probably a program crash... I didn't look at the generated code but I suppose dim-med vars are allocated on the heap, so probably could be doable something like this: if a Dim test.i(10000) fails, the array could be resized to 0 and one could test with ArraySize to see if the Dim was successful.
I have a few comments on your question, though no solutions.

An array of size 0 still takes up room for the array structure and 1 element and so would not be a great solution. Not having enough memory for an array when it is dimmed is similar to not having enough memory for any other variable, if it doesn't exist the program shouldn't run. I believe it also shares something with memory allocation in that the memory isn't used when the array is dimensioned until it is accessed for the first time.

For instance, without uncommenting the last line, this code works without error for me with Win XP and 2GB ram:

Code: Select all

Structure test
  q.q[999999999]
EndStructure


Dim a.test(999999999) ;this array would take up 7275957+ TB according to my rough calculations.

Debug a(0)\q[1] ;first element is fine
;Debug a(999999999)\q[999999998]  ;uncomment this and it will error trying to access the last element
I think it is a matter of determining how much memory is available and then dimensioning an array accordingly. Perhaps you could attempt to allocate the memory first and if it succeeds, unallocate the memory and dimension the array to a similar memory size.

Re: Dim and out of memory

Posted: Mon Oct 19, 2009 6:57 pm
by luis
Hi Demivec!
Demivec wrote:An array of size 0 still takes up room for the array structure and 1 element and so would not be a great solution. Not having enough memory for an array when it is dimmed is similar to not having enough memory for any other variable, if it doesn't exist the program shouldn't run.
I still believe it could be a good solution. To me seem the dimensioning of an array should be similar to a memory allocation with malloc(). Maybe I don't have enough memory for a big array but I can have plenty of memory for many simple variables.

If a big malloc() can fail, would be nice for a big dim to fail the same way, and the simplest way to communicate that back to the calling program would probably be to change my (10000) to (0).

A failed dim, as a failed malloc, doesn't necessarily mean the program must quit. Maybe it can work with other smaller data (for example a smaller image) or it can choose to work with many smaller chunks of the same data, if possible.
Demivec wrote:I think it is a matter of determining how much memory is available and then dimensioning an array accordingly. Perhaps you could attempt to allocate the memory first and if it succeeds, unallocate the memory and dimension the array to a similar memory size.
I thought about that too but I dislike it, allocating memory with malloc(), if ok deallocate and allocate an array of the same size... it's quite ugly. I'm also not sure it's equivalent. And you can add the fact another program can request the same amount of data between the two memory allocations. And all of this is complicated by the use of the virtual memory by the OS.

BUT the point is: if I can get a valid result (ok/fail) from a malloc() it should be possible to use the same principle with a Dim(), depending on how PB treat the memory allocation needed for a Dim.

Anyway, yes, nothing of this seem to offer a solution to the original problem.

Thank you for the reply :)


EDIT: oh, just got your remark about the size of 1 item in an array. You are talking of a BIG structured single item. Yes, you are right.
Anyway I was talking about simple arrays, primitive data types. Usually I use AllocateMemory() instead of BIG arrays, but sometimes an array is more handy, especially since some PB library functions works only with arrays and not with memory areas, even if they are structured in some way.

Re: Dim and out of memory

Posted: Mon Oct 19, 2009 7:09 pm
by luis
A simpler code snippets

Code: Select all

Dim q.q(999999999)

q(1) = 123 ; invalid memory access
it fails right away


relative code :

Code: Select all

; Dim q.q(999999999)
  PUSH   dword 13
  PUSH   dword a_q
  PUSH   dword 0
  MOV    edx,dword [a_q]
  CALL   SYS_FreeArray
  MOV    eax,1000000000
  PUSH   dword 8
  CALL   SYS_AllocateArray
; 
; q(1) = 123 ; invalid memory
  MOV    ebp,dword [a_q]
  LEA    eax,[ebp+8]
  MOV    dword [eax+4],0
  MOV    dword [eax],123
; 
Maybe CALL SYS_AllocateArray simply fails but there is no check so

Code: Select all

 
  MOV    ebp,dword [a_q]
  LEA    eax,[ebp+8]
  MOV    dword [eax+4],0
  MOV    dword [eax],123
results in the illegal memory access :?:

Re: Dim and out of memory

Posted: Mon Oct 19, 2009 8:19 pm
by Fred
I modified ArraySize(), so it will return -1 if the array isn't declared.

Re: Dim and out of memory

Posted: Mon Oct 19, 2009 8:33 pm
by luis
NO! This is great!

So we could do something like

Code: Select all

Dim q.q(999999999)

If ArraySize(q()) <> -1 
    q(12345) = 123 ; all is well
Else
   ; out of mem
EndIf
?

Thank you Fred!

Re: Dim and out of memory

Posted: Mon Oct 19, 2009 9:18 pm
by Fred
yep.

Re: Dim and out of memory

Posted: Mon Oct 19, 2009 9:48 pm
by Demivec
Awesome!

Re: Dim and out of memory

Posted: Tue Oct 20, 2009 8:57 am
by Little John
Demivec wrote:Awesome!
Yes, indeed. Thanks, Fred!
And thanks to luis for raising this issue!

Regards, Little John

Re: Dim and out of memory

Posted: Sat Nov 28, 2009 10:29 pm
by Andre
Fred wrote:I modified ArraySize(), so it will return -1 if the array isn't declared.
Added a note + example to the Docs. :)

Re: Dim and out of memory

Posted: Sat Nov 28, 2009 10:40 pm
by Kaeru Gaman
Demivec wrote:An array of size 0 still takes up room for the array structure and 1 element
does it?
Dim Help wrote:To delete the content of an array and release its used memory during program flow, call Dim with array name and 0 elements.
to me this is clear:
Dim called with 0 does not resize the Array to one Element, what would make no sense since nobody would use an Array instead of a single variable...
but it is exceptionally handled to erase the Array.


@topic
really cool completion of the ArraySize function! thnx!

Re: Dim and out of memory

Posted: Sat Nov 28, 2009 10:50 pm
by luis
Kaeru Gaman wrote: Dim called with 0 does not resize the Array to one Element [...] but it is exceptionally handled to erase the Array.
Good catch, I always thought the item (0) would be present but you are probably right, make sense considering the sentence from the manual.

[lazy]
Would be nice to know for sure (without trying to understand it by myself looking into pb code / libs !)
[/lazy]