Large Arrays and memory question

Just starting out? Need help? Post your questions and find answers here.
User avatar
charvista
Addict
Addict
Posts: 949
Joined: Tue Sep 23, 2008 11:38 pm
Location: Belgium

Large Arrays and memory question

Post by charvista »

I am wondering if creating a large integer or string Array will occupy a large amount of memory, when it is empty ? (or almost empty)

Say I am doing Dim A.s(1000000) and I put very few elements in it, will this large array be memory consuming ?
- Windows 11 Home 64-bit
- PureBasic 6.10 LTS (x64)
- 64 Gb RAM
- 13th Gen Intel(R) Core(TM) i9-13900K 3.00 GHz
- 5K monitor with DPI @ 200%
User avatar
idle
Always Here
Always Here
Posts: 6035
Joined: Fri Sep 21, 2007 5:52 am
Location: New Zealand

Re: Large Arrays and memory question

Post by idle »

It's going to be the num dimensions x size of Integer plus the strings you set
User avatar
Caronte3D
Addict
Addict
Posts: 1371
Joined: Fri Jan 22, 2016 5:33 pm
Location: Some Universe

Re: Large Arrays and memory question

Post by Caronte3D »

Maybe some ReDim use, would be good here :?
User avatar
jacdelad
Addict
Addict
Posts: 2032
Joined: Wed Feb 03, 2021 12:46 pm
Location: Riesa

Re: Large Arrays and memory question

Post by jacdelad »

charvista wrote: Sat Jan 20, 2024 11:45 am I am wondering if creating a large integer or string Array will occupy a large amount of memory, when it is empty ? (or almost empty)

Say I am doing Dim A.s(1000000) and I put very few elements in it, will this large array be memory consuming ?
The array always consumes the same amount of memory, no matter how many elements you actually use. Plus, like idle said, more memory if you use strings or structures.
You should maybe use a list or an array?
Good morning, that's a nice tnetennba!

PureBasic 6.21/Windows 11 x64/Ryzen 7900X/32GB RAM/3TB SSD
Synology DS1821+/DX517, 130.9TB+50.8TB+2TB SSD
User avatar
charvista
Addict
Addict
Posts: 949
Joined: Tue Sep 23, 2008 11:38 pm
Location: Belgium

Re: Large Arrays and memory question

Post by charvista »

Thank you for your replies.
jacdelad wrote: Sat Jan 20, 2024 12:59 pm The array always consumes the same amount of memory, no matter how many elements you actually use. Plus, like idle said, more memory if you use strings or structures.
You should maybe use a list or an array?
So I understand that creating an array with a huge dimension will not affect the memory or the processor, only a huge amount of data might have a bad impact.
That's good to know, because my intention is that when something comes at element number 468'501 for example, then I'm doing A.s(468501)="something". But only a few of them. So it is safe doing so. I was only afraid for the huge array size!!

Live long and prosper!
- Windows 11 Home 64-bit
- PureBasic 6.10 LTS (x64)
- 64 Gb RAM
- 13th Gen Intel(R) Core(TM) i9-13900K 3.00 GHz
- 5K monitor with DPI @ 200%
infratec
Always Here
Always Here
Posts: 7664
Joined: Sun Sep 07, 2008 12:45 pm
Location: Germany

Re: Large Arrays and memory question

Post by infratec »

I don't know if you understand the answer correctly.

In your eample, it is an array of strings.
A string is in this case a pointer to the memory where the data is stored.

So at time of creation your array needs the size of the elements multipied by the size of a pointer (x86 = 4, x64 = 8 bytes)
If you fill a string, then the size needed is increased only by the byte size of this string.
This happens for each string you fill with something.
User avatar
mk-soft
Always Here
Always Here
Posts: 6321
Joined: Fri May 12, 2006 6:51 pm
Location: Germany

Re: Large Arrays and memory question

Post by mk-soft »

Are there really all 1000000 elements or are there many gaps?
Otherwise maps also work ...

Code: Select all

Global NewMap myMap.s()

Element = 146501

AddMapElement(myMap(), Str(Element))
myMap() = "Hello"
If FindMapElement(myMap(), Str(Element))
  Debug myMap()
EndIf
My Projects ThreadToGUI / OOP-BaseClass / EventDesigner V3
PB v3.30 / v5.75 - OS Mac Mini OSX 10.xx - VM Window Pro / Linux Ubuntu
Downloads on my Webspace / OneDrive
AZJIO
Addict
Addict
Posts: 2226
Joined: Sun May 14, 2017 1:48 am

Re: Large Arrays and memory question

Post by AZJIO »

Compile, open the task manager and see the amount of memory used. It is not difficult to test and understand the behavior of the array.

Code: Select all

; Define Dim Arr(10000000) ; 76.7 Mb
Define Dim Arr(1000000) ; 8 Mb (your example. 8 * 1000000 = 8000000 bytes)
; Define Dim Arr(500000) ; 4.2 Mb
; Define Dim Arr(50000) ; 0.8 Mb
; Define Dim Arr(5000) ; 0.4 Mb
; Define Dim Arr(500) ; 0.4 Mb
; Define Dim Arr(50) ; 0.4 Mb
; Define Dim Arr(5) ; 0.4 Mb
Delay(10000)
User avatar
charvista
Addict
Addict
Posts: 949
Joined: Tue Sep 23, 2008 11:38 pm
Location: Belgium

Re: Large Arrays and memory question

Post by charvista »

Now I understand better, thank you.
I'm running on x64 machine, so if I create an array with 1 million elements, it creates 1 million indexes, 8 bytes each.
I tested the example from AZJIO, I have the same memory results, so it takes 8 Mb upon creation of the array.

AZJIO, I was wondering why you use "Define" in front of a "Dim" ? The Define is not necessary to declare arrays, only variables... :wink:
- Windows 11 Home 64-bit
- PureBasic 6.10 LTS (x64)
- 64 Gb RAM
- 13th Gen Intel(R) Core(TM) i9-13900K 3.00 GHz
- 5K monitor with DPI @ 200%
User avatar
mk-soft
Always Here
Always Here
Posts: 6321
Joined: Fri May 12, 2006 6:51 pm
Location: Germany

Re: Large Arrays and memory question

Post by mk-soft »

Use always ...

Code: Select all

;-TOP
EnableExplicit
My Projects ThreadToGUI / OOP-BaseClass / EventDesigner V3
PB v3.30 / v5.75 - OS Mac Mini OSX 10.xx - VM Window Pro / Linux Ubuntu
Downloads on my Webspace / OneDrive
AZJIO
Addict
Addict
Posts: 2226
Joined: Sun May 14, 2017 1:48 am

Re: Large Arrays and memory question

Post by AZJIO »

charvista wrote: Sat Jan 20, 2024 4:32 pm why you use "Define" in front of a "Dim" ?
Dim, NewList, NewMap define what the variable is, but do not define the scope. Global, Define, Protected - define the scope of visibility.
Define can also be used with arrays, the lists and the maps.
New maps are always locals by default, so Global or Shared commands have to be used if a map declared in the main source need to be used in procedures.
The new list are always locals, which means Global or Shared commands have to be used if a list declared in the main source need to be used in procedures.
The new arrays are always locals, which means Global or Shared commands have to be used if an array declared in the main source need to be used in procedures.
mk-soft wrote: Sat Jan 20, 2024 5:23 pm Use always ...

Code: Select all

;-TOP
EnableExplicit
I used EnableExplicit, and the compiler does not complain about the lack of "Global/Define". PB 6.04
DarkDragon
Addict
Addict
Posts: 2347
Joined: Mon Jun 02, 2003 9:16 am
Location: Germany
Contact:

Re: Large Arrays and memory question

Post by DarkDragon »

AZJIO wrote: Sat Jan 20, 2024 6:18 pmI used EnableExplicit, and the compiler does not complain about the lack of "Global/Define". PB 6.04
EnableExplicit enforces a declaration. For normal variables you need Define/Global/Protected/Shared/Static, but for lists, arrays and maps the NewXXX keywords are enough hint for being a declaration in which case the default scope as with Define is being used. It doesn't enforce the scope being declared explicitly, it enforces the declaration of the variable at all.
bye,
Daniel
User avatar
charvista
Addict
Addict
Posts: 949
Joined: Tue Sep 23, 2008 11:38 pm
Location: Belgium

Re: Large Arrays and memory question

Post by charvista »

Interesting and good to know.

Code: Select all

EnableExplicit
Define.s a,b,c,Dim d(3),e,f
d(0)=8 ; will raise an error, since d() was declared as string array
- Windows 11 Home 64-bit
- PureBasic 6.10 LTS (x64)
- 64 Gb RAM
- 13th Gen Intel(R) Core(TM) i9-13900K 3.00 GHz
- 5K monitor with DPI @ 200%
Post Reply