Okay here's another question from my long list of relatively insane ideas.
Is it possible to create a cross platform (Linux, MAC OSX, Windows) cross process communications structure for use with PureBASIC?
This question is actually complicated but it's a general query.
Frequently I find a handy piece of code (or huge piece of code) I want to exploit but it's written in PHP, Python, C++, Ruby and so on (I don't have anything specific in mind). Thing is, many times I want the ease of BASIC for the stupid stuff and the polish of something else for some function.
Given that PureBASIC can PEEK and POKE around in the memory of the computer, has anyone implemented a 'mailslot' system that basically allows a compatibly coded thread in a different language to pass (what could be large) amounts of data in a polled fashion? I figure it can be done with the IPC, but why not just use a structure directly in memory?
In effect this is somewhat similar to how a .DLL or .SO is 'wrapped' so that it can pass structures to PureBASIC, but what I'm looking for is a little more then that, because I want to be able to make this trick work with a language like Python or PHP that is relatively interpretted (yes I know there are ways to compile them...it's not the focus.)
Basically, one would 'wrap' the code with the language it requires and then it would pass it's data using this shared memory region. Of course, some kind of rudimentary locking would be required.
It occured to me that SQLite in a way would enable this. Write to database, have another program take from database on the same machine. This too seems kind of heavy if one process is generating a lot of transitory information.
In a very real way, the idea I'm thinking about is how we used to POKE assembly code into the memory of say a Commodore and then effectively CALL it. You end up using the stack and memory in this case in much the same manner as I've described.
Sigmason
Don't PEEK at me or I'll POKE you.
Cross process (cross platform) passing using memory maps
Trond, actually that's true but also partially incorrect at the same time.
I've written a few kernels (in assembler) while doing RTOS development (can you say painful?)
While generally your statement about the virtualization of memory is correct (within reason), it isn't a 100% truth.
For instance, MMAP in UNIX allows a space of memory to be shared, virtualized memory or not.
I believe the CreateFileMapping API can perform a similar function.
So while for example, Virtual memory API functions would be limited to the threads of a process in the virtual address space, I'm fairly certain CreateFileMappings are not.
On that note, I started digging in my links (I have way too many links) and found this in Visual BASIC:
http://www.simpits.org/programming/data ... aring.html
So I'm not crazy this can be done.
So for example, one could create a mailslot type system passing a simple semaphore to perform basic locking so long as they have access to the code base on both sides of the memory/file structure to define the context and layout of the mapping.
Sigmason
So I guess I'm only 99% insane on this one
I've written a few kernels (in assembler) while doing RTOS development (can you say painful?)
While generally your statement about the virtualization of memory is correct (within reason), it isn't a 100% truth.
For instance, MMAP in UNIX allows a space of memory to be shared, virtualized memory or not.
I believe the CreateFileMapping API can perform a similar function.
So while for example, Virtual memory API functions would be limited to the threads of a process in the virtual address space, I'm fairly certain CreateFileMappings are not.
On that note, I started digging in my links (I have way too many links) and found this in Visual BASIC:
http://www.simpits.org/programming/data ... aring.html
So I'm not crazy this can be done.
So for example, one could create a mailslot type system passing a simple semaphore to perform basic locking so long as they have access to the code base on both sides of the memory/file structure to define the context and layout of the mapping.
Sigmason
So I guess I'm only 99% insane on this one
In response to my own thread, and so I'm not leading others wrong....
Using the Windows API CreateFileMappings doesn't actually create a completely unmanaged area of shared memory between processes in virtual memory. It creates an operating system assisted (and therefore it doesn't violate the virtual memory design of the operating system like POKE and PEEK would have to if it would work at all) area of memory.
Tronds statement is very correct that if you forcibly accessed memory using the PEEK and POKE to allocate a chunck of memory you'd likely fail due to the virtual memory. However, if the operating system is happy to let you contruct a file mapping like this, in effect it does the job. In a very real sense, all mapping a file like this does is in effect provide the safety of allowing the operating system to consider this a file contruct while leaving the memory management unharmed. You're not really directly accessing the memory, but as stated, that's not a good thing for your operating system to let you do.
This works good enough for me.
Thank you Trond, I noticed you basically asked about this exact same thing a few years ago. Please look at the example link I provided. It details how to get Windows not to open an actual file, but just a file in memory.
Sigmason
Using the Windows API CreateFileMappings doesn't actually create a completely unmanaged area of shared memory between processes in virtual memory. It creates an operating system assisted (and therefore it doesn't violate the virtual memory design of the operating system like POKE and PEEK would have to if it would work at all) area of memory.
Tronds statement is very correct that if you forcibly accessed memory using the PEEK and POKE to allocate a chunck of memory you'd likely fail due to the virtual memory. However, if the operating system is happy to let you contruct a file mapping like this, in effect it does the job. In a very real sense, all mapping a file like this does is in effect provide the safety of allowing the operating system to consider this a file contruct while leaving the memory management unharmed. You're not really directly accessing the memory, but as stated, that's not a good thing for your operating system to let you do.
This works good enough for me.
Thank you Trond, I noticed you basically asked about this exact same thing a few years ago. Please look at the example link I provided. It details how to get Windows not to open an actual file, but just a file in memory.
Sigmason
Apparently, bangs head on desk, after chewing on the API a bit I found out the CreateFileMapping uses the PagingFile and seemingly only the paging file.
So it does write to the disk, which I guess is the reason it's not relevant still to Trond's previous post.
Okay, but this might work:
http://www.genesys-e.de/jwalter/mix4win.htm
http://en.wikipedia.org/wiki/Mmap
Sigmason
So it does write to the disk, which I guess is the reason it's not relevant still to Trond's previous post.
Okay, but this might work:
http://www.genesys-e.de/jwalter/mix4win.htm
http://en.wikipedia.org/wiki/Mmap
Sigmason
MMAP on Linux does something similar, but it's always memory so far as I know.
That said, that link I posted about VirtualAlloc is a better bet.
It shows how to simulate the MMAP function in *nix for Windows.
That said, it looks like VirtualAlloc also lets you control how memory is swapped out, which is a step up I think.
Thanks,
Sigmason
That said, that link I posted about VirtualAlloc is a better bet.
It shows how to simulate the MMAP function in *nix for Windows.
That said, it looks like VirtualAlloc also lets you control how memory is swapped out, which is a step up I think.
Thanks,
Sigmason
http://docs.python.org/lib/module-mmap.html
Mmap is available in Python, it might be a good way to exchange data between Python and PureBASIC, along with say something like POSH.
Sigmason
Mmap is available in Python, it might be a good way to exchange data between Python and PureBASIC, along with say something like POSH.
Sigmason
Well I have an idea (and yes my memory allocations and I are evil....it's only fitting that the multitasking environment knows that!)...
Lets say you have a communications process running to recieve and service data in another process. Let's say that it is the job of this other process to recieve (service) and effectively identify this data then sort it roughly into queues, essentially front ending your PureBASIC program. Now lets say this communications process were running REALLY fast and recieving somewhat large quantities of data.
I have little doubt that PureBASIC is fast, in fact my tests have confirmed it's quite efficient at what it does. That said, in this scenario, I need to get this data from the other process as fast as I can, whilst performing the sorting as efficiently as possible. Sorting every packet coming in at the terminus of the exchange (PureBASIC) could become a problem. So lets say the recieving process (presummably in another language) is recieving, sorting, threading (as it likes and the operating system permits), and placing this data into a storage structure (a VERY simple database of sorts) confined to possibly a dynamically allocated shared memory area (dynamic buffer or queue) which in turn has a known structure that PureBASIC can exploit.
Basically, the idea is to create a very large 'asynchronous' queuing system for recieved data (could be packets or not) so that quality of service (QOS) can be attained. For example, let's say the communications process is recieving packets that contain, say speech (digital PCM encoded and compressed), file transfers (compressed) and at the same time transaction data. With a large queue between the 2 process one could provide enough buffer to let PureBASIC run it's end user functions asynchronously from the communications subsystem, while being able to also prioritize the processing order of say one queue over the other (so for instance the speech doesn't get choppy when decoded.)
The other advantage as I see it, is that no matter how crazy you get with multiple processors, so long as your operating system is willing to thread and/or distribute the process load you can heavily tap the core of each additional processor.
Of course the downside, is that this kind of allocation, is as stated, not exactly friendly to the other unrelated processes in the system. However, I have this image of using this as a server side trick not as a client side trick. On the server, the essential theft of this memory from all but the kernel and memory management subsystem isn't really all that disruptive if it's considered during deployment.
Picture it like this, it's transacting data from say a Python based Twisted server to a PureBASIC program using memory as a buffer. Is there really anything faster then that other then say the processor registers?
Sigmason
Note, in a very real way the people using ETNA on BlitzBASIC or BlitzMAX are doing something similar.http://www.blitzbasic.com/Community/pos ... opic=65184
The only huge difference is that I am thinking of a memory structure with a simple semaphore rather then mySQL (which is not free) and a whole TCP/IP stack and server environment.
Lets say you have a communications process running to recieve and service data in another process. Let's say that it is the job of this other process to recieve (service) and effectively identify this data then sort it roughly into queues, essentially front ending your PureBASIC program. Now lets say this communications process were running REALLY fast and recieving somewhat large quantities of data.
I have little doubt that PureBASIC is fast, in fact my tests have confirmed it's quite efficient at what it does. That said, in this scenario, I need to get this data from the other process as fast as I can, whilst performing the sorting as efficiently as possible. Sorting every packet coming in at the terminus of the exchange (PureBASIC) could become a problem. So lets say the recieving process (presummably in another language) is recieving, sorting, threading (as it likes and the operating system permits), and placing this data into a storage structure (a VERY simple database of sorts) confined to possibly a dynamically allocated shared memory area (dynamic buffer or queue) which in turn has a known structure that PureBASIC can exploit.
Basically, the idea is to create a very large 'asynchronous' queuing system for recieved data (could be packets or not) so that quality of service (QOS) can be attained. For example, let's say the communications process is recieving packets that contain, say speech (digital PCM encoded and compressed), file transfers (compressed) and at the same time transaction data. With a large queue between the 2 process one could provide enough buffer to let PureBASIC run it's end user functions asynchronously from the communications subsystem, while being able to also prioritize the processing order of say one queue over the other (so for instance the speech doesn't get choppy when decoded.)
The other advantage as I see it, is that no matter how crazy you get with multiple processors, so long as your operating system is willing to thread and/or distribute the process load you can heavily tap the core of each additional processor.
Of course the downside, is that this kind of allocation, is as stated, not exactly friendly to the other unrelated processes in the system. However, I have this image of using this as a server side trick not as a client side trick. On the server, the essential theft of this memory from all but the kernel and memory management subsystem isn't really all that disruptive if it's considered during deployment.
Picture it like this, it's transacting data from say a Python based Twisted server to a PureBASIC program using memory as a buffer. Is there really anything faster then that other then say the processor registers?
Sigmason
Note, in a very real way the people using ETNA on BlitzBASIC or BlitzMAX are doing something similar.http://www.blitzbasic.com/Community/pos ... opic=65184
The only huge difference is that I am thinking of a memory structure with a simple semaphore rather then mySQL (which is not free) and a whole TCP/IP stack and server environment.
Last edited by sigmason on Wed Jun 06, 2007 9:32 pm, edited 2 times in total.
I decided to clarify my last post by adding this:
The point is not that PureBASIC couldn't out perform the other language, it's not really about that, so please no one think I'm faulting PureBASIC...that's not the case.
The point is, in this manner one can contruct a potentially vast buffer in actual memory, and exploit it to perform high speed interprocess communications without some of the limitations or overheads of say mailslots or perhaps even pipes to some extent.
This could be exploited to create a facility by which one can essentially join together the strengths of several other languages and packages into a modular system that lends itself to the end task. While, this idea could allow the programmer the modularity to eliminate a project while maintaining it's interface to the remainder of the code. While it does carry with it risk of memory exploits from the local machine, in some cases that might be a minimal concern. This is not to say other mechanisms don't work mind you.
Sigmason
The point is not that PureBASIC couldn't out perform the other language, it's not really about that, so please no one think I'm faulting PureBASIC...that's not the case.
The point is, in this manner one can contruct a potentially vast buffer in actual memory, and exploit it to perform high speed interprocess communications without some of the limitations or overheads of say mailslots or perhaps even pipes to some extent.
This could be exploited to create a facility by which one can essentially join together the strengths of several other languages and packages into a modular system that lends itself to the end task. While, this idea could allow the programmer the modularity to eliminate a project while maintaining it's interface to the remainder of the code. While it does carry with it risk of memory exploits from the local machine, in some cases that might be a minimal concern. This is not to say other mechanisms don't work mind you.
Sigmason
Adding to the pool of information in this thread, I thought it might be wise to reference that despite what VirtualAlloc() says, unless you use it with the AWE (very tricky from what I see) the VirtualLock() API doesn't actually lock the allocated memory into physical memory. Though that sentence is rather cryptic.
Apparently the cryptography folks (Peter Gutmann with them) have made note that items in memory that are protected by VirtualLock() form being swapped to the pagefile are NOT actually prevented at all times from being put in the Windows PageFile. Apparently, if the task becomes idle and starts to hibernate (looses significant focus) the VirtualLock() and the application page set that it is part of will swap to the pagefile.
This interesting tidbit is not really overly important to the scenario I created earlier, other then to say that you aren't completely ripping memory out of the operating system's use. Thought it does mean that one should be aware of it, so they still might find themselves pulling something from a pagefile instead of hardware memory if things get tight or usage of the application that set the VirtualLock gets low. (It sucks for cryptography because that means that if the private key is in the VirtualLock() it might get written to the pagefile for who knows how long.)
Sigmason
Apparently the cryptography folks (Peter Gutmann with them) have made note that items in memory that are protected by VirtualLock() form being swapped to the pagefile are NOT actually prevented at all times from being put in the Windows PageFile. Apparently, if the task becomes idle and starts to hibernate (looses significant focus) the VirtualLock() and the application page set that it is part of will swap to the pagefile.
This interesting tidbit is not really overly important to the scenario I created earlier, other then to say that you aren't completely ripping memory out of the operating system's use. Thought it does mean that one should be aware of it, so they still might find themselves pulling something from a pagefile instead of hardware memory if things get tight or usage of the application that set the VirtualLock gets low. (It sucks for cryptography because that means that if the private key is in the VirtualLock() it might get written to the pagefile for who knows how long.)
Sigmason
I'm still thinking about the twists of this a few days later and figure I post a few other twists I found to this.
It appears that the way Windows allocates the memory though the virtual memory API means that memory mapped to a file or allocated through it may not, likely by necessity, be mapped to the same places in other processes virtual address space. So apparently one needs to use base pointers to get to it.
In *nix address spaces (I'm a little more familiar with) something similar happens. Basically if a section of memory is MMAPed, as apposed to SHM directly mapped (SHM creates an actual common area in many of the implementations I've seen,) you end up with a section of memory that is also mapped into each processes virtual address space. In *nix, it seems the mechanism for combining these virtual areas is to use a page fault, so that the operating system can intervene when 2 processes operate in the overlap area (so alot of transactions causing page faults could be slower then using the SHM functions which appear not to be an option in Windows. Though one can write a kernel mode driver with the DDK, yikes.)
So based on this, one can create a process, and I've found code from projects like CYGWIN, that essentially provide string access to a 'shared' memory region mapped in diferring locations in each programs virtual address space. With that, one could also revert back to a similar function to MMAP's file functionality and use CreateFileMapping, which based on some tests I found might be just slightly slower (maybe only on Windows), but likely because of the file structure being imposed and not because of the virtual memory cross mappings (because so far as I can tell, CreateFileMapping still cross maps memory in Windows...but I may be wrong.) I'm not sure about using MMAPs file functions being slower then it's string functions in *nix because I suppose it's possible the implementation of MMAP might use SHM to make them work (really not sure on this.)
Sigmason
It appears that the way Windows allocates the memory though the virtual memory API means that memory mapped to a file or allocated through it may not, likely by necessity, be mapped to the same places in other processes virtual address space. So apparently one needs to use base pointers to get to it.
In *nix address spaces (I'm a little more familiar with) something similar happens. Basically if a section of memory is MMAPed, as apposed to SHM directly mapped (SHM creates an actual common area in many of the implementations I've seen,) you end up with a section of memory that is also mapped into each processes virtual address space. In *nix, it seems the mechanism for combining these virtual areas is to use a page fault, so that the operating system can intervene when 2 processes operate in the overlap area (so alot of transactions causing page faults could be slower then using the SHM functions which appear not to be an option in Windows. Though one can write a kernel mode driver with the DDK, yikes.)
So based on this, one can create a process, and I've found code from projects like CYGWIN, that essentially provide string access to a 'shared' memory region mapped in diferring locations in each programs virtual address space. With that, one could also revert back to a similar function to MMAP's file functionality and use CreateFileMapping, which based on some tests I found might be just slightly slower (maybe only on Windows), but likely because of the file structure being imposed and not because of the virtual memory cross mappings (because so far as I can tell, CreateFileMapping still cross maps memory in Windows...but I may be wrong.) I'm not sure about using MMAPs file functions being slower then it's string functions in *nix because I suppose it's possible the implementation of MMAP might use SHM to make them work (really not sure on this.)
Sigmason
Last edited by sigmason on Fri Jun 08, 2007 7:33 pm, edited 2 times in total.
Building on this (sorry, to those looking for working code right now, I'm kind of thinking out loud in this thread.)
In keeping with the idea of creating a database style place to exchange data in memory between 2 processes (perhaps not even writing in the same language or even compiled) perhaps a good idea might be to literally create a form of database in memory that runs as it's own process.
So for example one would create a 'database' process that allocates a block of shared memory using virtualalloc in Windows and MMAP (for compatibility) in Linux and Mac OSX. Now if one creates essentially a memory backed database process, one could just use local process memory and transact the data using anonymous or named pipes. The issue I see is that while transacting small amounts of information like this might be practical, it's likely to be slower then creating a process that acts like a shared memory area broker. The broker process could communicate between processes using pipes and then actually allow the transaction of data (that it is managing internally) by passing base offset pointers to the shared regions containing the data being focused on in the process requesting that area. The advantage, as I see it, is that the task of insuring that this data area is maintained properly and that the semaphores are kept operating correctly would not have to be replicated in the other processes. In effect it would be like making a transaction to a database server, but a database server that is essentially sitting in your own local memory.
The only thing I have to think about is what the security of using pipes in this manner is. I think creating a broker process would actually improve the security of a shared memory block (by internalizing the pointers) or even (if speed wasn't an issue) by encrypting and decrypting pages.
Sigmason
In keeping with the idea of creating a database style place to exchange data in memory between 2 processes (perhaps not even writing in the same language or even compiled) perhaps a good idea might be to literally create a form of database in memory that runs as it's own process.
So for example one would create a 'database' process that allocates a block of shared memory using virtualalloc in Windows and MMAP (for compatibility) in Linux and Mac OSX. Now if one creates essentially a memory backed database process, one could just use local process memory and transact the data using anonymous or named pipes. The issue I see is that while transacting small amounts of information like this might be practical, it's likely to be slower then creating a process that acts like a shared memory area broker. The broker process could communicate between processes using pipes and then actually allow the transaction of data (that it is managing internally) by passing base offset pointers to the shared regions containing the data being focused on in the process requesting that area. The advantage, as I see it, is that the task of insuring that this data area is maintained properly and that the semaphores are kept operating correctly would not have to be replicated in the other processes. In effect it would be like making a transaction to a database server, but a database server that is essentially sitting in your own local memory.
The only thing I have to think about is what the security of using pipes in this manner is. I think creating a broker process would actually improve the security of a shared memory block (by internalizing the pointers) or even (if speed wasn't an issue) by encrypting and decrypting pages.
Sigmason