CreatePack in memory?

[mikejs]

Hi all,

I have a case where a web service needs to create an archive and return that archive to a client via CGI. Currently I have this implemented such that it uses CreatePack() with a temp file, and then reads the temp file. This works after a fashion, but it seems like it would be better to do it entirely in memory.

Recently though, I've started getting intermittent failures with this process which I strongly suspect are to do with the antivirus software on the server (which I don't have any control over), causing file locking issues. Such that after calling ClosePack() and moving on to ReadFile(), I'm finding the file locked. I could probably work around this or add a retry mechanism or something, but again it seems like a neater solution would be to do it all in memory and avoid the temp file in the first place.

The archive is not large - 10s of KBs at most. It will typically contain multiple files, and I need their filenames stored such that the client can use ExaminePack to get the names and extract the files. This means I can't just use CompressMemory() as that just results in a compressed block with no filename information. The files may change at any time, so although caching the archive is an option, I will still need a way to reliably build the archive first in order to have something to cache.

One option that comes to mind is using a named pipe for the temp file, which I think should work, but are there better options?

[ricardo_sdl]

I think you could use CompressMemory() and then read the buffer and write it to the output.

[mikejs]

I think you could use CompressMemory() and then read the buffer and write it to the output.

I mention in the original post why I think that won't work - the client that receives this output uses ExaminePack() to get the filenames as well as the data. I don't see a way of using CompressMemory() to include the filenames and whatever else is needed for ExaminePack() to work - it just compresses a raw block.

I could make use of that, but it would then require some kind of header with names and offsets of the compressed blocks for each file. I have thought about that, but the main downside is that it would require client-side changes as well, meaning existing clients would not be compatible with it.

[Fred]

Moved to feature request as it could be useful

[mikejs]

Cheers, Fred - I'll try the named pipe approach in the meantime.

[Quin]

Giving this a +1. I've only ever had one case where I've had to personally do this, but I can definitely see the use.

[mikejs]

Cheers, Fred - I'll try the named pipe approach in the meantime.

Hmm. Having trouble getting the named pipe approach to work. I can create the named pipe successfully, but trying to use CreatePack() with it results in an error that "All pipe instances are busy."

Searching on this implies that it's not safe to close and reopen file handles on a named pipe, and I'm wondering if CreatePack() is doing this. The fact that CreatePack() is documented as overwriting any existing file that you specify (and the pipe probably counts as existing) makes me wonder whether it is possible to use a named pipe in place of a temp file for this case.