File locking and trapping

[DeanH]

This may have previously been brought up, but a feature I would like to request the ability to open standard files for either exclusive use or shared for multiple users. In particular, ReadFile to allow many users to have a file open and read at the same time but OpenFile and Createfile restricted to a single user. If the file is already opened for exclusive access, that condition can be identified or trapped for retry situations. Perhaps an optional extra parameter to these functions might do the job? I'm not sure about Linux and Mac, but Windows file open functions include these. I write applications that have files shared by anywhere from one to dozens of concurrent users on a network. I've investigated this in PureBasic and as far as I can tell it seems that files are only opened for exclusive use. True, I can do a workaround by using the Windows LockFile_ API in procedures, but that involves record locking as opposed to whole file locking and I read a Microsoft tech note that recommends file locking over record locking. To date, I've ended up writing all my own file handling procedures entirely using the Windows API, mainly to get around this problem. I used to use an old DOS/MAC/TRS80 language called ZBasic (a.k.a. Future Basic on the mac) extensively and it included this capability. I hope it can be considered, or at least all files can be opened in a shared mode.

[BorisTheOld]

We ran into this problem when we switched from Windows/PowerBasic to Windows/Linux/PureBasic. Unfortunately, file locking is a complex process, and each OS has its own way of handling file locking. The problem is compounded if multiple operating systems are sharing the same file.

The aim of file locking is to gain exclusive access to a file so that data can be read, written, or updated without compromising file integrity. And this can only be achieved if the locking and unlocking processes are atomic. In other words, another process must not be able to gain access to the file during the locking/unlocking process.

A discussion of file locking would fill a book, so I'll briefly explain how we solve the problem using standard PB features. And we do it in a way that allows other applications, written in other languages, to access our files. All they need to do is follow a couple of rules.

Firstly, files are only opened when access is needed.

Secondly, a lock file is used to synchronize the locking process.

The sequence of events is:

1) Attempt to rename the lock file until successful
2) Open the file, or multiple files
3) Perform the required file i/o
4) Close the files
5) Rename the lock file back to its oringinal name

Testing and setting flags in the lock file won't work because the process uses several i/o requests and is therefore not atomic.

There's a slight performance hit, but the technique is guaranteed to work on networks and with different operating systems. It's a simple process that works for most applications. The only other method would be to have a dedicated file server that handles all file i/o, but that's a heavy duty solution more appropriate to database applications.

This is a long-winded way of saying that I don't think file locking is needed in PB, and it's probably not there for a reason.

[DeanH]

Thank you for that. The technique you've outlined certainly works to allow only one user access at a time but in my situation I need to allow many (potentially dozens) to be able to read from the same file at the same time. My customers (over 2500) are schools and in a computer lab situation, for example, 25 kids may be trying to search the data at the same time. I have tried restricting to one user at a time and it can substantially slow things down to an unusable point. What I'd like to do is allow many users to access a file for concurrent reading but restrict acces to a single user when writing. If the file is locked for writing, then everyone else trying to open it waits until the lock is released (or a timeout applies). The wait is usually only a second or less. If users have a file open for reading and someone needs to write to it, the same happens. The lock is released when the file is closed. Parts of my school library software are written in PureBasic, and currently those parts use the Win32 API entirely for file handling. I did that as PB's native open functions did not appear to allow multiple user access (at least for reading). I am in the process of rewriting the entire system in PB. Yes, there are workarounds, but I thought it'd be useful if the native PB functions could cater for this.

Have to admit that I hadn't considered renaming a file, though. That's a nice trick.

[BorisTheOld]

The lockfile approach does allow for simultaneous access, because the file is not held locked while a user decides to go for a coffee.

User 1: lock, open, read record 1, close, unlock
User 2: lock, open, read record 42, close, unlock
User 1: lock, open, read record 2, close, unlock
User 1: lock, open, read record 3, close, unlock
User 3: lock, open, add record 99, read index record, update index record, close, unlock
User 1: lock, open, read record 4, close, unlock
User 2: lock, open, update record 42, close, unlock
User 1: lock, open, read record 5, close, unlock
etc

Usually, there is only one i/o request per lock/unlock cycle, so that many users can be accessing the file "simultaneously" without noticing any delay.

This technique can be fine tuned to suit the application, such as using large buffer sizes or bundling several i/o operations into a single lock/unlock cycle. But the main thing to remember is that no one should access a file without first locking it, and then unlocking it as soon as the i/o operation has been performed. That way you won't risk corrupting a file.

[RichAlgeni]

Dean, I've found that reading a file is not that much of a problem in Windows, but having multiple users write to the same file is not something Windows is really built for. Have you considered a database like PostgreSQL? It's free, supported natively by PureBasic, it's mature, and it's built to handle multiple users.

[DeanH]

I have been working with multi-user files for over 20 years and commercially published software over 30. My request was not made lightly nor due to an extreme problem I am having and is based on my experience working with large text-based databases in school situations with a couple of thousand users running on a variety of Windows versions. There are work-arounds which I have done but I do feel file locking is a 'hole' in this one small area in PureBasic. At the moment, I am implimenting SQLite throughout the system to replace another database DLL which turned out to have memory allocation problems in Windows 7 (when the DLL was called from PureBasic). That work is going fine but I am concerned about SQLite's ability to handle multiple users. If something goes completely wrong (unlikely) and that approach does not pan out my fallback is to go back to the flat-file system with which I am extremely familiar. In all cases, I only want one user to access a file when writing is involved but multiple users to be able to read at the same time. If a user needs to write and others are already reading, that user waits until clear then the entire file is locked. Everyone else waits until clear. The DB system I'd been using worked exactly like this (it was based on QDBM.) In all cases I never leave a file open for any longer than absolutely necessary. The filerename trick offered is a good solution in situations where only one user is allowed to read or write at a time, but I've already worked out some odd situations where that will not be practical.

As I indicated, I've written a range of Procedures based on the native Win32 API functions which accomplish all this, and which have been working successfully for a few years in several hundred installations. I just thought it would be useful to have had those functions as part of PureBasic's native commands.

[BorisTheOld]

This was essentially our experience -- and with printing too. However, the benefits of cross-platform support far outweigh the shortcomings in these two areas.

For now our Windows customers are using our File and Printer DLLs written in PowerBasic, which we had been using until two years ago. And our Linux and Windows/Linux customers are using simplified file and printer support written in PB. The file locking approach described above seems to be working ok, and it provides the simultaneous access that our customers require, although the continual opening and closing of files reduces performance a little.

Our main priority right now is to convert all our systems to PB, and we'll probably revisit file and printer support again towards the end of 2013. With luck PB will have a few more file and printer features by then.

In the late 1970's I got involved with desktop programming, using Z80 processors and the CP/M and MP/M operating systems. And to get around some of the file handling shortcomings I developed an indexed file module based on my experiences with the IBM mainframe access methods, ISAM and VSAM. That code, written in ASM, is still the basis of all our file handling. In fact, some of our customers are still running old DOS applications from the 1980's that can use files created by our current PB applications. Right from the beginning this code was designed to handle multi-user situations.

I agree with you that being able to open files in shared mode would solve a lot of problems, but it can also create problems.

I suspect that, as with most things in life, there's no correct answer and that the best way is the way that works best for the situation.

(edited)

[blueb]

... That work is going fine but I am concerned about SQLite's ability to handle multiple users. If something goes completely wrong (unlikely) and that approach does not pan out my fallback is to go back to the flat-file system with which I am extremely familiar. ...

Have you looked into SQLitening? (http://www.sqlitening.com)

It's a client server version specifically for SQLite that is open source and free.

I'm not aware of any existing PureBasic Include file, but SQLitening has 3 different versions of API DLL's
so it should be easy to create.

Basic information on the API's provided...

SQLitening Database System

SQLitening is a client/server implementation of the very popular SQLite
database.
SQLitening is a programmer's library in standard Win32 DLL form. It is installed
as a standard Windows Service. In addition to client/server mode, the library
allows the programmer to also access SQLite databases in local mode. In either
mode (local or client/server), the database is extremely fast and robust.
Installation is a breeze - you simply copy a couple of DLL's to the folder where
your application resides. If you work in client/server mode, you create a folder on
your server and start the SQLitening Windows Service from the Administration
program. You may need to modify the standard configuration text file to set
permissions and port numbers/host names. Simple.

Which API or DLL should you use?
Basic: If you language supports passing OLE
strings both ByVal and ByRef then you should be able to use the Basic API. OLE
strings are allocated using the Win32 OLE string engine. There are some
routines in the Basic API that will probably only work with PowerBASIC. slSelAry
is one that passes an array and therefore may only work in PowerBASIC. You
may need to modify the include file (SQLitening.Inc) to support your language.

Special: If your language supports OLE string passing but only ByRef (Visual
Basic) then the Special API will probable work for you. The language must also
support optional parameter passing. The slSelAry is not available in this API.
You will need to modify the include file (SQLiteningS.Inc) to support your
language This DLL is a front-end to the Basic API.

Universal: If your language does not support OLE strings then the Special API
will probable work for you. This API will work for any language that can call a
standard Dll. The parameter passing is patterned after the Windows API. The
slSelAry is not available in this API. You will need to modify the include file
(SQLiteningU.Inc) to support your language This DLL is a front-end to the Basic
API. Documentation about parameter passing is located in the include file.

It appears to do everything you are looking for.

[jassing]

I've written a range of Procedures based on the native Win32 API functions which accomplish all this, and which have been working successfully for a few years in several hundred installations.

Any chance you'd share those? I do not want to re-invent the wheel if there's some tried & true routines already done... lack of (native) Shared / readonly file i/o has been a problem for me...

I just thought it would be useful to have had those functions as part of PureBasic's native commands.

+1 ...