File locking
Encyclopedia
File locking is a mechanism that restricts access to a computer file
by allowing only one user
or process access at any specific time. Systems implement locking to prevent the classic interceding update scenario (see race condition).
The following example illustrates the interceding update problem:
File locking prevents this problem by enforcing the serialization of update processes to any given file. Most operating system
s support the concept of record locking
, which means that individual records within any given file may be locked, thereby increasing the number of concurrent
update processes.
Database maintenance uses file locking, whereby it can serialize access to the entire physical file underlying a database. Although this does prevent any other process from accessing the file, it can be more efficient than individually locking a large number of regions in the file by removing the overhead of acquiring and releasing each lock.
Poor use of file locks, like any computer lock
, can result in poor performance or in deadlock
s.
Windows inherits the semantics of share-access controls from the MS-DOS
system, where sharing was introduced in MS–DOS 3.3. Thus, an application must explicitly allow sharing; otherwise an application has exclusive read, write, and delete access to the file (other types of access, such as those to retrieve the attributes of a file are allowed.)
For a file with shared access, applications may then use byte-range locking to control access to specific regions of the file. Such byte-range locks specify a region of the file (offset and length) and the type of lock (shared or exclusive). Note that the region of the file being locked is not required to have data within the file, and applications sometimes exploit this ability to implement their functionality.
For applications that use the file read/write APIs in Windows, byte-range locks are enforced (also referred to as mandatory locks) by the file systems that execute within Windows. For applications that use the file mapping APIs in Windows, byte-range locks are not enforced (also referred to as advisory locks.) Byte-range locking may also have other side-effects on the Windows system. For example, the Windows file-sharing mechanism will typically disable client side caching of a file for all clients when byte-range locks are used on any client to control file access. The client will observe slower access because read and write operations must be sent to the server where the file is stored.
Improper error-handling in an application program can lead to a scenario where a file is locked (either using "share" access or with byte-range file locking) and cannot be accessed by other applications. If so, the user may be able to restore file access by manually terminating the malfunctioning program. This is typically done through the Task Manager
utility.
The sharing mode parameter in the CreateFile function used to open files determines file-sharing. Files can be opened to allow sharing the file for read, write, or delete access. Subsequent attempts to open the file must be compatible with all previously granted sharing-access to the file. When the file is closed, sharing-access restrictions are adjusted to remove the restrictions imposed by that specific file open.
Byte-range locking type is determined by the dwFlags parameter in the LockFileEx function used to lock a region of a file. The Windows API
function LockFile can also be used and acquires an exclusive lock on the region of the file.
Any file that is executing on the computer system as a program (e.g., an EXE
, COM
, DLL
, CPL or other binary program file format) is normally prevented by the file system from being opened for write or delete access, reporting a sharing violation, despite the fact that the program is not opened by any application. However, some access is still allowed. For example, a running application file can be renamed or copied (read) even when executing.
Files are accessed by applications in Windows by using file handles
. These file handles can be explored with the Process Explorer
utility. This utility can also be used to force-close handles without needing to terminate the application holding them.
Microsoft Windows XP and Server 2003
editions have introduced volume snapshot
(VSS) capability to NTFS
, allowing open files to be accessed by backup software
despite any exclusive locks. However, unless software is rewritten to specifically support this feature, the snapshot will be crash consistent only, while properly supported applications can assist the operating system in creating "transactionally consistent" snapshots. Other commercial software for accessing locked files under Windows include File Access Manager and Open File Manager. These work by installing their own drivers
to access the files in kernel mode.
Two kinds of locks are offered: shared locks and exclusive locks. In the case of fcntl, different kinds of locks may be applied to different sections (byte ranges) of a file, or else to the whole file. Shared locks can be acquired by an unlimited number of processes at the same time, but an exclusive lock can only be acquired by one process, and cannot coexist with a shared lock. To acquire a shared lock, a process must wait until no processes hold any exclusive locks. To acquire an exclusive lock, a process must wait until no processes hold either kind of lock. Unlike locks created by fcntl, those created by flock are preserved across forks, making them useful in forking servers. It is therefore possible for more than one process to hold an exclusive lock on the same file, provided these processes share a filial relationship and the exclusive lock was initially created in a single process before being duplicated across a fork.
Shared locks are sometimes called "read locks" and exclusive locks are sometimes called "write locks". However, because locks on Unix are advisory, this isn't enforced. Thus it is possible for a database to have a concept of "shared writes" vs. "exclusive writes"; for example, changing a field in place may be permitted under shared access, whereas garbage-collecting and rewriting the database may require exclusive access.
File locks apply to the actual file, rather than the file name. This is important since Unix allows multiple names to refer to the same file. Together with non-mandatory locking, this leads to great flexibility in accessing files from multiple or many processes. On the other hand, the cooperative locking approach can lead to problems when a process writes to a file without obeying file locks set by other processes.
For this reason, some Unix operating systems also offer limited support for mandatory locking. On such systems, a file whose setgid bit is on but whose group execution bit is off when that file is opened will be subject to automatic mandatory locking if the underlying filesystem supports it. However, non-local NFS partitions tend to disregard this bit. This is an exception to the earlier statement that Unix does not automatically lock open files. This strategy first originated in System V, and can be seen today in the Solaris, HP-UX, and Linux operating systems. It is not part of POSIX, however, and BSD-derived operating systems like FreeBSD, OpenBSD, NetBSD, and Apple's Mac OS X do not support it.
More than one process can hold an exclusive flock on a given file if the exclusive lock was duplicated across a later fork. This simplifies coding for network servers and helps prevent race conditions, but can be confusing to the unwary.
Mandatory locks have no effect on the unlink system call. As a result, certain programs may, effectively, circumvent mandatory locking.
The authors of Advanced Programming in the UNIX Environment (Second Edition) observed that the ed editor did so (page 456).
Whether and how flock locks work on network filesystems, such as NFS, is implementation dependent. On BSD systems, flock calls on a file descriptor open to a file on an NFS-mounted partition are successful no-ops
. On Linux
prior to 2.6.12, flock calls on NFS files would act only locally. Kernel 2.6.12 and above implement flock calls on NFS files using POSIX byte-range locks. These locks will be visible to other NFS clients that implement fcntl-style POSIX locks, but invisibile to those that do not.
Lock upgrades and downgrades release the old lock before applying the new lock. If an application downgrades an exclusive lock to a shared lock while another application is blocked waiting for an exclusive lock, the latter application will get the exclusive lock and lock the first application out.
All fcntl locks associated with a file for a given process are removed when any file descriptor for that file is closed by that process, even if a lock was never requested for that file descriptor. Also, fcntl locks are not inherited by a child process. The fcntl close semantics are particularly troublesome for applications that call subroutine libraries that may access files. Neither of these "bugs" occurs using real flock-style locks.
Preservation of the lock status on open file descriptors passed to another process using a Unix domain socket
is implementation dependent.
2.4 and later added notification of external changes to files with dnotify
mechanism (through the F_NOTIFY parameter in fcntl). Kernels from Linux 2.6.13 replaced and enhanced this mechanism with inotify
. Linux also supports mandatory locking through the special " -o mand" parameter for filesystem mounting, but this is rarely used.
On Amiga OS
A lock on a file (or directory) can be acquired using the Lock function (in the dos.library). A lock can be shared (other processes can read the file/directory, but can't modify or delete it), or exclusive so that only the process which successfully acquires the lock can access or modify the object. The lock is on the whole object and not part of it. The lock must be released with the UnLock function: unlike in Unix, the operating system does not implicitly unlock the object when the process terminates.
s and other programs often use a strategy similar to the use of file locking: creation of lock files, which are files whose contents are irrelevant (although often one will find the process identifier
of the holder of the lock in the file) and whose sole purpose is to signal by their presence that some resource is locked. A lock file is often the best approach if the resource to be controlled is not a regular file at all, so using methods for locking files does not apply. For example, a lock file might govern access to a set of related resources, such as several different files, directories, a group of disk partitions, or selected access to higher level protocols like servers or database connections.
When using lock files, care must be taken to ensure that operations are atomic. To obtain a lock, the process must verify that the lock file does not exist and then create it, whilst preventing another process from creating it in the meantime. Various methods to do this include:
Certain Mozilla
products (such as Firefox, Thunderbird, Sunbird) use this type of file resource lock mechanism (using a temporary file named "parent.lock".)
Computer file
A computer file is a block of arbitrary information, or resource for storing information, which is available to a computer program and is usually based on some kind of durable storage. A file is durable in the sense that it remains available for programs to use after the current program has finished...
by allowing only one user
User (computing)
A user is an agent, either a human agent or software agent, who uses a computer or network service. A user often has a user account and is identified by a username , screen name , nickname , or handle, which is derived from the identical Citizen's Band radio term.Users are...
or process access at any specific time. Systems implement locking to prevent the classic interceding update scenario (see race condition).
The following example illustrates the interceding update problem:
- Process A reads a customer record from a file containing account information, including the customer's account balance and phone number.
- Process B now reads the same record from the same file so it has its own copy.
- Process A changes the account balance in its copy of the customer record and writes the record back to the file.
- Process B, which still has the original stale value for the account balance in its copy of the customer record, updates the customer's phone number and writes the customer record back to the file.
- Process B has now written its stale account-balance value to the file, causing the changes made by process A to be lost.
File locking prevents this problem by enforcing the serialization of update processes to any given file. Most operating system
Operating system
An operating system is a set of programs that manage computer hardware resources and provide common services for application software. The operating system is the most important type of system software in a computer system...
s support the concept of record locking
Record locking
Record locking is the technique of preventing simultaneous access to data in a database, to prevent inconsistent results.The classic example is demonstrated by two bank clerks attempting to update the same bank account for two different transactions. Clerks 1 and 2 both retrieve the account's...
, which means that individual records within any given file may be locked, thereby increasing the number of concurrent
Concurrency (computer science)
In computer science, concurrency is a property of systems in which several computations are executing simultaneously, and potentially interacting with each other...
update processes.
Database maintenance uses file locking, whereby it can serialize access to the entire physical file underlying a database. Although this does prevent any other process from accessing the file, it can be more efficient than individually locking a large number of regions in the file by removing the overhead of acquiring and releasing each lock.
Poor use of file locks, like any computer lock
Lock (computer science)
In computer science, a lock is a synchronization mechanism for enforcing limits on access to a resource in an environment where there are many threads of execution. Locks are one way of enforcing concurrency control policies.-Types:...
, can result in poor performance or in deadlock
Deadlock
A deadlock is a situation where in two or more competing actions are each waiting for the other to finish, and thus neither ever does. It is often seen in a paradox like the "chicken or the egg"...
s.
In mainframes
IBM pioneered file locking in 1963 for use in mainframe computers using OS/360, where it was termed "exclusive control".In Microsoft Windows
Microsoft Windows uses three distinct mechanisms to manage access to shared files:- using share access controlsAccess controlAccess control refers to exerting control over who can interact with a resource. Often but not always, this involves an authority, who does the controlling. The resource can be a given building, group of buildings, or computer-based information system...
that allow applications to specify whole-file access-sharing for read, write, or delete - using byte-range locks to arbitrate read and write access to regions within a single file
- by Windows file systems disallowing executing files from being opened for write or delete access
Windows inherits the semantics of share-access controls from the MS-DOS
MS-DOS
MS-DOS is an operating system for x86-based personal computers. It was the most commonly used member of the DOS family of operating systems, and was the main operating system for IBM PC compatible personal computers during the 1980s to the mid 1990s, until it was gradually superseded by operating...
system, where sharing was introduced in MS–DOS 3.3. Thus, an application must explicitly allow sharing; otherwise an application has exclusive read, write, and delete access to the file (other types of access, such as those to retrieve the attributes of a file are allowed.)
For a file with shared access, applications may then use byte-range locking to control access to specific regions of the file. Such byte-range locks specify a region of the file (offset and length) and the type of lock (shared or exclusive). Note that the region of the file being locked is not required to have data within the file, and applications sometimes exploit this ability to implement their functionality.
For applications that use the file read/write APIs in Windows, byte-range locks are enforced (also referred to as mandatory locks) by the file systems that execute within Windows. For applications that use the file mapping APIs in Windows, byte-range locks are not enforced (also referred to as advisory locks.) Byte-range locking may also have other side-effects on the Windows system. For example, the Windows file-sharing mechanism will typically disable client side caching of a file for all clients when byte-range locks are used on any client to control file access. The client will observe slower access because read and write operations must be sent to the server where the file is stored.
Improper error-handling in an application program can lead to a scenario where a file is locked (either using "share" access or with byte-range file locking) and cannot be accessed by other applications. If so, the user may be able to restore file access by manually terminating the malfunctioning program. This is typically done through the Task Manager
Windows Task Manager
Windows Task Manager is a task manager application included with the Microsoft Windows NT family of operating systems that provides detailed information about computer performance and running applications, processes and CPU usage, commit charge and memory information, network activity and...
utility.
The sharing mode parameter in the CreateFile function used to open files determines file-sharing. Files can be opened to allow sharing the file for read, write, or delete access. Subsequent attempts to open the file must be compatible with all previously granted sharing-access to the file. When the file is closed, sharing-access restrictions are adjusted to remove the restrictions imposed by that specific file open.
Byte-range locking type is determined by the dwFlags parameter in the LockFileEx function used to lock a region of a file. The Windows API
Windows API
The Windows API, informally WinAPI, is Microsoft's core set of application programming interfaces available in the Microsoft Windows operating systems. It was formerly called the Win32 API; however, the name "Windows API" more accurately reflects its roots in 16-bit Windows and its support on...
function LockFile can also be used and acquires an exclusive lock on the region of the file.
Any file that is executing on the computer system as a program (e.g., an EXE
EXE
EXE is the common filename extension denoting an executable file in the DOS, OpenVMS, Microsoft Windows, Symbian, and OS/2 operating systems....
, COM
COM file
In many computer operating systems, a COM file is a type of executable file; the name is derived from the file name extension .COM. Originally, the term stood for "Command file", a text file containing commands to be issued to the operating system , on many of the Digital Equipment Corporation mini...
, DLL
Dynamic-link library
Dynamic-link library , or DLL, is Microsoft's implementation of the shared library concept in the Microsoft Windows and OS/2 operating systems...
, CPL or other binary program file format) is normally prevented by the file system from being opened for write or delete access, reporting a sharing violation, despite the fact that the program is not opened by any application. However, some access is still allowed. For example, a running application file can be renamed or copied (read) even when executing.
Files are accessed by applications in Windows by using file handles
File descriptor
In computer programming, a file descriptor is an abstract indicator for accessing a file. The term is generally used in POSIX operating systems...
. These file handles can be explored with the Process Explorer
Process Explorer
Process Explorer is a freeware computer program for Microsoft Windows created by Sysinternals, which has been acquired by Microsoft Corporation....
utility. This utility can also be used to force-close handles without needing to terminate the application holding them.
Microsoft Windows XP and Server 2003
Windows Server 2003
Windows Server 2003 is a server operating system produced by Microsoft, introduced on 24 April 2003. An updated version, Windows Server 2003 R2, was released to manufacturing on 6 December 2005...
editions have introduced volume snapshot
Volume Shadow Copy Service
Shadow Copy , is a technology included in Microsoft Windows that allows taking manual or automatic backup copies or snapshots of data, even if it has a lock, on a specific volume at a specific point in time over regular intervals...
(VSS) capability to NTFS
NTFS
NTFS is the standard file system of Windows NT, including its later versions Windows 2000, Windows XP, Windows Server 2003, Windows Server 2008, Windows Vista, and Windows 7....
, allowing open files to be accessed by backup software
Backup software
Backup software are computer programs used to perform backup; they create supplementary exact copies of files, databases or entire computers. These programs may later use the supplementary copies to restore the original contents in the event of data loss....
despite any exclusive locks. However, unless software is rewritten to specifically support this feature, the snapshot will be crash consistent only, while properly supported applications can assist the operating system in creating "transactionally consistent" snapshots. Other commercial software for accessing locked files under Windows include File Access Manager and Open File Manager. These work by installing their own drivers
Device driver
In computing, a device driver or software driver is a computer program allowing higher-level computer programs to interact with a hardware device....
to access the files in kernel mode.
In Unix
The Unix operating systems (including Linux and Apple's Mac OS X, sometimes called Darwin) do not normally automatically lock open files or running programs. Several kinds of file-locking mechanisms are available in different flavors of Unix, and many operating systems support more than one kind for compatibility. The two most common mechanisms are and . A third such mechanism is , which may be separate or may be implemented using either of the first two primitives. Although some types of locks can be configured to be mandatory, file locks under Unix are by default advisory. This means that cooperating processes may use locks to coordinate access to a file among themselves, but uncooperative processes are also free to ignore locks and access the file in any way they choose. In other words, file locks lock out other file lockers only, not I/O.Two kinds of locks are offered: shared locks and exclusive locks. In the case of fcntl, different kinds of locks may be applied to different sections (byte ranges) of a file, or else to the whole file. Shared locks can be acquired by an unlimited number of processes at the same time, but an exclusive lock can only be acquired by one process, and cannot coexist with a shared lock. To acquire a shared lock, a process must wait until no processes hold any exclusive locks. To acquire an exclusive lock, a process must wait until no processes hold either kind of lock. Unlike locks created by fcntl, those created by flock are preserved across forks, making them useful in forking servers. It is therefore possible for more than one process to hold an exclusive lock on the same file, provided these processes share a filial relationship and the exclusive lock was initially created in a single process before being duplicated across a fork.
Shared locks are sometimes called "read locks" and exclusive locks are sometimes called "write locks". However, because locks on Unix are advisory, this isn't enforced. Thus it is possible for a database to have a concept of "shared writes" vs. "exclusive writes"; for example, changing a field in place may be permitted under shared access, whereas garbage-collecting and rewriting the database may require exclusive access.
File locks apply to the actual file, rather than the file name. This is important since Unix allows multiple names to refer to the same file. Together with non-mandatory locking, this leads to great flexibility in accessing files from multiple or many processes. On the other hand, the cooperative locking approach can lead to problems when a process writes to a file without obeying file locks set by other processes.
For this reason, some Unix operating systems also offer limited support for mandatory locking. On such systems, a file whose setgid bit is on but whose group execution bit is off when that file is opened will be subject to automatic mandatory locking if the underlying filesystem supports it. However, non-local NFS partitions tend to disregard this bit. This is an exception to the earlier statement that Unix does not automatically lock open files. This strategy first originated in System V, and can be seen today in the Solaris, HP-UX, and Linux operating systems. It is not part of POSIX, however, and BSD-derived operating systems like FreeBSD, OpenBSD, NetBSD, and Apple's Mac OS X do not support it.
Problems
Both flock and fcntl have quirks that occasionally puzzle programmers more familiar with other operating systems.More than one process can hold an exclusive flock on a given file if the exclusive lock was duplicated across a later fork. This simplifies coding for network servers and helps prevent race conditions, but can be confusing to the unwary.
Mandatory locks have no effect on the unlink system call. As a result, certain programs may, effectively, circumvent mandatory locking.
The authors of Advanced Programming in the UNIX Environment (Second Edition) observed that the ed editor did so (page 456).
Whether and how flock locks work on network filesystems, such as NFS, is implementation dependent. On BSD systems, flock calls on a file descriptor open to a file on an NFS-mounted partition are successful no-ops
NOP
In computer science, NOP or NOOP is an assembly language instruction, sequence of programming language statements, or computer protocol command that effectively does nothing at all....
. On Linux
Linux
Linux is a Unix-like computer operating system assembled under the model of free and open source software development and distribution. The defining component of any Linux system is the Linux kernel, an operating system kernel first released October 5, 1991 by Linus Torvalds...
prior to 2.6.12, flock calls on NFS files would act only locally. Kernel 2.6.12 and above implement flock calls on NFS files using POSIX byte-range locks. These locks will be visible to other NFS clients that implement fcntl-style POSIX locks, but invisibile to those that do not.
Lock upgrades and downgrades release the old lock before applying the new lock. If an application downgrades an exclusive lock to a shared lock while another application is blocked waiting for an exclusive lock, the latter application will get the exclusive lock and lock the first application out.
All fcntl locks associated with a file for a given process are removed when any file descriptor for that file is closed by that process, even if a lock was never requested for that file descriptor. Also, fcntl locks are not inherited by a child process. The fcntl close semantics are particularly troublesome for applications that call subroutine libraries that may access files. Neither of these "bugs" occurs using real flock-style locks.
Preservation of the lock status on open file descriptors passed to another process using a Unix domain socket
Unix domain socket
A Unix domain socket or IPC socket is a data communications endpoint for exchanging data between processes executing within the same host operating system. While similar in functionality to...
is implementation dependent.
Linux
LinuxLinux kernel
The Linux kernel is an operating system kernel used by the Linux family of Unix-like operating systems. It is one of the most prominent examples of free and open source software....
2.4 and later added notification of external changes to files with dnotify
Dnotify
dnotify is a file system event monitor for the Linux kernel, one of the subfeatures of the fcntl call. It was introduced in the 2.4 kernel series...
mechanism (through the F_NOTIFY parameter in fcntl). Kernels from Linux 2.6.13 replaced and enhanced this mechanism with inotify
Inotify
inotify is a Linux kernel subsystem that acts to extend filesystems to notice changes to the filesystem, and report those changes to applications. It replaces an earlier facility, dnotify, which had similar goals....
. Linux also supports mandatory locking through the special " -o mand" parameter for filesystem mounting, but this is rarely used.
On Amiga OSAmigaThe Amiga is a family of personal computers that was sold by Commodore in the 1980s and 1990s. The first model was launched in 1985 as a high-end home computer and became popular for its graphical, audio and multi-tasking abilities...
A lock on a file (or directory) can be acquired using the Lock function (in the dos.library). A lock can be shared (other processes can read the file/directory, but can't modify or delete it), or exclusive so that only the process which successfully acquires the lock can access or modify the object. The lock is on the whole object and not part of it. The lock must be released with the UnLock function: unlike in Unix, the operating system does not implicitly unlock the object when the process terminates.Lock files
Shell scriptShell script
A shell script is a script written for the shell, or command line interpreter, of an operating system. It is often considered a simple domain-specific programming language...
s and other programs often use a strategy similar to the use of file locking: creation of lock files, which are files whose contents are irrelevant (although often one will find the process identifier
Process identifier
In computing, the process identifier is a number used by most operating system kernels to uniquely identify a process...
of the holder of the lock in the file) and whose sole purpose is to signal by their presence that some resource is locked. A lock file is often the best approach if the resource to be controlled is not a regular file at all, so using methods for locking files does not apply. For example, a lock file might govern access to a set of related resources, such as several different files, directories, a group of disk partitions, or selected access to higher level protocols like servers or database connections.
When using lock files, care must be taken to ensure that operations are atomic. To obtain a lock, the process must verify that the lock file does not exist and then create it, whilst preventing another process from creating it in the meantime. Various methods to do this include:
- Using the lockfile command (a conditional semaphore-file creator distributed in the procmail package).
- System calls that create a file, but fail if the file already exists. (System calls are available from languages such as C or C++, and shell scripts can make use of noclobber)
- Using the mkdir command and checking the exit code for failure
Certain Mozilla
Mozilla
Mozilla is a term used in a number of ways in relation to the Mozilla.org project and the Mozilla Foundation, their defunct commercial predecessor Netscape Communications Corporation, and their related application software....
products (such as Firefox, Thunderbird, Sunbird) use this type of file resource lock mechanism (using a temporary file named "parent.lock".)
Unlocker software
An unlocker is a utility used to determine what process is locking a file, and displays a list of processes as well as choices on what to do with the process (kill task, unlock, etc.) along with a list of file options such as delete or rename. On Unix and Unix-like systems, utilities such as the fstat and utilities can be used to inspect the state of file locks by process, by filename, or both.External links
- Everything you never wanted to know about file locking, a review of the Unix file locking options and their problems (dated December 13, 2010)