Large file support
Encyclopedia
Large file support, often abbreviated to LFS, is the term frequently applied to the ability to create files larger than 2 GiB
on 32-bit operating system
s.
implementations used 32-bit
integer
s to represent file
sizes and positions. Consequently no file could be larger than 232 bytes (4 GiB). The problem was exacerbated by treating the sizes as signed
numbers, which further lowered the limit to 231 bytes (2 GiB). Files larger than 2 GiB, too large for 32-bit operating systems to handle, came to be known as large files.
While the 2 GiB limit was quite acceptable at a time when hard disk
s were smaller, the general increase in storage capacity combined with increased server and desktop file usage, especially for database
and multimedia
files, led to intense pressure for OS vendors to remove the limitation.
In 1996, multiple vendors responded by forming an industry initiative known as the Large File Summit (thus "LFS" can be considered to stand for either "Large file support" or "Large File Summit"), tasked to define a standardized way to switch to 64-bit
numbers to represent file sizes. (Merely ensuring the sizes were treated as unsigned
numbers would only up the limit from 2 GiB to 4 GiB, which would have been only a stopgap measure given the explosive growth in data storage.) It is worth noting that 64-bit operating systems such as Tru64 UNIX
never had a 32-bit limit to begin with, and hence needed no additional "large file support".
This switch caused deployment issues and required design choices the consequences of which can still be seen:
Gibibyte
The gibibyte is a standards-based binary multiple of the byte, a unit of digital information storage. The gibibyte unit symbol is GiB....
on 32-bit operating system
Operating system
An operating system is a set of programs that manage computer hardware resources and provide common services for application software. The operating system is the most important type of system software in a computer system...
s.
Rationale
Traditionally, many operating systems and their underlying file systemFile system
A file system is a means to organize data expected to be retained after a program terminates by providing procedures to store, retrieve and update data, as well as manage the available space on the device which contain it. A file system organizes data in an efficient manner and is tuned to the...
implementations used 32-bit
32-bit
The range of integer values that can be stored in 32 bits is 0 through 4,294,967,295. Hence, a processor with 32-bit memory addresses can directly access 4 GB of byte-addressable memory....
integer
Integer
The integers are formed by the natural numbers together with the negatives of the non-zero natural numbers .They are known as Positive and Negative Integers respectively...
s to represent file
Computer file
A computer file is a block of arbitrary information, or resource for storing information, which is available to a computer program and is usually based on some kind of durable storage. A file is durable in the sense that it remains available for programs to use after the current program has finished...
sizes and positions. Consequently no file could be larger than 232 bytes (4 GiB). The problem was exacerbated by treating the sizes as signed
Signedness
In computing, signedness is a property of data types representing numbers in computer programs. A numeric variable is signed if it can represent both positive and negative numbers, and unsigned if it can only represent non-negative numbers .As signed numbers can represent negative numbers, they...
numbers, which further lowered the limit to 231 bytes (2 GiB). Files larger than 2 GiB, too large for 32-bit operating systems to handle, came to be known as large files.
While the 2 GiB limit was quite acceptable at a time when hard disk
Hard disk
A hard disk drive is a non-volatile, random access digital magnetic data storage device. It features rotating rigid platters on a motor-driven spindle within a protective enclosure. Data is magnetically read from and written to the platter by read/write heads that float on a film of air above the...
s were smaller, the general increase in storage capacity combined with increased server and desktop file usage, especially for database
Database
A database is an organized collection of data for one or more purposes, usually in digital form. The data are typically organized to model relevant aspects of reality , in a way that supports processes requiring this information...
and multimedia
Multimedia
Multimedia is media and content that uses a combination of different content forms. The term can be used as a noun or as an adjective describing a medium as having multiple content forms. The term is used in contrast to media which use only rudimentary computer display such as text-only, or...
files, led to intense pressure for OS vendors to remove the limitation.
In 1996, multiple vendors responded by forming an industry initiative known as the Large File Summit (thus "LFS" can be considered to stand for either "Large file support" or "Large File Summit"), tasked to define a standardized way to switch to 64-bit
64-bit
64-bit is a word size that defines certain classes of computer architecture, buses, memory and CPUs, and by extension the software that runs on them. 64-bit CPUs have existed in supercomputers since the 1970s and in RISC-based workstations and servers since the early 1990s...
numbers to represent file sizes. (Merely ensuring the sizes were treated as unsigned
Signedness
In computing, signedness is a property of data types representing numbers in computer programs. A numeric variable is signed if it can represent both positive and negative numbers, and unsigned if it can only represent non-negative numbers .As signed numbers can represent negative numbers, they...
numbers would only up the limit from 2 GiB to 4 GiB, which would have been only a stopgap measure given the explosive growth in data storage.) It is worth noting that 64-bit operating systems such as Tru64 UNIX
Tru64 UNIX
Tru64 UNIX is a 64-bit UNIX operating system for the Alpha instruction set architecture , currently owned by Hewlett-Packard . Previously, Tru64 UNIX was a product of Compaq, and before that, Digital Equipment Corporation , where it was known as Digital UNIX .As its original name suggests, Tru64...
never had a 32-bit limit to begin with, and hence needed no additional "large file support".
This switch caused deployment issues and required design choices the consequences of which can still be seen:
- The change to 64-bit file sizes frequently required incompatible changes to file system layout, which meant that large file support sometimes necessitated a file system change. For example, Microsoft WindowsMicrosoft WindowsMicrosoft Windows is a series of operating systems produced by Microsoft.Microsoft introduced an operating environment named Windows on November 20, 1985 as an add-on to MS-DOS in response to the growing interest in graphical user interfaces . Microsoft Windows came to dominate the world's personal...
' FAT32File Allocation TableFile Allocation Table is a computer file system architecture now widely used on many computer systems and most memory cards, such as those used with digital cameras. FAT file systems are commonly found on floppy disks, flash memory cards, digital cameras, and many other portable devices because of...
file system does not support files larger than 4 GiB; one has to use NTFSNTFSNTFS is the standard file system of Windows NT, including its later versions Windows 2000, Windows XP, Windows Server 2003, Windows Server 2008, Windows Vista, and Windows 7....
instead. - To support binary compatibilityBackward compatibilityIn the context of telecommunications and computing, a device or technology is said to be backward or downward compatible if it can work with input generated by an older device...
with old applicationsApplication softwareApplication software, also known as an application or an "app", is computer software designed to help the user to perform specific tasks. Examples include enterprise software, accounting software, office suites, graphics software and media players. Many application programs deal principally with...
, operating system interfacesApplication programming interfaceAn application programming interface is a source code based specification intended to be used as an interface by software components to communicate with each other...
had to retain their use of 32-bit file sizes and new interfaces had to be designed specifically for large file support. - To support writing portablePortingIn computer science, porting is the process of adapting software so that an executable program can be created for a computing environment that is different from the one for which it was originally designed...
code that makes use of LFS where possible, C standard libraryC standard libraryThe C Standard Library is the standard library for the programming language C, as specified in the ANSI C standard.. It was developed at the same time as the C POSIX library, which is basically a superset of it...
authors devised mechanisms that, depending on preprocessorC preprocessorThe C preprocessor is the preprocessor for the C and C++ computer programming languages. The preprocessor handles directives for source file inclusion , macro definitions , and conditional inclusion ....
constants, transparently redefined the functions to the 64-bit large file aware ones. - Many old interfaces, especially CC (programming language)C is a general-purpose computer programming language developed between 1969 and 1973 by Dennis Ritchie at the Bell Telephone Laboratories for use with the Unix operating system....
-based ones, explicitly specified argument types in a way that did not allow straightforward nor transparent transition to 64-bit types. For example, the C functionsfseekandftelloperate on file positions of typelong int, which is typically 32 bits wide on 32-bit architecture, and cannot be made larger without sacrificing backward compatibility. (This was resolved by introducing new functionsfseekoandftello. On Windows machines, under Visual C++, functions_fseeki64and_ftelli64are used.) - The above efforts notwithstanding, all applications had to be recompiledCompilerA compiler is a computer program that transforms source code written in a programming language into another computer language...
to make them LFS-aware. The resulting binariesExecutableIn computing, an executable file causes a computer "to perform indicated tasks according to encoded instructions," as opposed to a data file that must be parsed by a program to be meaningful. These instructions are traditionally machine code instructions for a physical CPU...
were typically not runnable on older releases of the same operating system. This was, and to some extent still remains, a problem for some application vendors.