Clustered file system
Encyclopedia
A clustered file system is a file system
which is shared by being simultaneously mounted
on multiple servers
. There are several approaches to clustering, most of which do not employ a clustered file system. While many computer clusters don't use clustered file systems, unless servers are underpinned by a clustered file system the complexity of the underlying storage environment increases as servers are added.
(SAN) or RAID
to provide direct disk access from multiple computers at the block level
. Translation from file-level operations that applications use to block-level operations used by the SAN must take place on the client node. The most common type of clustered file system, a shared disk file system adds a mechanism for concurrency control
which gives a consistent and serializable
view of the file system, avoiding corruption and unintended data loss
even when multiple clients try to access the same files at the same time. Shared disk file systems also usually employ some sort of a fencing
mechanism to prevent data corruption in case of node failures.
The underlying storage area network
might use any of a number of block-level protocols, including SCSI
, iSCSI
, HyperSCSI
, ATA over Ethernet
(AoE), Fibre Channel
, and InfiniBand
.
There are different architectural approaches to a shared disk file system. Some distribute file information across all the servers in a cluster (fully distributed). Others utilize a centralized metadata
server. Both achieve the same result of enabling all servers to access all the data on a shared storage device.
will be using a 4 Gbit/s Fibre Channel SAN
with a clustered file system to store the massive amount of data generated by the experiment (estimated at 1 GB/second for a month). Reasons quoted for this choice include "performance, scalability and vendor independence"
systems), SMB/CIFS (Server Message Block/Common Internet File System) (used with MS Windows systems), or AFP
(used with Apple Macintosh
computers).
that can result in data loss. To avoid this problem, a shared nothing architecture
can be employed. Each storage node communicates changes to other nodes or to a master, for replication purposes. If a single disk fails, other copies can be used to reconstruct or replace it on the fly so "nothing" is lost. To enable this feature, clients must be unaware of the physical location of a file. A single global file system is presented to clients, so the file system itself deals with allocations and low-level failures. Examples of this type of file system are found in products such as Ceph
, Lustre
, Isilon, IBRIX Fusion
, and Hadoop
.
's TOPS-20
and VAX
/VMS
clusters included shared disk filesystems.
File system
A file system is a means to organize data expected to be retained after a program terminates by providing procedures to store, retrieve and update data, as well as manage the available space on the device which contain it. A file system organizes data in an efficient manner and is tuned to the...
which is shared by being simultaneously mounted
Mount (computing)
Mounting takes place before a computer can use any kind of storage device . The user or their operating system must make it accessible through the computer's file system. A user can access only files on mounted media.- Mount point :A mount point is a physical location in the partition used as a...
on multiple servers
Server (computing)
In the context of client-server architecture, a server is a computer program running to serve the requests of other programs, the "clients". Thus, the "server" performs some computational task on behalf of "clients"...
. There are several approaches to clustering, most of which do not employ a clustered file system. While many computer clusters don't use clustered file systems, unless servers are underpinned by a clustered file system the complexity of the underlying storage environment increases as servers are added.
Shared-disk
A shared disk file system uses a storage area networkStorage area network
A storage area network is a dedicated network that provides access to consolidated, block level data storage. SANs are primarily used to make storage devices, such as disk arrays, tape libraries, and optical jukeboxes, accessible to servers so that the devices appear like locally attached devices...
(SAN) or RAID
RAID
RAID is a storage technology that combines multiple disk drive components into a logical unit...
to provide direct disk access from multiple computers at the block level
Block (data storage)
In computing , a block is a sequence of bytes or bits, having a nominal length . Data thus structured are said to be blocked. The process of putting data into blocks is called blocking. Blocking is used to facilitate the handling of the data-stream by the computer program receiving the data...
. Translation from file-level operations that applications use to block-level operations used by the SAN must take place on the client node. The most common type of clustered file system, a shared disk file system adds a mechanism for concurrency control
Concurrency control
In information technology and computer science, especially in the fields of computer programming , operating systems , multiprocessors, and databases, concurrency control ensures that correct results for concurrent operations are generated, while getting those results as quickly as possible.Computer...
which gives a consistent and serializable
Serialization
In computer science, in the context of data storage and transmission, serialization is the process of converting a data structure or object state into a format that can be stored and "resurrected" later in the same or another computer environment...
view of the file system, avoiding corruption and unintended data loss
Data loss
Data loss is an error condition in information systems in which information is destroyed by failures or neglect in storage, transmission, or processing. Information systems implement backup and disaster recovery equipment and processes to prevent data loss or restore lost data.Data loss is...
even when multiple clients try to access the same files at the same time. Shared disk file systems also usually employ some sort of a fencing
Fencing (computing)
Fencing is the process of isolating a node of a computer cluster when the former is malfunctioning. Isolating a node means ensuring that I/O can no longer be done from it. Fencing is typically done automatically, by cluster infrastructure such as shared disk file systems, in order to protect...
mechanism to prevent data corruption in case of node failures.
The underlying storage area network
Storage area network
A storage area network is a dedicated network that provides access to consolidated, block level data storage. SANs are primarily used to make storage devices, such as disk arrays, tape libraries, and optical jukeboxes, accessible to servers so that the devices appear like locally attached devices...
might use any of a number of block-level protocols, including SCSI
SCSI
Small Computer System Interface is a set of standards for physically connecting and transferring data between computers and peripheral devices. The SCSI standards define commands, protocols, and electrical and optical interfaces. SCSI is most commonly used for hard disks and tape drives, but it...
, iSCSI
ISCSI
In computing, iSCSI , is an abbreviation of Internet Small Computer System Interface, an Internet Protocol -based storage networking standard for linking data storage facilities. By carrying SCSI commands over IP networks, iSCSI is used to facilitate data transfers over intranets and to manage...
, HyperSCSI
HyperSCSI
HyperSCSI was a computer network protocol for accessing storage by sending and receiving SCSI commands.It was developed by researchers at the Data Storage Institute in Singapore in 2000 to 2003....
, ATA over Ethernet
ATA over Ethernet
ATA over Ethernet is a network protocol developed by the Brantley Coile Company, designed for simple, high-performance access of SATA storage devices over Ethernet networks. It is used to build storage area networks with low-cost, standard technologies.- Protocol description :AoE runs on layer 2...
(AoE), Fibre Channel
Fibre Channel
Fibre Channel, or FC, is a gigabit-speed network technology primarily used for storage networking. Fibre Channel is standardized in the T11 Technical Committee of the InterNational Committee for Information Technology Standards , an American National Standards Institute –accredited standards...
, and InfiniBand
InfiniBand
InfiniBand is a switched fabric communications link used in high-performance computing and enterprise data centers. Its features include high throughput, low latency, quality of service and failover, and it is designed to be scalable...
.
There are different architectural approaches to a shared disk file system. Some distribute file information across all the servers in a cluster (fully distributed). Others utilize a centralized metadata
Metadata
The term metadata is an ambiguous term which is used for two fundamentally different concepts . Although the expression "data about data" is often used, it does not apply to both in the same way. Structural metadata, the design and specification of data structures, cannot be about data, because at...
server. Both achieve the same result of enabling all servers to access all the data on a shared storage device.
Usage
Scientists working at ALICEA Large Ion Collider Experiment
ALICE is one of the six detector experiments at the Large Hadron Collider at CERN. The other five are: ATLAS, CMS, TOTEM, LHCb, and LHCf. ALICE is optimized to study heavy ion collisions. Pb-Pb nuclei collisions will be studied at a centre of mass energy of 2.76 TeV per nucleon...
will be using a 4 Gbit/s Fibre Channel SAN
Storage area network
A storage area network is a dedicated network that provides access to consolidated, block level data storage. SANs are primarily used to make storage devices, such as disk arrays, tape libraries, and optical jukeboxes, accessible to servers so that the devices appear like locally attached devices...
with a clustered file system to store the massive amount of data generated by the experiment (estimated at 1 GB/second for a month). Reasons quoted for this choice include "performance, scalability and vendor independence"
Examples
- Veritas Cluster File System HA
- DataPlow NasanNasanNasan is a shared disk file system created by DataPlow, that features the ability to give shared, block-level access to a disk volume from multiple client computers without causing file system corruption. DataPlow sells their software directly on a per-seat basis. Each client must purchase the...
File System - DataPlow SAN File System (SFS)
- IBM General Parallel File System (GPFS)
- Microsoft Cluster Shared VolumesCluster Shared VolumesCluster Shared Volumes is a feature of Failover Clustering first introduced in Windows Server 2008 R2 for use with the Hyper-V role. A Cluster Shared Volume is a shared disk containing an NTFS volume that is made accessible for read and write operations by all nodes within a Windows Server...
(CSV) - Oracle Cluster File SystemOCFSOCFS is a shared disk file system developed by Oracle Corporation and released under the GNU General Public License....
(OCFS) - PolyServe storage solutions
- Quantum StorNextStorNext File SystemStorNext File System is a shared disk file system made by Quantum Corporation. It is installed on hosts that are connected to the same disk array in a storage area network . Client systems are not required to run the same operating system to access a shared filesystem containing StorNext data...
FileSystem (SNFS), ex ADIC, ex CentraVision FileSystem (CVFS) - Blue Whale Clustered file systemBlue Whale Clustered file systemBlue Whale Clustered file system is a shared disk file system made by Tianjin Zhongke Blue Whale Information Technologies Company in China.-Overview:...
(BWFS) - Silicon GraphicsSilicon GraphicsSilicon Graphics, Inc. was a manufacturer of high-performance computing solutions, including computer hardware and software, founded in 1981 by Jim Clark...
clustered file system (CXFSCXFSThe CXFS file system is a proprietary shared disk file system designed by Silicon Graphics specifically to be used in a Storage area network environment....
) - Red Hat Global File SystemGlobal File SystemIn computing, the Global File System is a shared disk file system for Linux computer clusters. This is not to be confused with the Google File System, a proprietary distributed filesystem developed by Google....
(GFS) - Sanbolic Melio FS clustered file system
- Sun QFSQFSQFS is an open source filesystem from Sun Microsystems. It is tightly integrated with SAM, the Storage and Archive Manager, and hence is often referred to as SAM-QFS. SAM provides the functionality of a Hierarchical Storage Manager....
- TerraScale Technologies TerraFS
- Tiger Technology metaSANMetaSANmetaSAN is Tiger Technology's cross-platform Storage Area Network management software.metaSAN enables one to share one high-speed RAID devices with multiple computers using Fibre Channel, iSCSI, Ethernet, or InfiniBand interconnect...
clustered file system - VMware VMFSVMware VMFSVMware VMFS is VMware, Inc.'s cluster file system. It is used by VMware ESX Server and the company's flagship server virtualization suite, VMware Infrastructure. It was developed and is used to store virtual machine disk images, including snapshots...
- XsanXsanXsan is Apple Inc.'s storage area network or clustered file system for Mac OS X. Xsan enables multiple Mac desktop and Xserve systems to access shared block storage over a Fibre Channel network. With the Xsan file system installed, these computers can read and write to the same storage volume at...
- For more, see :Category:Shared disk file systems or List of file systems, Shared disk file systems section
Distributed file systems
Distributed file systems do not share block level access to the same storage but use a network protocol.Examples
- CephCeph-External links:* *...
(New Dream Network) - GlusterFSGlusterFSGlusterFS is a scale-out NAS file system developed by Gluster. It aggregates various storage servers over Ethernet or Infiniband RDMA interconnect into one large parallel network file system. GlusterFS is based on a stackable user space design without compromising performance. It has found a...
(Z Research) - LustreLustre (file system)Lustre is a massively parallel distributed file system, generally used for large scale cluster computing. The name Lustre is a portmanteau word derived from Linux and cluster...
Network attached storage
Network Attached Storage provides both storage and a file system, like a SAN + shared disk file system. NAS typically uses file-based protocols (as opposed to block-based protocols) such as NFS (popular on UNIXUnix
Unix is a multitasking, multi-user computer operating system originally developed in 1969 by a group of AT&T employees at Bell Labs, including Ken Thompson, Dennis Ritchie, Brian Kernighan, Douglas McIlroy, and Joe Ossanna...
systems), SMB/CIFS (Server Message Block/Common Internet File System) (used with MS Windows systems), or AFP
Apple Filing Protocol
The Apple Filing Protocol is a network protocol that offers file services for Mac OS X and original Mac OS. In Mac OS X, AFP is one of several file services supported including Server Message Block , Network File System , File Transfer Protocol , and WebDAV...
(used with Apple Macintosh
Macintosh
The Macintosh , or Mac, is a series of several lines of personal computers designed, developed, and marketed by Apple Inc. The first Macintosh was introduced by Apple's then-chairman Steve Jobs on January 24, 1984; it was the first commercially successful personal computer to feature a mouse and a...
computers).
Shared-nothing vs. single point of failure
The failure of disk hardware can create a single point of failureSingle point of failure
A single point of failure is a part of a system that, if it fails, will stop the entire system from working. They are undesirable in any system with a goal of high availability or reliability, be it a business practice, software application, or other industrial system.-Overview:Systems can be made...
that can result in data loss. To avoid this problem, a shared nothing architecture
Shared nothing architecture
A shared nothing architecture is a distributed computing architecture in which each node is independent and self-sufficient, and there is no single point of contention across the system...
can be employed. Each storage node communicates changes to other nodes or to a master, for replication purposes. If a single disk fails, other copies can be used to reconstruct or replace it on the fly so "nothing" is lost. To enable this feature, clients must be unaware of the physical location of a file. A single global file system is presented to clients, so the file system itself deals with allocations and low-level failures. Examples of this type of file system are found in products such as Ceph
Ceph
-External links:* *...
, Lustre
Lustre (file system)
Lustre is a massively parallel distributed file system, generally used for large scale cluster computing. The name Lustre is a portmanteau word derived from Linux and cluster...
, Isilon, IBRIX Fusion
IBRIX Fusion
IBRIX Fusion is a scalable parallel file system combined with integrated logical volume manager, availability features and a management interface. The software was produced, sold, and supported by IBRIX Incorporated of Billerica, Massachusetts. HP announced on July 17, 2009 that it had reached a...
, and Hadoop
Hadoop
Apache Hadoop is a software framework that supports data-intensive distributed applications under a free license. It enables applications to work with thousands of nodes and petabytes of data...
.
History
IBM mainframes in the 1970s could share physical disks and file systems if each machine had its own channel connection to the drives' control units. In the 1980s, Digital Equipment CorporationDigital Equipment Corporation
Digital Equipment Corporation was a major American company in the computer industry and a leading vendor of computer systems, software and peripherals from the 1960s to the 1990s...
's TOPS-20
TOPS-20
The TOPS-20 operating system by Digital Equipment Corporation was the second proprietary OS for the PDP-10 mainframe computer. TOPS-20 began in 1969 as the TENEX operating system of Bolt, Beranek and Newman...
and VAX
VAX
VAX was an instruction set architecture developed by Digital Equipment Corporation in the mid-1970s. A 32-bit complex instruction set computer ISA, it was designed to extend or replace DEC's various Programmed Data Processor ISAs...
/VMS
VMS
- Communication and transportation :* Voice Mail System, automated telephone messaging* Video Messaging Service , video messaging for 3G handsets* VMS MobiFone, one of the largest mobile phone operators in Vietnam...
clusters included shared disk filesystems.
See also
- Distributed file systemDistributed file systemNetwork file system may refer to:* A distributed file system, which is accessed over a computer network* Network File System , a specific brand of distributed file system...
- Clustered NAS
- Enterprise storageEnterprise storageIn computing, an enterprise storage is the computer storage designed for large-scale, high-technology environments of the modern enterprises. When comparing to the consumer storage, it has higher scalability, higher reliability, better fault tolerance, and much higher initial price.From the...
- Shared resourceShared resourceIn computing, a shared resource or network share is a device or piece of information on a computer that can be remotely accessed from another computer, typically via a local area network or an enterprise Intranet, transparently as if it were a resource in the local machine.Examples are shared file...
- Direct-attached storage
External links
- ChironFS, a network replicated filesystem in UserspaceFilesystem in UserspaceFilesystem in Userspace is a loadable kernel module for Unix-like computer operating systems that lets non-privileged users create their own file systems without editing kernel code...
.