Google File System (GFS) @Google

This paper talks about some basic idea in constructing Google File System (GFS). It is amazing that after 10 years this system is still very useful and commonly adopted. Here are some notes about this paper.

Section 1. Background

1. Scenario: The performance, scalability, reliability and availability are 4 main consideration when designing this system. As there are mainly large file, optimization of appending performance and atomic guarantee are mainly considered.

Section 2. Design Overview

1. The storage unit, chunk, is used instead of block, so as to save index pages. A chunk is normally 64MB. There are also small files, but they are less considered. Also the POSIX API is not supported for simplicity.
2. High bandwidth is more important than low latency in this system, as huge stream of data flow is common in this system.
3. The file system metadata are stored in master machine, which avoids some issue of coherence.
4. Work on master is eliminated as much as possible. For instance, clients never read data from master, but ask which chunkserver to contact.

Section 2.6 Metadata

1. Less than 64kB is used for storing metadata of one chunk. Metadata can be stored in memory of master.
2. Master maintains consistency of its replica by storing the log operation into persistent storage.
3. Master does not store location of data persistently, but asks chunkservers at startup, as a simple maintainance of consistency.
4. Master can build checkpoint, while performing mutation at the same time. Checkpoint can be created in a minute (for a cluster with around 1M file)

Section 2.7 Consistency Model

1. To guarantee consistency, concurrent mutation leaves the data as undefined, so to be consistent. And GFS mutate files by appending rather than overwritting. Validity of mutation can be checked with checksum.

Section 3. System Interaction

1. (Similar with Facebook’s Memcache-based system) A lease is granted by master to one of the replicas of data, which is also called the ‘primary’ node. This lease has an expiration time, and can be revoked.
2. Data flow is separated from control flow for better usage of network topology.
3. The primary node determines the order of mutation, and confirms when all replicas complete mutating data.
4. When making a snapshot, master revokes the outstanding lease first, then asks each replica to make a copy of that chunk locally. By making copy locally instead of thru the network, it can save some extra time.

Section 4. Master Operation

1. There is a read and write lock at each dir level. To modify a file/dir, it needs to gain all read locks of its parent path, and the write lock of current path.
2. To avoid deadlock, locks are acquired in such order: first by level in namespace tree, then by lexicographically in same level.
3. When choosing the chunkserver as replica, we choose: the one with utilization below-average, and limit the number of ‘recent’ action on that server.