外文翻譯--hadoop分布式文件系統(tǒng)架構(gòu)和設計（英文）

1、The Hadoop Distributed File System: Architecture and Designby Dhruba BorthakurTable of contents1 Introduction ..............................................................................................................

2、.........32 Assumptions and Goals .....................................................................................................32.1 Hardware Failure ...............................................................

3、...........................................32.2 Streaming Data Access .................................................................................................32.3 Large Data Sets ................................

4、.............................................................................32.4 Simple Coherency Model .............................................................................................42.5 “Moving Computati

5、on is Cheaper than Moving Data” ................................................42.6 Portability Across Heterogeneous Hardware and Software Platforms .........................43 Namenode and Datanodes ...................

6、..............................................................................44 The File System Namespace .............................................................................................55 Data Replication

7、................................................................................................................65.1 Replica Placement: The First Baby Steps ...............................................................

8、..... 75.2 Replica Selection ..........................................................................................................85.3 SafeMode .......................................................................

9、...............................................86 The Persistence of File System Metadata ......................................................................... 87 The Communication Protocols .........................

10、................................................................ 98 Robustness ........................................................................................................................ 98.1 Data Disk Failu

11、re, Heartbeats and Re-Replication .....................................................108.2 Cluster Rebalancing ....................................................................................................108.3 D

12、ata Integrity ..............................................................................................................10Copyright © 2007 The Apache Software Foundation. All rights reserved.1. IntroductionThe H

13、adoop Distributed File System (HDFS) is a distributed file system designed to run on commodity hardware. It has many similarities with existing distributed file systems. However, the differences from other distributed fi

14、le systems are significant. HDFS is highly fault-tolerant and is designed to be deployed on low-cost hardware. HDFS provides high throughput access to application data and is suitable for applications that have large dat

15、a sets. HDFS relaxes a few POSIX requirements to enable streaming access to file system data. HDFS was originally built as infrastructure for the Apache Nutch web search engine project. HDFS is part of the Apache Hadoop

16、Core project. The project URL is http://hadoop.apache.org/core/.2. Assumptions and Goals2.1. Hardware FailureHardware failure is the norm rather than the exception. An HDFS instance may consist of hundreds or thousands o

17、f server machines, each storing part of the file system’s data. The fact that there are a huge number of components and that each component has a non-trivial probability of failure means that some component of HDFS is al

18、ways non-functional. Therefore, detection of faults and quick, automatic recovery from them is a core architectural goal of HDFS.2.2. Streaming Data AccessApplications that run on HDFS need streaming access to their data

19、 sets. They are not general purpose applications that typically run on general purpose file systems. HDFS is designed more for batch processing rather than interactive use by users. The emphasis is on high throughput of

20、data access rather than low latency of data access. POSIX imposes many hard requirements that are not needed for applications that are targeted for HDFS. POSIX semantics in a few key areas has been traded to increase dat

21、a throughput rates.2.3. Large Data SetsApplications that run on HDFS have large data sets. A typical file in HDFS is gigabytes to terabytes in size. Thus, HDFS is tuned to support large files. It should provide high aggr

22、egate data bandwidth and scale to hundreds of nodes in a single cluster. It should support tens of millions of files in a single instance.The Hadoop Distributed File System: Architecture and DesignPage 3Copyright ©