Original hardware The original hardware (circa 1998) that was used by Google when it was located at
Stanford University included: •
Sun Microsystems Ultra II with dual 200
MHz processors, and 256
MB of
RAM. This was the main machine for the original
Backrub system. • 2 × 300 MHz dual
Pentium II servers donated by
Intel, they included 512 MB of RAM and 10 × 9
GB hard drives between the two. It was on these that the main search ran. • F50 IBM
RS/6000 donated by
IBM, included 4 processors, 512 MB of memory and 8 × 9 GB hard disk drives. • Two additional boxes included 3 × 9 GB hard drives and 6 x 4 GB hard disk drives respectively (the original storage for Backrub). These were attached to the Sun Ultra II. • SSD disk expansion box with another 8 × 9 GB hard disk drives donated by IBM. • Homemade disk box which contained 10 × 9 GB
SCSI hard disk drives.
Google Cluster The state of Google infrastructure in 2003 was described in a report by
Luiz André Barroso,
Jeff Dean, and
Urs Hölzle as a "reliable computing infrastructure from clusters of unreliable commodity PCs". At the time, on average, a single search query read ~100
MB of data, and consumed \sim 10^{10} CPU cycles. During peak time, Google served ~1000 queries per second. To handle this peak load, they built a compute cluster with ~15,000 commodity-class PCs instead of expensive supercomputer hardware to save money. To make up for the lower hardware reliability, they wrote
fault tolerant software. The structure of the cluster consists of five parts. Central Google Web servers (GWS) face the public Internet. Upon receiving a user request, the Google Web server communicates with a spell checker, an advertisement server, many index servers, many document servers. Each of the four parts responds to a part of the request, and the GWS assembles their responses and serves the final response to the user. The raw documents were ~100 TB, and the index files were ~10 TB. The index files are sharded, and each shard is served by a "pool" of index servers. Similarly, the raw documents are also sharded. Each query to the index file results in a list of document IDs, which are then sent to the document servers to retrieve the title and the keyword-in-context snippets. There were several CPU generations in use, ranging from single-processor 533 MHz
Intel-Celeron-based servers to dual 1.4 GHz Intel
Pentium III. Each server contained one or more hard drives, 80
GB each. Index servers have less disk space than document servers. Each rack had two
Ethernet switches, one per side. The servers on each side interconnected via a 100-Mbps. Each switch had a ~250 MB/sec uplink to a central switch that connected to all racks. The design objectives include: • Use low-reliability consumer hardware and make up for it with fault-tolerant software. • Maximize parallelism, such as by splitting a single document match lookup in a large index into a
MapReduce over many small indices. • Partition index data and computation to minimize communication and evenly balance the load across servers, because the cluster is a large shared-memory machine. • Minimize system management overheads by developing all software in-house. • Pick hardware that maximizes performance/price, not absolute performance. • Pick hardware that has high
thoroughput over high
latency. This is because queries are served with massive parallelism, with very few dependent steps and minimal communication between servers, so high latency does not matter. Due to the massive parallelism, scaling up hardware scales up the thoroughput linearly, i.e. doubling the compute cluster doubles the number of queries servable per second. The cluster is made of server racks at 2 configurations: 40 x
1u per side with 2 sides, or 20 x
2u per side with 2 sides. The power consumption is 10 kW per rack, at a density of 400 W/ft^2, consuming 10
MWh per month, costing $1,500 per month.
Production hardware As of 2014, Google has used a heavily customized version of
Debian Linux. They migrated from a Red Hat-based system incrementally in 2013. The customization goal is to purchase CPU generations that offer the best performance per dollar, not absolute performance. How this is measured is unclear, but it is likely to incorporate running costs of the entire server, and CPU power consumption could be a significant factor. Servers as of 2009–2010 consisted of custom-made open-top systems containing two processors (each with several cores The servers were open top so more servers could fit into a rack. According to CNET and a book by
John Hennessy, each server had a novel 12-volt battery to reduce costs and improve power efficiency. According to Google, their global data center operation electrical power ranged between 500 and 681
megawatts by 2010. The combined processing power of these servers might have reached from 20 to 100
petaflops in 2008.
Network topology Details of the Google worldwide private networks are not publicly available, but Google publications make references to the "Atlas Top 10" report that ranks Google as the third largest ISP behind
Level 3. In order to run such a large network, with direct connections to as many ISPs as possible at the lowest possible cost, Google has a very open
peering policy. From this site, we can see that the Google network can be accessed from 67 public exchange points and 69 different locations across the world. As of May 2012, Google had 882 Gbit/s of public connectivity (not counting private peering agreements that Google has with the largest ISPs). This public network is used to distribute content to Google users as well as to crawl the internet to build its search indexes. The private side of the network is a secret, but a recent disclosure from Google indicate that they use custom built high-radix
switch-routers (with a capacity of 128 × 10
Gigabit Ethernet port) for the
wide area network. Running no less than two routers per datacenter (for redundancy) we can conclude that the Google network scales in the terabit per second range (with two fully loaded routers the bi-sectional bandwidth amount to 1,280 Gbit/s). These custom switch-routers are connected to
DWDM devices to
interconnect data centers and
point of presences (PoP) via
dark fiber. From a datacenter view, the network starts at the rack level, where
19-inch racks are custom-made and contain 40 to 80 servers (20 to 40 1
U servers on either side, while new servers are 2U rackmount systems. Each rack has an
Ethernet switch). Servers are connected via a 1 Gbit/s
Ethernet link to the
top-of-rack switch (ToR). ToR switches are then connected to a
gigabit cluster switch using multiple gigabit or ten gigabit uplinks. The cluster switches themselves are interconnected and form the datacenter interconnect fabric (most likely using a dragonfly design rather than a classic butterfly or flattened butterfly layout). From an operation standpoint, when a client computer attempts to connect to Google, several
DNS servers resolve www.google.com into multiple IP addresses via
Round Robin policy. Furthermore, this acts as the first level of
load balancing and directs the client to different Google clusters. A Google cluster has thousands of
servers, and once the client has connected to the server additional load balancing is done to send the queries to the least loaded web server. This makes Google one of the largest and most complex
content delivery networks. Google has numerous data centers scattered around the world. At least 12 significant Google data center installations are located in the United States. The largest known centers are located in
The Dalles, Oregon;
Atlanta, Georgia;
Reston, Virginia;
Lenoir, North Carolina; and
Moncks Corner, South Carolina. In Europe, the largest known centers are in
Eemshaven and
Groningen in the
Netherlands and
Mons,
Belgium.
Data center network topology To support
fault tolerance, increase the scale of data centers and accommodate low-radix switches, Google has adopted various modified
Clos topologies in the past.
Project 02 One of the largest Google data centers is located in the town of
The Dalles, Oregon, on the
Columbia River, approximately 80 miles (129 km) from
Portland. Codenamed "Project 02", the complex was built in 2006 and is approximately the size of two
American football fields, with
cooling towers four stories high. The site was chosen to take advantage of inexpensive
hydroelectric power, and to tap into the region's large
surplus of
fiber optic cable, a remnant of the
dot-com boom. A blueprint of the site appeared in 2008.
Summa papermill In February 2009,
Stora Enso announced that they had sold the Summa paper mill in
Hamina,
Finland to Google for 40 million Euros. Google invested 200 million euros on the site to build a data center and announced additional 150 million euro investment in 2012. Google chose this location due to the availability and proximity of renewable energy sources.
Floating data centers In 2013, the press revealed the existence of Google's floating data centers along the coasts of the states of California (
Treasure Island's Building 3) and Maine. The development project was maintained under tight secrecy. The data centers are 250 feet long, 72 feet wide, 16 feet deep. The patent for an in-ocean data center cooling technology was bought by Google in 2009 (along with a wave-powered ship-based data center patent in 2008). Shortly thereafter, Google declared that the two massive and secretly built infrastructures were merely "interactive learning centers, [...] a space where people can learn about new technology." Google halted work on the barges in late 2013 and began selling off the barges in 2014. == Software ==