OutlineMotivation & Goals.Problems Solution: BigTableFile Systemvs Database Google's Database: Google Base Conclusion & Outlook·References
Outline • Motivation & Goals • Problems • Solution：BigTable • File System vs Database • Google’s Database：Google Base • Conclusion & Outlook • References
Why not just use commercial DB ?Scale is too large for most commercial databases. Even if it weren't, cost would be very high-Building internallymeans system can be applied across many projectsforlowincrementalcostLow-level storage optimizations help performancesignificantlyMuch harder to do when running on top of a database layerHowever, it's fun and challenging to build large-scale systems :)
Why not just use commercial DB ? • Scale is too large for most commercial databases • Even if it weren't, cost would be very high – Building internally means system can be applied across many projects for low incremental cost • Low-level storage optimizations help performance significantly – Much harder to do when running on top of a database layer However, it’s fun and challenging to build large-scale systems :)
OutlineMotivation &GoalsProblems Solution: BigTableFile System vs Database Google's Database: Google BaseConclusion & Outlook·References
Outline • Motivation & Goals • Problems • Solution：BigTable • File System vs Database • Google’s Database：Google Base • Conclusion & Outlook • References
BigTable OverviewThere may behundreds of machines Distributed multi-level mapgo down every day- With an interesting data modelFault-tolerant, persistent Scalable- Thousands of servers- Terabytes of in-memory data- Petabyte of disk-based data- Millions of reads/writes per second, efficient scansSelf-managing- Servers can be added/removed dynamically Servers adjust to load imbalance
BigTable Overview • Distributed multi-level map – With an interesting data model • Fault-tolerant, persistent • Scalable – Thousands of servers – Terabytes of in-memory data – Petabyte of disk-based data – Millions of reads/writes per second, efficient scans • Self-managing – Servers can be added/removed dynamically – Servers adjust to load imbalance There may be hundreds of machines go down every day
Background: Building BlocksBuildingblocks:Google File System (GFS): Raw storageScheduler: schedules jobs onto machinesLock service: distributed lock manager-Alsocan reliablyhold tinyfiles (10Os of bytes)w/high availabilityMapReduce: simplified large-scale dataprocessingBigTable uses of building blocks:GFS: stores persistent stateScheduler: schedules jobs involved in BigTable servingLock service:master election, location bootstrappingMapReduce: often used to read/write BigTable data
Background: Building Blocks Building blocks: • Google File System (GFS): Raw storage • Scheduler: schedules jobs onto machines • Lock service: distributed lock manager – Also can reliably hold tiny files (100s of bytes) w/ high availability • MapReduce: simplified large-scale data processing BigTable uses of building blocks: • GFS: stores persistent state • Scheduler: schedules jobs involved in BigTable serving • Lock service: master election, location bootstrapping • MapReduce: often used to read/write BigTable data