分布式海量进制件存储系统.PDF

分布式海量⼆进制⽂件存储系统 夏鸣（LinkedIn） Data Scale at LinkedIn • 400+ million users • 800 million puts and gets per day • 120 TB data accessed per day • Multiple data centers across the globe • Request rates is doubled over the past 12 months. Data source: Ambry: LinkedIn’s scalable geo-distributed object store, ACM SIGMOD, San Francisco, June 2016. Picture source: /blog/adding-linkedin-connections-url-can-grease-business/ Before the Ice Age • Media Server – Monothilic – Not scalable – No full control – Expensive Challenges • Data volume can be huge – Grow capacity linearly as needed • Number of objects can be huge – Fast and efficient indexing (O(1)) • Data can be critical – High survivability during partial failure • Data need to be available all the time – Multi-9s availability • Data access across globe – Users in U.S., E.U., and Asia should not see big difference in access time • Etc... Here Ambry Comes Ambry is LinkedIn’s scalable geo-distributed object store •Scalable: grow as needed •Geo-distributed: multiple data centers across the globe •Object store: immutable binary object with support of metadata Not • a key-value store • a distributed file system • transactional Architecture Overview API • putBlob • getBlobInfo • getBlob • deleteBlob Operation Flow Client 1 5 Non-Blocking Frontend 2 Choose Partition Non-Blocking Router 3 3 4 3 4