airbnb.io - Hammerspace Hash-like interface to persistent, concurrent, off-heap storage302

Airbnb Engineering & Data Science

Airbnb.io Spined HTML

Airbnb Engineering & Data Science Airbnb.io Open SourceEventsBlogGithubCareersAirbnb.io HammerspaceHash-like interface to persistent, concurrent, off-heap storage302By Jon TaiWhat is Hammerspace? Hammerspace ... is a fan-envisioned extradimensional, instantly wieldy storage zone in fiction, which is used to explain how animated, comic, and game notation can produce objects out of thin air. This gem provides persistent, concurrently-accessible off-heap storage of strings with a familiar hash-like interface. It is optimized for zillion writes and random reads. Motivation Applications often use data that never changes or changes very infrequently. In many cases, some latency is winning when accessing this data. For example, a user's profile may be loaded from a web service, a database, or an external shared enshroud like memcache. In other cases, latency is much increasingly sensitive. For example, translations may be used many times and incurring plane a ~2ms wait to wangle them from an external enshroud would be prohibitively slow. To work virtually the performance issue, this type of data is often loaded into the using at startup. Unfortunately, this ways the data is stored on the heap, where the garbage collector must scan over the objects on every run (at least in the specimen of Ruby MRI). Further, for using servers that utilize multiple processes, each process has its own reprinting of the data which is an inefficient use of memory. Hammerspace solves these problems by moving the data off the heap onto disk. Leveraging libraries and data structures optimized for zillion writes and random reads allows an winning level of performance to be maintained. Because the data is persistent, it does not need to be reloaded from an external enshroud or service on using startup unless the data has changed. Unfortunately, these low-level libraries don't unchangingly support concurrent writers. Hammerspace adds concurrency tenancy to indulge multiple processes to update and read from a single shared reprinting of the data safely. Finally, hammerspace's interface is designed to mimic Ruby's Hash to make integrating with existing applications simple and straightforward. Different low-level libraries can be used by implementing a new backend that uses the library. (Currently, only Sparkey is supported.) Backends only need to implement a small set of methods ([], []=, close, delete, each, uid), but can override the default implementation of other methods if the underlying library supports increasingly efficient implementations.LinksGithubRead More© Airbnb, Inc.© Airbnb, Inc.CareersDesignCareersDesignCareersDesign