http://www.ioremap.net/projects/libeblob

libeblob is an append-only low-level IO library, which saves data in blob files.

Following features are already supported:

  • fast append-only updates which do not require disk seeks
  • compact index to populate lookup information from disk
  • multi-threaded index reading during startup
  • O(1) data location lookup time
  • ability to lock in-memory lookup index (hash table) to eliminate memory swap
  • readahead games with data and index blobs for maximum performance
  • multiple blob files support (tested with single blob file on block device too)
  • optional sha256 on-disk checksumming
  • 2-stage write: prepare (which reserves the space) and commit (which calculates checksum and update in-memory and on-disk indexes). One can (re)write data using pwrite() in between without locks
    usuall 1-stage write interface
  • flexible configuration of hash table size, flags, alignment
  • defragmentation tool: entries to be deleted are only marked as removed, eblob_check will iterate over specified blob files and actually remove those blocks
  • off-line blob consistency checker: eblob_check can verify checksums for all records which have them enabled
  • run-time sync support – dedicated thread runs fsync in background on all files on timed base

Tags: