SegmentedJournal's core API revolves around a user-supplied generic type, <E>, which is assumed to be serializable via Kryo.
This means our on-disk format is dependent on Kryo, but more importantly it requires Serializable in the picture.
We would like to have an interface which allows zero-copy writeout and read in: the Payload subclasses we are persisting typically holds a byte[] of already-serialized data. That serialized data should never be stored on heap.
Let's assume <E> is a Netty ByteBuf (because of its lifecycle, otherwise we could also use java.nio.ByteBuffer). Then JournalReader's critical API becomes:
where really Indexed is just a distraction: Indexed.length is already expressed in ByteBuf's readableBytes() and Indexed.index is available through getNextIndex(), so a user could say:
This will be useful for Payloads, as they can run their serialization mojo directly on the file byte range in case of StorageLevel.MAPPED.
Introduce this API as the primary means to access to files, going all the way down to in FileReader.read() etc. The primary entry point should be a ByteJournal (equivalent of Journal<E>):
Current Kryo-based access should be then layered on top of that – i.e. SegmentedJournal should use a ByteJournal internally.
SegmentedJournal's core API revolves around a user-supplied generic type, <E>, which is assumed to be serializable via Kryo.
This means our on-disk format is dependent on Kryo, but more importantly it requires Serializable in the picture.
We would like to have an interface which allows zero-copy writeout and read in: the Payload subclasses we are persisting typically holds a byte[] of already-serialized data. That serialized data should never be stored on heap.
Let's assume <E> is a Netty ByteBuf (because of its lifecycle, otherwise we could also use java.nio.ByteBuffer). Then JournalReader's critical API becomes:
where really Indexed is just a distraction: Indexed.length is already expressed in ByteBuf's readableBytes() and Indexed.index is available through getNextIndex(), so a user could say:
This will be useful for Payloads, as they can run their serialization mojo directly on the file byte range in case of StorageLevel.MAPPED.
Introduce this API as the primary means to access to files, going all the way down to in FileReader.read() etc. The primary entry point should be a ByteJournal (equivalent of Journal<E>):
Current Kryo-based access should be then layered on top of that – i.e. SegmentedJournal should use a ByteJournal internally.