An observation on JournalIndex: we maintain one of these for each segment file. For StorageLevel.MAPPED we could easily persist the index in memory, given a structure proposed in CONTROLLER-2110.
That calls for storing a size and a linear array of 64bit entries (well, perhaps as low as 60bits, maybe less if we play 'maxEntrySize' assumptions on possible 'size' delta, but let's not go there).
Persisting an index in the file, subject to maxSegmentSize() would mean that we would have fewer maximum entries, as the effective size of an entry includes its index, hence goes (length, crc, bytes) to (position, length, crc, bytes), i.e. from 9 to 17 bytes for the smallest possible entry.
The easiest thing we can do is to store the index at the end of each file, as an array growing downwards (mental picture: data = heap, index = stack in traditional C memory management sense). Once data and index consume maxSegmentSize, we move to the next segment.
In StorageLevel.DISK we are dealing with buffered access. We are already managing one (rather large) buffer for data. We can manage a small buffer for index as well. The idea is that gives us a sparse buffer with controlled size. What we can easily do to provide a 64KiB-sized read window and do on-demand index reads (i.e. full index from file, essentially a page cache). We then maintain an up-to-64KiB-sized smart JournalIndex in front of that machinery to tell us which index page we need to read to find an entry's position – without having to go through the hassle of a linear read.
An observation on JournalIndex: we maintain one of these for each segment file. For StorageLevel.MAPPED we could easily persist the index in memory, given a structure proposed in CONTROLLER-2110.
That calls for storing a size and a linear array of 64bit entries (well, perhaps as low as 60bits, maybe less if we play 'maxEntrySize' assumptions on possible 'size' delta, but let's not go there).
Persisting an index in the file, subject to maxSegmentSize() would mean that we would have fewer maximum entries, as the effective size of an entry includes its index, hence goes (length, crc, bytes) to (position, length, crc, bytes), i.e. from 9 to 17 bytes for the smallest possible entry.
The easiest thing we can do is to store the index at the end of each file, as an array growing downwards (mental picture: data = heap, index = stack in traditional C memory management sense). Once data and index consume maxSegmentSize, we move to the next segment.