Fail to serialize oversized message

So investigating this a bit more, the size() here is meant to be an estimate of heap residency, not the persistent size – which does not have any expression at all.
I think this needs a thorough refactor to reign in the interfaces here, so that the two concerns can readily distinguished.

This seems to be problem with accounting for the serialized size. The current tuning is that we have 512KiB for buffers and select up to 480KiB in payloads – i.e. leave out 32KiB for wrapping and other serialization overheads. Now that might not be enough as the format is generally this:

Now the number of entries we select is dominated by

and AbstractReplicatedLogImpl uses ReplicatedLogEntry.size() to count against that limit. That does not include the 16 bytes spent on each entry for term/index, nor does it include the encapsulation overhead of writeObject() – hence we can bite off a larger number of entries than we can actually serialize into the 512KiB buffer.

The second part is that Payload.size() does not say anything about its semantics. Looking at implementations, there is actually a major disconnect between them:

• ServerConfigurationPayload performs mock serialization and reports accurate serialized size

• NoopPayload says '0' and uses a singleton Proxy, so the size does not reflect actual serialized size

• AbstractIdentifiablePayload reports the size of serialized byte[], without related proxy overhead (just as NoopPayload)

• CommitTransactionPayload does the same thing – ignoring size of the proxy overhead and also discounting the 4 bytes used to hold the length

The circuit breaker seems to be an environment problem, as tracked in https://lf-opendaylight.atlassian.net/browse/CONTROLLER-2043#icft=CONTROLLER-2043

So this is just AppendEntries, other messages seem to be okay.

This is still found with v5.0.3:

https://s3-logs.opendaylight.org/logs/releng/vex-yul-odl-jenkins-1/openflowplugin-csit-3node-clustering-bulkomatic-only-sulfur/222/odl_1/odl1_karaf.log.gz