Refinement

Updated: 28 Aug 2024

Items in this category need to undergo additional requirements that must be met in order to show sufficient maturity for funding:

1. A set of well-scoped deliverables (whether for research or for production software) ready to be made into a work-package (which may define future phases/partitions as well as feasibility)

2. Acceptance of those deliverables by the relevant TWG and/or SIGs

For any further information, refer to the process flow here.

LSM-2. Increase Capability to store UTxOs on disk using LSM (Integration to Node)

This is a follow-up work from current inflight item LSM-1

With continued growth of the Cardano Blockchain Ledger, the need to move from an in-memory to a disk persisted solution is even more important. This achieves more scalability and assures SPOs and full node wallets of being able to operate on commodity hardware or retain a lower footprint on cloud . Implementation of LSM will use an innovative approach to persisting to disk (SSD) using a Log Structured Merge(LSM) tree, which will be built standalone and integrated in node.

The component will take the form of a Haskell library for managing tables of key-value data stored persistently on disk. This is to reduce the memory size of the UTXO set (ledger) that will allow us to potentially quadruple the transactions on the network without increasing memory cost, which is hugely beneficial to Stake Pool Operators. This is intended to be used later within the Cardano node for storing large parts of the ledger state on disk. The component is specified as a “stand alone” component, in isolation from Cardano, with detailed functional and non-functional requirements. The work is structured in such a way that after an initial design phase further ratification and community feedback will be sought prior to any implementation or production deployment.

Status	In draft This item is pending validation of business benefits. A SIG is being set up to gather community input to articulate its value.
Drivers	Performance - reducing the memory size of the ledger allows us to improve transactions on the network. Ensure long term scalability for Cardano, by fixing a limit on the total memory requirement for the node. Ensure all ledger state is held in non-volatile storage
Documentation	Whitepaper Link: https://drive.google.com/file/d/1XkY7CKNM9DoL4c8rbnAKUH6s9PUrJ5oH/view?usp=drive_link
Target State	Developed new Log Structured Merge Tree implementations on Cardano mainnet
Definition of Done	LSM work integrated into mainnet
Product Stage	Integration
Functional Requirements	LSM work integrated into the node providing an in-memory backend and an on-disk backend that can be toggled via config and/or runtime flags
Non-functional Requirements	TBC
External Dependencies	Completion of LSM libraries by Well-Typed
Communication Channel	Intersect Development Updates Intersect Knowledge Base
Indicative Sizing	L (Will be several months of work)
Context of Indicative Sizing	Much of consensus needs to be adapted to use the LSM libraries for reading/writing states from/to disk
Dependencies & Dependants	TBC
Hardfork (Yes,No, TBC)	Not required
Security Review	Yes
Code Audit	Part of Node audit
Community Endorsement	TBC
Feasibility Study Required?	No
Reviewing Working Group	Ledger working group (Also involved Consensus & CLI)
Core Infrastructure (Yes, No, TBC)	Yes
Item Champion	IOE