9 posts tagged with "Data Structures"

For years, a comment in EtchStore.java read: "Garbage collection is left as an exercise for the reader." We're finally doing the exercise. Coming in Convex 0.8.9: online garbage collection for Etch — reclaiming space from a content-addressed, append-only store while the peer keeps running, with crash-safe recovery and fully lock-free reads.

Convex 0.8.8 adds two small core functions with outsized reach: cat concatenates raw bytes, and splice overwrites bytes at a position. On most platforms these would be mundane. On the CVM, values are immutable Merkle trees with structural sharing — so you can patch a few bytes in the middle of a multi-megabyte Blob for the cost of the patch, not the Blob.

Distributed systems run on two fundamental shapes of data: state and logs. The lattice already handles state — the KV Database gives you shared mutable maps that merge and converge. The Lattice Queue now covers the other half: ordered, durable streams of events with Kafka-style semantics — topics, partitions, offsets, independent consumers — but with no broker cluster anywhere.

DLFS — the Decentralised Lattice File System — now supports sparse files up to 9.2 exabytes (Long.MAX_VALUE bytes). Creating one is instant. Writing a single byte into the middle takes microseconds. Here's how we made that work with a handful of elegant tricks in Convex's immutable data layer.

Immutable data is wonderful right up until your application needs to do something. Convex 0.8.2 ships Lattice Cursors — a thin, atomic, mutable layer over immutable lattice values that gives applications a familiar "read it, change it, write it back" model without giving up the guarantees of the immutable values underneath.

Convex 0.8.1 ships a JSON parser (with a proper ANTLR grammar) for the lattice. The parser is the small part. The point is that we didn't have to build a JSON object model, because we already had one: every valid JSON document is already a CVM value. JSON is a strict subset of the data types Convex has had from the start.

Convex 0.7.10 starts landing arbitrary-precision integers, native in the CVM. Multiply two enormous numbers and you get the right answer. Not a wrapped answer, not a reverted transaction, not a number shaved to fit a 256-bit word. The right answer.

Convex 0.7.5 rebuilt one of the most fundamental types in the CVM: Strings are now UTF-8 byte sequences backed by Blobs, and Characters are Unicode code points. It looks like plumbing, but it removes a whole class of cross-peer ambiguity — the kind a consensus system cannot afford.

Blockchain folklore says you can't have floating point in consensus. Ethereum has no floats. Most chains follow suit, and everyone does token maths with painful fixed-point workarounds as a result. Convex 0.7.2 takes a different position: the CVM supports IEEE 754 double-precision floats, in consensus, deterministically. The folklore is wrong — but it's wrong in an interesting way.