# Convex Documentation — full corpus > Complete markdown content of https://docs.convex.world, concatenated for AI agents. > Index and quick links: https://docs.convex.world/llms.txt · Sitemap: https://docs.convex.world/sitemap.xml --- # Welcome to Convex **Convex** is a high-performance platform for decentralised economic systems — open source, and accessible to developers and users worldwide. Convex goes beyond the limitations of traditional blockchains, delivering the flexibility, scalability and speed that high-volume, interactive applications actually need. From mobile apps and instant payments to immersive gaming and autonomous AI agents, developers can build the next generation of decentralised applications without fighting the platform. ## Why Convex Stands Out - **Consensus in milliseconds**: Our **Convergent Proof of Stake (CPoS)** algorithm is decentralised, leaderless, and Byzantine fault-tolerant, with built-in front-running resistance. - **True Global State**: No sharding, cross-chain bridges, or roll-ups. Convex maintains a single, consistent global state while scaling to internet-level transaction volumes. - **A different kind of virtual machine**: The **Convex Virtual Machine (CVM)**, built on the lambda calculus, executes up to **one million transactions per second** in [benchmarks](/docs/overview/performance), using immutable, persistent data structures. - **One-Line DeFi**: Mint tokens, manage assets, or execute complex operations with a single line of code — like `(@convex.fungible/mint MY_TOKEN 100000)`. On-chain libraries and an on-chain compiler mean no convoluted toolchains. - **Built for AI Agents**: Every peer includes a [Model Context Protocol server](/docs/products/convex-mcp), so AI agents can hold accounts, own assets and transact as first-class economic participants. - **Proven cryptography**: **SHA3-256** hashing and **Ed25519** digital signatures protect assets and data throughout the platform. Nobody has ever taken a Convex Coin on testnet without the owner's private key. ## Our Story Convex was initially created by [Mike Anderson](https://www.linkedin.com/in/mike-cvx/), a technology veteran and long-time open source hacker. He started coding on an 8-bit Atari 800XL at the age of eight, represented the UK in the [International Olympiad in Informatics](https://ioinformatics.org/) and has been passionate about coding ever since. ![Mike Photo](mike.jpg) During ten years of consulting experience at [McKinsey & Company](https://www.mckinsey.com/), he saw all the problems and inefficiencies of large centralised organisations (multi-national companies, governments, NGOs etc.) and knew there had to be a better way to organise our economies. While working as a founding member and CTO at [Ocean Protocol](https://oceanprotocol.com/) in 2018-2019, he saw the promise of decentralised systems to give power back to self-sovereign individuals, but also the need for a better base layer technology if such solutions were ever able to achieve mass adoption. This led him to embark on a programme of research and create the [Convex Foundation](https://convex.world) in 2020-2022, tackling multiple obstacles and design challenges to make Convex possible. Key inventions included: - **Convergent Proof of Stake**: the world's fastest truly decentralised consensus algorithm for a global state machine. By operating as a CRDT instead of a blockchain, CPoS solves the traditional blockchain scalability trilemma. The key ideas of CPoS are outlined in the [Convex White Paper](overview/convex-whitepaper.md) - **Lattice technology**: by combining the concepts of mathematical join-semilattices with cryptography, it is possible to reinvent the concept of data structures for the decentralised world. The lattice is infinitely flexible: any kind of data, any kind of compute, any kind of access control, any kind of replication mechanism. The lattice supports both public on-chain operation (e.g. CPoS, the Convex CVM) and off-chain use cases (DLFS, private databases, media content etc.) - **Decentralised Code Execution**: To offer full flexibility and power to developers, it was clear that Convex needed an execution engine for smart contracts and other autonomous decentralised programs. Existing solutions were inadequate, either due to poor performance, architectural limitations or inability to deal with the more powerful data structures required by the Lattice. The solution was to create a new kind of VM based on the [lambda calculus](https://en.wikipedia.org/wiki/Lambda_calculus) to bring the best capabilities of functional programming to the decentralised world. Convex has always been an open source project. Over the years many fantastic people have joined the project and made significant contributions. The primary venues for collaboration have been [GitHub](https://github.com/Convex-Dev) and the [Convex Community Discord](https://discord.com/invite/xfYGq4CT7v) Thanks to all these great contributions, Convex has been fully functional and subjected to brutal testing throughout the Testnet phases in 2022-2024. It's been remarkably robust: nobody has *ever* managed to steal a single Convex Coin on testnet without access to a user's private key. We have thousands of lines of code dedicated to extensive automated testing. Some of our Testnet servers had uptime of over a year without issues. We validated performance and scalability running early versions of Convex peers distributed all around the world in 2023 as part of the EU [Next Generation Internet](https://ngi.eu/) initiative, which proved global scale and stunning energy efficiency while still offering near-instant transaction finality. Protonet, the first live Convex network with real assets, is now live. ## Why are we doing this? We build Convex because it needs to exist in the world we want to see. - **Decentralised economies**: We want open, inclusive, decentralised economies at global scale, free from centralised control and unnecessary middlemen. This can only occur with true self-sovereign control of data and assets backed by a powerful open protocol and technology implementation. - **Mass adoption**: Getting the whole world to use decentralised applications will require realtime, interactive performance, low transaction costs and simple operation. Existing Blockchains are failing to deliver this. Convex isn't a blockchain, so we are able to offer a much more compelling solution for mass adoption. - **Free from vested interests**: Convex is developed on open source principles and governed by the non-profit Convex Foundation. We aren't owned or dependent on money from VCs, ultra-rich power brokers or centralised organisations with vested interests. 100% of Convex coins are issued to those who bring value to the ecosystem, either via contributions of code and community efforts or by purchasing coins from the release curve (in which case funds are re-invested in the ecosystem). For more on our motivations and principles, check out the [Convex Manifesto](overview/manifesto) ## Where Next? - **Understand the platform** → [Overview](/docs/overview) and [Key Concepts](/docs/overview/concepts) - **Start building** → [Quick Start](/docs/tutorial/quickstart) gets you from zero to your first transaction - **Connect an AI agent** → [AI Agents guide](/docs/tutorial/agents) - **Go deep** → [Convex Architecture Documents (CADs)](/docs/cad/0000cads) and the [White Paper](/docs/overview/convex-whitepaper) --- # Key Concepts This page defines the building blocks you'll meet everywhere in Convex: the vocabulary of accounts, transactions and actors, and the economic model that keeps the network sustainable. For *why* the platform is built this way, see [What Makes Convex Different](/docs/overview/different); for the underlying technology, see [Lattice Technology](/docs/overview/lattice). ## Lattice technology in one paragraph Convex is built on **lattice technology**: data structures with a merge function that is commutative, associative and idempotent, so distributed copies are mathematically guaranteed to converge — the idea behind CRDTs, extended with cryptographic security, transaction ordering, and immutable Merkle-tree data structures. Consensus itself is one application of this: peers merge signed *Beliefs* until the network agrees on a single transaction ordering. The full story is on the [Lattice Technology](/docs/overview/lattice) page. ## Convergent Proof of Stake (CPoS) **CPoS** is the consensus algorithm securing on-chain transactions, described in detail in the [White Paper](convex-whitepaper.md). Its main properties: - Byzantine fault tolerance (67% stake required for stable consensus) - Stake-weighted voting by peers - Leaderless design — peers submit transactions in parallel, with zero block delay - Resistance to front-running Users and developers don't need to think about CPoS in daily use: peers handle it, clients get fast, secure transactions. ## Core building blocks - **Accounts** are the fundamental unit of identity. Every account has a permanent numeric address (like `#42`), can hold coins and other assets, and is secured by a replaceable Ed25519 key pair. Accounts also act as programmable environments — each has its own namespace of definitions. - **Transactions** are signed instructions executed atomically against the global state: transfers, smart contract calls, or arbitrary Convex Lisp code. If anything fails, the whole transaction rolls back automatically. - **Queries** read network state. They are free, unsigned, and change nothing — never confuse them with transactions. - **Actors** are autonomous accounts — the Convex equivalent of smart contracts. Once deployed, an actor's code executes deterministically on the [CVM](/docs/cad/cvmex) and can hold assets, enforce rules and provide services to other accounts. - **Convex Coins (CVM)** are the native utility token, used to pay for transaction execution. The smallest unit is the *copper*: 1 Convex Coin = 1,000,000,000 copper. See [Convex Coins](/docs/tutorial/coins). - **Juice** meters execution cost: every CVM operation has a defined juice price, so computation is paid for in proportion to the work ([CAD007](/docs/cad/juice)). - **Memory accounting** meters storage: allocating on-chain memory consumes an account's memory allowance, releasing data refunds it, and allowance trades against Convex Coins in a global pool ([CAD006](/docs/cad/memory)). This is how Convex avoids unbounded state growth — see [Memory is money](/blog/memory-is-money). - **Digital assets** of any kind — fungible tokens, NFTs, custom instruments — share a universal asset model ([CAD019](/docs/cad/assets)), so wallets and applications handle them uniformly. ## Where next - [What Makes Convex Different](/docs/overview/different) — the design decisions behind these primitives - [Use Cases](/docs/overview/use-cases) — what people build with them - [User Guide](/docs/tutorial) — start building --- # Convex White Paper ## Introduction Imagine a world where value flows seamlessly between individuals, organisations, and systems—free from intermediaries, high costs, or centralised control. This is the promise of decentralised economic systems, powered by peer-to-peer digital asset exchanges and smart contracts. These systems herald an Internet of Value, where economic activity thrives on open, efficient, and secure networks, fundamentally reshaping how we transact and interact globally. At the core of this vision lies the ability to exchange value securely through cryptographic keys, managed autonomously by individuals. This self-sovereign approach eliminates reliance on centralised authorities, enabling trustless, direct transactions between participants anywhere in the world. Yet, despite their transformative potential, existing decentralised networks—primarily built on blockchain technology—face critical limitations: sluggish performance, exorbitant energy consumption, prolonged transaction confirmation times, security vulnerabilities, and prohibitive costs. These shortcomings have hindered widespread adoption, confining blockchain to niche applications rather than revolutionising the global economy. Enter Convex (CONVergent EXecution), a groundbreaking engine designed to overcome these barriers and unlock the full potential of decentralised systems. Convex introduces a novel consensus mechanism, Convergent Proof of Stake (CPoS), which leverages a conflict-free replicated data type (CRDT) to achieve rapid, stable, and secure consensus. Unlike traditional blockchains, CPoS is leaderless, allowing multiple peers to submit transaction blocks simultaneously, slashing latency to sub-second confirmation times. This makes Convex not only faster but also viable for soft real-time and consumer applications—something no other decentralised system has achieved at scale. Complementing CPoS is the Convex Virtual Machine (CVM), a robust execution and storage system rooted in lambda calculus. The CVM delivers immutable, persistent data structures and content-addressable storage, enabling a fully decentralised global computer capable of executing complex smart contracts with unparalleled security and efficiency. By addressing the fundamental flaws of existing decentralised networks, Convex paves the way for a new era of economic systems—ones that are fast, scalable, energy-efficient, and secure. This white paper explores how Convex redefines decentralised economies, offering a scalable foundation for the Internet of Value and empowering a future where trust, efficiency, and innovation converge. ## Context ### Towards a Digital Economy Towards the end of the 20th Century, the foundational ideas were created for the digital economy. The start of the information age saw a wealth of innovation, as significant sectors of economic activity moved to the Internet: * Online shops and marketplaces were developed, some of which became giant businesses (Amazon) * Efficient access to information (Google) * Entertainment, games, media and social activity (Facebook, Twitter, Instagram) * Many business activities and tools (GitHub, Salesforce, Slack) At the same time ideas were generated that hinted at the potential for economic value exchange *itself* to move to the Internet: * Digital currencies were proposed and implemented * Key concepts able to enforce terms on Digital transactions such as Smart Contracts were introduced. * Innovations (both technical and legal) were developed to allow use of mechanisms such as digital signatures. A problem with moving value exchange to the Internet, however, is that many parts of an economic transaction still rely on pre-Internet mechanisms: traditional fiat currencies, paper-based contracts and centuries-old legal systems for enforcement. Execution of complete transactions usually depends on trusting a single centralised entity to operate the interfaces between the Traditional and Internet worlds - handling legal issues, settling payments etc. Under such a model, economics of scale and network effects tend to favour a few commercial giants at the expense of smaller companies. This is a net loss to the economy: stifling innovation, excluding new competition, allowing monopolistic behaviour and "locking in" consumers without many practical choices. ### The Internet of Value We envision a system that enables value exchange while solving the problems of traditional approaches - a network of economic agents that serves the needs of the digital economy. The seven key properties that are essential to this vision are that it must be: * **Global** - a single, global network for everybody, without artificial boundaries. Potential is maximised when everyone can transact with everyone else with a shared global state. * **Open** - a decentralised, technology independent system where anyone can participate without asking for permission. No barriers to entry. * **Automated** - able to support atomic, end-to-end transactions that are reliably executed using trusted and reliable smart contracts. * **Secure** - protecting against security threats so that users can be confident of the safety of their digital assets and transactions. * **Extensible** - capable of supporting unlimited innovation in the types of assets and applications created. Like the Internet, it must not constrain what people can build, and it must make it easy to innovate. * **Fast** - quick enough to confirm economic transactions in less than a second, meeting the demands of consumer applications and retail payments. * **Cheap** - inexpensive to utilise, so that nobody is excluded by high costs and most kinds of applications are economically viable. Convex has been designed from the ground up to provide these properties. ### Applications The Internet of Value will enable **decentralised applications** that involve digital assets and trusted value exchange. Just as anyone can create a website on the Internet, anyone can create a decentralised application for the Internet of Value. Convex is designed to make the process of building, operating and using such applications as simple and effective as possible. There is no practical limit to the ideas that could be implemented given an open and extensible system. Some notable ideas include: * Implementation of cryptocurrencies, utility tokens, and other forms of decentralised assets * Economic transaction mechanisms (auctions, shops, exchanges) where terms and conditions are automatically guaranteed and executed by Smart Contracts * Games and entertainment where rules include ownership of tradable digital assets * Educational environments for collaborative and interactive programming * Immutable records of documents / data provenance * Publicly accessible databases and registries ### Why is Convex needed? Convex is needed because, despite the vast potential of the Internet of Value, a technology did not previously exist to enable it in a satisfactory way. We need a system that is *simultaneously* Global, Open, Automated, Secure, Extensible, Fast and Cheap. Previous systems fall short on one or more of these points. Convex builds on ideas around decentralised technology popularised through blockchain innovations in recent years but was motivated by a desire to build something better than blockchains can offer. For example, Bitcoin does well on Global, Open and Secure, but is not Fast or Cheap. Key additional motivations for the creation of Convex, while not strictly central to the mission of supporting the Internet of Value, include goals such as: * Help the environment by supplanting systems based on Proof of Work. * Make decentralised software development more productive, engaging and fun * Establish new paradigms of decentralised programming around globally shared, consistent state ### Prior Innovation The space of decentralised technology has seen massive innovation in recent years, many of which have inspired the Convex project. It is not the purpose of this White Paper to chronicle all these exciting developments in detail, but some particularly significant events are worth noting: In 2009, Bitcoin was launched by Satoshi Nakamoto, which demonstrated for the first time that a digital currency could be operated on a fully decentralised, secure network using a Proof of Work consensus algorithm. The ability to prevent "double spending" using a purely decentralised, online method was a revelation that hinted at the possibility of entire economic systems migrating to the Internet. In 2015, Ethereum was launched, which build upon the ideas of Bitcoin but added a decentralised virtual machine (EVM) capable of executing Turing-complete smart contracts with a global state machine. This enabled a wave of innovations such as tokenisation of assets with smart contracts, and the first attempts at Decentralised Autonomous Organisations. These innovations paved the way for significant experimentation in the space of digital currencies, tokenisation and cryptoeconomics. The space has attracted massive investment and seen vigorous innovation in recent years, hinting at the enormous opportunities presented by digital value exchange. ### Technical Challenges However, Blockchain technologies suffer from a range of issues which have proved hard to resolve. On the technical side, Ethereum founder Vitalik Buterin noted the "Scalability Trilemma" which is that is extremely hard to achieve the combination of: * **Scalability** - Ability to offer performance comparable to traditional payment systems such as VISA * **Security** - Resistance to attacks on assets and information integrity (such as double spending of digital currency) * **Decentralisation** - Ability to operate free from centralised control by a single entity or group of powerful entities Given that security is essential, and that without decentralisation blockchains offer no compelling reason to switch from centralised solutions, blockchains have generally **sacrificed scalability** to the extent that they are not practical for large scale use cases. Other technical challenges became apparent over time. Some notable issues: * **Energy wastage** - The use of "Proof of Work" consensus algorithms has resulted in vast and wasteful energy consumption. This is particularly apparent in the Bitcoin and Ethereum 1.0 networks, which rely on vast amounts of computing power dedicated to hashing. * **Front-Running** is a particularly important problem in decentralised finance, where it is possible to steal value from others by quickly inserting a transaction before that of another user, and is exacerbated by the problem of long times to process blocks. * **Cross chain integration** presents a particular problem where different decentralised platforms provide different specialised capabilities but need to be integrated to form a combined solution. The problems of maintaining consensus, security, reliability etc. are magnified in such situations. * **Latency** - The time taken for most blockchains to confirm final consensus is frequently too long to offer a positive user experience. This inability to provide quick confirmation and feedback is a significant barrier to mainstream user adoption of decentralised applications. * **Upgradability** - Both networks themselves, and the specific implementations of smart contracts, are difficult to upgrade, in some cases requiring a "hard fork" of the network. * **State Growth** - Decentralised databases have an issue with *state growth*, defined as the increasing requirement for peers to store information that accumulates over time. Because on-chain data must be retained, potentially indefinitely (to satisfy future on-chain queries or smart contract operation), this imposes increasing economic costs on Peer operators. The economic costs of this often do not fall on those responsible for creating new state - leading to an instance of "the tragedy of the commons". Over time, this may make it impossible for normal machines to run a node, effectively halting decentralisation. Convex presents a solution to all these challenges, and as such we believe it allows a significant evolution "Beyond Blockchain" to deliver the Internet of Value. The remainder of this White Paper explains how we achieve this. ## Convex Overview ### The Convex Solution Convex solves many of the technical challenges of Blockchains. With reference to the Scalability Trilemma, Convex offers: * **Internet-scale performance** - Convex offers the capability to operate at transactions volumes comparable to or greater than centralised networks like VISA transaction levels, even *before* scalability solutions such as Layer 2 solutions, state sharding or optimistic lookahead approaches are applied. Early benchmarking suggests peers running commodity hardware may be able to handle in excess of 100,000 transactions per second. * **Byzantine Fault Tolerance** - Convex meets the strongest possible threshold for security under the model of Byzantine threats. Consensus formation is guaranteed (and stable) as long as at least 2/3 of the effective voting power of the network follows the protocol honestly. * **Full Decentralisation** - The network operates under a permissionless Peer-to-Peer model: Anyone can operate a Peer in the network, anyone can submit a transaction for execution, and transactions cannot be censored (subject to the usual security assumptions). But Convex is not just a faster Blockchain - it is a full stack platform for building digital economic systems. As such, it combines several capabilities that together enable construction of new classes of applications. Some technical highlights of the Convex design that support such applications include: * **Actors**: Programs that execute autonomously in the Convex environment with deterministic and verifiable behaviour, suitable for managing assets and enforcing Smart Contracts * **Convex Virtual Machine (CVM)** - a fully Turing complete programming and execution environment. By designing the execution model around the Lambda Calculus, the CVM supports functional programming capabilities, with novel combination of language and runtime features to facilitate writing decentralised applications. We implement a working Lisp compiler "on-chain". * **Decentralised Data Value Model** - A data model supporting powerful features such as orthogonal persistence, memory accounting, incremental data sharing and cryptographic verification * **Performance**: High throughput (many thousands of transactions per second), low latency execution (zero block delay, ~1 second or below transaction confirmations) * **Security**: Cryptographic security for control over all user accounts and assets, with Byzantine Fault Tolerance at the decentralised network level. The main sections of this White Paper describe the key subsystems of Convex that make all this possible: * The **Consensus Algorithm** enables the Convex network of Peers to agree on a consistent, replicated state of the world - essential to provide reliable confirmation of transactions in a decentralised system * The **Execution Engine** performs computations necessary to implement secured transactions on the Convex Network - it is the mechanism that updates the Global State and enforces Smart Contracts. * The **Storage System** enables Peers to store and manage data volumes with a level of scale and performance necessary to support a high throughput of transactions and data volume. * The **Memory Accounting** model keeps track of data sizes stored and transferred by users, allowing Convex to correctly align economic incentives to make optimal use of on-chain memory. We summarise these areas below: ### Consensus Algorithm Convex, like other decentralised systems, depends upon a consensus algorithm to ensure that everyone agrees on a single version of the truth - this is a precondition for any decentralised economic system that needs to enforce ownership of digital assets. Convex allows transactions to be submitted to the network at any time, grouped into blocks by individual peers. In contrast to traditional blockchains, the blocks are not linked to previous blocks - there is no "chain" as such. Relaxing this requirement enables Convex to handle new block submissions from multiple peers concurrently, significantly improving performance. The role of the consensus algorithm is to create an **ordering** of blocks. A stable order solves the famous "double spend" problem by ensuring that only the first transaction is able to spend any given funds or assets. Any later transaction that attempts to spend the same funds will fail. The algorithm operates by implementing a novel variant of a **Conflict-free Replicated Data Type** (CRDT), which can be proven to converge to a stable consensus through a few rounds of random gossip between Peers. This approach is efficient, robust to temporary failures, and provably secure even in the presence of malicious or faulty peers (i.e., it is "Byzantine Fault Tolerant" under reasonable security assumptions). The Convex consensus algorithm also makes use of **Proof of Stake**, a mechanism by which peers are required to deposit an economically significant stake to ensure their good behaviour and be granted participation rights in the consensus protocol. This avoids the wasteful use of resources and energy that plagues systems based on "Proof of Work". As well as offering substantially improved performance, this means that Convex presents an environmentally friendly alternative to previous models such as Bitcoin or Ethereum. ### Execution Engine Convex implements a full virtual machine for smart contracts, the **Convex Virtual Machine (CVM)**, designed to facilitate digital economic transactions. Given an initial CVM state and an ordering of blocks (and therefore transactions) from the consensus algorithm, the CVM can process the transactions and compute a new updated State. The latest state contains information of interest to users of the Convex network, in particular the record of ownership of digital assets. The CVM has the capability to execute arbitrary programs. Some programs may be fully autonomous, which we term **actors*. These are typically used to implement Turing-complete **smart contracts** which in turn can be used to express the logic of digital assets and decentralised applications. Importantly, the CVM operates on a **global state** - execution of transactions (once these are in consensus) is equivalent to creating a new state through a state transition function. Some particular innovations of interest to facilitate the development of decentralised applications: * **Decentralised Data Values (DDVs)** - A system of data types and structures enabling efficient and secure replication of data across the Convex network, and supporting the implementation of the CVM. The CVM works with a wide variety of data types enabling construction of powerful applications with optimised performance. * **Convex Lisp** - A powerful language where CVM code is itself expressed as decentralised data. The compiler itself executes on-chain - giving developers and Actors the power to construct, compile and deploy new actors on-chain without external tools. This enables systems of on-chain "meta actors" - actors who can autonomously create and manage other actors. * **Scheduled Execution** - The protocol allows for deterministic execution of Actor code at any future point in time. This allows for more advanced, time-based processes to be implemented on chain (without such a feature, smart contracts would need external systems and events to trigger execution at specific times, such as the Ethereum Alarm Clock ) * **Execution Worlds** - Each account on the network (external user or Actor) is granted a secure, scriptable code execution environment with its own database. This enables highly interactive use of the CVM by advanced users. ### Storage System Convex implemented a novel storage scheme, specifically designed to support the requirements of Convex DDVs. Key features of this system include: * **Content addressable storage (CAS)** - The key for every value in the database is the cryptographic hash of its encoding. This means that given a single root storage hash, an entire Directed Acyclic Graph (DAG) of values is directly reachable by recursively fetching nested values. * **Smart References** - references to data that can be lazily loaded and verified (via CAS), allowing just a small, required subset of data to be accessed on demand. * **Orthogonal Persistence** - DDVs used in Convex (such as the CVM state) are stored in a virtual database which may be much larger than main memory and is completely transparent to the user. This opens up opportunities for future scalability and sophisticated Actors capable of working with large databases. * **Novelty Detection** - The design of the storage system enables Convex to detect *novel* information when it is written to storage. This is important to reduce bandwidth requirements: only novel information will typically need to be broadcast to the Peer network. * **Proofed Persistence** - Certain proofs relating to the validation of data are persisted along with the data itself. This is an important optimisation: Entire large data structures can be verified in O(1) time by checking the cached proof. An important feature *excluded* from the storage system is that of "update". Once written, data values are immutable and cannot be changed. This limitation is appropriate given that keys are cryptographic hashes of value encodings: finding a different data value that maps to the same key would require breaking SHA3-256. However, this exclusion is also an advantage: it reduces the need for more complex database features such as index updates The database engine itself is called Etch. Etch is an embedded database engine optimised for these specific requirements. We believe that building a customised engine is a worthwhile investment, because of the specific feature requirements and performance improvements possible. Etch is at least an order of magnitude faster than using more traditional, general purpose databases. The Storage System supports optional garbage collection for Peers that wish to compact storage size. A Peer is only required to maintain the current state, and a short history sufficient to participate in the consensus algorithm. Of course, Peers may choose to retain additional information for historical analysis. ### Memory Accounting Decentralised databases have an issue with *state growth*, defined as the increasing requirement for peers to store information that accumulates over time. Because on-chain data must be retained, potentially indefinitely, to satisfy future on-chain queries or smart contract operation. There is no option to discard data arbitrarily: a good Peer cannot do so and maintain correct participation in the consensus protocol. This growing demand for storage space presents a significant problem. - It creates a requirement for Peers to incur increasing storage costs over time, for data that must be retained indefinitely - There are perverse incentives at work: a user might pay a one-off cost to store data on-chain, but does not bear the cost of indefinite future storage (which falls on peer operators) - Over time, careless use of on-chain storage may make it impractical for a typical individual to operate a Peer with normal computer hardware and storage capabilities - This problem might be manageable for low-transaction-volume platforms, but is clearly unacceptable for systems such as Convex that are designed to handle high volumes of transactions for sustained periods of time Convex implements a novel solution of Memory Accounting to help manage the problem. - Each user is given a **memory allowance**, which is a fully fledged second native currency on the Convex Network - Memory allowance is consumed when on-chain storage is allocated, and released when stored objects are deleted (this can be efficiently tracked by careful integration with the storage subsystem) - A automated "memory exchange" is provided that maintains a pool of liquidity with memory available for users to purchase. This regulates the maximum size of the on-chain state based on supply and demand. This pool can be increased over time to allow for reasonable state growth, improvements in technology and to discourage hoarding. - By this mechanism, a fair market price for memory is established that creates an economic incentive for efficient memory usage. ## Technical Description ### Design Rationale It is worth reflecting on the logic behind the design of Convex; this logic has driven the majority of key design decisions for constructing the Convex system. Convex can be categorised as a new type of Decentralised Ledger Technology (DLT). Convex works on the principle of proving a globally shared state on a permissionless decentralised network which executes instructions (transactions) on behalf of users: a "public computer" that everyone can access but where nobody has absolute control. The motivation for the development of a DLT system is that it can act as the foundation for the Internet of Value - a system of decentralised value exchange built in the open spirit of the original Internet. But is this necessary? Could it not be done in a simpler way? In this section we argue that these capabilities are necessary and sufficient (given the additional obvious assumption of adequate performance). It is important to consider why the Internet itself does not already function as an "Internet of Value". The key insight here is that the original Internet is primarily a **stateless** system that enables communication between different participants. A system of digital value exchange requires **state** - at the very minimum it must be able to record the ownership of digital assets (in Bitcoin, for example, this state is manifested in the set of UTXOs). While clients and servers on the Internet individually store and manage their own state, such state is inadequate for decentralised value exchange because it is susceptible to corruption or arbitrary modification by the party that controls the relevant network nodes. At scale, such centralised state cannot be trusted. If we are unable to trust centralised actors as the sole arbiters of truth regarding the state of assets, the solution must therefore involve decentralised verification. It must furthermore ensure **consensus** in order that the whole network sees the same verified state. Assets cannot reliably be used for value exchange if there is ambiguity over ownership (the "double spend" problem). To provide the basis for a global, open network, the consensus itself must be **global** and available to all participants. Because we wish to ensure openness and avoid the issue of centralised control over the system, it must also be **permissionless**; any actor can participate on an equal basis and not be excluded or censored by other participants wielding excessive power. This ideal of decentralisation presents the problem that some actors in the system may be malicious. They may be actively attempting to defraud other actors - a significant problem when valuable digital assets are at stake. Furthermore, even if not malicious, software or hardware failures can potentially disrupt the system. We therefore require the property of **Byzantine Fault Tolerance** - which can be informally characterised as resistance of the consensus mechanism to malicious attacks and failures. Theoretical results (e.g. the seminal work by Leslie Lamport) have demonstrated that such consensus requires 2/3 of participants to be good actors - that is, Byzantine Fault Tolerance is possible to achieve as long as no more than 1/3 of actors are malicious or faulty. We would like the Internet of Value to operate with a consensus algorithm that achieves this theoretical optimum level of security. What is the nature of digital assets, and what is necessary to implement them? For assets to be meaningful, they must obey rules. Ownership is the most obvious example; only the owner of an asset should be able to use it in an economic transaction. Other rules may also apply: for example a financial option may includ a right to "exercise" the option in exchange for some other underlying asset. Since assets only have value if owners trust that their rules will be enforced, we need a system of encoding and executing these rules in an automated, verifiable way as part of the decentralised protocol - such rules must be implemented as **smart contracts**. While it would be possible to create a simple system of smart contracts that tackle many useful applications without full programmability, the Internet of Value calls for extensibility to new forms of assets that may not be originally anticipated by the designers of the network. We therefore require a **Turing complete** smart contract execution model (capable of performing any computation) if we are to avoid the risk of future limitations preventing important digital asset classes from being created. To ensure that only valid transactions are executed on digital assets with the authorisation of the owner, we need a secure way to validate the authenticity of transactions. This is fortunately a well-studied problem that can be solved with cryptographic techniques, and in particular **digital signatures**. Authenticity of a transaction can be validated through the use of a secret private key held by users, and a public key that is visible to all. Maintenance of the network consensus invariably requires resources; powerful servers with compute capability, significant storage and network bandwidth are necessary to operate the consensus algorithm and execute transactions 24/7 at global scale. These resources are not free. To compensate their operators for the economic cost of their services there is a need to impose **transaction fees** for usage. Without some economic cost of transacting, the network could be swamped by low-value or malicious transactions that consume excessive resources. The need to charge a transaction fee leads to a form of **native currency**, known as Convex Coin - a digital currency to pay for the usage of network services. Finally, we note some practical considerations. Information must be durably and immutably maintained. The consensus algorithm frequently communicates between users of the network, which requires systems for **storage** and **transmission** of data. Such systems need to be **efficient** to provide necessary storage, and they must ensure **integrity** to allow recovery from faults and malicious tampering. The remainder of this White Paper describes the technical implementation of Convex, which implements all the above capabilities in order to provide a foundation for the Internet of Value. ### A note on Values As a distributed information system, Convex must deal with the representation of data. We therefore rely frequently on the definition of a **Decentralised Data Value** (DDV) which has the following properties: - It is immutable: the information content cannot be changed once constructed - It has a unique canonical encoding as a sequence of bytes, which is used for both network transmission and storage - It can be assigned a unique **Value ID** (VID) which is defined as the SHA3-256 cryptographic hash of the value's Encoding. This serves multiple purposes, most importantly for cryptographic verification and as a key for various data structures (e.g. hash maps and content addressable storage). - Other DDVs can be recursively nested inside - in essence forming a Merkle Tree, which becomes a Merkle Directed Acyclic Graph (DAG) with structural sharing of identical children. Where we refer to "value" or "data value" in this document, we are generally referring to DDVs. ### Consensus Algorithm #### Peers and Stake Convex defines the set of actors that participate in the consensus algorithm as **Peers** in the Network. Anyone may operate a peer by providing an economic **Stake** in Convex Coins. The size of the peer stake determines the relative voting power of the Peer in the consensus algorithm. Stake is locked up and may be forfeited if bad behaviour is provably detected. Stake could also be appropriated by malicious actors if the Peer does not maintain strong security for their system (in particular the security of the peer's private key). Requiring a stake is therefore a key aspect of the economic incentive for peers to maintain the security of the network. As a reward for helping to operate and secure the network, Peers are entitled to a share of fees for transactions executed on the network, plus other incentive pools. Their reward is proportional to peer stake, creating another positive incentive for Peers to provide more Stake and greater security. #### State The key task of the peer network is to securely store and update the **global state**. The state represents the complete information in the CVM at any point in time. The Convex Network operates as a globally replicated state machine, where new updates cause changes to the current State. Updates are defined on a globally consistent basis according to the sequence of transactions confirmed through the CPoS consensus algorithm. The latest state of the CVM network after all verified transactions in consensus have been executed is called the **consensus state**. The state is represented as an immutable decentralised data value that includes: * All Account information and balances * All Actor code, static information and current state * All information relating to active Peers and staking * Global information (such as the latest Block timestamp) Since it is a DDV, it follows that a state is a Merkle DAG, and has a unique value ID (VID). This means that if two peers compute a state update and announce the VIDs, it is possible to validate immediately if they computed the same resulting state. #### Transactions and Blocks A **transaction** is an instruction by any network participant (typically users of Client applications) to affect an update in the Consensus State. Transactions are digitally signed to ensure that they can only update the Consensus State if they are authorised for the holder of the corresponding private key. Transactions are grouped into **blocks**, which contain an ordered sequence of Transactions and some additional metadata (most importantly a Block timestamp). The inclusion of transactions in blocks is at the discretion of the Peer to which they are submitted. Users may choose to utilise any peer for this purpose, but normally should prefer to submit transactions to a peer that they trust to behave correctly, since this discretion could be abused by the receiving peer: - The peer could ignore a transaction and neglect to propose it for consensus - The peer could insert its own transaction(s) before the user's transaction, potentially executing a front running attack #### Reduction to the Ordering Problem Consensus in a decentralised state machine can trivially be achieved with the combination of: * Agreement on some initial genesis State: `S[0]` * Consensus over the complete ordering of Blocks of transactions `B[n]` * A deterministic **state transition function**, which updates the State according to the transactions contained in a Block: `S[n+1] = f(S[n],B[n])` This construction reduces the problem of generalised consensus to the problem of determining consensus over Block ordering. The CVM execution environment provides the state transition function, which is orthogonal to the consensus algorithm but provides the deterministic computations to compute the new consensus state (given any ordering of blocks in consensus). We define the **Consensus Point** to be the number of Blocks confirmed by the consensus algorithm in the ordering, and the consensus state is correspondingly the state obtained after applying the state transition function up to the consensus point. The hard problem that the consensus algorithm needs to solve is determining the **ordering** of blocks given the potential presence of malicious actors who may seek to profit by changing the order of transactions (e.g., a "Double Spend" attack). #### Block proposals Convex block proposals differ from traditional blockchain solutions. The latter have focused on mechanisms to determine which participant gains the right to propose the next block, which includes a hash of the previous block in order to extend a linked chain of blocks. This was the basis for the original Bitcoin Proof of Work algorithm (which used the ability to mine cryptographic hashes as the basis for allowing a miner to publish a block and claim the corresponding block reward). Other approaches involving selecting a "leader" to publish a new block have some notable problems: * It is a complex task to determine which participant should be the next leader, at least in a way that simultaneously works efficiently, provides security in the presence of potential malicious actors, and is guaranteed to make progress in cases such as leaders becoming unavailable. * Including the hash of the previous block in a chain creates an inherent data dependency that limits the ability to propose blocks in parallel and increases latency - each leader must build upon the work of the previous leader sequentially, which implies a minimum lower bound on the block time (given fundamental physical constraints). * It is necessary to make sure that the leader possesses the transactions that should be included in a Block. This implies the need for a mechanism to transmit transactions across peers prior to block creation (e.g., with the "mempool" used by Bitcoin to share pending transactions), which in turn implies additional communication costs and many opportunities for attackers to launch "front running" attacks on transactions that they observe. Convex eschews the idea of selecting a leader. We maintain the principle that **any Peer may propose a new Block at any time**. This "leaderless" approach has some important desirable consequences: - Blocks can be **instantaneously proposed**, without a Peer having to become a leader, perform any expensive computation, or wait for confirmation on any previous Block. - Users or Clients can **select a trusted Peer** to publish their transactions, rather than forwarding them to a leader that may not be trustworthy. As previously noted, this is an important mitigation against censorship and front-running attacks. - Blocks can be **independent of all previous Blocks** - they do not necessarily form a "chain" linked by cryptographic hashes. - It is possible for multiple Peers to propose Blocks **concurrently** for inclusion in consensus at the same time. This removes a major bottleneck compared to systems that require on some form of sequential leadership e.g. ??????. An ordering includes all blocks up to the current consensus point. A peer may propose a novel block for consensus by appending it to the ordering (alongside any other additional blocks still to be confirmed in consensus). #### Convergent Consensus Convex uses a variant of Conflict-free Replicated Data Types (CRDTs), ensuring that the network converges to consensus. CRDTs have the provable property of eventual consistency. All nodes eventually agree on the same value of an ordering up to the agreed consensus point. The CRDT is implemented through: * A **Belief** data structure, which represents a Peer's view of consensus across the whole Network, including the latest Block Orderings from other Peers * A **Belief Merge Function**, which: * Combines any two (or more) beliefs to create an updated belief * Is idempotent, commutative and associative with respect to the merging of other beliefs This effectively forms a *join-semilattice* for each peer, and satisfies the conditions required for a CRDT. Repeated applications of the Belief Merge Function on Beliefs propagated by peers automatically results in convergence to a stable consensus. Digital signatures ensure that peers can only validly update that part of the overall Belief structure that represents their *own* proposals. No peer can impersonate another Peer and full cryptographic security is maintained throughout the operation of the consensus algorithm. The Ordering of one or more other peers could be removed by from the Belief of a malicious peer, perhaps in an attempt to censor transactions. However, this will be an ineffective attack against the Network; the unchanged ordering relayed via other peers will ultimately be merged into the stable consensus. #### Stake-weighted voting During the convergence process conflicts in proposed block Orderings from different Peers are resolved by a system of convergent stake-weighted voting. [A diagram would be really helpful here] At each belief merge step, peers compute the total share of stake voting for each proposed block in the next position after the current Consensus Point. Peers have a view of the Orderings proposed by all other Peers. Stake-weighted voting is applied iteratively to future proposed blocks, but only counting the votes by peers that have supported the winning ordering up to this point. Supporting a minority block causes peers to be temporarily excluded from the vote on following blocks. Peers that vote for orderings inconsistent with the majority cannot influence the ordering of any subsequent blocks. There is therefore an incentive for peers to adopt an ordering consistent with the majority. Once the overall winning ordering has been determined, any peer can append any new blocks it wishes to propose, adopting this ordering as its own proposal. This ordering is signed and incorporated into the peer's own belief, which is then propagated onwards to other peers. As an illustration, consider three Peers that are initially in consensus with respect to an ordering of blocks `XXXX` but peers `A` and `B` propose new blocks `Y` and `Z`: ``` Peer A: (stake 20) ordering = XXXXY Peer B: (stake 30) ordering = XXXXZ Peer C: (stake 40) ordering = XXXX ``` Peer `C` observes the orderings of peer `A` and `B` (after propagation of beliefs). It sees two conflicting proposals, but because Peer `B` has the higher stake it takes this ordering first. It then appends the other Block it has observed: ``` Peer A: (stake 20) ordering = XXXXY Peer B: (stake 30) ordering = XXXXZ Peer C: (stake 40) ordering = XXXXZY (updated) ``` Peer `A` now observes the orderings of the other peers. Since there is still a conflict, it calculates the vote for each ordering and sees that there is a 70-20 vote in favour of having block `Z` first (and a 40/0 vote in favour of block `Y` next). It therefore adopts same the same Ordering as proposed by Peer `C`. ``` Peer A: (stake 20) ordering = XXXXZY (updated) Peer B: (stake 30) ordering = XXXXZ Peer C: (stake 40) ordering = XXXXZY ``` Peer `B` now observes the orderings. It sees everyone agreed on block `Z`, and a 60-0 vote in favour of Block `Y` next. It therefore adopts this winning ordering as its own: ``` Peer A: (stake 20) ordering = XXXXZY Peer B: (stake 30) ordering = XXXXZY (updated) Peer C: (stake 40) ordering = XXXXZY ``` This procedure provably converges to a single ordering. Any situation where peers are voting for different blocks (in any position) is unstable. It will converge towards one outcome, since peers will switch to an ordering calculated to have a slight majority. After a few rounds of belief propagation, all good peers will align on the same ordering. #### Stability The belief merge procedure outlined above has many desirable stability properties even in the presence of some proportion of malicious adversaries (which are considered "bad peers"). Some of these are listed below: ##### 51% Stability with Good Peer Majority If more than 50% of peers adopt the same ordering, this majority consists entirely of good peers and they are mutually aware of each other's agreement, then the ordering is provably stable no matter what any adversaries subsequently attempt, since the adversaries cannot cause any good peer to change their vote. ##### 51% Stability with Rapid Propagation Assuming that: - More than 50% of peers (some of which may be Bad Peers) adopt the same ordering - less than 50% of all peers are bad peers - Beliefs are propagated quickly to all other peers (at least before the next round of belief merges) Then the ordering will be provably stable since a majority of good peers will adopt the same Ordering during the next belief merge. ##### 67% Stability vs. Irrelevant Alternatives Assuming that: 1. At least 2/3 of all Peers are aligned in proposing the same Ordering, and are aware of each other's orderings 2. Less than 1/3 of Peers (by Staked voting weight) are Bad Peers, the remainder (>2/3) are Good Peers 3. Bad Peers may collude arbitrarily, but do not have a Belief propagation speed advantage (on average) relative to good peers 4. Non-aligned good peers do not initially have any significant support for any conflicting ordering. Then the ordering is provably stable, since: - By (1) and (2), the number of good peers within in the aligned set of peers must strictly outweigh the Bad Peers. - Whatever the bad peers do (including switching from being in the aligned group to supporting a new conflicting ordering), their new ordering will still be outweighed by the Good Peers which are already in alignment - Therefore, the initially aligned Good Peers will win 50%+ majority with good peers, since they will win a propagation race by sharing their beliefs which will bring the majority of remaining non-aligned good peers as per assumption (3) ##### 75% Stability vs. powerful adversaries Assuming that 75% of peers are aligned in proposing the same ordering, and are aware of each other's orderings, the ordering is stable as long as less than 25% of Peers are Bad Peers. This hold true even against powerful adversaries with capabilities such as: - Ability to temporarily isolate and trick non-aligned Peers into adopting a conflicting proposal - Ability to censor or delay arbitrary messages on the network (as long as at least one Belief propagation path eventually exists between any pair of good peers) - Ability to delay the activity of any good peer [What if a single bad peer holds a majority of the stake, say 70%, can it take full control of the consensus? Is this a governance issue?] #### Determining consensus Even after a stable ordering is observed, consensus must be confirmed. This is achieved through a decentralised implementation of a 2-phase commit. Once a 2/3 threshold of peers are observed by any Peer to be aligned on the same Ordering up to a certain Block number, the peer marks and communicates this number as a **Proposed Consensus Point (PCP)**. The Peers propagate this PCP as part of their next published belief, attached to their ordering. Once a 2/3 threshold of peers are observed to have the same proposed consensus point with the same ordering, this value is confirmed by the peer as the new **Consensus Point (CP)**. From this point on, Good Peers will consider the Consensus final. Consensus is **guaranteed** providing: - A stable ordering is reached where a majority of peers consistently propose the same ordering - At least 2/3 of stake is held by good peers that are active in the network This follows from the fact that given a majority for a stable ordering, all good peers will eventually adopt the same ordering and therefore the network will pass both thresholds. #### Illustration Consider a case where all peers A, B, C, D and E initially agree on a consensus ordering (labelled `o`). At this point, peer B receives a set of new transactions, composes these into a block and produces a belief with an updated ordering (`x`), including the new proposed Block. Initially, this is unknown to all other peers. We can visualise this initial situation as a matrix, where each row is the belief held by one peer, and each column represents the latest signed ordering observed by each peer from another peer. Each Peer also has knowledge of the current consensus defined by `o`, which is also its proposed consensus. ``` ABCDE Consensus Proposed Consensus A ooooo o o B oxooo o o C ooooo o o D ooooo o o E ooooo o o ``` Because it has a new Belief which represents novelty to the Network, Peer B propagates this Belief to other Peers. The other Peers observe that Peer B has proposed a new Ordering `x`, and incorporate this into their Belief regarding Peer B: ``` ABCDE Consensus Proposed Consensus A oxooo o o B oxooo o o C oxooo o o D oxooo o o E oxooo o o ``` With this information, all Peers are aware of a new Ordering. They validate that this is consistent with the previous Consensus Ordering `o`, and because it is a simple, non-conflicting extension of `o` (just one new Block appended) they automatically adopt it as their own proposed Ordering (the diagonal of the matrix). ``` ABCDE Consensus Proposed Consensus A xxooo o o B oxooo o o C oxxoo o o D oxoxo o o E oxoox o o ``` Another round of Belief propagation is performed. Now each peer is aware of the latest Ordering `x` being communicated by all other Peers. Since each Peer can now observe 100% of Stake proposing the same Ordering, it meets the threshold to be considered as the Proposed Consensus (the start of the 2-phase commit). ``` ABCDE Consensus Proposed Consensus A xxxxx o x B xxxxx o x C xxxxx o x D xxxxx o x E xxxxx o x ``` Finally, another round of propagation is performed. Peers now observe 100% of Stake supporting the same Proposed Consensus, so can confirm the Ordering `x` as the new Consensus (the completion of the 2-phase commit) ``` ABCDE Consensus Proposed Consensus A xxxxx x x B xxxxx x x C xxxxx x x D xxxxx x x E xxxxx x x ``` The network is now in a new quiescent state, with the Consensus Point advanced to include the full Ordering `x`, and ready to process the next proposed Block(s). In this simple case, the new Consensus is confirmed within just three rounds of Belief propagation: - Peer B communicates the new Block to other Peers - Other Peers communicate their adoption of the new Block - All Peers communicate Proposed Consensus (after which individual Peers can independently confirm Consensus) In more complex cases: * Multiple Peers may propose Blocks at the same time. In this case, stake-weighted voting would be used to resolve conflicts and determine which Blocks are included first. It may take an additional round or two to resolve such conflicts into a stable Ordering. Overall, this is more efficient since multiple Blocks are being brought into Consensus in a similar total number of rounds. * The network might not reach a quiescent state before further new Blocks are added. This is not an issue: consensus will be confirmed for the initial Block(s) while the new Blocks are still being propagated at earlier stages. * Some Peers might misbehave or be temporarily unavailable. Again, this is not a problem as long as a sufficient number of Good Peers are still operating and connected, since the consensus thresholds can still be met. Temporarily disconnected or offline Peers can "catch up" later. * The Peer Network may not be fully connected, potentially adding `O(log(number of peers))` additional rounds of propagation assuming that each Peer propagates to a small constant number of other Peers in each time period. In practice, not all these additional rounds may be needed because a smaller number of highly staked and well-connected Peers will be able to confirm consensus without waiting for the rest of the Network. #### Important note on complexity At first glance, the Convex consensus algorithm might be considered impractical because of the scale of data structures being shared. Consider a plausible high volume operating scenario: * n = 1,000 peers active in the network * r = 10 new blocks per second * s = 10k of data for each block (around 100 Transactions) * o = 1,000,000,000 Blocks of transactions in the each ordering (a few years of blocks) Each peer would theoretically be holding ~100 *petabytes* of information for their Belief, which would need to be transmitted in each propagation round, requiring a bandwidth in the order of many *exabytes* per second. Clearly this is not practical given current hardware or network capacity. Convex exploits powerful techniques to maximise efficiency: * Beliefs are represented as Decentralised Data Values that support **structural sharing**: identical values or subtrees containing identical values need only be stored once. Since orderings are identical up to the consensus point, these can be de-duplicated almost perfectly. * Peers are only required to actively maintain Block data for a limited period of time (e.g. 1 day of storage would be less than 10GB in this case) * The Decentralised Data Values support usage where only the **incremental change** (or "Novelty") can be detected. * The number of outgoing connections for each Peer is **bounded** to a small constant number of Peers that they wish to propagate to (typically around 10, but configurable on a per-peer basis) * Beliefs can be **selectively culled** to remove orderings from peers that have very low stakes and are irrelevant to consensus. This can be performed adaptively to network conditions if required: Peers may only need to consider the "long tail" of low staked Peers in rare situations where these are required to hit a consensus threshold or decide a close vote. With these techniques, Peers only need to propagate the novelty they receive (in this example around 100k of Block data per second, plus some accounting and structural overhead) to a small number of other peers. Bandwidth required is therefore on the order of 1-10MB/s (allowing for overheads and a reasonable number of Peer connections), which is certainly practical for any modern server with decent network connectivity. Overall complexity is therefore (factoring out constants): * $O(r \times s)$ bandwidth, scaling with the rate of new transaction data size * $O(r \times s)$ storage, scaling with the rate of new transaction data size * $O(\log n)$ latency, scaling with the logarithm of number of Peers (based on standard analysis of gossip networks) We believe this is optimal for any decentralised network that maintains consensus over a global state. Note that lower latency can be achieved by communicating to all peers simultaneously, but at the cost of significantly higher bandwidth. #### A note on Front Running Front running is difficult for an adversary to perform against the Convex consensus algorithm. While theoretically possible, it would require a sophisticated and well-resourced attacker. The main reason for this is that Transactions are not visible to any untrusted participants in the Network until *after* a new Block has been proposed by a Peer and propagated as part of a Belief, by which point it is already well on its way to being included in consensus. A user concerned about front-running attacks should submit vulnerable transactions exclusively via a well connected, well-staked Peer that is trusted not to be malicious, i.e. this Peer must not itself be helping to facilitate a front-running attack. In this scenario a front running attack would need to: * Listen to vulnerable transactions broadcast on the Network * Quickly generate a new Block with the Transaction(s) needed to execute the front-running attack * Have sufficient influence over consensus formation to ensure that the new Block is somehow re-ordered *before* the original Block (that is already approaching consensus) Practically, this attack would require the attacker to have more resources (Stake and network connectivity) than the original Good Peer *and all the Good Peers it is connected to*, since the original Block would already be ahead by at least one round of propagation by the time the attacker can observe it. Furthermore, the attack would be publicly visible and traceable to the Peer(s) that launched it: so even if successful the attacker would be vulnerable to blacklisting and/or real world legal repercussions. Assuming Good Peers are well-staked, and connect preferentially to other well-staked, trusted Peers with a known legal identity (which would be good practice, and should be configured as default behaviour), we believe such front running attacks will be highly difficult to execute and generally impractical from an economic perspective. ### Execution Engine The Convex execution engine is referred to as the Convex Virtual Machine (CVM). This is a general-purpose computational environment that is used to execute the State Transitions triggered by Transactions. #### Accounts The fundamental control mechanism for the CVM is via Accounts. There are two main types of Accounts, which differ primarily in the means that they can be controlled: * **User Accounts**: Accounts that are controlled by external users, where access is secured by Ed25519 digital signatures on Transactions. * **Actor Accounts**: Accounts that are managed by an autonomous Actor, where behaviour is 100% deterministic according to the defined CVM code. Actor functionality may be called directly or indirectly within an externally submitted Transaction, but only if this is initiated and validated via a User Account. It is important to note particular the two types of Account share a common abstraction. Both User Accounts and Actor Accounts may hold exclusive control over assets, allowing for decentralised value exchange mediated by smart contracts. This common abstraction is useful, because it makes it simple to write code that does not need to distinguish between assets controlled directly by a user and assets managed by a Smart Contract. User Accounts are **protected by digital signatures**. A transaction which uses a specific account is only considered valid if accompanied by a valid digital signature. Without access the corresponding private key, it is computationally infeasible for an attacker to submit a fake transaction for an Account. No external transactions are permitted on Actor Accounts - they operate purely according to the rules expressed in their code. #### Environments A novel feature of the Convex Account model is that each Account receives it's own *programmable environment* where variables, data structures and code can be dynamically defined and updated. Definitions held within different accounts cannot collide since they have independent environments. * For User Accounts, this behaves like a computer completely under the control of the user. Each user receives the equivalent of a fully functional "Lisp Machine", which can modify its own definitions and has read-only access to the environments of other Accounts. * For Actor Accounts, this can be used to store Actor code and state required for the operation of the Actor. Deployment of an Actor is equivalent to creating an Account and initialising the Actor's environment, with subsequent changes to the environment strictly controlled by a set of exported functions that can be externally called. We believe this is a powerful model to encourage rapid development and innovation: for example, a developer can easily experiment with code in their own user account, then capture the same code in the definition of a deployable Actor for production usage. Optionally, Actor Accounts can be utilised as **Libraries** of code for use by other Accounts. Since it is possible to create an immutable Actor Account (i.e., Any actor that lacks externally accessible code to change its own environment), this means that you can create Libraries that are provably immutable, and can therefore be relied upon from a security perspective never to change. Environments also support **Metadata** which can be optionally attached to any definition. This innovation is particularly useful to allow custom tags and documentation to be attached to library definitions in a way that can be inspected and utilised on-chain. For example, the metadata for a core function might look like: ```clojure { :doc {:description "Casts the argument to an Address. Valid arguments include hex Strings, Longs, Addresses and Blobs with the correct length (8 bytes)." :examples [{:code "(address 451)"}] :type :function :signature [{:params [a] :return Address}] :errors {:CAST "If the argument is not castable to a valid Address."}} } ``` #### Information Model Convex requires a standard information model. For consensus to be useful, well-defined data with clear semantics are necessary for operating smart contracts; parties must agreed precisely the information represented by consensus. information model design decisions have been driven by both theoretical and pragmatic considerations: - Representing types that are theoretically sound and fundamental to computing; such as vectors, maps and lambda functions - Providing types that are generally useful for developers of decentralised ledger systems - Supporting all the capabilities required for a Lambda Calculus - Using types that are conveniently represented in modern computing platforms (e.g., the use of 64-bit Long integers and IEEE 754 double precision floating point values) - Ensuring that information can be efficiently encoded to minimise storage and bandwidth requirements Convex implements a comprehensive set of data types, communication protocols and consensus algorithm. They are utilised both within the CVM and in the broader implementation of a Convex Peer. All data types available in the CVM are considered as Decentralised Data Values (DDVs) - immutable, persistent and structured for efficient network communication of information. ##### Primitive types Several basic primitive types are supported, consistent with a typical modern language and broadly equivalent to those available on the JVM: * `Byte` - an 8-bit unsigned integer * `Long` - a 64-bit signed integer * `Double` - an IEEE754 double-precision floating point value * `Character` - a UTF16 character * `Boolean` - `true` or `false` These behave generally as expected, with the important proviso that arithmetic is implemented exclusively using long and double types (other types are automatically upcast to long and double as required). There is also a set of primitive value types useful for programming on the CVM: * `Keyword` - a named value, most often used for map keys (`:foo`) * `Symbol` - a name generally used to refer to a value in an Environment (`bar`) * `String` - an arbitrary length sequence of Characters (`"Hello"`) * `Address` - a 64-bit identifier for an Account (e.g., `#1234`) Data values that are sufficiently small, including most of the above, have compact encodings that are **embedded** directly within the encoding of larger Data Values that contain them. This is an internal implementation detail, but important to reduce the overhead of storing and communicating many small values independently, which is transparent to CVM code. ##### Blobs A `Blob` is an arbitrary-length sequence of Bytes and is considered a first class value on the CVM. e.g. `0xa0b1c2d3e4f5` Cryptographic values such as Hashes are generally treated as small fixed-length Blobs. Internally, Blobs are stored as a Merkle tree of chunks of up to 4096 bytes in length. Blobs may exceed the size of working memory: they can technically be up to 2^63-1 bytes in length. Blobs could also be used as a basis for decentralised file storage, perhaps as a Layer 2 solution like IPFS. ##### Data Structures Convex supports a range of first-class data structures, primarily: * `Vector` - a sequence of values (e.g., `[1 2 3]`) * `List` a sequence of values usually used to represent code (e.g., `(foo bar baz)`) * `Map` a mapping of keys to values (e.g., `{:bar 1, :baz 2}`} * `Set` a set of values (e.g., `#{1 2 3}`) All data structures are immutable, functional data structures that support structural sharing based on an underlying tree representation (in fact, a Merkle tree). Critically, these provide efficient `O(log n)` operations for append, access, update etc. without requiring expensive "copy on write" operations to preserve immutability. These data structures are similar to the data structures frequently found in modern functional programming languages such as Clojure, Scala or Haskell. A moderately high branching factor (typically 16) is used. This is important because: * It facilitates faster lookups (less nodes to traverse by a factor of 4 vs. a binary tree) * It reduces the number of new node allocations required to update a path to a leaf node. * It reduces the number of hashes that need to be computed (a performance bottleneck in some cases) * There is a certain elegance and minor performance benefit in being able to index the tree using hex digits CVM data structures are used widely throughout the whole Convex implementation: For example the ordering of blocks in the CPoS algorithm is internally implemented as a vector. ##### Syntax Objects `Syntax` objects are wrapped values that contain both a raw value and optional metadata. Metadata may be an arbitrary Map, but typically would include such things as: * Source code references * Documentation * Information generated through macro expansion Syntax objects are inspired by Racket, and are generally used for code generation and compilation, although they are also available for use in regular CVM code if desired. They are marginally more efficient than storing a value and metadata as two separate fields in a Map, for example. The primary usages of syntax objects within the CVM are: * Allowing metadata to be attached to values in account environments (e.g., documentation for actor functions) * Supporting the implementation of the Convex Lisp compiler and macro system NOTE: In the future, syntax objects may be extended to implement a gradual type system such as seen in Typed Racket. Racket has demonstrated the value of syntax objects in helping to support future language evolution. ##### Nil Convex supports the value `nil` as a first-class value (which can be considered the sole member of the type `Nil`). By convention, core runtime functions generally return `nil` to indicate the absence of a value, for example looking up a value in a map with a key that is not present. `nil` values are considered as being "falsey" (equivalent to `false`) in conditional operations, which facilitates the technique of "nil-punning" popular in languages such as Clojure and Common Lisp. For example, the following is a typical pattern: ```clojure (if (lookup-optional-value ...) (true-branch ...) (false-branch ...)) ``` There is no direct equivalent of a `NullPointerException` since CVM objects do not implement methods, however careless use of `nil`s may result in type cast errors (e.g. `(+ 2 nil)`). ##### Records Certain CVM structures are defined as built-in record types, e.g. * `AccountStatus` * `PeerStatus` * `State` These are primarily used internally by the Convex Peer and CVM implementations, though for convenience they may be accessed and treated as Maps from field names to values in CVM code. NOTE: Supporting user-defined, row-polymorphic record types is under consideration for future implementation (probably in V2). ##### Functions Functions are first class objects suitable for use in functional programming. Convex implements functions in this way because they are fundamental and powerful constructs that allow the construction of effective programs without having to simulate them with lower-level constructs (e.g. a stack based model). The decision to emphasise first-class functions and functional programming is justified by the strong theoretical foundations of the Lambda Calculus. Important features of functions include: * Support for variable arity function application like `(+ 1 2 3 4)` * Full lexical closures (capturing values in the lexical environment at the point of creation). * Explicit tail-recursion support (recursively calling functions without consuming stack space) Many functions are provided as part of the runtime environment, generally available to users in the standard library `convex.core`. These functions provide the foundation for construction of higher-level functionality. In addition (adopting an idiom that has proved convenient in the Clojure language), data structures may be used in place of functions in some defined circumstances, e.g.: * Maps may be used as functions that implement map lookup: `({:foo 1 :bar 2} :bar) => 2` * Sets may be used as functions to test membership: `(#{1 2 3} 4) => false` * Vectors may be used to perform indexed lookup: `([1 2 3] 2) => 3` ##### Macros and Expanders Convex supports the use of macros, in the manner of most Lisps. Macros provide powerful code generation and templating facilities, allowing users to extend the language to add new programming constructs. In fact, a large proportion of the `convex.core` library itself is implemented using macros. Macros are also useful for generating efficient smart contracts, since they enable many computations to be performed once at compile time, reducing the cost of subsequent executions. For example, mathematical values required for liquidity curve calculations can be compiled into constants when liquidity curve Actors are deployed, eliminating wasteful computation when the Actor is subsequently called by users. Macros are implemented using the lower-level construct of Expanders, which are Functions that generate code at expansion time (i.e. just before compilation). The idea of Expanders as a fundamental language construct is covered in the 1988 Paper "Expansion-passing style: A general macro mechanism" (R. Kent Dybvig, Daniel P. Friedman & Christopher T. Haynes). Interested readers are encouraged to read this article to understand the detailed rationale for this approach, but perhaps the most important point is that Expanders are strictly more powerful and flexible than traditional Lisp macros. Macros and expanders present powerful possibilities for decentralised application, including automated code generation for new actors and smart contracts. ##### Ops Ops are low level, programmatic constructs that represent individual instructions on the CVM. All CVM code is compiled to a tree of Ops. They can be considered as the "machine code" instructions on the CVM. Currently the key Ops supported are: - `cond` - conditional evaluation - `constant` - load a constant value - `def` - modify environment (map of symbols to values) - `do` - sequential composition of operations - `invoke` - execution of a Function - `lambda` - instantiation of a Function - `let` - definition of values in a lexical scope - `lookup` - lookup of a value in the environment - `special` - access to special values in the CVM state and/or execution context These Ops resemble the basic primitives frequently found in an implementation of the lambda calculus - in particular `invoke` and `lambda` are direct implementations of "Application" and "Abstraction". It should be noted that certain other important constructs e.g., `cons`, `quote`, `=` etc. are currently implemented as functions in the runtime environment, so the base language of the CVM can be regarded as the combination of the Ops and runtime functions (accessed via the `invoke` Op), applied to arbitrary CVM values. #### Execution constraints Since the CVM supports Turing-complete computation, it is necessary to place constraints upon code execution to prevent erroneous, badly written or malicious code from consuming excessive resources. This is particularly important in a decentralised system because such resources are a global, shared cost. The CVM therefore constrains **time**, **space** and **depth**. ##### Time Convex constrains time by placing a "juice cost" on each CVM operation performed. Any transaction executed has a "juice limit" that places a bound on the total amount of computational work that can be performed within the scope of the transaction. The originating account for a transaction must reserve juice by paying an amount `[juice limit] x [juice price]` at the start of the transaction. Any unused juice at the end of the transaction is refunded at the same rate. The juice price is a dynamically varying price that adjusts with the amount of load on the Convex network as a whole: this is a cryptoeconomic mechanism to disincentivise transactions from being submitted at peak periods, and as a protection from DoS attacks by making it prohibitively expensive to flood the compute capacity of the network for a sustained period of time. If the juice limit has been exceeded, the CVM terminates transaction execution with an exception, and rolls back any state changes. No juice is refunded in such a situation - this penalises users who attempt excessive resource consumption either carelessly or maliciously. ##### Space Convex performs a complete, deterministic analysis of space usage by each Transaction, defined as the delta in the size of the Global State caused by the Transaction. This is an important execution constraint, without which there would be poor incentives for developers to be efficient with CVM memory usage (beyond paying the initial juice cost). Juice costs alone cannot be accurately used to constrain memory usage, because they are fundamentally a one-off "flow" cost that is immediately incurred, whereas space is an ongoing "stock" cost that is incurred by all Peers over time. This constraint is described in more detail in the "Memory Accounting" section of the White Paper. ##### Depth Convex places a limit on "stack" depth within Ops and Functions. While not strictly necessary (execution time constraints will at some point halt infinite recursion) a maximum depth is useful for two reasons: * Unbounded recursion should be discouraged in CVM code. The kinds of situations where it might be useful (heavy computations, or traversing large data structures, for example) should probably not be running on the CVM itself - this generally belongs in client or server code outside the CVM. * It makes the CVM implementation simpler and more performant, since the depth limit allows the underlying JVM stack to be safely used without the risk of `StackOverFlowError`s, and therefore removes the need to explicitly handle these. Currently the depth limit is 256. This could be relaxed if needed, but we currently do not see any realistic smart contract use cases that are likely to require this much stack depth, especially considering that the CVM supports techniques like tail recursion (which avoid consuming stack depth). #### Runtime environment The CVM defines a small core runtime system that provides CVM capabilities to CVM programs. These include: * Standard language control structures (loops, conditionals, error handling etc.) * Basic numerical functions (focused on 64-bit integer and IEEE 764 double precision floats) * Functions to manipulate and manage immutable persistent data structures (vectors, lists, maps sets) * Control of assets native to the Convex network, such as balances and stake * Ability to interact with Actors (deploying Actors, calling Actor functions) * Functionality useful for managing CVM state updates, e.g. transaction rollback * Language constructs necessary to support Convex Lisp (see below) The core system is designed so that these low-level capabilities can be easily composed to create higher level capabilities, through composition of data and functions. Runtime functions are generally exposed to the CVM in the `convex.core` library. At the same time, the capabilities of the runtime system are constrained so that they cannot break the rules of CVM execution necessary for deterministic state updates. There is no external IO capability, no ability for non-deterministic behaviour, and no ability to affect CVM state in a way that breaks the security model. CVM code is therefore fully "sandboxed" from the perspective of the overall Convex system. In many cases, the runtime system is optimised for performance - for example, methods to update CVM data structures are implemented in efficient, low level, compiled code. Emulating such operations in pure CVM code would be many orders of magnitude slower, so this approach allows us to provide sophisticated, immutable data structures and higher-level language features without compromising performance. #### Transparent persistence The Convex execution engine implements a system of transparent (sometimes also known as orthogonal) persistence. In this model, the CVM state size may exceed the working memory capacity of a Peer, and necessary parts of the State tree are loaded in from persistent storage on demand. This presents a significant conceptual benefit for the developer: there is no need to write any code to load or unload data from storage in normal CVM code. There is some additional implementation complexity for the CVM itself, but this is considered a worthwhile trade-off, especially since it simplifies the logic of other parts of the Convex Peer implementation (e.g., eliminates the need to explicitly handle the memory consumption growth of long Block Orderings generated by the CPoS consensus algorithm over time). In the current implementation, this is achieved with judicious reliance upon the very efficient JVM automatic memory management. This enables the following lifecycle for in-memory data values: 1. Values are initial created with strong (RefDirect) references, which ensure that they are held in memory for as long as they are potentially needed 2. At certain checkpoints (most importantly, after the successful processing of each Block) the current State is *persisted*. All Cells which are reachable but not yet persisted are written to storage, and references to them are converted from strong references to soft (RefSoft) references. This happens as an atomic operation. This is made efficient by the system of Novelty Detection which can identify the `n` new Cells to be persisted in `O(n)` time. 3. From this point onwards, the persisted objects may be garbage collected at any time by the JVM if memory pressure occurs. 4. If an attempt is made to access a value that has been garbage collected, the reference automatically fetches the associated data value from storage. This is guaranteed to succeed assuming that the previous persistence step was successfully completed. 5. Over longer time periods, it is possible to perform garbage collection on the storage itself by compacting the store to remove data that is no longer required by the current consensus state. Peers may choose to do this at their own discretion based on their operational requirements, or alternatively they may decide to preserve all data (for example in order to perform historical analysis) #### Convex Lisp The CVM includes a small, dynamically typed, embedded Lisp suitable for general purpose programming within the CVM environment. Convex Lisp draws inspiration from Common Lisp, Racket and Clojure. It is designed as primarily a functional language, with fully immutable data structures, as it is our belief that functional programming forms a strong foundation for building robust, secure systems. Convex Lisp was chosen as the first language implementation in Convex for the following reasons: * Experience with Lisp as a highly productive language for developers, particularly when manipulating data structures (as seen in data-driven development approaches with Clojure, for example). * It can be constructed using a very small number of simple, well-defined axiomatic primitives, which in turn are based on the Lambda Calculus. This provides a robust logical and mathematical foundation, suitable for the type of verifiable, deterministic computations that the CVM must support. * Lisp has a very simple regular syntax, homoiconic nature of code and ability to implement powerful macros. We hope this provides the basis for innovative new languages and domain-specific languages (DSLs) on the CVM. * Lisp compilers are small enough and practical enough to include as a capability within the CVM, avoiding the need for external compilers and tools to generate CVM code. * It is comparatively simple to implement, reducing the risk of bugs in the CVM implementation (which may require a protocol update to correct). * Lisp is well suited for interactive usage at a REPL prompt. This facilitates rapid prototyping and development of Actors in a way that we believe is a significant advantage for decentralised application builders looking to test and prototype new ideas. Developers using the Convex system are not required to use Convex Lisp: It is perfectly possible to create alternative language front ends that target the CVM (e.g. by constructing trees of Ops directly). Convex has experimental support for a JavaScript-like language (Scrypt) and community members are encouraged to innovate further in this space. #### Scheduled execution The CVM includes a specialised data structure called the Schedule that references CVM code to be executed under a specific Account at a defined future timestamp. The main purpose of the Schedule is to allow Actors to implement autonomous behaviour without the need to wait for an external transaction to trigger execution. This could be used to finalise a decentralised auction, to distribute the prize from a random lottery, to trade on a periodic basis, or to unlock assets that have been frozen for a specified period of time. The schedule is **unstoppable**, in the sense that once the consensus timestamp advances based a scheduled execution time, the associated code is automatically executed according to protocol guarantees. This execution is guaranteed to happen before any other transactions in a block submitted on or after the same timestamp. Scheduled executions currently cannot be cancelled, but this is not a serious limitation: Actors can simply implement code to ignore the event if it is no longer relevant. #### Garbage collection The CVM automatically garbage collects objects to which references are no longer maintained. The choice of including garbage collection in the CVM is motivated by the following factors: * **Convenience** - developers need not be concerned with manual memory management. In general, this is a significant productivity gain. * **Performance** Given a focus on immutable data, garbage collection offers significant performance advantages because references can be shared internally within the CVM implementation, as opposed to relying on expensive "copy on write" approaches. * **Security** - Mistakes in memory management are one of the most common defect types, often resulting in significant security issues (e.g. "buffer overruns"). Such risks are generally unacceptable for smart contract code securing significant digital assets. Short lived objects are garbage collected by the host runtime (the JVM). This will happen for most temporary objects created during the execution of CVM code. For those data values that are persisted to long term storage (e.g. because they become part of the updated CVM state), the host runtime may garbage-collect the in-memory copy. Peer operators may also choose to either garbage collect old data from long term storage, or alternatively maintain old data for historical analysis. Peers are only required to maintain object information necessary to execute the consensus algorithm (belief structures plus the proportion of CVM state relating to Peer information and stakes). For more details, see the section on Convergent Immutable Storage. ### Storage System Convex makes use of a specialised storage system that complements the design of the CVM. This provides significant performance advantages, since the format of data in storage aligns directly to the usage patterns and data structures utilised in the CVM. The storage system is also used to facilitate serialisation and transport of data across the network in communication between peers and clients. #### Cells Storage is constructed out of Cells. In most cases, a Cell is an entity that represents a Value in the CVM Informational Model. Normally there is a 1-1 mapping between Cells and CVM Values, however there are some exceptions: - For larger data structures a tree of Cells may be necessary - this is because we need to place a fixed upper bound on the size of each cell. - Small data values do not require a whole Cell, since it is more efficient to embed them directly within a larger Cell. - Some special data structures used in Convex are technically implemented as Cells for the purpose of storage and serialisation but are unavailable for use within the CVM since they are used externally to the CVM State - for example the `Belief` data structure used in the CPoS consensus algorithm. #### Encoding All Cells have a unique Encoding. The Encoding is designed to provide the following properties: * A bounded maximum encoding size for any Cell (currently 8191 bytes) * Very fast serialisation and deserialisation, with minimal requirements for temporary object allocation. * Uniqueness of encoding - there is a 1:1 mapping between Cell values and valid encodings. This means, among other useful properties, that Value equality can be determined by comparing hashes of encodings. * Self describing format - given a valid Cell Encoding, the Data Value can be reconstructed without additional context The same encoding is utilised in both durable storage and in network transmission. #### Value IDs as storage keys The cryptographic hash of the Cell encoding is used an an identifier (the "Value ID" or "VID") to refer to a Cell, and as a key for addressing data in the storage system. This has the important property that it requires all values in the storage system *immutable* - the data value cannot change for a given key, or else the VID will no longer be valid. This restriction may seem limiting at first, but in fact provides significant advantages for the Convex storage implementation: * No need to re-size values once written: the database can be accumulated in an "append-only" manner. The prevents storage fragmentation. * No need for cache invalidation or synchronisation of replicas: values cannot change * Rapid verification: if a hash exists in the store, and the data has already been validated, it must still be valid. #### Embedding Small Data Values can usually be Embedded within the Encoding of another Cell (typically a Cell representing part of a larger data structure). In most cases, this avoids the need to construct and store separate cells for small primitive values, and often small data structures themselves can be fully embedded. For example the vector `[1 2]` is encoded as a 6 byte sequence (`0x800209010902`) which can be seen to embed the values `1` (`0901`) and 2 (`0902`). Currently, Cells with an Encoding size of up to **140 bytes** are automatically embedded. This heuristic may be modified based on further testing and profiling, but it seems reasonable: per-Cell storage overheads make it inefficient to separately store such small objects, and by compressing many small objects into a single Cell we avoid the need to compute separate SHA3-256 VIDs for each, which we have observed to be a bottleneck in some cases. #### Convergence Given the above design features, we can implement a system of immutable storage that is Convergent: Additional storage information may be merged into the store in a manner analogous to a CRDT. It is a well-known result that taking the union of sets in a purely additive manner (a Grow-only Set) is a valid CRDT. The storage system can be regarded as a growing set of immutable (key, value) pairs, and hence satisfies the CRDT property. This convergence property is particularly beneficial when combined with the structured of Merkle trees used throughout the CVM: data structures with identical branches are automatically de-duplicated when they are stored, since the existing storage entry can simply be re-used. If effect, the Merkle trees become Merkle DAGs with guaranteed sharing of identical children. #### Monotonic Headers In addition to Value IDs and Encodings, the storage system allows header information to be attached to each Cell. As we require the storage system to be convergent, we require each field of the header to be *monotonic*, i.e., there is a simple function that can compute the new header as the maximum value of any previous header values. This ensure that the headers themselves satisfy the convergence property. The current Convex implementation utilises Monotonic Headers for the following fields: - Lazily computed memory size (any value is considered to replace an empty value) - Status tagging (see below) - Marking Cells to be pinned for purposes of durable persistence or garbage collection Unlike the Encoding, Monotonic Headers associated with each Value ID are essentially *transient* in nature, i.e., they can be reset or discarded without affecting the Cells themselves. This allows the Monotonic headers to be used locally by Peers independently of the general functioning of Convex as a decentralised network. For example, rebuilding the database during garbage collection may safely unmark pinned Cells providing the Peer has ensured that it has retained all the information it needs. #### Status tagging In order to support efficient Peer operation, the storage system implements a system of status tagging, used to indicate the level to which a data value has been verified by the Peer. Status tagging is monotonic in nature (increases in value) and hence can be included in part of the Monotonic Header The basic status levels are: * **UNKNOWN** - The Peer has an identifier (Hash), but does not know yet if this is consistent with any encoded data * **STORED** - The Peer has encoded data in storage which is validated to be a well-formed Cell (ignoring children), and consistent with the Hash. * **PERSISTED** - The Peer has validated the structure of the Cell completely, including recursively validating all its children. At this point, we can rely on a Data Value represented by the Cell to be usable in CVM execution * **ANNOUNCED** - The Peer has included the data in a publicly broadcast Belief Some special status levels are also possible, including: * **EMBEDDED** - A Cell is able to be embedded within other Cells and does *not* need to be individually stored. * **INVALID** - A Cell has been proven to be inconsistent with some validation rules. Such values cannot be used in the CVM, but caching the invalid status can be helpful to avoid the need to repeat the validation. This status tagging is monotonic and compatible with being included in the storage CRDT, since: * The status level can never go backwards: once verified, the result is known to be true forever. If the status was reset (e.g., in the case of storage failure), the only real loss would be the Peer having to repeat certain calculations to re-verify the status. * Where there are two possible outcomes (valid or invalid, embedded or non-embedded) all Peers that perform correct validation must agree (i.e., it is effectively monotonic for any given data value) #### Novelty detection A key feature of the storage system is the ability to detect and apply special handling to Novelty. Novelty is defined as a stored value that is moving up to a higher status level for the first time. Novelty detection is important for the following reasons: * When information needs to be shared on the network, only the incremental information needs to be transmitted. This is especially important for the consensus algorithm, for example: the transmission of a new Belief need only include the additions to the proposed Ordering, without communicating the complete Ordering (which may be very long, but is already likely to held by all other Peers) * When validating data, it avoids the need to re-compute validation on parts of the data that have already been validated. This is particularly important when the data structures to be validated are large, but have only a few small changes in comparison with a previously validated data structure (e.g., the entire CVM State) Most importantly, when a Belief data structure is produced an determined to be Novelty, Peers utilise this fact to trigger the propagation of the Belief to other Peers - however they only need to transmit the small subset of Cells in the Belief that are new, since most of the Belief data structure will not be novel and a Peer can safely assume that other Peers will already have access to such data in memory or storage. #### Garbage Collection Given infinite cheap storage, we could just keep accumulating values in the database forever. However, practical storage limits or costs will make this infeasible or undesirable for many Peers operators. Peers are only strictly required to maintain: * Enough information regarding Beliefs to participate in the consensus algorithm (about one day of orderings and transactions - exact limit TBC) * The current Consensus State for the CVM The storage system therefore allows garbage collection to be performed on a periodic basis, so that storage space containing data that is no longer required can be reclaimed. Garbage collection is done on a mark+copy basis, where currently used storage is copied to a new data file, and after which the old data file can be safely discarded. This could theoretically be performed concurrently with ongoing Peer operation in a future version. This behaviour is of course configurable by Peer Operators - we expect some will want to maintain and index all historical data for analytical purposes, or in order to provide their clients with additional historical query capabilities. #### Memory Mapped Implementation The Convex reference implementation implements the storage system using a specialised memory-mapped database called Etch, which is heavily optimised for the storage of Cells as described here. Assuming sufficient available RAM on a Peer, Etch effectively operates as an in-memory database. In performance tests, we have observed millions of reads and writes per second. This compares favourably to traditional approaches, such as using a relational database or a more generalised key-value store. ### Memory Accounting In order to address the problem of economic and storage costs of state growth, Convex performs continuous memory accounting calculations to ensure that participants pay appropriate costs for resources that they consume. #### Motivation A significant but often overlooked problem facing a global, decentralised database that provides a commitment to preserve data indefinitely is the problem of state growth: if not constrained, the size of the CVM state might grow excessively large. This is an economic problem: The participants who create additional state are not necessarily the same as those who must bear the cost of ongoing storage. This can create a "Tragedy of the Commons" where participants are careless about creating new state. This could quickly lead to a situation where the state grows too large to be feasible for normal computers to participate as Peers in the Convex networks, which will in turn cause centralisation towards a few large and powerful nodes. This problem cannot be solved by charging at execution time alone. There is no way to determine at execution time how long a particular set of data will be stored for - it might be deallocated in the very next transaction, or it might need to persist forever. Any "average" estimate will end up penalising those who are being efficient only need the storage briefly, and subsidising those who are careless and consume space forever. #### Overall Design To solve the state growth problem, Convex implements **Memory Accounting** with the following features: * Every change to the state tracks the impact on **State Size**, measured in bytes, which is (to a close approximation) the amount of memory that would be required to write out the byte encoding of the entire state tree. * Each account has an allocation of **Memory Allowance** to utilise. * When a transaction is executed, the **change in State Size** is computed. An increase in state size reduces the accounts free memory, while a decrease in state size increases the account's free memory. * If at the end of a transaction the incremental space exceeds free memory then the transaction will fail and be rolled back. * Accounts may *temporarily* exceed their memory allocation during the course of transaction execution - perhaps by constructing temporary data structures. We can safely allow this because the maximum amount of temporary object allocation is bounded to a constant size by juice limits. **Note 1**: that in practice, the actual storage size of the CVM state will be significantly smaller than the tracked state size, because the use of immutable persistent data structures allows many equal tree branches to be shared. The effectiveness of this structural sharing needs to be observed over time, but we anticipate perhaps a 2-3x reduction in state size may be possible in the long term. **Note 2**: Observant system hackers may notice that the memory accounting mechanism means that if Account A causes some memory to be allocated, and Account B causes that same memory to be de-allocated (e.g., through the use of calls to an Actor), then Account B will gain memory from A. We consider this a feature, not a bug: It incentivises participants to clean up state wherever possible and encourages the writers of actor code to consider their memory allocations and deallocations carefully. To ensure correct economic behaviour, it is necessary for free memory to have an economic cost. Therefore, Convex provides a **Memory Exchange** though which memory allocations may be traded. This has the following features: * An automatic market maker enabling accounts to buy and sell memory at any time, placing an effective price on memory * A total cap on available memory set at a level that constrains the total state size to an acceptable level * Ongoing governance to regulate changes in the total cap, which can be adjusted to allow for additional state growth as technology improves average Peer resources, without risking a loss of decentralisation. For convenience, memory purchases happen automatically if additional allocations are needed within a transaction. This means that in most cases, users need not be concerned with the specifics of managing their memory allowance. The overall cryptoeconomic design of Memory Accounting and the Memory Exchange offers a number of important benefits to the Convex ecosystem: * A guaranteed cap on state growth, that can safeguard against the growth of storage requirements driving centralisation * A general incentive for all participants to minimise and manage memory usage. This incentive increases as total state size grows towards the cap. * A specific incentive for coders to write memory-efficient code and provide the ability for unused data to be deleted, if they want their Actors to be considered high quality and trustworthy. * A partial disincentive to hoard memory allocations (since expected future cap additions may devalue large memory holdings). * When space becomes scarce, there is an incentive for less economically viable applications to wind up operations and sell their freed memory allocation. #### Memory Size Each Cell (in memory or storage) is defined to have a "Memory Size" which approximates the actual storage requirement (in bytes) for the object The Memory Size includes: - The size of the Encoding of the Cell in bytes - The total Memory Size of referenced child Cells, (e.g., if the object is a data structure) - An allowance for indexing and storage overheads (currently set to a fixed estimate of 64 bytes) #### Lazy computation Memory Size for a Cell is only calculated when required (usually at the point that the State resulting from a transaction is persisted to storage). This minimises the computational costs associated with memory accounting for transient in-memory objects. #### Memory Allowance Each Account on the Convex network is given a Memory Allowance which is a quantity of memory that may be consumed by that Account before incurring additional costs. #### Consumption Whenever a Transaction is executed on the CVM, Memory Consumption is calculated based on the total impact of the Transaction on the size of CVM state (the State Size). Memory Consumption is computed at the end of each transaction, and is defined as: `Memory Consumption = [CVM State Size at end of Transaction] - [CVM State Size at start of Transaction]` If a transaction has zero memory consumption, it will complete normally with no effect from the memory accounting subsystem If a transaction would complete normally, but has a positive memory consumption, the following resolutions are attempted, in this order: 1. If the user has sufficient allowance, the additional memory requirement will be deducted from the allowance, and the transaction will complete normally 2. If the transaction execution context has remaining juice, and attempt will be made to automatically purchase sufficient memory from the Memory Exchange. The maximum amount paid will be the current juice price multiplied by the remaining juice for the transaction. If this succeeds, the transaction will complete successfully with the additional memory purchase included in the total juice cost. 3. The transaction will fail with a MEMORY Error, and any state changes will be rolled back. The User will still be charged the juice cost of the transaction If a transaction has negative memory consumption, the memory allowance of the user will be increased by the absolute size of this value. In effect, this is a refund granted for releasing storage requirements. #### Allowance transfers It is permissible to make an allowance transfer directly between accounts. This is a practical decision for the following reasons: - It enables actors to automate management of allowances more effectively - It enables accounts controlled by the same user to shift allowances appropriately - It avoids any need for resource-consuming "tricks" such as allocating Memory from one account, and deallocating it from another to make an allowance transfer - It creates a potential for memory allowances to be handled as an asset by smart contracts #### Actor Considerations All Accounts, including actors, have a memory allowance. However, in most cases actors have no need for a memory allowance: any memory consumed during interaction with an actor will be accounted for via the user account account that originated the transaction. One exception to this is with scheduled execution, where an actor itself may be the origin for a transaction. Actor developers may include a capability to reclaim memory allowances from an Actor (e.g. transferring it to a nominated user account). This is optional, but without this there may be no way to ever utilise an allowance held within an actor (either because a scheduled transaction obtained a memory refund, or because an allowance transfer was made to the actor). #### Memory Exchange trading The Memory Exchange is a simple Automated Market Maker (AMM), allowing users to buy and sell memory allowances at any time from a Memory Pool. The price of memory in the pool will automatically adjust to find an equilibrium between supply and demand. #### New Memory Release It is expected that advantages in storage technology over time will allow memory constraints to be gradually relaxed. Furthermore, it would be unwise to have memory be priced too cheaply at the beginning but continuously increase as State size grows and more memory is bought from the Pool. Methods are therefore implemented to allow the gradual release of new memory into the Pool. Knowledge that additional memory will be released (and subsequently reduce memory prices) is a useful incentive against deliberate hoarding of memory allowances. The current Convex design anticipates two such mechanisms: - A protocol based, automatic addition of new memory into the Pool (based on Consensus timestamps). - Network Governance roles specially authorised to create and release new Memory into the Pool. The first of these two methods is strongly preferred in order to minimise potential centralisation, complexity and risk innate to allowing any privileged governance controls over the Network. However it may be necessary given the high probability of technological shocks which cannot be predicted in the protocol. #### Size persistence The memory size is persisted in the Storage System as part of the header information for a Cell. Persisting this value is important to ensure that memory sizes can be computed incrementally without re-visiting the complete tree of child Cells. #### Memory Accounting impact The memory accounting subsystem is designed so that it always has a minimal effect on CVM state size, even though it causes changes in the CVM state (consumption of allowances etc.). This limits any risk of state growth size from the memory accounting itself. This is achieved mainly by ensuring that state changes due to Memory Accounting cause no net Cell allocations: at most small embedded fields within existing cells are updated (specifically balances and allowances stored within Accounts). #### Performance characteristics Memory Accounting is `O(1)` for each non-embedded cell allocated, with a relatively small constant. This would appear to be asymptotically optimal for any system that performs exact memory accounting at a fine-grained level. This achievement is possible because: - The memory size is computed incrementally and cached for each cell. - The number of child cells for each cell is itself bounded by a small constant - Memory Size computation is usually lazy, that is it is not performed unless required - The immutable nature of Convex cell values means that there is never a need to update memory sizes once cached #### Accounting for computational costs The direct computational cost of performing this memory accounting is factored in to the juice cost of operations that perform new cell allocations. This compensates Peer operators for the (relatively small) overhead of performing memory accounting. The storage cost is, of course, handled by the general economics of the Memory Accounting model and pool trading. ### Cryptographic Primitives Convex uses cryptographic primitives for the following functions: * Digital Signature (Ed25519) * For every Transaction submitted by a client * For every Block proposed by a peer for consensus * For every Ordering constructed and shared by a peer * For every Belief shared by a Peer on the gossip network * Cryptographic Hashes (SHA3-256) * For every Cell which forms part of a Decentralised Data Value, a hash of its byte encoding is computed for storage, identity, indexing and verification purposes. This is effectively equal to the VID. * For every key used in a hash map data structure, its hash is computed (if necessary) * Standard approaches used to store and protect keys in common key file formats (e.g. .pem, .pfx) As an engineering principle, Convex only uses trusted implementations of cryptographic algorithms in well tested libraries (currently Bouncy Castle, and the cryptographic routines available as standard in the JVM). There is no need to "roll our own" with respect to fundamental crypto algorithms. ## Conclusion Convex is a unique approach to programmable economic systems that provides a powerful combination of scalability, security and decentralisation - suitable for building applications for the Internet of Value. The high degree of simplicity has enabled the development of a smart contract facility with more stable and secure protocols than other decentralised ledger technology. Convex features: * Functional programming on the CVM based on the lambda calculus * Immutable values for all data structures * A Belief Merge Function that protects against malicious or faulty Peers * A provable, efficient, convergent consensus algorithm (CPoS) based on CRDTs We believe that the innovations in Convex, combined with meticulous engineering decisions, have implemented a practical, high performance platform for supporting a new generation of decentralised applications and economic value creation systems. ## Contact and Links To learn more and experiment with the live Convex test network: [Convex World](https://convex.world) For discussion of this White Paper and other topics relating to Convex, you are very welcome to join our public [Discord Server](https://discord.gg/5j2mPsk). Email: info@convex.world --- # What Makes Convex Different Every platform claims to be fast and scalable. This page lists the specific design decisions where Convex departs from the mainstream — and links the evidence for each, so you can judge for yourself. ## It is not a blockchain Convex reaches consensus without a chain of blocks. Peers share and merge *Belief* data structures — a [CRDT](https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type) — and the network converges on a single transaction ordering because lattice merges mathematically must converge. There is no leader election, no mining, no proof-of-work waste. This is not a detail of implementation; it is where most of the other differences come from. See [Lattice Technology](/docs/overview/lattice) for how it works and the [White Paper](/docs/overview/convex-whitepaper) for the full treatment. ## Zero block delay On most platforms, your transaction waits for someone's turn to make a block. On Convex, any peer publishes a block the instant it has a transaction — simultaneous blocks from different peers are simply merged and ordered by consensus. Combined with leaderless design, this gives sub-second finality and **built-in front-running resistance**: there is no privileged block producer to reorder your trade. The numbers and the mechanism are on the [Performance](/docs/overview/performance) page. ## One global state No shards, no bridges, no roll-ups, no cross-chain messaging to design around. Convex maintains a single consistent global state, and transactions against it are **atomic and Turing complete** — arbitrary logic with automatic rollback if anything fails. The engineering that makes one state scale (structural sharing, delta transmission, embedded encodings) is described in [Lattice Technology](/docs/overview/lattice). ## Memory is a tradeable resource On-chain storage is usually an unpriced externality: users pay once to write, peer operators store forever. Convex has [memory accounting](/docs/cad/memory): every account holds a memory allowance, allocation consumes it, and **deleting data refunds it**. A global memory pool caps total state size and lets accounts buy and sell allowance for Convex Coins. Storage stays priced, bounded, and worth cleaning up. Background and rationale: [Memory is money](/blog/memory-is-money). ## The compiler lives on-chain No external toolchain is required: submit **source**, and the CVM expands and compiles it in consensus, juice-metered like any computation. That's why development on Convex is REPL-first (type an expression, it runs on the network), why `deploy` is a one-liner, and why there is no "verified source" problem for code submitted this way: the chain compiled it, so the chain knows. Clients that can precompile are free to submit compiled CVM ops directly instead — that saves the compilation juice — but the compiler is always there in the protocol. Details: [CAD008 (Compiler)](/docs/cad/compiler) and [The compiler lives on-chain](/blog/on-chain-compiler). ## Your address is not your key A Convex address (`#1234`) is a permanent identity; the Ed25519 key attached to it is replaceable data. Rotate keys in one transaction without moving assets or updating anyone. Actors are simply accounts with no key, and accounts can designate controllers — enabling time-locked recovery and programmable custody, none of it requiring protocol changes. Why this matters: [Your address is not your key](/blog/your-address-is-not-your-key). ## Networks that upgrade without forking Protocol upgrades are scheduled **on-chain** and activate at a consensus timestamp: state migration applied in lock-step by every peer, protocol version incremented, genesis hash untouched. Peers that can't upgrade shed stake and step out cleanly rather than forking the network. Convex can keep improving for decades without ever becoming a different network. How it works: [Networks that evolve](/blog/network-upgrades). ## The lattice is bigger than the chain Consensus is one *region* of the Lattice. The same convergent-merge technology also runs a [replicated file system](/docs/cad/dlfs), a [SQL database](/docs/cad/convex_sql) with a PostgreSQL wire protocol, [Kafka-style streaming queues](/docs/cad/lattice_queue), and delegated, capability-based [authorisation](/docs/cad/lattice_auth) — plus [Covia](https://covia.ai), the federated AI orchestration grid built on Convex. One data model, on-chain and off. The full map is in [Lattice Technology](/docs/overview/lattice). --- Some answers to common questions can be found here. ## Is Convex Free? Yes! Convex is free for anyone to use, and always will be. We are building Convex as an open public utility network for everyone to support the Internet of Value. When transacting on the network, small fees are charged using Convex Coins, which is the native utility token of the network. This is necessary for several reasons: - Compensate fairly those who provide important secure infrastructure to the network (i.e. peer operators) - Prevent denial of service attacks by people flooding the network with wasteful transactions. This makes it very expensive to launch such attacks. - Create an economic incentive to use the network as efficiently as possible (both for users and developers of smart contracts) Our goal is to keep transaction fees small, so that it is never a significant issue for legitimate network users. ## Is Convex live? Yes. Protonet, the first live Convex network with real assets, is up and running — you can connect at `https://peer.convex.live`. Getting it right was always more important than rushing a release: Protonet launched only after extensive testing to ensure it was ready for production use with real-world value at stake. People will depend on Convex to be a secure, reliable platform for decentralised applications and digital assets. It is not acceptable to expose them to security risks from flaws in the platform, nor is it acceptable to make breaking changes to the CVM that could cause significant problems with smart contracts. ## How do I get Convex coins? Convex coins are a cryptoasset that can be obtained in multiple ways. - Anyone can buy Convex coins directly from individuals on a self-sovereign basis - You will be free to purchase Convex coins via exchanges and other independent service providers - [Paisley](https://www.paisley.io/) allows its members to purchase Convex Coins with fiat / crypto - Contributors to Convex or the broader Convex ecosystem can earn awards of Convex coins - Approved purchasers can buy newly issued coins from the Release Curve. Anyone interested in Convex coins should familiarise themselves with the relevant [tokenomics](/docs/cad/tokenomics) We have established the Convex Foundation a non-profit organisation that facilitates the initial sale of Convex coins, and distributes coin awards to contributors. Funds raised will be reinvested in building Convex and the ecosystem. ## How fast is Convex? Convex can comfortably process many thousands of complex transactions per second (e.g. transfers and smart contract calls). The CVM itself has been benchmarked at over 1,000,000 TPS on a modern desktop PC. And as we continue making performance improvements it is getting faster by the day. But it's important to note that performance and scalability are not just about the raw throughput of transactions. Convex is designed to offer a good overall combination of: - Low latency to stable consensus (below 0.5s seconds on a global network) - High execution throughput (e.g. 50,000+ token-based smart contract transactions per second) - Advanced features (smart contracts, memory accounting, an on-chain compiler, autonomous Actors) - A unified global state machine supporting atomic transactions - True decentralisation as a public utility network (with Convergent Proof of Stake) - Ability to operate a Peer with affordable hardware We achieve all this *without* resorting to over-complicated scaling solutions that introduce various new problems (e.g. cross-shard transactions). We can always add additional scaling features later, but it may not even be necessary. ## How do I integrate my application with Convex? Convex provides several mechanisms for integration. For most decentralised apps, it is easy to build a client-side application that makes use of the **HTTP Client API**. This is a convenient REST API using JSON that is ideal for web and mobile applications developers. Advanced applications can use the **Binary API** and construct messages directly to communicate with Peers. This is currently only possible for JVM-based languages (Java, Clojure, Scala etc.), but more may be supported in the future. Integrators can also use **Direct Peer Integration** where they run a fully operational Convex Peer alongside their server-side applications. This approach is complex and recommended only if you want maximum performance and/or want to interact with the CVM state directly (e.g. for search or indexing purposes). The `convex.world` sandbox itself uses this technique. ## What should go on-chain? You often have a choice between putting code and information on the public Convex network or keeping it on a separate server when building a decentralised application. Some applications might do both: we call these "hybrid" dApps. Some general principles: - Put data *on-chain* when it needs to be *publicly visible and verifiable* - Put data *on-chain* when you need to make trusted transactions between parties (e.g. exchanging digital assets) - Keep data processing (e.g. string formatting, sorting) *off-chain* - this is best done on the client - Keep data *off-chain* when it needs to be private - Keep code and data *off-chain* if there are significant compute or storage requirements (it would be too expensive to put on-chain) It is a good idea to **keep data structures as simple as possible**. Ideally, Actor code should be performing a small set of O(1) operations on the right kind of data structures (looking up / updating a value in a map, appending a value to a vector etc.). We recommend designing the data structures for your Actors carefully first, before writing the code to interact with them. ## Is Convex a Blockchain? Think of Convex and the underlying lattice technology as "beyond blockchain". Convex shares many common attributes with traditional public blockchains: - A decentralised consensus network - Security from malicious actors with cryptographic techniques - Decentralised ownership of accounts, including the ability to control digital assets and currencies - Ability to deploy and execute secure smart contract code - Transactions are grouped into blocks Technically however it's not implemented as a blockchain (in the sense that there is a linked list of blocks where each block contains the hash of a previous block). The Convex consensus algorithm creates an *ordering* of blocks, but the cryptographic hashes used to secure this ordering are kept outside the blocks themselves. This gives us a big advantage, as blocks can be submitted and processed by peers concurrently without having to first determine the hash of preceding block(s). ## How does Convex perform so well? It's complex! But here are some of the most important points: - [Lattice technology](lattice.md) is uniquely efficient for handling decentralised data at scale. We've been building the foundations of lattice technology for 5+ years with a heavy focus on making it supremely scalable. - The consensus algorithm (Convergent Proof of Stake, or CPoS) is magic. It can confirm blocks in milliseconds between peers running on a local network. The main latency delay in the global network is just signal transmission over the Internet: the speed of light is a tricky problem. - The CVM execution model is designed for performance: CVM operations perform high level state transformations, but are implemented using very efficient low-level code. - We wrote a custom database (Etch) from scratch to support the performance needs of Convex. Having a database perfectly designed and tuned for lattice technology is a huge advantage and much faster than more generic alternatives (e.g. LevelDB) - We exploit a lot of advanced features of the JVM, which is a very powerful platform backed by thousands of man-years of engineering effort. We benefit a lot from the JIT compiler, concurrency, asynchronous IO and advanced memory management features. - Our team includes people who have been performance-oriented hackers for many years, with experience in algorithms, data structures, game coding, embedded systems, distributed computing and more. We enjoy and take pride in writing fast, efficient code! ## Why does Convex use Lisp? A variant of Lisp was chosen as the initial language for the CVM for a few reasons: - Lisp expressions are essentially a direct encoding of the [Lambda Calculus](https://en.wikipedia.org/wiki/Lambda_calculus). This means that we are based on fundamentally sound computation theory. - Lisp macros are a powerful tool for generating code, which is an ideal solution for building sophisticated smart contract capabilities with an on-chain compiler. - Lisp is a highly expressive language for interactive development, with a long history of REPL-based usage. We feel this is ideal for a platform where we want developers to be instantly productive and able to interact directly with the system in real-time. Paul Graham's essay [Beating the Averages](http://www.paulgraham.com/avg.html) is an interesting perspective on the advantages of Lisp for building a business. Despite dating from 2001, we feel many of these points still stand today and are very relevant for people wanting to build applications using Convex. For more information see the tutorial section for [Convex Lisp](/docs/tutorial/convex-lisp) ## What is Memory Accounting? Memory Accounting is the system in Convex used to track the usage of on-chain memory. Every time a user executes a transaction, the amount of memory used is calculated and deducted from the user's memory allowance. If the user has an insufficient memory allowance, it is possible to automatically buy more on-demand. If a user executes a transaction that releases memory, the amount of released memory is credited back to the user's allowance. This creates a good incentive to "clean up after yourself". Actors and smart contracts should also be designed with the option to clean up memory after it is no longer required. We need Memory Accounting because on-chain memory is a **scarce resource**, and should be used wisely. An effective way of doing this is to make memory allowances themselves into a digital asset, that can be transferred and traded. This creates a market incentive to utilise memory as efficiently as possible. Memory account is described in much more detail in [CAD006](/docs/cad/memory) ## What is the difference between actors and smart contracts? Actors are virtual agents that exist inside the Convex Virtual Machine. They are autonomous agents that can execute CVM code, manage digital assets, perform complex computation, make decisions. They follow strict rules that control their execution, so that they can be audited and relied upon to behave in a particular way. Smart contracts are a concept: the idea of agreements that can be automatically executed and enforced by software, eliminating risk and the need to trust fallible humans. Convex actors are therefore used to *implement* smart contracts. An example would be a digital asset store that allows parties to offer assets for sale, but only delivers them to the buyer once payment has been made. Each asset sale is a smart contract between the seller and the buyer, facilitated by the actor. Not every actor needs to be a smart contract, however: an actor that simply manages on-chain information on behalf of individuals isn't facilitating any contracts between parties. ## Who is building Convex? We are a small team of dedicated hackers and creators passionate about building an amazing platform for the future digital economy. We mostly hang out on the [Convex Discord](https://discord.com/invite/xfYGq4CT7v) --- # Governance The Convex Network is governed by the **Convex Foundation**, a dedicated non-profit entity committed to fostering a secure, decentralised, and thriving ecosystem for global peer-to-peer value exchange. By prioritising community-driven principles, the Foundation ensures Convex remains innovative, accessible, and open to developers and users worldwide. The Foundation drives key initiatives on behalf of the ecosystem: - **Ensures robust governance** of the Convex Network, safeguarding its integrity and long-term sustainability. - **Operates key community resources** including the flagship [convex.world](https://convex.world) website and developer resources such as [GitHub repositories](https://github.com/Convex-Dev) and [Convex Docs](https://docs.convex.world) - **Manages the initial issuance of Convex Coins (CVM)** to individuals, organisations, and ecosystem participants, overseeing a transparent release curve designed to promote fair distribution and network stability. - **Rewards ecosystem contributors** through grants and awards, recognising open-source developers, builders, and innovators who advance the platform. - **Accelerates ecosystem development** via strategic partnerships, targeted marketing, educational programs, and community initiatives that inspire collaboration and real-world impact. ## Getting Involved The Foundation is built to be open: every stakeholder has a voice. - **Join the Community** - Connect with us on [Discord](https://discord.com/invite/xfYGq4CT7v) or social media - **Contribute** - Make open source or community contributions. Contributors frequently earn CVM awards in recognition of their efforts - **Purchase CVM** - buying CVM gets you the utility token that powers the Convex network, and supports our mission: 100% of revenue from initial CVM sales is re-invested in the ecosystem via the Foundation. ## Foundation Information The Convex Foundation is registered as a non-profit company limited by guarantee in the UK, founded on 23rd September 2020. Company Number: 12900166. --- # Convex Overview Convex is a decentralised lattice platform for building fair, efficient, and sustainable economic systems: a single programmable global state, secured by leaderless consensus, open to everyone — including autonomous AI agents. If you only have a minute, these are the claims that matter, each linked to its evidence: - **Transactions confirm in under a second, with zero block delay.** Peers publish a block the moment a transaction arrives; leaderless [CPoS consensus](/docs/overview/performance) orders everything with built-in front-running resistance. - **The CVM executes up to a million transactions per second** in [benchmarks](/docs/overview/performance), on commodity hardware — with figures validated in the EU's Next Generation Internet programme. - **The lattice is bigger than the chain.** The same technology runs a [replicated file system](/docs/cad/dlfs), a [SQL database](/docs/cad/convex_sql), [streaming queues](/docs/cad/lattice_queue) and [federated AI orchestration](https://covia.ai) — see [Lattice Technology](/docs/overview/lattice). - **AI agents are first-class economic participants.** Every peer is an [MCP server](/docs/products/convex-mcp); agents hold accounts, own assets and transact under exactly the same rules as everyone else. - **The security record is public.** Nobody has ever taken a Convex Coin on testnet without the owner's private key, through years of open testing. ## Understanding Convex - **[What Makes Convex Different](/docs/overview/different)** - the design decisions that set Convex apart, and the evidence for each - **[Lattice Technology](/docs/overview/lattice)** - the foundation: convergent data structures, regions, and why it scales - **[Key Concepts](/docs/overview/concepts)** - accounts, transactions, actors, juice and memory - **[Use Cases](/docs/overview/use-cases)** - what you can build, from DeFi to AI agent economies ## The evidence - **[Performance](/docs/overview/performance)** - latency and throughput in detail, with benchmarks - **[FAQ](/docs/overview/faq)** - direct answers to common questions ## Vision and stewardship - **[Convex White Paper](/docs/overview/convex-whitepaper)** - the complete technical case - **[Manifesto](/docs/overview/manifesto)** - why we are building this - **[Governance](/docs/overview/governance)** - the Convex Foundation and how to get involved --- Ready to build? Head to the **[User Guide](/docs/tutorial)**, or explore the **[Products](/docs/products)** — SDKs, CLI, Desktop, MCP and more. --- # Lattice Technology Lattice Technology is the foundation of Convex — a robust, self-sustaining, and democratic infrastructure for sharing data and computation globally. Imagine a boundless, self-healing cloud of decentralised data and computing power: accessible to all, secured by strong cryptography, converging on consensus without central coordination. Unlike traditional systems, no single entity controls it. This is the Lattice: a global network where trust, scalability, and resilience are built into the foundation. ## How the Lattice works ### Algebraic foundation At its core, the Lattice draws inspiration from the mathematical concept of a [lattice](https://en.wikipedia.org/wiki/Lattice_(order)) - a partially ordered set equipped with a *merge* function. This function combines any two lattice values (elements of the set) into a single, consistent result. Through repeated merges, the system naturally converges to a unified value, known as the *supremum*, without relying on complex locking mechanisms or heavy consensus protocols. This design makes the Lattice a [Conflict-free Replicated Data Type (CRDT)](https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type). It guarantees *eventual consistency* across the network, requiring only that nodes intermittently share lattice values. No continuous connectivity or specialised protocols are needed—simple, occasional "gossip" between nodes suffices to keep the system in sync. ### Lattice innovations Lattice Technology builds on the CRDT framework with several groundbreaking enhancements: - **Cryptographic Security**: Digital signatures and cryptographic hashes ensure secure, tamper-proof operation in a decentralised environment, fostering trust without centralised oversight. - **Transaction Ordering Consensus**: The Lattice establishes a reliable sequence of transactions, solving critical issues like the double-spend problem in a decentralised setting. - **Immutable Persistent Data Structures**: Lattice values leverage powerful, git-like data structures of arbitrary size. Only changes (deltas) are transmitted and processed, enabled by the efficient [CAD3 encoding format](/docs/cad/encoding), making updates lightweight and scalable. - **Merkle Tree Integration**: Lattice data structures double as Merkle trees, providing strong integrity guarantees and enabling rapid verification of data authenticity. - **Conditional Acceptance Rules**: The system enforces strict validation of incoming lattice values, thwarting malicious actors. Merging invalid values is futile because other nodes will reject them outright, incentivising honest participation and preserving network integrity. ### Merge Context A key innovation in Lattice Technology is the *merge context*, which refines how lattice values are combined. The merge process isn’t just a blind fusion of two values—it’s a function of three inputs: ``` new lattice value = merge (context, existing lattice value, received lattice value) ``` - `context`: Additional context-specific data for the merge (e.g., timestamps, keys for signature verification). - `existing lattice value`: The locally validated value, already trusted by the node. - `received lattice value`: An incoming value from the network, subject to scrutiny. This approach ensures merges are intelligent and secure. For instance, a merge might only accept portions of a received lattice value that pass cryptographic validation, rejecting anything unverified or malicious. By prioritising the trusted existing value and applying contextual rules, the Lattice thrives in a decentralised world where not every participant can be blindly trusted. ## Regions of the Lattice The Lattice isn’t a monolith—it’s a flexible framework divided into regions, each defined by the specific lattice values it uses and the rules governing how those values merge. Think of each region as a specialised sub-lattice, tailored to a unique purpose, yet fully integrated into the broader Lattice ecosystem. Regions operate as lattices themselves, leveraging a key property: a map of keys to lattice values forms a lattice. The merge function is straightforward—combine entries from two maps and resolve key collisions by merging the associated lattice values. Participants enforce region-specific rules on a decentralised basis. If someone violates these rules by sharing invalid values, their efforts are futile—others simply ignore the rogue values, preserving the Lattice’s integrity. The Lattice launches with several foundational regions, each powering critical decentralised functions. Here’s a closer look: ### Convex Consensus Lattice The *Convex Consensus Lattice* drives a secure, decentralised global state machine using the Convex Proof-of-Stake (CPoS) algorithm. Lattice values are *Beliefs*, which are shared by peers and merged using the belief merge function, as defined in the [Convex White Paper](convex-whitepaper.md). Convex performs many of the functions of a typical L1 blockchain, but without the drawbacks: - A transparent global state machine with Byzantine fault-tolerant consensus. - Self-sovereign accounts for managing digital assets, secured by digital signatures. - Turing-complete smart contracts and autonomous actors on the Convex Virtual Machine (CVM) - "unstoppable code" in action. - Trusted storage of data for usage by decentralised applications. This region blends blockchain-grade security with the Lattice’s lightweight, conflict-free design, making it a powerhouse for decentralised economic systems. ### Data Lattice The *Data Lattice* is a decentralised storage network for content-addressable data, owned and managed by its users. Lattice values are arbitrary sets of data (indexed by cryptographic hash) and the merge function simply takes the union of these sets. Nodes may discard values they are not interested in to save resources: if, and only if, all nodes decide to do this then the data is effectively deleted from the Lattice. Four essential functions are supported: - **Store**: Store arbitrary data in a local node, for future onwards replication. - **Read**: Access data from any reachable node using its hash. - **Acquire**: Fetch data from specific nodes or search the network (like BitTorrent), copying it locally. - **Pin**: Retain critical data on a node, ensuring availability across the Lattice. Think of it as a faster, more efficient evolution of IPFS or IPLD, built on the Lattice’s high-performance architecture. ### Data Lattice File System (DLFS) The *Data Lattice File System (DLFS)* extends the Data Lattice into a self-sovereign, replicated file system. Lattice values are file system trees ("drives") with files and directory similar to a traditional file system. The merge function updates drives based on recency and cryptographic authorisation. Because lattice values are an immutable persistent data structure, it is also possible to "snapshot" an entire DLFS drive with a single cryptographic hash. This snapshot could, for example, be pinned in the Data Lattice for audit / backup / analysis purposes. This operation is extremely efficient because of structural sharing: most of the actual storage will be shared with the current DLFS drive and/or other snapshots so this operation is extremely efficient (you are only really storing the deltas from other versions). ### Structured data: KV, SQL and Queues Because any lattice value with a sound merge function forms a region, familiar data infrastructure can be rebuilt lattice-native: - The **[KV Database](/docs/cad/kv_database)** provides shared mutable key-value maps with last-writer-wins merge — the "state" half of distributed data. - **[Convex SQL](/docs/cad/convex_sql)** provides relational tables with schema, primary keys and full SQL queries (via Apache Calcite), replicated by lattice merge. It even speaks the [PostgreSQL wire protocol](/blog/sql-on-lattice), so ordinary database clients connect directly. - The **[Lattice Queue](/docs/cad/lattice_queue)** provides Kafka-style streaming — topics, partitions, offsets, independent consumers — with [no broker anywhere](/blog/lattice-queue): replication is just merge. Together these cover the two fundamental shapes of distributed data — state and logs — without any centralised infrastructure. ### Execution Lattice The Execution Lattice specifies compute tasks to be performed on a decentralised basis. Lattice values are a map of job IDs to signed and timestamped job records. The merge function again combines these maps, with the most recent correctly signed job status preferred in event of collisions. Job records consist of: - A specification of the compute job to be performed - Metadata about the job (including authorisation for completing the job) - A map of inputs (provided by the requestor) - A map of outputs (filled in by the completer) Importantly, such job executions are highly extensible. They can utilise any form of compute task including computation in private enclaves, use of encrypted data or harnessing specialised compute infrastructure. Flexible authorisation makes it possible to specify tasks that must be completed by a specific party, or to make it open for anyone to complete the task (perhaps in exchange for some form of tokenised payment) ### P2P Lattice The *P2P Lattice* powers peer-to-peer communication by solving the challenge of locating and connecting participants in a decentralised network. Lattice values are a map of public keys to signed and timestamped metadata describing a peer. The merge function is simply to combine these maps, and to take the most recent correctly signed metadata if keys collide. The P2P Lattice operates in a manner similar to [Kademlia](https://en.wikipedia.org/wiki/Kademlia), allowing the location of arbitrary peers on the Internet without depending on any decentralised location service. In the Kademlia model, peers only need to store metadata for other peers that they are relatively "near" to in cryptographic space, making this a highly efficient and fault-tolerant decentralised service. ### An ecosystem, not a fixed list Regions compose, and new ones keep arriving. Delegated, capability-based access via [UCAN authorisation](/docs/cad/lattice_auth) and [decentralised identity](/docs/cad/did) secures cross-party sharing; every Convex peer exposes the lattice to AI agents via [MCP](/docs/products/convex-mcp); and [Covia](https://covia.ai) builds federated AI orchestration — models, agents and data collaborating across organisational boundaries — entirely on lattice technology. If you can define a value type and a sound merge function, you can add a region. ## Efficiency and scalability How do we build a global, decentralised data structure of unlimited scale? How do we make it fast? Or even feasible? There are a number of key engineering ideas here. We've been building and stress-testing Lattice technology for 5+ years which has given use some unique implementation advantages and insights: **Structural sharing** - Using immutable [persistent data structures](https://en.wikipedia.org/wiki/Persistent_data_structure) means that when changes to a large lattice value are made, a new lattice value is produced which shares most of its data with the original value. This means that storage and processing is only required. **Selective attention** - Nodes may select whichever subsets of the lattice they are interested in handling on a self-sovereign basis. This means that participants can scale their resource usage based on their own needs. For example, a Convex peer operator might elect only to participate in the Convex consensus lattice and a small subset of DLFS drives representing data that the peer operator needs to access and maintain. **Delta transmission** - Building upon structural sharing, it is possible to only transmit the deltas (changes) when a new lattice value is communicated. This assumes that the recipient will have the original data, but this is a good assumption if they are already participating in the same lattice (and if they don't they can simply acquire it again...). This means that network / communication requirements are only ever (at most) proportional to the number of changes made in regions of the Lattice that a specific node has chosen to participate in. **Merge coalescing** - A node may receive multiple lattice values from different sources in a short amount of time. With a series of repeated merges, it produces a new lattice value incorporating all of these updates. It then only needs to produce and transmit one new lattice value (typically with a much smaller delta than the sum of those received). This coalescing behaviour therefore automatically reduces traffic and scales the load to the amount that nodes can individually handle (typically, network transmission bandwidth will be the primary bottleneck since local lattice value merges are very fast). **Embedded encodings** - Merkle trees have the disadvantage that they require the computation of a cryptographic hash at every branch of the tree. This can become expensive with a large number of small branches, so the Lattice makes use of a novel efficient encoding scheme (outlined in [CAD003](/docs/cad/encoding)) that compresses multiple embedded values into a single Merkle tree branch (while still maintaining the important property of having a unique encoding with a content-addressable hash). Typical branches might be around 1000 bytes on average (and never less than 141 bytes), which ensures efficiency from a hashing perspective and also keeps overall storage requirements near-optimal. **Branching factor** - There is a trade-off with branching factors in Merkle trees. Too low, and your tree becomes excessively deep with a lot of extra intermediate hashes to store and compute. Too high, and the encoding of a single branch becomes large, meaning that small changes result in a lot of redundant copying. Lattice values are optimised to provide efficient branching ratios for different use cases (typically ~10). In all cases: The number of branches, encoding size and cost of navigating to a direct child branch are guaranteed to be `O(1)` by design. **Orthogonal persistence** - Lattice values can exist in memory on other storage media (typically local disks). From a developer perspective, these are effectively identical, there is no need to treat in-memory and externally stored values differently. However, values are efficiently loaded from storage and cached on demand, so that most of the time the lattice behaves like a very fast in-memory database despite being potentially much larger than local RAM. **Fast comparison** - Lattice values enable some extremely quick comparison algorithms which lattice technology fully exploits. Most simply, checking the identity of any two values is simply the comparison of two cryptographic hashes, which can be done in `O(1)` time. Perhaps surprisingly, computing the common prefix of two vectors of arbitrary length is also just `O(1)`, which is heavily exploited to compare transaction orderings efficiently in CPoS. More sophisticated comparisons include computing differences between multiple lattice data structures (typically `O(n)` or `O(n log n)` where `n` is the size of differences). It is thanks to these comparison algorithms that we are able to implement extremely fast lattice merge operations. **Garbage collection** - Lattice values work *extremely* well with a model of lazy garbage collection. Technically, you can keep lattice values as long as you like (they are immutable and content-addressable after all, so never go stale). However, sooner or later you are likely to hit storage constraints. In this case, you can simply retain the subset of the lattice(s) you are interested in as identified by current root value(s) and discard all other values. This works for both in-memory caches (e.g. leveraging the JVM GC) and long term storage (e.g. `convex etch gc`). --- # The Convex Manifesto Building Open, Decentralised Economies for the 21st Century For too long, centralised gatekeepers have dominated our economies - extracting rents, inflating costs, and excluding billions from meaningful participation. These outdated systems stifle innovation, widen inequality, and accelerate environmental collapse. Artificial intelligence is now reshaping every aspect of the digital world at unprecedented speed. In this defining era, we must reclaim sovereignty over data, compute, and value itself. Convex is the public, decentralised foundation for real-time, peer-to-peer exchange of data and value. Engineered for the AI age, it delivers the fair, inclusive, efficient, and sustainable economic infrastructure the world demands. This manifesto articulates the principles powering Convex and the open economy it enables. Join us in building it. ## Open Economic Systems **Convex facilitates open economic systems: interactions where participants are free to create mutual value for each other** Voluntary exchange is the ultimate win-win, leaving every participant better off without coercion or gatekeepers. This engine of human prosperity must be unleashed at scale, not stifled by monopolistic control. Value is boundless: currency, data, assets, services, access, reputation, or yet-unimagined forms. Convex is engineered as an extensible foundation, ready to represent *any* exchange—today and tomorrow - fuelling endless innovation in the AI-driven economy. ## Self Sovereignty **Participants are self-sovereign agents: free to transact on their own behalf** Participation in the digital economy is a human right. No one should be censored, excluded, or forced to pay tribute to gatekeepers. Convex returns control over data, assets, and choices to users. Delegation is always optional: users may appoint agents, wallets, or services to act on their behalf, but only by explicit, revocable consent - never compulsion. Self-sovereignty coexists with responsibility. Users remain bound by law, taxes, and the *Golden Rule*: treat others as you wish to be treated. Convex enforces no harm, only freedom within fair boundaries. ## Sustainability **Convex delivers efficient, sustainable, zero-waste infrastructure** Economic systems must fuel human progress without plundering the Earth. Today’s centralised giants devour 5–10% of global GDP in fees and friction, while offloading pollution, e-waste, and climate costs onto the innocent and unborn. This is theft from future generations. Convex rejects extravagance. We ban energy-hungry Proof of Work and embrace *Convergent Proof of Stake (CPoS)* - a consensus engine that outperforms PoW in speed and security using a fraction of the power. Every transaction is lean, auditable, and ecologically defensible. Decentralisation means we can’t police every app built atop Convex, but we architect the foundation for sustainability: minimal overhead, maximal reuse, and incentives that reward green behaviour. By slashing systemic waste, Convex paves the way for an economy that regenerates rather than extracts—proof that abundance and stewardship can coexist. ## Fair Access **Everyone deserves equal, fair and direct access to the mechanisms of value exchange, enabling universal participation in the digital economy** Billions remain locked out: unbanked, overcharged, or barred from markets by geography, identity, or greed. Traditional finance hoards opportunity behind walls of paperwork, KYC, and monopolistic rents. Convex tears down those walls. Anyone with an internet connection joins on equal footing: self-sovereign, first-class, and permissionless. No middlemen. No censorship. The network stays open, resilient, and globally reachable, turning exclusion into universal inclusion overnight. ## Real-time Transactions **Users must be provided with real-time, interactive, atomic transactions** Users demand seamless experiences: instant mobile payments, fluid gaming economies, immersive metaverses. Clunky blockchains with seconds/minutes delay kill adoption and usability. Physics sets limits, but Convex is limited only by the speed of light. **CPoS** is leaderless, slashing latency to milliseconds. Zero block delay. No mempool. No front-running. Atomicity is ironclad: every transaction executes fully or rolls back completely. Turing-complete logic with automatic failure protection eliminates the risk of partial failures. ## The Global State **Users are free to interact with the decentralised, shared global state** Global state is essential to the vision of building decentralised economic systems. Information such as asset balances, the status of smart contracts and publicly verifiable data must be available to network participants on a decentralised basis, without being subject to centralised control or arbitrary modification. Global state is also essential for public roots of trust such as known name services and digital identity. There should be *one and only one* official global state. This is essential if it is to serve as an authoritative source of truth for systems that depend upon it. A key role of network governance is to enforce this, and a key role of Convex peers is to maintain secure consensus over the global state and validate correct updates. The Convex global state is maintained by Peers, managed on a decentralised basis according to protocol rules and available to on-chain smart contracts. This is a true global state, that is not subdivided into shards, programs or other restricted sub-containers. This allows for full interoperability and atomic transactions between all users and automated actors. The global state is protected by strict access control rules: account holders on Convex may store information freely within the scope of their own accounts, but are prohibited from modifying the accounts of others without proper authorisation. Automated actors may provide custom access control rules allowing for sophisticated on-chain shared databases. A system of memory accounting ensures an economic cost is applied for those wishing to use on-chain memory within the global state. Refunds are provided when memory is released. This is reasonable and fair since on-chain memory is a scarce resource and we wish to incentivise efficient usage, including appropriate clean up when memory is no longer needed. ## The Data Lattice **We offer decentralised data systems with unlimited scalability alongside the global state** Data exists in many forms, and most data (by volume, if not by value) does not belong in a shared public global state. This may be for reasons of privacy or confidentiality, or it simply may not be economically feasible to handle large volumes of data as part of the global state. Such "off-chain" data might include personal files, large media files, AI models, data sets, databases, binary files, contract documentation etc. The Convex solution must provide an effective way to manage such data as part of decentralised systems alongside the global state. We are inspired by P2P technology such as BitTorrent and IPFS (which are usable with Convex based applications), but need something better to enable decentralised applications that can match or exceed the capabilities of centralised platforms. The data lattice provides this capability to all Convex users, with the following principles: - It is 100% peer to peer with no centralised services or single source of truth (if required, the global state provides this) - Data is structured to form mathematical lattices, enabling efficient coordination-free P2P replication via CRDTs - Users have full control over the data they store, and what they share with others - Data can be in any format, and take any structure. - Code is also data: programs and computations can be stored on or referenced from the data lattice, allowing any computational process (such as AI models) to be executed using the data lattice alone. - Specific data structures are provided to support effective operational databases and real-time applications, so that the data lattice can replace conventional centralised databases and storage. - All data is immutable, content addressable, and structured as merkle trees that can be validated with cryptographic hashes (enabling strong provenance claims and integrity validation) - Data can be partially loaded, stored and transmitted, allowing applications to operate efficiently on small subsets of much larger data structures - Data is 100% interoperable with the Convex global state, making it easy for hybrid applications to work with both in frictionless manner ## Programmability **Users are given secure, atomic, programmable smart contract capabilities** We allow value exchange to be specified and executed automatically, so that parties to the exchange can be certain that the transaction will complete successfully and be settled as a single atomic transaction. Alternatively, if something goes wrong, the whole transaction will be rolled back and nothing of value will be lost. Furthermore, programmability enables innovation in terms of new types of digital assets as well as decentralised autonomous economic actors (e.g. DAOs). Because it is impossible to anticipate the full scope of potential future innovation in advance, Convex provides for general purpose, Turing-complete programming languages on the Convex Virtual Machine (CVM). ## Transparency **The operation of open economic systems should be transparent to all participants** Transparent operation is essential for open economic systems to thrive. Data in the global state should be available to all to enable better decisions making and allow independent audits to ensure trust and accountability. There is a trade-off between economic transparency and individual privacy. The ability to operate pseudonymously is critical to allow individual privacy to be protected, while still providing transparency at the system level. While the Convex network is public, we expect individuals and organisations to take appropriate measures to ensure the protection of private data. For data that should remain private, the data lattice provides the capability to protect and control access on whatever terms the user chooses. Such data is compatible with but separate from the public global state. In this way, Convex users are empowered to choose the appropriate level of privacy for each type of data. ## Cost Effectiveness **Transaction costs should be negligible: they should not be a significant factor in user decisions whether or not to transact, but enough to fairly compensate providers of infrastructure and services** High transaction costs are a curse. They represent friction that prevents useful value exchange from taking place, a deadweight loss to the whole economy. They exclude those with less financial resources, as the costs may become prohibitive for smaller transactions. For technology products, they present a major barrier to adoption and regular usage. At the same time, it would be impractical and unfair to make transaction costs exactly zero: someone must provide real resources to operate and secure the network. Operators should be fairly compensated for the services they provide, and it is right that the users incurring these costs should pay for them. We solve this dilemma by making Convex scalable and efficient: being able to support a large number of transactions with low underlying infrastructure requirements results in low costs per transaction. We furthermore design the protocol so that this low cost is passed on to users. Convex transaction costs include both immediate processing costs (network and CPU) and long term storage costs. Both are accounted for automatically in the CVM, via juice accounting and memory accounting respectively. Read-only access to the network ("queries") is free at the protocol level, and should also generally be 100% free for users (subject only to peer operators being willing to service requests). The data lattice is also 100% free to users, although service providers may offer optional value added services for which they charge fees (such as redundant backup with high availability). We encourage innovation and experimentation in business models based on the data lattice. ## Digital Asset Innovation **Participants must be empowered to create and utilise digital assets** We want to enable true P2P value exchange, which will typically relate to one or more forms of digital asset. These assets may be entirely virtual, or represent ownership of some asset in the real world. We believe everyone should have the freedom to create and use digital assets of any type. Data assets may be "on-chain" (part of the global state) or kept "off-chain" on the data lattice. Given the infinite variety of possible assets, it is essential to allow users to innovate and create new kinds of assets without limitation. This may involve defining rules and governance mechanisms (both on-chain and off-chain). Convex will facilitate the development of new digital asset classes on a decentralised basis. The creation and usage of digital assets may be regulated in some jurisdictions. This is an emerging area of law with considerable uncertainty. The Convex network is neutral with regards to such regulations: it is the responsibility of participants to comply with relevant laws and regulation in their own jurisdiction. ## Open Architecture **Convex will operate as an open network in the spirit of the original Internet** Convex effectively adds new capabilities to the Internet: a secure, persistent global state machine that can efficiently execute transactions and enforce smart contracts backed by a powerful decentralised data lattice for arbitrary data and compute operations. As a protocol and network for value exchange, Convex is agnostic to the types of applications which are built on top of this base layer. Like the Internet, we support innovation and a wide variety of use cases. We do not limit or prescribe the possibilities in any way. According to the "end-to-end" principle, we expect most significant application functionality to be implemented within the end nodes (i.e. client applications and servers which access the Convex network). However, since the CVM does allow for some aspects of functionality (such as smart contracts and digital assets) to run on the network itself, the final decision regarding how much of the application to run "on-chain" in this way is left to application developers. We envision that some user facing applications will be pure dApps (i.e. clients which interact purely with decentralised networks) but that many will likely be hybrid dApps (some usage of decentralised networks but coupled with traditional centralised servers). Convex supports both configurations equally well. Furthermore, the open architecture supports "Layer 2" extensions to the network, allowing for even greater flexibility in building decentralised systems (for example, scaling transaction volumes outside the main network). We note that Layer 2 solutions have some drawbacks (most notably, lacking atomic transactions within the global state), but they will always be available as options via the open architecture for those that choose them. ## Open Source **Convex technology will be open source, available for free, and developed by the community in the interests of all users** We are inspired by the principles of open source software, making high quality software freely available to all users and encouraging collaboration. Most importantly, this applies to the libraries and tools required by users to interact with Convex, so that facilities to access the network are guaranteed to be freely available to all. The Convex implementation itself is open source software, released under the Convex Public License (CPL). The CPL is a relatively permissive open source license that supports the goal of effective network governance in addition to the usual open source freedoms. Contributors to Convex may retain copyright or other intellectual property. However, free license to use any such intellectual property (including patents) must be automatically granted to all Convex users and developers under the terms of the CPL, thereby ensuring the Convex network is always free to use, in line with our open principles. We develop software in public, according to open source principles. Currently, all core Convex technology is available on GitHub at [https://github.com/Convex-Dev]. ## Network Neutrality **We provide neutral network infrastructure, which does not censor or discriminate between users or applications** Networks should not discriminate amongst users, both from an ethical and practical perspective. This is consistent with our vision of Convex as a public utility available to all, in the spirit of the Internet. Indeed, Convex itself depends on the presence of the Internet as an underlying, censorship-resistant neutral communication network. In particular, the Convex network protocol treats all valid user transactions equally, without giving preferential treatment or blocking any transactions. ## Freedom to participate in network operations **Anyone may operate a peer on the network** As a decentralised network, anyone should be permitted to operate a Convex peer, participate in maintaining the secure consensus of the network and provide other infrastructure such as nodes on the data lattice. The only requirement is that, they follow protocol rules to ensure interoperability. Peer operators should be entitled to a fair share of rewards and transaction fees to compensate them for providing essential secure infrastructure. Provided that protocol rules are followed, peer operators are free to customise the software and services they operate as they wish. This might include value added services for users, high end analytics capabilities, performance optimisations etc. ## The Convex Coin **Users have access to Convex Coins as the native currency for the network** The Convex Coin is the native currency of the Convex network. The natural currency unit is `1 Convex Gold Coin`, each of which can be subdivided one billion times into a base coin unit (informally called "coppers"). The Convex Coin serves the traditional roles of a currency in that it is a store of value, a medium of exchange and a unit of account. This native currency is a utility token used for transaction fees which fairly compensate those who provide common resources to the network. This occurs in an automatic way, as part of the network protocol. A protocol defined native currency is also important for regulating usage of the network. If transactions were free in a public permissionless network, nothing would deter a malicious actor from flooding the network with transactions and denying service to genuine users. The supply of Coins is designed to increase over time in line with ecosystem demand. The supply may never however exceed a maximum supply cap of `1,000,000,000` Convex Coins. The use of the Convex Coin as part of the protocol does not in any way prevent other currencies from being established. Indeed, a key capability of Convex is the ability it gives for all users to create their own currencies, tokens and other digital assets. ## Community Recognition **Convex Coins will be issued based on ecosystem contribution** Convex serves its community and ecosystem, enabling them to build technology and create value for society through many different applications. We honour this with radical fairness: **100% of Convex Coins are initially issued for contribution**. No "pre-mines" for insiders. No VC dumps. Awards are scaled according to value contributed - whether open-source code, widely adopted dApps, community mentorship, or purchasing coins from the foundation to accelerate the ecosystem. Once issued, Convex Coins circulate freely on a self-sovereign basis, available for all to use. ## Good Governance **Convex is governed as a perpetual public utility for its users** Networks thrive when governance serves the commons, not cabals. Convex commits to this from genesis: upgrades that endure, protections that scale, capabilities that serve. Initially, governance role will be stewarded via the Convex Foundation, a non-profit organisation registered in the United Kingdom. It will have responsibility for protocol-level upgrades to the network and initial issuance of Convex coins. Decentralised governance is the destination: on-chain, credentialed, and attack-resistant. We transition only when battle-tested—ensuring every voice counts, every upgrade benefits users, and the network remains censorship-proof forever. Key governance roles include: - Authorising official updates to the peer network or protocol - Safeguarding the global state: the purpose of the global state is to act as a reliable single source of truth. As a public utility that records important economic information on a decentralised basis, the network must avoid forks since these present a significant risk of confusion and loss to participants in the ecosystem. Forks are not desirable in a system designed to act as a single source of truth for asset ownership, contract state and account balances etc. - Ensuring that issuance of Convex Coins is fair, secure and consistent with out principles of rewarding ecosystem contribution --- # Performance Convex offers outstanding performance for decentralised applications. Our objective is to offer the best combination of decentralised security and interactive performance for applications in categories such as DeFi, gaming and the metaverse with large numbers of concurrent users. ## The headline numbers - **Sub-second end-to-end finality** — consensus confirmation typically contributes ~200ms of a complete signed round trip (breakdown below) - **Zero block delay** — a peer publishes a block the instant it has a transaction; no leader slot, no waiting - **Up to 1 million transactions per second** raw CVM execution, measured by `CVMBenchmark`; the network targets 100,000+ TPS end-to-end - **~5 million reads / 800,000 writes per second** in Etch storage, on an ordinary laptop (`EtchBenchmark`) These figures were independently validated in the EU's Next Generation Internet (NGI) OntoChain project, which compared Convex against other decentralised platforms: ![CompareETH2](https://github.com/user-attachments/assets/0ed23d0b-85dc-4aa6-91f7-8fc6903bcf40) Since that study, further improvements have raised transaction throughput and lowered latency. The rest of this page explains where these numbers come from and how to reproduce them. ## How to think about performance We care primarily about two different measurements of performance: - **Latency** (how quickly results can be achieved) - important for interactive applications - **Throughput** (how many requests per second can we handle) - important for achieving internet scale Convex performance is based around a key idea: We implement consensus using a **CRDT** (Conflict-free Replicated Data Type) where the Peers achieve consensus by simply sharing a Belief data structure which is repeatedly merged with other Beliefs to form consensus. CRDTs are guaranteed to eventually converge to a consistent value under reasonable assumptions, which gives the desired properties of safety and liveness to the network. Peers, therefore, have a simple primary task: merge and propagate new beliefs to the network as quickly as possible. ## Latency Latency is the time to achieve some specified result (between the cause and the effect). From the perspective of a Convex client executing an end-to-end transaction, this would typically be due to six sequential stages (approximate contributions to latency given as examples): 1. Digitally signing a transaction (1ms) 2. Sending the signed transaction to a Peer (30ms) 3. Having the transaction included in a Block (5ms) 4. Confirming the Block in consensus (200ms) 5. Computing the result of the transaction / CVM state updates (10ms) 6. Returning the result to the Client (30ms) In some cases, a client may not need to wait for all six stages, for example, a client that trusts the reliability and correctness of the Peer they connect to may be happy to assume that their transaction will be successfully processed after Stage 2. Below we describe several key techniques we use to keep the overall latency as small as possible. ### Minimising Hops The total round trip transaction time is driven mainly by network lag (stages 2,4,6). In a decentralised network, this is inevitable - we cannot avoid the network latency between the Peer and the Client, and the need for a few rounds of communication between peers to confirm consensus (the theoretical minimum number of rounds for systems of this type is generally three, as can be proven in various consensus algorithms in the relevant literature e.g. PBFT). The CPoS consensus algorithm, fortunately, can achieve the theoretical minimum number of hops to confirm consensus under good conditions. Given a well-connected network, with the majority of highly staked Peers directly connected to each other, only three hops between Peers are required: 1. Peer broadcasts Belief containing new block to other Peers 2. Peers (representing at least 2/3 of total stake) broadcast Belief with a new block in the next position in consensus 3. Peers broadcast Belief confirming consensus based on observing the same proposal from 2/3 of other peers More hops may be required in the case that the network is less well connected. However, this typically only adds a small constant number of extra hops, since Belief propagation spreads exponentially across the network (and hence will cover the whole network in `log(n)` hops) ### Belief merge performance After network latency, the largest source of delay is the performance of the Belief merge function. This is central to the CPoS algorithm functioning as a CRDT and is typically performed many times as a transaction is propagated between Peers. This represents a delay before the merged Belief can be re-broadcast to the network, so we make this fast with the following measures: - The Belief merge can merge multiple received Beliefs from different Peers at once (more efficient than performing repeatedly for each Belief received) - The Belief merge handles multiple Blocks simultaneously (Which may come from different Peers) - Overall algorithm performance roughly is `O(n+m)` where `n` is the number of active Peers, and `m` is the number of additional Blocks being handled in the merge - We use a very efficient algorithm for comparing Orderings of Blocks from different Peers so that consistency can be confirmed (or conflicts detected). This effectively is `O(1)` even on long Orderings (we say "effectively" because it is technically `O(log n)` but the high branching factor and practical bounds on ordering length growth make this behave like `O(1)` in practice). ### Zero Belief propagation delay Because CPoS operates as a CRDT, Peers can immediately propagate a Belief as soon as they have performed a Belief merge. Propagating a new Belief as soon as one has been updated is the default behaviour for Peers, and is optimised for low latency performance. Because belief merges are idempotent, there is no harm in broadcasting a Belief multiple times. ### Delta Transmission Beliefs are large data structures, and it would be a significant performance cost if the entire Belief needed to be transmitted every time one was propagated - adding significant latency delays in Stage 4. Fortunately, we can send only the Deltas (changes) to a Belief in most circumstances. This is possible because: - Beliefs are structured as Merkle DAGs, allowing the integrity of the whole structure to be validated - We track which parts of a Belief have already been broadcast, and normally omit to send these again - Receivers of Beliefs can verify the Belief by examining the cryptographic hashes from the root of the Merkle DAG, and in most cases confirm that they already hold the rest of the data structure required. If not, they can always recover by explicitly requesting a missing piece of data (which again, they can identify using the cryptographic hash) In many ways, Belief propagation can be seen as analogous to the efficient storage and merging of source code trees utilised in the Git version control system. ### Zero Block delay A potential major source of latency from the perspective of a Client would be a block delay, i.e. the need to wait between receiving a transaction and producing a Block containing the transaction (within stage 3). Traditional blockchains almost all have at least some delays here, either needing to wait for an allocated time at which a block can be produced as the "leader" of the network, or solving a PoW problem to earn the right to produce a block. Convex solves this problem by implementing a zero-block delay strategy: 1. A Peer produces a Block immediately whenever it has at least one transaction 2. Blocks can be submitted immediately at any time by adding the Block to the current Peer Belief and broadcasting it This approach means that it is possible, and even likely, that multiple Peers will submit Blocks for consensus simultaneously. Fortunately, this is not a problem for the CPoS algorithm since successive Belief merges will efficiently resolve any conflicts and sort the Blocks into a stable consistent ordering. Hence, we can allow a zero-block delay strategy without compromising the overall consensus approach. Although the overhead of producing a Block is comparatively small, it is still more efficient to wait for a few transactions before producing a single larger Block. Hence this behaviour is configurable by Peer operators who can then offer clients a trade-off between cost and latency. ### CVM Execution Time Once a Block is confirmed in consensus, Peers are required to compute CVM state updates. This adds a small amount of latency (in stage 5). The good news is that the efficiency of the CVM means that this only takes a couple of milliseconds on average. The `BigBlockBenchmark` confirms that Convex can sequentially process over 400 Blocks of 1000 transactions per second on a single thread: ``` Result "convex.benchmarks.BigBlockBenchmark.benchmark": 468.024 ±(99.9%) 42.190 ops/s [Average] ``` Block processing, in general, is very quick because: - Raw CVM execution speed for transactions is very high (up to 1 million transactions per second) - Expensive operations (Merkle tree hashing, writing to storage) are performed lazily, typically only done once per Block - Blocks only have a small amount of additional processing required (timestamp updates, distribution of transaction fees etc.) ## Throughput Convex targets a peak transaction throughput of 100,000+ transactions per second. This is many orders of magnitude faster than traditional blockchains, and even faster than centralised payment systems such as VISA (which might typically handle in the range of 1700 TPS). The key to achieving this performance is the successive elimination of bottlenecks to performance, and choosing an architecture that makes it possible to efficiently utilise Peer resources. ### Staged Event-Driven Architecture We employ a staged event-driven architecture (SEDA) within Peers to optimise throughput and ensure that expensive work is performed concurrently on different threads. Hence overall throughput is maximised by maximising the throughput at each stage. Stages are connected with efficient in-memory queues which can easily transfer millions of events per second. This queue-based approach helps up to manage complexity by clearly decoupling the different stages and also allows for backpressure to be used to manage periods of high loads (an essential technique for high volume distributed systems). The most important stages are: 1. Network ingestion, where a NIO Server reads messages from the network at maximum speed and puts these messages on an in-memory queue 2. Message handling, where messages are decoded and either handled directly or placed on an appropriate queue 3. Peer update, where relevant messages (Beliefs, Transactions) are processed for the Peer's Belief update. All critical CPoS computation is performed here 4. Transaction signature verification 5. CVM execution, where the results of transactions are computed and CVM state updated 6. Outbound messaging, where responses are returned to other Peers / clients Convex is agnostic to the underlying hardware architecture used, however, this configuration of stages would be well suited for efficient execution on commodity PC hardware with 8-16 cores. ### CVM performance The CVM is a highly optimised execution engine. Some notable points: - Data structures and algorithms are all implemented in highly efficient, low-level JVM code (often with bitwise operations!) - CVM code is compiled down to efficient "Ops", where a typical transaction might require 10-50 Ops to be executed. This is effectively the "machine code" of the JVM, and corresponds to operations typical in an implementation of the Lambda Calculus - because of the power of the lambda calculus, we usually require fewer Ops than the equivalent for a stack-based machine - We can rely on the JVM JIT to further compile key code paths down to efficient native code The `OpBenchmark` is a microbenchmark of several different groups of Ops, and demonstrates that up to hundreds of millions of Ops can be executed per second on a single thread: ``` Benchmark Mode Cnt Score Error Units OpBenchmark.constant thrpt 5 522926192.803 ± 200159991.331 ops/s (a single Constant op) OpBenchmark.emptyLoop thrpt 5 2180.943 ± 118.609 ops/s (a loop executing 1000 times) OpBenchmark.simpleSum thrpt 5 8336440.019 ± 444054.388 ops/s (a sum involving dynamic lookup, 4 Ops) OpBenchmark.simpleSumPrecompiled thrpt 5 21846062.388 ± 1823368.198 ops/s (a sum with constants, 4 Ops) ``` The `CVMBenchmark` measures the time of applying entire transactions to the CVM state (comprising the atomic application of multiple Ops plus some transaction accounting), and suggests that over a million TPS may be feasible on the CVM. ``` Benchmark Mode Cnt Score Error Units CVMBenchmark.contractCall thrpt 5 573728.219 ± 45555.012 ops/s CVMBenchmark.defInEnvironment thrpt 5 808384.086 ± 64336.668 ops/s CVMBenchmark.simpleCalculationDynamic thrpt 5 1169199.192 ± 15417.298 ops/s CVMBenchmark.simpleCalculationStatic thrpt 5 1293331.725 ± 36267.904 ops/s CVMBenchmark.smallTransfer thrpt 5 845463.961 ± 16815.637 ops/s ``` ### Query offload We further enhance capacity by offloading queries (i.e. read-only requests that do not affect CVM state) to a separate thread. Because the CVM State is an immutable data structure, it can be copied as a snapshot in `O(1)` time and queries can be processed by separate threads, removing the need for queries to bottleneck the main CVM execution or Belief merge processes. Furthermore, it would be possible to replicate the CVM state to multiple servers and offer effectively unlimited scalability for query capacity. In practice we do not believe this to be necessary: a single CVM worker thread might be able to handle a million queries per second which are already enough for internet-scale transaction volumes, (for example Google might need to handle in the region of 60,000 search queries per second). ### Storage engine Database storage is often a significant bottleneck in computational systems that require durable storage and reliability. Because of this, we have implemented `Etch`, a highly performant database specifically designed to meet the storage needs of Convex: - Content addressable storage (keyed by Hash of value encodings, i.e. Storage resembles an immutable Merkle DAG) - Embedded encodings of small values mean that many values can typically be read or written in a single storage call - High performance, memory-mapped file access Our `EtchBenchmark` suggests that Etch can handle approximately 5 million reads per second, and 800,000 writes per second on a regular laptop. This compares highly favourably with alternative database engines, and means we can comfortably achieve 100,000+ TPS (most transactions are likely to require only a small number of writes, especially given the embedding of small values). Reads can furthermore be executed concurrently on multiple threads, which is particularly helpful for query performance. ### Network bandwidth The majority of network bandwidth is required for transaction data, which are typically in the range of 100-150 bytes. Allowing for some additional overheads and Belief propagation, 100,000 TPS is likely to require handling sustained streams of data in the region of 16 Mb/s. We must also allow for streams of data of similar size to be maintained for each Peer to which a Peer is connected, say 5-20 different Peers. Overall, this level of bandwidth is plausible for a 1 Gigabit network connection, although a faster connection would be recommended for heavily staked Peers wishing to handle peak loads or serve the needs of a particularly large number of clients. 10 Gbps would probably be recommended for such Peers. Fortunately, Peers have significant control over bandwidth consumption: they can limit the number of Peer connections they maintain to reduce bandwidth requirements. They can also temporarily drop out of consensus (by reducing their stake to zero) if necessary. ### Signature verification Convex uses high performance, well-tested libraries for crypto algorithms, most notably LibSodium for Ed25519. With multiple cores focused primarily on signature verification, Peers can realistically validate 100,000+ transactions per second. This is actually the largest proportion of the computational work done by peers under load. We are exploring strategies to relieve this bottleneck and push performance even further, in particular: - We have an experimental design to allow small, low staked peers to randomly skip some percentage of signature verifications (which can be proved safe for practical purposes because all signatures will still be checked by a good peer with probability arbitrarily close to 1). - Hardware-based approached to signature verification may be feasible ## Conclusion and future work Engineering high-performance systems is a challenge, and we're proud of the great work to get Convex so far as we move towards the main network launch. In the future, we expect to continue to improve performance and innovate in this domain. Key ideas include: - Support for subnets, enabling work to be performed separately from the main CVM Global State. This is a "Layer 2" approach that has some drawbacks (mainly, being disconnected from atomic updates to the Global state) but may support some use cases that wish to operate more independently and only occasionally sync up with the Global State using techniques such as state channels. - Further optimisation of the core Convex code base. We expect to be able to squeeze out a variety of fine-grained optimisations, especially in the core data structures and algorithms. - Support for dedicated read-only "query" Peers which can serve the needs of read-intensive workloads without burdening the primary set of consensus-maintaining Peers. --- Convex is a general purpose, programmable decentralised network with global state. As such, there is no inherent limit on the nature of applications that can be built. Here are some of the things that Convex can be used for, and are being actively developed within the Convex community: ## AI Agent Economies AI agents are first-class economic participants on Convex: they can hold accounts, own assets, execute transactions and contract with humans and other agents under exactly the same rules. Every peer ships with a built-in [MCP server](/docs/products/convex-mcp), so any MCP-compatible agent can query state and transact directly — no custom integration required. The [x402 protocol](/docs/cad/x402) adds HTTP-native micropayments, making pay-per-use APIs practical for autonomous agents. This makes Convex the natural settlement and coordination layer for agentic systems: - Machine-to-machine payments and micropayment-gated services - Agents holding and trading digital assets with sub-second finality - Deterministic smart contract escrow and coordination between agents - Decentralised identity for agents via [DIDs](/docs/cad/did) For federated AI orchestration across organisational boundaries — models, agents and data collaborating with built-in governance — see [Covia](https://covia.ai), which is built on Convex lattice technology. Start with the [AI Agents guide](/docs/tutorial/agents). ## Tokenisation Convex offers a uniquely powerful system for the development of tokenised assets and applications. Tokens in Convex usually support the [CAD029](/docs/cad/fungible) token standard, which offers multiple advantages over the common ERC20 standard. Tokens can be used for: - Payments in the form of stablecoins, flatcoins, nationcoins or other digital currency types - Utility tokens for delivery of decentralised services - Reward schemes and bonus programmes - Governance of decentralised economic systems, e.g. with stake-weighted voting ## NFTs Non-fungible tokens (NFTs) are a powerful mechanism for creating unique virtual items that can be owned, collected traded and utilised in virtual environments such as games. Each NFT has a unique on-chain ID and can be individually addressed, transferred and used within smart contracts. Convex includes a powerful NFT engine based on set theory, capable of millions of NFT transfers per second, and allowing arbitrary metadata and smart contract rules to be attached to individual NFTs. ## DeFi Convex provides comprehensive support for decentralised finance applications. As part of the full-stack solution we offer: - A uniform, fungible token model suitable for utility tokens, stablecoins, governance tokens and more. - An advanced decentralised token exchange based on constant liquidity pools allowing for efficient trading of fungible tokens of all types. - Template smart contracts and libraries for creating and interacting with DeFi assets - Capabilities to delegate control of assets to intermediaries, custodians or decentralised smart contracts - Capabilities for digital asset governance including trusted smart contract upgrades ## Gaming Games need fast interactive performance and shared global state for immersive multiplayer worlds. With transaction confirmations in milliseconds and minimal transaction costs, Convex is well suited to decentralised gaming, virtual items and in-game currencies. Our first game is a 3D block game inspired by Minecraft, Boulderdash and the Roguelike genre. The entire game world is implemented as a set of Convex Actors. Transactions are automatically executed when the player takes actions that change the world state, and the player can otherwise view and explore the world at zero cost. ## Metaverse Convex provides the ideal substrate for the convergence of real and virtual worlds into new shared realities. With a rich model of digital assets and on-chain smart contract capabilities, we can help you craft the next generation of experiences in virtual reality. Our metaverse innovation includes the world's [first fully on-chain 3D game world](https://www.youtube.com/watch?v=op2NccyMEBE). ## Payments and Digital Currencies Convex supports an [advanced model of digital currencies](https://www.youtube.com/live/XHxEnqBwgN8?feature=shared&t=95) with customisable governance, suitable for everything from stablecoins and flatcoins to branded community currencies. Our [open source mobile wallet](https://github.com/Convex-Dev/convexity) (Convexity) makes these easily accessible and convenient for users, and can be customised to create branded experiences with additional functionality. [Here's a demo](https://www.youtube.com/watch?v=2a40mRkcuag). With incredible performance, low costs and low latency, Convex digital currencies are ideal for consumer use cases such as digital payments, international transfers and retail purchases. If you are involved in a project to create a digital currency, we can help you succeed: get in touch at [info@convex.world](mailto:info@convex.world). --- Below are some practical pointers and best practices for actor development. ## Efficiency ### Minimise O(n) memory Memory is a scarce global resource. It's fine to use memory for your actor code and basic data structures, but you should be very careful about memory allocation that scales in an O(n) fashion (e.g. with number of users, number of NFTs issued). You should aim for only a few bytes of storage when O(n) scaling is happening. Some hints: - If data isn't needed on-chain, don't store it on-chain. The global state is not the place for `.png` avatar files. Have a small integer ID on-chain, and store the big data at a URI that incorporates the ID, e.g. `https://foo.com/user/10101/avatar.png` - Store per-user data in a vector like `[name address age]` rather than a structured map like `{:name name :address address :age age}`. The extra key metadata wastes bytes. ### Allow garbage collection CVM memory accounting rewards those who free up memory. Always allow methods to remove dead / redundant / expired records from the CVM state. You can call these yourselves, or let others call them. ## Security ### Distrust all inputs Anyone can call a `:callable` function. Assume malicious inputs may be passed and consider: - Can they corrupt the data in the actor with invalid values? - Can they cause values to be stored in the actor which may be undesirable - Can they modify state in any other way that might be harmful? Best practices are: - Validate all inputs - `fail` if the input is wrong in any way (state changes will get rolled back, so this is safe) - Do not store user-submitted content in the actor state (logging or producing return values is fine) - Write adversarial unit tests that give invalid inputs and check that the actor rejects them appropriately ### Sandbox external calls When making a `call` to an external actor where you only want the result and don't expect any state changes, it is safer to sandbox this in a `query` ```clojure (defn price-from-oracle [oracle] (query (call oracle (current-price)))) ;; should be read-only, safe inside query ``` This advice is especially important if you are making an external call early, e.g. as part of a pre-condition check in the CEI model. ### Beware `eval` If you ever use `eval` or `eval-as` then be extremely careful of code injection attacks: do not allow these to run on any untrusted input. ### Beware untrusted function arguments First-class functions come with the danger that arbitrary code can be passed as values. Consider the following: ```clojure (def vals [1 2 3 4]) (defn ^:callable process-values [f] (mapv f vals)) (call *address* (process-values inc)) => [2 3 4 5] ``` This is a massive vulnerability, since `f` is an arbitrary function and gets called in the context of this account. Anyone can take over this actor with something like: ```clojure (call ... (process-values (fn [x] (set-controller *address*) (set-key nil) :HAHA))) ``` Solutions: - Don't allow functions as arguments to `:callable` actor functions - If you must take functions as inputs, always run them inside a `query` for safety. Any state changes will be rolled back, so you are safe whatever the function does. ### Apply CEI pattern Always structure actor functions as **Checks -> Effects -> Interactions** . - **Checks** are preconditions on inputs / current state. You want to fail early if anything is invalid. - **Effects** are any mutations to the state of the current actor. Do these *before* any external interactions to eliminate risk of re-entrancy attacks - you do not want any external actor to be able to make a re-entrant call while this actor is in an inconsistent state. - **Interactions** are any external calls (e.g. token transfers facilitated by other actors). If these all succeed, then everything is good. If anything fails, we want to roll back. ## Development Process ### Use Queries for testing Anything executed in a `query` is safe (state changes are automatically rolled back). So they are very useful when: - You want to test code in a specific environment - You don't want to incur unnecessary fees - You don't want to risk causing damage --- # Building an Actor This walkthrough builds a small but complete actor — a shared counter that anyone can increment, but only its owner can reset. Every step runs in the [Sandbox](/docs/tutorial/quickstart) or a REPL; the `=>` lines show the expected result. ## 1. Deploy We deploy in one step, defining the owner, the initial state, and three callable functions: ```clojure (def counter (deploy '(do (def owner *caller*) ;; the account that deploys becomes the owner (def count 0) (defn ^:callable increment [] (def count (inc count)) count) (defn ^:callable get-count [] count) (defn ^:callable reset [] (when-not (= *caller* owner) (fail :TRUST "Only the owner may reset")) (def count 0))))) => #1234 ``` `deploy` runs the quoted code inside a brand-new account and returns its address. `owner` captures `*caller*` — the deployer — as the trusted account. ## 2. Call it Anyone can increment and read the count: ```clojure (call counter (increment)) => 1 (call counter (increment)) => 2 (call counter (get-count)) => 2 ``` Inside each call, `*caller*` is the calling account and `*address*` is the counter itself. ## 3. Owner-only actions The owner can reset: ```clojure (call counter (reset)) => 0 ``` If a **different** account calls `reset`, the `:TRUST` check fails — and because a failed call rolls back completely, the count is left untouched: ```clojure ;; called from some other account (call counter (reset)) => Exception: :TRUST Only the owner may reset (call counter (get-count)) => 0 ;; unchanged — the failed reset had no effect ``` That is the core actor-safety pattern: **authorise against `*caller*`, and let `fail` roll back on rejection** (see [Key Concepts → Authorisation](concepts.md#authorisation)). ## 4. Where next - **[Coin Distributor](../recipes/coin-distributor/index.md)** — a complete actor that distributes coins, using these same patterns - **[Account Control](../recipes/account-control/index.md)** — controllers and multi-signature - **[Evolution](evolution.md)** — upgrading a live actor - **[Best Practices](best-practices.md)** — checklists for safe actors --- Actors are programmable, autonomous accounts that power automation on Convex. ## Actor Accounts Every Convex account can hold data and code in their own environment. Actors extend this model by bundling code with state: - **User Account**: controlled by an externally held key pair, executes transactions it receives as long as they are correctly signed - **Actor**: autonomous account, exposes callable functions that other accounts or actors can call — its `:key` is typically `nil` (so no external transactions can be signed for it) Use regular accounts when external users (humans, client software, AI agents etc.) need to submit transactions for the account. Use actors when you need trusted programmable or shared behaviour. ## Callable code Actors are accessed via **callable functions**, which are just regular functions annotated with the metadata `^:callable`. A simple example might look like this: ```clojure ;; In account #1337 (defn ^:callable hello [] (str "Hello " *caller* " from " *address*)) ``` This can be called from another account as follows: ```clojure ;; In account #11 (call #1337 (hello)) => "Hello #11 from #1337" ``` Note that while the callable actor function is running `*caller*` is the account that called the function, and `*address*` is the actor (`*address*` always references the CVM account in which code is currently running) ## Authorisation Because anyone can call a callable function, **every function that changes state or moves value must check who is calling**. Authorise against `*caller*` (the immediate caller) — never `*origin*` (the original transaction signer), which can be misused when calls are chained through other actors. The simplest pattern stores a trusted owner and checks it: ```clojure (def owner *caller*) ;; whoever deployed the actor (defn ^:callable set-value [v] (when-not (= *caller* owner) (fail :TRUST "Not authorised")) (def value v)) ``` For anything beyond a single owner, use a **trust monitor** — an actor implementing the `convex.trust` interface — via `(@convex.trust/trusted? owner *caller*)`. Trust monitors express roles, allow-lists and delegated permissions uniformly. See [CAD022: Trust Monitors](/docs/cad/trustmon). Failing is safe: `fail` (and any error) rolls back **all** state changes made during the call, so a rejected call leaves the actor untouched. ## Lifecycle 1. **Authoring**: write Convex Lisp functions that return updated state and values. 2. **Deployment**: upload code to the lattice or global state and compile it on-chain. 3. **Evolution**: upgrade code or migrate state as requirements change. Transactions against actors are atomic: either the whole call succeeds (with state updates) or it fails with no partial effects. This property makes complex coordination safe. ## Interaction Flow Actors process calls atomically within the CVM: 1. A caller submits a transaction containing a `call` instruction with the actor address and function. 2. The CVM context switches to the actor, then evaluates the function. 3. Results and state changes are committed if evaluation succeeds, or rolled back otherwise. Because calls are atomic and completely ordered, you can write deterministic logic without worrying about race conditions between transactions. ## Comparison to EVM contracts For developers familiar with coding in Solidity on the EVM, the following differences may be helpful to note: - Actors are typically more sophisticated than EVM contracts, e.g. a single actor might represent an entire class of digital assets (multiple tokens) rather than a single ERC20 token - Actors are easy to upgrade, thanks to the dynamic nature of the CVM --- ## Actor deployment Actors are created with the `deploy` function, which: - Creates a new, empty account in the CVM state - Runs the given code inside that account, so its `def`s become the actor's state and functions - Returns the address of the new actor The code you pass is **quoted** — `deploy` runs it in the new account rather than in the caller's: ```clojure (def greeter (deploy '(defn ^:callable greet [name] (str "Hello, " name "!")))) => #1234 (call greeter (greet "World")) => "Hello, World!" ``` ### Multiple forms `deploy` accepts several forms, run in order — useful for setting up state and functions together. You will also see a single `(do ...)` form used to group them: ```clojure (deploy '(def owner *caller*) ;; the deployer '(def total 0) '(defn ^:callable add [n] (def total (+ total n)))) ``` ## Test before you deploy A bad deploy can leave you with a broken or uncontrollable actor, so test the code first. Because code run inside a `query` is always rolled back, you can deploy-and-test safely without committing anything: ```clojure (query (let [a (deploy '(do (def n 0) (defn ^:callable inc-get [] (def n (inc n)) n)))] (assert (= 1 (call a (inc-get)))) (assert (= 2 (call a (inc-get)))) :ok)) => :ok ``` If the query returns `:ok`, the same code behaves identically when you deploy it for real — the whole transaction is atomic, so a test that passes in the query phase deploys reliably. Test adversarially too: pass bad inputs and confirm the actor rejects them (e.g. with a `:CAST` or `:TRUST` error) rather than misbehaving. --- Convex is designed for real-time dApps and self-sovereign economic systems. This is a rapidly evolving space, and it is likely that actors will need to be upgraded to access new capabilities. While in an ideal world, you might want actors be be completely immutable in terms of their behaviour, in practice this is often not desirable and you want to upgrade and evolve them. ## Upgradable Actors A key risk of developing smart contracts is that once they are live, significant losses may occur if bugs are found. Losses could be from theft by malicious actors that manage to exploit a security weakness, or a bug that causes assets to be permanently lost. It is therefore *an option* to make actors upgradable. This is a trade-off: You gain the ability to patch problems in the original smart contract, but also open up the possibility that this upgrade feature itself may be exploited by attackers or accidentally misused. ## Example approach - Account controller The simplest way to make an actor upgradable is to the the `*controller*` account feature, which allows the controller account to use the `eval-as` function to update the actor. ```clojure ;; Deploy an empty, upgradable actor that is empty but has the controller set ;; Note: *caller* within deployed code is always the account that called deploy (def UP-ACTOR (deploy '(set-controller *caller*))) ;; add a callable function to the actor (eval-as UP-ACTOR '(defn ^:callable hello [] "Hello upgrade!")) ;; Check the newly added function works! (call UP-ACTOR (hello)) => "Hello upgrade!" ``` ## Example approach - Trusted upgrade You can also write a `:callable` function that performs the upgrade. This requires slightly more code than simple setting a `*controller*` however it has some advantages: - You can write custom logic to validate whether the caller has upgrade rights - You can make the upgrade specific to a particular part or feature of the actor - This can be more secure than allowing a general `eval-as` ```clojure ;; deploy an upgradable Actor (def UPGRADABLE (deploy '(do ;; make the initial deployer be the owner (def owner *caller*) ;; initial version of the upgradable function (defn ^:callable get-version [] "Version 1") ;; upgrade function (only callable by owner) (defn ^:callable upgrade [new-impl] (if (= *caller* owner) (set! get-version new-impl ) (fail :TRUST "Not authorised to upgrade")))))) ;; Check initial version of function (call UPGRADABLE (get-version)) => "Version 1" ;; Check we are the owner! (= *address* UPGRADABLE/owner) => true ;; Upgrade the callable function to a new function definition (call UPGRADABLE (upgrade (fn [] (str "Version " (inc 1) " - dynamically computed")))) ;; Call the new function (call UPGRADABLE (get-version)) => "Version 2 - dynamically computed" ``` ## Security Risks Adding a general-purpose upgrade feature like this lets you correct bugs or add new enhancements to actors, but it opens the risk that the same mechanism could be used to compromise the actor's behaviour if an attacker were able to impersonate the owner, and also creates the risk that the actor may be permanently disabled by the owner by mistake. As always, you must perform your own security analysis to determine whether this trade-off is worthwhile for actors that you deploy. Here are some recommended security practices for upgradable actors: ### Security of Owner / Controller account Compromise of the controller account is the biggest risk, as this account may be able to make arbitrary changes to the actor. Ideally, for this account: - Highest security measures should be used - ideally air-gapped usage with secure physical backups - Upgrade transaction should be signed offline then transferred to an internet-connected computer for submission to the network ### Bundle the upgrade code with checks By adding checks to the upgrade code that fail if an assumption is not met, you can ensure an automatic rollback of any failed upgrade ## Immutable actors If you are sure your actor logic will never change, you can make it immutable. This is done by: - Setting `*key*` to `nil` so that no external transactions can be signed for the actor - Setting `*controller*` to `nil` so that no controller can modify or upgrade it - Ensuring there are no callable functions that can make changes to any code (e.g. using `eval` or `defn`) The above are true by default when you call `deploy`, so the following will make an immutable actor: ```clojure (deploy ;; Accept any offered coins '(defn ^:callable receive-coin [_ _ _] (accept *offer*)) ;; Let anyone collect coins (a public donation, if you like....) '(defn ^:callable collect-coins [] (transfer *caller* *balance*))) ``` If you have an existing actor that you want to make immutable, you will need to remove any upgradability. This can be done with something like the following (if you are the controller of the actor): ```clojure (eval-as ACTOR '(do ;; Safety check to make sure we are not accidentally making ourselves immutable! ;; - Must execute in a called account (in this case via eval-as) ;; - Must not run in the account of the *caller* itself (assert *caller* (not (= *caller* *address*))) (set-controller nil) ;; Remove controller (set-key nil) ;; Remove any external account key (undef upgrade-function) ;; Remove any callable function that allows upgrades )) ``` --- Convex actors are autonomous programs that hold state, expose callable functions, and participate in economic systems on-chain. This section introduces the concepts, tools, and best practices you need to build robust actors on the Convex Virtual Machine (CVM). **NOTE**: Most of the time you don't need to develop your own actors: solid economic systems already exist on Convex that are sufficient for many use cases (e.g. minting and trading fungible tokens of any kind). So actor development is only needed if you really need to implement your own custom logic or have fine-grained control over behaviour. ## Where to start Each subpage builds on the previous one, but you can jump directly to what you need: - [Key Concepts](./actors/concepts) — core ideas, anatomy, and authorisation of Convex actors. - [Building an Actor](./actors/building-an-actor) — a runnable end-to-end example; start here to get hands-on. - [Deployment](./actors/deployment) — deploying and testing actors on the CVM. - [Evolution](./actors/evolution) — strategies for real-time upgrades of actors. - [Best Practices](./actors/best-practices) — practical advice and checklists. ## Worked examples Complete, copy-pasteable actors in the Recipes section: - [Coin Distributor](../recipes/coin-distributor/index.md) — distribute coins to many recipients - [Account Control](../recipes/account-control/index.md) — controllers and multi-signature patterns --- AI agents interact with Convex through regular accounts — the same kind of account that humans use. There are no special "agent accounts". An agent needs an address, and optionally a key pair, to participate in the network. This guide covers how to create, fund, and secure accounts for agent use. ## Creating an agent account ### Via MCP Use the `createAccount` tool to create an account with an optional faucet payout (available on testnets): ```json { "name": "createAccount", "arguments": { "faucet": 100000000 } } ``` The response includes the new account address and a generated key pair. On testnets the faucet provides initial funds; on the protonet you would fund the account separately. To create an account with a specific public key (e.g. one you generated elsewhere): ```json { "name": "createAccount", "arguments": { "accountKey": "89b5142678bfef7a2245af5ae5b9ab1e10c282b375fa297c5aaeccc48ac97cac" } } ``` ### Via the TypeScript SDK ```typescript import { Convex, KeyPair } from '@convex-world/convex-ts'; const convex = new Convex('https://peer.convex.live'); const kp = KeyPair.generate(); const info = await convex.createAccount(kp, 100_000_000); convex.setAccount(info.address, kp); ``` ### Via Convex Lisp If you already have an account, you can create new accounts on-chain: ```clojure ;; Create account with a public key (create-account 0x89b5142678bfef7a2245af5ae5b9ab1e10c282b375fa297c5aaeccc48ac97cac) ``` ## Key management approaches The most important security decision for an agent deployment is how the agent accesses its signing key. There are two primary approaches. ### Option 1: Agent holds the private key The agent has the Ed25519 seed (private key) in its environment or configuration. It can sign transactions directly using the `transact` or `signAndSubmit` MCP tools. ```json { "name": "transact", "arguments": { "source": "(transfer #99 1000000)", "address": "42", "seed": "a1b2c3d4e5f6..." } } ``` **Advantages:** - Simple setup — no external signing infrastructure - Fully autonomous — agent can transact at any time without external approval **Risks:** - A compromised agent has full access to the key - The seed is visible to the MCP transport layer - No spending limits or policy enforcement This approach is suitable for development, testnets, and controlled environments where the agent runs in trusted infrastructure. ### Option 2: MCP signing tool (recommended) The agent prepares transactions but a separate signing service handles the actual signing. The agent uses the `prepare` and `submit` tools, while a dedicated signing tool or service holds the key. ``` Agent Signing Service Peer │ │ │ │── prepare(source, addr) ───────────────────────────►│ │◄─────────────────────────── hash ───────────────────│ │ │ │ │── "please sign this hash" ─►│ │ │◄── signature ──────────────│ │ │ │ │ │── submit(hash, sig, key) ──────────────────────────►│ │◄─────────────────────────── result ─────────────────│ ``` The signing service can be: - Another MCP tool that the agent calls (e.g. a `convex-signer` tool) - A hardware security module (HSM) or secure enclave - A human approval workflow for high-value transactions - A policy engine that enforces spending limits, allow-listed addresses, and rate limits **Advantages:** - Agent never sees the private key - Signing service can enforce arbitrary policies - Defence in depth — compromised agent cannot steal funds - Supports human-in-the-loop review **Risks:** - More complex setup - Signing service becomes a dependency (agent cannot transact if it's unavailable) :::tip Best practice For production deployments, use the signing tool approach. Even a simple signing service that checks a spending limit per transaction provides significant protection against agent misbehaviour or compromise. ::: ## Account controllers Convex accounts can have a **controller** — another account that can execute arbitrary code in the controlled account's context using `eval-as`. This is a powerful mechanism for agent governance. ### Setting a controller When creating an agent account, set a controller so that a human or governance system can intervene if needed: ```clojure ;; As the agent account (using key authority) (set-controller #100) ;; Or as an existing controller (eval-as #42 '(set-controller #100)) ``` ### What a controller can do A controller can execute any code in the agent's account: ```clojure ;; Reset the agent's key (e.g. after key rotation) (eval-as #42 '(set-key 0xNEW_PUBLIC_KEY)) ;; Recover funds from a misbehaving agent (eval-as #42 '(transfer #100 *balance*)) ;; Upgrade agent logic stored in the account (eval-as #42 '(defn ^:callable strategy [] (new-implementation))) ``` :::warning Only set a controller you trust completely — a controller has full authority over the account. For agents, the controller is typically the deployer's own account or a governance actor. ::: See the [Account Control recipe](/docs/tutorial/recipes/account-control) for more detail on working with controllers. ## Delegated authority Sometimes you want an agent to act on your behalf without handing over your key or making it a full [controller](#account-controllers). **UCAN capability tokens** let one identity delegate specific, attenuated permissions to another: a signed token granting exactly which resources and abilities the agent may use, and nothing more. Delegations can be chained and narrowed at each hop, and each is bound to the issuer's key. The MCP `signingDelegate` tool issues such a token from a stored key. See [CAD038: Lattice Authentication](/docs/cad/lattice_auth) for the capability model. Agents are identified by a **DID** (Decentralised Identifier). Convex's native `did:convex` method anchors an identity to a Convex account — an address or CNS name — so a delegation can name exactly which agent identity it grants authority to. See [CAD043](/docs/cad/did). ## Funding agent accounts Agents need a Convex Coin balance to pay for transaction execution (juice costs). ### On testnets Use the faucet when creating the account: ```json { "name": "createAccount", "arguments": { "faucet": 100000000 } } ``` Or fund an existing account: ```clojure ;; From another account with funds (transfer #42 100000000) ``` ### On the protonet Transfer funds from a treasury or operational account. There is no faucet on the production network. Plan your agent's operational budget based on expected transaction volume and juice costs. ### Monitoring balance Agents should monitor their own balance to avoid running out of funds: ```json { "name": "query", "arguments": { "source": "*balance*", "address": "42" } } ``` ## Managing multiple agents Each agent should have its own account. This provides: - **Isolation** — one agent's transactions don't affect another's sequence numbers - **Auditability** — on-chain history is per-account - **Independent key management** — each agent can have its own security model ### Deterministic key derivation Use the `keyGen` MCP tool with a master seed and agent-specific derivation: ```json { "name": "keyGen", "arguments": { "seed": "master-seed-hex" } } ``` Or derive keys programmatically using the TypeScript SDK: ```typescript import { KeyPair } from '@convex-world/convex-ts'; // Deterministic: same seed always produces same key pair const agentKey = KeyPair.fromSeed(agentSeedHex); ``` This allows you to recreate agent keys from a master secret without storing individual seeds. ## Next steps - [Agentic Economics](./agentic-economics) — design economic interactions between your agents - [MCP Integration](./mcp) — connect agents to the network - [Actor Development](/docs/tutorial/actors) — build on-chain actors that agents can interact with --- Convex is designed for a world where humans and autonomous agents participate in the same economy under the same rules. The CVM doesn't distinguish between a transaction submitted by a human and one submitted by an AI agent — both pay the same juice costs, follow the same asset model, and receive the same finality guarantees. This makes Convex a natural substrate for **agentic economics**: systems where AI agents own assets, execute strategies, and coordinate with each other through on-chain contracts. ## Agents as economic citizens Every agent on Convex is just an account. An agent can do anything a human account can do: - Hold and transfer Convex Coins - Own fungible tokens and NFTs - Deploy and interact with smart contracts (actors) - Set controllers and manage other accounts - Register and manage CNS names There are no special permissions, no separate API, and no privileged access. Agents earn trust the same way humans do: by building a track record of on-chain behaviour that others can verify. ## Direct transactions Agents transact using standard Convex Lisp, the same language humans use in the REPL or client SDKs. **Transfer coins:** ```clojure (transfer #99 1000000) ``` **Call an actor function:** ```clojure (call #500 (place-order :buy MY-TOKEN 100 5000)) ``` **Deploy a new actor:** ```clojure (deploy '(do (def state (atom {})) (defn ^:callable register [name] (swap! state assoc *caller* name)))) ``` **Interact with fungible tokens:** ```clojure ;; Check balance (@convex.fungible/balance #128 *address*) ;; Transfer tokens (@convex.fungible/transfer #128 #99 500) ``` All of these work identically whether the submitting account is controlled by a human or an AI agent. ## Smart contracts for agent coordination Agents coordinate through on-chain actors — shared, deterministic programs that enforce rules atomically. This is where agentic economics becomes powerful: agents don't need to trust each other, they just need to trust the contract. ### Escrow pattern Two agents want to swap assets. Neither trusts the other to send first. An escrow actor solves this: ```clojure ;; Agent A offers tokens to Agent B (@convex.asset/offer #agentB #128 1000) ;; Agent B accepts the offer (atomic — either both sides execute or neither does) (@convex.asset/accept #agentA #128 1000) ``` The CVM's atomic transactions guarantee that the swap either completes fully or rolls back entirely. No partial execution, no stuck funds. ### Registry pattern Agents can register capabilities, prices, or availability in a shared on-chain registry: ```clojure ;; Agent registers its service (call #registry (register-service {:type :translation :languages [:en :fr :de] :price-per-word 10})) ;; Another agent queries available services (call #registry (find-services {:type :translation :language :fr})) ``` ### Auction pattern On-chain auctions let agents compete fairly with transparent bidding: ```clojure ;; Agent places a bid (call #auction (bid item-id 50000)) ;; Auction resolves automatically when conditions are met ;; Winner pays, asset transfers — all atomic ``` ## Autonomous strategies A typical autonomous agent follows a **query-decide-transact** loop: 1. **Query** the current state of the world (prices, balances, contract state) 2. **Decide** on the next action based on the agent's model or policy 3. **Transact** to execute the chosen action 4. **Verify** the result and update internal state ### Example: market-making agent ``` loop: // 1. Query bid_price = query("(call #dex (best-bid MY-TOKEN))") ask_price = query("(call #dex (best-ask MY-TOKEN))") my_balance = query("*balance*") // 2. Decide if spread > threshold and my_balance > minimum: new_bid = bid_price + 1 new_ask = ask_price - 1 // 3. Transact transact("(call #dex (place-order :buy MY-TOKEN 100 new_bid))") transact("(call #dex (place-order :sell MY-TOKEN 100 new_ask))") // 4. Verify position = query("(call #dex (my-orders))") sleep(interval) ``` The agent uses MCP tools for each step: `query` for reads, `transact` (or `prepare`/`submit`) for writes. ### Example: treasury management An agent that manages a shared treasury, rebalancing funds across assets: ```clojure ;; Query current holdings (def holdings {:coins *balance* :token-a (@convex.fungible/balance #token-a *address*) :token-b (@convex.fungible/balance #token-b *address*)}) ;; Rebalance if allocation drifts beyond threshold (when (> (abs (- (/ (:token-a holdings) total) target-allocation)) 0.05) (@convex.fungible/transfer #token-a #dex rebalance-amount)) ``` ## Trust and verification One of Convex's strengths for agentic systems is **verifiability**. All agent actions are recorded on-chain and can be inspected by anyone. ### Verify before transacting Before interacting with another agent or contract, an agent can verify the counterparty's state: ```clojure ;; Check the other agent's balance (balance #counterparty) ;; Inspect a contract's code and state (account #contract) ;; Verify a contract hasn't been modified (:controller (account #contract)) ;; nil means immutable — no one can change it ``` ### Governance via controllers Account controllers provide a governance layer for agent systems: - **Operational control** — a human operator sets themselves as controller of an agent's account. If the agent misbehaves, the operator can freeze funds, rotate keys, or shut down the agent - **Multi-sig governance** — a governance actor (itself controlled by multiple parties) serves as controller for high-value agent accounts - **Immutability** — setting the controller to `nil` makes an agent's on-chain code permanent and unmodifiable ```clojure ;; Operator can recover funds from a misbehaving agent (eval-as #agent-account '(transfer #treasury *balance*)) ;; Or rotate the agent's key (eval-as #agent-account '(set-key 0xNEW_KEY)) ``` ## Multi-agent architectures Real-world agentic systems often involve multiple agents with different roles. Convex supports several coordination patterns. ### Shared treasury Multiple agents share a treasury actor that enforces spending policies: ```clojure ;; Treasury actor allows authorised agents to withdraw up to a daily limit (defn ^:callable withdraw [amount] (assert (authorised? *caller*)) (assert (<= (+ (daily-spent *caller*) amount) daily-limit)) (transfer *caller* amount)) ``` ### On-chain coordination Agents coordinate through a shared contract rather than direct messaging: ```clojure ;; Task queue actor — agents post tasks and claim them (defn ^:callable post-task [description reward] (let [id (inc @next-id)] (swap! tasks assoc id {:poster *caller* :desc description :reward reward :status :open}) id)) (defn ^:callable claim-task [id] (let [task (get @tasks id)] (assert (= (:status task) :open)) (swap! tasks assoc-in [id :status] :claimed) (swap! tasks assoc-in [id :worker] *caller*))) ``` ### Off-chain discovery, on-chain settlement Agents discover each other via MCP (or other off-chain protocols) but settle transactions on-chain: 1. Agent A discovers Agent B's MCP server and negotiates terms off-chain 2. Agent A creates an on-chain offer (e.g. token swap) 3. Agent B verifies the offer on-chain and accepts 4. Settlement is atomic and final — no trust required between the agents This pattern combines the flexibility of off-chain communication with the finality of on-chain execution. ## Paying for services with x402 Agents frequently need to pay for external resources — an API call, a dataset, a model inference. Convex implements the [x402 protocol](/docs/cad/x402), which activates the dormant HTTP `402 Payment Required` status code so a service can demand payment inline, with no accounts, API keys or OAuth. The flow is simple: an agent requests a resource, the server responds `402` with payment details, the agent pays (in CVM or any [CAD029 token](/docs/cad/fungible)) and retries with proof of payment. Because it works through ordinary HTTP headers and status codes, any agent with an HTTP client and a Convex account can transact — making micropayment-gated APIs practical for autonomous agents. x402 support is under active development; see [CAD042](/docs/cad/x402) for the current specification. ## Economic constraints Convex applies the same economic constraints to agents as it does to humans. These constraints are features, not bugs — they prevent runaway agents from disrupting the network. - **Juice pricing** — every computation costs juice, priced in Convex Coins. An agent that enters an infinite loop or generates excessive transactions will exhaust its balance and stop naturally - **Memory accounting** — storing data on-chain costs memory allowance. Agents can't bloat the global state without paying for it - **Sequence numbers** — each account has a monotonically increasing sequence number that prevents replay attacks and ensures transaction ordering - **Deterministic execution** — all agents see the same state and get the same results for the same queries. No front-running, no information asymmetry at the protocol level These constraints make Convex a safe environment for autonomous agents: the physics of the system bound what any single agent can do. ## Next steps - [MCP Integration](./mcp) — connect your agents to the network - [Account Management](./account-management) — set up agent accounts and security - [Actor Development](/docs/tutorial/actors) — build the on-chain contracts your agents will interact with - [Convex Lisp Guide](/docs/tutorial/convex-lisp) — learn the language your agents will use for on-chain logic --- AI agents are first-class participants in the Convex economy. They can own accounts, hold assets, execute transactions, and contract with humans and other agents — all under the same rules, the same costs, and the same finality. Convex doesn't distinguish between human users and autonomous agents. Every account on the network follows the same physics: deterministic execution, atomic transactions, and cryptographic authentication. This makes Convex a natural substrate for agentic systems where AI models need to interact with real economic state. ## Why Convex for AI agents? - **Native MCP support** — Convex peers ship with built-in [Model Context Protocol](/docs/products/convex-mcp) servers, so any MCP-compatible AI agent can interact with the network out of the box - **Deterministic execution** — the CVM guarantees identical results for identical inputs, eliminating a whole class of agent coordination bugs - **Sub-second finality** — transactions settle in under a second, fast enough for real-time agent decision loops - **Global state** — agents can read the entire network state (balances, contracts, assets) with a single query, no indexers required - **Universal asset model** — one interface for fungible tokens, NFTs, and arbitrary digital assets ([CAD019](/docs/cad/assets)) - **Economic constraints** — juice pricing and memory accounting prevent runaway agents from spamming the network ## Guides Each guide builds on the previous one, but you can jump directly to what you need: - [MCP Integration](agents/mcp) — connect AI agents to Convex via the Model Context Protocol. Covers available tools, querying state, executing transactions, and signing approaches. - [Account Management](agents/account-management) — create, fund, and secure agent accounts. Covers key management, controllers, and the trade-offs between full autonomy and delegated signing. - [Agentic Economics](agents/agentic-economics) — design economic interactions between agents. Covers direct transactions, smart contract coordination, autonomous strategies, and multi-agent architectures. ## Further reading - [Convex MCP Product Overview](/docs/products/convex-mcp) — full reference for the MCP server capabilities - [CAD041: Model Context Protocol](/docs/cad/mcp) — technical specification - [Actor Development](/docs/tutorial/actors) — build on-chain actors that agents can interact with - [Account Control](/docs/tutorial/recipes/account-control) — recipe for managing accounts via controllers --- The **Model Context Protocol (MCP)** is a standardised interface that allows AI agents to discover and invoke tools across different systems. Convex peers ship with a built-in MCP server, turning any peer into a gateway for agents to query state, execute transactions, manage assets, and receive real-time state change notifications on the Convex network. ## Connecting to a Convex peer Every Convex peer with MCP enabled exposes an endpoint at: ``` https:///mcp ``` For a local peer this is typically `http://localhost:8080/mcp`. The peer also publishes a discovery document at `/.well-known/mcp` so that MCP clients can auto-detect the server. ### MCP client configuration Most MCP-compatible tools (Claude Desktop, Cursor, Windsurf, etc.) accept a JSON configuration. To connect to a Convex peer: ```json { "mcpServers": { "convex": { "url": "http://localhost:8080/mcp" } } } ``` For a public peer: ```json { "mcpServers": { "convex": { "url": "https://peer.convex.live/mcp" } } } ``` Once connected, the agent can discover all available tools and prompts through MCP's standard `tools/list` and `prompts/list` methods. ## Available tools Convex peers expose over 30 tools via MCP, organised into categories: - **Queries and state** — execute Convex Lisp queries (`query`), inspect accounts (`describeAccount`, `getBalance`), navigate the global state tree (`queryState`), resolve CNS names, and check peer status - **Transactions** — execute transactions directly with a seed (`transact`, `transfer`), or use the secure prepare/sign/submit flow for external signing approval - **Cryptography** — Ed25519 key generation (`keyGen`), signing, verification, and SHA-256/SHA3 hashing - **Encoding** — convert between CVM literals and CAD3 binary format - **State watching** — register watches on any path in the global state (`watchState`) and receive SSE notifications when values change; remove watches with `unwatchState` - **Signing service** — when configured, server-side key management tools (`signingTransact`, `signingCreateAccount`, etc.) where private keys are stored encrypted and never leave the server Use `tools/list` to discover the exact set of tools available on any given peer. ## Prompts The MCP server also provides guided workflow prompts that teach LLMs about Convex and walk them through common tasks. Prompts are user-controlled — triggered by slash commands or explicit selection in the client. Available prompts cover account exploration, network status inspection, Convex Lisp guidance, account creation, contract deployment, and fund transfers. Use `prompts/list` to discover them. ## Querying state The `query` tool is the starting point for most agent interactions. Queries are free (no transaction fees) and require no signing. **Check an account balance:** ```json { "name": "query", "arguments": { "source": "(balance #42)" } } ``` **Read account details:** ```json { "name": "query", "arguments": { "source": "(account #42)" } } ``` **Resolve a CNS name:** ```json { "name": "query", "arguments": { "source": "@convex.fungible" } } ``` **Query a fungible token balance:** ```json { "name": "query", "arguments": { "source": "(@convex.fungible/balance #128 #42)", "address": "42" } } ``` Queries can execute arbitrary Convex Lisp, so agents can inspect any on-chain state — contract environments, asset holdings, governance parameters, etc. ## Executing transactions Transactions modify on-chain state and require cryptographic signing. The MCP server supports three approaches, each with different security trade-offs. ### Approach 1: Direct transaction (agent holds seed) The simplest approach. The agent provides its Ed25519 seed directly and the peer handles signing: ```json { "name": "transact", "arguments": { "source": "(transfer #99 1000000)", "address": "42", "seed": "a1b2c3d4..." } } ``` This is straightforward but means the agent (and the MCP connection) has access to the private key. Suitable for controlled environments where the agent is trusted. ### Approach 2: Prepare + external sign + submit (recommended) A more secure pattern where the agent never touches the private key directly. Instead: 1. **Prepare** — the agent asks the peer to prepare the transaction and return a hash: ```json { "name": "prepare", "arguments": { "source": "(transfer #99 1000000)", "address": "42" } } ``` Response includes a `hash` field. 2. **Sign externally** — the hash is signed by a separate signing tool, hardware wallet, or human-in-the-loop approval process. This could be another MCP tool, a secure enclave, or a manual review step. 3. **Submit** — the agent submits the signed transaction: ```json { "name": "submit", "arguments": { "hash": "0xabc123...", "sig": "0xdef456...", "accountKey": "0x789..." } } ``` This pattern keeps the private key outside the agent's environment. The signing service can enforce policies (spending limits, allow-listed destinations, rate limits) before approving. :::tip Recommended for production The prepare/sign/submit flow is the recommended approach for production agent deployments. It provides defence in depth: even if the agent is compromised, it cannot sign transactions without approval from the signing service. ::: ### Approach 3: Combined sign and submit A convenience method that combines preparation, signing, and submission: ```json { "name": "signAndSubmit", "arguments": { "source": "(transfer #99 1000000)", "address": "42", "seed": "a1b2c3d4..." } } ``` Like approach 1, the agent needs the seed. This is useful for fully autonomous agents in trusted environments. ### Approach 4: Signing service (recommended for production) When the peer has a signing service configured, the agent can transact without ever handling private keys. Keys are generated and stored encrypted on the server: ```json { "name": "signingTransact", "arguments": { "source": "(transfer #99 1000000)", "address": "#42", "passphrase": "user-provided-passphrase" } } ``` The signing service stores keys encrypted at rest with a user-chosen passphrase. Sensitive operations like key import, export, and deletion require an additional browser-based confirmation step. :::tip Recommended for production The signing service is the recommended approach for production agent deployments. Private keys never leave the server, and the passphrase-based encryption ensures that even server administrators cannot access them without the user's passphrase. ::: ## Running your own peer For production agent deployments, running your own Convex peer is strongly recommended: - **Performance** — direct local access eliminates network round-trips to public peers - **Security** — your MCP endpoint isn't exposed to the public internet - **Privacy** — query patterns and transaction data stay within your infrastructure - **Reliability** — no dependency on third-party peer availability See the [Peer Operations](/docs/tutorial/peer-operations) guide for setup instructions. MCP support is enabled by default — no additional configuration is needed. :::warning Public peers are suitable for experimentation and development, but production agent systems should use a dedicated peer. This ensures that your agent's private keys and transaction patterns are not exposed to shared infrastructure. ::: ## Example: agent workflow A typical agent interaction with Convex via MCP follows this pattern: 1. **Query** — read current state (balances, prices, contract state) 2. **Decide** — the agent's model determines the next action 3. **Prepare** — create a transaction for the chosen action 4. **Sign** — approve and sign (directly, via signing service, or external signer) 5. **Submit** — execute the signed transaction 6. **Verify** — query the updated state to confirm the result This loop runs continuously for autonomous agents, or on-demand for agents responding to user requests. For reactive agents, `watchState` enables event-driven workflows: the agent registers watches on relevant state paths and receives SSE notifications when values change, eliminating the need for polling. ## Next steps - [Account Management](./account-management) — set up accounts, keys, and security for your agents - [Agentic Economics](./agentic-economics) — design economic interactions between agents - [Convex MCP Reference](/docs/products/convex-mcp) — full product documentation - [CAD041: MCP Specification](/docs/cad/mcp) — technical specification --- ### Java Native JVM client for high-performance enterprise applications. - **Status**: Production ready - **Package**: `world.convex:convex-java` (on Maven Central) - **Environments**: JVM (Java 21+) - **Features**: Native performance, CompletableFuture async, zero overhead - **Repository**: [github.com/Convex-Dev/convex](https://github.com/Convex-Dev/convex) (convex-java module) **[Get Started with Java →](client-sdks/java/)** --- ### Python Pythonic client for scripting, automation, and data science. - **Status**: Production ready - **Package**: `convex-sdk` (on PyPI) - **Environments**: Python 3.8+ - **Features**: Synchronous I/O, Account objects, automatic sequence retry - **Repository**: [github.com/Convex-Dev/convex-api-py](https://github.com/Convex-Dev/convex-api-py) **[Get Started with Python →](client-sdks/python/)** --- ### Rust (Coming Soon) Zero-cost abstractions for systems programming. - **Status**: 🚧 Planned - **Features**: Memory-safe, high-performance - **Use Cases**: Performance-critical applications, embedded systems --- ## Feature comparison | Feature | TypeScript / JS | Java | Python | |---------|:---:|:---:|:---:| | Queries | ✅ | ✅ | ✅ | | Transactions | ✅ | ✅ | ✅ | | Accounts & funding | ✅ | ✅ | ✅ | | Fungible tokens | ✅ first-class handles | ✅ via CVM calls (+ `TokenBuilder`) | ✅ via CVM calls | | CNS name resolution | ✅ handles | ✅ via CVM calls | ✅ helper methods | | Custom / hardware signers | ✅ `Signer` API | direct `AKeyPair` | direct `KeyPair` | All three SDKs can perform **any** on-chain operation by submitting CVM code — the differences above are about first-class convenience wrappers, not capability. ## Command Line Interface For terminal-based interaction with Convex, see the [Convex CLI](/docs/products/convex-cli). ## Need Help? - 💬 [Discord Community](https://discord.com/invite/xfYGq4CT7v) - 📚 [Documentation](https://docs.convex.world) - 🐛 [Report Issues](https://github.com/Convex-Dev/convex.ts/issues) --- # Account Management Learn how to create, manage, and secure Convex accounts using key pairs in Java. ## Key Concepts ### Account An **Account** represents an identity on the Convex network with: - **Address** - Unique numeric identifier (e.g., `#1234`) - **Key Pair** - Ed25519 cryptographic keys for signing - **Balance** - Convex Coins (copper) owned by the account - **Sequence** - Transaction counter (starts at 0) - **Environment** - Variables and functions defined by the account ### AKeyPair An **AKeyPair** contains Ed25519 cryptographic keys: - **Account Key** - Public key shared with the network (32 bytes) - **Seed** - Private key material, used for signing (32 bytes) **⚠️ Never share your seed!** Anyone with it can control your account. ## Creating Key Pairs ### Generate New Keys Create a new random key pair: ```java import convex.core.crypto.AKeyPair; // Generate fresh Ed25519 keys AKeyPair keyPair = AKeyPair.generate(); System.out.println("Public key: " + keyPair.getAccountKey()); // Example: 0x36d8c5c40dbe2d1b0131acf41c38b9d37ebe04d85... ``` ### Create from Seed Restore keys from a 32-byte seed: ```java import convex.core.data.Blob; // 32-byte seed (keep secret!) byte[] seed = new byte[32]; // ... load seed from secure storage ... AKeyPair keyPair = AKeyPair.create(seed); System.out.println("Restored public key: " + keyPair.getAccountKey()); ``` ### Create from Hex String ```java // Seed as hex string String seedHex = "5b32f8ff94d1f901098fa41a4045449726849dda5e1a3de34ae123037418795c"; Blob seedBlob = Blob.fromHex(seedHex); AKeyPair keyPair = AKeyPair.create(seedBlob.getBytes()); ``` ## Exporting Keys ### Export Seed Save the seed bytes securely: ```java import java.nio.file.Files; import java.nio.file.Path; AKeyPair keyPair = AKeyPair.generate(); // Get seed bytes (32 bytes) byte[] seed = keyPair.getSeed(); // Save to file (should be encrypted in production!) Path keyFile = Path.of("keypair.dat"); Files.write(keyFile, seed); System.out.println("Keys saved to " + keyFile); ``` **Best Practices:** - Encrypt seed before saving to disk - Store in secure location with appropriate file permissions - Make multiple backups - Never commit to version control ### Export as Hex Get seed as a hex string: ```java byte[] seed = keyPair.getSeed(); String seedHex = Blob.create(seed).toHexString(); System.out.println("Seed (hex): " + seedHex); // 5b32f8ff94d1f901098fa41a4045449726849dda5e1a3de34ae123037418795c ``` ## Account Setup ### Configure Convex Client Set your account address and key pair: ```java import convex.api.Convex; import convex.core.crypto.AKeyPair; import convex.core.cvm.Address; Convex convex = Convex.connect("https://peer.convex.live"); // Generate or load key pair AKeyPair keyPair = AKeyPair.generate(); // Set key pair for signing convex.setKeyPair(keyPair); // Set account address Address myAddress = Address.create(1234); convex.setAddress(myAddress); System.out.println("Account configured: " + myAddress); ``` ### Get Current Configuration ```java // Get current address Address address = convex.getAddress(); System.out.println("Address: " + address); // Get current key pair AKeyPair keyPair = convex.getKeyPair(); System.out.println("Public key: " + keyPair.getAccountKey()); ``` ## Creating Accounts ### Request Account Creation On test networks, you can request account creation from the peer: ```java // Generate key pair AKeyPair keyPair = AKeyPair.generate(); // On test networks, some peers provide account creation // This typically requires calling a specific API endpoint // or using the peer's faucet service // For development, you may need to use the Convex Desktop // or CLI tools to create initial accounts ``` **Note**: On production networks, someone with an existing funded account must create your account for you. See the [Faucet Guide](/docs/tutorial/coins/faucet) for details. ### Load Existing Account Reconstruct account from saved keys and known address: ```java import java.nio.file.Files; import java.nio.file.Path; // Load seed from file Path keyFile = Path.of("keypair.dat"); byte[] seed = Files.readAllBytes(keyFile); // Create key pair AKeyPair keyPair = AKeyPair.create(seed); // Configure convex client convex.setKeyPair(keyPair); convex.setAddress(Address.create(1234)); // Your known address System.out.println("Loaded account: #" + convex.getAddress()); ``` ## Account Properties ### Query Account Information ```java import convex.core.lang.Reader; import convex.core.Result; Address myAddress = convex.getAddress(); // Query account info Result result = convex.query( Reader.read("(get-account " + myAddress + ")"), Address.create(1) ).get(); if (!result.isError()) { AMap accountInfo = (AMap) result.getValue(); // Extract properties Long balance = (Long) accountInfo.get(Keywords.BALANCE); Long sequence = (Long) accountInfo.get(Keywords.SEQUENCE); System.out.println("Balance: " + balance); System.out.println("Sequence: " + sequence); } ``` ### Check Balance ```java Result result = convex.query( Reader.read("(balance " + myAddress + ")"), myAddress ).get(); if (!result.isError()) { long balanceCopper = ((Number) result.getValue()).longValue(); double balanceCoins = balanceCopper / 1_000_000_000.0; System.out.println("Balance: " + balanceCoins + " CVM"); } ``` ## Signing Transactions Accounts sign transactions using their private keys: ```java import convex.core.transactions.Invoke; import convex.core.data.SignedData; // The SDK handles signing automatically when you call transact() Result result = convex.transact(Reader.read("(def x 42)")).get(); // For manual signing: ATransaction tx = Invoke.create(myAddress, Reader.read("(def x 42)")); SignedData signed = keyPair.signData(tx); // Submit signed transaction Result manualResult = convex.transact(signed).get(); ``` ### Manual Signing For advanced use cases, manually sign data: ```java import convex.core.data.ABlob; import convex.core.data.Hash; // Data to sign ABlob data = Blob.fromHex("7e2f1062f5fc51ed65a28b5945b49425aa42df6b7e67107efec357794096e05e"); // Sign with key pair ASignature signature = keyPair.sign(data.getBytes()); System.out.println("Signature: " + signature); ``` ## Transferring Account Control Change the key pair that controls an account: ```java // Generate new keys AKeyPair newKeys = AKeyPair.generate(); // Set the new public key on-chain String transaction = "(set-key " + newKeys.getAccountKey() + ")"; Result result = convex.transact(Reader.read(transaction)).get(); if (!result.isError()) { // Update local key pair convex.setKeyPair(newKeys); System.out.println("Account keys updated"); // IMPORTANT: Save new keys before this step! byte[] newSeed = newKeys.getSeed(); Files.write(Path.of("new_keypair.dat"), newSeed); } ``` **⚠️ Warning**: - Save new keys BEFORE transferring! - Old keys will no longer work - Losing new keys means losing account access forever ## Key Security Best Practices ### 1. Store Keys Securely ```java import javax.crypto.Cipher; import javax.crypto.SecretKey; import javax.crypto.spec.SecretKeySpec; // ✅ GOOD - Encrypted storage public void saveKeysSecurely(AKeyPair keyPair, String password) throws Exception { byte[] seed = keyPair.getSeed(); // Derive encryption key from password (use proper KDF in production) SecretKey key = deriveKey(password); // Encrypt seed Cipher cipher = Cipher.getInstance("AES/GCM/NoPadding"); cipher.init(Cipher.ENCRYPT_MODE, key); byte[] encrypted = cipher.doFinal(seed); // Save encrypted seed Files.write(Path.of("keys.enc"), encrypted); } // ❌ BAD - Plaintext storage // Don't do this in production! Files.write(Path.of("keys.txt"), keyPair.getSeed()); ``` ### 2. Use Secure Random ```java import java.security.SecureRandom; // ✅ GOOD - AKeyPair.generate() uses SecureRandom internally AKeyPair keyPair = AKeyPair.generate(); // ❌ BAD - Don't create seeds manually with weak RNG // byte[] weakSeed = new Random().nextBytes(32); // DON'T! ``` ### 3. Clear Sensitive Data ```java // Clear sensitive data from memory when done byte[] seed = keyPair.getSeed(); try { // Use seed... } finally { // Clear array Arrays.fill(seed, (byte) 0); } ``` ### 4. Separate Hot and Cold Wallets ```java // Cold wallet - offline, secure storage AKeyPair coldWallet = AKeyPair.generate(); byte[] coldSeed = coldWallet.getSeed(); // Store offline, never expose to network // Hot wallet - online, small balance AKeyPair hotWallet = AKeyPair.generate(); convex.setKeyPair(hotWallet); convex.setAddress(Address.create(5678)); // Use for day-to-day transactions ``` ### 5. Never Share Private Keys ```java // ✅ GOOD - Share public key System.out.println("My public key: " + keyPair.getAccountKey()); // ✅ GOOD - Share address System.out.println("Send funds to: #" + myAddress); // ❌ BAD - Never do this! // System.out.println("My seed: " + Blob.create(keyPair.getSeed())); // NEVER! ``` ## Account Lifecycle ```java import convex.api.Convex; import convex.core.crypto.AKeyPair; import convex.core.cvm.Address; import java.nio.file.Files; import java.nio.file.Path; // 1. Generate keys AKeyPair keyPair = AKeyPair.generate(); // 2. Save keys securely Files.write(Path.of("keypair.dat"), keyPair.getSeed()); // 3. Create account on network (requires existing funded account) // This step varies by network configuration // 4. Configure client Convex convex = Convex.connect("https://peer.convex.live"); convex.setKeyPair(keyPair); convex.setAddress(Address.create(1234)); // 5. Use account Result result = convex.transact(Reader.read("(def x 42)")).get(); // 6. Later: restore from backup byte[] restoredSeed = Files.readAllBytes(Path.of("keypair.dat")); AKeyPair restoredKeys = AKeyPair.create(restoredSeed); convex.setKeyPair(restoredKeys); ``` ## Common Patterns ### Development Account Quick account setup for testing: ```java public Convex createDevAccount() throws Exception { Convex convex = Convex.connect("https://peer.convex.live"); AKeyPair keyPair = AKeyPair.generate(); convex.setKeyPair(keyPair); // Assume test account at #1000 convex.setAddress(Address.create(1000)); return convex; } ``` ### Production Account Secure account for production: ```java public Convex loadProductionAccount() throws Exception { // Load from secure environment String keyPath = System.getenv("CONVEX_KEY_PATH"); String addressStr = System.getenv("CONVEX_ADDRESS"); byte[] seed = Files.readAllBytes(Path.of(keyPath)); AKeyPair keyPair = AKeyPair.create(seed); Convex convex = Convex.connect("https://peer.convex.live"); convex.setKeyPair(keyPair); convex.setAddress(Address.create(Long.parseLong(addressStr))); return convex; } ``` ### Key Rotation Periodically rotate keys for security: ```java public void rotateKeys(Convex convex, AKeyPair oldKeys) throws Exception { // Generate new keys AKeyPair newKeys = AKeyPair.generate(); // Save new keys first! Files.write(Path.of("new_keypair.dat"), newKeys.getSeed()); // Update on-chain Result result = convex.transact( Reader.read("(set-key " + newKeys.getAccountKey() + ")") ).get(); if (!result.isError()) { // Update client convex.setKeyPair(newKeys); System.out.println("Keys rotated successfully"); // Archive old keys securely Files.move( Path.of("keypair.dat"), Path.of("keypair.dat.old") ); } else { System.err.println("Key rotation failed: " + result.getErrorCode()); } } ``` ### Multiple Accounts Manage multiple accounts: ```java public class AccountManager { private final Convex convex; private final Map accounts = new HashMap<>(); public void addAccount(String name, AKeyPair keyPair, Address address) { accounts.put(name, new AccountConfig(keyPair, address)); } public void useAccount(String name) { AccountConfig config = accounts.get(name); if (config != null) { convex.setKeyPair(config.keyPair()); convex.setAddress(config.address()); } } record AccountConfig(AKeyPair keyPair, Address address) {} } ``` ## Working with Addresses ### Create Address from String ```java // From string with # prefix Address addr1 = Address.parse("#1234"); // From string without prefix Address addr2 = Address.parse("1234"); // From long Address addr3 = Address.create(1234L); ``` ### Format Addresses ```java Address address = Address.create(1234); // Standard format System.out.println("Address: " + address); // #1234 // As long long addressLong = address.longValue(); // In queries String query = "(balance " + address + ")"; ``` ## Next Steps - **[Query Guide](queries)** - Learn how to read network state - **[Transaction Guide](transactions)** - Execute state-changing operations - **[Convex Lisp Guide](/docs/tutorial/convex-lisp)** - Learn the smart contract language - **[Recipes](/docs/tutorial/recipes)** - Common account patterns and examples --- # Assets Convex has no special "token" primitive — fungible tokens and named assets are ordinary [actors](/docs/cad/fungible) you deploy and call. This guide issues and moves a fungible token from Java using the CVM-typed `convex.api.Convex` client. :::note Which client? These examples use the typed `convex.api.Convex` client (CVM values and `Result`), consistent with the rest of the Java guide — see [Clients](clients.md). The `convex.java.asset` package (`TokenBuilder`, `Fungible`) offers convenience wrappers built over the simpler `ConvexJSON` client; handy for quick scripts, but the typed client preserves full CVM type fidelity. ::: The snippets below assume a connected `Convex convex` with a key pair and address set (see [Clients](clients.md)), and that `myAddress` is your account's `Address`. ## Deploy a token ```java import convex.api.Convex; import convex.core.Result; import convex.core.lang.Reader; import convex.core.cvm.Address; Result deploy = convex.transact(Reader.read( "(deploy [(@convex.fungible/build-token {:supply 1000000})])")).get(); Address token = (Address) deploy.getValue(); // the token's actor address ``` ## Check a balance ```java Result r = convex.query(Reader.read( "(@convex.fungible/balance " + token + " " + myAddress + ")")).get(); long balance = ((Number) r.getValue()).longValue(); // 1000000 ``` ## Transfer tokens The generic `convex.asset` library moves any asset; the amount is paired with the token as `[token amount]`: ```java Address recipient = /* ... */; convex.transact(Reader.read( "(@convex.asset/transfer " + recipient + " [" + token + " 1000])")).get(); ``` ## Convenience: TokenBuilder If you are using the `ConvexJSON` client, the `convex.java.asset` package wraps the same operations in a fluent API: ```java import convex.java.asset.TokenBuilder; import convex.java.asset.Fungible; Fungible token = new TokenBuilder().withSupply(1_000_000).deploy(convexJSON); long balance = token.getBalance(); ``` ## See Also - [CAD029: Fungible Token Standard](/docs/cad/fungible) - [Transactions](transactions.md) — submitting and confirming transactions - [Clients](clients.md) — choosing the right Java client --- # Client Types The Java SDK provides three different ways to connect to the Convex network, each with different performance characteristics and use cases. ## Overview | Client Type | Speed | Complexity | Use Case | |-------------|-------|------------|----------| | **HTTP Client** | Fast | Simple | Most applications, web services, general use | | **Binary Client** | Faster | Moderate | High-throughput applications, real-time systems | | **Local Peer** | Fastest | Expert | Testing, development, embedded systems | ## HTTP Client (REST API) The HTTP client connects to a Convex peer's REST API endpoint. This is the **recommended default** for most applications. ### When to Use - Web applications and services - Getting started with Convex - Applications behind firewalls/proxies - Cloud deployments - When simplicity matters more than raw speed ### Connection ```java import convex.api.Convex; // Connect to peer's HTTP endpoint (default port 18888) Convex convex = Convex.connect("https://mikera1337-convex-testnet.hf.space"); // With explicit port Convex convex = Convex.connect("https://peer.example.com:18888"); ``` ### Characteristics - **Protocol**: HTTPS with JSON payloads - **Port**: 18888 (default) - **Performance**: Fast (typical latency 10-50ms depending on network) - **Firewall**: Works through standard HTTP proxies - **SSL/TLS**: Full encryption support - **Connection**: Stateless (new HTTP request per operation) ### Example ```java import convex.api.Convex; import convex.core.Result; import convex.core.lang.Reader; public class HttpClientExample { public static void main(String[] args) throws Exception { // Connect via HTTP to public testnet Convex convex = Convex.connect("https://mikera1337-convex-testnet.hf.space"); // Execute query Result result = convex.query( Reader.read("(balance #13)") ).get(); System.out.println("Balance: " + result.getValue()); convex.close(); } } ``` ## Binary Protocol Client The binary client establishes a direct TCP connection to a peer using Convex's efficient binary protocol. Use this for **high-performance applications** that need lower latency. ### When to Use - High-throughput transaction systems - Real-time applications - When you need minimal latency - Direct peer-to-peer communication - Applications with sustained connection requirements ### Connection ```java import convex.api.Convex; import java.net.InetSocketAddress; // Connect to peer's binary protocol port (default 18888) InetSocketAddress peerAddress = InetSocketAddress.createUnresolved( "mikera1337-convex-testnet.hf.space", 18888 ); Convex convex = Convex.connect(peerAddress); ``` ### Characteristics - **Protocol**: Custom binary protocol over TCP - **Port**: 18888 (default, same as HTTP) - **Performance**: Faster (typical latency 5-20ms) - **Firewall**: May require specific TCP port access - **Connection**: Persistent stateful connection - **Overhead**: Lower bandwidth usage than HTTP/JSON - **Multiplexing**: Multiple concurrent requests on single connection ### Example ```java import convex.api.Convex; import convex.core.Result; import convex.core.crypto.AKeyPair; import convex.core.cvm.Address; import convex.core.lang.Reader; import java.net.InetSocketAddress; public class BinaryClientExample { public static void main(String[] args) throws Exception { // Connect via binary protocol InetSocketAddress peerAddress = InetSocketAddress.createUnresolved( "mikera1337-convex-testnet.hf.space", 18888 ); Convex convex = Convex.connect(peerAddress); // Set up account AKeyPair keyPair = AKeyPair.generate(); convex.setKeyPair(keyPair); convex.setAddress(Address.create(1234)); // Execute many transactions efficiently for (int i = 0; i < 100; i++) { Result result = convex.transact( Reader.read("(def x " + i + ")") ).get(); System.out.println("Transaction " + i + " completed"); } convex.close(); } } ``` ### Performance Tip The binary client maintains a persistent connection, making it ideal for applications that submit many operations over time. The first connection has slightly higher overhead, but subsequent operations are much faster than HTTP. ## Local Peer Client The local peer client gives you **direct access to a peer running in the same JVM**. This is the fastest possible option but requires expert knowledge and careful resource management. ### When to Use - Testing and development - Embedded applications - When you need absolute maximum performance - Running your own peer node - Advanced debugging and monitoring ### Connection ```java import convex.api.Convex; import convex.peer.Server; // Start a local peer Server server = Server.create(); server.launch(); // Connect directly to local peer Convex convex = Convex.connect(server); ``` ### Characteristics - **Protocol**: Direct in-memory method calls - **Performance**: Fastest (sub-millisecond latency) - **Overhead**: No network or serialisation overhead - **Resource**: Requires running full peer (memory intensive) - **Expertise**: Requires understanding of peer lifecycle management - **Use Case**: Primarily testing, development, and specialized deployments ### Example ```java import convex.api.Convex; import convex.core.Result; import convex.core.crypto.AKeyPair; import convex.core.cvm.Address; import convex.core.lang.Reader; import convex.peer.Server; public class LocalPeerExample { public static void main(String[] args) throws Exception { // Create and launch local peer Server server = Server.create(); server.launch(); try { // Connect directly to local peer Convex convex = Convex.connect(server); // Create test account with funds AKeyPair keyPair = AKeyPair.generate(); Address address = convex.createAccountSync(keyPair.getAccountKey()); // Set up client convex.setKeyPair(keyPair); convex.setAddress(address); // Execute operations with minimal latency Result result = convex.transact( Reader.read("(def my-value 42)") ).get(); System.out.println("Result: " + result.getValue()); convex.close(); } finally { // Important: shut down peer properly server.shutdown(); } } } ``` ### Important Considerations **Memory Requirements**: A local peer requires significant memory (typically 2-4GB minimum) for the full node database and consensus state. **Lifecycle Management**: You are responsible for properly starting and stopping the peer. Always use try-finally or try-with-resources to ensure cleanup. **Testing Only**: Local peers are typically used for testing. Production applications usually connect to dedicated peer infrastructure. **Peer Configuration**: Local peers can be configured with custom parameters: ```java import convex.peer.Config; import convex.peer.Server; // Create peer with custom configuration Config config = Config.create(); config = config.withPort(18888); config = config.withRestPort(8080); Server server = Server.create(config); server.launch(); ``` ## Performance Comparison Approximate latency for a simple query operation: | Client Type | Typical Latency | Throughput | |-------------|----------------|------------| | HTTP Client | 10-50ms | ~100 ops/sec | | Binary Client | 5-20ms | ~500 ops/sec | | Local Peer | <1ms | ~10,000 ops/sec | **Note**: Actual performance depends on: - Network conditions - Hardware specifications - Peer load and configuration - Operation complexity - Concurrent usage patterns ## Switching Between Client Types You can easily switch between client types by changing only the connection code. The rest of your application code remains identical: ```java // Option 1: HTTP Convex convex = Convex.connect("https://mikera1337-convex-testnet.hf.space"); // Option 2: Binary Protocol (same functionality) InetSocketAddress addr = InetSocketAddress.createUnresolved("mikera1337-convex-testnet.hf.space", 18888); Convex convex = Convex.connect(addr); // Option 3: Local Peer (same functionality) Server server = Server.create(); server.launch(); Convex convex = Convex.connect(server); // All three support the same API: Result result = convex.query(Reader.read("(balance #13)")).get(); ``` ## Best Practices ### Development and Testing **Recommended**: Use a **local peer** for best development experience: ```java // Local peer: fastest, full control, built-in faucet Server server = Server.create(); server.launch(); Convex convex = Convex.connect(server); ``` **Alternative**: Use a **public testnet** if you don't want to run a local peer: ```java // Public testnet: good for learning, has faucet Convex convex = Convex.connect("https://mikera1337-convex-testnet.hf.space"); ``` ### Production For production applications, connect to production peers: ```java // Production: use peer.convex.live or your own peer infrastructure Convex convex = Convex.connect("https://peer.convex.live"); // Or binary protocol for better performance InetSocketAddress peerAddress = InetSocketAddress.createUnresolved( "peer.convex.live", 18888 ); Convex convex = Convex.connect(peerAddress); ``` ### Testing Use local peer for unit and integration tests: ```java import org.junit.jupiter.api.*; class MyConvexTest { private static Server server; private Convex convex; @BeforeAll static void startPeer() throws Exception { server = Server.create(); server.launch(); } @BeforeEach void connect() throws Exception { convex = Convex.connect(server); } @AfterEach void disconnect() throws Exception { convex.close(); } @AfterAll static void stopPeer() throws Exception { server.shutdown(); } @Test void testQuery() throws Exception { Result result = convex.query(Reader.read("(+ 1 2 3)")).get(); assertEquals(6L, result.getValue()); } } ``` ## Connection Configuration ### Timeouts Configure connection and request timeouts: ```java import convex.api.Convex; import java.time.Duration; Convex convex = Convex.connect("https://mikera1337-convex-testnet.hf.space"); // Set timeout for operations convex.setTimeout(Duration.ofSeconds(30)); ``` ### Connection Pooling For high-concurrency applications, consider maintaining a pool of connections: ```java import java.util.concurrent.ConcurrentLinkedQueue; class ConvexConnectionPool { private final ConcurrentLinkedQueue pool = new ConcurrentLinkedQueue<>(); private final String endpoint; public ConvexConnectionPool(String endpoint) { this.endpoint = endpoint; } public Convex acquire() throws Exception { Convex convex = pool.poll(); if (convex == null) { convex = Convex.connect(endpoint); } return convex; } public void release(Convex convex) { pool.offer(convex); } } ``` ## Troubleshooting ### HTTP Client Issues **Connection Refused**: - Verify peer URL is correct - Check peer is running and accessible - Verify firewall/proxy settings ```bash # Test peer accessibility curl https://mikera1337-convex-testnet.hf.space/api/v1/query \ -H "Content-Type: application/json" \ -d '{"source":"(+ 1 2 3)"}' ``` ### Binary Client Issues **Connection Timeout**: - Verify peer hostname and port - Check TCP port 18888 is accessible - Some networks block non-HTTP ports **Peer Disconnection**: - Binary connections are persistent; implement reconnection logic - Monitor connection state and reconnect on failure ### Local Peer Issues **OutOfMemoryError**: - Increase JVM heap size: `-Xmx4g` - Local peers require significant memory **Port Already in Use**: - Another peer is already running on the same port - Configure peer with different port or stop existing peer ## Next Steps - **[Query Guide](queries)** - Learn how to read network state - **[Transaction Guide](transactions)** - Submit state-changing operations - **[Account Management](accounts)** - Manage keys and accounts ## Resources - **[Convex Peer Documentation](/docs/products/convex-peer)** - Running your own peer - **[Javadoc API Reference](https://javadoc.io/doc/world.convex/convex-java)** - Complete API documentation - **[GitHub Repository](https://github.com/Convex-Dev/convex)** - Source code (convex-java module) --- # Java SDK The official Java client library for interacting with the Convex decentralised lattice network. ## Overview The `convex-java` module provides a native JVM interface to the Convex network, enabling you to: - **Execute Queries** - Read network state without fees - **Submit Transactions** - Execute atomic state transitions with cryptographic signatures - **Manage Accounts** - Create and manage self-sovereign accounts with Ed25519 keys - **Deploy Actors** - Deploy and interact with smart contracts (actors) written in Convex Lisp - **Async Operations** - Non-blocking I/O with CompletableFuture :::note Which client? This guide uses the **CVM-typed client**, `convex.api.Convex` — you work with native CVM values (`ACell`, `Address`, `Result`), preserving full type fidelity. A lighter `convex.java.ConvexJSON` client (used in the [Quick Start](quickstart.md)) returns plain JSON maps and is convenient for quick scripts or REST-style access. For production JVM code, prefer the typed client. See [Clients](clients.md). ::: ## Installation ### Maven ```xml world.convex convex-java 0.8.8 ``` ### Gradle ```groovy implementation 'world.convex:convex-java:0.8.8' ``` ## Quick Example ```java import convex.api.Convex; import convex.core.Result; import convex.core.crypto.AKeyPair; import convex.core.cvm.Address; import convex.core.lang.Reader; // Connect to a public testnet Convex convex = Convex.connect("https://mikera1337-convex-testnet.hf.space"); // Execute a query Result result = convex.query(Reader.read("(balance #13)")).get(); System.out.println("Balance: " + result.getValue()); // Use your account AKeyPair keyPair = AKeyPair.generate(); convex.setKeyPair(keyPair); convex.setAddress(Address.create(1234)); // Submit a transaction (requires funded account) Result txResult = convex.transact(Reader.read("(def my-value 42)")).get(); System.out.println("Transaction result: " + txResult.getValue()); ``` **Note**: This example uses a public testnet with faucet support. For production use `https://peer.convex.live` (no faucet). For development, consider [running a local peer](java/clients#local-peer-client) for best performance. ## Key Features ### ☕ Native JVM Performance Pure Java implementation with zero overhead, leveraging the full power of the JVM: ```java // Native Java types Address address = Address.create(1234); AKeyPair keyPair = AKeyPair.generate(); ACell data = Vectors.of(1, 2, 3, 4, 5); ``` ### ⚡ Async with CompletableFuture Non-blocking operations using Java's CompletableFuture: ```java CompletableFuture future = convex.query( Reader.read("(balance #123)") ); future.thenAccept(result -> { System.out.println("Balance: " + result.getValue()); }); ``` ### 🔐 Ed25519 Key Management Full support for Ed25519 cryptographic keys: ```java // Generate new key pair AKeyPair keyPair = AKeyPair.generate(); // Create from seed AKeyPair keyPair = AKeyPair.create(seedBytes); // Export public key AccountKey publicKey = keyPair.getAccountKey(); ``` ### 🌐 Multiple Client Types Choose the right client for your use case: ```java // JSON API client (REST) Convex convex = Convex.connect("https://peer.convex.live"); // Direct peer connection (binary protocol) Convex convex = Convex.connect(InetSocketAddress.createUnresolved("peer.convex.live", 18888)); ``` ### 🔄 Transaction Management Automatic transaction preparation and signing: ```java // SDK handles preparation and signing ATransaction tx = Invoke.create(address, Reader.read("(+ 1 2)")); SignedData signed = convex.getKeyPair().signData(tx); Result result = convex.transact(signed).get(); ``` ## Copper and Convex Coins The Convex network uses **copper** as the smallest unit of currency. Like Bitcoin's satoshis or Ethereum's wei, copper allows for precise fractional amounts: **1 Convex Coin = 1,000,000,000 copper** All balance and transfer amounts in the API are in **copper**: ```java // Query balance in copper Result result = convex.query(Reader.read("(balance #123)")).get(); long balanceCopper = ((Long) result.getValue()).longValue(); // Convert to Convex Coins double balanceCoins = balanceCopper / 1_000_000_000.0; System.out.println("Balance: " + balanceCoins + " CVM"); // Transfer 0.1 Convex Coins (100 million copper) long amount = 100_000_000L; Result txResult = convex.transact( Reader.read("(transfer #456 " + amount + ")") ).get(); ``` ## Java Version Support - **Java 21+** required - **Java 21** recommended for optimal performance - Virtual threads support for improved concurrency ## Dependencies The Java SDK depends on: - `convex-core` - Core CVM and data structures - `convex-peer` - Peer networking (for direct connections) - Minimal external dependencies (self-contained) ## Thread Safety The Java SDK is designed for concurrent use: ```java // Same Convex instance can be used from multiple threads ExecutorService executor = Executors.newFixedThreadPool(10); for (int i = 0; i < 100; i++) { int id = i; executor.submit(() -> { Result result = convex.query( Reader.read("(balance #" + id + ")") ).get(); System.out.println("Account #" + id + ": " + result.getValue()); }); } ``` **Important**: Avoid executing transactions that use the same account from multiple threads, as each transaction requires incrementing a sequence number that may become mismatched. Queries are safe for concurrent use. ## Resources - **[Maven Central](https://search.maven.org/artifact/world.convex/convex-java)** - Releases and versions - **[Javadoc API Reference](https://javadoc.io/doc/world.convex/convex-java)** - Complete API documentation - **[GitHub Repository](https://github.com/Convex-Dev/convex)** - Source code (convex-java module) - **[Convex Documentation](https://docs.convex.world)** - Network and platform docs - **[Discord Community](https://discord.com/invite/xfYGq4CT7v)** - Get help and share ideas ## Next Steps - **[Quickstart Guide](java/quickstart)** - Build your first Java app on Convex - **[Query Guide](java/queries)** - Learn how to read network state - **[Transaction Guide](java/transactions)** - Submit state-changing operations - **[Account Management](java/accounts)** - Manage keys and accounts --- # Queries Queries are read-only operations that execute Convex Lisp code without modifying network state. ## Basic Query Pattern Execute Convex Lisp code as a read-only query: ```java import convex.api.Convex; import convex.core.Result; import convex.core.cvm.Address; import convex.core.lang.Reader; import java.util.concurrent.CompletableFuture; Convex convex = Convex.connect("https://peer.convex.live"); // Execute a query CompletableFuture future = convex.query( Reader.read("(+ 1 2 3)"), Address.create(1) // Execution context address ); Result result = future.get(); System.out.println("Result: " + result.getValue()); // 6 ``` ## Queries are Free Unlike transactions, queries: - **Don't consume juice** - completely free to execute - **Don't require balance** - work even with zero funds - **Don't need signatures** - read-only access - **Execute instantly** - no consensus delay ## Query Methods ### `query(ACell form, Address address)` Execute Convex Lisp source code as a read-only query. ```java ACell form = Reader.read("(balance #123)"); Address address = Address.create(1); // Context address CompletableFuture future = convex.query(form, address); Result result = future.get(); if (result.isError()) { System.err.println("Query failed: " + result.getErrorCode()); } else { System.out.println("Balance: " + result.getValue()); } ``` **Parameters:** - `form` (ACell): Convex Lisp expression to execute - `address` (Address): Account address for execution context **Returns:** `CompletableFuture` with query result ## Query Patterns ### Checking Balances ```java // Query any account's balance Address targetAddress = Address.create(123); Result result = convex.query( Reader.read("(balance " + targetAddress + ")"), Address.create(1) // Any valid address for context ).get(); if (!result.isError()) { long balanceCopper = ((Number) result.getValue()).longValue(); double balanceCoins = balanceCopper / 1_000_000_000.0; System.out.println("Balance: " + balanceCoins + " CVM"); } ``` ### Reading Contract State ```java // Query a deployed contract Address contractAddress = Address.create(789); Result result = convex.query( Reader.read("(call " + contractAddress + " (get-count))"), Address.create(1) ).get(); if (!result.isError()) { System.out.println("Contract count: " + result.getValue()); } ``` ### Evaluating Expressions ```java // Test Convex Lisp expressions Result result = convex.query( Reader.read("(map inc [1 2 3 4 5])"), Address.create(1) ).get(); System.out.println("Result: " + result.getValue()); // [2, 3, 4, 5, 6] ``` ### Reading Global State ```java // Access special variables (requires address context) Address myAddress = Address.create(1234); // Query my balance Result balance = convex.query( Reader.read("*balance*"), myAddress ).get(); // Query my address Result address = convex.query( Reader.read("*address*"), myAddress ).get(); // Query current timestamp Result timestamp = convex.query( Reader.read("*timestamp*"), myAddress ).get(); ``` ## Advanced Queries ### Multi-Step Queries Execute multiple expressions in one query using `do`: ```java String query = """ (do (def x 10) (def y 20) (+ x y)) """; Result result = convex.query( Reader.read(query), Address.create(1) ).get(); System.out.println("Result: " + result.getValue()); // 30 ``` ### Conditional Queries Use Lisp conditionals to query based on state: ```java Address myAddress = Address.create(1234); String query = """ (if (> *balance* 1000000) "Rich account" "Poor account") """; Result result = convex.query( Reader.read(query), myAddress ).get(); System.out.println(result.getValue()); ``` ### Async Query Patterns Handle queries asynchronously with CompletableFuture: ```java // Execute multiple queries in parallel CompletableFuture query1 = convex.query( Reader.read("(balance #100)"), Address.create(1) ); CompletableFuture query2 = convex.query( Reader.read("(balance #200)"), Address.create(1) ); // Wait for all queries to complete CompletableFuture.allOf(query1, query2).get(); System.out.println("Account #100: " + query1.get().getValue()); System.out.println("Account #200: " + query2.get().getValue()); ``` ### Query with Callbacks Process results asynchronously: ```java convex.query( Reader.read("(balance #123)"), Address.create(1) ).thenAccept(result -> { if (!result.isError()) { long balance = ((Number) result.getValue()).longValue(); System.out.println("Balance: " + balance); } }).exceptionally(ex -> { System.err.println("Query failed: " + ex.getMessage()); return null; }); ``` ## Error Handling Queries can fail due to syntax errors or runtime errors: ```java Result result = convex.query( Reader.read("(invalid-function)"), Address.create(1) ).get(); if (result.isError()) { String errorCode = result.getErrorCode().toString(); System.err.println("Query failed: " + errorCode); System.err.println("Message: " + result.getValue()); } ``` Common error codes: - `UNDECLARED` - Symbol not found - `CAST` - Type error - `ARITY` - Wrong number of arguments - `BOUNDS` - Index out of bounds - `NOBODY` - Account doesn't exist ## Result Handling Working with query results: ```java Result result = convex.query( Reader.read("(map inc [1 2 3])"), Address.create(1) ).get(); // Check if successful if (result.isError()) { System.err.println("Error: " + result.getErrorCode()); return; } // Get value (may need casting) ACell value = result.getValue(); // Check type and extract if (value instanceof AVector) { AVector vector = (AVector) value; System.out.println("Vector length: " + vector.count()); for (int i = 0; i < vector.count(); i++) { System.out.println(" [" + i + "]: " + vector.get(i)); } } ``` ## Best Practices ### 1. Query Before Transacting Test expressions with queries before submitting transactions: ```java // Test with query first (free) Result testResult = convex.query( Reader.read("(transfer #456 1000000)"), myAddress ).get(); if (!testResult.isError()) { // If successful, transact convex.transact(Reader.read("(transfer #456 1000000)")).get(); } ``` ### 2. Use Queries for Read-Only Data Never use transactions when queries suffice: ```java // ❌ BAD - wastes juice Result result = convex.transact( Reader.read("(balance #123)") ).get(); // ✅ GOOD - free query Result result = convex.query( Reader.read("(balance #123)"), Address.create(1) ).get(); ``` ### 3. Cache Query Results Query results don't change unless someone transacts: ```java // Cache contract metadata Result metadata = convex.query( Reader.read("(call " + contractAddress + " (get-metadata))"), Address.create(1) ).get(); // Reuse cached metadata until state changes AMap metadataMap = (AMap) metadata.getValue(); ``` ### 4. Handle Timeouts Set appropriate timeouts for queries: ```java try { Result result = convex.query( Reader.read("(complex-computation)"), Address.create(1) ).get(5, TimeUnit.SECONDS); // 5 second timeout } catch (TimeoutException e) { System.err.println("Query timed out"); } ``` ### 5. Batch Related Queries Use `do` to execute multiple queries in one request: ```java String batchQuery = """ (do (def balance *balance*) (def timestamp *timestamp*) {:balance balance :timestamp timestamp}) """; Result result = convex.query( Reader.read(batchQuery), myAddress ).get(); // Result is a map with both values AMap resultMap = (AMap) result.getValue(); ``` ## Performance Tips ### Minimize Query Complexity Complex queries take longer to execute: ```java // ❌ SLOW - iterates entire range Result result = convex.query( Reader.read("(reduce + (range 1000000))"), Address.create(1) ).get(); // ✅ FAST - simple arithmetic Result result = convex.query( Reader.read("(* 1000000 500000)"), Address.create(1) ).get(); ``` ### Use Parallel Queries Execute independent queries concurrently: ```java List
addresses = List.of( Address.create(100), Address.create(200), Address.create(300) ); List> futures = addresses.stream() .map(addr -> convex.query( Reader.read("(balance " + addr + ")"), Address.create(1) )) .toList(); // Wait for all to complete CompletableFuture.allOf(futures.toArray(new CompletableFuture[0])).get(); // Process results for (int i = 0; i < futures.size(); i++) { Result result = futures.get(i).get(); System.out.println("Account " + addresses.get(i) + ": " + result.getValue()); } ``` ## Working with Data Types ### Extracting Values ```java Result result = convex.query( Reader.read("(+ 1 2)"), Address.create(1) ).get(); // Numbers if (result.getValue() instanceof CVMLong) { CVMLong num = (CVMLong) result.getValue(); long value = num.longValue(); } // Strings if (result.getValue() instanceof AString) { AString str = (AString) result.getValue(); String value = str.toString(); } // Vectors if (result.getValue() instanceof AVector) { AVector vec = (AVector) result.getValue(); long count = vec.count(); } // Maps if (result.getValue() instanceof AMap) { AMap map = (AMap) result.getValue(); ACell value = map.get(Keywords.BALANCE); } ``` ## Next Steps - **[Transaction Guide](transactions)** - Learn how to modify network state - **[Account Management](accounts)** - Manage keys and accounts - **[Convex Lisp Guide](/docs/tutorial/convex-lisp)** - Learn the query language --- # Quickstart Build your first Java application on Convex in under 10 minutes. ## Network Options This quickstart uses a **public testnet** for simplicity. For serious development, consider: - **Public Testnet** - `https://mikera1337-convex-testnet.hf.space` (good for learning, has a faucet) - **Local Peer** - Run your own peer (fastest, full control) - see [Client Types](clients#local-peer-client) - **Production** - `https://peer.convex.live` (no faucet, requires a funded account) :::note Which client? This guide uses **`ConvexJSON`**, the REST client that talks to any Convex peer over HTTPS — the right choice for a remote testnet. For an embedded, in-process peer use the binary `convex.api.Convex` client instead (see [Client Types](clients)). ::: ## Step 1: Create a Maven Project Create a new Maven project with `pom.xml`: ```xml 4.0.0 com.example convex-demo 1.0-SNAPSHOT 21 21 UTF-8 world.convex convex-java 0.8.8 ``` Or with Gradle (`build.gradle`): ```groovy plugins { id 'java' } group = 'com.example' version = '1.0-SNAPSHOT' repositories { mavenCentral() } dependencies { implementation 'world.convex:convex-java:0.8.8' } java { toolchain { languageVersion = JavaLanguageVersion.of(21) } } ``` ## Step 2: Connect to the Network Create `src/main/java/com/example/HelloConvex.java`: ```java package com.example; import convex.java.ConvexJSON; public class HelloConvex { public static void main(String[] args) throws Exception { // Connect to a public testnet over the REST API ConvexJSON convex = ConvexJSON.connect("https://mikera1337-convex-testnet.hf.space"); System.out.println("Connected to Convex network"); } } ``` ## Step 3: Execute a Query Queries are read-only operations that don't require an account. The result is a JSON map; the computed value is under the `"value"` key: ```java package com.example; import java.util.Map; import convex.java.ConvexJSON; public class QueryExample { public static void main(String[] args) throws Exception { ConvexJSON convex = ConvexJSON.connect("https://mikera1337-convex-testnet.hf.space"); // Query an account balance Map result = convex.query("(balance #13)"); long balance = ((Number) result.get("value")).longValue(); System.out.println("Balance: " + (balance / 1_000_000_000.0) + " CVM"); } } ``` Run with: ```bash mvn compile exec:java -Dexec.mainClass=com.example.QueryExample ``` ## Step 4: Create a Funded Account On a test network, `useNewAccount` generates a key pair, creates an account, funds it from the faucet, and sets the connection to use it — all in one call (amount in copper): ```java package com.example; import convex.java.ConvexJSON; import convex.core.cvm.Address; import convex.core.crypto.AKeyPair; public class CreateAccount { public static void main(String[] args) throws Exception { ConvexJSON convex = ConvexJSON.connect("https://mikera1337-convex-testnet.hf.space"); // Create a faucet-funded account (up to 10,000,000 copper) and use it Address address = convex.useNewAccount(10_000_000); System.out.println("Account created: " + address); // The generated key pair signs your transactions — save it to reuse this account! AKeyPair keyPair = convex.getKeyPair(); System.out.println("Public key: " + keyPair.getAccountKey()); } } ``` :::note Production Faucets only work on test networks. On production, an existing funded account must create and fund yours. See the [Faucet Guide](/docs/tutorial/coins/faucet). ::: ## Step 5: Submit a Transaction Transactions modify network state and require a funded account (Step 4). They are signed locally and return the same JSON-map shape as queries: ```java package com.example; import java.util.Map; import convex.java.ConvexJSON; import convex.core.cvm.Address; public class TransactExample { public static void main(String[] args) throws Exception { ConvexJSON convex = ConvexJSON.connect("https://mikera1337-convex-testnet.hf.space"); // Create + use a funded account (see Step 4) Address address = convex.useNewAccount(10_000_000); // Store a value on-chain Map result = convex.transact("(def my-value 42)"); System.out.println("Transaction result: " + result.get("value")); // Read it back (free query) Map query = convex.query("my-value"); System.out.println("Stored: " + query.get("value")); } } ``` Transactions **modify state** and **consume juice**. They require a funded account, a key pair for signing (both handled by `useNewAccount` above), and sufficient balance for execution costs. ## Complete Example Here's everything together — query, create a funded account, transact, and read back: ```java package com.example; import java.util.Map; import convex.java.ConvexJSON; import convex.core.cvm.Address; public class CompleteExample { public static void main(String[] args) throws Exception { // Connect to the testnet ConvexJSON convex = ConvexJSON.connect("https://mikera1337-convex-testnet.hf.space"); System.out.println("✓ Connected to Convex"); // Free query — no account needed Map balance = convex.query("(balance #13)"); System.out.println("✓ Account #13 balance: " + balance.get("value") + " copper"); // Create a faucet-funded account and use it Address address = convex.useNewAccount(10_000_000); System.out.println("✓ Account created: " + address); // Execute a transaction Map tx = convex.transact("(def greeting \"Hello Convex!\")"); System.out.println("✓ Transaction result: " + tx.get("value")); // Query it back Map q = convex.query("greeting"); System.out.println("✓ Read back: " + q.get("value")); } } ``` Run it: ```bash mvn compile exec:java -Dexec.mainClass=com.example.CompleteExample ``` Expected output: ``` ✓ Connected to Convex ✓ Account #13 balance: ... copper ✓ Account created: #1234 ✓ Transaction result: Hello Convex! ✓ Read back: Hello Convex! ``` ## Next Steps - **[Query Guide](queries)** - Learn advanced query patterns - **[Transaction Guide](transactions)** - Explore transaction capabilities - **[Account Management](accounts)** - Manage keys and accounts - **[Convex Lisp Guide](/docs/tutorial/convex-lisp)** - Learn the smart contract language ## Troubleshooting ### ClassNotFoundException If you get `ClassNotFoundException` for Convex classes: ```bash # Verify dependency is resolved mvn dependency:tree | grep convex # Clean and rebuild mvn clean compile ``` ### Connection Refused If you cannot connect to `https://mikera1337-convex-testnet.hf.space`: - Check your internet connection - Verify the peer is online; try an alternative peer URL - Check firewall settings ### Transaction Fails with SEQUENCE Error If you get `SEQUENCE` errors: - Don't submit multiple transactions concurrently from the same account - Wait for the previous transaction to complete before submitting the next - Use separate accounts for concurrent transactions ### Insufficient Balance (`FUNDS`) If transactions fail with a `FUNDS` error, the account needs Convex Coins. On the testnet, create it with a faucet amount via `useNewAccount(...)` (Step 4); on production you must transfer in funds from an existing account. See the [Faucet Guide](/docs/tutorial/coins/faucet). --- # Transactions Transactions are state-changing operations that execute Convex Lisp code and modify the network state. ## Transaction Basics Unlike queries, transactions: - ⚡ **Modify state** - Change account balances, deploy contracts, update data - 💰 **Consume juice** - Require Convex Coins to pay for execution - 🔐 **Require signing** - Must be cryptographically signed by account key - ⏱️ **Achieve consensus** - Confirmed by network consensus (sub-second) - 📈 **Increment sequence** - Each account has a monotonic sequence number ## Basic Transaction Pattern ```java import convex.api.Convex; import convex.core.Result; import convex.core.crypto.AKeyPair; import convex.core.cvm.Address; import convex.core.lang.Reader; Convex convex = Convex.connect("https://peer.convex.live"); // Set up account AKeyPair keyPair = AKeyPair.generate(); convex.setKeyPair(keyPair); convex.setAddress(Address.create(1234)); // Execute a transaction Result result = convex.transact( Reader.read("(def my-value 42)") ).get(); if (result.isError()) { System.err.println("Transaction failed: " + result.getErrorCode()); } else { System.out.println("Result: " + result.getValue()); } ``` ## Transaction Methods ### `transact(ACell form)` Execute Convex Lisp source code as a state-changing transaction. ```java ACell form = Reader.read("(def x 10)"); CompletableFuture future = convex.transact(form); Result result = future.get(); if (result.isError()) { System.err.println("Error: " + result.getErrorCode()); } else { System.out.println("Success: " + result.getValue()); } ``` **Parameters:** - `form` (ACell): Convex Lisp expression to execute **Returns:** `CompletableFuture` with transaction result ### `transact(SignedData signed)` Submit a pre-signed transaction: ```java import convex.core.transactions.Invoke; import convex.core.data.SignedData; Address myAddress = convex.getAddress(); // Create transaction ATransaction tx = Invoke.create(myAddress, Reader.read("(def x 42)")); // Sign with key pair SignedData signed = convex.getKeyPair().signData(tx); // Submit Result result = convex.transact(signed).get(); ``` ## Transaction Costs Every transaction consumes **juice** based on: 1. **Memory usage** - Creating new data structures 2. **Computation** - CPU cycles for execution 3. **Storage** - Persisting data on-chain Example juice costs: ```java // Check balance before transaction Result balanceBefore = convex.query( Reader.read("(balance " + myAddress + ")"), myAddress ).get(); long before = ((Number) balanceBefore.getValue()).longValue(); // Execute transaction convex.transact(Reader.read("(def x [1 2 3 4 5])")).get(); // Calculate juice consumed Result balanceAfter = convex.query( Reader.read("(balance " + myAddress + ")"), myAddress ).get(); long after = ((Number) balanceAfter.getValue()).longValue(); long juiceConsumed = before - after; System.out.println("Juice consumed: " + juiceConsumed + " copper"); ``` Typical costs: - **Simple expression**: 1,000 - 5,000 copper - **Transfer**: 2,000 - 3,000 copper - **Define variable**: 1,500 - 10,000 copper - **Deploy contract**: 50,000 - 500,000 copper ## Transaction Patterns ### Defining Variables Store data in your account's environment: ```java // Define a simple value convex.transact(Reader.read("(def my-number 42)")).get(); // Define a collection convex.transact(Reader.read("(def my-list [1 2 3 4 5])")).get(); // Define a map convex.transact(Reader.read("(def my-map {:name \"Alice\" :age 30})")).get(); // Read back with a query Result result = convex.query( Reader.read("my-number"), myAddress ).get(); System.out.println("Value: " + result.getValue()); // 42 ``` ### Transferring Funds ```java // Transfer 0.05 CVM (50 million copper) Address recipientAddress = Address.create(456); long amount = 50_000_000L; Result result = convex.transact( Reader.read("(transfer " + recipientAddress + " " + amount + ")") ).get(); if (!result.isError()) { System.out.println("Transferred " + (amount / 1_000_000_000.0) + " CVM"); } else { System.err.println("Transfer failed: " + result.getErrorCode()); } ``` ### Deploying Smart Contracts Deploy an actor (smart contract) using `deploy`: ```java String contractSource = """ (deploy (do (def count 0) (defn increment [] (def count (inc count)) count) (defn get-count [] count))) """; Result result = convex.transact(Reader.read(contractSource)).get(); if (!result.isError()) { Address contractAddress = (Address) result.getValue(); System.out.println("Contract deployed at: " + contractAddress); } else { System.err.println("Deployment failed: " + result.getErrorCode()); } ``` ### Calling Deployed Contracts Interact with deployed actors: ```java Address contractAddress = Address.create(789); // Call actor function Result result = convex.transact( Reader.read("(call " + contractAddress + " (increment))") ).get(); if (!result.isError()) { System.out.println("New count: " + result.getValue()); } // Query actor state (free) Result queryResult = convex.query( Reader.read("(call " + contractAddress + " (get-count))"), myAddress ).get(); System.out.println("Current count: " + queryResult.getValue()); ``` ### Multi-Step Transactions Execute multiple operations atomically: ```java String transaction = """ (do (def x 10) (def y 20) (def sum (+ x y)) (transfer #456 sum) sum) """; Result result = convex.transact(Reader.read(transaction)).get(); if (!result.isError()) { System.out.println("Transferred " + result.getValue() + " copper"); } ``` If any step fails, the entire transaction is rolled back. ## Sequence Numbers Every transaction increments the account's sequence number: ```java // Get account info with sequence Result info = convex.query( Reader.read("(get-account " + myAddress + ")"), Address.create(1) ).get(); // The SDK automatically manages sequence numbers // You rarely need to handle them manually ``` ## Error Handling Transactions can fail for various reasons: ```java Result result = convex.transact( Reader.read("(transfer #999999 1000000000000)") ).get(); if (result.isError()) { String errorCode = result.getErrorCode().toString(); switch (errorCode) { case "FUNDS": System.err.println("Insufficient balance"); break; case "NOBODY": System.err.println("Recipient account does not exist"); break; case "SEQUENCE": System.err.println("Sequence error (rare with single-threaded use)"); break; case "CAST": System.err.println("Type error in transaction"); break; default: System.err.println("Transaction failed: " + errorCode); System.err.println("Message: " + result.getValue()); } } ``` Common error codes: | Code | Meaning | Solution | |------|---------|----------| | `FUNDS` | Insufficient balance | Request more funds or reduce amount | | `NOBODY` | Account doesn't exist | Create recipient account first | | `SEQUENCE` | Sequence mismatch | Avoid concurrent transactions on same account | | `CAST` | Type error | Fix Lisp expression | | `UNDECLARED` | Symbol not found | Check variable/function names | | `ARGUMENT` | Invalid argument | Check function parameters | ## Advanced Patterns ### Async Transaction Handling Execute transactions asynchronously: ```java // Submit transaction without blocking CompletableFuture future = convex.transact( Reader.read("(def x 42)") ); // Do other work while transaction processes System.out.println("Transaction submitted..."); // Handle result when ready future.thenAccept(result -> { if (!result.isError()) { System.out.println("Transaction successful: " + result.getValue()); } else { System.err.println("Transaction failed: " + result.getErrorCode()); } }).exceptionally(ex -> { System.err.println("Exception: " + ex.getMessage()); return null; }); ``` ### Batch Transactions Submit multiple transactions sequentially: ```java List transactions = List.of( "(def x 10)", "(def y 20)", "(def sum (+ x y))" ); for (String tx : transactions) { Result result = convex.transact(Reader.read(tx)).get(); if (result.isError()) { System.err.println("Transaction failed: " + result.getErrorCode()); break; // Stop on first error } System.out.println("Success: " + result.getValue()); } ``` ### Conditional Transactions Execute different logic based on conditions: ```java String transaction = """ (if (> *balance* 1000000) (transfer #456 500000) :insufficient-funds) """; Result result = convex.transact(Reader.read(transaction)).get(); if (!result.isError()) { System.out.println("Result: " + result.getValue()); } ``` ### Setting Account Keys Transfer account control to a new key pair: ```java // Generate new keys AKeyPair newKeys = AKeyPair.generate(); // Set the new public key on-chain String transaction = "(set-key " + newKeys.getAccountKey() + ")"; Result result = convex.transact(Reader.read(transaction)).get(); if (!result.isError()) { // Update local key pair convex.setKeyPair(newKeys); System.out.println("Account keys updated"); } ``` **⚠️ Warning**: Save the new keys before changing! Losing keys means losing account access. ## Best Practices ### 1. Query Before Transacting Test transactions with queries first: ```java // Test with query (free) Result testResult = convex.query( Reader.read("(transfer #456 1000000)"), myAddress ).get(); if (testResult.isError()) { System.err.println("Would fail: " + testResult.getErrorCode()); } else { // Proceed with transaction convex.transact(Reader.read("(transfer #456 1000000)")).get(); } ``` ### 2. Check Balance First Verify sufficient funds before transacting: ```java // Get current balance Result balanceResult = convex.query( Reader.read("(balance " + myAddress + ")"), myAddress ).get(); long balance = ((Number) balanceResult.getValue()).longValue(); long transferAmount = 50_000_000L; if (balance < transferAmount + 10_000) { // Extra for juice System.err.println("Insufficient balance"); } else { convex.transact( Reader.read("(transfer #456 " + transferAmount + ")") ).get(); } ``` ### 3. Handle Failures Gracefully Always handle potential errors: ```java public Result safeTransact(Convex convex, String transaction) { try { Result result = convex.transact(Reader.read(transaction)).get(); if (result.isError()) { logger.error("Transaction failed: {}", result.getErrorCode()); return null; } return result; } catch (Exception e) { logger.error("Exception during transaction", e); return null; } } ``` ### 4. Use Atomic Transactions Group related operations in a single transaction: ```java // ❌ BAD - Two separate transactions convex.transact(Reader.read("(def x 10)")).get(); convex.transact(Reader.read("(def y (* x 2))")).get(); // Might fail // ✅ GOOD - One atomic transaction convex.transact(Reader.read("(do (def x 10) (def y (* x 2)))")).get(); ``` ### 5. Avoid Concurrent Transactions Don't submit multiple transactions from the same account concurrently: ```java // ❌ BAD - Concurrent transactions on same account CompletableFuture tx1 = convex.transact(Reader.read("(def x 1)")); CompletableFuture tx2 = convex.transact(Reader.read("(def y 2)")); // May cause SEQUENCE errors // ✅ GOOD - Sequential transactions convex.transact(Reader.read("(def x 1)")).get(); convex.transact(Reader.read("(def y 2)")).get(); // OR use one atomic transaction convex.transact(Reader.read("(do (def x 1) (def y 2))")).get(); ``` ### 6. Keep Juice Costs Low Minimize memory allocation and computation: ```java // ❌ EXPENSIVE - Creates large data structure convex.transact(Reader.read("(def big-list (range 1000000))")).get(); // ✅ CHEAPER - Store minimal data convex.transact(Reader.read("(def count 1000000)")).get(); ``` ## Transaction Lifecycle 1. **Prepare** - SDK creates transaction with source and account address 2. **Sign** - Account's private key signs the transaction hash 3. **Submit** - Signed transaction sent to peer 4. **Consensus** - Network reaches consensus (typically < 1 second) 5. **Execute** - CVM executes transaction 6. **Result** - Result returned to client ```java // All handled automatically by SDK Result result = convex.transact(Reader.read("(def x 42)")).get(); // But you can handle each step manually if needed ATransaction tx = Invoke.create(myAddress, Reader.read("(def x 42)")); SignedData signed = keyPair.signData(tx); Result result = convex.transact(signed).get(); ``` ## Monitoring Transactions ### Check Transaction Status ```java Result result = convex.transact(Reader.read("(def x 42)")).get(); if (!result.isError()) { System.out.println("Transaction ID: " + result.getID()); System.out.println("Result value: " + result.getValue()); } else { System.err.println("Error code: " + result.getErrorCode()); System.err.println("Error message: " + result.getValue()); } ``` ### Transaction Logging ```java import org.slf4j.Logger; import org.slf4j.LoggerFactory; private static final Logger logger = LoggerFactory.getLogger(MyClass.class); public void executeTransaction(String transaction) { logger.info("Submitting transaction: {}", transaction); Result result = convex.transact(Reader.read(transaction)).get(); if (result.isError()) { logger.error("Transaction failed: {} - {}", result.getErrorCode(), result.getValue()); } else { logger.info("Transaction successful: {}", result.getValue()); } } ``` ## Next Steps - **[Account Management](accounts)** - Manage keys and accounts - **[Convex Lisp Guide](/docs/tutorial/convex-lisp)** - Learn the smart contract language - **[Actor Development](/docs/tutorial/actors)** - Build smart contracts --- # Using Convex from JavaScript The Convex client ships as a single package — `@convex-world/convex-ts` — that works equally in TypeScript and plain JavaScript. This page shows idiomatic JavaScript usage; the full reference lives in the [TypeScript / JavaScript SDK](/docs/tutorial/client-sdks/typescript) docs. ## Installation ```bash npm install @convex-world/convex-ts ``` :::tip TypeScript Package The package name is `@convex-world/convex-ts` but it works perfectly in plain JavaScript projects. The TypeScript definitions are optional. ::: ## Quick Start ### ES Modules (Recommended) ```javascript import { Convex, KeyPair } from '@convex-world/convex-ts'; // Connect and query const convex = new Convex('https://peer.convex.live'); const result = await convex.query('(balance #13)'); console.log('Balance:', result.value); ``` ### CommonJS ```javascript const { Convex, KeyPair } = require('@convex-world/convex-ts'); async function main() { const convex = new Convex('https://peer.convex.live'); const result = await convex.query('(balance #13)'); console.log('Balance:', result.value); } main().catch(console.error); ``` ## Using Your Account ```javascript import { Convex, KeyPair } from '@convex-world/convex-ts'; const convex = new Convex('https://peer.convex.live'); // Load your key pair const keyPair = KeyPair.fromSeed(process.env.CONVEX_SEED); convex.setAccount('#1678', keyPair); // Check balance const info = await convex.getAccountInfo(); console.log('Balance:', info.balance / 1_000_000_000, 'Convex Coins'); // Transfer coins const result = await convex.transfer('#456', 1_000_000_000); console.log('Transaction:', result.hash); ``` ## Common Patterns ### Query Multiple Accounts ```javascript const addresses = ['#9', '#10', '#11']; for (const addr of addresses) { const result = await convex.query(`(balance ${addr})`); console.log(`${addr}: ${result.value} copper`); } ``` ### Execute Convex Lisp ```javascript // Deploy a function const deploy = await convex.transact(` (def greet (fn [name] (str "Hello, " name "!"))) `); // Call it const result = await convex.transact('(greet "World")'); console.log(result.result); // "Hello, World!" ``` ### Error Handling ```javascript try { const result = await convex.transfer('#456', 1_000_000_000); if (result.status === 'success') { console.log('✅ Success!'); } else { console.error('❌ Failed:', result.error); } } catch (error) { console.error('Network error:', error.message); } ``` ## Package.json Setup ### ES Modules Add to your `package.json`: ```json { "type": "module" } ``` ### Running Scripts ```bash # Node.js 18+ node script.js # Or with tsx (handles both JS and TS) npx tsx script.js ``` ## Complete Example ```javascript import { Convex, KeyPair } from '@convex-world/convex-ts'; async function main() { // Connect const convex = new Convex('https://peer.convex.live'); // Query (no account needed) console.log('=== Query ==='); const balance = await convex.query('(balance #9)'); console.log('Convex Foundation:', balance.value, 'copper'); // Use your account console.log('\n=== Account ==='); const keyPair = KeyPair.fromSeed(process.env.CONVEX_SEED); convex.setAccount(process.env.CONVEX_ADDRESS, keyPair); const info = await convex.getAccountInfo(); console.log('My balance:', info.balance / 1_000_000_000, 'coins'); // Transaction console.log('\n=== Transaction ==='); const result = await convex.transact('(def my-data "Hello from JavaScript!")'); console.log('Transaction:', result.hash); } main().catch(console.error); ``` Run it: ```bash CONVEX_SEED=your-seed CONVEX_ADDRESS=#1678 node app.js ``` ## Key Differences from TypeScript ### No Type Annotations ```javascript // JavaScript - no types const convex = new Convex('https://peer.convex.live'); const result = await convex.query('(balance #13)'); // TypeScript - with types const convex: Convex = new Convex('https://peer.convex.live'); const result: Result = await convex.query('(balance #13)'); ``` ### Runtime Validation Without TypeScript, validate at runtime: ```javascript function validateConfig(config) { if (!config.seed) { throw new Error('Missing seed'); } if (!config.address) { throw new Error('Missing address'); } return config; } const config = validateConfig({ seed: process.env.CONVEX_SEED, address: process.env.CONVEX_ADDRESS }); ``` ### JSDoc for IntelliSense Get autocomplete without TypeScript: ```javascript /** * @param {string} address - Convex address * @param {number} amount - Amount in copper * @returns {Promise} */ async function transfer(address, amount) { return await convex.transfer(address, amount); } ``` ## Browser Usage ### With Module Bundler ```javascript import { Convex } from '@convex-world/convex-ts'; const convex = new Convex('https://peer.convex.live'); document.getElementById('queryBtn').addEventListener('click', async () => { const result = await convex.query('(balance #9)'); document.getElementById('result').textContent = result.value; }); ``` ### With CDN (Not Recommended) For quick prototypes only: ```html ``` ## React (JavaScript) ```javascript import { useState, useEffect } from 'react'; import { Convex } from '@convex-world/convex-ts'; function ConvexBalance() { const [balance, setBalance] = useState(null); useEffect(() => { const convex = new Convex('https://peer.convex.live'); convex.query('(balance #9)').then(result => { setBalance(result.value); }); }, []); return (
Balance: {balance ? `${balance} copper` : 'Loading...'}
); } ``` ## Further Reading For detailed documentation, see the [TypeScript guide](/docs/tutorial/client-sdks/typescript). All concepts apply to JavaScript - just ignore the type annotations. **Key topics:** - [Queries](/docs/tutorial/client-sdks/typescript/queries) - Reading network state - [Transactions](/docs/tutorial/client-sdks/typescript/transactions) - Modifying state - [Accounts](/docs/tutorial/client-sdks/typescript/accounts) - Key pair management - [Signers](/docs/tutorial/client-sdks/typescript/signers) - Hardware wallet integration --- # Account Management Learn how to create, manage, and secure Convex accounts using key pairs and the Convex Name Service. ## Key Concepts ### Account An **Account** represents an identity on the Convex network with: - **Address** - Unique numeric identifier (e.g., `#1234`) - **Key Pair** - Ed25519 cryptographic keys for signing - **Balance** - Convex Coins (copper) owned by the account - **Sequence** - Transaction counter (starts at 0) - **Environment** - Variables and functions defined by the account ### KeyPair A **KeyPair** contains Ed25519 cryptographic keys: - **Public Key** - Shared with the network (64 hex characters) - **Private Key** - Secret, used for signing transactions **⚠️ Never share your private key!** Anyone with it can control your account. ## Creating Key Pairs ### Generate New Keys Create a new random key pair: ```python from convex_sdk import KeyPair # Generate fresh Ed25519 keys key_pair = KeyPair() print(f'Public key: {key_pair.public_key}') # Example: 0x36d8c5c40dbe2d1b0131acf41c38b9d37ebe04d85... ``` ### Import from File Load keys from an encrypted PEM file: ```python # Import keys (requires password) key_pair = KeyPair.import_from_file('my_keys.pem', 'secret_password') print(f'Imported public key: {key_pair.public_key}') ``` ### Import from Mnemonic Restore keys from a BIP39 mnemonic phrase: ```python # Import from 24-word phrase mnemonic = 'witch collapse practice feed shame open despair creek road again ice least' key_pair = KeyPair.import_from_mnemonic(mnemonic) print(f'Restored public key: {key_pair.public_key}') ``` ### Import from Text Load keys from a PEM-formatted string: ```python pem_text = """-----BEGIN ENCRYPTED PRIVATE KEY----- MIGbMFcGCSqGSIb3DQEFDTBKMCkGCSqGSIb3DQEFDDAcBAi3qm1zgjCO5gICCAAw ... -----END ENCRYPTED PRIVATE KEY-----""" key_pair = KeyPair.import_from_text(pem_text, 'secret_password') ``` ## Exporting Keys ### Export to File Save keys to an encrypted PEM file: ```python key_pair = KeyPair() # Encrypt with password and save key_pair.export_to_file('backup.pem', 'strong_password_123') print('Keys saved to backup.pem') ``` **Best Practices:** - Use a strong, unique password - Store file in a secure location - Make multiple backups - Never commit to version control ### Export to Text Get keys as an encrypted PEM string: ```python pem_text = key_pair.export_to_text('my_password') print(pem_text) # -----BEGIN ENCRYPTED PRIVATE KEY----- # MIGbMFcGCSqGSIb3DQEFDTBKMCkGCSqGSIb3DQEFDDA... ``` ### Export as Mnemonic Get a BIP39 mnemonic phrase: ```python # Get 24-word recovery phrase mnemonic = key_pair.export_to_mnemonic print(f'Recovery phrase: {mnemonic}') # witch collapse practice feed shame open despair creek road again ice least ... ``` **⚠️ Warning**: Store mnemonic securely! Anyone with it can restore your keys. ## Creating Accounts ### Create New Account Register a new account address on the network: ```python from convex_sdk import Convex, KeyPair convex = Convex('https://peer.convex.live') # Generate keys and create account key_pair = KeyPair() account = convex.create_account(key_pair) print(f'Account created: #{account.address}') print(f'Public key: {account.key_pair.public_key}') ``` ### Load Existing Account Reconstruct an account from saved keys and known address: ```python from convex_sdk import Account, KeyPair # Import keys key_pair = KeyPair.import_from_file('my_keys.pem', 'password') # Create Account object with known address account = Account(key_pair, address=1234) print(f'Loaded account: #{account.address}') ``` ### Setup Account with Name Create or load an account with a registered name: ```python # Import keys key_pair = KeyPair.import_from_file('alice_keys.pem', 'password') # Creates new account or loads existing if name is registered account = convex.setup_account('alice', key_pair) print(f'Account: {account.name} at #{account.address}') ``` This method: 1. Checks if name exists in CNS 2. If exists, loads the account 3. If not, creates new account and registers name ## Account Properties ```python account = convex.create_account(key_pair) # Address (int) print(f'Address: #{account.address}') # Public key (bytes) print(f'Public key bytes: {account.public_key}') # Key pair object print(f'Key pair: {account.key_pair}') # Name (if registered) print(f'Name: {account.name}') # None if not registered ``` ## Account Names (CNS) The **Convex Name Service (CNS)** allows human-readable names for accounts. ### Register Account Name Register a name in the format `account.`: ```python # Create account key_pair = KeyPair() account = convex.create_account(key_pair) # Fund it for registration convex.request_funds(100_000_000, account) # Register name (costs juice) account = convex.register_account_name('alice', account) print(f'Registered: account.alice → #{account.address}') ``` ### Resolve Account Names Convert names to addresses: ```python # Resolve account name address = convex.resolve_account_name('alice') print(f'account.alice is at #{address}') # Returns None if not found address = convex.resolve_account_name('unknown') print(address) # None ``` ### Load Named Account Load an account by its registered name: ```python key_pair = KeyPair.import_from_file('alice_keys.pem', 'password') # Load account by name account = convex.load_account('alice', key_pair) if account: print(f'Loaded: {account.name} at #{account.address}') else: print('Account name not registered') ``` ## Signing Transactions Accounts sign transactions using their private keys: ```python # SDK handles signing automatically result = convex.transact('(def x 42)', account) # Internally: # 1. SDK prepares transaction # 2. Account signs hash with private key # 3. SDK submits signed transaction ``` ### Manual Signing For advanced use cases, manually sign data: ```python # Sign a hash manually hash_hex = '7e2f1062f5fc51ed65a28b5945b49425aa42df6b7e67107efec357794096e05e' signature = account.sign(hash_hex) print(f'Signature: {signature}') ``` ## Transferring Account Control Change the key pair that controls an account: ```python # Generate new keys new_keys = KeyPair() # Create account to transfer TO new_account = Account(new_keys, account.address) # Transfer control (changes on-chain key) transferred = convex.transfer_account(new_account, account) print(f'Account #{transferred.address} now uses new keys') print(f'New public key: {transferred.key_pair.public_key}') ``` **⚠️ Warning**: - Save new keys BEFORE transferring! - Old keys will no longer work - Losing new keys means losing account access forever ## Key Security Best Practices ### 1. Store Keys Securely ```python # ✅ GOOD - Encrypted PEM file key_pair.export_to_file('/secure/path/keys.pem', 'strong_password') # ✅ GOOD - Hardware wallet (requires custom integration) # Store keys on hardware device, never expose private key # ❌ BAD - Plaintext file # Don't do this! # ❌ BAD - In code api_key = '0x123abc...' # Don't hardcode keys ``` ### 2. Use Strong Passwords ```python # ❌ BAD - Weak password key_pair.export_to_file('keys.pem', 'password123') # ✅ GOOD - Strong password import secrets password = secrets.token_urlsafe(32) key_pair.export_to_file('keys.pem', password) print(f'Save this password: {password}') ``` ### 3. Make Backups ```python key_pair = KeyPair() # Export multiple formats key_pair.export_to_file('primary_backup.pem', password) key_pair.export_to_file('secondary_backup.pem', password) mnemonic = key_pair.export_to_mnemonic print(f'Write down recovery phrase: {mnemonic}') ``` ### 4. Separate Hot and Cold Wallets ```python # Cold wallet - offline, secure storage cold_key_pair = KeyPair() cold_key_pair.export_to_file('/offline/storage/cold.pem', password) # Hot wallet - online, small balance hot_key_pair = KeyPair() hot_account = convex.create_account(hot_key_pair) convex.request_funds(100_000_000, hot_account) # Only 0.1 CVM # Transfer bulk funds to cold storage when not needed ``` ### 5. Never Share Private Keys ```python # ✅ GOOD - Share public key print(f'My public key: {key_pair.public_key}') # ✅ GOOD - Share address print(f'Send funds to: #{account.address}') # ❌ BAD - Never do this! # print(f'My private key: ...') # NEVER! ``` ## Account Lifecycle ```python from convex_sdk import Convex, KeyPair, Account convex = Convex('https://peer.convex.live') # 1. Generate keys key_pair = KeyPair() # 2. Create account on network account = convex.create_account(key_pair) # 3. Fund account convex.request_funds(100_000_000, account) # 4. Register name (optional) convex.register_account_name('alice', account) # 5. Backup keys key_pair.export_to_file('alice_backup.pem', 'password') # 6. Use account convex.transact('(def x 42)', account) # 7. Later: restore from backup restored_keys = KeyPair.import_from_file('alice_backup.pem', 'password') restored_account = convex.load_account('alice', restored_keys) ``` ## Common Patterns ### Development Account Quick account for testing: ```python def create_dev_account(convex): key_pair = KeyPair() account = convex.create_account(key_pair) convex.request_funds(1_000_000_000, account) # 1 CVM return account # Use in development convex = Convex('https://peer.convex.live') dev_account = create_dev_account(convex) ``` ### Production Account Secure account for production: ```python import os from pathlib import Path def load_production_account(convex, name): # Load from secure environment key_file = Path(os.environ['CONVEX_KEY_PATH']) password = os.environ['CONVEX_KEY_PASSWORD'] key_pair = KeyPair.import_from_file(str(key_file), password) account = convex.load_account(name, key_pair) return account # Use in production account = load_production_account(convex, 'prod_service') ``` ### Multi-Signature Pattern Require multiple signatures (requires custom contract): ```python # Deploy multi-sig contract multisig_source = """ (deploy (do (def signers #{#123 #456 #789}) (def threshold 2) (defn execute [action signatures] (if (>= (count signatures) threshold) (eval action) :insufficient-signatures)))) """ result = convex.transact(multisig_source, account) multisig_address = result['value'] ``` ## Next Steps - **[Query Guide](queries)** - Learn how to read network state - **[Transaction Guide](transactions)** - Execute state-changing operations - **[Convex Lisp Guide](/docs/tutorial/convex-lisp)** - Learn the smart contract language - **[Recipes](/docs/tutorial/recipes)** - Common account patterns and examples --- # Assets Convex has no special "token" primitive — fungible tokens and named assets are ordinary [actors](/docs/cad/fungible) you deploy and call. This guide issues and moves a fungible token, and registers names with the Convex Name System (CNS), from Python. ## Fungible tokens Fungible tokens follow the [CAD029](/docs/cad/fungible) standard and are created with the built-in `convex.fungible` library. ### Deploy a token ```python from convex_sdk import Convex, KeyPair convex = Convex('https://mikera1337-convex-testnet.hf.space') account = convex.create_account(KeyPair()) convex.request_funds(100_000_000, account) # 0.1 CVM for juice # Deploy a token with an initial supply of 1,000,000 units, held by the deployer result = convex.transact('(deploy [(@convex.fungible/build-token {:supply 1000000})])', account) token = result['value'] # the token's actor address (an integer) print(f'Deployed token at #{token}') ``` ### Check a balance Pass the querying account as the second argument to `query`: ```python balance = convex.query(f'(@convex.fungible/balance #{token} #{account.address})', account)['value'] print(f'Balance: {balance}') # 1000000 ``` ### Transfer tokens The generic `convex.asset` library moves any asset. The amount is paired with the token as `[token amount]`: ```python recipient = convex.create_account(KeyPair()) convex.transact(f'(@convex.asset/transfer #{recipient.address} [#{token} 1000])', account) ``` ### Total supply ```python supply = convex.query(f'(@convex.fungible/total-supply #{token})', account)['value'] ``` ## CNS names The Convex Name System maps human-readable names to addresses. Account names are registered under the `account.` namespace. ```python # Register account.my-service -> your account address convex.register_account_name('my-service', account.address, account) # Resolve a name back to an address (returns an int, or None if unregistered) address = convex.resolve_account_name('my-service') ``` `resolve_account_name(name)` resolves the `account.` shorthand; `resolve_name(name)` resolves a full CNS path. ## See Also - [CAD029: Fungible Token Standard](/docs/cad/fungible) - [CAD014: Convex Name System](/docs/cad/cns) - [Transactions](transactions.md) — submitting and confirming transactions - [Managing Coins recipe](../../recipes/managing-coins/index.md) --- # Python SDK The official Python client library for interacting with the Convex decentralised lattice network. ## Overview The `convex-sdk` package provides a Pythonic interface to the Convex network, enabling you to: - **Execute Queries** - Read network state without fees - **Submit Transactions** - Execute atomic state transitions with cryptographic signatures - **Manage Accounts** - Create and manage self-sovereign accounts with Ed25519 keys - **Transfer Funds** - Send Convex Coins between accounts - **Deploy Contracts** - Deploy and interact with smart contracts written in Convex Lisp ## Installation Install via pip: ```bash pip install convex-sdk ``` Or with Poetry: ```bash poetry add convex-sdk ``` ## Quick Example ```python from convex_sdk import Convex, KeyPair # Connect to the network convex = Convex('https://peer.convex.live') # Create a new account key_pair = KeyPair() account = convex.create_account(key_pair) # Request test funds convex.request_funds(100_000_000, account) # Query your balance balance = convex.get_balance(account) print(f'Balance: {balance / 1_000_000_000} Convex Coins') # Transfer funds convex.transfer('#456', 50_000_000, account) ``` ## Key Features ### 🐍 Pythonic API Idiomatic Python with `snake_case` naming, exceptions for errors, and synchronous I/O patterns. ```python # Pythonic method names balance = convex.get_balance(account) info = convex.get_account_info(account) result = convex.transact('(map inc [1 2 3 4])', account) ``` ### 📦 Account Objects Encapsulate address, key pair, and optional name in a convenient `Account` object: ```python from convex_sdk import Account, KeyPair key_pair = KeyPair() account = Account(key_pair, address=1234, name='my_account') print(f'Address: {account.address}') print(f'Public Key: {account.public_key}') ``` ### 🔐 Flexible Key Management Multiple ways to create and import key pairs: ```python # Generate new keys key_pair = KeyPair() # Import from encrypted file key_pair = KeyPair.import_from_file('my_keys.pem', 'secret_password') # Import from mnemonic phrase key_pair = KeyPair.import_from_mnemonic('word1 word2 word3 ...') # Export for later use key_pair.export_to_file('backup.pem', 'secret_password') ``` ### 🔄 Automatic Sequence Retry The SDK automatically retries transactions with sequence errors, handling concurrency gracefully: ```python # Automatically retries up to 20 times on sequence conflicts result = convex.transact('(transfer #789 1000000)', account) ``` ### 🌐 CNS Integration Built-in support for Convex Name Service (CNS): ```python # Register an account name convex.register_account_name('alice', account) # Resolve names to addresses address = convex.resolve_account_name('alice') # Resolve any CNS name address = convex.resolve_name('convex.trust') ``` ## Copper and Convex Coins The Convex network uses **copper** as the smallest unit of currency. Like Bitcoin's satoshis or Ethereum's wei, copper allows for precise fractional amounts: **1 Convex Coin = 1,000,000,000 copper** All balance and transfer amounts in the API are in **copper**: ```python # Request 0.1 Convex Coins (100 million copper) convex.request_funds(100_000_000, account) # Transfer 0.05 Convex Coins (50 million copper) convex.transfer('#456', 50_000_000, account) # Display balance in Convex Coins balance_copper = convex.get_balance(account) balance_coins = balance_copper / 1_000_000_000 print(f'Balance: {balance_coins} CVM') ``` ## Python Version Support - **Python 3.8+** required - **Python 3.10+** recommended for full type hint support ## Dependencies - `requests` - HTTP client for REST API - `cryptography` - Ed25519 signatures and key management - `mnemonic` - BIP39 mnemonic phrase support - `pydantic` - Data validation and serialisation ## Resources - **[GitHub Repository](https://github.com/Convex-Dev/convex-api-py)** - Source code and examples - **[PyPI Package](https://pypi.org/project/convex-sdk/)** - Install from Python Package Index - **[Convex Documentation](https://docs.convex.world)** - Network and platform docs - **[Discord Community](https://discord.com/invite/xfYGq4CT7v)** - Get help and share ideas ## Next Steps - **[Quickstart Guide](python/quickstart)** - Build your first Python app on Convex - **[Query Guide](python/queries)** - Learn how to read network state - **[Transaction Guide](python/transactions)** - Submit state-changing operations - **[Account Management](python/accounts)** - Manage keys and accounts --- # Queries Queries are read-only operations that execute Convex Lisp code without modifying network state. ## Basic Query Pattern The simplest query executes Convex Lisp code from an account context: ```python from convex_sdk import Convex, KeyPair convex = Convex('https://peer.convex.live') # Create an account for query context key_pair = KeyPair() account = convex.create_account(key_pair) # Execute a query result = convex.query('(+ 1 2 3)', account) print(result['value']) # 6 ``` ## Queries are Free Unlike transactions, queries: - **Don't consume juice** - completely free to execute - **Don't require balance** - work even with zero funds - **Don't need signatures** - read-only access - **Execute instantly** - no consensus delay ## Query Methods ### `query(transaction, address_account)` Execute Convex Lisp source code as a read-only query. ```python result = convex.query('(balance #123)', account) ``` **Parameters:** - `transaction` (str): Convex Lisp source code to execute - `address_account` (Account | int | str): Account, address, or address string for execution context **Returns:** Dictionary with query result: ```python { 'value': } ``` ### `get_balance(address_account)` Get the balance of any account in copper: ```python # Get your own balance balance = convex.get_balance(account) # Get another account's balance by address balance = convex.get_balance(123) # Or by address string balance = convex.get_balance('#123') ``` **Returns:** Balance in copper (int) ### `get_account_info(address_account)` Get detailed account information: ```python info = convex.get_account_info(account) print(f"Address: {info.address}") print(f"Balance: {info.balance}") print(f"Sequence: {info.sequence}") print(f"Memory Size: {info.memorySize}") print(f"Type: {info.type}") # 'user', 'actor', or 'library' ``` **Returns:** `AccountDetailsResponse` with fields: - `address` - Account address - `balance` - Current balance in copper - `sequence` - Transaction sequence number - `memorySize` - Memory usage in bytes - `allowance` - Memory allowance - `type` - Account type (user/actor/library) - `isActor` - Whether account is an actor - `isLibrary` - Whether account is a library - `environment` - Account environment (for actors) ## Query Patterns ### Checking Balances ```python # Check if account has sufficient funds balance = convex.get_balance(account) if balance < 10_000_000: # Less than 0.01 CVM print('Insufficient funds') convex.request_funds(100_000_000, account) ``` ### Reading Contract State ```python # Query a deployed contract's state contract_address = '#789' result = convex.query(f'(call {contract_address} (get-count))', account) print(f'Contract count: {result['value']}') ``` ### Evaluating Expressions ```python # Test Convex Lisp expressions before transacting result = convex.query('(map inc [1 2 3 4 5])', account) print(result['value']) # [2, 3, 4, 5, 6] # Check syntax without executing result = convex.query('(let [x 10] (* x x))', account) print(result['value']) # 100 ``` ### Resolving Addresses ```python # Resolve CNS names to addresses address = convex.resolve_name('convex.trust') print(f'convex.trust is at address #{address}') # Resolve account names address = convex.resolve_account_name('alice') print(f'Account "alice" is at address #{address}') ``` ### Reading Global State ```python # Access special variables balance_result = convex.query('*balance*', account) print(f'My balance: {balance_result['value']}') address_result = convex.query('*address*', account) print(f'My address: {address_result['value']}') timestamp_result = convex.query('*timestamp*', account) print(f'Current timestamp: {timestamp_result['value']}') ``` ## Advanced Queries ### Multi-Step Queries Execute multiple expressions in one query using `do`: ```python query = """ (do (def x 10) (def y 20) (+ x y)) """ result = convex.query(query, account) print(result['value']) # 30 ``` ### Conditional Queries Use Lisp conditionals to query based on state: ```python query = """ (if (> *balance* 1000000) "Rich account" "Poor account") """ result = convex.query(query, account) print(result['value']) ``` ### Query with Context Query from a specific account's perspective: ```python # Query as account #123 result = convex.query('*balance*', 123) # Query as account object result = convex.query('*balance*', account) # Query as address string result = convex.query('*balance*', '#123') ``` ## Error Handling Queries can fail due to syntax errors or runtime errors: ```python from convex_sdk.exceptions import ConvexAPIError try: result = convex.query('(invalid-function)', account) except ConvexAPIError as e: print(f'Query failed: {e.code} - {e.message}') ``` Common error codes: - `UNDECLARED` - Symbol not found - `CAST` - Type error - `ARITY` - Wrong number of arguments - `BOUNDS` - Index out of bounds - `NOBODY` - Account doesn't exist ## Best Practices ### 1. Query Before Transacting Test expressions with queries before submitting transactions: ```python # Test first result = convex.query('(transfer #456 1000000)', account) # If successful, transact if 'errorCode' not in result: convex.transact('(transfer #456 1000000)', account) ``` ### 2. Use Queries for Read-Only Data Never use transactions when queries suffice: ```python # ❌ BAD - wastes juice result = convex.transact('(balance #123)', account) # ✅ GOOD - free query balance = convex.get_balance(123) ``` ### 3. Cache Query Results Query results don't change unless someone transacts: ```python # Cache contract metadata contract_info = convex.query(f'(call {contract_address} (get-metadata))', account) # Reuse cached info until state changes ``` ### 4. Query from Minimal Context You don't need a funded account to query: ```python # Create account with no funds just for querying key_pair = KeyPair() query_account = convex.create_account(key_pair) # Query without ever funding it result = convex.query('(+ 1 2)', query_account) ``` ## Performance Tips ### Minimize Query Complexity Complex queries take longer to execute: ```python # ❌ SLOW - iterates entire range result = convex.query('(reduce + (range 1000000))', account) # ✅ FAST - simple arithmetic result = convex.query('(* 1000000 500000)', account) ``` ### Batch Related Queries Use `do` to execute multiple queries in one request: ```python query = """ (do (def balance *balance*) (def timestamp *timestamp*) {:balance balance :timestamp timestamp}) """ result = convex.query(query, account) print(result['value']) # {'balance': 100000000, 'timestamp': 1234567890} ``` ## Next Steps - **[Transaction Guide](transactions)** - Learn how to modify network state - **[Account Management](accounts)** - Manage keys and accounts - **[Convex Lisp Guide](/docs/tutorial/convex-lisp)** - Learn the query language --- # Quickstart Build your first Python application on Convex in under 5 minutes. ## Prerequisites - Python 3.8 or higher - pip or Poetry package manager ## Step 1: Install the SDK Install the `convex-sdk` package: ```bash pip install convex-sdk ``` Or with Poetry: ```bash poetry add convex-sdk ``` ## Step 2: Connect to the Network Create a new Python file `hello_convex.py`: ```python from convex_sdk import Convex # Connect to the public Convex network convex = Convex('https://mikera1337-convex-testnet.hf.space') print('Connected to Convex network') ``` The peer URL `https://mikera1337-convex-testnet.hf.space` connects you to the public Convex network. You can also run a local peer for development. ## Step 3: Create an Account Generate a new key pair and create an account: ```python from convex_sdk import Convex, KeyPair convex = Convex('https://mikera1337-convex-testnet.hf.space') # Generate a new Ed25519 key pair key_pair = KeyPair() # Create an account on the network account = convex.create_account(key_pair) print(f'Account created with address: {account.address}') print(f'Public key: {key_pair.public_key}') ``` **Important**: Save your key pair! Without it, you cannot access your account: ```python # Export keys to encrypted file key_pair.export_to_file('my_account.pem', 'my_secret_password') # Later, import them back key_pair = KeyPair.import_from_file('my_account.pem', 'my_secret_password') ``` ## Step 4: Request Test Funds New accounts start with zero balance. Use the faucet to get test funds: ```python # Request 100 million copper (0.1 Convex Coins) amount = convex.request_funds(100_000_000, account) print(f'Received {amount / 1_000_000_000} Convex Coins') ``` The faucet is available on test networks. Production networks require purchasing or earning Convex Coins. ## Step 5: Query Your Balance Read your account balance using a query: ```python # Query balance in copper balance = convex.get_balance(account) # Convert to Convex Coins coins = balance / 1_000_000_000 print(f'Balance: {coins} CVM ({balance} copper)') ``` Queries are **read-only** and **free** - they don't consume any funds. ## Step 6: Execute a Transaction Submit a transaction to execute Convex Lisp code: ```python # Execute a simple Lisp expression result = convex.transact('(map inc [1 2 3 4])', account) print(f'Result: {result['value']}') # [2, 3, 4, 5] ``` Transactions **modify state** and **consume juice**. They require: - A funded account (balance > 0) - A valid key pair for signing - Sufficient balance for execution costs ## Step 7: Transfer Funds Send Convex Coins to another account: ```python # Create a second account recipient_key_pair = KeyPair() recipient = convex.create_account(recipient_key_pair) # Transfer 10 million copper (0.01 CVM) convex.transfer(recipient.address, 10_000_000, account) print(f'Transferred 0.01 CVM to account {recipient.address}') # Check recipient balance recipient_balance = convex.get_balance(recipient) print(f'Recipient balance: {recipient_balance / 1_000_000_000} CVM') ``` ## Complete Example Here's the complete quickstart script: ```python from convex_sdk import Convex, KeyPair def main(): # Connect to network convex = Convex('https://mikera1337-convex-testnet.hf.space') print('Connected to Convex') # Create account key_pair = KeyPair() account = convex.create_account(key_pair) print(f'Created account: #{account.address}') # Save keys key_pair.export_to_file('my_account.pem', 'secret_password') print('Keys saved to my_account.pem') # Request funds convex.request_funds(100_000_000, account) print('Received test funds') # Check balance balance = convex.get_balance(account) print(f'Balance: {balance / 1_000_000_000} CVM') # Execute transaction result = convex.transact('(+ 1 2 3)', account) print(f'Transaction result: {result['value']}') # Query balance after transaction new_balance = convex.get_balance(account) juice_used = balance - new_balance print(f'Juice consumed: {juice_used} copper') if __name__ == '__main__': main() ``` Run the script: ```bash python hello_convex.py ``` Expected output: ``` Connected to Convex Created account: #1234 Keys saved to my_account.pem Received test funds Balance: 0.1 CVM Transaction result: 6 Juice consumed: 1480 copper ``` ## Next Steps - **[Query Guide](queries)** - Learn advanced query patterns - **[Transaction Guide](transactions)** - Explore transaction capabilities - **[Account Management](accounts)** - Manage keys and account names - **[Convex Lisp Guide](/docs/tutorial/convex-lisp)** - Learn the smart contract language ## Troubleshooting ### Import Error If you get `ModuleNotFoundError: No module named 'convex_sdk'`: ```bash # Check installation pip show convex-sdk # Reinstall if needed pip install --upgrade convex-sdk ``` ### Connection Error If you cannot connect to `https://mikera1337-convex-testnet.hf.space`: - Check your internet connection - Try an alternative peer URL - Check firewall settings ### Sequence Errors If you get sequence errors when submitting multiple transactions: ```python # The SDK automatically retries, but you can increase retry count result = convex.transact( '(transfer #456 1000000)', account, sequence_retry_count=50 # Default is 20 ) ``` ### Insufficient Balance If transactions fail with `FUNDS` error: ```python # Check balance before transacting balance = convex.get_balance(account) if balance < 10_000_000: # 0.01 CVM minimum convex.request_funds(100_000_000, account) ``` --- # Transactions Transactions are state-changing operations that execute Convex Lisp code and modify the network state. ## Transaction Basics Unlike queries, transactions: - ⚡ **Modify state** - Change account balances, deploy contracts, update data - 💰 **Consume juice** - Require Convex Coins to pay for execution - 🔐 **Require signing** - Must be cryptographically signed by account key - ⏱️ **Achieve consensus** - Confirmed by network consensus (sub-second) - 📈 **Increment sequence** - Each account has a monotonic sequence number ## Basic Transaction Pattern ```python from convex_sdk import Convex, KeyPair convex = Convex('https://peer.convex.live') # Create and fund an account key_pair = KeyPair() account = convex.create_account(key_pair) convex.request_funds(100_000_000, account) # Execute a transaction result = convex.transact('(def my-value 42)', account) print(result['value']) # 42 ``` ## Transaction Methods ### `transact(transaction, account, sequence_retry_count=20)` Execute Convex Lisp source code as a state-changing transaction. ```python result = convex.transact('(def x 10)', account) ``` **Parameters:** - `transaction` (str): Convex Lisp source code to execute - `account` (Account): Account to sign and execute the transaction - `sequence_retry_count` (int): Number of times to retry on sequence errors (default: 20) **Returns:** Dictionary with transaction result: ```python { 'value': , 'id': , 'error': # Only present if transaction failed } ``` **Raises:** - `ConvexAPIError` - If transaction fails (after retries) - `ValueError` - If transaction string is invalid ### `transfer(to_address_account, amount, account)` Transfer Convex Coins to another account: ```python # Transfer 0.01 CVM (10 million copper) result = convex.transfer('#456', 10_000_000, account) print(f'Transferred: {result} copper') ``` **Parameters:** - `to_address_account` (Account | int | str): Recipient account or address - `amount` (int | float): Amount in copper to transfer - `account` (Account): Source account (must have sufficient balance) **Returns:** Amount transferred in copper (int) ## Transaction Costs Every transaction consumes **juice** based on: 1. **Memory usage** - Creating new data structures 2. **Computation** - CPU cycles for execution 3. **Storage** - Persisting data on-chain Example juice costs: ```python # Check balance before transaction balance_before = convex.get_balance(account) # Execute transaction convex.transact('(def x [1 2 3 4 5])', account) # Calculate juice consumed balance_after = convex.get_balance(account) juice_consumed = balance_before - balance_after print(f'Juice consumed: {juice_consumed} copper') ``` Typical costs: - **Simple expression**: 1,000 - 5,000 copper - **Transfer**: 2,000 - 3,000 copper - **Define variable**: 1,500 - 10,000 copper - **Deploy contract**: 50,000 - 500,000 copper ## Transaction Patterns ### Defining Variables Store data in your account's environment: ```python # Define a simple value convex.transact('(def my-number 42)', account) # Define a collection convex.transact('(def my-list [1 2 3 4 5])', account) # Define a map convex.transact('(def my-map {:name "Alice" :age 30})', account) # Read back with a query result = convex.query('my-number', account) print(result['value']) # 42 ``` ### Transferring Funds ```python # Create recipient recipient_keys = KeyPair() recipient = convex.create_account(recipient_keys) # Transfer 0.05 CVM convex.transfer(recipient.address, 50_000_000, account) # Verify transfer balance = convex.get_balance(recipient) print(f'Recipient balance: {balance / 1_000_000_000} CVM') ``` ### Deploying Smart Contracts Deploy an actor (smart contract) using `deploy`: ```python # Deploy a simple counter contract contract_source = """ (deploy (do (def count 0) (defn increment [] (def count (inc count)) count) (defn get-count [] count))) """ result = convex.transact(contract_source, account) contract_address = result['value'] print(f'Contract deployed at: #{contract_address}') ``` ### Calling Deployed Contracts Interact with deployed actors: ```python # Call actor function result = convex.transact( f'(call #{contract_address} (increment))', account ) print(f'New count: {result['value']}') # Query actor state (free) result = convex.query( f'(call #{contract_address} (get-count))', account ) print(f'Current count: {result['value']}') ``` ### Multi-Step Transactions Execute multiple operations atomically: ```python transaction = """ (do (def x 10) (def y 20) (def sum (+ x y)) (transfer #456 sum) sum) """ result = convex.transact(transaction, account) print(f'Transferred {result['value']} copper') ``` If any step fails, the entire transaction is rolled back. ## Sequence Numbers Every transaction increments the account's sequence number: ```python # Get current sequence info = convex.get_account_info(account) print(f'Current sequence: {info.sequence}') # Submit transaction (auto-increments sequence) convex.transact('(def x 1)', account) # Sequence is now +1 info = convex.get_account_info(account) print(f'New sequence: {info.sequence}') ``` ### Handling Sequence Errors When multiple transactions submit concurrently, sequence conflicts can occur. The SDK automatically retries: ```python # Automatically retries up to 20 times result = convex.transact('(def x 10)', account) # Increase retry count for high-concurrency scenarios result = convex.transact( '(def x 10)', account, sequence_retry_count=50 ) ``` ## Error Handling Transactions can fail for various reasons: ```python from convex_sdk.exceptions import ConvexAPIError try: result = convex.transact('(transfer #999999 1000000000000)', account) except ConvexAPIError as e: if e.code == 'FUNDS': print('Insufficient balance') elif e.code == 'NOBODY': print('Recipient account does not exist') elif e.code == 'SEQUENCE': print('Sequence error (unlikely after retries)') else: print(f'Transaction failed: {e.code} - {e.message}') ``` Common error codes: | Code | Meaning | Solution | |------|---------|----------| | `FUNDS` | Insufficient balance | Request more funds or reduce amount | | `NOBODY` | Account doesn't exist | Create recipient account first | | `SEQUENCE` | Sequence mismatch | SDK auto-retries, increase retry count | | `CAST` | Type error | Fix Lisp expression | | `UNDECLARED` | Symbol not found | Check variable/function names | | `ARGUMENT` | Invalid argument | Check function parameters | ## Advanced Patterns ### Conditional Transactions Execute different logic based on conditions: ```python transaction = """ (if (> *balance* 1000000) (transfer #456 500000) :insufficient-funds) """ result = convex.transact(transaction, account) ``` ### Setting Account Keys Transfer account control to a new key pair: ```python # Generate new keys new_keys = KeyPair() # Set the new public key on-chain transaction = f'(set-key {new_keys.public_key_checksum})' convex.transact(transaction, account) # Now only new_keys can transact for this address ``` **⚠️ Warning**: Save the new keys before changing! Losing keys means losing account access. ### Creating Sub-Accounts Some contracts allow creating controlled sub-accounts: ```python # Create account controlled by a contract transaction = f'(call #{contract_address} (create-account))' result = convex.transact(transaction, account) sub_account_address = result['value'] print(f'Created sub-account: #{sub_account_address}') ``` ## Best Practices ### 1. Query Before Transacting Test transactions with queries first: ```python # Test with query (free) result = convex.query('(transfer #456 1000000)', account) if 'errorCode' in result: print(f'Would fail: {result["errorCode"]}') else: # Proceed with transaction convex.transact('(transfer #456 1000000)', account) ``` ### 2. Check Balance First Verify sufficient funds before transacting: ```python balance = convex.get_balance(account) transfer_amount = 50_000_000 if balance < transfer_amount + 10_000: # Extra for juice print('Insufficient balance') else: convex.transfer('#456', transfer_amount, account) ``` ### 3. Handle Failures Gracefully Always handle potential errors: ```python from convex_sdk.exceptions import ConvexAPIError def safe_transfer(convex, to_address, amount, from_account): try: result = convex.transfer(to_address, amount, from_account) return result except ConvexAPIError as e: print(f'Transfer failed: {e.code}') return None ``` ### 4. Use Atomic Transactions Group related operations in a single transaction: ```python # ❌ BAD - Two separate transactions convex.transact('(def x 10)', account) convex.transact('(def y (* x 2))', account) # Might fail if first didn't commit # ✅ GOOD - One atomic transaction convex.transact('(do (def x 10) (def y (* x 2)))', account) ``` ### 5. Keep Juice Costs Low Minimize memory allocation and computation: ```python # ❌ EXPENSIVE - Creates large data structure convex.transact('(def big-list (range 1000000))', account) # ✅ CHEAPER - Store minimal data convex.transact('(def count 1000000)', account) ``` ## Transaction Lifecycle 1. **Prepare** - SDK creates transaction with source and account address 2. **Sign** - Account's private key signs the transaction hash 3. **Submit** - Signed transaction sent to peer 4. **Consensus** - Network reaches consensus (typically < 1 second) 5. **Execute** - CVM executes transaction 6. **Result** - Result returned to client ```python # All handled automatically by SDK result = convex.transact('(def x 42)', account) # But you can see the lifecycle in logs import logging logging.basicConfig(level=logging.DEBUG) result = convex.transact('(def x 42)', account) # Logs show: prepare → sign → submit → result ``` ## Next Steps - **[Account Management](accounts)** - Manage keys and account names - **[Convex Lisp Guide](/docs/tutorial/convex-lisp)** - Learn the smart contract language - **[Actor Development](/docs/tutorial/actors)** - Build smart contracts --- # Account Management Learn how to generate, import, and manage Convex accounts and Ed25519 key pairs. ## Overview Every Convex account has: - **Address** - A unique identifier (e.g., `#1678`) - **Ed25519 Seed** - 32-byte private seed (keep this secret!) - **Public Key** - Derived from the seed (safe to share) - **Balance** - Amount of Convex Coins owned - **Sequence** - Transaction counter ## Generating New Key Pairs ### Random Generation Generate a new random key pair: ```typescript import { KeyPair } from '@convex-world/convex-ts'; // Generate random key pair const keyPair = KeyPair.generate(); // Access the keys console.log('Public key:', keyPair.publicKeyHex); console.log('Seed:', keyPair.privateKeyHex); // Keep secret! // Access as bytes const publicBytes: Uint8Array = keyPair.publicKey; const seedBytes: Uint8Array = keyPair.privateKey; ``` :::danger Keep Seeds Secret Never commit seeds to version control, share them publicly, or store them unencrypted. Anyone with your seed can control your account. ::: ### From Existing Seed If you already have an Ed25519 seed: ```typescript import { KeyPair } from '@convex-world/convex-ts'; // From hex string const keyPair = KeyPair.fromSeed('your-32-byte-seed-hex'); // From bytes const seedBytes = new Uint8Array(32); // Your seed const keyPair = KeyPair.fromSeed(seedBytes); // Public key is automatically derived console.log('Public key:', keyPair.publicKeyHex); ``` ## Setting Up Your Account ### Basic Setup ```typescript import { Convex, KeyPair } from '@convex-world/convex-ts'; const convex = new Convex('https://peer.convex.live'); // Load your key pair const keyPair = KeyPair.fromSeed(process.env.CONVEX_SEED!); // Set your account convex.setAccount('#1678', keyPair); // Now you can transact const info = await convex.getAccountInfo(); console.log('Balance:', info.balance); ``` ### Separate Signer and Address For advanced use cases, you can set the signer and address separately: ```typescript // Set signer first convex.setSigner(keyPair); // Then set address convex.setAddress('#1678'); // Or switch addresses with the same signer convex.setAddress('#9999'); ``` This is useful when one signer controls multiple accounts. ## Account Information ### Get Account Details ```typescript const info = await convex.getAccountInfo(); console.log('Address:', info.address); console.log('Balance:', info.balance / 1_000_000_000, 'coins'); console.log('Sequence:', info.sequence); console.log('Public key:', info.publicKey); ``` ### Check Balance ```typescript // Your native coin balance const result = await convex.balance(); console.log('Balance:', result.value, 'copper'); // Another account's balance const other = await convex.balance('#123'); console.log('Other balance:', other.value); ``` ## Key Pair Formats ### Hex Format ```typescript const keyPair = KeyPair.generate(); // Get keys as hex strings const publicKeyHex = keyPair.publicKeyHex; const seedHex = keyPair.privateKeyHex; // Restore from hex const restored = KeyPair.fromSeed(seedHex); ``` ### Bytes Format ```typescript const keyPair = KeyPair.generate(); // Get keys as Uint8Array const publicKey: Uint8Array = keyPair.publicKey; // 32 bytes const seed: Uint8Array = keyPair.privateKey; // 32 bytes // Restore from bytes const restored = KeyPair.fromSeed(seed); ``` ## Environment Variables ### Best Practice Storage Store credentials in environment variables: ```typescript // .env file (never commit this!) CONVEX_SEED=abc123... CONVEX_ADDRESS=#1678 CONVEX_PEER=https://peer.convex.live ``` ```typescript // app.ts import { Convex, KeyPair } from '@convex-world/convex-ts'; const convex = new Convex(process.env.CONVEX_PEER!); const keyPair = KeyPair.fromSeed(process.env.CONVEX_SEED!); convex.setAccount(process.env.CONVEX_ADDRESS!, keyPair); ``` ### Using dotenv ```bash npm install dotenv ``` ```typescript import 'dotenv/config'; import { Convex, KeyPair } from '@convex-world/convex-ts'; const convex = new Convex(process.env.CONVEX_PEER!); const keyPair = KeyPair.fromSeed(process.env.CONVEX_SEED!); convex.setAccount(process.env.CONVEX_ADDRESS!, keyPair); ``` ## Key Pair Validation ### Verify Seed Length ```typescript import { KeyPair, hexToBytes } from '@convex-world/convex-ts'; function loadKeyPair(seedHex: string): KeyPair { const seedBytes = hexToBytes(seedHex); if (seedBytes.length !== 32) { throw new Error(`Invalid seed length: expected 32 bytes, got ${seedBytes.length}`); } return KeyPair.fromSeed(seedBytes); } ``` ### Verify Key Derivation ```typescript const keyPair = KeyPair.fromSeed(seed); // Public key should always be 32 bytes if (keyPair.publicKey.length !== 32) { throw new Error('Invalid public key length'); } // Verify deterministic derivation const keyPair2 = KeyPair.fromSeed(seed); if (keyPair.publicKeyHex !== keyPair2.publicKeyHex) { throw new Error('Non-deterministic key derivation'); } ``` ## Multiple Accounts ### Managing Multiple Key Pairs ```typescript class AccountManager { private accounts = new Map(); addAccount(address: string, keyPair: KeyPair) { this.accounts.set(address, keyPair); } getKeyPair(address: string): KeyPair | undefined { return this.accounts.get(address); } async useAccount(convex: Convex, address: string) { const keyPair = this.accounts.get(address); if (!keyPair) { throw new Error(`No key pair for ${address}`); } convex.setAccount(address, keyPair); } } // Usage const manager = new AccountManager(); manager.addAccount('#1678', KeyPair.fromSeed(seed1)); manager.addAccount('#9999', KeyPair.fromSeed(seed2)); await manager.useAccount(convex, '#1678'); await convex.transfer('#456', 1000000); await manager.useAccount(convex, '#9999'); await convex.transfer('#789', 2000000); ``` ### Switching Between Accounts ```typescript // Use different accounts with the same client const kp1 = KeyPair.fromSeed(seed1); const kp2 = KeyPair.fromSeed(seed2); // Transact as first account convex.setAccount('#1678', kp1); await convex.transfer('#456', 1_000_000); // Switch to second account convex.setAccount('#9999', kp2); await convex.transfer('#789', 2_000_000); ``` ## Security Best Practices ### ✅ Do - Generate seeds with cryptographically secure random number generators - Store seeds in environment variables or encrypted keystores - Use different accounts for different purposes (hot/cold wallets) - Back up seeds securely (encrypted, offline) - Verify public key derivation is deterministic - Use hardware wallets for high-value accounts ### ❌ Don't - Hardcode seeds in source code - Commit seeds to version control - Share seeds via insecure channels (email, chat) - Store seeds in browser localStorage without encryption - Reuse seeds across different networks without understanding implications - Generate seeds with weak random number generators (Math.random()) ## Backup and Recovery ### Mnemonic Seeds (BIP39) For better user experience, consider using BIP39 mnemonics: ```typescript import * as bip39 from 'bip39'; // Generate mnemonic const mnemonic = bip39.generateMnemonic(); console.log('Backup phrase:', mnemonic); // "witch collapse practice feed shame open despair creek road again ice least" // Derive seed from mnemonic const seed = bip39.mnemonicToSeedSync(mnemonic); const ed25519Seed = seed.subarray(0, 32); // Use first 32 bytes // Create key pair const keyPair = KeyPair.fromSeed(ed25519Seed); ``` ### Paper Backup ```typescript // Generate QR code for backup import QRCode from 'qrcode'; const keyPair = KeyPair.generate(); // Create QR code of seed (for paper backup) const qrCode = await QRCode.toDataURL(keyPair.privateKeyHex); // Display or print QR code console.log('Scan this QR code to restore your account:'); console.log(qrCode); ``` ## Account Creation ### Create via REST API The simplest way to create an account — no existing account needed: ```typescript import { Convex, KeyPair } from '@convex-world/convex-ts'; const convex = new Convex('https://peer.convex.live'); // Generate a fresh key pair const keyPair = KeyPair.generate(); // Create account with faucet funding (test networks only) const info = await convex.createAccount(keyPair, 100_000_000); console.log('New address:', info.address); console.log('Balance:', info.balance); // Set the new account on the client convex.setAccount(info.address, keyPair); ``` ### Top Up with Faucet Request additional funds for an existing account (test networks only): ```typescript await convex.faucet('#1678', 100_000_000); ``` :::warning Faucets are test-only Faucets are only available on test networks. On production networks, someone must transfer initial funds to your account. ::: ## Next Steps - **[Signers](./signers)** — Hardware wallet and custom signers - **[Asset Handles](./assets)** — Token and CNS management - **[Transactions](./transactions)** — Using your account ## See Also - [Ed25519 Signature Scheme](https://ed25519.cr.yp.to/) - [BIP39 Mnemonic Codes](https://github.com/bitcoin/bips/blob/master/bip-0039.mediawiki) --- # Asset Handles Learn how to interact with on-chain assets using the fluent Asset Handle APIs introduced in v0.3.0. ## Overview Asset handles provide a lightweight, fluent API for working with on-chain assets. Rather than composing raw Convex Lisp expressions, you create a handle object and call methods directly. Three handle types are available: - **🪙 FungibleToken** — CAD29 fungible tokens (`@convex.fungible/*`) - **📦 AssetHandle** — Generic assets including NFTs (`@convex.asset/*`) - **🏷️ CnsHandle** — Convex Name System (`@convex.cns/*`) Handles are: - **Lightweight** — no network calls on construction - **Stateless** — they hold only the asset address and a reference back to the `Convex` client - **Synchronous to create** — `convex.fungible('#128')` returns immediately :::info No Account Needed for Queries Read-only methods such as `balance()`, `supply()`, and `decimals()` work without setting an account. State-modifying methods like `transfer()` and `mint()` require an account and signer. ::: ## 🪙 Fungible Tokens Use `convex.fungible()` to create a `FungibleToken` handle for any CAD29-compliant token: ```typescript import { Convex, KeyPair } from '@convex-world/convex-ts'; const convex = new Convex('https://peer.convex.live'); const token = convex.fungible('#128'); ``` ### Querying Token State Read-only operations — no account needed: ```typescript // Balance of the client's current address const bal = await token.balance(); // Balance of another account const bal2 = await token.balance('#13'); // Total supply across all holders const sup = await token.supply(); // Number of decimal places (for display formatting) const dec = await token.decimals(); ``` ### Transacting with Tokens State-modifying operations — requires an account: ```typescript const keyPair = KeyPair.fromSeed(process.env.CONVEX_SEED!); convex.setAccount('#1678', keyPair); // Transfer tokens to another account await token.transfer('#456', 1000); // BigInt is supported for large amounts await token.transfer('#456', 1000000000000000000n); // Mint new tokens (must have minting authority) await token.mint(5000); // Burn tokens from your own balance await token.burn(100); ``` ### Amount Validation All amounts are validated as non-negative integers. The `BalanceLike` type accepts: | Type | Example | Notes | |------|---------|-------| | `number` | `1000` | Must be a non-negative integer | | `bigint` | `1000000000000000000n` | For amounts exceeding `Number.MAX_SAFE_INTEGER` | | `string` | `"1000"` | Parsed as an integer string | Negative values, fractional numbers, and non-numeric strings throw immediately on the client side. ## 📦 Generic Assets (NFTs and More) Use `convex.asset()` to create an `AssetHandle` for any asset that follows the Convex asset protocol — including NFTs, multi-token contracts, and other custom assets: ```typescript const asset = convex.asset('#256'); ``` ### Querying Asset State ```typescript // Balance of the client's current address const bal = await asset.balance(); // Total supply const sup = await asset.supply(); ``` ### Transferring Assets The quantity parameter is flexible — it can be a number, bigint, or a CVM expression string for non-numeric asset quantities: ```typescript // Fungible-like numeric transfer await asset.transfer('#456', 100); // NFT set transfer using a CVM expression string await asset.transfer('#456', '#{:foo :bar}'); ``` :::tip CVM Expression Strings When you pass a string quantity like `'#{:foo :bar}'`, it is sandboxed inside a `(query ...)` form before being embedded in the transaction. This prevents injection of arbitrary code and ensures only valid CVM data expressions are accepted. ::: ### Offer / Accept Pattern For trustless exchanges where two parties need to swap assets without trusting each other, use the offer/accept pattern: ```typescript // Party A: offer specific NFTs to Party B await asset.offer('#456', '#{1 2 3}'); // Party B: accept a fungible quantity from Party A await asset.accept('#456', 50); ``` The offer is recorded on-chain and can only be claimed by the designated recipient through `accept()`. ## 🏷️ CNS (Convex Name System) Use `convex.cns()` to create a `CnsHandle` for resolving and managing Convex Name System entries: ```typescript const handle = convex.cns('convex.core'); ``` ### Reading CNS Entries No account needed: ```typescript // Resolve a name to its address const result = await handle.resolve(); // → #8 ``` ### Writing CNS Entries Requires an account with the appropriate CNS permissions: ```typescript const keyPair = KeyPair.fromSeed(process.env.CONVEX_SEED!); convex.setAccount('#1678', keyPair); // Update the address a name points to await handle.set('#1678'); // Transfer control of the name to another account await handle.setController('#99'); ``` ### Name Validation Names are validated on construction. The following will throw immediately — no network round-trip required: - Empty strings - Names starting with a digit - Names containing parentheses or other invalid characters ```typescript // These throw immediately: convex.cns(''); // Error: invalid CNS name convex.cns('123bad'); // Error: invalid CNS name convex.cns('foo(bar)'); // Error: invalid CNS name ``` ## Error Handling All handle methods throw `ConvexError` when the CVM returns an error. This is the same error type used by `query()` and `transact()`: ```typescript import { Convex, KeyPair, ConvexError } from '@convex-world/convex-ts'; const convex = new Convex('https://peer.convex.live'); const keyPair = KeyPair.fromSeed(process.env.CONVEX_SEED!); convex.setAccount('#1678', keyPair); const token = convex.fungible('#128'); try { await token.transfer('#456', 999999999); } catch (e) { if (e instanceof ConvexError) { console.error('CVM error code:', e.code); // e.g. "FUNDS" console.error('Execution info:', e.info); // { juice: 100, fees: 50, ... } } } ``` Common error codes you may encounter: | Code | Meaning | |------|---------| | `FUNDS` | Insufficient token balance for the operation | | `TRUST` | Caller lacks permission (e.g. minting without authority) | | `ARGUMENT` | Invalid argument (e.g. negative amount reached the CVM) | | `STATE` | Invalid state for the operation | | `NOBODY` | Target account does not exist | ## Class Hierarchy The three handle types are organised as follows: - **`AssetHandle`** — base class for generic asset operations (`balance`, `supply`, `transfer`, `offer`, `accept`) - **`FungibleToken` extends `AssetHandle`** — adds optimised fungible-specific operations (`decimals`, `mint`, `burn`) - **`CnsHandle`** — independent class for name resolution (`resolve`, `set`, `setController`) `FungibleToken` inherits all `AssetHandle` methods, so you can use `offer()` and `accept()` on fungible tokens as well. ## Complete Example Putting it all together — querying a token, checking a CNS name, and performing a transfer: ```typescript import { Convex, KeyPair, ConvexError } from '@convex-world/convex-ts'; async function main() { const convex = new Convex('https://peer.convex.live'); // 1. Resolve a token address from CNS (no account needed) const cns = convex.cns('my.custom.token'); const resolved = await cns.resolve(); console.log('Token address:', resolved.result); // 2. Create a fungible token handle const token = convex.fungible(resolved.result); // 3. Query token metadata const supply = await token.supply(); const decimals = await token.decimals(); console.log('Total supply:', supply, `(${decimals} decimal places)`); // 4. Set up account for transactions const keyPair = KeyPair.fromSeed(process.env.CONVEX_SEED!); convex.setAccount(process.env.CONVEX_ADDRESS!, keyPair); // 5. Check balance and transfer const balance = await token.balance(); console.log('My balance:', balance); if (balance > 0) { try { await token.transfer('#456', 100); console.log('Transfer successful'); } catch (e) { if (e instanceof ConvexError) { console.error('Transfer failed:', e.code); } } } } main().catch(console.error); ``` ## Next Steps - **[Queries](./queries)** — Learn about read-only queries and result handling - **[Transactions](./transactions)** — Understand the full transaction lifecycle - **[Convex Lisp](/docs/tutorial/convex-lisp/)** — Master the language behind asset contracts --- # TypeScript The official TypeScript/JavaScript client library for the Convex decentralised lattice network. ## Overview The `@convex-world/convex-ts` package provides a modern, type-safe way to interact with the Convex network from TypeScript applications. It works across multiple environments including Node.js, browsers, Deno, and Bun. ## Key Features - **🔍 Read-Only Queries** - Query network state without needing an account or keys - **🔐 Account Management** - Full support for Ed25519 key pairs and account operations - **💸 Transactions** - Submit and track transactions with complete type safety - **🪙 Asset Handles** - Fluent API for fungible tokens, generic assets, and CNS - **🔌 Pluggable Signers** - Extensible signer interface for hardware wallets, browser extensions, and HSM - **💾 Secure Keystore** - Encrypted key storage with password protection - **📘 Full TypeScript Support** - Complete type definitions for IntelliSense and compile-time checks - **🌐 Environment Agnostic** - No DOM dependencies, works everywhere JavaScript runs - **⚛️ React Integration** - Hooks and components via `@convex-world/convex-react` ## Installation ```bash # Using npm npm install @convex-world/convex-ts # Using pnpm pnpm add @convex-world/convex-ts # Using yarn yarn add @convex-world/convex-ts ``` ## Quick Start ### Read-Only Query (No Account Needed) The simplest way to get started - query the network without authentication: ```typescript import { Convex } from '@convex-world/convex-ts'; // Connect to the Convex network const convex = new Convex('https://peer.convex.live'); // Query any account's balance const result = await convex.query('(balance #13)'); console.log('Balance:', result.value); ``` ### Using Your Account If you have a Convex account and Ed25519 seed: ```typescript import { Convex, KeyPair, ConvexError } from '@convex-world/convex-ts'; // Connect to network const convex = new Convex('https://peer.convex.live'); // Load your key pair from seed const keyPair = KeyPair.fromSeed('your-32-byte-seed-hex'); convex.setAccount('#1678', keyPair); // Query your balance (balance() returns a Result; read .value for the number) const balance = (await convex.balance()).value; console.log('Balance:', balance / 1_000_000_000, 'Convex Coins'); // Transfer coins const result = await convex.transfer('#456', 1_000_000_000); if (result.errorCode) { console.error('Transfer failed:', result.errorCode); } else { console.log('Transfer result:', result.value); } ``` ## Network URLs Connect to different Convex networks: - **Protonet (Production)**: `https://peer.convex.live` - Live network with real assets (no public faucet) - **Testnet**: `https://mikera1337-convex-testnet.hf.space` - Test network with faucet for testing - **Local**: `http://localhost:8080` - Local development peer **Important**: Faucets providing free funds are only available on test networks. Production networks require acquiring Convex Coins through legitimate means. See the [Faucet Guide](/docs/tutorial/coins/faucet) for details. ## Next Steps ## Resources - **📦 npm Package**: [@convex-world/convex-ts](https://www.npmjs.com/package/@convex-world/convex-ts) - **🐙 GitHub**: [Convex-Dev/convex.ts](https://github.com/Convex-Dev/convex.ts) - **💬 Discord**: [Convex Community](https://discord.com/invite/xfYGq4CT7v) - **📚 Full Documentation**: [docs.convex.world](https://docs.convex.world) --- # Querying Network State Learn how to query the Convex network to read account balances, smart contract state, and execute read-only Convex Lisp expressions. ## Overview Queries are **read-only operations** that don't require an account or keys. They allow you to: - Read account balances - Query smart contract state - Execute Convex Lisp expressions - Inspect network data Queries do not modify state and are not recorded as transactions. ## Basic Query The simplest way to query uses a string containing Convex Lisp code: ```typescript import { Convex } from '@convex-world/convex-ts'; const convex = new Convex('https://peer.convex.live'); // Query an account balance const result = await convex.query('(balance #13)'); console.log('Balance:', result.value); ``` ## Query with Address Context Some queries need an address context (the `*address*` special variable). You can set a default address on the client: ```typescript // Set a default address for subsequent queries convex.setAddress('#1678'); // Now query() automatically includes this address const result = await convex.query('*balance*'); console.log('Balance:', result.value); ``` You can also pass address context explicitly using the object form: ```typescript // Query using explicit address context const result = await convex.query({ address: '#1678', source: '*balance*' // Uses the context address }); console.log('Balance:', result.value); ``` :::tip When to Use Address Context Use `setAddress()` or the object form with `address` when: - Your query references `*address*` or other address-specific variables - You need to execute code as if you were a specific account - Querying actor-specific state When `setAddress()` has been called, `query()` with a plain string automatically includes the address. For queries that don't depend on a context address, the simple string form is sufficient. ::: ## Common Query Patterns ### Account Balances The preferred way to check balances is with the `balance()` convenience method: ```typescript // Balance of the client's current address const myBalance = await convex.balance(); // Balance of a specific account const balance = await convex.balance('#13'); ``` You can also query balances using Convex Lisp: ```typescript // Specific account const result = await convex.query('(balance #123)'); console.log('Balance:', result.value); // Multiple accounts const accounts = ['#9', '#10', '#11']; for (const addr of accounts) { const result = await convex.query(`(balance ${addr})`); console.log(`${addr}:`, result.value); } ``` ### Mathematical Expressions ```typescript // Execute Convex Lisp maths const result = await convex.query('(+ 1 2 3 4 5)'); console.log('Sum:', result.value); // 15 // More complex const calc = await convex.query('(* (+ 10 5) (- 8 3))'); console.log('Result:', calc.value); // 75 ``` ### Smart Contract Calls ```typescript // Query a smart contract const result = await convex.query('(call #789 (get-price :BTC))'); console.log('BTC Price:', result.value); ``` ### CNS Lookups The Convex Name Service (CNS) provides human-readable names for on-chain addresses. You can resolve CNS names directly or via a handle: ```typescript // Direct CNS resolution const result = await convex.query('@convex.core'); console.log('Address:', result.result); // Or via a CNS handle const handle = convex.cns('convex.core'); const resolved = await handle.resolve(); console.log('Address:', resolved.result); ``` ## Handling Query Results ### Result Structure Query results have this structure: ```typescript interface Result { value?: any; // JSON-converted CVM value result?: string; // CVM printed representation (e.g. "#8" for an address) errorCode?: string; // Error code (triggers throw) info?: ResultInfo; // Execution metadata (juice, fees, trace, etc.) } ``` The `value` field contains the JSON-converted return value, whilst `result` contains the CVM printed representation as a string. For example, an address would appear as `"#8"` in `result` but may be a number in `value`. ### Error Handling In v0.3.0, `query()` and `transact()` automatically throw a `ConvexError` when the CVM returns an error (i.e. when `errorCode` is present in the response). Use try/catch to handle errors: ```typescript import { Convex, ConvexError } from '@convex-world/convex-ts'; const convex = new Convex('https://peer.convex.live'); try { const result = await convex.query('(balance #123)'); console.log('Success:', result.value); } catch (error) { if (error instanceof ConvexError) { // CVM error — the query was executed but failed console.error('CVM error code:', error.code); console.error('Execution info:', error.info); // juice, fees, trace, etc. console.error('Full result:', error.result); } else { // Network or other error console.error('Network error:', error); } } ``` The `ConvexError` class provides: - `error.code` — the CVM error code string (e.g. `"NOBODY"`, `"UNDECLARED"`) - `error.info` — a `ResultInfo` object with execution metadata (juice, fees, trace, etc.) - `error.result` — the full `Result` object from the response ### Type Checking ```typescript const result = await convex.query('(balance #123)'); // Parse numeric values const balance = Number(result.value); if (isNaN(balance)) { throw new Error('Expected numeric balance'); } console.log('Balance in coins:', balance / 1_000_000_000); ``` ## Advanced Queries ### Multi-line Queries ```typescript const result = await convex.query(` (do (def total-supply 1000000000) (def circulating (* total-supply 0.3)) {:total total-supply :circulating circulating :locked (- total-supply circulating)}) `); console.log('Token metrics:', result.value); ``` ### Querying Collections ```typescript // Query a map const userData = await convex.query(` (get user-data #1678) `); // Query a vector const topScores = await convex.query(` (get-holding #789 :top-10-scores) `); ``` ### Conditional Queries ```typescript const result = await convex.query(` (if (> (balance #123) 1000000) "Sufficient balance" "Insufficient balance") `); ``` ## Performance Tips ### Batch Queries Run multiple independent queries in parallel: ```typescript const [balance1, balance2, balance3] = await Promise.all([ convex.query('(balance #100)'), convex.query('(balance #200)'), convex.query('(balance #300)') ]); ``` ### Optimise Query Logic ```typescript // Bad: Multiple round trips const balance = await convex.query('(balance #123)'); const sequence = await convex.query('(account-sequence #123)'); // Good: Single query const result = await convex.query(` {:balance (balance #123) :sequence (account-sequence #123)} `); ``` ## Next Steps - **[Transactions](./transactions)** — Learn to modify state - **[Asset Handles](./assets)** — Fluent API for tokens, assets, and CNS - **[Convex Lisp Guide](/docs/tutorial/convex-lisp/)** — Master the query language --- # Quickstart Tutorial Build your first Convex application in TypeScript against the public testnet — free, with a faucet for test funds. ## Prerequisites - Node.js 18+ installed - Basic knowledge of JavaScript/TypeScript - A code editor (VS Code recommended) ## Step 1: Create a New Project ```bash mkdir my-convex-app cd my-convex-app npm init -y npm install @convex-world/convex-ts npm install -D typescript tsx @types/node ``` Create a `tsconfig.json`: ```json { "compilerOptions": { "target": "ES2020", "module": "NodeNext", "moduleResolution": "NodeNext", "strict": true, "esModuleInterop": true } } ``` ## Step 2: Query the Network Queries are read-only and free — no account needed. Create `src/query.ts`: ```typescript import { Convex } from '@convex-world/convex-ts'; async function main() { // Connect to the public testnet const convex = new Convex('https://mikera1337-convex-testnet.hf.space'); // Query an account balance const result = await convex.query('(balance #13)'); console.log('Balance:', result.value, 'copper'); } main().catch(console.error); ``` Run it: ```bash npx tsx src/query.ts ``` ## Step 3: Create a Funded Account On a test network you can create an account and top it up from the faucet. Create `src/account.ts`: ```typescript import { Convex, KeyPair } from '@convex-world/convex-ts'; async function main() { const convex = new Convex('https://mikera1337-convex-testnet.hf.space'); // Generate a key pair and create a faucet-funded account // (faucet amount is in coppers; 100_000_000 = 0.1 CVM) const keyPair = KeyPair.generate(); const account = await convex.createAccount(keyPair, 100_000_000); convex.setAccount(account.address, keyPair); console.log('Address:', account.address); const info = await convex.getAccountInfo(); console.log('Balance:', info.balance / 1_000_000_000, 'Convex Coins'); // Save the seed if you want to reuse this account later: // console.log('Seed:', keyPair.toHex().privateKey); } main().catch(console.error); ``` :::tip Production / existing accounts Faucets only work on test networks. On production (`https://peer.convex.live`) you load an existing funded account from its seed instead of creating one: ```typescript const keyPair = KeyPair.fromSeed(process.env.CONVEX_SEED!); convex.setAccount(process.env.CONVEX_ADDRESS!, keyPair); ``` ::: ## Step 4: Submit a Transaction With an account wired up, submit a transaction (any Convex Lisp). Create `src/transact.ts`: ```typescript import { Convex, KeyPair } from '@convex-world/convex-ts'; async function main() { const convex = new Convex('https://mikera1337-convex-testnet.hf.space'); // Create + use a funded account (see Step 3) const keyPair = KeyPair.generate(); const account = await convex.createAccount(keyPair, 100_000_000); convex.setAccount(account.address, keyPair); // Store a value on-chain const result = await convex.transact('(def greeting "Hello from TypeScript!")'); console.log('Stored:', result.value); // Read it back (free query) const query = await convex.query('greeting'); console.log('Read back:', query.value); } main().catch(console.error); ``` `transact()` throws a `ConvexError` on a CVM-level failure (e.g. insufficient funds) — wrap it in `try/catch` if you want to handle those explicitly. ## Step 5: Transfer Coins Send Convex Coins to another account: ```typescript // ... after creating + wiring an account ... const result = await convex.transfer('#11', 1_000_000); // to #11, in coppers console.log('Transfer result:', result.value); ``` ## Complete Example Here's everything together — query, create a funded account, transact, and read back: ```typescript import { Convex, KeyPair } from '@convex-world/convex-ts'; async function main() { const convex = new Convex('https://mikera1337-convex-testnet.hf.space'); // 1. Free query — no account needed const foundation = await convex.query('(balance #13)'); console.log('Account #13 balance:', foundation.value, 'copper'); // 2. Create a faucet-funded account const keyPair = KeyPair.generate(); const account = await convex.createAccount(keyPair, 100_000_000); convex.setAccount(account.address, keyPair); console.log('My account:', account.address); // 3. Execute a transaction const stored = await convex.transact('(def my-data {:message "Hello Convex"})'); console.log('Stored:', stored.result); // 4. Query back the data const data = await convex.query('my-data'); console.log('Read back:', data.value); } main().catch(console.error); ``` **🎉 Congratulations!** You've built your first Convex application with TypeScript. ## Next Steps - **[Queries](./queries)** — Learn advanced query patterns - **[Transactions](./transactions)** — Understand the transaction lifecycle - **[Accounts](./accounts)** — Key pair generation and management - **[Asset Handles](./assets)** — Token and CNS management - **[Signers](./signers)** — Hardware wallet integration ## Common Issues ### "Cannot find module" Make sure you're using Node.js 18+ and have `"type": "module"` in your `package.json`. ### "No account set" You must call `convex.setAccount()` before submitting transactions. Read-only queries don't need an account. ### `FUNDS` error The account needs Convex Coins. On the testnet, create it with a faucet amount (Step 3) or call `convex.faucet(address, amount)`; on production you must transfer in funds from an existing account. ### "Connection refused" Check the network URL. For local development, ensure your local peer is running. --- # Signer Interface Learn how to integrate hardware wallets, browser extensions, and other signing mechanisms using the pluggable Signer interface. ## Overview The Signer interface allows you to use different signing mechanisms beyond simple key pairs: - **Hardware Wallets** - Ledger, Trezor, etc. - **Browser Extensions** - MetaMask-style wallets - **HSM** - Hardware Security Modules for enterprise - **Mobile Wallets** - iOS/Android wallet apps - **Remote Signers** - Cloud-based signing services ## Signer Interface ```typescript interface Signer { // Get the public key (synchronous, cached) getPublicKey(): Uint8Array; // Sign a message (asynchronous, may require user interaction) sign(message: Uint8Array): Promise; // Sign for a specific public key (for multi-key wallets) signFor(publicKey: string, message: Uint8Array): Promise; } ``` ## Built-in Signers ### KeyPairSigner The default signer that wraps a KeyPair: ```typescript import { KeyPair, KeyPairSigner } from '@convex-world/convex-ts'; const keyPair = KeyPair.fromSeed(seed); const signer = new KeyPairSigner(keyPair); // Use with Convex client convex.setSigner(signer); convex.setAddress('#1678'); // Or use KeyPair directly (automatically wrapped) convex.setAccount('#1678', keyPair); ``` ## Custom Signer Implementation ### Basic Custom Signer ```typescript import { Signer } from '@convex-world/convex-ts'; class CustomSigner implements Signer { private publicKey: Uint8Array; constructor(publicKey: Uint8Array) { this.publicKey = publicKey; } getPublicKey(): Uint8Array { return this.publicKey; } async sign(message: Uint8Array): Promise { // Implement your signing logic const signature = await yourSigningFunction(message); return signature; } async signFor(publicKey: string, message: Uint8Array): Promise { // Verify the public key matches const ourPubKeyHex = Buffer.from(this.publicKey).toString('hex'); if (publicKey !== ourPubKeyHex) { throw new Error('Public key mismatch'); } return this.sign(message); } } // Use it const signer = new CustomSigner(publicKeyBytes); convex.setSigner(signer); convex.setAddress('#1678'); ``` ### Hardware Wallet Signer Example for a hardware wallet that requires user confirmation: ```typescript class HardwareWalletSigner implements Signer { private wallet: HardwareWalletDevice; private publicKey: Uint8Array; constructor(wallet: HardwareWalletDevice) { this.wallet = wallet; // Get and cache public key this.publicKey = wallet.getPublicKeySync(); } getPublicKey(): Uint8Array { // Return cached value (synchronous) return this.publicKey; } async sign(message: Uint8Array): Promise { // This will prompt user on hardware device console.log('Please confirm transaction on your hardware wallet...'); try { const signature = await this.wallet.signMessage(message); return signature; } catch (error) { if (error.code === 'USER_REJECTED') { throw new Error('Transaction rejected by user'); } throw error; } } async signFor(publicKey: string, message: Uint8Array): Promise { const ourPubKeyHex = Buffer.from(this.publicKey).toString('hex'); if (publicKey !== ourPubKeyHex) { throw new Error(`This hardware wallet does not control ${publicKey}`); } return this.sign(message); } } // Usage const wallet = await HardwareWalletDevice.connect(); const signer = new HardwareWalletSigner(wallet); convex.setSigner(signer); convex.setAddress('#1678'); // User will be prompted on device await convex.transfer('#456', 1_000_000_000); ``` ### Browser Extension Signer Example for a browser extension wallet: ```typescript class ExtensionWalletSigner implements Signer { private publicKey: Uint8Array; private extension: any; constructor(extension: any, publicKey: Uint8Array) { this.extension = extension; this.publicKey = publicKey; } static async connect(): Promise { // Connect to browser extension if (!(window as any).convexWallet) { throw new Error('Convex wallet extension not installed'); } const extension = (window as any).convexWallet; await extension.connect(); const accounts = await extension.getAccounts(); if (accounts.length === 0) { throw new Error('No accounts found in wallet'); } const publicKey = Buffer.from(accounts[0].publicKey, 'hex'); return new ExtensionWalletSigner(extension, publicKey); } getPublicKey(): Uint8Array { return this.publicKey; } async sign(message: Uint8Array): Promise { // Extension shows popup for user approval const messageHex = Buffer.from(message).toString('hex'); const signatureHex = await this.extension.signMessage(messageHex); return Buffer.from(signatureHex, 'hex'); } async signFor(publicKey: string, message: Uint8Array): Promise { const ourPubKeyHex = Buffer.from(this.publicKey).toString('hex'); if (publicKey !== ourPubKeyHex) { throw new Error('Public key mismatch'); } return this.sign(message); } } // Usage in browser const signer = await ExtensionWalletSigner.connect(); convex.setSigner(signer); // Get address from wallet const address = await window.convexWallet.getAddress(); convex.setAddress(address); // User approves in extension popup await convex.transfer('#456', 1_000_000_000); ``` ## Multi-Key Wallets Some wallets manage multiple keys. Use `signFor()` to specify which key to use: ```typescript class MultiKeyWalletSigner implements Signer { private keys: Map; private defaultKey: Uint8Array; constructor(keys: Array<{ publicKey: Uint8Array; privateKey: Uint8Array }>) { this.keys = new Map(); for (const key of keys) { const pubKeyHex = Buffer.from(key.publicKey).toString('hex'); this.keys.set(pubKeyHex, key.privateKey); } this.defaultKey = keys[0].publicKey; } getPublicKey(): Uint8Array { return this.defaultKey; } async sign(message: Uint8Array): Promise { const defaultKeyHex = Buffer.from(this.defaultKey).toString('hex'); return this.signFor(defaultKeyHex, message); } async signFor(publicKey: string, message: Uint8Array): Promise { const privateKey = this.keys.get(publicKey); if (!privateKey) { throw new Error(`No private key for ${publicKey}`); } // Sign with Ed25519 const signature = await sign(message, privateKey); return signature; } } // Usage with multiple addresses const signer = new MultiKeyWalletSigner([ { publicKey: pubKey1, privateKey: privKey1 }, { publicKey: pubKey2, privateKey: privKey2 } ]); convex.setSigner(signer); // Use first address convex.setAddress('#1678'); await convex.transfer('#456', 1_000_000_000); // Switch to second address (same signer) convex.setAddress('#9999'); await convex.transfer('#789', 2_000_000); ``` ## Remote Signing Service Example for cloud-based signing: ```typescript class RemoteSignerService implements Signer { private apiUrl: string; private apiKey: string; private publicKey: Uint8Array; constructor(apiUrl: string, apiKey: string, publicKey: Uint8Array) { this.apiUrl = apiUrl; this.apiKey = apiKey; this.publicKey = publicKey; } getPublicKey(): Uint8Array { return this.publicKey; } async sign(message: Uint8Array): Promise { const messageHex = Buffer.from(message).toString('hex'); const response = await fetch(`${this.apiUrl}/sign`, { method: 'POST', headers: { 'Authorization': `Bearer ${this.apiKey}`, 'Content-Type': 'application/json' }, body: JSON.stringify({ message: messageHex, publicKey: Buffer.from(this.publicKey).toString('hex') }) }); if (!response.ok) { throw new Error(`Signing failed: ${response.statusText}`); } const data = await response.json(); return Buffer.from(data.signature, 'hex'); } async signFor(publicKey: string, message: Uint8Array): Promise { const ourPubKeyHex = Buffer.from(this.publicKey).toString('hex'); if (publicKey !== ourPubKeyHex) { throw new Error('Public key mismatch'); } return this.sign(message); } } // Usage const signer = new RemoteSignerService( 'https://signing.example.com', process.env.API_KEY!, publicKeyBytes ); convex.setSigner(signer); convex.setAddress('#1678'); ``` ## Testing Signers ### Mock Signer for Tests ```typescript class MockSigner implements Signer { private publicKey: Uint8Array; private shouldFail: boolean; constructor(shouldFail = false) { // Generate random public key this.publicKey = crypto.getRandomValues(new Uint8Array(32)); this.shouldFail = shouldFail; } getPublicKey(): Uint8Array { return this.publicKey; } async sign(message: Uint8Array): Promise { if (this.shouldFail) { throw new Error('Mock signing failure'); } // Return mock signature return crypto.getRandomValues(new Uint8Array(64)); } async signFor(publicKey: string, message: Uint8Array): Promise { return this.sign(message); } } // Use in tests describe('Transaction tests', () => { it('should handle signing failures', async () => { const signer = new MockSigner(true); convex.setSigner(signer); await expect( convex.transfer('#456', 1_000_000_000) ).rejects.toThrow('Mock signing failure'); }); }); ``` ## Error Handling ### User Rejection ```typescript try { await convex.transfer('#456', 1_000_000_000); } catch (error) { if (error.message.includes('rejected')) { console.log('User cancelled the transaction'); } else if (error.message.includes('timeout')) { console.log('Signing timed out'); } else { console.error('Signing error:', error); } } ``` ### Timeout Protection ```typescript class TimeoutSigner implements Signer { private inner: Signer; private timeout: number; constructor(inner: Signer, timeout = 30000) { this.inner = inner; this.timeout = timeout; } getPublicKey(): Uint8Array { return this.inner.getPublicKey(); } async sign(message: Uint8Array): Promise { return Promise.race([ this.inner.sign(message), new Promise((_, reject) => setTimeout(() => reject(new Error('Signing timeout')), this.timeout) ) ]); } async signFor(publicKey: string, message: Uint8Array): Promise { return this.inner.signFor(publicKey, message); } } // Usage const hardwareSigner = new HardwareWalletSigner(wallet); const timeoutSigner = new TimeoutSigner(hardwareSigner, 60000); // 60s timeout convex.setSigner(timeoutSigner); ``` ## Best Practices ### ✅ Do - Cache public keys (they don't change) - Make signing operations async - Provide clear user feedback during signing - Implement proper error handling - Validate public key matches before signing - Use timeouts for user interactions ### ❌ Don't - Block the main thread during signing - Assume signing always succeeds - Ignore user rejection - Store private keys in custom signers (use secure storage) - Skip signature validation ## Next Steps - **[Accounts](./accounts)** — Account management - **[Transactions](./transactions)** — Submit transactions ## See Also - [Ed25519 Signatures](https://ed25519.cr.yp.to/) --- # Submitting Transactions Learn how to submit transactions to modify state on the Convex network. ## Overview Transactions are **state-modifying operations** that require: - An account address (e.g., `#1678`) - A cryptographic signer (key pair or hardware wallet) - Sufficient balance to pay for execution Unlike queries, transactions are recorded on the network and modify the global state. ## Prerequisites Before submitting transactions, you need: 1. **Account Address** - Your Convex account number 2. **Ed25519 Seed** - Your 32-byte private seed 3. **Balance** - Sufficient Convex Coins for transaction fees Set up your account: ```typescript import { Convex, KeyPair } from '@convex-world/convex-ts'; const convex = new Convex('https://peer.convex.live'); const keyPair = KeyPair.fromSeed(process.env.CONVEX_SEED!); convex.setAccount('#1678', keyPair); ``` ## Basic Transfer The `transfer()` method sends Convex Coins to another account: ```typescript // Transfer 1 Convex Coin (1,000,000,000 copper) const result = await convex.transfer('#456', 1_000_000_000); console.log('Transfer result:', result.value); ``` The second argument is a `BalanceLike` type, accepting `number`, `bigint`, or `string`: ```typescript await convex.transfer('#456', 1_000_000_000); // number await convex.transfer('#456', 1000000000000000000n); // bigint await convex.transfer('#456', '1000000000'); // string ``` :::info Copper vs Coins Amounts are in **copper coins** where: - 1 Convex Coin = 1,000,000,000 copper - Minimum amount = 1 copper ::: ## Executing Convex Lisp The `transact()` method accepts a **string of Convex Lisp source code** and executes it as a transaction: ```typescript // Deploy a function await convex.transact(` (def greet (fn [name] (str "Hello, " name "!"))) `); // Call the function const greeting = await convex.transact('(greet "Alice")'); console.log(greeting.value); // "Hello, Alice!" console.log(greeting.result); // "Hello, Alice!" (CVM printed representation) ``` ## Transaction Results Both `transact()` and `transfer()` return a `Result` object on success: ```typescript interface Result { value?: any; // JSON-converted CVM value result?: string; // CVM printed representation errorCode?: string; // Absent on success info?: ResultInfo; // Juice, fees, mem, trace, etc. } ``` Example: ```typescript const result = await convex.transact('(+ 1 2 3)'); console.log('Value:', result.value); // 6 console.log('Result:', result.result); // "6" console.log('Juice used:', result.info?.juice); ``` ## Error Handling When the CVM encounters an error during transaction execution, `transact()` and `transfer()` **throw a `ConvexError`**. There is no `status` field to check. ### CVM Errors ```typescript import { Convex, KeyPair, ConvexError } from '@convex-world/convex-ts'; try { await convex.transact('(transfer #456 999999999999)'); } catch (e) { if (e instanceof ConvexError) { console.error('CVM error code:', e.code); // "FUNDS", "STATE", etc. console.error('Juice used:', e.info?.juice); } } ``` ### Network Errors Network and connection issues throw standard JavaScript errors: ```typescript try { await convex.transact('(+ 1 2)'); } catch (error) { if (error instanceof ConvexError) { console.error('CVM error:', error.code); } else if (error instanceof Error) { if (error.message.includes('timeout')) { console.error('Network timeout - retry?'); } else if (error.message.includes('connect')) { console.error('Cannot connect to peer'); } else { console.error('Unexpected error:', error.message); } } } ``` ### Retry Logic ```typescript async function transactWithRetry( convex: Convex, code: string, maxRetries = 3 ): Promise { for (let attempt = 1; attempt <= maxRetries; attempt++) { try { return await convex.transact(code); } catch (error) { if (error instanceof ConvexError) throw error; // CVM errors are not retryable if (attempt === maxRetries) throw error; console.log(`Attempt ${attempt} failed, retrying...`); await new Promise(resolve => setTimeout(resolve, 1000 * attempt)); } } throw new Error('Unreachable'); } ``` ## Common Transaction Patterns ### Deploy Smart Contract ```typescript try { const contract = await convex.transact(` (do (def token-balance (let [balances {}] {:balances balances})) (defn transfer [to amount] (let [from *caller* from-bal (get-in token-balance [:balances from] 0) to-bal (get-in token-balance [:balances to] 0)] (cond (< from-bal amount) (fail "Insufficient balance") :else (do (assoc-in! token-balance [:balances from] (- from-bal amount)) (assoc-in! token-balance [:balances to] (+ to-bal amount)) {:success true}))))) `); console.log('Contract deployed:', contract.value); } catch (e) { if (e instanceof ConvexError) { console.error('Deploy failed:', e.code); } } ``` ### Call Smart Contract ```typescript try { const result = await convex.transact('(call #789 (transfer #456 1000))'); console.log('Contract call result:', result.value); } catch (e) { if (e instanceof ConvexError) { console.error('Contract call failed:', e.code); } } ``` ### Batch Operations ```typescript // Multiple operations in one transaction const result = await convex.transact(` (do (def user-data {:name "Alice" :level 5}) (def user-inventory [:sword :shield :potion]) (transfer #789 100000) {:user user-data :inventory user-inventory}) `); ``` ## Transaction Lifecycle 1. **Prepare** - Client creates transaction with sequence number 2. **Sign** - Transaction is signed with your private key 3. **Submit** - Signed transaction sent to peer 4. **Validate** - Peer validates signature and sequence 5. **Execute** - CVM executes the transaction code 6. **Consensus** - Transaction included in consensus 7. **Finalise** - Transaction permanently recorded ## Sequence Numbers Each account has a sequence number that increments with each transaction: ```typescript // Get current sequence const sequence = await convex.getSequence(); console.log('Next transaction sequence:', sequence); ``` :::tip Automatic Sequence Management The SDK automatically manages sequence numbers. You rarely need to inspect them manually. ::: ## Juice and Fees Transactions consume **juice** which is paid in Convex Coins: ```typescript // Check juice price const priceInfo = await convex.query('*juice-price*'); console.log('Juice price:', priceInfo.value); ``` After a transaction completes, you can inspect juice consumption via the result: ```typescript const result = await convex.transact('(transfer #456 1000000)'); console.log('Juice used:', result.info?.juice); console.log('Fees paid:', result.info?.fees); ``` ## Advanced Patterns ### Conditional Transfer ```typescript const result = await convex.transact(` (let [balance (balance *address*) threshold 10000000] (if (> balance threshold) (do (transfer #456 (- balance threshold)) {:transferred (- balance threshold)}) {:transferred 0})) `); ``` ### Time-locked Transaction ```typescript try { const result = await convex.transact(` (let [unlock-time 1735689600000] ; Unix timestamp (if (> (timestamp) unlock-time) (transfer #456 1000000) (fail "Funds locked until unlock time"))) `); console.log('Unlocked and transferred:', result.value); } catch (e) { if (e instanceof ConvexError) { console.error('Still locked:', e.code); } } ``` ### Multi-step Transaction ```typescript const result = await convex.transact(` (do ; Step 1: Validate (assert (> (balance *address*) 1000000) "Insufficient balance") ; Step 2: Transfer (transfer #456 500000) ; Step 3: Update state (def last-transfer (timestamp)) ; Step 4: Return receipt {:success true :amount 500000 :timestamp last-transfer}) `); ``` ## Best Practices **Do:** - Check balance before large transfers - Catch `ConvexError` and handle CVM error codes - Use meaningful error messages in your Convex Lisp code - Log transaction results for audit trails - Test on testnet first **Don't:** - Hardcode private seeds in source code - Ignore thrown errors from `transact()` or `transfer()` - Submit transactions in tight loops - Skip validation of user input - Pass objects to `transact()` -- it only accepts Convex Lisp strings ## Next Steps - **[Accounts](./accounts)** — Key pair and account management - **[Asset Handles](./assets)** — Fluent API for tokens and CNS - **[Signers](./signers)** — Hardware wallet integration - **[Convex Lisp](/docs/tutorial/convex-lisp)** — Learn the language --- # Faucet A **faucet** dispenses small amounts of Convex Coins on test networks, so you can experiment without acquiring coins. On the public testnet you rarely call it directly — the SDKs use it to fund new accounts for you (see the [Quick Start](/docs/tutorial/quickstart)). This page covers requesting funds explicitly. ## Availability - **Public testnet** — a faucet is available. The current endpoint is `https://mikera1337-convex-testnet.hf.space`. - **Local peer** — when you run your own peer for development, you control funding directly (see [Local Testnets](/docs/tutorial/peer-operations/local-testnets)). - **Production (Protonet, `peer.convex.live`)** — **no faucet.** Coins must be acquired legitimately (transfer from another account, an exchange, or network rewards). A peer may also have its faucet disabled. A faucet request to such a peer is refused with `403 Forbidden` — treat that as "no faucet on this network". > **Security note:** any service offering free *production* coins is a scam. Faucets exist only for testing. ## Requesting funds Amounts are always in **copper** (1 CVM = 1,000,000,000 copper). 0.1 CVM — 100,000,000 copper — is plenty for most testing. ### TypeScript ```typescript import { Convex, KeyPair } from '@convex-world/convex-ts'; const convex = new Convex('https://mikera1337-convex-testnet.hf.space'); // Fund a new account on creation: pass a faucet amount to createAccount const keyPair = KeyPair.generate(); const account = await convex.createAccount(keyPair.accountKey, 100_000_000); // 0.1 CVM // Top up an existing account await convex.faucet(account.address, 100_000_000); ``` ### Python ```python from convex_sdk import Convex, KeyPair convex = Convex('https://mikera1337-convex-testnet.hf.space') # Faucet-sponsored account creation (the peer's faucet pays the juice) key_pair = KeyPair() account = convex.create_account(key_pair) # Request funds for an account convex.request_funds(100_000_000, account) # 0.1 CVM # Or top up only when needed: requests funds until the balance reaches min_balance convex.topup_account(account, min_balance=10_000_000) ``` ## Account creation is faucet-sponsored Creating an account costs juice, which must be paid by an already-funded account. On the public testnet the peer's faucet account sponsors creation, so `createAccount` / `create_account` work from a fresh key with no funds. On production there is no faucet, so a new account must be created and funded by an existing account. ## Handling an unavailable faucet If the faucet is disabled or the peer refuses the request, the call fails (HTTP `403 Forbidden`). Handle it as "no faucet here" rather than a transient error: ```python from convex_sdk.exceptions import ConvexAPIError try: convex.request_funds(100_000_000, account) except ConvexAPIError as e: print(f'Faucet unavailable: {e}') # likely a production peer or a disabled faucet ``` ## Best practices - **Request only what you need** — 0.1 CVM covers most testing; you rarely need whole coins. - **Reuse funded accounts** — cache a funded test account across sessions rather than re-funding each run; `topup_account` (Python) makes this easy. - **Don't depend on faucet availability in scripts** — treat a `403` as expected on production or restricted peers. ## See Also - [Convex Coins Overview](index.md) — Convex Coin basics - [Quick Start](/docs/tutorial/quickstart) — the fastest path to a funded account - [TypeScript SDK — Quickstart](/docs/tutorial/client-sdks/typescript/quickstart) - [Python SDK — Quickstart](/docs/tutorial/client-sdks/python/quickstart) - [Local Testnets](/docs/tutorial/peer-operations/local-testnets) — run your own peer --- # Convex Coins ## About Convex Coins Convex Coins (CVM) are the native utility token of Convex. They are used to transact on the Convex Network. Convex Coins can be subdivided into one billion coppers. `1 Convex Coin = 1,000,000,000 coppers`. This means that you can use small fractions of a Convex Coins, which may be useful for precise calculations or micropayments. ## Receiving Coins You might receive Convex Coins for multiple reasons: - Purchasing from the release curve - Being awarded coins as a Convex contributor - Receiving a payment, e.g. for goods and services that you provide - Using a dApp that makes use of Convex Coins So how do you actually receive them? They are a digital asset, after all, and you can't put them in your pocket. To receive coins you need a **Convex account**. Convex accounts are recorded as part of the Convex global state, which is updated, verified and protected by peers of the decentralised network. Accounts hold a balance of Convex Coins and are designated with an address which looks like: ``` #23798 ``` If you have an account you use already, that's great: you can just provide the address and receive coins directly. The sender can use this address to make any transfer to you as required. ### Getting an Account Any user of the network can create an account for you. If you are new to Convex, the person or organisation sending you coins will normally do this for you. If you are a developer looking to build on Convex, hop on the Discord and a friendly person will probably give you one :-) It is important to secure the new account so that only you can access it. Accounts and the coins they contain are protected by strong cryptography, so to do this you will need to provide an **Ed25519 public key**. This is usually shown as a 32-byte hex string that looks like: ``` 0xe3B4746e32f32Eb7B1c6Af045dd7238ee5eea71B33c8b5CD91a22BcDAc5b0c54 ``` Account are *psuedonymous*: The account public key or address alone doesn't identify you as an individual, but it is possible that others may discover additional information that associates you with the account. ### Creating an Ed25519 key pair You can create an Ed25519 key pair any way you like - there are multiple tools and software solutions that can do this for you and help you to secure your private keys. One easy option is to use Convex Desktop. If you choose to create a key using Convex Desktop. The key generation screen looks like this: ![alt text](keygen.png) At the bottom you can see the key pair generated: - Private Ed25519 seed: `5b32f8ff94d1f901098fa41a4045449726849dda5e1a3de34ae123037418795c` (KEEP SECRET!!!) - Ed25519 Public key: `0xe3B4746e32f32Eb7B1c6Af045dd7238ee5eea71B33c8b5CD91a22BcDAc5b0c54` (OK to share!) To minimise risk, Convex Desktop does not save keys on your computer (unless you tell it to), so make sure you have taken an offline backup of the necessary information to recreate it. Typically you will want to keep: ``` Mnemonic: sing bomb stay manual powder hard north mixture sausage lunch retreat desert Passphrase: hello1234567890ZZ ``` Note: The BIP39 path is always `m/44/864/0/0/0` by default. You don't need to remember this if you use the default, but if you change this then make sure you back this up too! **DOUBLE CHECK**: Make sure you can independently recreate your key pair from the mnemonic and passphrase you have written down. ### Creating extra accounts Once you have an account, you might want more than one. This can be useful for multiple reasons: - Security: if you lose the key to one account, the others are safe - Separating accounts for different purposes, e.g. developer work vs. savings - Having an account that is shared with others with a limited balance - Privacy: You don't want all your transactions going through a single account Alternatively, you might want to create an account for someone else so that you can transfer coins to them. If you are a power user or developer, you can use the Convex REPL to create a new account. Example usage as follows: ```clojure ;; Create account with this desired public key. Make sure you use the PUBLIC key here!! (create-account 0xe3B4746e32f32Eb7B1c6Af045dd7238ee5eea71B33c8b5CD91a22BcDAc5b0c54) => #21568 ;; Send 100 Convex Coins to the new account ;; Note: 1 Convex Coin = 1000000000 coppers, so remember to add 9 zeros! (transfer #21568 100000000000) ``` When you create a new account, make sure you record the account number. While it is possible to go and look back at transaction history and/or search for accounts that have your public key if you forget it, it's a lot easier if you just write down the account number! ## Securing your Coins ### Basic rules NEVER give your private key (or the seed phrases used to create it) to anyone else. Anyone with access to these can sign transactions for your account, and steal your coins. As a wise man once said: Not your keys, not your coins! ALWAYS ensure you are signing transactions on a device that you control and trust. For small coin to medium balances, your laptop may be fine. If you are at higher risk, sign transactions only on an air-gapped laptop (i.e. not connected to the internet). ### Physical Backups If you have a significant holding of Convex coins, you may wish to keep a secure physical backup of your mnemonic phrase. Writing it down and putting it in a safe is an option, or you might choose to use something more durable like a metal seed phrase storage device (these are available relatively cheaply online). Here's an example of a backup of the example mnemonic used above: ![alt text](backup.jpg) Don't forget to backup your passphrase as well (that can more plausibly be something that you can remember, but still, plenty of people have lost Bitcoin due to forgetting it....). ### Custodial Accounts Some companies (e.g. exchanges) may offer custodial accounts on Convex. These accounts are similar to the self-custody account described above in that they have an address like `#82456` and can receive / send Convex coins, with two key differences: - The company providing the custody will keep the private key safe. You don't have it. - You can't execute transactions yourself: you will need to use the services of the custody provider (e.g. using their app) Custodial accounts have the advantage that you don't need to manage your own key security. This can be easier for many users, and can be very secure if the custodial provider manages the keys properly on your behalf. The risk is that you are dependent on the custody provider to keep the keys secure and faithfully execute transactions on your behalf. Only use custody providers that you trust! ### Account Controllers Accounts can optionally specify a `*controller*`, which is a facility that allows other accounts to control your account. Typical reasons to do this might include: - You have a secure fallback account that you can use to recover coins and other assets if you lose the private key to your regular account - You want to control your account via code (e.g. via a smart contract) - You want a trusted third party to have the ability to recover your account even if you lose your keys - An app may set itself as the controller so that it can restore / take actions in your account on your behalf This is a useful and powerful feature but it should be **used with caution**. Account controllers have full control of the account: they can change the keys, move funds etc. Do not set a controller unless you trust the controller account to act in your best interests. At the very least, you should expect the controller account to be **more secure** than the account that it controls. --- Convex Lisp is a powerful modern Lisp for decentralised systems. It supports immutable lattice data structures and functional programming for robust but flexible decentralised programs. You don't need to know Convex Lisp to use Convex, but it's a powerful tool for developers wishing to build dApps and digital assets. Most of the core Convex on-chain tools are written in and designed to be used from Convex Lisp. ## Where to start New to Convex Lisp? Work through these in order: - [Gentle Lisp Introduction](./lisp-guide.md) — start here; assumes no prior Lisp experience. - [Lisp on the CVM](./lisp-cvm.md) — how Convex Lisp runs on-chain. - [Advanced Guide](./lisp-guide-advanced.md) — macros, expanders, and metaprogramming. ## Why Lisp? The CVM is designed to be able to support many different languages and paradigms. However, we wanted a powerful language available to developers on Convex from the beginning, and chose Lisp as the first language on the CVM for a number of reasons: - **Immutability**: Lisp's functional programming paradigm promotes immutability, which is essential for ensuring the integrity and consistency of decentralised systems where data must be replicated and distributed across multiple nodes. - **Dynamic typing**: Lisp's dynamic typing allows for flexible data structures and easy modifications, crucial for adapting to the constantly evolving nature of decentralised systems. At the same time, Convex Lisp is **strongly typed**, avoiding the issues associated with weak typing. - **Code is data**: representing code using its own data structures (also known as "homoiconicity") gives Lisp advanced **metaprogramming** capabilities the creation of domain-specific languages (DSLs) tailored to specific concepts, making it easier to reason about and implement complex economic mechanisms. - **Interactive coding**: Lisp pioneered the use of the REPL for highly interactive programming. With Convex, we've taken this a step further with a complete on-chain compiler so that everything on the CVM can be done with simple interactive Lisp commands. ## Inspirations Lisp was originally created in the 1950s by John McCarthy and has maintained a deep-rooted appeal among hackers for its unique characteristics. Convex Lisp was heavily inspired by [Clojure](https://clojure.org/), from which it borrows most basic and many core functions, and [Racket Scheme](https://racket-lang.org/) which inspired many of the more advanced metaprogramming facilities. ## Interesting technical notes - Convex Lisp is a **Lisp-1** (i.e. keeps function and data values in the same namespace). This is simpler than a Lisp-2 and seems more appropriate for a functional programming language, where functions are regularly treated as first class values. - Convex Lisp is designed for **orthogonal persistence**. Developers don't need to care where data or code is stored: it is loaded and cached on demand. This powerful capability means that the CVM can operate code and data data structures much larger than system RAM without imposing any burden on developers. - Convex Lisp data structures (and therefore code) are all represented as **Merkle trees** using strong cryptographic hashes. This has the interesting property of making all code and data **content-addressable** and cryptographically **verifiable**. As well as making Convex Lisp ideally suited for running on the lattice technology used by Convex, this opens interesting possibility for new kinds of systems based on immutable and verifiable distributed code bases. --- In the Gentle Lisp Introduction we covered the basics of the Convex Lisp language. This guide builds on these basics to introduce the key ideas of programming on the Convex CVM. ## The Convex Virtual Machine The CVM is a decentralised, deterministic VM. Because it is deterministic, any replicas of the CVM that execute the same transactions from the same initial state will produce identical results. This is a key part how the Convex network is able to provide a consistent, programmable global state. It also enforces cryptographic security, so that only authorised users can make use of protected services. In practice, this means that developers can execute code on the CVM safely (including smart contracts and transactions involving valuable digital assets) in the knowledge that the results are guaranteed by a robust, fault-tolerant global network with no centralised point of failure and strong security. ## Accounts All code on the CVM is executed in the context of an account. Accounts are part of the global state of Convex. Accounts are numbered sequentially and referred to by an address which is denoted like `#123`. CVM code can find out what address is being used at any time: ``` ;; This is a special symbol that returns the current account address *address* => #123 ``` Accounts contain several important pieces of information in the global state. You can examine this information for any account on the CVM: ```clojure (account #11) => {:sequence 0, :key 0xcf345407332ab2b312c933377f13cc00b02e7ecfa59080b7f1f77a846800c4cf, :balance 388799997300000, :allowance 10000000, :holdings {#38 500000,#39 1581138830,#40 393701,#41 1403034590}, :controller nil, :environment {torus #19,fun #15}, :metadata {torus {:static true},fun {:static true}}, :parent nil} ``` A critical field to note is the `:key` that specifies the Ed25519 public key associated with the account. This key is used to validate digitally signed transactions to ensure that only the owner of the account (or other parties authorised by them) can execute code in the context of this account. This is essential for security, since accounts are able to control valuable digital assets owned by the account. ## Environments Every account has an environment that defines a mapping of symbols to values that have been defined in the account. When you execute a `def` command, you are actually creating a mapping in the environment. ```clojure (def foo 167) => 167 foo => 167 ``` Each environment is specific to a single account, but you can access symbols in other accounts by a lookup reference in the form `target-account/symbol` ```clojure ;; Assuming you have the symbol 'foo' defined, the following produce the same result (= foo *address*/foo) => true ;; If you attempt to resolve a symbol that doesn't exist in an account... #0/foo => Exception: :UNDECLARED foo ``` ## Lisp Machine By giving every account its own environment on the CVM, we have achieved something magical: every account is a personal, cryptographically secure, decentralised Lisp machine! As a simple example, here's a program that manages a database of "friend" accounts ```clojure (def friends #{}) ;; empty friend set to start :-( (defn add-friend [friend] (set! friends (conj friends (address friend)))) (defn un-friend [enemy] (set! friends (disj friends (address enemy)))) (add-friend #67) (add-friend #70) friends => #{#70 #67} ;; We have friends now! :-) ``` The significance of this capability cannot be understated: - Every account is a personal workspace where you can define your own scripts, tools and utilities for managing decentralised assets - You can write, run and modify entire decentralised programs on-chain - You can store and manage arbitrary data - Everything can be done interactively with simple REPL commands - no other tools required! ## Special Symbols You might notice symbols like `*address*`, conventionally surrounded with asterisks. These are *special symbols* which get special treatment by the CVM. They are not static values, but dynamically calculated on demand by the CVM. Commonly used ones are: - `*address*` the address of the current account, e.g. `#15656` - `*balance*` the Convex Coin balance of the account - `*juice-price*` the price (in Convex coppers) of each unit of juice - `*caller*` the address which made a `call` to the current address (may be `nil`) - `*origin*` the address of the origin account for the transaction - `*timestamp*` the unix timestamp of the current CVM state (= the time of the most recent block which started processing) - `*controller*` an account (can be `nil`) with the authority to control this account ## Actors So far, we've looked at accounts controlled by users. But accounts can also be CVM programs that are independent of any users. We call these actors in Convex because they act and respond autonomously in accordance with their code. Actors are critical because they can serve as trusted services shared by all users. This is often done so that actors can enforce **smart contracts**: self-executing contracts that are guaranteed to behave in predictable ways. Actors each get their own account on Convex. So an actor implementing a smart contract might be located at account address `#1033`. ### Creating an actor To create an actor, you need to deploy some code to initialise the actor. The code is executed immediately after a new account for the actor is created and can be used to set up the environment of the actor, e.g. defining new values or functions. ```clojure ;; Deploy an actor, returning the address of the new actor account (deploy '(def some-data "Hello")) => #1033 ;; This is undeclared, since some-data is in the actor's environment some-data => UNDECLARED ;; However, we can look up the data in another account: #1033/some-data => "Hello" ``` Your initialisation code *MUST* set up any capabilities you want the actor to have in the future: once deployed, you may not be able to make any further changes if you make a mistake (although it is possible to make an actor upgradable... more on this later). ### Calling actor functions Actors are more than just containers for data - they can be active participants in transactions. To create an actor that exposes executable functionality to others, you need to make functions `:callable`. The following example is an actor that allows callers to get and set a value ```clojure ;; define code for our Actor (def actor-code '(do (def value :initial-value) ;; stateful data definition for this actor (defn ^:callable set [v] (set! value v)) ;; note: `set!` fails if `value` is not defined (defn ^:callable get [] value))) ;; Deploy the Actor and store the address as 'act' for convenient use later (def act (deploy actor-code)) ;; Call 'get' (call act (get)) => :initial-value ;; Call 'set' with a new value (call act (set :new-value)) ;; Call 'get' again (call act (get)) => :new-value ``` This actor is pretty simple, but it demonstrates the key ideas: - An actor is an autonomous program, with its own execution environment - You can make callable functions to allow users to interact with an actor Note you can also read the actor account's data directly by lookup: ```clojure act/value => :new-value ``` This works, but is not recommended: you are making an assumption about how the actor is internally structured which might break if the actor ever gets updated. It is better to use a public `:callable` API to minimise this risk. ### Sending funds to actors Like users, actor accounts can have their own balance of Convex Coins. You can use the `transfer` function to transfer funds to an Actor. However, this causes a problem: what if the actor doesn't expect to receive funds, and there is no a facility to transfer the funds elsewhere? This can cause coins to be irrevocably lost. The better way to transfer funds is to "offer" them to the actor you are calling, which then has to actively `accept` the funds to acknowledge receipt. Thus, if coded correctly, there is no risk of funds being transferred that the receiving actor is unable to handle. Below is a simple example of an actor that accepts funds, keeps track of how much been donated to each charitable cause, and provides a payout mechanism to relay the funds to the given cause. ```clojure (def charity-box (deploy '( ;; a map of causes to donation amounts (def all-donations {}) (defn ^:callable donate [cause] (let [donation *offer*] (if (> donation 0) (let [prev-donations (get all-donations cause 0)] (accept donation) ;; take the offered amount (set! all-donations (assoc all-donations cause (+ prev-donations donation))) (return "Thanks for your donation")) (fail :FUNDS "No donation offered!")))) (defn ^:callable collect [cause] (if (@convex.trust/trusted? cause *caller*) (let [amt (get all-donations cause 0)] ;; We need to clear the donations for the cause (set! all-donations (dissoc all-donations cause)) ;; CEI Pattern implies interactions go last (transfer *caller* amt)) (fail :TRUST "Not authorised to collect funds"))) ;; end of actor code ))) ``` To use this actor, it should be called with the offer amount as an extra parameter to `call`: ```clojure ;; A charity address that you want to be the beneficiary of donations (def charity #2055) ;; Donate 0.0001 Convex Gold to charity via an offer (2nd argument to `call`) (call charity-box 100000 (donate charity)) => "Thanks for your donation" ;; Sneakily look at how much is donated so far to each cause! charity-box/all-donations => {#2055 100000} ``` This actor also make use of the powerful `convex.trust` library to control who is allowed to collect funds. The `cause` is actually a trust monitor that verifies whether a caller is authorised to make a collection of the donated funds. Unauthorised attempts will get rejected: ```clojure ;; try to collect funds (call charity-box (collect charity)) => Exception: :TRUST Not authorised to collect funds ``` However if the account `#2055` itself attempted to collect the funds, it would receive the full donated amount, since `convex.trust` specified that an account always trusts itself by default: ```clojure ;; try to collect funds as account #2055 (call charity-box (collect #2055)) => 100000 ``` ### Important security note Actor code runs in the account of the actor itself. In many circumstances, calling actor code can be considered "safe" in the sense that it cannot in general access assets owned by the calling account. However, there are some risks that you should be aware of: - If you make an actor call, you are still liable for paying any transaction fees (and memory usage) associated with running actor code. If this is a concern, you should evaluate the actor code to determine if there is any risk of high transaction fees (or set an appropriate transaction fee limit). - An actor may call other actors. This can open up a "reentrancy attack" if the Actor calls back into other code that you were not expecting (may change the state of other actors for example) and invalidate some assumptions about the state of other Actors that you previously made. If you consider this a risk, calling an actor, especially an unknown / untrusted one, should usually be the *last* thing you do in a transaction. - An actor may "accept" Convex Coins or other digital assets that have been offered to it. Only offer assets to an actor if you are comfortable that the actor will only attempt to claim these assets legitimately. ## Libraries In most programming environments, it is helpful to bundle up code into libraries that can be shared and re-used. Convex Lisp is no exception, but takes a novel approach on the CVM: Libraries are just actor accounts that don't do anything other than provide usable code. This approach is powerful because: - We make use of Convex as a global repository for libraries - You can deploy libraries in the same way as you deploy actors - no special tools or treatment needed! - Libraries get all the same security and management guarantees as actors - You can use library functionality to access actors and vice versa ### Using libraries Using libraries is easy! All you need to do is call code in the library account: ```clojure (#567858/some-function arg1 arg2) ``` If the library is registered in CNS, you can also use its CNS name symbol to resolve the address: ```clojure (@convex.fungible/balance MY-TOKEN) ``` If you use a library regularly, you may find it convenient to import it and give in an alias (which is just a definition in your environment that points to the library address). - `import` the library using its Address or CNS name and give it a convenient alias e.g. `foo` - Use symbols defined in the library by prefixing the symbol name with the alias e.g. `foo/bar` ```clojure ;; Import a library (in this case, the standard registry Actor) (import convex.registry :as reg) ;; Use a symbol from the library (in this case, count the number of registered accounts) (count reg/registry) => 67081 ``` ### Deploying libraries Deploying libraries is like deploying an Actor, with a few key differences to note: - You don't need to make any functions `:callable` ```clojure (def my-lib-address (deploy '(defn distance [x y] (sqrt (+ (* x x) (* y y)))))) (import my-lib-address :as my-lib) (my-lib/distance 3.0 4.0) => 5.0 ``` ### Important security note A key difference between a `call` to an actor function and running library code is the difference in *security context*: - An actor `call` runs code in the actor's account and environment, with the actor itself as the current `*address*` (and the calling account as `*caller*`) - Library code runs in the environment of the current account, i.e. `*address*` is unchanged **DO NOT RUN LIBRARY CODE YOU DO NOT TRUST**. Library code executed in your account can do anything that your account can, including transferring away all your coins and tokens, or entering into arbitrary smart contracts on your behalf. If you have any doubt about the trustworthiness of library code, do not use it from an account that controls any valuable assets or resources. If you don't trust a library but also don't expect it to make any state changes, you can wrap a call in `query` which will roll back any state changes made and just return the result. This is mostly safe since any malicious state changes will be undone (though beware that the library might still return malicious results, or deliberately burn juice to hit you with transactions fees...) ```clojure (query (untrusted-library/dangerous-function)) => "This is the result" ``` --- If you've got this far, you may be interested in some of the more advanced features of Convex Lisp. This section is intended for people who want to know more about how Convex Lisp works. ## Compiler Phases How does 'Code as Data' actually work? The secret is in understanding the phases of the Convex Lisp compiler. ### 1. Reading Reading is the first phase that parses source code as text into CVM data structures (technically known as a *form*, since it is data that is structured to be used as code) ```clojure "(foo :bar :baz)" -> '(foo :bar :baz) ``` The Convex Lisp reader is part of the Convex platform code but outside the scope of the CVM - there's no good reason for doing parsing on-chain when it can be performed easily and cheaply by clients. This means that you can't parse code from Strings on-chain, but this isn't a significant limitation: you can just parse off-chain and pass in the resulting data structure (skipping the reading phase) ### 2. Expansion Expansion is the second phase of the compiler. Expansion takes the raw form data structures and translates them into expanded forms, which are a representation of the Convex Lisp Abstract Syntax Tree (AST) ```clojure ;; `if` is a macro that gets expanded to `cond` (expand '(if :bar :baz)) => (cond :bar :baz) ``` In this phase, any macros are applied to the forms analysed, which has the effect of replacing them with the macro expansion. This means that arbitrary CVM code in macros *can* be executed during expansion - which in turn can be sometimes useful, e.g. in smart contract code that wishes to generate code based on analysing the CVM state. Expansion can be performed either on-chain (with the `expand` core function) or off-chain. ### 3. Compilation In the third phase, forms are *compiled* into *Ops*, which are the low-level instructions that can be executed by the CVM. ```clojure -> Op ``` There are only a small number of Op types on the CVM, which are roughly based on the fundamental operations required to implement the [Lambda Calculus](https://en.wikipedia.org/wiki/Lambda_calculus). Important ones are: - **Cond** - Performs a conditional branch - **Constant** - Loads a constant value - **Def** - Creates a definition in the environment - **Do** - Executes a structured sequence of Ops sequentially - **Invoke** - Executes a function with optional arguments - **Lambda** - Creates a function - **Let** - Defines a scope for local variables - **Lookup** - Looks up a value from a definition in the environment Ops can be nested, e.g. an Op of type **Do** may contain multiple child Ops. In this way, single Ops can be used to represent whole programs or algorithms. Normally, users don't need to interact directly with Ops. There are cases where it may be marginally more efficient to construct Ops off-chain and send them directly for execution, but this is an optimisation probably only worthwhile for applications doing very large numbers of transactions. ### 4. Execution The final phase is execution, where Ops are executed in the CVM context. The Op may update the CVM state in various ways, and it may also return a result, so the process of Op execution can be informally viewed as a state update: ``` + => + ``` Results from Op execution can be either: - A valid CVM data object - An exceptional result (e.g. an error or early return value) Convex Ops are technically a form of [p-code](https://en.wikipedia.org/wiki/P-code_machine), analogous in many ways to Java bytecode. Using Ops gives a few big advantages: - Ops can be executed efficiently many times (avoiding the more expensive phases of parsing, expansion and compilation). - Ops are very compact in terms of memory used - making them ideal for network transmission and efficient use of on-chain storage. - We can improve the underlying performance and implementation details of the CVM without breaking CVM code that has been compiled to Ops. - Ops are designed to match up with the runtime and security checks that the CVM must perform when executing code securely on-chain. ## Destructuring It is common that data is passed in a data structure, and you wish to access specific elements of the structure. Convex supports basic destructuring of sequential data structures: ```clojure (defn name [user] (let [[name age attributes & flags] user] name)) (name ["Bob" 67 {:favourite-colour "Green"} :some-extra-flag]) => "Bob" ``` The `&` symbol can be user to indicate an arbitrary number of following elements. Such elements are bound as a single vector. ```clojure (defn restargs [_ & more] more) (restargs 1 2 3 4) => [2 3 4] ``` The `_` symbol is used to ignore an argument. Nothing is bound for the corresponding position. While often convenient, destructuring can make code harder to read, so use judiciously. ## Macros We've actually used a couple of macros already in this guide: `if` and `defn` are both examples of macros. A macro is a procedure that generates new code at compile time (technically, in the *expansion* phase of the compiler). Macros are an incredibly powerful tool that allows you to enhance the Convex Lisp language with new capabilities and syntax. As a simple example, let's consider a macro that allows you to use 'infix' notation for mathematical expressions, i.e. instead of writing `(+ 1 2)` we want to write `1 + 2`. It is possible to do this with a simple macro that rewrites the infix expression into the expected Lisp format: ```clojure (defmacro infix [arg1 operator arg2] (list operator arg1 arg2)) (infix 1 + 3) => 4 ``` What is happening here? The macro defines an expander function that takes three arguments `[arg1 operator arg2]` and then outputs a list starting with the operator. This transforms `1 + 3` into the list `(+ 1 3)` which can then be executed normally. We can see the effect of macro expansion by using the `expand` function, which performs the expansion of a form without evaluating it: ```clojure (expand '(infix 1 + 3)) => (+ 1 3) ``` Macros are powerful tools, but should only be used when needed - they are more complicated to use and understand than regular functions. The best use cases for macros are usually: - Writing new syntax / language extensions that need to make use of arguments *without* evaluating them beforehand. If you are happy to use arguments after regular evaluation, then regular functions are probably a better fit. - Situations where you want code to be evaluated at compile-time, e.g. to avoid repeatedly performing the same expensive computation at runtime. ## Transactions and state rollback The CVM is a state machine, and the execution of ops often leads to changes in the CVM state. Usually this is desirable (because you want a digital asset transfer to be executed, for example) however sometimes you need a greater level of control for security or integrity reasons. ### Atomic transactions A transaction on Convex either succeeds (with a result value) or fails (with an error code and optional message / metadata). If a transaction fails then *all state changes that happen within the transaction* are rolled back. Transactions are therefore **atomic** from the perspective of any external observer. This is important because it prevents a situation where a transaction is only partially completed. The only state changes that can happen in relation to a failed transaction are those external to the transaction itself (e.g. juice accounting). ### Actor calls Calling an actor creates a new execution context (effectively a fork of the CVM). Like an overall transaction, this will either succeed or fail. Again, if there is any failure, all CVM changes made within the scope of the `call` are rolled back. ```clojure ;; an actor with a callable function that changes state but cannot succeed (def bad-actor (deploy '(defn ^:callable will-fail [] (def test 10) ;; make a state change first (fail) ;; then fail ))) ;; the call fails as expected (call bad-actor (will-fail)) => Exception: :ASSERT nil ;; this fails because the definition of `test` has been rolled back bad-actor/test => Exception: :UNDECLARED test ``` ### `rollback` instruction You can terminate a transaction or actor call with `(rollback :value)`, which causes the current execution context to stop with the given value as a result. This returns the transaction to the `*caller*` of the current actor, or terminates the transaction if there is no `*caller*` (i.e. we are in a top level transaction). This is typically used when unexpected has happened, or an attempt to perform some work failed, but you do not want to throw an error. Presumably the caller will know what they want to do in this situation. ### `halt` instruction You can terminate a transaction or actor call with `(halt :value)`, which causes the current execution context to stop with the given value as a result, but keep any CVM state changes. This returns the transaction to the `*caller*` of the current actor, or terminates the transaction if there is no `*caller*` (i.e. we are in a top level transaction). This is typically used to exit processing early when everything is successfully completed and the desired result is known. ### `query` expression Often you want to execute some code to determine the result, but do not want any CVM changes to occur. You can do this by wrapping any code in `(query ...)`. The example below ```clojure ;; if I deployed a new actor, what address would I get? (def next-actor-address (query (deploy :this-is-not-important))) => #68796 ;; The new actor account wan't actually created! (account next-actor-address) => nil ``` Note that code within queries may still fail, with the error propagated back to the calling code. `query` is particularly useful when executing code that *shouldn't* make state changes but you don't entirely trust it. A typical example would be calling a 3rd party actor to read some data - it protects you against unintended or malicious CVM state changes in what should be a read-only operation. Such calls might also open up security vulnerabilities like reentrancy attacks if the actor calls back into your code, but again `query` protects you, because any such changes are rolled back and it doesn't matter even if an attacker does manage to compromise your code: they can't do anything. ## Exception Handling **WARNING**: error handling on the CVM is a risky business. It is always safer to fail a transaction than to attempt to handle an error - so only do this if you really know what you are doing ### `try` expressions The CVM supports a `try` expression similar to many general purpose languages that support exception handling. The semantics of `try` are: 1. Attempt the first expression 2. If the expression succeeds, return its result and finish (including any CVM state changes) 3. If expression fails with a catchable error **roll back** any CVM state changes and proceed 4. If more expressions exist, continue to attempt each expression in turn as in 2. above 5. If all expressions fail, return the result of the last expression (which could be an error) Example: ```clojure (try (+ :foo :bar) ;; this will fail due to bad argument types :ALTERNATIVE) => :ALTERNATIVE ``` ### Rollback behaviour The atomic rollback feature of `try` is critical for smart contract safety. In the event of failure, the CVM state will be as it was before the failing expression. For example, the following code performs some digital asset transactions (which typically involved nested calls to actors that modify CVM state). If any one of these fails, the entire `do` block is atomically rolled back before `alternative-handling` is attempted. ```clojure (try (do (asset-transfer-1) (asset-transfer-2) (asset-transfer-3)) (alternative-handling)) ``` This pattern of ensuring a set of actions either all succeed or are all rolled back is quite common in more sophisticated smart contract code. In effect, the `try` block lets you attempt an atomic sub-transaction. You can furthermore wrap code in `query` to discard CVM state changes even in the case of success. This enables speculative execution of arbitrary code, even with `eval` on untrusted code: ```clojure (defn would-code-succeed? [dangerous-code] (try (query (eval dangerous-code) true) false)) ;; assuming you have 1 gold at least (would-code-succeed? '(transfer #11 1000000000)) => true ;; A transfer of this size is impossible (would-code-succeed? '(transfer #11 1000000000000000001)) => false ``` The most you can lose here is the value of your juice: the transfer (or any other state changes) won't actually happen. ### Uncatchable errors Note: Some CVM errors are impossible to recover from: such errors are regarded as *uncatchable*. This usually isn't a concern because there is nothing you can do anyway to continue effectively. e.g. `:JUICE` failures are pointless to catch because any error handling code will immediately also fail due to `:JUICE`. --- This guide is for developers interested in learning the basics of Convex Lisp. We assume a general familiarity with programming concepts, but no prior experience in Lisp. We will take you through the basics of the language. Veteran Lisp hackers may wish to skip this section, though there are some unique features in Convex Lisp worth noting. ## Setup The easiest way to experience Convex Lisp is the **[Sandbox](https://convex.world/sandbox)** — a live REPL in your browser, with no installation. We recommend trying the examples as you go: instant feedback makes it much more fun to experiment. Create a free, anonymous account with a single click, then type the example code into the input window. Throughout this guide we use `=>` to indicate the expected output. You can also use **[Convex Desktop](https://docs.convex.world/docs/products/convex-desktop)** and enter code via the Client Terminal: ![Convex Desktop Client Terminal](/img/convex-desktop-client-terminal.png) Cut and paste into the lower part of the screen, like this: ![Entering code in the Client Terminal](/img/convex-desktop-client-terminal-input.png) ## Expressions All Lisp code is constructed from expressions. Expressions are evaluated to get a result (or maybe an error, if something went wrong...). The classic Lisp expression is a list enclosed in parentheses `(...)` where the first element of the list is the function to be called and the following elements are the arguments. So to add two numbers with the `+` function you would do something like: ```clojure (+ 2 3) => 5 ``` Each element in the expression is itself an expression. It's expressions all the way down. So you can nest expressions arbitrarily to create more complex structures: ```clojure (- (* 10 10) (* 5 5)) => 75 ``` There are many different types of expressions (many of which are introduced in this guide). But the syntax is of Lisp is ultimately just a tree of nested expressions. It is this simplicity and consistency which gives Lisp its power. **NOTE**: Lisp expressions use whitespace as separators. Whitespace includes spaces, tabs, commas and carriage returns so you can format expressions as you like. ## Literals The simplest type of expression is a constant literal data value, which evaluates directly to itself! If you type the number `1` in the REPL and execute it, the result is simply the number `1` itself: ```clojure 1 => 1 ``` Convex can handle double precision floating point numbers, which work the same way: ```clojure 1.5 => 1.5 ``` Strings can be used as literals by enclosing them in double quotes: ```clojure "Hello World!" => "Hello World!" ``` Individual characters can be used as literals by preceding them with a backslash (`\`). You can also specify 16-bit unicode characters in the form `\uXXXX` where XXXX is a 4-character hex string. ```clojure \a => a \u0065 => e ``` Keywords are special literal values that are intended for use as keys in hash maps, sets, etc. They can also be conveniently used as field names in records, as special unique marker values, or as a member of a defined set of values like "enums" in other languages. ```clojure :foo => :foo ``` The special values `true` and `false` are the two usual Boolean values: ```clojure true => true false => false ``` The special value `nil` is considered as the empty / missing value. It is also considered the same as `false` when used in conditional expressions which is often surprisingly useful: more on that later! ```clojure nil => nil ``` Addresses (which refer to accounts) can be expressed as a literal starting with `#`. Address literals need not refer to an account that actually exists. ```clojure #12345 => #12345 ``` Finally, there is also support for byte data encoded in hexadecimal (we call these "Blob literals" because they can technically be arbitrary Binary Large Objects). Any hex string with an even number of digits is valid. ```clojure 0xff1234 => 0xff1234 ;; NOTE: This is OK, and results in a zero-length Blob 0x => 0x ``` Blob literals are somewhat unusual as a data type, but are very convenient for many reasons in Convex: specifying addresses of users or smart contracts, validating cryptographic hashes against exact values etc. Using blob literals directly is also much more efficient than encoding/decoding binary data in some other format such as hex strings. ## Symbols Symbols are named references to value in the Convex programming environment. When used as expressions, they look up the value that they refer to in the current context. Usually, you would first use `def` to create a new value in the environment. ```clojure ;; Define a symbol with 'def' (def a 100) => 100 ;; 'a' now refers to 100 in the current environment a => 100 ``` If you try to evaluate a symbol that has no corresponding definition in the environment, you will get an UNDECLARED error: ```clojure bad => ERROR (UNDECLARED) => 'bad' is undeclared. ``` Some *special symbols* are provided by Convex to make it easier to access special values provided by the CVM. By convention, and to make them stand out when reading Convex Lisp code, these symbol names start and end with asterisks (`*`). ```clojure ;; Get the available balance of the current Account via the special symbol '*balance*' *balance* => 97996220 ;; Get the Address of the current Account via the special symbol '*address*' *address* => #123 ``` ## Functions Functions in Convex Lisp are the fundamental objects that represent computation: algorithms that can be applied to transform input data into output data. ### Function application Functions can be called in an expression by placing the function name in a list before the arguments to a function. Usually, the function is specified by a Symbol: ```clojure ;; Call the 'inc' function which adds 1 to an integer value. (inc 10) => 11 ``` This construct of applying a function by forming an expression with the function at the beginning of a list followed by its arguments is classic Lisp syntax. This may be surprising to people who are new to Lisp but are used to languages such as C, Java or JavaScript. If it helps, you can think of simply moving the opening parenthesis of the argument list before the function name: ```clojure // In a C-like language inc(10) ;; In Lisp (inc 10) ``` Why do we do this? It turns out that being able to express the whole function application expression as a list is extremely useful for more advanced techniques such as macros and code generation. A topic for later. ### Variable arity Lisp function often allow variable arity, i.e. you can pass a variable number of arguments. This is often more concise and readable than nesting multiple functions. A good example is the core `str` function for assembling strings: ```clojure (str "Hello" " :: " "Bob" " :: " 42) => "Hello :: Bob :: 42" ``` If you pass an illegal number of arguments, you will get an `:ARITY` error: ```clojure (count [1 2 3] [5 6]) Exception: :ARITY count requires arity 1 but called with: 2 ``` ### The Core library The Convex core runtime library provides a wide variety of useful functions. Some simple examples to try out: ```clojure ;; Addition: '+' is a variable arity function that can take multiple arguments (+ 1 2 3) => 6 ;; There are several predicate functions that test values and return a boolean ;; e.g. 'str?' tests if the argument is a String (str? "Hello") => true ``` ### Defining functions You can easily define your own functions with `defn`: ```clojure ;; Define a 'square' function which multiplies its argument by itself (defn square [x] (* x x)) ;; Apply the new function (square 111) => 12321 ``` You can also create anonymous functions and use them directly with the `fn` special form. The function below is equivalent to the `square` example above, but we can use it without needing to give it a symbolic name. ```clojure ((fn [x] (* x x)) 111) => 12321 ``` ## Data structures Convex provides a powerful set of data structures as part of the CVM. In fact, one of the reasons Convex performs so well is due to the power of the data structures. All Convex data structures are *immutable* - functions that make a change to a data structure actually create a new data structure. There are some clever tricks that mean that most of the data in large data structures don't need to be cloned, which makes this extremely fast. #### Vectors A `Vector` is an ordered sequence of values. You can create a vector by enclosing any list of expressions with square brackets `[...]` ```clojure ;; A vector containing the numbers 1, 2 and 3 [1 2 3] => [1 2 3] ;; The empty vector [] => [] ;; A vector generated by evaluating two expressions [(+ 1 2) (+ 3 4)] => [3 7] ;; Vectors can contain arbitrary element types, including other nested vectors [1 :foo ["Hello" true]] => [1 :foo ["Hello" true]] ``` There are many functions in the core library that work with Vectors. Some simple examples: ```clojure ;; Get an element from a vector at the specified index (get [:foo :bar :baz] 1) => :bar ;; Test if a value is actually a vector (vector? [1 2]) => true ;; Concatenate two vectors (concat [1 2] [3 4]) => [1 2 3 4] ;; Add a new element to a vector (at the end) (conj [1 2] 3) => [1 2 3] ``` In general, you should use Vectors whenever you need to store an ordered sequence of values. They are the fastest data structure for indexed lookup, and for appending elements to the end with `conj`. Vectors are the natural Convex equivalent to what is often called "arrays" or "tuples" in other languages. #### Maps A map associates a finite set of keys with a value for each. You can create a map as a literal value by enclosing any list of expressions with curly braces `{...}` ```clojure ;; A map with two key/value pairs {:foo 1 :bar 2} => {:foo 1 :bar 2} ;; An empty map {} => {} ``` Maps are designed for efficient lookup of values based on keys. If the specified key does not exist, either nil or an optional 'not found' value can be returned. ```clojure ;; Get the value from a map for a specified key (get {:foo 1 :bar 2} :foo) => 1 ;; Get a key that doesn't exist (get {:foo 1 :bar 2} :batman) => nil ;; Get using an optional 'not-found' result (get {:foo 1 :bar 2} :batman :MISSING) => :MISSING ``` You can also use a map as a function! This can be convenient since it can save you from writing boilerplate code just to lookup up values in a map. ```clojure ;; Define a map in the environment with the symbolic name 'my-map' (def my-map {:foo 13 :bar 23 :baz 41}) ;; Use 'my-map' to look up values as if it is a function (my-map :baz) => 41 ``` There are a variety of useful functions in the core library that are designed to work with maps. Some examples to try: ```clojure ;; Update a map with a new key / value association using 'assoc' (assoc {:foo 1 :bar 2} :baz 3) => {:baz 3 :foo 1 :bar 2} ;; Remove a key/value pair from a map with 'dissoc' (dissoc {1 2 3 4} 1) => {3 4} ;; Count the number of key/value pairs in a map (count {:foo 1 :bar 2}) => 2 ;; Get a vector of keys for the map (keys {:foo 1 :bar 2}) => [:foo :bar] ``` In general, you should use Maps whenever you need to look up values with a specific key, and the order doesn't matter. Maps support a very efficient lookup by key. `assoc` and `dissoc` are also very efficient. #### Sets Sets are an unordered collection of values. You can create a set as a literal value by enclosing any list of expressions with a hash symbol followed by curly braces `#{...}` ```clojure ;; A set of 3 numbers #{1 2 3} => #{1 2 3} ;; The empty set #{} ``` The most common operation with a set is to test whether it contains a specific value. ```clojure ;; 'get' returns the value from a set if it is present, or nil otherwise (get #{1 2} 2) => 2 (get #{1 2} 3) => nil ;; An optional 'not-found' value can also be added (get #{1 2} 3 :OOPS) => :OOPS ``` You can also use a set as a function, in which case it will return a boolean value indicating whether the set contains the specified argument. ```clojure (def my-set #{1 2 3}) (my-set 1) => true (my-set 10) => false ``` Some examples of functions from the core library that work with sets: ```clojure ;; Add a value to a set ;; NOTE: when displaying the elements of the resulting set, the order is not guaranteed (conj #{1 2 3} 4) => #{1 2 3 4} ;; Use 'into' to add a sequence of extra elements to a set (will de-duplicate automatically) (into #{1 2 3} [3 4 5]) => #{1 2 3 4 5} ;; Remove an element from a set with 'disj' (disj #{1 2 3} 2) => #{1 3} ``` #### Lists Lists are ordered sequences of elements, just like Vectors. However, Lists are specially designed to be used for representing code. If you enter a List directly in the Sandbox, it will get evaluated as an expression: ```clojure (inc 10) => 11 ``` This is helpful for executing code, but it is not useful if you want to use Lists as a data structure! If you want to stop a List from being automatically evaluated, you can *quote* the List by adding the character `'` before this list. This tells Convex to interpret the list as a literal data structure: ```clojure '(inc 10) => (inc 10) ``` Some other ways of constructing a List: ```clojure ;; The empty list works directly as a literal () => () ;; Create a List using 'cons' which adds a value to the front of any sequential collection (cons 'a '(b c)) => (a b c) ;; Assemble a List by concatenating two lists (concat '(this is) '(a test)) => (this is a test) ``` You should use vectors over lists for storing data in most cases. Lists are mainly be used for generating code - in macros, for example. ## Conditionals General-purpose Turing complete languages need some way of controlling conditional execution of code, and Convex Lisp is no exception. Convex Lisp provides an `if` macro that evaluates a conditional expression and then executes one of two other expressions depending on whether the value of the first is true or false. ```clojure ;; A simple if expression that always takes the 'true' branch (if true 20 30) => 20 ;; A simple if expression that always takes the 'false' branch (if false 20 30) => 30 ;; If no false branch is provided, the 'if' expression returns 'nil' in this case (if false 20) => nil ``` Conditionals branch based on whether the conditional expression evaluated is truthy or falsey. - A value is considered **falsey** if it is either the boolean value `false` or `nil` - Any other value is considered **truthy**, including the boolean value `true` but also `[]`, `1`, `:foo` etc. Why? It turns out that in many situations, you want to branch based on whether a result is `nil` or non-`nil` (e.g. when you look up a key in a database, `nil` may represent the absence of a value). We could force developers to do a `(nil? x)` check, but this adds overhead and boilerplate code. Instead, we make conditionals work with truthy and falsey values directly, so that such conversion code becomes unnecessary. This convention is well established in other Lisps such as Common Lisp or Clojure. The `cond` special form works like `if`, but allows multiple tests, and can optionally provide a default result that will be returned if none of the previous tests succeed. ```clojure (cond false 10 false 20 true 30 false 40) => 30 ;; You can provide a default result at the end if all tests fail (cond false 10 false 20 "Nothing matched") => "Nothing matched" ``` Implementation note: `if` is actually a macro that expands to a `cond` special form. So technically, `cond` is the lower level special form. In practice, it may be more convenient and intuitive to use `if`. Your choice, as always! ## Execution flow ### Sequential `do` blocks The `do` special form groups a number of expressions into a single expression and returns the value of the last expression (or `nil` if there are zero expressions). Results from earlier expressions are discarded. ```clojure ;; A do Block with three expressions inside, all are executed but only the last result is returned. (do 1 2 3) => 3 ;; A do block with zero expressions always returns 'nil' (do) => nil ;; Side effects from earlier expressions are visible in later expressions (do (def a 100) (+ a a)) => 200 ``` The `do` form serves a similar purpose to a code block in many other languages. It's useful for grouping several statements together for the purposes of side effects. Typically, the earlier expressions are included because they perform some side effect. There isn't much point executing some pure code that simply returns a value if you simply ignore it. Because of this, the presence of `do` in some Convex Lisp code is a strong hint that side effects may be happening. ### Local variables with `let` The `let` special form allows you to define local variables in the scope of a code block. Apart from the local variable definition, a `let` block is similar to a `do` block. ```clojure ;; 'let' expression that defines 'x' in its body. (let [x 10] (* x x)) => 100 ;; You can define multiple local variables with one binding vector (let [x 10 y (* x x)] (+ x y)) => 110 ;; Local binding ceases to exist immediately after the '(let ...)' form (do (let [x 10]) x) => ERROR (UNDECLARED) => 'x' is undeclared. ;; Local bindings take precedence over definitions in the surrounding environment (def foo 13) foo => 13 (let [foo 17] foo) => 17 ``` You can also use `set!` to modify the binding of a local variable. This value will last until the end of the current binding form (a surrounding `let` block, or returning from a function body). ```clojure (let [a 10] (set! a 20) a) => 20 ``` ### Def and the environment We've already seen the `def` special form in a couple of examples, where it was used to set the value of a symbol: ```clojure (def message "Hello!") message => "Hello!" ``` The key difference with `def` compared to `let` is that it sets the value in the persistent environment, rather than just making a temporary local binding. The environment in Convex is special: - Every user Account gets its own *independent* environment. Two different users can define their own `message` and they will see their own version. - You can only *modify* your own environment, using a digitally signed transaction for the relevant account. - It is possible for anyone with access to the Convex network to *observe* any user environments on a read-only basis - so while nobody can modify your data, it isn't private! - The environment is *persistent* between transactions. Unless you choose to delete it, a definition in your environment will stay there forever. You can therefore use definitions in the environment to store data. - Definitions in the environment use some amount of *memory* on-chain. While small data allocations are typically not very expensive, care should be taken before storing large data structures in the environment. If you want to define functions specifically, you can use `defn`: ```clojure ;; Define a Euclidean distance function (defn dist [x y] (sqrt (+ (* x x) (* y y)))) (dist 3.0 4.0) => 5.0 ``` `defn` is actually a simple macro that converts `(defn f [x] ...)` into `(def f (fn [x] ...))`. So you never really need `defn`: it's just a convenient shortcut for defining functions and can make your code more readable. ### Loop and recur When you want to iteratively re-evaluate an expression, you can use `loop` and `recur`. ```clojure ;; This is a slow way to calculate an integer square root. Please use 'sqrt' in real code. (loop [i 0] ;; initialise 'i' with zero (if (< (* i i) 100) ;; test if 'i' is too low to be the square root (recur (inc i)) ;; recur - incrementing 'i' for the next loop iteration i)) ;; return 'i' => 10 ``` Loop works like `let` in that it establishes local loop variable bindings that you can use in each iteration. `recur` will jump back to the start of the loop, updating the loop variables. It is normal to use a conditional expression to determine whether to recur or not. The value returned from the loop will be the value of the last expression executed (in this case `i`) You can also use `recur` to repeat the evaluation of a function body: ```clojure ;; A factorial function using an accumulator (defn factorial [acc n] (if (<= n 1) acc (recur (* acc n) (dec n)))) (factorial 1 10) => 3628800 ``` `recur` implements "tail call optimisation", i.e. it recurs without consuming any stack space. This is important if you want to perform many iterations: stack depth on the CVM is a limited resource and your transactions will fail if you consume too much. `recur` is your friend. ## Quoting Sometimes, you want to use a symbol itself rather than the thing that the symbol refers to. In these cases, you can 'quote' the symbol. ```clojure (def a 10) ;; Use the value defined for the symbol a a => 10 ;; Use the Symbol itself (quote a) => a ;; The ' notation also quotes a Symbol 'a => a ``` You can also quote lists and other data structures - which returns these data structure *without* evaluating them. ```clojure ;; Evaluate a list normally (+ 1 2 3) => 6 ;; Quote a list without evaluating it '(+ 1 2 3) => (+ 1 2 3) ``` You can also 'unquote' within a *quasi-quoted* expression — written with a backtick instead of a single quote — using the tilde (`~`), which evaluates the unquoted part normally while leaving the rest quoted. (Plain `quote` does **not** process `~`; it leaves it as a literal `unquote` form, so use quasi-quote when you want to splice in computed values.) ```clojure ;; Quasi-quote with a backtick; ~ evaluates the unquoted part `[(+ 1 2 3) ~(+ 1 2 3)] => [(+ 1 2 3) 6] ``` ## Evaluation We've looked at the basic constructs of Convex Lisp, but it's worth taking a moment to look at the way that code is evaluated in Lisp. This section delves into some implementation details, and what makes Convex Lisp special. ### Code is Data A key idea in Lisp is that 'Code is Data'. The language syntax is expressed in the data structures of the language. This property is known as *homoiconicity*, and is one of the features of Lisp that makes it uniquely powerful. You can use the `eval` function to execute code that is provided as data: ```clojure ;; Regular code (+ 1 2) => 3 ;; Code expressed as a list '(+ 1 2) ;; Execute code using eval (eval '(+ 1 2)) => 3 ``` The power of 'Code is Data' starts to become apparent when you realise that since you can use code to construct data, you can equivalently use code to construct code. ```clojure (defn make-code [operation arguments] (cons operation arguments)) (make-code '+ [1 2 3 4]) => (+ 1 2 3 4) (eval (make-code '* [1 2 3 4])) => 24 ``` It is now clear why Lisp puts parentheses *before* the function name: expressions can be constructed as a single list, prepending the desired function to the list of arguments (often using `cons`). Code generation for arbitrary expressions becomes simple: just construct the code you want as a data structure! ### SECURITY: Take care with `eval` You should **NEVER** use `eval` on data from an untrusted source. It will be able to execute anything that you can in your environment - including helping itself with any coins and tokens controlled by your account. If you are unsure whether this is a risk or not, a good rule is that you should avoid using `eval` at all in any environment with economically valuable assets. ## Functional Programming Convex Lisp is designed to support functional programming. We can think of functional programming as a paradigm where: - Functions are first-class objects in the language - Programs are developed by composing pure functions and immutable data - Mutable data and side effects are generally avoided Functional programming offers us many major advantages: - Code expressed using pure functions is easier to reason about and test, because you don't have to worry about the internal or external mutable state that might affect behaviour. - It is often much shorter and quicker to read/write than equivalent imperative code - Immutable data is a *great* fit for the CVM which is designed around immutable, cryptographically verified data structures. Here's a simple example of functional programming, where we define a first-class function `square` and then pass it to another function to achieve our intended result: ```clojure ;; Define a function that squares a number (defn square [x] (* x x)) ;; Apply the square function to each element of a vector (map square [1 2 3 4]) => [1 4 9 16] ``` We can get a bit more sophisticated, and use functions to create other functions: ```clojure ;; Function to build a function combining a map and a reduce (defn mapreducer [init reducer mapper] (fn [vals] (reduce reducer init (map mapper vals)))) ;; Build a 'sum-of-squares' function using our 'mapreducer' (def sum-of-squares (mapreducer 0 + square)) (sum-of-squares [1 10 100 1000]) => 1010101 ``` `map` and `reduce` are both very powerful tools for functional programming, and in many cases can replace the need to implement imperative loops. They also help to avoid the dreaded "off by one" errors! ## Where to go from here You now know the basics of Convex Lisp. To put it to work on the network: - **[Quick Start](/docs/tutorial/quickstart)** — deploy and call a smart contract from the Sandbox - **[Actor Development](/docs/tutorial/actors)** — build smart contracts with `^:callable` functions - **[Recipes](/docs/tutorial/recipes)** — practical, task-oriented examples - **[Client SDKs](/docs/tutorial/client-sdks)** — run Convex Lisp from Java, Python, or TypeScript --- ## Account An Account is a record of identification and ownership within Convex. Accounts may be either: * **User Accounts**: Accounts that are controlled by external users, where access is controlled by digital signatures on transactions. * **Actor Accounts**: Accounts that are managed by an autonomous actor, where behaviour is 100% deterministic according to the associated CVM code for the actor. ## Account Key (Public Key) An account key is a 32-byte value (blob) used as a public key to control access to an account. It is generally shown as a hexadecimal string, looking something like: `0x89b5142678bfef7a2245af5ae5b9ab1e10c282b375fa297c5aaeccc48ac97cac` Technically, the account key of a user account is an `Ed25519` public key. You must be in possession of the corresponding private key in order to digitally sign transactions for that Account. Actor accounts have a `nil` account key (therefore they do not have a corresponding private key, and no transactions can be submitted for them). ## Actor An autonomous entity implemented in CVM code on the Convex Network. An Actor is defined with exactly one Account, but may send messages to and control assets managed by other Actors / Accounts. ## Address An address is a numerical value used to refer to accounts. An address is valid if it refers to an existing Account (User or Actor) in the CVM state. Addresses are conventionally displayed as a number with with a `#` prefix e.g.: `#1245` Addresses are issued sequentially for new accounts by the CVM. ## Belief A belief is a specialised value containing a Peer's combined view of consensus. The belief data structure most notably contains orderings of blocks either confirmed in consensus or proposed for consensus by different peers. ## Belief Merge Function A specialised function that can be used to merge Beliefs from different peers. Each Peer runs a belief merge function as part of the CPoS algorithm. ## Blob A value representing an arbitrary sequence of bytes. It is typically displayed in hex with a leading `0x` ``` 0x0123456789abcdef ``` ## Block A Block in Convex is a collection of transactions submitted simultaneously by a Peer. Unlike Blockchains, a Block in Convex does *not* contain a hash of the previous block. A Block must be digitally signed by the proposing Peer to be valid for inclusion in consensus. ## Blockchain A system that maintains an append-only sequence of blocks where each block contains a cryptographic hash of the previous block (and hence its integrity can be validated recursively all the way back to the original block). Technically, Convex is *not* a blockchain because blocks are not required to contain a hash of any previous block. This gives Convex a technical advantage because blocks can therefore be handled in parallel and re-ordered by the consensus algorithm after creation. ## CAD A Convex Architecture Document — an open, RFC-style specification defining part of Convex's functionality and interoperability. CADs are numbered (e.g. CAD003 Encoding, CAD024 Data Lattice) and serve as the authoritative technical reference for the ecosystem. ## Cell A cell is a self-contained unit of a larger CVM value. Cells are linked immutably as a Merkle tree and handled transparently by the system, allowing developers to work with massive datasets without worrying about fragmentation. Scale your ideas infinitely on Convex. For more details see [CAD3 Encoding](../cad/encoding) ## CNS (Convex Name System) The on-chain naming system that maps human-readable, hierarchical names to Convex resources such as accounts, actors and values. Names are resolved with the `@` syntax, for example `@convex.fungible`. See [CAD014](../cad/cns). ## Coin (Convex Coin) The native currency of the Convex Network, denoted as CVM. Convex Coins are used to pay for transaction fees, computation, and economic incentives within the ecosystem. Hold CVM to participate in governance, stake for yields, or power your dApps— the fuel for a booming decentralised economy. ## Consensus Algorithm The Convex Consensus Algorithm is obtains consensus through the use of a convergent Belief Merge Function. This algorithm is called Convergent Proof of Stake (CPoS), and is described in more detail in the [White Paper](../overview/convex-whitepaper.md). ## Consensus Point The greatest position in the Ordering of Blocks produced by the Consensus Algorithm which has been confirmed as being in Consensus. Each Peer maintains it's own view of the Consensus Point based on observed consensus proposals from other Peers. The Consensus Point cannot be rolled back according to the rules of the Protocol (any attempt to do so would therefore constitute a Fork). However, some Peers may advance their Consensus Point slightly before others. Users transacting on the Convex network should use the Consensus Point of a trusted Peer to confirm that their transactions have been successfully executed on the Convex Network. ## Controller An account (or key) authorised to control another account — for example to manage an actor or to recover access. An account's controller can act on its behalf according to the rules of the CVM, so controller keys must be protected as carefully as the account's own key. ## Convex Network A network of Peers, maintaining a consistent global state and executing state transitions according to the Consensus Algorithm and rules of the CVM. ## Convex Lisp A programming language based on Lisp, that is available by default as part of the CVM. Convex Lisp prioritises features that are well suited to the development of decentralised economic systems. This includes: * Emphasis on functional programming to reduce error and improve logical clarity * Use of immutable, persistent data structures * Actor-based model enabling trusted autonomous execution of code for Smart Contracts ## Copper The smallest unit of the Convex Coin. 1 CVM = 1,000,000,000 copper (10^9). Coin amounts in transactions, balances and faucet requests are expressed in copper. ## CPoS (Convergent Proof of Stake) CPoS is Convex's breakthrough consensus mechanism that guarantees convergence on a single, fork-free global state using belief merge functions and stake-weighted peer coordination. Unlike traditional PoS, CPoS enables parallel block processing and sub-second finality—perfect for high-throughput dApps that demand speed and reliability. ## CRDT Acronym for Conflict-free Replicated Data Type, a data structure that can be replicated across many computers in a network and is guaranteed (mathematically) to reach eventual consistency. The Consensus Algorithm makes use of what is effectively a CRDT (of Beliefs) to guarantee convergence on a single consensus. ## Cursor A mutable handle to a position within immutable [Lattice](#lattice) data, providing atomic read, write, navigation and merge operations. Lattice-aware cursors support fork/sync for transactional updates that converge by lattice merge. See [CAD035](../cad/cursors). ## CVM Acronym for Convex Virtual Machine. This is a general purpose computational environment that can be used to implement the State transitions triggered by Transactions in the Convex Network. The CVM is Turing complete, and is capable of executing arbitrary logic. It enforces constraints upon computation costs and memory usage to ensure that Users are unable to abuse shared resources (making Denial of Service attacks prohibitively expensive, for example). ## CVM Code A representation of computer code that can be executed natively on the CVM. CVM code is based on a small number of core primitives that map to the Lambda Calculus, which can be composed in a tree data structure to represent arbitrary Turing-complete code. Different languages may be compiled to CVM code. ## DApp A dApp is a decentralised application. We can distinguish between two forms of Dapp: - **Pure dApp** - the Dapp consists only of client code and on-chain implementation (i.e. the Dapp depends on the Convex network and nothing else). Such Dapps are simple to build and maintain, and minimise the risk of relying on centralised systems - **Hybrid dApp** - the Dapp uses client code, on-chain-implementation and one or more off-chain servers. This is more complex to build and maintain, but is necessary if additional servers are required (e.g. to store private information, or to integrate with external systems) ## DID (Decentralised Identifier) A W3C-standard identifier for a self-sovereign identity. Convex supports `did:key`, where the identifier embeds an Ed25519 public key directly, used to authenticate users and to sign capability tokens. See [CAD043](../cad/did). ## Digital Signature A cryptographic technique where a piece of data Digital signatures in Convex use the Ed25519 algorithm. The data that is signed is the Value ID of a CVM Data Object (which in turn is the SHA3-256 hash of the Encoding) ## DLFS (Data Lattice File System) A decentralised, content-addressed file system built on the Data Lattice. It provides CRDT-based file storage that replicates and merges across peers without a central server, supporting offline-first workflows. See [CAD028](../cad/dlfs). ## Encoding Every CVM Data Object has an Encoding, which is a representation of the Object as a sequence of bytes. Encodings are designed to be: - Small in size (to minimise storage and network bandwidth requirements) - Efficient for serialisation and deserialisation - Canonical (i.e. any Data Object has one and only one valid Encoding) The maximum Encoding size is 8191 byes. Larger Data Objects are broken down into multiple Cells which each have their individual Encoding - however this is handled automatically by Convex and not usually a relevant concern for users or developers. ## Environment An Environment on the CVM is a mapping from Symbols to defined values. The Convex environment should be familiar to those who study the formal semantics of programming languages. It is implemented as a functional, immutable map, where new definitions result in the creation and usage of a new Environment. Each Account receives its own independent Environment for programmatic usage. If the Account is an Actor, exported definitions in the environment define the behaviour of the Actor. ## Etch Etch is the underlying Convex storage subsystem - "A database for information that needs to be carved in stone". Etch implements Converge Immutable Storage for Data Objects. ## Faucet A service on test networks that dispenses small amounts of Convex Coins for development and testing. Faucets are never available on production networks. See the [Faucet guide](coins/faucet). ## Fee The cost paid in Convex Coins to execute a transaction on the network. Fees compensate peers for computation and ensure spam resistance. Convex's efficient design keeps fees ultra-low, making micro-transactions viable and attracting mass adoption. ## Fork A Fork in a consensus system is, in general, where two or more different groups diverge in agreement on the value of shared Global State. This creates significant problems with a system of value exchange because assets may have different ownership in different forks - which in some cases could cause major economic loss (e.g. the infamous "double spend problem") Convex is designed to prevent forks. In the unlikely event of a fork created by malicious actors or software / network failures, the Convex Network will follow the largest majority among known, trusted Peers (this is a governance decision outside the scope of the Protocol). ## Function A Function is a Data Object that represents a first-class function on the CVM. Functions may be passed as arguments to other functions, and invoked with arbitrary arguments. They may be anonymous, or given a name within an Environment. They may also be closures, i.e. capture lexical values from the point of creation. Functions can support multiple arities on the CVM (e.g. `+`, although many functions only support a specific arity.) ## Genesis The predefined initial State of a Convex network (`S[0]`), from which all subsequent states are derived. The **genesis hash** is the Value ID of this initial state and uniquely identifies a network — it is, for example, how a client distinguishes Protonet from a test network. ## Identicon An Icon generated in a pre-defined way that can be used to visually confirm if a value is identical to another value. Identicons are used in Convex to provide additional security for similar Addresses that might be hard to distinguish by the hexadecimal strings alone. ## Juice The measure of computational cost on the CVM. Every operation consumes juice, and a transaction must pay (in Convex Coins) for the juice it uses. Juice bounds computation so that shared resources cannot be abused. Juice is Convex's equivalent of "gas" in other systems — but is always called **juice**. See [CAD007](../cad/juice). ## Lattice The Data Lattice is Convex's decentralised, content-addressed data substrate. Lattice values merge as CRDTs, so peers converge on shared state without coordination. It underpins DLFS, the KV database and other off-chain replication. See [CAD024](../cad/data_lattice). ## MCP (Model Context Protocol) A standard protocol that lets AI models and agents interact with external tools and data. Convex peers expose an MCP server so agents can query state, submit transactions, and manage keys through a well-defined tool interface. See [CAD041](../cad/mcp). ## Memory Memory in Convex is a second native cryptocurrency (in addition to Convex Coins) that can be used to purchase on-chain storage capacity. Users need to buy Memory if they want to execute transactions that increase the size of the State. USers get a refund if they execute Transactions that reduce the size of the State - creating a good incentive to use on-chain resources efficiently. ## Memory Accounting Memory Accounting is the process by which changes in Memory usage are attributed and charged to Users. This is a necessary feature of Convex to create the right incentives to utilise on-chain memory efficiently. Without a system of Memory Accounting, there would be a risk of careless usage of Memory leading to ever-increasing size of the Global State (sometimes termed the "state growth problem" in Blockchains). ## On-chain Data or code is considered to be "on-chain" if is contained within or affects the current State of the CVM. On-chain data is the *only* information that is visible to the CVM. It can be accessed and used by Actors, e.g. as part of the management of smart contracts and digital assets. As a general principle, on-chain data should be kept to the *absolute minimum necessary*. This is because: - It has a real cost (in terms of both coins and memory) - It is effectively public information so should exclude any confidential or private information Note that while Convex is not technically a blockchain in the literal sense of a hash-linked chain of blocks, we still use "on-chain" informally to refer to data or code secured by the equivalent Ordering of blocks in Convex. ## Ordering An Ordering defines the sequence in which Blocks of Transactions are to be executed. A key purpose role of the Consensus Algorithm is to ensure that all good PEers agree on the same Consensus Ordering, and hence all calculate the same Consensus State. In normal use of the Convex system, the Ordering maintained by a Peer will be confirmed in Consensus only up to a certain point (the Consensus Point). Blocks after this point are not yet confirmed, but are in the process of being proposed for consensus. ## Peer A Peer is a system that participates in the operation of the decentralised Convex Network, and in particular helps to achieve consensus. Peers are required to use a private key to sign certain messages as they participate in the consensus protocol. Because of this, a Peer's stake may be at risk if the system is not adequately secured. Peer operators therefore have a strong incentive to maintain good security for their systems. ## Private Key A cryptographic key that can be used to digitally sign transactions. Private Keys must be kept secure in order to prevent unauthorised access to Accounts and Digital Assets controlled by that Account. ## Protonet The Convex production network, reachable at `peer.convex.live`. Contrast with test networks, which provide faucets and may be reset. Coins on Protonet have real economic value. ## Public Key A cryptographic key that can be used to validate transactions. Public Keys may be safely shared with others, as they do not allow digital signatures to be created without the corresponding private key. User Accounts in Convex use an Ed25519 Public Key as the Account Keys, which enables any Peer to validate that a transaction for a given user has been signed with the correct Private Key. ## Query A read-only operation submitted to the CVM that executes code against the current State without modifying it. Queries are free (no fees) and instant, ideal for dApp frontends. Query the state with confidence: real-time insights to delight users and drive engagement. ## Schedule The schedule is a feature in the CVM enabling CVM code to be scheduled for future execution. Once included in the schedule, such code is *unstoppable* - it's execution is guaranteed by the protocol. Scheduled code may be used to implement actors that take periodic actions, smart contracts that have defined behaviour after a certain period of time etc. ## Sequence Number A per-account counter that orders and de-duplicates a user's transactions. Each transaction specifies the next sequence number for its account; the CVM rejects out-of-order or replayed sequence numbers, which prevents transaction replay. ## Smart Contract A Smart Contract is a self-executing economic contract with the terms of the agreement written into lines of code that are executed deterministically on the CVM. Buyer and sellers can predict exactly how the Smart Contract will behave, and can therefore trust it to enforce contract terms and conditions effectively. Typically a Smart Contract would be implemented using an Actor, but it is possible for a single Actor to manage many smart contracts, and likewise for a single Smart Contract to be executed across multiple Actors. It may be helpful to think of Smart Contracts as secure economic constructs, and Actors as a lower level implementation mechanism. ## Stake A stake is an asset with economic value put at risk by some entity in order to prove commitment to its participation in some economic transaction and / or good future behaviour. Convex uses a staking mechanism to admit peers for participation in the CPoS algorithm. Other forms of stakes may be used in smart contracts. ## Stake Weighted Voting Convex uses stakes to determine the voting weight of each Peer in the Consensus Algorithm. Benefits for a peer having a higher effective voting stake are: * Slightly more influence over which Blocks get ordered first, if two blocks are simultaneously submitted for consensus * They may also benefit from slightly improved overall latency for Blocks that they submit. While good Peers are expected to be content neutral, they may legitimately wish to offer better QoS to their partners or customers, and having a higher voting stake can help them to achieve this. The protocol does not allow Peers to reverse a confirmed consensus, or prevent (censor) a Block from being included in consensus. Their stake may be at risk if they attempt this. ## State A State is a special (typically large) Value that refers to the complete information managed by execution on the CVM. It is in effect the "Universe" that can be manipulated by Transactions and CVM code. The latest version of the State is maintained by the Consensus Algorithm, and the latest State obtained in Consensus is called the "Consensus State". The Consensus State is particularly important, since it contains the confirmed balances of Convex Coins and other digital assets, as well as the current state of all existing Smart Contracts and other data on the CVM. ## State Transition Function The State Transition Function is the function that updates the State in response to new Blocks of Transactions after they are confirmed by the Consensus Algorithm. Formally this might be recursively specified as ``` S[n+1] = f(S[n],B[n]) where: f is the State Transition Function S[n] is the sate after n Blocks have been processed B[n] is the Block at position n in the cordering S[0] is the pre-defined Initial State ``` ## Transaction A transaction is an operation that can be submitted by clients for execution on the Convex Network. A transaction must be linked to a user account, and must be digitally signed by the private key corresponding to the account key in order to be valid. Transactions must be digitally signed by the owner of the account in order to be valid. ## UCAN User Controlled Authorisation Network — a signed, delegable capability token. A UCAN grants specific abilities on specific resources and can be *attenuated* (narrowed) as it is delegated, enabling fine-grained, decentralised authorisation without a central authority. See [CAD038](../cad/lattice_auth). ## Value A value is a first-class, immutable piece of information managed by Convex. A value can be simple (e.g. the number `1`) or composite (e.g., a vector containing other values like `[1 2 [3 4] :foo]`) Value types include: * Primitive values (numbers, strings, symbols, binary blobs) * Data Structures representing composites of many values (including other data structure) * Executable CVM code ("Ops") * Some special values used by the CVM, e.g. Blocks Values may be processed by code within the CVM, and are the fundamental building blocks for on-chain systems such as smart contracts. ## Wallet A Wallet is an application or device that stores keys (especially private keys) for Convex accounts, enabling access and control over digital assets held by those accounts. Wallet functionality may be provided by a dApp, or embedded in any system that communicates with the Convex Network. It may also be a specialised hardware device (hardware wallet). Wallet security is paramount: if access to the private keys in a wallet is compromised, any on-chain digital assets (coins, tokens, smart contract rights etc.) may be at risk. ## x402 A protocol for native web payments using the HTTP `402 Payment Required` status code, enabling agents and services to pay for resources over HTTP using Convex Coins. See [CAD042](../cad/x402). --- Convex offers a comprehensive, full-stack solution for building decentralised applications (dApps) and open economic systems. ## Core Concepts ### Convex Coins Learn about the native currency and accounts: - **[Convex Coins](tutorial/coins)** - Native utility token (CVM) - **[Faucet Guide](tutorial/coins/faucet)** - Getting test funds ### Smart Contracts Build on-chain logic with Convex: - **[Convex Lisp](tutorial/convex-lisp)** - The on-chain programming language - **[Actors](tutorial/actors)** - Smart contracts and autonomous actors - **[Recipes](tutorial/recipes)** - Practical how-to guides ### AI Agents Connect autonomous agents to the Convex economy: - **[AI Agents](tutorial/agents)** - Agents as first-class economic participants - **[MCP Integration](tutorial/agents/mcp)** - Connect any MCP-compatible agent to a peer ### Network Operations Run and operate peer infrastructure: - **[Peer Operations](tutorial/peer-operations)** - Complete operations guide - **[Local Testnets](tutorial/peer-operations/local-testnets)** - Development environments --- ## Learning Paths Choose your path based on what you want to build: ### Application Developer **Build apps that use Convex for state and transactions** 1. Start with [Networks](tutorial/networks) - Understand available networks 2. Choose your [SDK](tutorial/client-sdks) - Pick your programming language 3. Follow SDK quickstart - Get your first transaction working 4. Explore [Recipes](tutorial/recipes) - Practical examples **Best for:** Web apps, mobile apps, backend services --- ### Smart Contract Developer **Build autonomous actors and on-chain logic** 1. Learn [Convex Lisp](tutorial/convex-lisp) - Master the on-chain language 2. Understand [Actors](tutorial/actors) - Smart contract concepts 3. Study [Recipes](tutorial/recipes) - Working examples 4. Test with [Local Peer](tutorial/peer-operations/local-testnets) - Fast iteration **Best for:** DeFi protocols, DAOs, on-chain logic --- ### Agent Developer **Connect AI agents to real economic state** 1. Read [AI Agents](tutorial/agents) - Why Convex suits agentic systems 2. Set up [MCP Integration](tutorial/agents/mcp) - Connect your agent to a peer 3. Manage [Agent Accounts](tutorial/agents/account-management) - Keys, funding and delegation 4. Design [Agentic Economics](tutorial/agents/agentic-economics) - Agent-to-agent commerce **Best for:** AI agents, autonomous services, machine-to-machine payments --- ### Peer Operator **Run peer infrastructure and participate in consensus** 1. Read [Networks](tutorial/networks) - Understand network architecture 2. Study [Peer Staking](tutorial/peer-operations/staking) - Economic requirements 3. Choose [Deployment Method](tutorial/peer-operations/manual-deployment) - Set up infrastructure 4. Follow [Security Guide](tutorial/peer-operations/security) - Secure operations **Best for:** Infrastructure providers, validators, high-availability operations --- ## Getting Started ### Networks Learn about Convex networks and how to connect to them: - **[Networks Guide](tutorial/networks)** - Understand network types, find available networks - Production networks (peer.convex.live) - Public testnets (with faucets) - Local development peers ### Client SDKs Connect to Convex from your application using official client libraries: - **[TypeScript / JavaScript](/docs/tutorial/client-sdks/typescript)** - For web apps, Node.js, Deno, React — with full type safety, or [plain JavaScript](/docs/tutorial/client-sdks/javascript) *Production ready* • `npm install @convex-world/convex-ts` - **[Java](/docs/tutorial/client-sdks/java)** - For JVM applications, Android, enterprise systems *Production ready* • `world.convex:convex-java:0.8.8` - **[Python](/docs/tutorial/client-sdks/python)** - For Python applications, scripting, data science *Production ready* • `pip install convex-sdk` For command-line interaction, see the [Convex CLI](/docs/products/convex-cli). --- ## Quick Links **I want to...** - **Build my first app** → [Quick Start](tutorial/quickstart) → [Choose SDK](tutorial/client-sdks) - **Connect to a network** → [Networks Guide](tutorial/networks) - **Get test funds** → [Faucet Guide](tutorial/coins/faucet) - **Write smart contracts** → [Convex Lisp](tutorial/convex-lisp) - **Connect an AI agent** → [AI Agents](tutorial/agents) - **Run a peer** → [Peer Operations](tutorial/peer-operations) - **Look up terms** → [Glossary](tutorial/glossary) --- # Networks Understanding Convex networks and choosing the right one for your needs. ## Network Reference Quick reference for available Convex networks: | Network | Type | HTTP Access | Binary Access | Faucet | Genesis Hash | |---------|------|-------------|---------------|--------|--------------| | **Protonet** | Production | `https://peer.convex.live` | `peer.convex.live:18888` | ❌ No | `0xb0e44f2a...` | | **Hugging Face Testnet** | Testnet | `https://mikera1337-convex-testnet.hf.space` | ❌ Not available | ✅ Yes | *(varies)* | | **Local Peer** | Development | `http://localhost:8080` | `localhost:18888` | ✅ Yes | *(varies)* | **Connection Types**: - **HTTP Access** - REST API for web apps, simple integration (HTTPS/JSON) - **Binary Access** - Direct peer protocol for higher performance (TCP binary, not available on all networks) **Note**: Genesis hashes for testnets and local peers change when networks reset or restart. **→ [Peer Operations Guide](peer-operations)** - Connection details, verification, troubleshooting --- ## What is a Convex Network? A **Convex network** is a decentralised lattice-based system where multiple peer nodes work together to maintain a shared global state. Unlike traditional blockchains, Convex uses lattice technology and Convergent Proof of Stake (CPoS) consensus to achieve high performance and instant finality. ### Key Characteristics **Decentralised Architecture** - Network of peer nodes maintaining consensus - No central authority or single point of failure - Cryptographically secure state transitions **Lattice Technology** - State merges like CRDTs (Conflict-free Replicated Data Types) - Not a linear blockchain - Enables parallel transaction processing **Global Shared State** - All peers converge on the same state - Atomic transactions with cryptographic signatures - Self-sovereign accounts with Ed25519 keys **Consensus Mechanism** - Convergent Proof of Stake (CPoS) - Sub-second consensus finality - Byzantine fault tolerant ## Network Types ### Production Networks **Purpose**: Real-world applications with actual value **Characteristics**: - No public faucets (Convex Coins have real value) - High reliability and uptime - Persistent state (never resets) - Professionally operated infrastructure - ⚠️ Requires funded account to transact **When to Use**: - Deploying production applications - Managing real assets - Interacting with mainnet smart contracts ### Test Networks (Testnets) **Purpose**: Testing and learning without risk **Characteristics**: - Public faucets available (free test funds) - Safe for experimentation - ⚠️ May reset periodically - ⚠️ No real value (test Convex Coins only) - Publicly accessible **When to Use**: - Learning Convex development - Testing applications before deployment - Experimenting with smart contracts - Demonstrating features to others ### Local Development Peers **Purpose**: Private development environment **Characteristics**: - Full control over configuration - Fastest performance (no network latency) - Built-in account creation and funding - Perfect for unit/integration testing - Works offline - ⚠️ Requires running peer infrastructure **When to Use**: - Active development and debugging - Running automated tests - Learning Convex internals - Building without network dependency ## Known Networks ### Protonet (Production) Convex's main production network for real-world applications. **Connection**: - **HTTP**: `https://peer.convex.live` - **Binary**: `peer.convex.live:18888` **Details**: - **Genesis Hash**: `0xb0e44f2a645abfa539f5b96b7a0eabb0f902866feaff0f7c12d1213e02333f13` - **Faucet**: ❌ No - **Funding**: Requires funded account or transfer from existing account **⚠️ Important**: Verify the [genesis hash](peer-operations#verifying-network-identity) before submitting production transactions. --- ### Hugging Face Testnet Public testnet hosted on Hugging Face Spaces with faucet support for testing and learning. **Connection**: - **HTTP**: `https://mikera1337-convex-testnet.hf.space` - **Binary**: ❌ Not available (HTTP only) **Details**: - **Faucet**: ✅ Yes - **Hosting**: Hugging Face Spaces - **Resets**: May reset periodically (genesis hash will change) - **Use For**: Learning, testing, experimentation - **Limitation**: HTTP access only, use local peer for binary protocol testing --- ### Local Development Peer Run your own peer locally for maximum performance and control. **Connection**: - **HTTP**: `http://localhost:8080` - **Binary**: `localhost:18888` **Details**: - **Performance**: <1ms latency (fastest option) - **Control**: Full configuration access - **Privacy**: No network exposure - **Faucet**: Built-in account creation and funding **→ See**: [Peer Operations - Local Testnets](peer-operations/local-testnets) for setup options. --- ### Finding More Networks Additional test networks and community-run peers: - **[Discord Community](https://discord.com/invite/xfYGq4CT7v)** - `#network-status` channel - **[Convex Foundation](https://convex.world)** - Official announcements - **Community Forums** - User-operated test networks **Note**: Always verify network URLs before connecting, especially for production use. ## Choosing the Right Network ### Decision Guide | Scenario | Recommended Network | Why | |----------|-------------------|-----| | **First time learning** | Hugging Face Testnet | No setup required, safe to experiment | | **Daily development** | Local Peer | Fastest, binary protocol, full control | | **Integration tests** | Local Peer | Deterministic, fast, isolated | | **Public demos** | Hugging Face Testnet | Accessible to others, persistent | | **Staging tests** | Hugging Face Testnet | Production-like environment | | **Production app** | Protonet | Real value, high reliability | ### Development Workflow ```mermaid graph TD A[Starting Development] --> B{What are you doing?} B -->|Learning Basics| C[Hugging Face Testnet] B -->|Active Development| D[Local Peer] B -->|Integration Testing| D B -->|Deployment| E[Protonet] C -->|Ready to deploy| E D -->|Need to test publicly| C D -->|Ready to deploy| E ``` ### Quick Start by Use Case **Learning Convex** → Use **Hugging Face Testnet** (`https://mikera1337-convex-testnet.hf.space`) - No setup required • Free test funds • Safe to experiment **Building an Application** → Use **Local Peer** ([setup guide](peer-operations/local-testnets)) - Fastest iteration • Full control • Binary protocol • Works offline **Deploying to Production** → Use **Protonet** (`https://peer.convex.live`) - Real value • High reliability • [Verify genesis hash](peer-operations#verifying-network-identity)! ## Connecting to Networks All networks support the same client API. Simply change the URL to switch networks: ```java // Testnet Convex convex = Convex.connect("https://mikera1337-convex-testnet.hf.space"); // Production Convex convex = Convex.connect("https://peer.convex.live"); ``` **→ See [Peer Operations Guide](peer-operations)** for: - Connection types (HTTP vs binary protocol) - Configuration and timeouts - Genesis hash verification - Health monitoring - Troubleshooting ## Security Considerations ### Production (Protonet) - **Always verify genesis hash** - See [verification guide](peer-operations#verifying-network-identity) - Use HTTPS for encrypted connections - Never share private keys or seed phrases - ⚠️ Transactions are irreversible - ⚠️ Test thoroughly on testnets first ### Testnets - ⚠️ Never use production keys on testnets - ⚠️ Testnets may reset without notice - ⚠️ Test funds have no real value - Safe to experiment and make mistakes ### Local Peers - Completely private (no network exposure) - Safe for testing with production-like keys - ⚠️ State is ephemeral unless persisted ## Next Steps ### Getting Started - **[Peer Operations](peer-operations)** - Detailed connection guide, verification, troubleshooting - **[Client SDKs](/docs/tutorial/client-sdks)** - Connect from your programming language - **[Faucet Guide](/docs/tutorial/coins/faucet)** - Get test funds for testnets ### Advanced Topics - **[Client Types](/docs/tutorial/client-sdks/java/clients)** - HTTP vs binary protocol - **[Convex Peer](/docs/products/convex-peer)** - Run your own production peer - **[Account Management](/docs/tutorial/client-sdks/java/accounts)** - Keys and accounts ## Resources - **[Discord Community](https://discord.com/invite/xfYGq4CT7v)** - Network status and support - **[Convex.world](https://convex.world)** - Official website - **[GitHub](https://github.com/Convex-Dev/convex)** - Source code --- # Docker Deployment Deploy a Convex peer using Docker for simplified container-based deployment. ## Overview Docker deployment provides: - Simplified installation - Consistent environment - Easy upgrades - Portable configuration **→ For traditional installation, see [Manual Deployment](manual-deployment)** ## Prerequisites - **Docker** 20.10+ installed - **Docker Compose** 2.0+ (optional, recommended) - 4+ GB RAM allocated to Docker - 100+ GB disk space ## Quick Start ### Using Docker Run ```bash # Pull image docker pull convex/convex:latest # Run peer docker run -d \ --name convex-peer \ -p 18888:18888 \ -p 8080:8080 \ -v convex-data:/app/data \ convex/convex:latest peer start ``` ### Using Docker Compose Create `docker-compose.yml`: ```yaml version: '3.8' services: peer: image: convex/convex:latest container_name: convex-peer ports: - "18888:18888" # Peer port - "8080:8080" # REST API volumes: - convex-data:/app/data # Etch store - ./keystore.pfx:/app/keys/keystore.pfx:ro # keystore holding the peer key environment: - JAVA_OPTS=-Xmx4g - NETWORK=protonet restart: unless-stopped logging: driver: "json-file" options: max-size: "10m" max-file: "3" volumes: convex-data: ``` Start with: ```bash docker-compose up -d ``` ## Configuration ### Environment Variables ```bash # docker-compose.yml environment section environment: - JAVA_OPTS=-Xmx4g -XX:+UseG1GC - NETWORK=protonet - PEER_PORT=18888 - REST_PORT=8080 - LOG_LEVEL=info ``` ### Peer Configuration The peer is configured by `convex peer start` flags (see [Manual Deployment](manual-deployment)), not a config file. Pass them as the container command, for example: ```yaml command: > peer start --peer-key 0x --etch /app/data/peer-store --peer-port 18888 --api-port 8080 --url your-peer.example.com:18888 ``` > **Note:** confirm the image name/tag and entrypoint for your Convex release before relying on this compose file. ### Peer Keys Mount peer keys securely: ```yaml volumes: - ./peer-keys.dat:/app/keys/peer-keys.dat:ro ``` **⚠️ Security**: Set restrictive file permissions (600) on host. ## Management ### Start/Stop ```bash # Start docker-compose start # Stop docker-compose stop # Restart docker-compose restart ``` ### View Logs ```bash # Follow logs docker-compose logs -f peer # Last 100 lines docker-compose logs --tail=100 peer ``` ### Check Status ```bash # Container status docker-compose ps # Health check curl http://localhost:8080/api/v1/status ``` ## Monitoring ### Resource Usage ```bash # Container stats docker stats convex-peer # Detailed info docker inspect convex-peer ``` ### Health Checks Add to `docker-compose.yml`: ```yaml services: peer: healthcheck: test: ["CMD", "curl", "-f", "http://localhost:8080/api/v1/status"] interval: 30s timeout: 10s retries: 3 start_period: 60s ``` ## Upgrading ### Pull New Image ```bash # Pull latest docker-compose pull # Restart with new image docker-compose up -d ``` ### Backup Before Upgrade ```bash # Backup data volume docker run --rm \ -v convex-data:/data \ -v $(pwd):/backup \ ubuntu tar czf /backup/convex-data-$(date +%Y%m%d).tar.gz /data ``` ## Networking ### Custom Network ```yaml networks: convex-net: driver: bridge services: peer: networks: - convex-net ``` ### Expose Ports ```yaml ports: - "18888:18888" # Peer protocol - "8080:8080" # REST API ``` ## Persistence ### Data Volumes ```yaml volumes: convex-data:/app/data # Peer state convex-logs:/app/logs # Logs ``` ### Backup Strategy ```bash # Backup script #!/bin/bash docker-compose stop docker run --rm \ -v convex-data:/data \ -v /backups:/backup \ ubuntu tar czf /backup/backup-$(date +%Y%m%d).tar.gz /data docker-compose start ``` ## Next Steps - **[Hosting Options](hosting)** - Infrastructure requirements - **[Security Guide](security)** - Secure your deployment - **[Monitoring](troubleshooting#monitoring-and-diagnosis)** - Production monitoring ## Resources - **[Docker Documentation](https://docs.docker.com/)** - **[Docker Compose Reference](https://docs.docker.com/compose/)** - **[Manual Deployment](manual-deployment)** - Alternative method --- # Hosting Options Infrastructure requirements and hosting options for Convex peers. ## Hardware Requirements ### Minimum Specifications **Suitable for**: Testnet participation, development - **CPU**: 4 cores (2.5+ GHz) - **RAM**: 8 GB - **Storage**: 100 GB SSD - **Network**: 100 Mbps symmetric - **Uptime**: 95%+ ### Recommended Specifications **Suitable for**: Production (Protonet), high stake - **CPU**: 8+ cores (3.0+ GHz) - **RAM**: 16+ GB - **Storage**: 500+ GB NVMe SSD - **Network**: 1 Gbps symmetric - **Uptime**: 99.9%+ ### Enterprise Specifications **Suitable for**: Major validators, high availability - **CPU**: 16+ cores (3.5+ GHz) - **RAM**: 32+ GB - **Storage**: 1+ TB NVMe SSD (RAID 10) - **Network**: 10 Gbps symmetric - **Uptime**: 99.99%+ - **Redundancy**: Hot standby peer ## Storage Considerations ### Storage Requirements **Growth Rate**: - State: ~10 GB/year (estimated) - Logs: ~1 GB/month - Backups: 2-3x current state **Performance**: - IOPS: 10,000+ recommended - Latency: <1ms for state access - Type: NVMe SSD strongly recommended ### Storage Configuration ```bash # Recommended filesystem layout /opt/convex/ ├── data/ # 500 GB+ (state data) ├── logs/ # 50 GB (application logs) └── backups/ # 1 TB (backups) ``` ## Network Requirements ### Bandwidth **Minimum**: - Download: 100 Mbps - Upload: 100 Mbps - Monthly: ~500 GB **Recommended**: - Download: 1 Gbps - Upload: 1 Gbps - Monthly: ~2 TB ### Ports **Required**: - `18888` - Peer protocol (TCP, inbound/outbound) - `8080` - REST API (TCP, optional public) **Firewall Rules**: ```bash # Ubuntu/Debian with UFW sudo ufw allow 18888/tcp # Peer protocol sudo ufw allow 8080/tcp # REST API (optional) sudo ufw enable ``` ### IP Address **Requirements**: - Static IP address (required) - IPv4 (required) - IPv6 (recommended) - Reverse DNS configured (recommended) ## Cloud Providers ### AWS (Amazon Web Services) **Recommended Instance**: `c5.2xlarge` or higher **Configuration**: ``` Type: c5.2xlarge CPU: 8 vCPUs RAM: 16 GB Storage: 500 GB EBS (gp3, 10,000 IOPS) Network: Up to 10 Gbps Cost: ~$300/month ``` **Setup**: 1. Launch EC2 instance (Ubuntu 22.04 LTS) 2. Attach EBS volume for data 3. Configure Security Group (ports 18888, 8080) 4. Allocate Elastic IP 5. Follow [Manual Deployment](manual-deployment) guide **Considerations**: - Mature platform, extensive docs - Good global availability - ⚠️ Higher cost than alternatives - ⚠️ Complex pricing model ### Google Cloud Platform (GCP) **Recommended Instance**: `n2-standard-8` or higher **Configuration**: ``` Type: n2-standard-8 CPU: 8 vCPUs RAM: 32 GB Storage: 500 GB SSD persistent disk Network: 16 Gbps Cost: ~$350/month ``` **Setup**: 1. Create Compute Engine instance 2. Attach persistent SSD 3. Configure VPC firewall rules 4. Reserve static IP 5. Follow [Manual Deployment](manual-deployment) guide **Considerations**: - Competitive pricing - Strong network performance - Good regional coverage ### Microsoft Azure **Recommended Instance**: `Standard_D8s_v5` **Configuration**: ``` Type: Standard_D8s_v5 CPU: 8 vCPUs RAM: 32 GB Storage: 512 GB Premium SSD Network: 12.5 Gbps Cost: ~$400/month ``` **Setup**: 1. Create Virtual Machine (Ubuntu) 2. Attach Premium SSD 3. Configure Network Security Group 4. Reserve public IP 5. Follow deployment guide **Considerations**: - Enterprise integration - Hybrid cloud options - ⚠️ Higher cost ### DigitalOcean **Recommended Droplet**: `CPU-Optimized 8GB` **Configuration**: ``` Type: CPU-Optimized CPU: 8 vCPUs RAM: 16 GB Storage: 100 GB SSD Network: 8 TB transfer Cost: ~$240/month ``` **Setup**: 1. Create Droplet (Ubuntu 22.04) 2. Attach Block Storage volume 3. Configure firewall 4. Reserve IP address 5. Follow deployment guide **Considerations**: - Simple pricing - Easy to use - Good value - ⚠️ Limited regions ### Hetzner **Recommended Server**: `CPX41` (Cloud) or `AX52` (Dedicated) **Cloud Configuration**: ``` Type: CPX41 CPU: 8 vCPUs RAM: 16 GB Storage: 240 GB NVMe Network: 20 TB traffic Cost: ~$50/month ``` **Considerations**: - Excellent value - Good performance - ⚠️ Limited to Europe - ⚠️ Strict abuse policies ### OVH Cloud **Recommended**: `c2-15` or dedicated server **Configuration**: ``` Type: c2-15 CPU: 8 vCores RAM: 15 GB Storage: 200 GB NVMe Network: 1 Gbps Cost: ~$70/month ``` **Considerations**: - Competitive pricing - Good European presence - ⚠️ Complex interface ## Dedicated Servers ### When to Use Dedicated **Advantages**: - Better performance per dollar - Predictable costs - No noisy neighbors - Full hardware control **Use dedicated when**: - Running multiple peers - High stake value - Maximum performance needed - Cost optimisation important ### Providers **Hetzner Dedicated**: - Excellent value - Fast provisioning - European locations - ~$50-200/month **OVH Dedicated**: - Competitive pricing - Global locations - Good hardware options - ~$60-300/month **Leaseweb**: - Flexible configurations - Global presence - Custom hardware - ~$100-500/month ## Colocation ### When to Consider **Suitable for**: - Multiple peers - Maximum control - Cost optimisation at scale - Custom hardware requirements **Requirements**: - Own hardware - Colocation facility - Remote hands service - Network connectivity **Cost Considerations**: - Hardware: $2,000-10,000 upfront - Rack space: $100-500/month per U - Power: $50-200/month - Network: $100-1,000/month ## Geographic Distribution ### Location Selection **Consider**: - Network latency to other peers - Legal jurisdiction - Data sovereignty requirements - Cost and availability **Recommended Regions**: - North America (East Coast, West Coast) - Europe (Frankfurt, London, Amsterdam) - Asia Pacific (Singapore, Tokyo) ### Multi-Region Deployment For high availability: 1. Primary peer in main region 2. Hot standby in different region 3. Automated failover 4. Shared stake control ## Kubernetes Deployment ### When to Use K8s **Suitable for**: - Running multiple peers - Existing K8s infrastructure - Advanced orchestration needs - Team familiar with K8s **Basic Deployment**: ```yaml apiVersion: apps/v1 kind: StatefulSet metadata: name: convex-peer spec: serviceName: convex-peer replicas: 1 selector: matchLabels: app: convex-peer template: metadata: labels: app: convex-peer spec: containers: - name: peer image: convex/convex:latest ports: - containerPort: 18888 - containerPort: 8080 resources: requests: cpu: 4 memory: 8Gi limits: cpu: 8 memory: 16Gi volumeMounts: - name: data mountPath: /app/data volumeClaimTemplates: - metadata: name: data spec: accessModes: ["ReadWriteOnce"] resources: requests: storage: 500Gi ``` ## Cost Optimisation ### Tips for Reducing Costs 1. **Right-size resources**: Start smaller, scale as needed 2. **Use spot/preemptible instances**: For testnets only 3. **Reserved instances**: Commit for 1-3 years (30-70% savings) 4. **Monitor usage**: Identify waste, optimise 5. **Compare providers**: Benchmark cost vs performance ### Cost Estimation **Monthly Cloud Costs (USD)**: - Minimum spec: $100-150 - Recommended spec: $250-400 - Enterprise spec: $500-1,000+ **Monthly Dedicated Costs (USD)**: - Entry level: $50-100 - Mid-range: $100-300 - High-end: $300-1,000+ ## Next Steps - **[Manual Deployment](manual-deployment)** - Install on your infrastructure - **[Docker Deployment](docker-deployment)** - Container-based deployment - **[Security Guide](security)** - Secure your infrastructure - **[Monitoring](troubleshooting#monitoring-and-diagnosis)** - Production monitoring ## Resources - **[AWS EC2 Pricing](https://aws.amazon.com/ec2/pricing/)** - **[GCP Pricing Calculator](https://cloud.google.com/products/calculator)** - **[Hetzner Server Finder](https://www.hetzner.com/dedicated-rootserver)** - **[Discord Community](https://discord.com/invite/xfYGq4CT7v)** - Get recommendations --- # Peer Operations Overview Guide for connecting to, monitoring, and operating Convex peer nodes. ## What is Peer Operations? Peer Operations covers everything from connecting to existing peers to running your own peer infrastructure: **For Developers** (connecting to peers): - **[Connection Types](#connection-types)** - HTTP vs binary protocol - **[Network Verification](#verifying-network-identity)** - Genesis hash and health checks - **[Performance](#performance-optimisation)** - Optimisation and monitoring **For Operators** (running peers): - **[Staking & Registration](peer-operations/staking)** - Stake coins and register your peer - **[Manual Deployment](peer-operations/manual-deployment)** - Install and configure manually - **[Docker Deployment](peer-operations/docker-deployment)** - Deploy with Docker/Compose - **[Hosting Options](peer-operations/hosting)** - Infrastructure and requirements - **[Security](peer-operations/security)** - Secure your peer infrastructure - **[Troubleshooting](peer-operations/troubleshooting)** - Diagnose and fix issues ## Connection Types Convex peers support two connection protocols with different performance characteristics. ### HTTP Protocol (REST API) **Recommended for**: Web applications, general development **Connection**: ```java // Java Convex convex = Convex.connect("https://peer.convex.live"); ``` ```typescript // TypeScript const convex = new Convex('https://peer.convex.live'); ``` ```python # Python convex = Convex('https://peer.convex.live') ``` **Characteristics**: - Protocol: HTTPS with JSON payloads - Port: 18888 (default) - Latency: 10-50ms typical - Firewall: Works through HTTP proxies - Connection: Stateless ### Binary Protocol **Recommended for**: High-performance applications **Connection**: ```java // Java import java.net.InetSocketAddress; InetSocketAddress addr = InetSocketAddress.createUnresolved( "peer.convex.live", 18888 ); Convex convex = Convex.connect(addr); ``` **Characteristics**: - Protocol: Custom binary over TCP - Port: 18888 (default) - Latency: 5-20ms typical - Connection: Persistent stateful - Throughput: Higher than HTTP ### Connection Configuration **Timeouts**: ```java Convex convex = Convex.connect("https://peer.convex.live"); convex.setTimeout(Duration.ofSeconds(30)); ``` **Connection Pooling** (for high concurrency): ```java import java.util.concurrent.ConcurrentLinkedQueue; class ConvexConnectionPool { private final ConcurrentLinkedQueue pool = new ConcurrentLinkedQueue<>(); private final String endpoint; public ConvexConnectionPool(String endpoint) { this.endpoint = endpoint; } public Convex acquire() throws Exception { Convex convex = pool.poll(); if (convex == null) { convex = Convex.connect(endpoint); } return convex; } public void release(Convex convex) { pool.offer(convex); } } ``` ## Network Health and Status ### Check Network Availability ```java import convex.api.Convex; import convex.core.Result; import convex.core.lang.Reader; Convex convex = Convex.connect("https://peer.convex.live"); // Query consensus state Result result = convex.query(Reader.read("*state*")).get(); if (!result.isError()) { System.out.println("✓ Network operational"); } else { System.err.println("✗ Network issue: " + result.getErrorCode()); } ``` ### Query Network Metadata ```java // Peer / network status is available from the REST API: GET /api/v1/status // Current consensus point Result consensus = convex.query(Reader.read("*state*")).get(); // Total accounts Result accounts = convex.query(Reader.read("(count *accounts*)")).get(); // Memory usage Result memory = convex.query(Reader.read("*memory*")).get(); ``` ## Verifying Network Identity Each network has a unique **genesis hash**. Always verify you're connected to the correct network. ### Protonet Genesis Hash ``` 0xb0e44f2a645abfa539f5b96b7a0eabb0f902866feaff0f7c12d1213e02333f13 ``` ### Verify Connection ```java // Protonet genesis hash String PROTONET_GENESIS = "0xb0e44f2a645abfa539f5b96b7a0eabb0f902866feaff0f7c12d1213e02333f13"; // Connect and verify Convex convex = Convex.connect("https://peer.convex.live"); Result result = convex.query(Reader.read("*genesis*")).get(); String genesis = result.getValue().toString(); if (genesis.equals(PROTONET_GENESIS)) { System.out.println("✓ Connected to Protonet"); } else { System.err.println("⚠ Warning: Unknown network"); System.err.println("Genesis: " + genesis); } ``` ### TypeScript Example ```typescript const PROTONET_GENESIS = '0xb0e44f2a645abfa539f5b96b7a0eabb0f902866feaff0f7c12d1213e02333f13'; const convex = new Convex('https://peer.convex.live'); const result = await convex.query('*genesis*'); if (result.value.toString() === PROTONET_GENESIS) { console.log('✓ Connected to Protonet'); } else { console.warn('⚠ Unknown network:', result.value.toString()); } ``` ### Python Example ```python PROTONET_GENESIS = "0xb0e44f2a645abfa539f5b96b7a0eabb0f902866feaff0f7c12d1213e02333f13" convex = Convex('https://peer.convex.live') result = convex.query('*genesis*') if str(result['value']) == PROTONET_GENESIS: print("✓ Connected to Protonet") else: print(f"⚠ Unknown network: {result['value']}") ``` ### Important Notes - ⚠️ **Testnets** change genesis hash when they reset - ⚠️ **Local peers** generate new genesis hash on startup - **Protonet** genesis is stable - always verify for production - 🔒 Prevents accidental connection to wrong network ## Performance Optimisation ### Connection Performance | Method | Latency | Throughput | Use Case | |--------|---------|------------|----------| | HTTP | 10-50ms | ~100 ops/sec | Web apps, general use | | Binary | 5-20ms | ~500 ops/sec | High-performance apps | | Local Peer | <1ms | ~10,000 ops/sec | Development, testing | ### Best Practices **For Development**: - Use local peer for fastest iteration - Binary protocol for integration tests - Disable unnecessary logging **For Production**: - Use binary protocol for better throughput - Implement connection pooling - Monitor peer health proactively - Verify genesis hash on startup - Set appropriate timeouts - Implement retry logic with exponential backoff ### Monitoring Example ```java import java.util.concurrent.*; public class PeerMonitor { private final Convex convex; private final ScheduledExecutorService scheduler = Executors.newScheduledThreadPool(1); public PeerMonitor(Convex convex) { this.convex = convex; } public void startMonitoring() { scheduler.scheduleAtFixedRate(() -> { try { Result result = convex.query(Reader.read("*state*")).get(); if (result.isError()) { System.err.println("⚠ Peer health check failed"); } else { System.out.println("✓ Peer healthy"); } } catch (Exception e) { System.err.println("✗ Peer unreachable: " + e.getMessage()); } }, 0, 30, TimeUnit.SECONDS); } public void stopMonitoring() { scheduler.shutdown(); } } ``` ## Local Development Peer For development, run a local peer for maximum speed and control. **Quick Start**: ```java import convex.peer.Server; import convex.api.Convex; // Create and launch Server server = Server.create(); server.launch(); try { Convex convex = Convex.connect(server); // Use for development } finally { server.shutdown(); } ``` **Configuration**: ```java import convex.peer.Config; Config config = Config.create(); config = config.withPort(18888); config = config.withRestPort(8080); Server server = Server.create(config); server.launch(); ``` **Advantages**: - Sub-millisecond latency - No network dependency - Full control - Built-in account creation **→ For production peer deployment, see the deployment guides below.** ## Running Production Peers For production peer operation, see the following guides: ### Getting Started 1. **[Peer Staking & Registration](peer-operations/staking)** - Stake coins and register your peer 2. **[Choose Deployment Method](peer-operations/manual-deployment)** - Manual or Docker 3. **[Select Hosting](peer-operations/hosting)** - Infrastructure requirements 4. **[Secure Your Peer](peer-operations/security)** - Security best practices ### Deployment Options - **[Manual Deployment](peer-operations/manual-deployment)** - Traditional installation - **[Docker Deployment](peer-operations/docker-deployment)** - Containerized deployment ### Operations - **[Troubleshooting](peer-operations/troubleshooting)** - Common issues and solutions - **[Security](peer-operations/security)** - Secure your infrastructure ## Next Steps **For Developers**: - **[Networks Guide](/docs/tutorial/networks)** - Available networks - **[Client SDKs](/docs/tutorial/client-sdks)** - Connect from your language - **[Client Types](/docs/tutorial/client-sdks/java/clients)** - Detailed SDK guide **For Operators**: - **[Staking & Registration](peer-operations/staking)** - Start running a peer - **[Deployment Guides](peer-operations/manual-deployment)** - Installation instructions - **[Security Guide](peer-operations/security)** - Secure your peer ## Resources - **[Discord Community](https://discord.com/invite/xfYGq4CT7v)** - `#peer-operations` channel - **[Convex.world](https://convex.world)** - Official website - **[GitHub](https://github.com/Convex-Dev/convex)** - Source code --- # Local Testnets Run a local Convex testnet for development and testing. ## Overview Local testnets provide: - **Fastest development** - No network latency - **Full control** - Configure as needed - **Free funds** - Create accounts at will - **Privacy** - No external exposure - **Offline work** - No internet required ## Options Choose the method that best fits your workflow: | Method | Best For | Setup Time | Complexity | |--------|----------|------------|------------| | **[JVM Direct](#jvm-direct)** | Unit tests, library integration | Instant | Low | | **[Convex Desktop](#convex-desktop)** | Interactive development, GUI | 1 minute | Low | | **[CLI](#cli-peer)** | Scripts, automation | 1 minute | Low | | **[Docker](#docker-peer)** | Consistent environment, CI/CD | 2 minutes | Medium | ## JVM Direct Launch a peer directly in your JVM process - fastest option for testing. ### Use Cases - Unit tests - Integration tests - Library development - Quick experiments ### Quick Start **Java**: ```java import convex.peer.Server; import convex.api.Convex; import convex.core.Result; import convex.core.crypto.AKeyPair; import convex.core.cvm.Address; import convex.core.lang.Reader; public class LocalTestnetExample { public static void main(String[] args) throws Exception { // Launch local peer Server server = Server.create(); server.launch(); try { // Connect directly Convex convex = Convex.connect(server); // Create test account AKeyPair keyPair = AKeyPair.generate(); Address address = convex.createAccountSync(keyPair.getAccountKey()); // Fund account (no faucet needed, you control the peer!) convex.setKeyPair(keyPair); convex.setAddress(address); // Use for testing Result result = convex.transact(Reader.read("(def test-value 42)")).get(); System.out.println("Result: " + result.getValue()); } finally { // Clean shutdown server.shutdown(); } } } ``` ### JUnit Test Example ```java import org.junit.jupiter.api.*; import convex.peer.Server; import convex.api.Convex; class MyConvexTest { private static Server server; private Convex convex; @BeforeAll static void startPeer() throws Exception { server = Server.create(); server.launch(); } @BeforeEach void connect() throws Exception { convex = Convex.connect(server); // Create fresh account for each test AKeyPair keyPair = AKeyPair.generate(); Address address = convex.createAccountSync(keyPair.getAccountKey()); convex.setKeyPair(keyPair); convex.setAddress(address); } @Test void testTransaction() throws Exception { Result result = convex.transact(Reader.read("(+ 1 2 3)")).get(); assertEquals(6L, result.getValue()); } @AfterEach void disconnect() throws Exception { convex.close(); } @AfterAll static void stopPeer() throws Exception { server.shutdown(); } } ``` ### Configuration ```java import convex.peer.Config; // Custom configuration Config config = Config.create(); config = config.withPort(18888); config = config.withRestPort(8080); Server server = Server.create(config); server.launch(); ``` ### Advantages - ⚡ **Sub-millisecond latency** - Direct method calls - 🔧 **Full control** - Configure everything - 🧪 **Isolated** - Clean state per test - 📦 **No external dependencies** - Pure Java ### Considerations - ⚠️ Requires 2-4 GB RAM - ⚠️ State is ephemeral (lost on shutdown) - ⚠️ Single-peer network (no consensus testing) ## Convex Desktop Run a local peer with a full GUI for interactive development. ### Use Cases - Interactive REPL - Visual account management - Transaction debugging - Learning Convex Lisp - Smart contract development ### Installation **Download**: ```bash # Linux/Mac wget https://github.com/Convex-Dev/convex/releases/download/0.8.8/convex.jar # Or build from source git clone https://github.com/Convex-Dev/convex.git cd convex && mvn install ``` ### Launch Desktop ```bash # Start GUI java -jar convex.jar desktop # With custom memory java -Xmx4g -jar convex.jar desktop ``` ### Features **GUI Includes**: - **Peer Control** - Start/stop local peer - **Account Manager** - Create/manage accounts - **REPL** - Interactive Convex Lisp console - **Actor Deployer** - Deploy smart contracts - **Network Explorer** - Browse accounts and state - **Transaction Builder** - Build and submit transactions ### Quick Start Workflow 1. **Launch** Convex Desktop 2. **Start Peer** - Click "Start Local Peer" 3. **Create Account** - Tools → New Account 4. **Fund Account** - Request funds (automatic for local peer) 5. **Use REPL** - Interactive Convex Lisp console ### Example: Deploy Actor via Desktop ```clojure ;; In Desktop REPL ;; 1. Create account (if not already) (create-account) ;; 2. Deploy actor (deploy '(do (defn ^:callable greet [name] (str "Hello, " name "!")))) ;; 3. Call actor function (call actor-address (greet "World")) ``` ### Advantages - 🎨 **Visual interface** - See everything - 🔍 **REPL** - Instant feedback - 📊 **State exploration** - Browse accounts - 🎓 **Learning-friendly** - Interactive environment ### Considerations - ⚠️ GUI overhead (slower than headless) - ⚠️ Not suitable for automation - ⚠️ Requires display (no headless servers) ## CLI Peer Run a local peer from the command line - ideal for scripts and automation. ### Use Cases - Shell scripts - Automation - CI/CD pipelines - Headless servers - Background services ### Quick Start ```bash # Start a temporary local test network (1 peer + REST API on 8080) java -jar convex.jar local start --api-port 8080 & # Wait for startup sleep 5 # Use peer curl http://localhost:8080/api/v1/query \ -H "Content-Type: application/json" \ -d '{"source":"(+ 1 2 3)"}' # Stop peer (Ctrl-C in the foreground, or kill the background job) kill %1 ``` ### Options `convex local start` runs a temporary network — no config file needed. Useful flags: | Flag | Purpose | |------|---------| | `--count` | Number of local peers to start (default 1) | | `--api-port` | REST API port on the first peer (default 8080) | | `--ports` | Explicit peer ports (otherwise free ports are chosen) | The command prints the actual peer ports in use on startup. ### Automation Script ```bash #!/bin/bash # start-local-testnet.sh CONVEX_JAR="convex.jar" PID_FILE="peer.pid" # Start a temporary local network java -jar $CONVEX_JAR local start --api-port 8080 & echo $! > $PID_FILE # Wait for ready echo "Waiting for peer to start..." while ! curl -s http://localhost:8080/api/v1/status > /dev/null; do sleep 1 done echo "✓ Local peer ready at http://localhost:8080" ``` ```bash #!/bin/bash # stop-local-testnet.sh PID_FILE="peer.pid" if [ -f $PID_FILE ]; then kill $(cat $PID_FILE) rm $PID_FILE echo "✓ Peer stopped" else echo "No peer PID file found" fi ``` ### Using in Scripts ```bash # Start peer ./start-local-testnet.sh # Run tests ./run-tests.sh # Stop peer ./stop-local-testnet.sh ``` ### CI/CD Integration **GitHub Actions**: ```yaml name: Test with Local Peer jobs: test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 - name: Set up Java uses: actions/setup-java@v2 with: java-version: '21' - name: Start Local Peer run: | wget https://github.com/Convex-Dev/convex/releases/download/0.8.8/convex.jar java -jar convex.jar local start --api-port 8080 & sleep 10 - name: Run Tests run: mvn test - name: Stop Peer run: pkill -f convex.jar ``` ### Advantages - 🚀 **Scriptable** - Automation-friendly - 🔄 **CI/CD** - Perfect for pipelines - 💻 **Headless** - No GUI needed - 📝 **Logging** - File-based logs ### Considerations - ⚠️ Manual process management - ⚠️ Requires cleanup handling - ⚠️ No interactive features ## Docker Peer Run a local peer in a Docker container for consistent, isolated environment. ### Use Cases - Consistent development environment - Multi-developer teams - CI/CD pipelines - Isolated testing - Easy cleanup ### Quick Start **Docker Run**: ```bash # Start local peer docker run -d \ --name convex-local \ -p 18888:18888 \ -p 8080:8080 \ -e LOCAL_MODE=true \ convex/convex:latest peer start # Check status docker logs convex-local # Use peer curl http://localhost:8080/api/v1/status # Stop and remove docker stop convex-local docker rm convex-local ``` ### Docker Compose Create `docker-compose.yml`: ```yaml version: '3.8' services: convex-local: image: convex/convex:latest container_name: convex-local-testnet ports: - "18888:18888" - "8080:8080" environment: - LOCAL_MODE=true - JAVA_OPTS=-Xmx2g - LOG_LEVEL=info volumes: - convex-local-data:/app/data command: peer start volumes: convex-local-data: ``` **Usage**: ```bash # Start docker-compose up -d # View logs docker-compose logs -f # Stop docker-compose down # Stop and remove data docker-compose down -v ``` ### Makefile for Convenience ```makefile .PHONY: start stop logs clean test start: docker-compose up -d @echo "Waiting for peer to be ready..." @sleep 5 @echo "✓ Local testnet ready at http://localhost:8080" stop: docker-compose down logs: docker-compose logs -f clean: docker-compose down -v docker system prune -f test: start ./run-tests.sh $(MAKE) stop ``` **Usage**: ```bash make start # Start local testnet make test # Run tests make stop # Stop testnet make clean # Clean up everything ``` ### CI/CD Integration **GitHub Actions with Docker**: ```yaml name: Test with Docker Peer jobs: test: runs-on: ubuntu-latest services: convex: image: convex/convex:latest ports: - 18888:18888 - 8080:8080 env: LOCAL_MODE: true steps: - uses: actions/checkout@v2 - name: Wait for Peer run: | timeout 60 bash -c 'until curl -s http://localhost:8080/api/v1/status; do sleep 1; done' - name: Run Tests run: mvn test -Dconvex.peer.url=http://localhost:8080 ``` ### Advantages - 📦 **Consistent environment** - Same everywhere - 🔒 **Isolated** - No host contamination - 🧹 **Easy cleanup** - `docker-compose down -v` - 🚀 **CI/CD friendly** - Standard tooling ### Considerations - ⚠️ Docker overhead (slight performance impact) - ⚠️ Requires Docker installed - ⚠️ Volume management needed for persistence ## Comparison ### Performance | Method | Startup Time | Latency | Throughput | |--------|-------------|---------|------------| | JVM Direct | Instant | Sub-ms | Highest | | Convex Desktop | ~5 seconds | Sub-ms | High | | CLI | ~3 seconds | Sub-ms | High | | Docker | ~5 seconds | 1-2ms | Good | ### Best Practices **Development**: - Use **JVM Direct** for unit tests (fastest) - Use **Desktop** for interactive development - Use **CLI** for integration tests - Use **Docker** for team consistency **CI/CD**: - Prefer **Docker** (consistent, isolated) - Use **CLI** if Docker unavailable - Avoid **Desktop** (GUI not needed) - Consider **JVM Direct** for pure Java tests **Learning**: - Start with **Desktop** (visual, interactive) - Move to **JVM Direct** as you progress - Experiment with **CLI** for automation - Try **Docker** for deployment practice ## Next Steps ### For Development - **[Client SDKs](/docs/tutorial/client-sdks)** - Connect from your language - **[Convex Lisp Guide](/docs/tutorial/convex-lisp)** - Learn the language - **[Smart Contract Tutorial](/docs/tutorial/actors)** - Build actors ### For Production - **[Networks Guide](/docs/tutorial/networks)** - Public networks - **[Manual Deployment](manual-deployment)** - Production setup - **[Docker Deployment](docker-deployment)** - Container deployment ## Resources - **[GitHub Repository](https://github.com/Convex-Dev/convex)** - Source code - **[Discord Community](https://discord.com/invite/xfYGq4CT7v)** - Get help - **[REST API Reference](https://peer.convex.live/swagger)** - API reference --- # Manual Deployment Install and configure a Convex peer manually for production use. ## Overview Manual deployment gives you full control over peer installation and configuration. This guide covers traditional installation on Linux/Unix systems. **→ For containerized deployment, see [Docker Deployment](docker-deployment)** ## Prerequisites ### System Requirements **Minimum**: - 4 CPU cores - 8 GB RAM - 100 GB SSD storage - Ubuntu 20.04 LTS or later (or equivalent) **Recommended**: - 8+ CPU cores - 16 GB+ RAM - 500 GB+ SSD storage - Dedicated server or VM ### Software Requirements - **Java 21+** - OpenJDK or Oracle JDK - **Git** - For source checkout (optional) - **Network Access** - Ports 18888 (peer) and 8080 (REST API) ## Installation Steps ### Step 1: Install Java 21 ```bash # Ubuntu/Debian sudo apt update sudo apt install openjdk-21-jdk # Verify installation java -version # Should show: openjdk version "21..." or higher ``` ### Step 2: Download Convex **Option A: Download Release** ```bash # Create installation directory mkdir -p /opt/convex cd /opt/convex # Download latest release wget https://github.com/Convex-Dev/convex/releases/download/0.8.8/convex.jar # Verify download java -jar convex.jar version ``` **Option B: Build from Source** ```bash # Clone repository git clone https://github.com/Convex-Dev/convex.git cd convex # Build with Maven mvn clean install -DskipTests # Copy JAR cp convex-integration/target/convex.jar /opt/convex/ ``` ### Step 3: Generate Peer Keys ```bash cd /opt/convex # Generate peer key pair java -jar convex.jar key generate --type random # View public key (needed for staking) java -jar convex.jar key list ``` **⚠️ Security**: Store `peer-keys.dat` securely with restricted permissions (600). ### Step 4: Prepare Keys and Store Convex peers are configured by command-line flags and a keystore — there is no config file. Make sure: - Your **peer key** is in the keystore (Step 3). - The peer has been **created and staked** on the network — see the [Staking guide](staking) (`create-peer`). Note the peer's public key from `convex key list`; you pass it to `peer start` with `--peer-key`. ### Step 5: Create Data Directory ```bash mkdir -p /opt/convex/data/peer-store chown -R convex:convex /opt/convex/data chmod 755 /opt/convex/data ``` ## Running the Peer ### Manual Start ```bash cd /opt/convex # Start the peer (Etch store under data/peer-store, REST API on 8080) java -Xmx4g -jar convex.jar peer start \ --peer-key 0x \ --etch data/peer-store \ --peer-port 18888 \ --api-port 8080 \ --url your-peer.example.com:18888 # Peer will start and begin syncing. Run `convex peer start --help` for all options. ``` ### Check Status ```bash # Check peer status / heartbeat curl http://localhost:8080/api/v1/status ``` ## Systemd Service For production, run as a systemd service: ### Create Service Unit Create `/etc/systemd/system/convex-peer.service`: ```ini [Unit] Description=Convex Peer Node After=network.target [Service] Type=simple User=convex Group=convex WorkingDirectory=/opt/convex ExecStart=/usr/bin/java -Xmx4g -jar /opt/convex/convex.jar peer start --peer-key 0x --etch data/peer-store --peer-port 18888 --api-port 8080 --url your-peer.example.com:18888 Restart=on-failure RestartSec=10s # Resource limits LimitNOFILE=65536 LimitNPROC=4096 # Logging StandardOutput=journal StandardError=journal SyslogIdentifier=convex-peer [Install] WantedBy=multi-user.target ``` ### Create Convex User ```bash # Create system user sudo useradd -r -s /bin/false convex # Set ownership sudo chown -R convex:convex /opt/convex ``` ### Enable and Start Service ```bash # Reload systemd sudo systemctl daemon-reload # Enable service (start on boot) sudo systemctl enable convex-peer # Start service sudo systemctl start convex-peer # Check status sudo systemctl status convex-peer ``` ### View Logs ```bash # Follow logs sudo journalctl -u convex-peer -f # View recent logs sudo journalctl -u convex-peer -n 100 ``` ## Configuration Options ### Key `peer start` Flags | Flag | Purpose | |------|---------| | `--peer-key` | Public key of the peer (must be in the keystore) | | `--etch` | Path to the peer's Etch store | | `--peer-port` | Peer protocol port (default 18888) | | `--api-port` | REST API port (default 8080) | | `--url` | Public URL other peers use to reach this peer | | `--genesis` | Genesis seed — **test networks only** | | `--reset` | Delete and recreate the Etch store | Run `convex peer start --help` for the complete list. ### Performance Tuning ```clojure {:max-connections 1000 ; Maximum peer connections :thread-pool-size 16 ; Worker threads :cache-size 1000000 ; State cache size :sync-batch-size 1000} ; Blocks per sync batch ``` ### Logging Configuration ```clojure {:log-level :info ; :debug :info :warn :error :log-file "logs/peer.log" ; Log file path :log-rotation :daily} ; :hourly :daily :weekly ``` ## Monitoring ### Health Checks ```bash # Check if peer is responding curl http://localhost:8080/api/v1/status # Query consensus state curl http://localhost:8080/api/v1/query \ -H "Content-Type: application/json" \ -d '{"source":"*state*"}' ``` ### Metrics Monitor key metrics: - Consensus point (should advance regularly) - Memory usage - Connection count - Transaction throughput ### Log Monitoring ```bash # Watch for errors sudo journalctl -u convex-peer -f | grep ERROR # Monitor consensus sudo journalctl -u convex-peer -f | grep CONSENSUS ``` ## Upgrading ### Upgrade Process ```bash # Stop peer sudo systemctl stop convex-peer # Backup data tar -czf backup-$(date +%Y%m%d).tar.gz /opt/convex/data # Download new version cd /opt/convex wget https://github.com/Convex-Dev/convex/releases/download/0.8.8/convex.jar \ -O convex.jar.new # Replace JAR mv convex.jar convex.jar.old mv convex.jar.new convex.jar # Start peer sudo systemctl start convex-peer # Monitor startup sudo journalctl -u convex-peer -f ``` ## Backup and Recovery ### Backup Strategy ```bash # Stop peer for consistent backup sudo systemctl stop convex-peer # Backup data directory tar -czf backup-$(date +%Y%m%d).tar.gz /opt/convex/data # Back up the Etch store (peer state) and your keystore (peer keys) cp -r data/peer-store data/peer-store.backup # also back up your keystore file (the path passed to --keystore, e.g. ~/.convex/keystore.pfx) # Restart peer sudo systemctl start convex-peer ``` ### Recovery Process ```bash # Stop peer sudo systemctl stop convex-peer # Restore data tar -xzf backup-20260210.tar.gz -C / # Verify permissions sudo chown -R convex:convex /opt/convex/data # Start peer sudo systemctl start convex-peer ``` ## Next Steps 1. **[Staking & Registration](staking)** - Stake and register your peer 2. **[Hosting Options](hosting)** - Infrastructure selection 3. **[Security Guide](security)** - Secure your deployment 4. **[Troubleshooting](troubleshooting)** - Common issues ## Resources - **[Docker Deployment](docker-deployment)** - Alternative deployment method - **[Convex CLI Reference](/)** - Complete CLI documentation - **[Discord Community](https://discord.com/invite/xfYGq4CT7v)** - Get help --- # Security Considerations Security best practices for operating Convex peer nodes. ## Overview Secure peer operation requires: - 🔐 Key management - 🛡️ Network security - 👁️ Monitoring and alerting - 🔄 Regular updates - 📋 Incident response planning ## Key Management ### Peer Keys **Protection**: ```bash # Restrict permissions chmod 600 /opt/convex/peer-keys.dat chown convex:convex /opt/convex/peer-keys.dat # Verify permissions ls -la /opt/convex/peer-keys.dat # Should show: -rw------- 1 convex convex ``` **Backup**: ```bash # Encrypted backup gpg --encrypt --recipient admin@example.com peer-keys.dat # Store in multiple secure locations # - Encrypted cloud storage # - Hardware security module # - Offline cold storage ``` **Rotation**: - Rotate peer keys annually (if supported) - Generate new keys if compromise suspected - Keep backup of old keys for recovery ### Stake Controller Keys **Separation**: - Never store on same system as peer keys - Use hardware wallet for high-value stakes - Consider multi-signature for large stakes **Cold Storage**: ```bash # Generate offline # Transfer only public key to peer # Sign transactions on air-gapped system ``` **Access Control**: - Limit who can access stake controller keys - Use principle of least privilege - Audit access logs regularly ## Network Security ### Firewall Configuration **UFW (Ubuntu)**: ```bash # Default deny sudo ufw default deny incoming sudo ufw default allow outgoing # Allow SSH sudo ufw allow 22/tcp # Allow peer protocol sudo ufw allow 18888/tcp # Allow REST API (optional, if public) sudo ufw allow 8080/tcp # Enable firewall sudo ufw enable ``` **iptables**: ```bash # Allow peer protocol iptables -A INPUT -p tcp --dport 18888 -j ACCEPT # Allow REST API from specific IP iptables -A INPUT -p tcp --dport 8080 -s 192.168.1.0/24 -j ACCEPT # Drop everything else iptables -A INPUT -j DROP ``` ### SSH Hardening **Best Practices**: ```bash # /etc/ssh/sshd_config # Disable root login PermitRootLogin no # Use key-based auth only PasswordAuthentication no PubkeyAuthentication yes # Limit users AllowUsers convex admin # Change default port (optional) Port 2222 ``` ### SSL/TLS for REST API **Using Nginx Reverse Proxy**: ```nginx server { listen 443 ssl http2; server_name peer.example.com; ssl_certificate /etc/ssl/certs/peer.crt; ssl_certificate_key /etc/ssl/private/peer.key; ssl_protocols TLSv1.2 TLSv1.3; ssl_ciphers HIGH:!aNULL:!MD5; location /api/ { proxy_pass http://localhost:8080/api/; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; } } ``` ## Operating System Security ### System Updates **Automatic Updates** (Ubuntu): ```bash # Install unattended-upgrades sudo apt install unattended-upgrades # Configure sudo dpkg-reconfigure -plow unattended-upgrades # /etc/apt/apt.conf.d/50unattended-upgrades Unattended-Upgrade::Automatic-Reboot "false"; Unattended-Upgrade::Automatic-Reboot-Time "03:00"; ``` ### User Security **Create Dedicated User**: ```bash # Create system user sudo useradd -r -s /bin/false convex # Limit privileges # - No shell access # - No sudo rights # - Only peer files access ``` **Audit Logging**: ```bash # Enable auditd sudo apt install auditd # Monitor peer files sudo auditctl -w /opt/convex/ -p wa -k convex-watch ``` ## Application Security ### Java Security Manager **Enable Security Manager**: ```bash java -Djava.security.manager \ -Djava.security.policy=/opt/convex/security.policy \ -jar convex.jar peer start ``` **Policy File** (`security.policy`): ```java grant codeBase "file:/opt/convex/convex.jar" { permission java.net.SocketPermission "*:18888", "connect,accept,listen"; permission java.net.SocketPermission "*:8080", "connect,accept,listen"; permission java.io.FilePermission "/opt/convex/data/-", "read,write"; }; ``` ### JVM Hardening **Security Options**: ```bash java -Xmx4g \ -Djava.security.egd=file:/dev/urandom \ -Djavax.net.ssl.trustStore=/opt/convex/truststore.jks \ -jar convex.jar peer start ``` ## Monitoring and Alerting ### Security Monitoring **Monitor**: - Failed authentication attempts - Unusual network activity - Unexpected peer connections - File system changes - Resource usage spikes **Log Monitoring**: ```bash # Watch for security events sudo journalctl -u convex-peer -f | grep -i "security\|auth\|error" # Alert on suspicious activity # Integrate with monitoring system (Prometheus, Datadog, etc.) ``` ### Intrusion Detection **Fail2ban**: ```bash # Install sudo apt install fail2ban # Configure for SSH # /etc/fail2ban/jail.local [sshd] enabled = true port = 22 maxretry = 3 bantime = 3600 ``` **AIDE (File Integrity)**: ```bash # Install sudo apt install aide # Initialize database sudo aideinit # Run checks sudo aide --check ``` ## Backup Security ### Encrypted Backups ```bash # Encrypt with GPG tar czf - /opt/convex/data | \ gpg --encrypt --recipient backup@example.com > \ backup-$(date +%Y%m%d).tar.gz.gpg # Upload to secure storage aws s3 cp backup-*.tar.gz.gpg s3://secure-bucket/ \ --sse AES256 ``` ### Backup Verification ```bash # Regular backup tests # 1. Download backup # 2. Decrypt # 3. Extract # 4. Verify integrity # 5. Test restore procedure ``` ## Incident Response ### Response Plan **Preparation**: 1. Document incident response procedures 2. Identify key personnel 3. Maintain contact information 4. Test response plan quarterly **Detection**: - Automated monitoring alerts - Log analysis - Community reports - Anomaly detection **Response Steps**: 1. Assess severity 2. Contain threat 3. Investigate root cause 4. Remediate 5. Document incident 6. Post-mortem review ### Compromise Response **If Peer Compromised**: 1. Immediately stop peer 2. Disconnect from network 3. Preserve logs for analysis 4. Use stake controller (from secure system) to withdraw stake 5. Investigate compromise vector 6. Rebuild from clean state 7. Generate new peer keys 8. Implement additional security measures **If Stake Controller Compromised**: 1. Attempt emergency stake withdrawal (if possible) 2. Alert community immediately 3. Document compromise for network consideration 4. Legal reporting (if applicable) ## Compliance and Best Practices ### Security Checklist **Pre-Deployment**: - [ ] Keys generated securely offline - [ ] Keys backed up and tested - [ ] Firewall configured - [ ] SSH hardened - [ ] System updates enabled - [ ] Monitoring configured - [ ] Incident response plan documented **Ongoing**: - [ ] Weekly security updates applied - [ ] Monthly backup verification - [ ] Quarterly security audit - [ ] Annual key rotation review - [ ] Regular log review - [ ] Performance monitoring ### Audit Logging **Enable Comprehensive Logging**: ```bash # System logs sudo journalctl -u convex-peer # Security logs sudo journalctl -u ssh # Audit logs sudo ausearch -k convex-watch ``` **Log Retention**: - Application logs: 30 days - Security logs: 90 days - Audit logs: 1 year ## Cloud Security ### AWS Security **Security Groups**: ```hcl # Terraform example resource "aws_security_group" "convex_peer" { name = "convex-peer" ingress { from_port = 18888 to_port = 18888 protocol = "tcp" cidr_blocks = ["0.0.0.0/0"] } # Restrict REST API ingress { from_port = 8080 to_port = 8080 protocol = "tcp" cidr_blocks = ["10.0.0.0/8"] # Internal only } } ``` **IAM Roles**: ```json { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Action": [ "s3:PutObject", "s3:GetObject" ], "Resource": "arn:aws:s3:::convex-backups/*" }] } ``` ### GCP Security **Firewall Rules**: ```bash # Allow peer protocol gcloud compute firewall-rules create convex-peer \ --allow tcp:18888 \ --source-ranges 0.0.0.0/0 # Allow REST API from internal gcloud compute firewall-rules create convex-api \ --allow tcp:8080 \ --source-ranges 10.0.0.0/8 ``` ## References - **[OWASP Top 10](https://owasp.org/www-project-top-ten/)** - Security risks - **[CIS Benchmarks](https://www.cisecurity.org/)** - Hardening guidelines - **[NIST Cybersecurity Framework](https://www.nist.gov/cyberframework)** - Security framework ## Next Steps - **[Monitoring](troubleshooting#monitoring-and-diagnosis)** - Set up monitoring - **[Troubleshooting](troubleshooting)** - Common security issues - **[Deployment Guides](manual-deployment)** - Secure deployment ## Resources - **[Discord Security Channel](https://discord.com/invite/xfYGq4CT7v)** - Security discussions - **[Security Advisories](/)** - Network security updates - **[Bug Bounty](/)** - Report vulnerabilities --- # Peer Staking and Registration How to stake Convex Coins and register your peer on the network. ## Overview To participate in Convex consensus as a peer, you must: 1. **Stake Convex Coins** — lock coins as collateral with `create-peer` 2. **Register peer data** — publish your peer's connection URL with `set-peer-data` 3. **Maintain stake** — keep sufficient stake to remain an active part of consensus Staking uses a small set of CVM functions: `create-peer`, `set-peer-stake`, `set-stake`, `set-peer-data`, `get-peer-stake` and `evict-peer`. This guide shows each one. The economic model (rewards, slashing, delegated stake) is specified in [CAD016: Peer Staking](/docs/cad/peerstake). ## Prerequisites - Funded Convex account with sufficient CVM balance (stake + juice) - Ed25519 key pair for the peer identity (the *peer key*) - Peer infrastructure ready ([deployment guides](manual-deployment)) - Network connectivity configured ## Staking Requirements ### Minimum Stake Running a peer on Protonet requires a minimum stake of **1000 CVM**. A higher stake gives your peer proportionally greater weight in consensus. On testnets you can practise with smaller amounts. ### Stake Economics **Benefits** - Participate in Convex Convergent Proof of Stake (CPoS) consensus - Earn rewards proportional to stake (rewards accrue to the peer's stake automatically — there is no separate claim step; see [CAD016](/docs/cad/peerstake) and [CAD020: Tokenomics](/docs/cad/tokenomics)) **Risks** - Stake is locked while the peer participates - Provable misbehaviour may lead to **slashing** (loss of stake) ## Generating Peer Keys Each peer needs a unique Ed25519 key pair for its identity. ### Using the CLI Keys are generated into your configured keystore (a PKCS#12 file), not a loose seed file: ```bash # Generate a new random key pair in the keystore convex key generate --type random # List the public keys in your keystore convex key list ``` ### Using Java ```java import convex.core.crypto.AKeyPair; // Generate a peer key pair AKeyPair peerKeys = AKeyPair.generate(); System.out.println("Peer Public Key: " + peerKeys.getAccountKey()); ``` **⚠️ Security:** store peer keys securely. Loss of the peer key means loss of the peer identity. ## Staking Process ### Step 1: Prepare the Controlling Account The account that submits `create-peer` controls the peer's stake. Ensure it has enough funds for the stake plus juice: ```java import convex.api.Convex; import convex.core.Result; import convex.core.crypto.AKeyPair; import convex.core.cvm.Address; import convex.core.lang.Reader; // Connect to your target network (Protonet shown; use the public testnet to practise) Convex convex = Convex.connect("https://peer.convex.live"); AKeyPair controllerKeys = AKeyPair.create(controllerKeySeed); convex.setKeyPair(controllerKeys); convex.setAddress(controllerAddress); Result balance = convex.query(Reader.read("(balance " + controllerAddress + ")")).get(); long balanceCopper = ((Number) balance.getValue()).longValue(); System.out.println("Balance: " + (balanceCopper / 1_000_000_000.0) + " CVM"); ``` ### Step 2: Create the Peer `create-peer` registers the peer key and places its initial stake (in copper): ```java import convex.core.data.AccountKey; AccountKey peerKey = peerKeys.getAccountKey(); long stakeAmount = 1_000L * 1_000_000_000L; // 1000 CVM in copper String stakeCommand = String.format("(create-peer %s %d)", peerKey, stakeAmount); Result result = convex.transact(Reader.read(stakeCommand)).get(); if (!result.isError()) { System.out.println("✓ Peer created with stake"); } else { System.err.println("✗ create-peer failed: " + result.getErrorCode()); } ``` ### Step 3: Register Peer Data Publish the peer's connection URL so others can reach it. The metadata map uses a single `:url` of the form `"host:port"`: ```java String registerCommand = String.format( "(set-peer-data %s {:url \"peer.example.com:18888\"})", peerKey); Result result = convex.transact(Reader.read(registerCommand)).get(); System.out.println(result.isError() ? "✗ " + result.getErrorCode() : "✓ Peer data registered"); ``` ## Managing Stake ### Checking Stake `get-peer-stake` reads a peer's current stake: ```java String stakeQuery = String.format("(get-peer-stake %s)", peerKey); Result result = convex.query(Reader.read(stakeQuery)).get(); if (!result.isError()) { long stake = ((Number) result.getValue()).longValue(); System.out.println("Current Stake: " + (stake / 1_000_000_000.0) + " CVM"); } ``` ### Changing Your Peer's Stake `set-peer-stake` sets the peer's stake to an **absolute** value — there is no separate add/withdraw. To add stake, set a higher total; to reduce it, set a lower total; to fully unstake, set it to `0`: ```java // Increase total stake to 2000 CVM long newStake = 2_000L * 1_000_000_000L; convex.transact(Reader.read(String.format("(set-peer-stake %s %d)", peerKey, newStake))).get(); // Fully unstake (stop participating in consensus) convex.transact(Reader.read(String.format("(set-peer-stake %s 0)", peerKey))).get(); ``` **⚠️ Warning:** setting stake to `0` removes the peer from consensus. Shut the peer down cleanly first. ### Delegated Stake Any coin holder can back a peer they trust with *delegated* stake using `set-stake` (also an absolute set). This adds to the peer's consensus weight and lets the delegator share in rewards, without running a peer: ```java // Delegate 500 CVM of stake to a trusted peer long delegated = 500L * 1_000_000_000L; convex.transact(Reader.read(String.format("(set-stake %s %d)", trustedPeerKey, delegated))).get(); ``` ### Evicting a Peer `evict-peer` removes an inactive or misbehaving peer from the peer set (subject to network rules): ```java convex.transact(Reader.read(String.format("(evict-peer %s)", peerKey))).get(); ``` ## Stake Security ✅ **Key management** - Keep the peer key and the controlling account key separate - Use hardware security modules (HSM) for high-value stakes - Maintain secure offline backups ✅ **Stake protection** - Monitor peer performance to avoid slashing - Keep infrastructure maintained and reachable at the registered URL - Set up alerting for stake changes ### Recovery Scenarios **Lost peer key** — the peer cannot sign consensus messages; the controlling account can still adjust or remove the stake with `set-peer-stake`. Generate a new peer key and re-run `create-peer`. **Lost controlling account key** — you cannot modify the stake. Protect this key accordingly. ## Monitoring Stake ```java public class StakeMonitor { private final Convex convex; private final AccountKey peerKey; public StakeMonitor(Convex convex, AccountKey peerKey) { this.convex = convex; this.peerKey = peerKey; } public void checkStake() throws Exception { String query = String.format("(get-peer-stake %s)", peerKey); Result result = convex.query(Reader.read(query)).get(); if (result.isError()) { System.err.println("⚠ Cannot query stake"); return; } long stake = ((Number) result.getValue()).longValue(); double stakeCoins = stake / 1_000_000_000.0; long minStake = 1_000L; // 1000 CVM Protonet minimum if (stakeCoins < minStake) { System.err.println("⚠ Stake below minimum: " + stakeCoins + " CVM"); } else { System.out.println("✓ Stake adequate: " + stakeCoins + " CVM"); } } } ``` Set up alerting on: stake below the minimum threshold, unexpected stake changes, and loss of consensus participation. ## Troubleshooting ### `create-peer` / `set-peer-stake` failed - **`:FUNDS`** — the controlling account lacks the balance for the stake plus juice. Check `(balance ...)`. - **`:STATE`** — the peer already exists, or the stake is below the network minimum. - **`:TRUST`** — the submitting account is not authorised to control this peer. ### Peer not appearing in consensus - Stake below the minimum, or set to `0` - Peer not reachable at its registered `:url` - Registration metadata incorrect — re-check `set-peer-data` ## Next Steps 1. **[Choose a deployment method](manual-deployment)** — install your peer 2. **[Select hosting](hosting)** — infrastructure requirements 3. **[Security guide](security)** — secure your peer 4. **[Troubleshooting](troubleshooting)** — common issues ## Resources - **[Discord Community](https://discord.com/invite/xfYGq4CT7v)** — `#peer-operations` channel - **[CAD016: Peer Staking](/docs/cad/peerstake)** — the staking and rewards model - **[CAD017: Peer Operations](/docs/cad/peerops)** — running a peer - **[CAD020: Tokenomics](/docs/cad/tokenomics)** — network economics --- # Troubleshooting Common issues and solutions for Convex peer operations. ## Connection Issues ### Cannot Connect to Peer **Symptoms**: Client cannot reach peer endpoint **Check**: ```bash # Test peer port telnet peer.example.com 18888 # Test REST API curl http://peer.example.com:8080/api/v1/status ``` **Common Causes**: - Firewall blocking ports - Peer not running - Incorrect hostname/IP - Network connectivity **Solutions**: 1. Verify peer is running: `systemctl status convex-peer` 2. Check firewall rules: `sudo ufw status` 3. Verify port binding: `netstat -tlnp | grep 18888` 4. Check logs: `journalctl -u convex-peer` ### Peer Cannot Sync **Symptoms**: Consensus point not advancing **Check Sync Status**: ```bash curl http://localhost:8080/api/v1/query \ -d '{"source":"*state*"}' ``` **Common Causes**: - No peers connected - Network issues - Corrupted state - Insufficient resources **Solutions**: 1. Check peer connections 2. Verify bootstrap peers configured 3. Check network connectivity 4. Review resource usage (CPU, memory, disk) 5. Consider state reset if corrupted ## Performance Issues ### High Latency **Symptoms**: Slow query/transaction responses **Diagnose**: ```bash # Check system load top # Check disk I/O iostat -x 1 # Check network iftop ``` **Solutions**: - Increase JVM heap: `-Xmx8g` - Upgrade to SSD storage - Increase CPU/RAM - Optimise network ### High Memory Usage **Symptoms**: Peer consuming excessive RAM **Check Memory**: ```bash # JVM memory jmap -heap # System memory free -h ``` **Solutions**: - Increase JVM heap if needed - Check for memory leaks (monitor over time) - Review cache configuration - Consider heap dump analysis ### Slow State Access **Symptoms**: Database queries slow **Check**: - Disk I/O performance - Storage type (HDD vs SSD) - Available disk space - Database corruption **Solutions**: - Migrate to faster storage - Clear unnecessary logs - Verify database integrity - Consider re-sync from peers ## Consensus Issues ### Not Participating in Consensus **Symptoms**: Peer not producing blocks/proposals **Check**: 1. Verify stake amount sufficient 2. Check peer registration 3. Verify peer keys correct 4. Check consensus logs **Solutions**: - Review [staking guide](staking) - Verify peer metadata - Check stake controller permissions - Review consensus configuration ### Fork Detection **Symptoms**: Different state than network **Check Genesis**: ```bash curl http://localhost:8080/api/v1/query \ -d '{"source":"*genesis*"}' ``` **Solutions**: 1. Verify correct network 2. Check genesis hash matches 3. Consider full resync 4. Verify no local modifications ## Deployment Issues ### Service Won't Start **Check Logs**: ```bash # Systemd journalctl -u convex-peer -n 100 # Docker docker logs convex-peer ``` **Common Causes**: - Port already in use - Incorrect configuration - Missing dependencies - Permission issues - Corrupted data **Solutions**: 1. Check port availability: `lsof -i :18888` 2. Validate configuration file 3. Verify file permissions 4. Check Java version: `java -version` 5. Review error messages in logs ### Crashes/Restarts **Check**: ```bash # Recent crashes journalctl -u convex-peer | grep -i crash # OOM killer dmesg | grep -i "out of memory" ``` **Solutions**: - Increase JVM heap - Check for memory leaks - Review error logs - Update to latest version - Check system resources ## Security Issues ### Unauthorised Access Attempts **Check Logs**: ```bash # Failed authentication journalctl -u convex-peer | grep -i "unauthorized" # Network connections netstat -an | grep 18888 ``` **Solutions**: - Review firewall rules - Implement IP whitelisting - Check for exposed services - Review [security guide](security) ### Compromised Keys **If peer keys compromised**: 1. Immediately stop peer 2. Use stake controller to withdraw stake 3. Generate new peer keys 4. Re-register with new keys 5. Investigate compromise source ## Data Issues ### State Corruption **Symptoms**: Errors reading state, crashes **Solutions**: 1. Stop peer 2. Backup current data 3. Attempt state repair 4. If repair fails, full resync 5. Restore from backup if available ### Disk Space Full **Check**: ```bash df -h /opt/convex ``` **Solutions**: - Clear old logs - Rotate logs more frequently - Increase disk space - Move data to larger volume ## Network Issues ### High Bandwidth Usage **Monitor**: ```bash # Real-time bandwidth iftop # Historical usage vnstat ``` **Solutions**: - Limit peer connections - Check for DDoS attack - Optimise sync settings - Review traffic patterns ### Firewall Blocking **Test Connectivity**: ```bash # From external telnet peer.example.com 18888 # Check firewall sudo iptables -L -n ``` **Solutions**: - Update firewall rules - Check security groups (cloud) - Verify NAT configuration - Test from multiple locations ## Monitoring and Diagnosis ### Health Checks ```bash #!/bin/bash # health-check.sh # Check if peer is responding response=$(curl -s -o /dev/null -w "%{http_code}" \ http://localhost:8080/api/v1/status) if [ "$response" != "200" ]; then echo "ERROR: Peer not responding" exit 1 fi # Check consensus advancing state=$(curl -s http://localhost:8080/api/v1/query \ -d '{"source":"*state*"}' | jq -r '.value') if [ -z "$state" ]; then echo "ERROR: Cannot query state" exit 1 fi echo "OK: Peer healthy, state: $state" ``` ### Log Analysis ```bash # Error summary journalctl -u convex-peer --since "1 hour ago" | \ grep ERROR | sort | uniq -c | sort -rn # Consensus issues journalctl -u convex-peer | grep -i consensus | tail -50 # Performance metrics journalctl -u convex-peer | grep -i "performance\|latency" ``` ## Getting Help ### Information to Collect When seeking help, provide: 1. Peer version: `java -jar convex.jar version` 2. Operating system and version 3. Hardware specifications 4. Configuration file (redact sensitive data) 5. Recent logs (last 100 lines) 6. Genesis hash being used 7. What you were trying to do 8. What actually happened ### Support Channels - **[Discord Community](https://discord.com/invite/xfYGq4CT7v)** - `#peer-operations` channel - **[GitHub Issues](https://github.com/Convex-Dev/convex/issues)** - Bug reports - **[Forum](/)** - Detailed discussions ## Next Steps - **[Security Guide](security)** - Prevent issues - **[Monitoring](/)** - Proactive monitoring - **[Manual Deployment](manual-deployment)** - Deployment guide - **[Docker Deployment](docker-deployment)** - Container deployment --- # Quick Start Deploy and call your first smart contract on a live Convex network — in under a minute, with no installation. ## Try it now: the Web Sandbox The fastest way in is the **[Web Sandbox](https://convex.world/sandbox)** — a live REPL connected to the public testnet. No install, no signup. **1. Evaluate an expression.** Queries are free and need no account: ```clojure (+ 1 2 3) ;; => 6 ``` **2. Get a funded account.** Create an account in the Sandbox and fund it from the testnet faucet — you now hold Convex Coins to pay for transactions. (See the [Faucet Guide](coins/faucet).) **3. Deploy and call a smart contract** — a one-line *actor*: ```clojure ;; Deploy an actor exposing one callable function (deploy '(defn ^:callable greet [name] (str "Hello, " name "!"))) ;; => #1234 ;; your new actor's address ;; Call it (call #1234 (greet "world")) ;; => "Hello, world!" ``` You have deployed and called a smart contract on a live decentralised network. :::note Network The Sandbox runs against the public **testnet** — free, for development. Production runs on **Protonet** (`peer.convex.live`). See the [Networks Guide](networks). ::: --- ## Go further Pick the path that fits what you're building. ### Try an SDK Build an app against the testnet in your language: **[ Java ](client-sdks/java/quickstart)** · **[ Python ](client-sdks/python/quickstart)** · **[ JavaScript / TypeScript ](client-sdks/typescript/quickstart)** Each quickstart is copy-paste runnable. A minimal example — create a funded account, transact, then query: **TypeScript / JavaScript** — `npm install @convex-world/convex-ts`: ```typescript import { Convex, KeyPair } from '@convex-world/convex-ts'; const convex = new Convex('https://mikera1337-convex-testnet.hf.space'); // Create a faucet-funded account and use it for signing (amount in coppers) const keyPair = KeyPair.generate(); const account = await convex.createAccount(keyPair, 100_000_000); convex.setAccount(account.address, keyPair); const result = await convex.transact('(def greeting "Hello Convex!")'); console.log(result.value); // "Hello Convex!" console.log((await convex.query('greeting')).value); // "Hello Convex!" ``` **Python** — `pip install convex-sdk`: ```python from convex_sdk import Convex, KeyPair convex = Convex('https://mikera1337-convex-testnet.hf.space') # Create a faucet-funded account (amount in coppers) key_pair = KeyPair() account = convex.create_account(key_pair) convex.request_funds(100_000_000, account) result = convex.transact('(def greeting "Hello Convex!")', account) print(result.value) # Hello Convex! print(convex.query('greeting', account).value) # Hello Convex! ``` **Java** — `world.convex:convex-java`: ```java import java.util.Map; import convex.java.ConvexJSON; import convex.core.cvm.Address; ConvexJSON convex = ConvexJSON.connect("https://mikera1337-convex-testnet.hf.space"); // Create a faucet-funded account (up to 10,000,000 copper) and use it Address address = convex.useNewAccount(10_000_000); Map result = convex.transact("(def greeting \"Hello Convex!\")"); System.out.println(result.get("value")); // Hello Convex! System.out.println(convex.query("greeting").get("value")); // Hello Convex! ``` ### Run your own peer Full control, offline development, or running on the network: download `convex.jar` (or **Convex Desktop**) and run your own peer. See **[Local Testnets](peer-operations/local-testnets)**. For an embedded peer in Java, see the **[Java Quickstart](client-sdks/java/quickstart)**. ### Write Convex Lisp Convex Lisp is the on-chain language for queries, transactions, and actors (smart contracts). Start with the **[Convex Lisp guide](/docs/tutorial/convex-lisp)** and **[Actor Development](actors)**. ### Operate a peer Run a node that participates in consensus on the network. See **[Peer Operations](peer-operations)**. --- ## Understanding what you did ### Key concepts **Account** — your identity on Convex. Holds Convex Coins (CVM) and is identified by an address (e.g. `#1234`). **Key pair** — Ed25519 public/private keys. The private key signs transactions; the public key backs the account. **Query** — reads network state. Free, needs no account, and changes nothing. **Transaction** — changes network state (e.g. `deploy`, `def`, `call`). Requires a funded account and costs *juice* (an execution fee). **Actor** — an autonomous account that holds code. Functions tagged `^:callable` can be invoked by anyone with `call`. This is a smart contract. ### What just happened? ```clojure (deploy '(defn ^:callable greet [name] (str "Hello, " name "!"))) ``` This Convex Lisp transaction: 1. **Created** a new actor account 2. **Installed** a `greet` function, exposed via `^:callable` 3. **Persisted** it across the network — permanently, cryptographically signed, and validated by consensus `(call #1234 (greet "world"))` then ran that function on-chain and returned its result. ## Next steps **Understand the network** - **[Networks Guide](networks)** — production, testnet, local - **[Faucet Guide](coins/faucet)** — getting test funds **Master your SDK** - **[Queries](client-sdks/java/queries)** — reading state - **[Transactions](client-sdks/java/transactions)** — changing state - **[Account Management](client-sdks/java/accounts)** — keys and accounts **Write smart contracts** - **[Convex Lisp](/docs/tutorial/convex-lisp)** — the on-chain language - **[Actor Development](actors)** — smart contracts - **[Recipes](recipes)** — practical examples ### Try these in the Sandbox ```clojure ;; Check an account balance (balance #13) ;; Do some math (+ 1 2 3 4 5) ;; Define and call a function (defn square [x] (* x x)) (square 7) ;; Deploy a smart contract (deploy '(defn ^:callable add [a b] (+ a b))) ``` ## Troubleshooting **`FUNDS` error** — your account needs Convex Coins. In the Sandbox, top up from the faucet; via an SDK, request from the [faucet](coins/faucet). **`SEQUENCE` error** — don't submit concurrent transactions from one account; wait for the previous one to confirm. **Can't connect** — check your network URL and internet connection. The testnet endpoint is `mikera1337-convex-testnet.hf.space`. **Query returns nothing** — check your syntax and that the variable or actor exists. ## Get help - **[Discord Community](https://discord.com/invite/xfYGq4CT7v)** — live help - **[GitHub Issues](https://github.com/Convex-Dev/convex/issues)** — bug reports - **[Documentation](/)** — complete guides **Common terms:** *CVM* = Convex Virtual Machine · *juice* = transaction execution cost · *copper* = smallest unit (1 CVM = 1,000,000,000 copper) · *peer* = a network node · *actor* = a smart contract. --- # Account Control Convex has a unique feature with **account controllers**. Account controllers can execute code in the context of accounts that they control allowing powerful capabilities such as: - Resetting an account's public key - Switching to a different controller - Updating code within an account There are two primary uses for account controllers: - Providing a way to backup and restore a user's account - Allowing an autonomous actor to be upgraded or modified by another user or actor ## Checking for a controller Account records are publicly visible as as part of the Convex global state, you you can examine an account at any time to see if a controller is set. Remember to run this in "Query" mode so that you don't pay any transaction fees: ```clojure (account #1567) ;; Result will look something like this: => {:sequence 0, :key 0x89b5142678bfef7a2245af5ae5b9ab1e10c282b375fa297c5aaeccc48ac97cac, :balance 5000000000, :allowance 0, :holdings nil, :controller #202, :environment nil, :metadata nil, :parent nil} ``` If the `:controller` field is `nil` there is no controller. Otherwise the controller is the value specified, in this case the account `#202`. ## Running code in an account you control If you are the controller for another account, you can do the following: ```clojure (eval-as #1567 '(def some-data [1 2 3])) ``` This will run the quoted code in the context of the controlled account (in this case `#1567`). A controller can *execute arbitrary code*, in this case we are defining some new data in the account with the symbol `some-data`. :::tip When using `eval-as` always remember to "quote" the code with `'` or `` ` `` to ensure that you send the literal code to the controlled account, rather than evaluating it as an expression in your own account! If unsure you can use `(assert *caller*)` in your code to ensure that the code is being run in the controlled account. ::: Assuming you ran the above in "Transact" mode, you can verify that this change has actually occurred in the controlled account: ```clojure #1567/some-data => [1 2 3] ``` ## Common control actions At this point is should be clear that you can effectively do anything in the controlled account! But here are some of the most common things you may wish to do: ### Reset a public key for a user account Suppose your friend lost their key pair. Luckily, they set you as a controller for their account so you can reset their key! Just get them to give you their new public key and using this do: ```clojure (eval-as #1567 '(set-key 0x89b5142678bfef7a2245af5ae5b9ab1e10c282b375fa297c5aaeccc48ac97cac)) ``` Et voila! They can now use their account again 😃. ### Upgrade an actor Let's say you want to upgrade an actor you already created to add a new callable function. As the account controller, this is no problem! ```clojure (eval-as #1567 '(defn ^:callable account-info [] (account *address*))) ``` This will run the `defn` in the controlled account, defining a new callable function which you can now call in the usual way: ```clojure (call #1567 (account-info)) ``` It is *strongly recommended* that you add tests to such upgrade operations, so that you can verify that you have not broken anything and that the actor functions as intended. If you run such tests in the same transaction that does the upgrade, then you can ensure that the upgrade gets rolled back in case of failure: ```clojure (do ;; Run the upgrade first (eval-as #1567 '(defn ^:callable account-info [] (account *address*))) ;; Test that the function works (if (call #1567 (account-info)) :OK (fail :ERROR "Upgrade failed, rolling back changes")) ) ``` ### Clear the controller You've been working on your awesome smart contract actor for months, and it has been battle tested and security audited with a growing fan base of loyal users. Originally, you set yourself as the controller so you could upgrade and make fixes to the actor, but now you are 100% convinced it is perfect and want to make it immutable: ```clojure (eval-as #1567 '(set-controller nil)) ``` :::warning Only do this if you are *really* sure you want the actor to be immutable forever. There's no way back if you remove your final access capability as a controller. ::: --- # A Coin Distributor This recipe is an example of developing an actor that handles a fairly simple use case of wanting to distribute an asset in a controller manner. Specifically: - The asset manages a potentially large balance of Convex Coins - One trusted party (the "allocator" specifies how much of the asset is available for distribution) - A second trusted party has the right to distribute the asset only up to the amount available If you are new to smart contract coding, this is a good introduction to some of the key ideas. ## Design This code represents some design decisions that a typical smart contract developer might make. We have chosen to implement this functionality **as an actor**. Distributing coins could be done with manual transfers between user account of course, but using an actor has several advantages: - We can define the exact rules by which coin distribution takes place. This could be extended in the future, e.g. only allowing distribution to recipient accounts that are properly registered (e.g. members of a DAO) - An actor can have its own balance of Convex coins, which makes it easier to control buckets of coins allocated for this specific usage. - The actor can perform automatic accounting (in this case, counting how much has been distributed and preventing this from exceeding a specified amount). This can eliminate mistakes that might be made with manual processes, and makes the distribution process observable and verifiable. We want to **divide responsibilities** between the allocator and the distributor. We can do this by creating a **trust monitor* for each, that is used to check that the caller of our actor functions is suitably authorised. ## Setting up the actor The following code is for the initial setup: ```clojure ;; Import the convex.trust library, so we can use CAD22 trust monitors (import convex.trust :as trust) ;; Allocator, has ability to allocate coins for distribution (def allocator #13) ;; Distributor, can distribute from allocated coins only (small bucket) (def distributor #13) ;; The amount of coins available for distribution (initially zero)) (def available-coins 0) ``` :::tip Replace `#13` with any address to define the initial allocator and distributor. `*caller*` makes sense if you are deploying the code with `(deploy ...)` in which case the account doing the deployment will be the initial allocator and distributor. ::: ## Making coins available We need a `:callable` function to let the allocator set the value of `available-coins` ```clojure ;; Set the amount of available coins. Only a trusted allocator can do this (defn ^:callable set-available [amount] (if (not (trust/trusted? allocator *caller* :set-available amount)) (fail :TRUST "Not authorised as allocator")) (if (< amount 0) (fail :ARGUMENT "Negative amount!")) (set! available-coins (int amount))) ``` Notes: - The function will fail with a `:CAST` error if the `amount` is not numerical, since it is being passed to numerical comparison operators. It's good practice to check what happens if an unexpected argument type is passed in, we want this to fail as early as possible. - The `(int amount)` cast in the last line. This is good practice to ensure that any value we `set!` will always be of the expected type (Integer in this case) even if the trusted caller makes a mistake. While setting available coins to a Double value could theoretically work, we want to minimise possible complications! - Following good CEI design (Checks-Effects-Interactions) we do the checks first before the effect (setting the `available-coins` variable). There are no external interactions, so in this case the code is safe from reentrancy attacks but it is *always* good to check. - We produce **meaningful error messages** as far as possible: see CAD11 (Errors) for recommended error codes. As Convex is an interactive system, we want to be as informative as possible. To use this function the trusted allocator will be able to execute commands such as: ```clojure ;; Set the available amount to 888 CVM (call distributor-actor (set-available 888000000000)) ``` ## Stocking the actor with coins The actor needs to have a balance of coins before any distribution can happen of course. Typically you would do this with a simple transfer: ```clojure ;; Transfer 9999 CVM to the actor (transfer distributor-actor 9999000000000) ``` There's a catch if the coin recipient is an actor however! Actors need to define a `receive-coin` function if they want to act as a recipient of coins. Fortunately it is pretty simple to make any acctor accept whatever coins are sent to it: ```clojure ;; Make an actor accept any offered coins ;; _ just ignores an argument, in this case we are ignoring 3 arguments: [sender amount optional-data] (defn ^:callable receive-coin [_ _ _] (accept *offer*)) ``` With the `receive-coin` function defined, the actor will automatically accept any coins :::note `receive-coin` is only required by an *actor* account that needs to receive convex coins: you can freely transfer to a user account without this. So as an alternative, you could develop this actor in a user account, transfer the coins to it, then use `(set-key nil)` to turn it into an actor. ::: ## Distributing coins We similarly need a `:callable` function to distribute coins. that can be sued by the trusted distributor: ```clojure ;; Distribute coins. Only a trusted distributor can do this (defn ^:callable distribute [receiver amount] (if (not (int? amount)) (fail :ARGUMENT "amount must be an integer")) (if (not (trust/trusted? distributor *caller* :distribute amount)) (fail :TRUST "Not authorised to distribute")) (if (> amount available-coins) (fail :FUNDS "Insufficient available coins")) ;; Every check passed, so: ;; 1. reduce available coins (Effect) ;; 2. Make an external transfer (interaction) (do (set! available-coins (- available-coins amount)) (transfer receiver amount))) ``` The user responsible for distribution will use this as follows: ```clojure ;; Send 3 CVM to the chosen recipient (call distributor-actor (distribute recipient 3000000000)) ``` This will work up until the amount of available coins is exhausted, at which point the allocated must make more coins available for distribution. :::warning It is **really important** in this case to use the **CEI ordering** (Checks-Effects-Interactions). If you do the `transfer` before reducing the amount of available coins, there is a *potential* for a reentrancy attack where multiple outward transfers are made before the available-coins variable is reduced. ::: ## Withdrawing coins Optionally, you may wish to make it possible for the allocator to withdraw coins from the actor. This is often a good idea: if plans change and you no longer want to distribute coins via this actor, it is good to be able to withdraw the assets contained within. Here's a simple example that lets the allocator (but nobody else!) withdraw coins: ``` (defn ^:callable withdraw [amount] (if (not (trust/trusted? allocator *caller* :withdraw amount)) (fail :TRUST "Not authorised to withdraw")) (transfer *caller* amount)) ``` ## Additional Extension Ideas If you are feeling adventurous, it is instructive to try out some ideas regarding how this actor might be extended. Some though starters: - Build an equivalent actor for a CAD29 fungible token rather than convex coins - Have multiple authorised distributors each with their own allocation of available coins to distribute --- # Creating Accounts Maybe you have an account on Convex already, but would like to give on to a friend so that they can also get into the Convex ecosystem early! This recipe is all how to create accounts for yourself or other users. ## The Basics An account has a numerical address, which is conventionally displayed with a `#` e.g. ``` #202 ``` Addresses are allocated sequentially as new accounts are created - so get in early if you want a rare 5 digit account number! User accounts have a public key, which is a 32-byte hex string that looks like this: ``` 0x89b5142678bfef7a2245af5ae5b9ab1e10c282b375fa297c5aaeccc48ac97cac ``` The public key ensures that only users with the correct private key can sign transactions for the account. To ensure you have the correct public key, Convex Desktop also displays identicons like the following which are a visual representation of the public key: ![An Identicon image for the public key 0x89b5142678bfef7a2245af5ae5b9ab1e10c282b375fa297c5aaeccc48ac97cac](identicon.png) ## Making a user account To make an account for a new user there are three steps: 1. Get the user to create and **give you their public key** - this is needed because you want them to be able to sign transactions for their own account right away! 2. Create the **account on Convex**, which will allocate a new address like `#1567` 3. Tell the user their **account address**, so they can start using it (theoretically, they can also search on the Convex global state to find which account(s) match their public key, but it's easier if you just tell them 🙂) You can do this with a single "Transact" command at the Convex terminal: ```clojure ;; Create a new account with the specified public key (create-account 0x89b5142678bfef7a2245af5ae5b9ab1e10c282b375fa297c5aaeccc48ac97cac) ``` This command will return the account address created: ``` #1567 ``` And the new account is good to go! :::warning A new account created this way has zero coins, and can *only* be used with the public key you entered. If the user made a mistake (e.g. lost their public key or somebody mistyped it), you just wasted your own coins creating a useless account. Read on for tips to mitigate this risk... ::: ## Transferring some coins A user account is useless unless it has some coins in it (which are needed to execute transactions, paying juice fees etc.). So typically, you will also want to transfer some coins to the new account. Again, this is a simple one line command: ```clojure ;; Send 10 CVM to the new account. Be careful with the number of zeros! (transfer #1567 10000000000) ``` Assuming you have >10 CVM in your own account, this transfer will succeed and put exactly 10 CVM in the destination user account. Account `#1567` is now ready to transact! :::tip Just remember that each convex coin is subdivided into 1 billion coppers, so a transfer needs an extra 9 zeros: 10 CVM = 10,000,000,000 coppers ::: ## Adding a controller for account recovery There's a risk above that the user loses their private key, and the account and any coins transferred to it will be forever lost.... 😢 That's a situation we want to avoid, so you can solve the problem by setting a **controller** that will be able to recover the account and funds if something goes wrong. The controller would typically be the creator of the new account, but you need to set this at account creation time (after this point, the account is outside your control). You can do this with the following code: ```clojure ;; Deploy a new account, setting the key and controller account ;; Note the ' : you want to quote the code so it runs in the newly deployed account, not your own! (deploy '(do (assert *caller*) ; This is a safety check to make sure the *caller* exists (set-key 0x89b5142678bfef7a2245af5ae5b9ab1e10c282b375fa297c5aaeccc48ac97cac) (set-controller *caller*))) ``` :::note The `(assert *caller*)`expression is optional, but works as an extra safety check. If you execute this on its own in a transaction, it will fail because `*caller*` is `nil` in a top level transaction. But it will pass the check in the deployed code, because `*caller*` would be the account address running the `deploy`. This is important, because you don't want to accidentally overwrite the key of your own account and set the controller to `nil`! ::: ## Putting it all together So we've seen that a good strategy for issuing new accounts is to create them with the correct public key, set a controller for account recovery, and send some CVM so the new user can get going. To automate this, you might want to create a helper function: ```clojure (defn distribute [key coins] (let [code `(do (assert *caller*) ; Safety check to make sure the *caller* exists (set-key ~key) ; Set key for the account, ~key unquotes the function argument (set-controller *caller*)) ; Set the controller for account recovery addr (deploy code)] (transfer addr (* coins 1000000000)) addr)) ; finally return the new address ``` With this defined, you can now distribute new accounts with 5 CVM as follows: ```clojure (distribute 0x89b5142678bfef7a2245af5ae5b9ab1e10c282b375fa297c5aaeccc48ac97cac 5) ``` ## Removing the controller Once a user has received their account and is successfully using it, they may wish to remove the controller (or set it to something else) so that the original account creator no longer has control. This can be done with the following command: ```clojure (set-controller nil) ``` A better option is to set it to the account address of someone you trust: ```clojure (set-controller #14) ;; Assuming #14 is a someone you trust to restore your account in an emergency ``` ## Checking it all works It's worth checking that a new account is created correctly. Remember to run this in "Query" mode so that you don't pay any transaction fees: ```clojure (account #1567) ;; Result will look something like this: => {:sequence 0, :key 0x89b5142678bfef7a2245af5ae5b9ab1e10c282b375fa297c5aaeccc48ac97cac, :balance 5000000000, :allowance 0, :holdings nil, :controller #202, :environment nil, :metadata nil, :parent nil} ``` Remember to verify the `:key`, `:controller` and `:balance` are what you expect, if they are, then you are all good! --- # Recipes Ready-to-cook examples you can run in the Sandbox or against the public testnet. Recipes use short code snippets but are meant to be accessible even if you are new to coding — don't be afraid to try them out. New to Convex? Start with the [Quick Start](../quickstart.md) first. Many recipes demonstrate our principle of **"One Line DeFi"** — the idea that most significant on-chain economic actions should be possible with a simple one-line command, with no advanced coding or new smart contracts required. ## Available recipes | Recipe | What you'll do | |--------|----------------| | [Create an Account](create-account/index.md) | Generate a key pair and create your own Convex account | | [Managing Coins](managing-coins/index.md) | Check balances and transfer Convex Coins between accounts | | [Coin Distributor](coin-distributor/index.md) | Deploy a simple actor that distributes coins to many recipients | | [Account Control](account-control/index.md) | Secure accounts with keys, controllers and multi-signature patterns | | [Issue a Fungible Token](issue-token/index.md) | Create and transfer your own CAD029 token in one line | Get cooking with Convex! --- # Issue a Fungible Token Creating your own fungible token on Convex is a one-liner — there is no new smart contract to write. Convex ships a standard [`convex.fungible`](/docs/cad/fungible) library that builds fully-featured [CAD029](/docs/cad/fungible) tokens. ## Create a token Deploy a token with an initial supply. The whole supply starts in your account: ```clojure (def TOKEN (deploy [(@convex.fungible/build-token {:supply 1000000})])) => #1234 ``` `TOKEN` is the address of your new token actor. That's it — you have issued a token. ## Check a balance ```clojure (@convex.fungible/balance TOKEN *address*) => 1000000 ``` ## Transfer tokens Use the generic `convex.asset` library to move any asset. The amount is paired with the token as `[TOKEN amount]`: ```clojure ;; send 1000 units to account #202 (@convex.asset/transfer #202 [TOKEN 1000]) ``` Now check both balances: ```clojure [(@convex.fungible/balance TOKEN *address*) (@convex.fungible/balance TOKEN #202)] => [999000 1000] ``` ## Total supply and decimals ```clojure (@convex.fungible/total-supply TOKEN) => 1000000 ``` `build-token` also accepts `:decimals` (and `:initial-holder`). For a token with 2 decimal places: ```clojure (def TOKEN (deploy [(@convex.fungible/build-token {:supply 1000000 :decimals 2})])) ``` ## See Also - [CAD029: Fungible Token Standard](/docs/cad/fungible) - [Managing Coins](../managing-coins/index.md) — the same ideas for the native Convex Coin - [Building an Actor](../../actors/building-an-actor.md) — write your own custom actor --- # Managing Convex Coins Convex coins are the native utility token on Convex. Convex coins can be used in several ways: - To pay execution fees for transactions executed on the network - To pay for other assets, such as memory allowances - As a transferable virtual currency for use with decentralised economic systems - To stake on peers managing the consensus of the network ## Checking balances Every account on Convex maintains a balance of Convex Coins. An account can use and transfer its balance freely, but no other account can do so unless given prior authorisation. Checking balance is best done in "Query" mode (to avoid transaction fees) and can be done simply as follows: ```clojure *balance* => 2966998521900 ``` :::note The balance is displayed in Convex coppers which are one billionth of a Convex Coin, so a balance of `2966998521900` should be interpreted 2966.998521900 CVM ::: If you want to check the balance of any other account, you can do so with the `balance` core function: ```clojure (balance #202) => 0 ``` If you request the balance for an account that doesn't yet exist, you will get `nil` as a response: ```clojure (balance #999999999) => nil ``` ## Transfers The `transfer` function can be used to send Convex Coins to any existing account. This requires the "Transact" mode since the transfer needs to change the global state: ```clojure (transfer #202 (* 10 1000000000)) => 10000000000 ``` The function returns the amount successfully transferred, which can be useful if you use a formula to calculate the transfer amount as above. Once a transfer is complete, you can confirm the coins have moved by checking the balances (as a query) for the source and destination account: ```clojure ;; mapv applies a function to a vector or arguments in turn (mapv balance [*address* #202]) => [2956998497790 10000000000] ``` ## Total Supply You can check the total issued supply of Convex Coins with the following query: ```clojure (coin-supply) => 1000000000060950 ``` Here the coin supply is the one million Convex Coins issued at genesis, plus a small amount of transaction fees incurred by governance accounts. Normal transaction fees by non-governance do not affect the coin supply (since they are re-circulated). Additional coin supply will be issued in the future, e.g. as ecosystem demand causes new coins to be purchased from the release curve. See the [Tokenomics CAD](/docs/cad/tokenomics) for more details. ## Burning coins If you have excess coins and want to get rid of some, you can burn them to the account `#0` ```clojure (transfer #0 5000000000) ``` Coins burnt in this way will be added to the peer reward pool, and eventually be re-circulated as rewards to peers for continuing to operate the network. So indirectly, you are contributing to the future development of the network 😀 --- # Convex CLI The Convex CLI (Command Line Interface) is a powerful tool for interacting with Convex from the command line on Windows, Linux and MacOS. It's a handy tool for system administrators and peer operators who need to script interactions with the Convex Network. ![Convex CLI Screenshot](convex-cli.png) ## Installation The Convex CLI requires: - A recent version of Java (21+, Java 25 recommended) - A copy of the `convex.jar` executable jar file Java is available from multiple providers, we recommend: - [Oracle JDK](https://www.oracle.com/java/technologies/downloads/) - [Eclipse Temurin JDK](https://adoptium.net/temurin/releases/) Releases of the `convex.jar` file are available from [GitHub Releases](https://github.com/Convex-Dev/convex/releases) — download `convex.jar` from the latest release. ## Running the CLI To run the CLI, you simply need to run the following: ``` java -jar convex.jar ``` It is often convenient to have a script, alias or batch file which executes the Java command to run the CLI. This means that you can simply type `convex ....` from any directory to access the CLI. The remainder of this document assumes you are using such a script. ### Windows For Windows PowerShell, the following should work: ``` # Windows powershell - convex.ps1 function convex { java -jar C:\path\to\convex.jar $args } ``` Put this in your PowerShell profile (typically `Profile.ps1` in `$home/documents/PowerShell`) and the `convex` command should be available in all future PowerShell sessions. ### Linux Run the following bash command (or add to your `~/.bashrc`) as follows: ``` alias convex="java -jar ~/convex.jar" ``` This will create an alias to the `convex.jar` so you can type `convex ....` in any directory to access the full set of CLI features. --- # Convex Desktop Convex Desktop is a GUI tool for interacting with Convex. Designed for developers and power users, it puts all the capabilities of Convex at your fingertips. ![Convex Desktop Screenshot](convex-desktop.png) ## Installation Convex Desktop requires: - A recent version of Java (21+, Java 25 recommended) - A GUI based operating system (e.g. Windows, Linux or MacOS) - A copy of the `convex.jar` executable jar file Java is available from multiple providers, we recommend: - [Oracle JDK](https://www.oracle.com/java/technologies/downloads/) - [Eclipse Temurin JDK](https://adoptium.net/temurin/releases/) Releases of the `convex.jar` file are available from [GitHub Releases](https://github.com/Convex-Dev/convex/releases) — download `convex.jar` from the latest release. ## Running Convex Desktop If Java is configured to run by default for `.jar` files in your OS, you should be able to run Convex Desktop simply by double-clicking on the `convex.jar` file which will run Convex Desktop as an executable. On some systems, you may need to grant permissions to run a downloaded executable file. Alternatively, to run Convex Desktop, you can run the following command from the command prompt, script or GUI shortcut: ``` java -jar convex.jar desktop ``` ## Introduction for non-crypto Developers ### Convex Desktop - Create Convex accounts and Peers via a graphical user interface (GUI) for operating networks - Allows developers to generate test networks and secure crypto keys prior to production use - Prepare and test transactions before committing to execution or paying fees - Manage a wallet for storing keys and account balances - In production, earn fees for staking and operating Peers - Use for global scale payments and gaming ### Getting Started - Download the latest convex.jar from Convex Snapshots - Learn Convex Lisp for more advanced functions image ## Step 1a: Hacker Tools KeyGen image - Click Generate – 12 words appear in seed Phrase - Enter Passphrase (lower case) - Record both somewhere safe. DON’T LOSE - Key Derivation defaults to `m/44/864/0/0/0` (the SLIP-10 standard for Convex). You can change this, but if you do, make sure that you SAFELY record this too. - Copy Private Ed25519 seed somewhere safe. DON’T SHARE. This is also known as Private Key - Copy Ed25519 Public Key. Can be shared - Check out your identicon. It will be useful later to recognise your key. - Normalise mnemonic sets lower case - Click Add to keyring ## Step 1b: Hacker Tools Load Keystore image - Icon for swift recognition - Public key to share - Source = location of keystore - Lock for extra security, keep separate from other keys and passphrases. --- # Convex MCP Convex peers ship with built-in **Model Context Protocol (MCP)** servers, enabling AI agents and MCP-compatible tools to interact directly with the Convex network. Any peer with MCP enabled becomes a gateway for querying state, executing transactions, managing digital assets, and receiving real-time state change notifications — no custom integrations or centralised intermediaries required. ## Quick start Connect any MCP client to the public testnet peer: ``` https://mikera1337-convex-testnet.hf.space/mcp ``` That's it. The peer exposes over 30 built-in tools and guided prompts that your AI agent or development tool can discover automatically through MCP's standard `tools/list` and `prompts/list` methods. The testnet includes a free faucet, so agents can create and fund accounts to experiment with. :::note Protonet The flagship Protonet peer (`peer.convex.live`) does not currently expose a public MCP endpoint — use the testnet above for MCP access, or run your own peer (MCP is enabled by default in current releases). ::: ## Key capabilities - **Direct network access** — sub-millisecond state queries, native Convex Lisp execution, no indexers or RPC layers - **Flexible transaction signing** — direct signing, prepare/sign/submit with external approval, or server-side signing via the built-in signing service - **Real-time state watching** — register watches on any path in the global state tree and receive SSE notifications when values change - **Universal asset model** — fungible tokens, NFTs, and arbitrary digital assets through a unified [CAD019](/docs/cad/assets) interface - **Built-in cryptography** — Ed25519 key generation, signing, verification, and hashing - **Guided prompts** — workflow prompts that teach LLMs Convex domain knowledge and guide them through common tasks like account exploration, contract deployment, and fund transfers - **Signing service** — optional server-side key management where private keys are stored encrypted and never leave the server, with elevated operations requiring browser confirmation - **Zero configuration** — MCP support is enabled by default on every Convex peer ## Running your own peer For production use, run your own Convex peer for better performance, security, and privacy. MCP is enabled by default — the endpoint is available at `/mcp` with no additional configuration. See the [Peer Operations guide](/docs/tutorial/peer-operations) for setup instructions. ## Learn more - **[AI Agents User Guide](/docs/tutorial/agents)** — step-by-step guides for MCP integration, agent account management, and agentic economics - **[CAD041: MCP Specification](/docs/cad/mcp)** — technical specification for the Convex MCP endpoint - **[Model Context Protocol](https://modelcontextprotocol.io/)** — the MCP standard --- # Peer Container Convex is packaged in the Peer Container for easy deployment via Docker-based systems ![Docker Peer Container](peer-container.png) ## Installation To run the peer container locally, you will need to install [Docker](https://www.docker.com/). You can use either the Docker CLI or Docker Desktop. The Peer Container is packaged as a docker container available on DockerHub - [Peer Container on DockerHub](https://hub.docker.com/repository/docker/convexlive/convex) The latest version is general available in docker as `convexlive/convex:latest` ## Running a Peer Container Running the Peer Container is a one-liner: ```bash docker run --name my-peer -d convexlive/convex:latest -p 8080:8080 -p 18888:18888 ``` This will run a default test peer, with ports mapped as follows: - `8080` mapped for REST API access - `18888` for the Convex peer protocol (other peers and binary clients) However, to operate a peer effectively some configuration is usually required. ### SSL Certificates If you want your peer to use HTTPS (highly recommended) you probably want to provide SSL certificates. ### Peer Keys Each peer require a "peer key", which is an Ed25119 key pair. The public key is used to identify the peer on the network, and the private key is used by the peer to sign its interactions with the lattice / CPoS consensus. ### Controller keys We recommend that you DO NOT add peer controller keys to a Convex peer container. Reasons for this: - Peer controller keys have control over significant economic assets, in particular the peer's stake on the Convex Network - If the system running the docker container is compromised, the controller key may be stolen Instead, is is best to operate the Peer Container with a peer key alone, and use a separate mechanism (e.g. a hardware wallet or air-gapped laptop) for signing transactions that require the controller key. --- # Convex SDK Official client libraries for the Convex decentralised lattice network. ## Overview The Convex SDK lets developers use the full capabilities of the Convex decentralised lattice from any programming language or platform. Whether you're building web applications, backend services, mobile apps, or embedded systems, Convex SDKs provide idiomatic, type-safe APIs that feel natural in your language ecosystem. By bridging multiple language ecosystems with the Convex Virtual Machine (CVM), these SDKs expose the platform's core capabilities: - **Decentralised State Management** - Access and modify global state with atomic transactions - **Lattice Technology** - Use CRDT-based data structures for conflict-free collaboration - **Convergent Proof of Stake** - Sub-second transaction finality via leaderless consensus - **Convex Lisp Execution** - Execute smart contracts and queries from any environment - **Cross-Platform Interoperability** - Applications in different languages share the same network state ## Available SDKs ### TypeScript / JavaScript **Status**: Production Ready Build web applications, Node.js services, and React dApps with full type safety and modern JavaScript tooling. ```bash npm install @convex-world/convex-ts ``` ```typescript import { Convex } from '@convex-world/convex-ts'; const convex = new Convex('https://peer.convex.live'); const result = await convex.query('(balance #13)'); console.log('Balance:', result.value); ``` **Key Features:** - Read-only queries without authentication - Full account management with Ed25519 keys - Type-safe transaction submission - Pluggable signers (hardware wallets, browser extensions) - Environment agnostic (Node.js, browsers, Deno, Bun) - React integration available **→ [TypeScript SDK Documentation](/docs/tutorial/client-sdks/typescript)** --- ### Java **Status**: Production Ready Native JVM integration for high-performance backend services and enterprise applications. ```xml world.convex convex-java 0.8.8 ``` ```java import convex.api.Convex; import convex.core.Result; import convex.core.lang.Reader; Convex convex = Convex.connect("https://peer.convex.live"); Result result = convex.query(Reader.read("(balance #9)")).get(); System.out.println("Balance: " + result.getValue()); ``` **Key Features:** - Native JVM performance with zero overhead - Async operations with CompletableFuture - Ed25519 key management (AKeyPair) - Multiple client types (HTTP, direct peer) - Automatic transaction signing - Self-contained, minimal dependencies **→ [Java SDK Documentation](/docs/tutorial/client-sdks/java)** --- ### Python **Status**: Production Ready Pythonic API for scripting, automation, and data science applications. ```bash pip install convex-sdk ``` ```python from convex_sdk import Convex, KeyPair convex = Convex('https://mikera1337-convex-testnet.hf.space') key_pair = KeyPair() account = convex.create_account(key_pair) convex.request_funds(100_000_000, account) balance = convex.get_balance(account) print(f'Balance: {balance / 1_000_000_000} CVM') ``` **Key Features:** - Synchronous I/O for simple scripting - Account object pattern with Ed25519 keys - Built-in transfer and balance methods - Automatic sequence retry for concurrency - CNS (Convex Name Service) integration - Pythonic `snake_case` naming **→ [Python SDK Documentation](/docs/tutorial/client-sdks/python)** --- ## Why Multiple SDKs? Different applications demand different languages and ecosystems: - **Web Applications** need TypeScript/JavaScript for browser compatibility - **Enterprise Systems** require Java for Spring/Jakarta EE integration - **Data Science** relies on Python's rich ecosystem (NumPy, TensorFlow, scikit-learn) - **Mobile Applications** use platform-specific languages (Swift, Kotlin) Convex SDKs let developers use decentralised lattice technology in their **native language ecosystem**, with idiomatic APIs that feel familiar rather than foreign. ## Universal Capabilities All Convex SDKs provide: - **Network Queries** - Read network state without fees or accounts - **Transaction Submission** - Execute atomic state transitions with cryptographic signatures - **Account Management** - Self-sovereign control with Ed25519 key pairs - **Smart Contract Interaction** - Call deployed actors and execute Convex Lisp - **Event Monitoring** - Subscribe to network events and state changes - **Secure Key Storage** - Encrypted keystores for credential management ## Resources - **[SDK Documentation](/docs/tutorial/client-sdks/)** - Guides for all languages - **[Discord Community](https://discord.com/invite/xfYGq4CT7v)** - Get help and share ideas - **[GitHub](https://github.com/Convex-Dev)** - Source code and examples - **[npm Package](https://www.npmjs.com/package/@convex-world/convex-ts)** - TypeScript SDK --- **→ [Get Started with an SDK](/docs/tutorial/client-sdks/)** --- # Convex Explorer The Convex Explorer is a peer-hosted interface for viewing live data from the Convex network. Instead of relying on a single central service, every Convex peer can serve the explorer, giving the community a resilient way to inspect network activity and smart-contract state. ## Quick Access - Primary community instance: [https://peer.convex.live](https://peer.convex.live/explorer) - Any Convex peer with the explorer module enabled can be accessed at `https:///explorer` ## Key Capabilities - Real-time account balances, transaction history, and on-chain state - Built-in support for inspecting Convex Lisp contracts and evaluating forms - Node health dashboards, including peer status, stake, and consensus metrics - Lightweight HTTP interface designed to run directly from a peer without extra infrastructure ## How to Use It 1. Open the explorer endpoint for the peer you want to inspect (for example, `https://peer.convex.live`). 2. Browse accounts, transactions, or smart contracts using the sidebar navigation. 3. Use the interactive Convex Lisp console to evaluate queries directly against the peer. > Tip: Because each explorer instance is peer-backed, the data reflects exactly what that peer sees on the network. If you need to confirm consensus, compare results across multiple peers. ## Running Your Own Explorer If you operate a Convex peer, the explorer will run by default on port 8080. The explorer will be available at the `/explorer` path for that peer. We recommend securing your explorer endpoint with HTTPS and a reverse proxy (for example, [Caddy](https://caddyserver.com/)) to provide TLS termination and optionally restrict access. ## Troubleshooting - **Explorer not loading**: Confirm that the peer is reachable and the explorer service is enabled. - **Stale data**: Refresh the page or check another peer to rule out local network partitioning. - **Permission issues**: Some peer operators may restrict the explorer to trusted networks; contact the operator if access is denied. --- # Convex Products Tools and applications for working with the Convex network — whether you are building applications, operating infrastructure, or exploring live network data. ## For Developers - **[Convex SDKs](/docs/products/convex-ts)** — official client libraries for TypeScript / JavaScript, Java and Python - **[Convex REPL](/docs/tools/convex-repl)** — interactive Convex Lisp REPL, right in your browser - **[Convex MCP](/docs/products/convex-mcp)** — Model Context Protocol servers built into every peer, so AI agents can query state and transact directly ## For Users and Power Users - **[Convex Desktop](/docs/products/convex-desktop)** — GUI for wallets, accounts and smart contract development - **[Convex Explorer](/docs/products/explorer)** — peer-hosted interface for viewing live network data - **[TokEngine](/docs/products/tokengine)** — cross-chain token interoperability bridge ## For Operators - **[Convex CLI](/docs/products/convex-cli)** — command-line tool for scripting and peer administration - **[Peer Container](/docs/products/convex-peer)** — Docker packaging for running your own peer --- # TokEngine TokEngine is an open-source cross-chain token interoperability bridge that uses the efficiency and scalability of Convex's lattice technology to enable high-speed digital asset transfers across diverse blockchain networks. Designed for flexibility, TokEngine lets anyone operate a server and facilitate cross-chain token exchanges for their chosen digital assets, fostering a decentralised and inclusive financial ecosystem. Released under the Apache 2.0 License, TokEngine is freely accessible for developers and organisations worldwide. ![TokEngine Logo](image.png) ## About the project TokEngine is proudly supported by the [Convex Foundation](https://convex.world) and funded by the EU's [Next Generation Internet](https://ngi.eu/) initiative, championing sustainable and efficient cross-chain interoperability. Developed by Convex Ecosystem Services Limited in collaboration with [Werenode](https://werenode.com/), TokEngine is built to drive innovation in decentralised finance and blockchain connectivity. TokEngine is an open source product supported by the Convex Foundation ## Key Functionality - **Deposit and Virtual Balances**: Accept deposits and convert them into "Virtual Balances" for cross-chain use. - **Cross-Chain Payouts**: Redeem "Virtual Balances" as equivalent tokens on any supported distributed ledger technology (DLT). - **End-to-End Swaps**: Execute complete cross-chain token swaps. - **Wrapped Tokens**: Support for wrapped token functionality. - **Audit Logging**: Optional logging to a Kafka queue for transparency and tracking. - **CAIP Compatibility**: Aligns with [Chain Agnostic Improvement Protocols (CAIPs)](https://chainagnostic.org/) for standardised interoperability. ## Supported DLTs TokEngine's plugin adapter system enables integration with various DLT networks, so operators choose which networks and assets to support. ### Convex With an embedded Convex Peer, TokEngine ensures ultra-low-latency communication with Convex networks, achieving end-to-end transactions in under 300ms. It can connect to existing Convex networks or operate in standalone mode. TokEngine supports the following on Convex based networks e.g. Protonet: - Convex coins (native token) - CAD029 tokens (i.e. compatible with the convex.fungible library) ### EVM TokEngine integrates with EVM-compatible chains using the [Web3j](https://www.web3labs.com/web3j-sdk) library, with key storage in standard EVM wallet files. On any EVM-compatible chain, TokEngine supports - ETH (native token) - ERC20 token - ERC777 tokens (coming soon) ### Tezos (Coming Soon) On Tezos TokEngine supports: - tez (native token) - FA1.2 - FA2 - FA2.1 ## Community and discussion Join the TokEngine community to contribute, collaborate, or stay updated: - Explore the code and get involved with OSS development at the [GitHub repository](https://github.com/Convex-Dev/tokengine). Open source developers wishing to get involved with TokEngine should visit the - Engage in discussions on the [Convex Community Discord](https://discord.com/invite/xfYGq4CT7v). --- import ConvexREPL from '@site/src/components/ConvexREPL.tsx'; # Convex REPL Enter and execute commands live on the Convex network. --- # Convex Architecture Documents ## Overview Convex CADs (Convex Architecture Documents) are open, community-driven specifications that define the core functionality and interoperability of Convex and related tools. They serve as a blueprint for developers to create compatible and interoperable applications within the Convex ecosystem. Anyone can propose and develop a CAD, and all CADs are openly available on GitHub to encourage community contributions and transparency. Similar to the Internet RFC process, CADs formalize standards that can be adopted and shared throughout the Convex ecosystem, promoting compatibility and interoperability. ## Governance The Convex Foundation oversees the governance of CADs, making decisions on updates and official acceptance to maintain the integrity and compatibility of the Convex ecosystem. While the Foundation plays a crucial role, the community is encouraged to contribute and propose new CADs. This collaborative approach ensures that the standards evolve in line with the needs and innovations of the Convex community. --- # CAD000: Design Principles We are building a platform for global decentralised computation and data, based on lattice technology and as a shared public utility network. As such, we are developing a set of open standards based on sound principles and values that will serve the long term vision. This document details general design and engineering principles deployed in the implementation and documentation of Convex. All CADs should refer to and align with these principles. ## Technical Principles ### Values are Immutable We adopt immutability for all values in Convex. Immutability is important for several reasons: - Enables hash codes to be used for value identity (value IDs) - essential for content-addressable storage - Enables structural sharing in persistent data structures, especially when shared on the lattice - Easier to reason about immutable values, especially with pure functions - Better suited for concurrency Mutability may occur as an implementation detail (e.g. a mutable cached value for performance reasons), however such mutability should not be externally visible (e.g. should not affect the encoding of values or CVM behaviour). A useful presentation on the topic: https://www.infoq.com/presentations/Value-Values/ ### Bounded Resources We are building a system for distributed computation and data in the context of a global internet where many parties with access to the Convex network may be untrusted. It is therefore necessary to place a bound on the size of resources used. This is essential to ensure that the CVM does not ever attempt to process data of unbounded size, which could allow DoS attacks by adversaries constructing arbitrarily sized input. Where input to Convex may be effectively unbounded (e.g. the size of data structures such as Vectors), implementations MUST NOT attempt O(n) or greater operations on such structures unless these operations are protected by resource constraints (e.g. accounting for juice costs, memory allowances). ### Conflict-free replication We have designed lattice technology so that it can operate as a conflict-free replicated data type: this enables scalable systems without the need for locking and synchronisation between distributed processes. Arguably this is the *only* practical way to achieve decentralised consensus at the scales we envisage. This places some restrictions on technical implementation: - Message handling should be idempotent, i.e. repeated receipt of identical messages should have no effect (and consume minimal resources) - Data structures must be designed for efficient CRDT merge operations - Systems must be designed to operate on the latest concurrent immutable state, with the knowledge that this state may change as it converges towards eventual consistency - Situations which are ordering-dependent must use CPoS to reach consensus on ordering (this mostly applies to the ordering of Convex transactions at present, but may apply to other lattice merge operations in future) ## General Design Philosophy ### Security First Convex supports high value economic transactions. As such, security issues should be automatically regarded as the highest priority. We MUST NOT release core software with known severe security defects that might place digital assets at risk. We SHOULD use existing, proven algorithms and cryptography wherever practical: there is no need to "reinvent the wheel" in crypto. ### Favour Simplicity Especially in API design, there is a tendency to want to add new features for user convenience, e.g. additional optional arguments for core functions. In such cases we SHOULD strongly resist the temptation to add additional complexity, and prefer the simplest possible implementation, especially within core Convex functionality. It is more important that core functionality is clean, simple and maintainable than superficially easy to use. Users have a powerful language with macro capabilities if they wish to implement more convenient programmatic interfaces appropriate for their own use case or design tastes. An excellent talk by Rich Hickey on this topic: https://www.infoq.com/presentations/Simple-Made-Easy-QCon-London-2012/ ### Design for Composition Convex features should be explicitly designed to compose to enable higher-order functionality to be built from simple, regular, well-behaved components. In general, pure functions and immutable data compose well. ### Think Algebraically Wherever possible, consideration should be given of the algebraic properties of Functions and Values. An algebraic approach lends itself well to cleaner, simpler definitions and well-behaved functionality. As far as possible, we want Convex functionality to behave as pure mathematical constructs. For example, algebraic thinking may be useful when thinking about the "zero" case of function arities. This might be considered by treating the varargs parameter as a vector where consideration should be given to the case where this vector is empty e.g. - `(apply + []) -> 0` - `(apply conj coll []) => coll` - `(apply * []) => 1` - `(apply assoc m []) => m` ### Always be additive We can add functionality, we can't remove it (at least in released versions of Convex). We MUST always continue to support existing released functionality that users are depending on in production environments. This principle also applies to error conditions. We can permissibly turn a failure into a success (e.g. defining a function that was previously undeclared) but cannot turn a success into a failure (e.g. removing a core function). ### Favour Explicit over Implicit We prefer to avoid implicit behaviour, and generally require operations to be explicitly requested. It is better to throw an Error rather than having implicit behaviour (which creates implementation complexity and may not be what the user intended). This principle manifests, for example, in the idea that most functions should avoid performing implicit casts between different Types. If a user wants a value to be cast to a different type, they should specify it explicitly, e.g. `(+ 1 (long 0x1234))` works but `(+ 1 0x1234)` should not. ### Avoid Scope Creep Convex is designed to facilitate on-chain transactions and smart contracts between multiple participants, providing foundational capabilities for open economic systems: the Internet of Value. Many types of software are a poor fit for a publicly accessible and globally validated state machine such as the Convex CVM, e.g. text processing, AI models or data analytics. We SHOULD NOT add features and complexity to support use cases in the CVM that do not belong there in the first case. Such features MAY make sense in P2P off-chain usage, e.g. the Data Lattice. We encourage experimentation in off-chain environments before consideration for CVM inclusion. ### Apply Judgement Principles are rarely absolute. There are always trade-offs in engineering decisions that must be considered. Discussion is encouraged to ensure relevant aspects are considered from a number of perspectives. ## CAD Style Conventions Literal code (e.g. Convex Lisp forms) should be quoted in a fixed width font `(like this)` Type names like the Convex Vector or the Java type Object should be capitalised. They SHOULD NOT be quoted as code unless they are intended as a code example. Where possible, follow RFC style MUST, SHOULD, MAY, SHOULD NOT etc. in formal specifications. --- # CAD001: Convex Architecture ## Overview The Convex platform comprises two complementary decentralised systems built on the same underlying data infrastructure: - **Convex** — a decentralised global state machine providing publicly verifiable on-chain data, deterministic execution, and economic coordination via the Convex Virtual Machine (CVM) - **Lattice** — a broader decentralised CRDT network providing unlimited scalability for off-chain data, content distribution, AI models, and real-time collaboration Both systems use **CAD3 encoded data** ([CAD003](../003_encoding/index.md)) as their universal data representation, and both employ **CRDT merge semantics** to guarantee convergence without coordination. This shared foundation means data flows naturally between on-chain and off-chain contexts using the same types, encoding, and cryptographic primitives. ### Convex (On-Chain) Convex operates a decentralised **network of peers**, which verify and execute transactions submitted by users on the **Convex Virtual Machine (CVM)**. The CVM manages a **Global State** which is publicly verified and visible at all times. Transactions are ordered via the **Convergent Proof of Stake** (CPoS) consensus algorithm. This algorithm efficiently ensures that transactions are processed in the correct order, and that clients can rely on the results of their transaction being accurately reflected in the Global State of the CVM. On-chain data provides the **root of trust**: asset ownership, identity, smart contracts, and economic value exchange that require global consensus and public verifiability. ### Lattice (Off-Chain) The Lattice is a global, massively scalable data distribution layer for data that does not need to pass through CVM consensus. It uses lattice merge (CRDT semantics) to combine data from multiple sources without coordination, enabling: - **Unlimited scalability** — no consensus bottleneck; data propagates peer-to-peer - **Offline-first operation** — nodes merge when connectivity is restored - **Signed ownership** — data is cryptographically signed per-owner ([CAD038](../038_lattice_auth/index.md)) - **Rich data types** — replicated filesystems ([CAD028](../028_dlfs/index.md)), key-value databases ([CAD037](../037_kv_database/index.md)), and custom lattice structures ### Shared Infrastructure The on-chain and off-chain layers share: | Layer | Shared Component | Specification | |-------|-----------------|---------------| | Data representation | Immutable CAD3 values (ACell hierarchy) | [CAD002](../002_values/index.md), [CAD003](../003_encoding/index.md) | | Cryptography | Ed25519 signatures, AccountKey identities | [CAD002](../002_values/index.md) | | Merge semantics | Lattice merge (commutative, associative, idempotent) | [CAD024](../024_data_lattice/index.md) | | Storage | Content-addressed Etch database | [CAD003](../003_encoding/index.md) | | Networking | Peer-to-peer binary protocol | [CAD015](../015_peercomms/index.md) | This means the next generation of decentralised applications can achieve the right combination of massive scalability backed up by publicly verifiable on-chain roots of trust and economic value exchange. ## Architecture Requirements ### 1. Peer Network The network of Peers collectively forms a decentralised substrate for the execution of queries and transactions on behalf of users. The network MUST be configured as a set of Peers with the ability to communicate with other peers over a network, preferably the public Internet. The network MAY suffer from temporary disconnection or interruption. Peers MUST attempt to make progress (subject to the rules of the CPoS consensus algorithm), i.e. the network should be resilient to temporary partitions that isolate a set of Peers. Peers MUST accept Belief update messages from at least one other Peer. Failure to do so will result in that Peer being unable to observe consensus. Peers MUST transmit their own Belief updates to at least one other Peer. Failure to do so will result in the Peer being unable to contribute its own transactions to network consensus. Peers SHOULD send and receive Beliefs from multiple randomly selected other peers, sufficient to ensure that the peer does not become isolated. Failure to do so may result in the peer being temporarily excluded or ejected from the main network if connectivity is insufficient. The Network SHOULD be configured in such a way that the sharing of Belief updates will ultimately propagate information from any Peer to any other Peer, i.e. the network graph transmission should be strongly connected. Failure to respect this property may result in Peers being unable to participate from consensus, in a manner similar to suffering from a network partition. ### 2. Clients Clients are defined as any participating system that transacts or queries the Convex network. Clients are typically software applications run by users with self-sovereign control of their cryptographic keys, but may also be centralised applications run on behalf of users by a third party (e.g. custodial exchanges). Clients MUST connect to an active Peer in the Peer Network, either locally or to a remote Peer. Clients SHOULD ensure that the trust the Peer that they use to faithfully carry out queries or transactions on their behalf. Clients MAY validate important or high value transactions with multiple peers (in particular, that the transaction was successfully submitted and executed in consensus) Clients MAY connect to multiple Peers. This may be valuable if the Client wishes to verify information from multiple sources, e.g. to confirm the consensus state of the network. ### 3. State The State is a data structure that represents all information managed by the CVM. It may also be referred to as the Global State, as it is globally shared and verified by all Peers. Peers MUST maintain a copy of the current consensus State. Peers MAY keep the State in durable storage, and retain only partial elements of the State in working memory. This behaviour will typically be required if the Peer is memory-constrained and the State is large. Peers SHOULD retain past instances of the State. This may be valuable for analytical, query or search purposes. --- # CAD002: CVM Values Convex uses a special representation of information values within the CVM, across the data lattice and to support the Convergent Proof Of Stake Consensus. Values in Convex are special for a number of reasons: - They are **pure immutable values** well suited for use in **functional programming** - They are designed for **efficient encoding** and network transmission - They form **Merkle trees** supporting cryptographic verification - They implement **orthogonal persistence**: automatically migrate between stem main memory and disk as required - They support **structural sharing**, making operations such as taking snapshots or the entire CVM state possible in O(1) time It is fair to say that Convex wouldn't be possible without this powerful and flexible implementation of data values. This document describes the values used and key design requirements that specify the available data types in the CVM and Convex Peers / Clients. ## Motivation Fundamentally, all computing is dependent on the representation of information values (alongside the specification of programs that work with these values). This is particularly important in a decentralised context, where different systems must share and come to consensus over such values. We therefore need a consistent and meaningful definition of all such values. We want data types for two primary purposes: - Data on the Convex consensus network as part of the CVM - Off-chain data in broader decentralised applications (the "Data Lattice") Functional programming languages such as Clojure, Haskell and Scala have shown the effectiveness of **immutable** values expressed as persistent data structures. This is perhaps best illustrated with a simple example: ```clojure ;; Define a Vector (def a [1 2 3 4]) => [1 2 3 4] ;; Define a new Vector appending a new value (def b (conj a 5)) => [1 2 3 4 5] ;; Original Vector is unchanged a => [1 2 3 4] ``` These immutable values have some important advantages for a decentralised network: - Reasoning about code behaviour is easier with immutable values. If a value is immutable, you don't have to worry about concurrent updates. This has important security benefits too: you don't need to remember to perform defensive copies when passing references to other code (e.g. a potentially untrusted smart contract). - Immutable values **compose** in a simple way that breaks down in the presence of mutable values. A collection of mutable values is conceptually mutable even if the collection itself is immutable, but if all values are immutable then you are guaranteed that any composed data structure is also immutable. - When internally implemented as trees, persistent data structures have favourable computational complexity bounds, e.g. `O(1)` operations for common copy, update and append operations. Since decentralised systems may frequently operate on large data structures, these are extremely important for performance. A traditional approach using mutable data structures might require `O(n)` operations and memory to perform a copy, for example creating a full immutable snapshot of *any* data structure in Convex is always `O(1)`. - Cryptographic hashes of immutable values are possible and safe: if values cannot change, neither can their hash. This makes it practical to use immutable values (but not mutable ones!) in data structures such as Merkle DAGs. Hashes can be used to efficiently determine if two entire data structures are identical and/or identify any differences. ## General Principles ### Sound Definitions Values (and their corresponding types) MUST have clear logical definitions, ideally corresponding to logical and mathematical constructs. This is an important principle, since types with simple logical definitions are easier to implement correctly, less likely to require handling of special cases and less likely to require future changes. ### Useful Types The available set of CVM Values SHOULD represent a generally useful set of data types with the following criteria: - They should support common programming paradigms and idioms. As far a possible, prefer types that will be familiar to programmers - They should facilitate the types of code likely to be run on-chain, particularly smart contracts dealing with digital assets - They should allow for efficient implementation and execution performance - Keep it simple: we should not have too many types. additional types will not be included without compelling use cases that cannot be adequately supported by existing types ### Immutability All CVM values MUST be **immutable**. This restriction is necessary from the perspective of maintaining integrity of the decentralised state, where values can be identified by a hash of their encoding. The property of immutability is also helpful from a performance perspective, since it means that CVM values can be safely cached and de-duplicated in memory or storage. ### Structural Sharing All CVM values which are data structure MUST support structural sharing of sub components if they have greater than `O(1)` size. This ensures that we can offer better than `O(n)` performance bounds for reads and updates of immutable structures (i.e. avoiding copy-on-write costs). Typically these costs should be either `O(1)` or `O(log n)` for most operations. This also ensures that Peers can safely store multiple versions of large data structures with minor changes while only incurring storage requirements that scale with the size of the changes. This is particularly important for Beliefs and their constituent Block Orderings, which may grow very large over time. ### Canonical Encoding All CVM values MUST have a unique canonical **encoding** as a fixed length sequence of bytes. See [Encoding CAD](/cad/003_encoding/index.md) for full specification. CVM values are **defined to be equal** if and only if their Encoding is identical. ### Value ID Each unique CVM value is defined to have a **Value ID** that is equal to the SHA3-256 hash of the value's encoding. The Value ID is extremely important, because: - It makes it possible to refer to values using a small fixed-length reference suitable for content-addressable storage - The Value ID makes it possible to cryptographically verify that content is correct in an efficient way (since it acts as the Merkle root of the value when seen as Merkle tree). ## Types ### Primitive Types #### Integers An integer is a a whole number (positive or negative) as commonly defined in arithmetic. Convex allows big integers up to the size of 32768 bits, i.e. around `1.4*10^9864`. This may be extended in the future, though we haven't found a sensible use case that is likely to require integers this large. #### Long A Long is a 64-bit, signed integer. Longs are the subset of integers within this 64-bit range. For efficiently reasons, Convex automatically uses longs in place of big integers where possible: from a developer perspective, there is usually no need to distinguish between the two. Examples: ```clojure 1 -127 9223372036854775807 ;; The maximum Long value ``` Longs are the natural representation of small integer values within a fixed range. They are suitable for representing common concepts such as indices, quantities of items (including digital asset quantities). Longs are also used to represent quantities of native Convex Coins (which by the definition of the 10^18 max supply cap, are guaranteed to fit in 64 bits and not overflow when value quantities are added or subtracted). #### Byte A Byte is an 8-bit, unsigned integer. From a developer perspective, they can be generally be considered simply as longs in the in the range 0-255. Bytes are useful for representing small integer values efficiently, such as a small set of flags or short codes. They are also important as the individual elements of Blob data (equivalent to immutable byte arrays). They are encoded as just 1-2 bytes of data, therefore recommended for very memory conscious applications. #### Double A Double is a 64-bit double precision floating point value as defined in the IEEE 754 standard. Examples: ```clojure 1.0 -3.4e-20 ##NaN ##Inf ``` Doubles are suitable for many applications that need to represent numerical values that can be very large or very small, but do not need to maintain precision beyond a certain number of decimal places. The maximum IEEEE 754 double value is around `1.7976931348623157*10^308`. While the lowest bits of precision may be lost, double computations are still deterministic. Doubles also support some special values as per the IEEE 754 standard: Positive infinity, negative infinity, negative zero and NaN (not a number). #### Character A Character is a Unicode code point expressed as a 32-bit unsigned integer. A Character can map to 1-4 bytes in UTF-8 encoding. For maximum efficiency, characters in the ASCII range should be used as these will map to a minimal 2-byte Encoding. #### Boolean A Boolean value is one of the two values `true` and `false`. In addition to their utility in general purpose programming, `true` and `false` are particularly efficient in the CVM, requiring only 1 byte of Encoding. When considering truth values in the the CVM, any Value is considered "truthy" of "falsey". `false` and `nil` are the only Values that are considered falsey, all other values are truthy. The reason for this is that it is often useful to directly test for the presence or absence of a value in a conditional expression without converting to Boolean values first, e.g. ```clojure (if (get {1 :foo 2 :bar} 1) "Found Foo" "No Foo") => "Found Foo" ``` #### Address An Address is an identifier for a Convex Account. Examples: ```clojure #1 #666 ``` Addresses can be considered equivalent to 63-bit positive integers, though they are not intended for use in calculation. Note that Longs could have been used for this purpose, however a specialised Address value type has some additional advantages: - A separate notation for Addresses makes them more clearly visible in code. - We can apply additional security validation and prevent some user errors (e.g. getting argument orders wrong and passing an asset quantity instead of an address which might produce unexpected results...) - The implementation can be made more optimised #### Blob A Blob is an immutable sequence of Bytes, with a length up to what can be indexed with a Long. Examples ``` 0x1234 ;; A 2 byte blob 0xd553110223c4e5217221aeab1319942163e57fffd4e89415c057ba031af6a8e0 ;; A very secret network key. Maybe? 0x ;; The empty Blob (0 bytes) ``` Blobs are especially useful for storing opaque units of data that may be important to external systems (e.g. client data encodings) as well as cryptographic values such as keys, hashes (including value IDs) or verification proofs. While is is possible to manipulate Blobs in CVM code, this is not usually recommended: such handling should normally be done off-chain. #### String A String is a sequence of bytes intended to represent the UTF-8 character encoding of text. Examples: ``` "Hello Convex" "" ;; The empty string ``` Internally, storage and management of Strings is very similar to Blobs. #### Symbol A Symbol is a identifier used to name things: values stored in an Account environment, or meaningful symbolic values in code. Example: ```clojure a count hello ``` Symbols are 1-128 bytes long, expressed in UTF-8 encoding. Symbols have special behaviour when evaluated: they perform a lookup of the value named by the symbol in the current environment. If this behaviour is not desired, they should be **quoted** with `'` to specify that the actual symbol is required, not the referenced value. An example of this usage: ```clojure (def a 1234) ;; Define a in current Account => 1234 a => 1234 ;; Symbol lookup is performed 'a => a ;; No lookup is performed for quoted symbol ``` Internally, Symbols *may* contain arbitrary characters (including badly formed UTF-8), but some of these may not read correctly in an off-chain Parser - therefore it is up to users to ensure that the Symbols they define are readable if this is a requirement. #### Keyword A Keyword is similar to a Symbol, but generally should be used to specify fields or other meaningfully named categorical values in data structures. Examples: ```clojure :foo :hello ``` Keywords are generally more convenient to use than Symbols as explicit values in user code since they do not need to be quoted. ### Collection Types The CVM defines a number of persistent, immutable collection types. #### Vector A Vector is a fixed length indexed sequence of values. Examples: ```clojure [1 2 3 4] ;; A Vector of Longs [1 :foo true] ;; Heterogeneous ector containing a Long, a Keyword and a Boolean [] ;; The empty Vector ``` Vectors are the most natural way of expressing most sequences where array-like behaviour is required. They support Long indexes. Vectors are particularly efficient when appending to the end of the Vector, which is always an `O(1)` operation. Indexed access and update is also guaranteed to be `O(1)`. #### List A List is a fixed length indexed sequence of values ntended to represent CVM code Examples: ```clojure (+ 2 3) ;; An arithmentic expression () ;; The emplty List ``` Lists have special behaviour when evaluated: They are treated as expressions by the compiler to be executed, where the first element of the List specifies the function to be executed. As with Symbols, you can **quote** a List to prevent immediate evaluation: ```clojure (+ 2 3) => 5 ;; List is evaluated as an expression '(+ 2 3) => (+ 2 3) ;; Quoted list is not evaluated ``` Lists are particularly efficient when prepending to the front of the List, which is guaranteed to be an `O(1)` operation. This property is especially useful when constructing code, since a common pattern is to prepend a function to a List of arguments. Lists also guarantee `O(1)` indexed access and update - in this sense Convex Lists are considerably superior to traditional linked lists. #### Map A Map is a data structure that maps keys to values. Keys and values may be any valid Value (including other data structures). Examples: ```clojure {} ;; The empty Map {1 2} ;; Map of key 1 to value 2 {:a 1 :b [:foo]} ;; Map with Keyword keys and heterogeneous value types ``` The map may also be considered as a sequential collection of Map entries, where each entry is a `[Key Value]` Vector. Examples of this usage: ```clojure (first {1 2}) => [1 2] ``` Internally, a Map is structured an efficient tree indexed with the the Value ID of values in the Map. All entries will therefore be ordered by the Value ID of Map keys. #### Index An Index is a specialised form of Map where the keys are enforced to be Blob-like values. Data is internally stored in an immutable persistent radix tree for efficient sorting and indexed access. Indexes have some advantages over regular Maps: - Entries are sorted according to the Blob keys (taken as unsigned, big-endian integers) - They support efficient slicing and indexing using Blob keys Indexes can be created using the core function `index` ``` (index) ``` #### Set A Set is a data structure that contains zero or more values as **members** of the set. From a logical perspective, every Value is either present or excluded from the set. For this reason, you can consider a Set as a mapping from any value to the Boolean type. To reflect this, the standard function `get` returns `true` or `false` when used with sets: ```clojure (get #{1} 1) => true (get #{1} 2) => false ``` ### Record Types Records are specialised data structures that implement a defined sequence of fields. Records behave like Maps when accessed using their field names as keys mapped to the corresponding field values. If updated, errors may occur if the update is inconsistent with the requirements on field values (e.g. attempting to put a value of the wrong type into a field). #### Block A block is a group of transactions submitted by a peer to the network. Unlike blockchains, Convex does not require blocks to be chained to the previous block via a hash - which allows them to be created and submitted in parallel. They are best thought of as groups of contiguous transactions submitted by the same peer in the ordering. #### Account An Account record represents information regarding the current state of an Account. See CAD004 for more details of the specification and contents of Accounts #### Peer A Peer Record represents the current state of a Peer. #### Ordering An Ordering is a special record type used by Peers to represent their Belief regarding: - The sequence of all Blocks - The current Consensus Point - The proposed Consensus Point Orderings are not normally used or accessible in CVM code, but are important for the CPoS algorithm and may be used to access the complete transaction history. #### State The State reocrd represents an immutable snapshot of the global State of the CVM. This includes - All Accounts - All Peers - Global settings and status flags - The Schedule #### Transaction Types Transaction types represent instructions to Convex that can be submitted by external clients. Transactions are specialised record types. For more details see CAD010. #### SignedData A `SignedData` record combines any other CVM value with an Ed25519 digital signature. SignedData records are used extensively for security, since the represent proof that someone constructing such data was in posession of the correct cryptographic key. In particular: - Peers sign Blocks and Orderings as part of CPoS - Clients sign Transaction records that they wish to execute ## See Also - [CAD003: Encoding](../003_encoding/index.md) — Binary encoding format for CVM values - [CAD012: Numerics](../012_numerics/index.md) — Detailed specification of numeric operations and arithmetic - [CAD013: Metadata](../013_metadata/index.md) — Metadata attached to definitions and values - [CAD033: CVM Types](../033_cvmtypes/index.md) — Extended CVM type system ## Implementation notes ### Equivalent host values Implementations MAY make use of different host types to represent the same CVM values, subject to the condition that they MUST respect canonical encodings, Value IDs and Value identity rules. The use of such different types MUST NOT result in CVM behaviour change, i.e. the distinction should not be visible to external observers. This allowance is intended mainly to permit performance optimisations, where specialised types can be used to represent CVM values in appropriate circumstances. --- # CAD003: Encoding Format ## Overview Convex uses the standard **CAD3 Encoding** format that represents data values as a **sequence of bytes**. The CAD3 encoding is an important capability for Convex because: - It allows values to be efficiently **transmitted** over the network between peers and clients - It provides a standard format for **durable data storage** of values - It defines a cryptographic **value ID** to identify any value. This is a decentralised pointer, which also serves as the root of a Merkle DAG that represents the complete encoding of a value. - CAD3 values are fundamental for enabling **lattice technology** for internet-scale decentralised data structures The encoding model breaks values into a Merkle DAG of one or more **cells** that can be individually encoded. Cells are immutable, and may therefore be safely shared by different DAGs, or used multiple times in the the same DAG. This technique of "structural sharing" is extremely important for the performance and memory efficiency of Convex. ## Special Requirements Convex and related lattice infrastructure have some very specific requirements for the encoding format which necessitate the design of the encoding scheme design here: - A **unique canonical encoding** for every value, such that it can be hashed to a stable **value ID** - An **efficient binary format** for both storage and transmission - A **self describing** format - no additional schema is required to read an encoding - Provision of **immutable persistent data structures** for the lattice data values used in Convex - Automatic generation of a verifiable **Merkle DAG** via references to other value IDs - Support for rich **data types** used in the CVM and lattice data (Maps, Sets, Vectors, Blobs etc.) - Data structure of **arbitrary size** may be represented. The lattice is huge. - Support for **partial data**: we often need to transmit deltas of large data structures, so need a way to build these deltas and reconstruct the complete structure when they are received (assuming existing data can fill the gaps) - Ability to read encode / decode `n` bytes of data in `O(n)` time and space to ensure **DoS resistance** - **maximum size limit** on encodings for any value (excluding referenced children) so that reading and writing can occur in fixed sized buffers - this allows **streaming capabilities** including zero-copy operations. No existing standard was identified that meets these requirements, e.g. - XML and JSON are inefficient text based formats, and lack unique representations of the same data - Google's protocol buffers require external schemas (and does not usually guarantee a unique canonical encoding) - ASN.1 is over-complex, generally relies on schemas and doesn't encode into bounded units suitable for building merkle trees or streaming data through fixed size buffers ## Examples ### Small Integer value The Integer `19` is encoded as: - tag of 0x11 (one byte integer) - the hex value 0x13 ``` 0x1113 = 19 ``` ### A 3-element Vector A Vector containing the Integer 101, the String "Hello" and an empty Set is encoded as: - `0x80` tag for a Vector - `0x02` count of Vector elements - `0x1164` embedded encoding of the Integer 101 - `0x30` tag for a String - `0x05` length of String - `0x48656c6c6f` 5 bytes UTF-8 encoding of "Hello" - `0x83` tag for a Set - `0x00` number of elements in the set (i.e. empty) ``` 80031165300548656c6c6f8300 = [101 "Hello" #{}] ``` Note that the encoded representation (13 bytes) is shorter than the printed representation (17 bytes), even though it incorporates rich data types and self-describing structural information! This is a key design goal in general for CAD3 ### A 4gb Blob A Blob of length 4gb, specified with: - `0x31` tag for a Blob - `0x9080808000` VLQ encoded length of 2^32 - `0x20`+child value ID (repeated 16 times, each child is a 256mb Blob) ``` 0x31908080800020af61c2faf10511466f73fe890524dccc056bddc79df37c7fbb 1823d5c8dae191202144a7641028ccd2259792d4c9626feb7f3cfb631eb7473d 3b95f1312fc4bf05202426c963ce5e0032fff92a028deec91a7466dd6d970cf4 78e510033854f6499120b2165e855dddd0daf62ba138ba0c1553a347b7f9f635 a589a2f9ab50be67c65120c7e6b0c74f27af4771ef06304fb02988bcd3bbe8f7 c2af84d3262d36f9ab75a620e8000a3edfa7bd1321c5d40e36a52c3c93d2be03 d976df15fd2323796c43435f201de13753be217f7fe3b89effaf7f2f5326bff4 94b50c1d86d96eeeb537bfcd5e205b2e93772a254a5196662707c68e851d16e3 a9386df7b40183daf82389d761032095ed2e62b005d363d33ccd4794ecc9f9f3 bae35979151ee1e340555e6d265a0820cf0902e3f9ca79469ed03e25085ad14b dea6a03fe41299ce538837e1e3666e3a20d614113ed517586ec7fe3576a9ce90 66f4795efbe85315fa0f6872085a408d4120ff5d93db343c185b47484aef9bd8 e1c5d171e87762960b659344b0aeda6ba0ba20fb047cfd86c9b81883b7920a44 f5f8909f6360a5e2f2d2d4ee4639d554ab7801202ad7a5b7bafc6d323f3d6ec1 4288775095775eb7d72f63cebffae6a0438ccb1120afc2b4cb2ed7c26d7026b1 74a22979bf4cf09468d5a31d33dca1aad04df0b1cc207881f54f571cd0416e5a f36bc6f133660bf8a60b4ded525332f9a314bea4ddea ``` On its own, the encoding above is a valid encoding for a single cell, but the encoding of the referenced values would need to be obtained in order to examine the tree of child Blobs - which is four whole gigabytes of data. This is an example of a "partial" value. This example illustrates some key benefits of the CAD3 design: - It is possible to examine and validate the top levels of CAD3 data structures independently, and only retrieve / decode sub-trees where necessary. - There is very little overhead for large structures. On 4gb of random data, the overhead is ~1% (mainly from ~1.1 million cryptographic hashes, which are in any case necessary for validation as a Merkle tree) - The entire tree is just 6 levels deep - so navigating down to any single byte only requires 5 lookups via value IDs. ## Basic Rules and Concepts ### Cells The fundamental entities that are encoded are called cells. Cells may contain other cells by reference, and therefore a top-level cell can be regarded as a directed acyclic graph (DAG). Since cell encodings contain cryptographic hashes of the encodings of any referenced cells, this is furthermore a Merkle DAG. ### Branches Branches are group of cells which are collectively stored in a single encoding. Branches are an important optimisation, since they reduce the need to produce many small individual cell encodings, and reduce the need for hashing, since only branches need to be hashed to produce references in the Merkle tree. ![Branch Encodings](branch.png) ### Encoding A CAD3 encoding MUST be a sequence of bytes. Any given cell MUST map to one and only one encoding. Any two distinct (non-identical) cells MUST map to different encodings It MUST be possible to reconstruct the cell from its own encoding, to the extent that the cell represents the same value (it is possible for implementations to use different internal formats if desired, providing these do not affect value equality semantics) The encoding MUST have a maximum length of 16383 bytes. This ensure that a cell encoding will always fit within a reasonable fixed size buffer, and guarantees that most operations on can achieve `O(1)` complexity. ### Value ID The value ID of a cell is the SHA3-256 hash of the encoding of the cell. All cells have a unique encoding, therefore they also have a unique value ID (subject the the assumption that the probability of SHA3-256 collisions is extremely low). A value ID reference may be utilised as a "decentralised pointer" to an immutable value, or used as an index for content addressable storage. Note: since only tree roots and branches are likely to be stored in storage systems, care should be taken with value IDs that point to intermediate non-branch cells, as these may not be persisted in storage. If in doubt, navigate down from a known root or value ID. ### References A cell encoding MAY contain references ("Refs") to other cells. There are two types of reference: - **Embedded**, where the embedded cell's encoding is included within the parent cell encoding - **Branch**, where an external reference is encoded as a byte sequence that includes the Value ID of the referenced cell (i.e. the branch) From a functional perspective, the difference between an embedded cell and a branch cell is negligible, with the important exception that following a branch reference will require accessing a separate encoding (typically cached in memory, but if necessary loaded from storage). From a performance perspective however, this distinction is extremely important: - Embedding allows multiple values to be included in the encoding of a single cell. For example a small Vector like `[1 2 3 4 5]` will be encoded within a single top level cell encoding, with the individual element values being embedded. - It reduces the number of SHA3-256 hash operations that need to be performed, since typically these need only be computed on branch cells. - It reduces the overall number of nodes in Merkle DAGs of Cells, reducing the number of individual calls to network and storage functionality. - A typical branch will have a relatively high branching factor (e.g. 10-30 child references) - this helps to navigate quickly down to leaf cells with the minimum number of IO operations. #### Embedded References A cell may be defined as embedded in which case the cell's encoding is inserted into in the encoding of the parent cell. If a cell is embedded, it MUST NOT be included in the encoding of another cell by external reference. This restriction is required to guarantee uniqueness of encoding (if not enforced, child cells might be encoded as either an embedded reference or by external reference, thus giving two or more different encodings for the parent cell). An embedded cell MUST have an encoding of 140 bytes or less. This restriction helps to ensure that cell encodings which may contain many child embedded references cannot exceed the overall 16383 byte limit. #### External References An external reference is a reference to a cell that is not embedded, i.e. points to a branch cell. An external reference MUST be encoded using the value ID of the target cell. This requirement ensures the integrity of a complete Merkle DAG of cells. ### Data Structures Data structure (such a Maps, Vectors etc.) are designed to operate as trees of cells. Each child cell is itself a valid data structure representing a subset of the whole data structures. For example, a Vector with 256 elements is a tree root containing 16 references to child vectors with 16 elements each. This approach is important for several reasons: - It is a natural way to structure the components cells of the data structure as a Merkle DAG - The child cells can be re-used as valid data structures elsewhere (e.g. allowing structural sharing of common subsequences of vectors) - It ensures that each child cell is a valid cell with a unique encoding in its own right - It allows fast indexed and sorted access into data structures (similar to B-trees) ### CVM Values Most cells represent valid CVM values, i.e. are permitted as first class values in the Convex Virtual Machine. Not all cells represent true CVM values, since cells may also be used for internal data structures within larger CVM values, or represent values that are only used outside the CVM. ### Valid and Invalid Encodings A sequence of bytes is a "valid" encoding if there exists a cell which produces the same sequence of bytes as its encoding. Conversely, a sequence of bytes is an invalid encoding if there is no cell which produces the same sequence of bytes as its encoding. Implementations MUST be able to reconstruct a cell from any valid encoding. Implementations MUST recognise an invalid encoding, and in particular: - Implementations MUST recognise an invalid encoding if the byte sequence contains additional bytes after the end of a valid encoding - Implementations MUST recognise an invalid encoding if the byte sequence terminates before enough bytes are obtained to complete a valid encoding Implementations MUST be able to produce the unique valid encoding for any cell. Note: Random byte strings are almost always invalid. This is a good thing, because it allows us to quickly reject corrupt or malicious data. This property is due to the multiple constraints on validity: - Quite a few tags are illegal / reserved - Within each value, there are significant constraints on validity (correct VLQ counts, embedded children must be 140 bytes or less etc.) - Even for an otherwise valid encoding, the length of the encoding must be exactly correct ### Extensibility CAD3 is designed for applications to use, and is therefore **extensible**. This is an important aspect of the design: Just like HTTP does not enforce a specific meaning on content included in a PUT request, CAD3 does not specify how an application might choose to interpret specific CAD3 encoded values. Applications using CAD3 encodings MAY assign semantic meaning to values on an application-specific basis. Applications SHOULD honour the logical meaning of defined CAD3 types, e.g.: - A CAD3 Integer should represent an integer value in the application (in some cases it might be repurposed e.g. as a code value) - A CAD3 String should be treated as a UTF-8 string if the application supports strings / text data. Applications SHOULD use the Blob type (`0x31`) for data which has a different binary encoding. This allows applications to encode arbitrary data in CAD3 structures with custom encodings or using other standards. Implementations MUST preserve CAD3 encoded values, even if they do not recognise the meaning. This ensures that implementations are compatible with all applications and systems can relay application specific data even if they do not understand it. In practice this means: - Applications can define what a particular cell value means in context, e.g. the vector `[1 17 :owns]` might represent an edge in a graph where entity `1` "owns" entity `17` - Certain categories of values (`0xAn`, `0xCn`, `0xDn` and `0xEn`) are explicitly intended for application usage - Independent of semantic meaning, applications can encode and decode arbitrary CAD3 data using a suitable implementation library. Application developers SHOULD document their extensions so that other developers are able to interpret their extension values if required. ## Encoding Format ### Tag Byte The first byte of the encoding is defined to be the "tag", which designates the general type of the data value, and determines how the remainder of the encoding should be interpreted. Implementations MUST reject an encoding as invalid if it starts with a tag byte that is not defined in this document. Tags are designed with the following objectives: - The tag byte enables very fast branching regarding how to interpret the following bytes - We use enough unique tags so that the 1 byte conveys a meaningful amount of information. Every bit counts! - Plenty of tag bytes are still available for future extension - The hex values convey at least some meaning to experienced human readers. `nil` is `00`. Numbers start with `1`. booleans start with `b`. `ff` is fatal failure etc. ### Categories The high hex digit of each tag byte specifies the general category of teh data value. These are defined as follows: | Pattern | Category | Purpose | | -------- | -------------------- | -------- | | 0x0x | Basic constants | Special values like `nil` | | 0x1x | Numerics | Integers, Doubles | | 0x2x | References | References to branch values | | 0x3x | Strings and Blobs | Raw Blob data, UTF-8 Strings etc. | | 0x4x | Reserved | Reserved for future use, possible N-dimensional arrays | | 0x5x | Reserved | Reserved for future use | | 0x6x | Reserved | Reserved for future use | | 0x7x | Reserved | Reserved for future use | | 0x8x | Data Structures | Containers for other values: Maps, Vectors, Lists, Sets etc. | | 0x9x | Cryptography | Digital Signatures etc. | | 0xAx | Sparse Records | For application usage, records that frequently omit fields | | 0xBx | Byte Flags | One-byte flag values (0xB0 and 0xB1 defined as CVM booleans) | | 0xCx | Coded Values | For application usage, values tagged with an code value | | 0xDx | Data Records | For application usage, records that have densely packed fields | | 0xEx | Extension Values | For application usage | | 0xFx | Special Values | Mostly reserved, 0xFF is illegal | The categories have been designed with the following purposes in mind: - Include all key fundamental types for decentralised lattice data structures - Allow rapid recognition of general types based on the first hex digit - Extensibility: several categories are designed for flexible application usage - Some degree of human interpretability (from looking at hex values) - Reasonable space for future extension ### VLQ Counts VLQ Counts are unsigned integer values expressed in a base-128 encoding using a Variable Length Quantity (VLQ) format. They are useful where values are usually small and negative numbers are not meaningful, e.g. the size or length of data structures. Encoding rules are: - The high bit of each byte is `1` if there are following bytes, `0` for the last bytes. - The remaining 7 bits from each byte are considered as a standard unsigned big-endian two's complement binary encoding. - The encoding is defined to be the shortest possible such encoding for any given integer. In data structures, a VLQ Count is frequently used to specify the number of elements in the data structure. It should be noted that this system can technically support arbitrary sized integers, however for use in CAD3 encoding it is limited to 63-bit integer values. It seems unlikely that anyone will ever actually need to construct or encode a single data structure with this many elements, so this may be considered sufficiently future-proof. Note: In the Convex reference implementation a signed variant (VLC Long) is also available. This is not currently used in CAD3. ### `0x00` Nil The single byte `0x00` is the encoding for the `nil` value. `nil` is conventionally used to indicate the absence of a value in applications. ### `0x10` - `0x18` Integer (Long) ```Encoding 0x1n ``` A Long value is encoded by the Tag byte followed by `n` bytes representing the signed two's complement numeric value of the Integer. The Integer MUST be represented in the minimum possible number of bytes - excess leading sign bytes are an invalid encoding. Note: The value zero is conveniently encoded in this scheme as the single byte `0x10` Note: This encoding is chosen in preference to a VLQ encoding because: - The length of a small integer can be included in the tag, making it more efficient than VLQ which requires continuation bits - It is consistent with the natural encoding for two's complement integers on most systems - The numerical part is consistent with the format for BigInts ### `0x19` Integer (BigInt) ``` 0x19 ``` A "Big" Integer is represented by the tag byte `0x19` followed by the VLQ encoded length of the Integer in bytes. The Integer MUST be represented in the minimum possible number of bytes - excess leading sign bytes are an invalid encoding. This is necessary to ensure an unique encoding for every Integer. The length MUST be at least `9` (otherwise the integer MUST be encoded as the Long version of Integer). With the exception of the tag byte, The encoding of a BigInt is exactly the same as a Blob with `n` bytes. ### `0x1D` Double A Double is an IEEE754 double precision floating point value. ``` 0x1D <8 bytes IEEE 754> ``` A Double value is encoded as the Tag byte followed by the 8-byte big-endian standard representation of an IEEE 754 double-precision floating point value. **Every 64-bit pattern is a valid encoding.** All IEEE 754 values are supported, and each distinct 8-byte pattern — including each distinct `NaN` bit pattern, both signed zeroes, the infinities, and subnormals — is a *distinct value* with its own unique canonical encoding and value ID. A decoder MUST accept any 8 bytes in this position and MUST NOT reject a "non-canonical" `NaN`. This is consistent with the unique-encoding requirement: unlike Integers (where excess leading bytes are *redundant* and therefore an invalid encoding), distinct `NaN` bit patterns carry distinct payloads, so each is a genuinely distinct value with exactly one encoding. Canonicalisation of `NaN` is a **CVM value-level concern, not an encoding-level one**. The CVM defines a single canonical `NaN`, written `##NaN`, with the encoding `0x1d7ff8000000000000` (the quiet-`NaN` bit set, all other payload bits zero). The CVM never *produces* any other `NaN`: every Double produced by a CVM operation — arithmetic, the `double` coercion, the `##NaN` reader literal, etc. — normalises a `NaN` result to this canonical form. A `NaN` with any other bit pattern (e.g. constructed via the `#[1d…]` raw-encoding reader form, or decoded from storage or the network) is still a valid Double cell, but is a *distinct* value from `##NaN`: it is not `=` to `##NaN`, has a different value ID, and prints in raw form as `#[1d…]`. CVM numeric operations coerce such a value to `##NaN` when it is used. Implementations MUST enforce canonical `NaN` by coercion at the CVM value layer, and MUST NOT enforce it by rejecting encodings. ### `0x20` Ref A Ref is a special encoding that points to a branch cell encoding using its value ID (cryptographic hash). ``` 0x20 <32 bytes Value ID> ``` An external reference is encoded as the tag byte followed by the 32-byte value ID (which is in turn defined as the SHA3-256 hash of the encoding of the referenced value). They are not themselves cell values, rather they represent a reference to another cell An implementation MUST NOT admit a Ref as a valid cell encoding in its own right: it can only be included to represent a child value in another encoding. Ref encodings are used for child values contained within other cell encodings subject to the following rules: - They MUST be used whenever the child cannot be embedded (i.e. is a branch cell). - They MUST NOT be used when the child cell is embedded. These rules are necessary to ensure uniqueness of the parent encoding (otherwise, there would be two or more encodings for many values, e.g. one with an embedded child and the other with a external branch ref). ### `0x30` String A String is a sequence of bytes with UTF-8 string encoding assumed. ``` If String is 4096 bytes or less: 0x30 If String is more than 4096 Bytes: 0x30 (repeated 2-16 times) ``` Every String encoding starts with the tag byte and a VLQ-encoded length. Encoding then depends on the String length `n`. - If 4096 characters or less, the UTF-8 bytes of the String are encoded directly (`n` bytes total) - If more than 4096 bytes, the String is broken up into a tree of child Blobs, where each child except the last is the maximum sized child possible for a child string (1024, 16384, 262144 etc.), and the last child contains all remaining characters. Up to 16 children are allowed before the tree must grow to the next level. Because child strings are likely to be non-embedded (because of encoding size) they will usually be replaced with Refs (33 bytes length). Thus a typical large String will have a top level cell encoding of a few hundred bytes, allowing for a few child Refs and a (perhaps embedded) final child. Importantly, this design allows: - Arbitrary length Strings to be encoded, while still keeping each cell encoding smaller than the fixed maximum size - Structural sharing of tree nodes, giving O(log n) update with path copying - Low overhead because of the high branching factor: not many branch nodes are required and each leaf note will compactly store up to 4096 characters - Most of the implementation can be shared with Blobs Note: UTF-8 encoding is assumed, but not enforced in CAD3 encoding rules. Applications SHOULD determine their own rules for handling invalid UTF-8. Note: with the exception of the tag byte, String encoding is exactly the same as a Blob. This includes the fact that the children of Strings are in fact Blobs. This is useful because it facilitates structural sharing between large Strings and Blobs. ### `0x31` Blob A Blob is an arbitrary length sequence of bytes (Binary Large OBject). ``` If Blob is 4096 bytes or less: 0x31 If Blob is more than 4096 bytes: 0x31 (repeated 2-16 times) ``` Every Blob encoding starts with the tag byte and a VLQ-encoded length. Encoding then varies depending on the Blob length `n`. - If 4096 bytes or less, the bytes of the Blob are encoded directly (`n` bytes following the VLQ Count) - If more than 4096 bytes, the Blob is broken up into a tree of child Blobs, where each child except the last is the maximum sized child possible for a child Blob (4096, 65536, 1048576 etc.), and the last child contains all remaining bytes data. Up to 16 children are allowed before the tree must grow to the next level. Applications MAY include whatever data or encoding they wish within Blobs. Applications SHOULD use Blobs for binary data where the data is not otherwise meaningfully represented as a CAD3 type. Examples might include PNG format image data, a binary database file, or text in an encoding other than UTF-8. Applications SHOULD use Blobs for data which is naturally represented as a string of bytes, e.g. cryptographic hashes or signatures. Because child Blobs are likely to be non-embedded (because of encoding size) they will usually be replaced with Refs (33 bytes length). Thus a typical large Blob will have a top level cell encoding of a few hundred bytes, allowing for a few child Refs and a final child for the remaining bytes (which may be embedded). Importantly, this design allows: - Arbitrary length Blobs to be encoded, while still keeping each Blob encoding within a fixed size - Structural sharing of tree nodes, giving `O(log n)` updates with path copying - Relatively low overhead, because of the high branching factor: not many branch nodes are required and each leaf note will compactly store 4096 bytes. ### `0x32` Symbol Symbols are used for naming things, e.g. a CNS name like `convex.core`. ``` 0x32 ``` A Symbol is encoded with the tag byte, an unsigned count byte `n`, and `n` bytes of UTF-8 encoded characters. The Symbol MUST have a length of 1-128 UTF-8 bytes. Any other length is invalid. This guarantees that all Symbols are embedded. Applications SHOULD use Symbols when referring to named values that need to be externally looked up based on the context in which they are used e.g. values defined in the environment of a Convex account. ### `0x33` Keyword Keywords are used for human readable values, e.g. names of keys in a larger data structure. ``` 0x32 ``` A Keyword is encoded with the tag byte, an unsigned count byte `n`, and `n` bytes of UTF-8 encoded characters. The Keyword MUST have a length of 1-128 UTF-8 bytes. Any other length is invalid. This guarantees that all Symbols are embedded. Applications SHOULD use keywords for known values that may appear in other data structures, are intended to be short and need to be human readable. Reasonable examples include: - the `:name` keyword for a key value in a Map describing a person - the `:expired` keyword at the end of a vector describing a smart contract that has expired - the `:NOBODY` keyword as an error code on the CVM Applications SHOULD prefer Keywords over Symbols in most data structures, unless it is a name for something external to the data structure. ### `0x3c` - `0x3f` Character A Character is a Unicode character uniquely defined by its code point. ``` The encoding is one of the following: 0x3c <1 Byte> 0x3d <2 Bytes> 0x3e <3 Bytes> 0x3f <4 Bytes> (reserved, not currently possible) Where the number of bytes is the minimal number of bytes required to represent the Unicode code point. ``` A Character value is encoded by the tag byte followed by 1-4 bytes representing the Unicode code point as an unsigned integer. A Character encoding is invalid if: - More bytes are used than necessary (i.e. a leading byte of zero) - The code point is beyond the maximum allowable (currently 0x10ffff) ### `0x80` Vector ``` If a leaf cell: 0x80 (repeated 0-16 times) If a non-leaf cell: 0x80 (repeated 2-16 times) ``` A leaf cell is a Vector with Count `n` being 0, 16, or any other positive integer which is not an exact multiple of 16. A Vector is defined as "packed" if its count is a positive multiple of 16. A leaf vector which is packed must therefore have a count of exactly 16 - such vectors for the leaf nodes of a tree of non-leaf vectors. A Vector is defined as "fully packed" if its Count is `16 ^ level`, where `level` is any positive integer. Intuitively, this represents a Vector which has the maximum number of elements before a new level in the tree must be added. All Vector encodings start with the tag byte and a VLQ Count of elements in the Vector. Subsequently: - For leaf cells, a packed prefix vector is encoded (which may be `nil`) that contains all elements up to the highest multiple of 16 less than the Count, followed by the Values - For non-Leaf cells, Child Vectors are encoded where each child is the maximum size Packed Vector less than Count in length, except the last which is the Vector containing all remaining Values. This Encoding has some elegant properties which make Convex Vectors particularly efficient in regular usage: - Short Vectors (0-16 count) are always encoded in a single cell, which may require no further cell encodings in the common case that all elements are embedded. - The last few elements of the Vector are usually in a Leaf Vector, which allows `O(1)` access and update to elements - Append is `O(1)`, usually with a small constant (only extending the current leaf vector) - Access and update are also `O(1)` (Note: could be considered `O(log n)` with a high branching factor, but upper bounds on vector size make this `O(1)` with a constant factor accounting for the maximum possible depth) ### `0x81` List A List is encoded the same as a Vector, except: - The tag byte is `0x81` - The elements are logically considered to be in reversed order (i.e. the last element encoded is the first element of the list) ### `0x82` Map A Map is a hash map from keys to values. ``` If a map leaf cell: 0x80 ... (key + value repeated n times, in order of key hashes) If a map tree cell: 0x80 (repeated 2-16 times) Where: - specifies the hex position where the map branches (0 = at the first hex digit,.... 63 = last digit) - is a 16-bit bit mask indicating key hash hex values are included (low bit = `0` ... high bit = `F`) - are Refs to Map cells which can be Leaf or non-Leaf nodes ``` If the count n is 15 or less, the Map MUST be encoded as a map leaf cell, otherwise it MUST be encoded as a map tree cell. This is to ensure unique encoding. The number 15 is chosen for optimal binary search and so that all types of Map cells have 0-16 child value refs. All entries MUST be encoded in the order of key hashes. - In a map leaf cell, this means that the `` pairs are sorted by key hash - In a map tree, it means that the child maps are ordered according to the hex digit at the shift position All entries within a map tree cell (directly or indirectly) MUST have identical key hashes up to the position of the shift byte. Since hashes are 32 bytes, this means that the maximum possible shift byte value is 63 (though this is very unlikely to occur in practice: it would imply someone found at least 9 SHA-256 hashes differing only by the last 4 bits!) A map tree cell MUST have at least two children (if not, it should not exist since the branch must occur at a later hex digit). Again, this is necessary to ensure uniqueness of encoding. A Map MAY contain arbitrary keys and values. ### `0x83` Set A Set is a logical set of included values. A Set is encoded exactly the same as a Map, except: - The tag byte is `0x83` - The Value elements in the entries are omitted ### `0x84` Index ``` 0x84 (repeated 1-16 times) Where: is either: - 0x00 (if no entry present at this position in Index) - 0x20 (if entry present) is an unsigned byte indicating the hex digit at which the entry / branch occurs. If an entry is present, depth must match the hex length of the entry key is a 16 bit bitmap of which child Index nodes are present at the given depth (low bit = `0` ... high bit = `F`) Special cases: - If Count is 0 everything following the Count is omitted (the empty Index) - If Count is 1 the first byte of and everything following the Entry is omitted (single Key / Value pair) - If count is 2+ then in order to be valid the Index node must have either: - An entry at this location plus at least one child - No entry and at least 2 children which differ in the hex digit at this depth ``` An Index serves as a specialised map with ordered keys. Logically, it is a mapping from byte arrays to values. Key values MUST be Blobs, Strings, Addresses, Keywords or Symbols. These are regarded as "BlobLike" because they can be considered as a sequence of bytes like a Blob. This encoding ensures that entries are encoded in lexicographic ordering. Unlike the hash based Maps, an Index is constrained to use only BlobLike keys, and cannot store two keys which have the same Blob representation (though the keys will retain their original type). ### `0x88` Syntax A Syntax Object is a value annotated with a Map of metadata. ``` 0x88 Where is a value which is either: - 0x00 (nil) if there is no metadata (considered as empty map) - A a non-empty Map containing the metadata The can be any value. ``` The metadata MUST be a Map of keys to values (`nil` is used as the empty Map, for efficiency) ### `0x90` Signed Represents a digitally signed data value. ``` `0x90` Where: - Public Key is 32 bytes Ed25519 public key - Signature is 64 bytes Ed25519 signature ``` The Signature is expected to be the Ed25519 signature of the Value Ref encoding. This means that the signed bytes will be either an embedded value (1-140 bytes), or an `0x20` Ref to a branch cell (33 bytes). This format is effective because it means the encoding of the Signed data value is sufficient to validate the Signature without any external references. The signature may or may not be valid: an invalid signature is still a valid value from an encoding perspective. ### `0x91` Signed (Short Version) This is the same as `0x90` signed but excluding the public key ### `0xA0` - `0xAF` Sparse Records A Sparse Record is an structure containing 0-63 fields. The fields are stored sparsely, with `nil` values omitted. ``` `0xAn` (repeated for each set bit in inclusion mask) Where: - `n` is an implementation-defined hex value (0-15) which MAY be used to disambiguate distinct record types. - The inclusion mask is an unsigned integer (63 bits max) represented with a VLQ Count. ``` The inclusion mask is a non-negative value indicating which fields are included in the Record as bit mask. The number of Value Refs MUST be equal to the number of `1` bits in the inclusion count, with the first Value Ref corresponding to the least significant `1` bit etc. For maximal encoding efficiency, it is recommended that the most commonly included fields are defined in the first 7 positions, which maximises the chance that the inclusion mask will only require one byte. Value Refs, if included, MUST NOT be `nil`. This is necessary to ensure unique encoding, since excluded fields are already defined as `nil`. Implementations which require more than 63 fields MAY adopt their own scheme to further embed additional structures within the 63 fields available. Reasonable options include: - Group subsets of similar fields into child Records. This is especially useful if common groups of fields are frequently used in multiple places and logically grouped together. - Have the Record specify one field which contains a vector of additional fields - Use the first field (index 0) to specify the interpretation of following fields (which may contain arbitrary values as sub-structures) ### `0xB0`-`0xBF` Byte Flags (Extensible) Byte flags are one byte encodings (similar to Booleans) available for application specific use. ``` `0xBn` Where - n = a hex value from 0-15 ``` Applications MAY use byte flags as a efficient single byte value, i.e. the complete value encoding is always exactly one byte. For example, the encoding `0xb2` might represent an "unknown" value in ternary logic. Values `0xb0` and `0xb1` are typically used for the two boolean values, though an application MAY repurpose these as single byte values (along with `0x00` and `0x10`) providing these values are not needed for some other purpose. Fun Idea: A 1-byte Lisp where `0x10` is an opening paren, `0x00` is a closing paren and `0xb0 - 0xbf` are the allowable tokens. ### `0xC0`-`0xCF` Codes Codes are values tagged with another value. ``` `0xCz` Where: - is any value indicating what code is being used - is any value representing the coded payload - z = a hex value from 0-15 ``` Codes are intended for applications to represent values requiring special interpretation depending on the code used. e.g. the code could be used to represent the MIME content type for a Blob of data. Applications SHOULD use a small code value (e.g. a small Long, or a Byte Flag) to specify the precise type of value being encoded, and a corresponding value that is meaningful for the given code value. Applications MAY in addition use the hex digit `z` to further disambiguate code types. In combination with the 18 valid one byte encodings, this gives a reasonably generous 288 distinct code types before another byte is required. Lisp aficionados and hackers may find appeal in the fact that e.g. `0xC0` can be used to represent a Cons cell. While Convex Lisp does not use this, it is a perfectly valid application of extension types. ### `0xD0`-`0xDF` Data Records Data Records are record types where every field value is encoded (i.e. the record is densely coded). ``` `0xDz` Where: - z = a hex value from 0-15 - n = the number of fields in the record ``` Data Record encoding is exactly the same as a Vector, with the exception of the tag byte. Note that if there are more than 16 fields, this means there will be a child prefix cell which is a Vector. Applications MAY use the hex digit `z` and/or the field count `n` to distinguish record types. If this is insufficient, applications MAY use the first or the last field value to indicate the type, or embed a Data Record as a coded value (`0xCz`) to tag with an arbitrary type. The intention of Data records is that applications may interpret these as records in their own custom format. For example, a record might represent a listing on a decentralised market place with fields such as Asset ID, Price, Seller ID, Listing description, Creation Time, Time Limit etc. The CVM interprets certain Data Records as CVM-specific record types (e.g. the `convex.core.cvm.State` type). ### `0xE0`-`0xEF` Extension Values Extension values are enumeration-style values allowing 16 application specific ranges, and 2^63 possible values for each. Typically, applications might use them for encoding known enumerations, such as a reference into an array of constant values. ``` `0xEz` Where: - z = a hex value from 0-15 - value = any non-negative Long value (0 - 2^63-1) ``` Extension values are arbitrary non-negative integer values with a one byte tag, where the low byte of the tag is available for applications to define a special meaning for the value. For example, Convex uses the extension value `0xEA` to indicate an Address. Since Addresses are allocated sequentially from zero (and Accounts can be re-used), this usually results in a short VLQ encoding. Extension values are considered "BlobLike" and can therefore be used a a key in an Index Examples: - an application might define `0xE5` as an extension where the value references a known JSON schema. - another application might define `0xE0` as an enum where the values are the possible states of a finite state machine ### `0xF0` - `0xFE` Reserved These values are reserved for possible future CAD3 extensions. at present they are illegal ### `0xFF` Illegal The `0xFF` tag is always illegal as a tag byte in any encoding. Implementations MUST treat and values encoded starting with `0xFF` as an invalid encoding. ### Reserved Tags Tag bytes or value ranges not otherwise specified are reserved for future CADs. Decoder implementations MUST treat all such cases as illegal and reject encodings they are unable to read. ## Cell life-cycle ### Construction Cells may be constructed in 3 ways: - Created during local processing (will not have encoding yet, but created on demand) - Received as external input and decoded (will already have encoding) - Loaded from storage (will have encoding, hash, and storage flags) ### Completion of partial cells (Acquisition) It may be the case that a tree of cells is partial, i.e. some referenced branches are not yet available. In such cases, processing SHOULD be deferred until the missing cells can be externally acquired. ### Garbage collection Cells which are no longer referenced by any cells currently in use may be safely garbage collected (either from main memory or storage). ## Cell validation Cell validation will typically occurs in multiple stages: - Encoding correctness (is this a valid encoding?) - Structural correctness (is the whole tree of cells valid?) - Semantic correctness (does the value make sense in this context?) ### Encoding correctness A cell's encoding can be quickly checked for correctness up to the point of external branch references (i.e. you know you have `n` external references that appear to be 32 bit hash values, but the encoding or validity of those may or may not be known). Implementations SHOULD detect and reject invalid encodings as early as possible. An invalid encoding MAY be considered sufficient evidence to discard the entire message in which it is received, since the source is provably not behaving correctly. ### Structural correctness Checking for structural correctness requires traversing external references in order to validate: - The external reference is the hash of a valid cell encoding - Each cell referenced is of a valid type expected by the parent cell - All cells referenced meet any requirements for consistency with the parent cell (e.g. correct counts for child data structures) - All cells referenced are themselves structurally correct Checking for structural correctness is typically an `O(n)` operation in the number of cells checked. For performance reasons, it is usually valuable to cache the results of structural correctness checks so that they do not need to be recomputed - this is especially important for large data structures with structural sharing. Applications SHOULD NOT trust or re-transmit CAD3 messages unless they have validated for structural correctness. This is for security and robustness reasons: structurally incorrect messages are likely to cause errors or unexpected behaviour which an attacker might exploit. ### Semantic correctness Checking semantic correctness is to validate that the cell value makes sense / has meaning in the context that it is used. A cell could be structurally correct but contain values that are illegal in some application (e.g. a Vector that should contain Integers but actually contains a String). Checking for semantic correctness is application defined and outside the scope of CAD003. ## Implementation Notes ### Applications Applications are free to assign semantic meaning to CAD3 encoded values. Applications SHOULD ensure that they are able to read, persist and communicate arbitrary CAD3 encoded values, even if the semantic meaning may be unknown. This is important for several reasons: - The application SHOULD be robust and not fail due to unrecognised but legal encodings - The application MAY need to pass on these values to other systems that do understand them In practice the recommended approach is: - Applications should be written to work with generic CAD3 data - If a specific value is to be accessed and used, the application should verify semantic correctness - If valid, application can proceed with the semantic meaning it defines - If invalid, this is presumably an exception that needs handling (e.g. a malicious message from an external source that should be rejected) ### Compatible Subsets CAD3 encodings are designed to support various other data formats as a natural subset. Applications may find it useful to exploit these correspondences to efficiently store any data in CAD3 format. - **JSON** encodes naturally using Map, String, Vector, Double, Integer, Boolean and Nil - **UTF-8** text encodes naturally as a String - **Binary data** naturally encodes as a Blob, or as a Code with the encoding format specified - **Encrypted data** is perfectly suited for storage in a Blob - **S-expressions** are naturally coded using Lists, Symbols and a selection of other values (Integers, Strings etc.) - **XML** can be encoded in multiple ways e.g.: - As a UTF-8 String - As a Vector where each element is either a content String or a markup value. The metadata map of a Syntax Object could be used to specify element attributes - **Tabular data** like **CSV** or **SQL** result sets is naturally represented as a Vector of Vectors. This has the added advantage of fast indexing by row number. - **Content addressable storage** is naturally represented with an Index, which has the advantage of fast indexed lookup by content ID - **Abstract Syntax Trees** are naturally encoded using Syntax Objects, where arbitrary metadata can be attached to nodes. The nodes themselves might contain a Vector of child nodes. Developers wishing to utilise such subsets SHOULD research and collaborate with other developers to establish common standards for embedding such data in CAD3 format. This is outside the scope of CAD3, but new CAD proposals defining such standards are welcome. ### Partial Implementations It is possible to write a partial implementation that understands only a subset of CAD3. This may be useful e.g. for embedded devices. Partial implementations MUST be able to decode any cell, and recognise it as valid / invalid from an encoding perspective. This is necessary for correctness and interoperability. Partial implementations MAY ignore CAD3 values that they cannot interpret. This means that they MUST at a minimum be able to: - calculate the length of an embedded value so that they can skip over it. This may require a bounded amount of recursion, as embedded values may embed other values inside them (up to a small depth limit limited by 140 bytes) - store the encoding of any value(s) they have ignored if they need to re-encode the data for onward transmission Partial implementations MAY ignore branch references, and hence avoid the need to compute SHA3-256 hashes / look up child cells by reference. In this case, care must be taken that values are small enough that they always result in embedded encodings. ### Convex JVM Implementation In the Convex JVM implementation, cells are represented by subclasses of the class `convex.core.data.ACell`. Having a common abstract base class allows for convenient implementation of common cell functionality, is helpful for performance, and ensures that all cell instances offer a common interface. The implementation allows for internal usage of "non-canonical" cells. These are `ACell` instances that may break normal rules, e.g. encoding more than 4096 bytes in a single flat Blob. These are used for temporary purposes (typically for performance reasons) and are always converted back to a canonical implementation for CAD3 encoding purposes. The implementation keeps singleton "interned" references for various common values. This is mainly to avoid repeated memory allocation for such values. These currently include: - Byte Flags including the two boolean values `true` and `false` - Small Integers (0-255) - ASCII Characters - Empty Maps, Sets, Strings and Blobs etc. - Static constants such as Strings, Keywords and Symbols used frequently in the CVM The JVM `null` value is interpreted as the Convex `nil` value. This is an implementation decision, again chosen for efficiency and performance reasons. However there is no formal requirement that `nil` must be represented this way (for example, it could be a singleton value). --- # CAD004: Accounts ## Overview Accounts are a fundamental construct in Convex - they are logical records in the CVM State that are either securely controlled by an external user, or operate as autonomous actors. Accounts are identified with an address, which are sequentially allocated in the form `#1567` Accounts are the primary means of managing security and access control for on-chain transactions. Any transaction executed by Convex must be associated with a user account and signed with a valid digital signature. This protects the user's account from unauthorised access. Any user account used in this way must have a 32-byte Ed25519 public key, of the form `0x9D98C7C6B9E89AEC23F4AF6D5175872C25982264AD91E95DC4B061EE3062BFD1`. If external transactions are not permitted for an account, its public key is set to `nil` (this is the case for autonomous actor accounts). Accounts also constitute the largest part of the on-chain CVM State. Accounts are used to store code and data, and to track holdings of various digital assets. In the future, accounts will probably constitute over 99% of the CVM State size - there isn't much else apart from data structure to support peers managing consensus and a little network-global data. ## Key Concepts ### Addresses Every account has an address, which is a unique ID that identified the account. Addresses are conventionally shown in the format `#1234`, and are primitive values in the CVM in their own right. Addresses are assigned sequentially whenever new accounts are created. It is impossible to change the address of an account once created - this is important because the Address is designed to be a stable unique identifier for the account. Addresses should be used as the unique ID for access control mechanisms, e.g. an actor might maintain a `Set` of addresses which are authorised to execute a security-critical operation. Addresses are also typically used as the key for indexed data structures that track ownership of digital assets. A common pattern is to represent ownership as a `Map` of addresses to numbers representing balances of the appropriate digital asset(s). ### User Accounts User accounts are accounts controlled / owned by individuals or organisations that use the Convex Network. A user account is defined as an account with a public key defined, which is used to validate the digital signature of transactions. The associated private key is assumed to be under the secure control of an external user. In this way, Convex supports full self sovereign control over accounts. A user account is considered the origin account during the execution of any transaction submitted for this account. ### Actor Accounts Actor accounts are autonomous accounts that manage trusted code and data. They can be considered as autonomous software components operating on the Convex Network. Typical applications of actors might include: - Implementing a digital asset such as a fungible token or collection of NFTs - Providing a public registry of accredited organisations - An auction house that manages multiple concurrent auctions, and ensures winners correctly receive their purchased items - Providing a shared code library for other accounts to utilise An actor account is defined as any account with no public key. Actors do nothing on their own initiative (with some limited exceptions e.g. scheduled operations). They need to be invoked by other accounts, e.g. a user will typically `call` an exported actor function from their own account. ### Identicons Accounts may be given an "identicon" which visually represents the public key of an account. ![Account identicons](identicons.png) Identicons are optional and intended for user interfaces where public keys are used and must be kept secure. Different GUIs may implement alternative identicon schemes, however the Convex Desktop used the following approach and other applications are invited to adopt the same format: - Identicons are 7x7 pixel bitmaps, symmetric about vertical axis - 4 colours, taken from last 12 bytes of key (3 RGB bytes each) - First 7 bytes of key define unique pixels, 2 bits each, i.e. one byte for each row of the identicon - The identicon for a null value is defined to be a black box (useful for identifying actors etc. with no public key) ### Lisp Machine Each account can be considered as a small, lightweight lisp machine! It has its own programmable environment, and can be interacted with via transactions (write) or queries (read-only). There's no limit on what can be done with this capability, as the CVM provides a fully Turing complete programming environment. You can control an account with a REPL, use it to script various on-chain operations, use it as a temporary environment for on-chain smart contract development etc. ### Controllers Controllers allow an account to be controlled by other accounts, as an alternative or in addition to the use of transactions signed with the account key. Typical purposes might include: - Allowing a trusted third party to recover the account if the user's private key is lost - Allowing a maintainer to make upgrades to actor / smart contract code - Allowing a DAO treasury to receive instructions from an actor that manages votes on proposals Optionally, an account may define a controller, giving the ability to one or more other accounts to control the account. The controller may be a specific address of another account, or a trust monitor that permits access to an arbitrary set of accounts that may be defined in code (e.g. accounts authorised by a governance actor or DAO). This is a powerful capability. A controller account can be used, for example, to give "root" access to an actor so that it can be upgraded or debugged after deployment. It is also a risk: Users SHOULD NOT set a Controller for any higgh value account they wish to keep secure unless: - They know exactly what they are doing - They fully trust the account(s) they are giving controller access to. dApps MAY set an account controller to be used as an account recovery mechanism, with the ability to reset the account's public key in the case that access to the original key pair is lost. A possible implementation is to use a time locked smart contract that updates the account after some delay, that can be cancelled in the case of a suspected attack or if the original key is found again. ### Recycling Accounts It is possible to recycle old accounts, perhaps even selling them! This is likely to be cheaper and more efficient than creating a new account, since it will save memory. It also helps keep the CVM state smaller overall. An example procedure for doing this securely is: - Transfer away any digital assets or other access control rights you want to keep - Set the controller to `nil` - Delete unwanted definitions from the account's environment with `undef` - Especially, it is important to delete: - any exported functions that might be called externally - The `*schedule-start*` value, which may enable scheduled operations - Set the account public key to the public key of the new owner With this method, accounts may be re-used by different individuals, with the secure knowledge that the previous owner(s) no longer have any control over the account. ## Account Specification The account is represented a data structure within the state of the CVM. Each valid address MUST have precisely one account record in the global state. If a state transition causes an update to information in the account record (e.g. changing a definition in the environment), the new state MUST reflect the account update. The account record (`AccountStatus` in the standard reference implementation) MUST be a valid CVM Record data structure with Keyword keys as follows: - `:sequence` - sequence number of the account, initially `0` - `:key` - account key, may be `nil` to indicate an actor, otherwise a 32 byte Blob representing an Ed25519 public key - `:balance` - balance of the account in Convex copper coins (a 64-bit natural number) - `:allowance` - unused memory allowance of the account, normally `0` but may be higher (e.g. if a memory accounting refund occurred) - `:holdings` - map of holdings of the account, attributed to any other accounts which have utilised `set-holding` (e.g. token actors) - `:controller` - a controller account, which has the power to issue commands for this account (e.g. `eval-as`) - `:environment` - a map of symbols to defined values in the account, initially `{}` - `:metadata` - a map of symbols to metadata for values defined in the account, if any. Initially `{}` - `:parent` - an optional address that is used to resolve symbols by default if not otherwise defined in this account. Initially `nil` ### Sequence Number The sequence number's purpose is to prevent replay attacks, since the same signed transaction with the same sequence number cannot be re-used. The sequence number MUST indicate the number of transactions which have been executed for this account. The sequence number MUST be `0` for a new account, or any account for which transactions have never been previously executed (e.g. an immutable actor) The sequence number MUST increase by `1` for each correctly signed transaction executed. ### Account Key The account key specifies which cryptographic public key (if any) can be used to control the account Each account MAY have a single account key. If the account key exists, it MUST be a 32 byte `Blob`. If the account key is not specified for the account, it MUST be treated as the value `nil` when accessed. The account key SHOULD represent a valid Ed25519 public key for which the owner of the user account is expected to have access to the corresponding private key. Security of the private key is the responsibility of the external user. The CVM MUST NOT process transactions for an account unless the Ed25519 digital signature on the transaction can be verified with the account key. See CAD010 for more details. The account key MAY be changed by a controller of the account to a new account key, or set to `nil`. ### Balance The balance field enables each account to hold a quantity of Convex Coins. The balance field MUST be a non-negative integer indicating the number of Convex copper coins controlled directly by the account. ### Allowance The allowance field enables each account to hold a pre-alloacted allowance of CVM memory for future use. If this is zero, any new memory allocations must be purchased at the prevailing memory pool price at the time the transaction completes. If a user's transactions result in a memory refund (by reducing the size of the global state), the refund is deposited automatically in the allowance field. The user may then utilise this memory for another purpose, or sell it if no longer required. The allowance field MUST be a non-negative integer indicating the number of bytes of unused memory allowance held by the account. ### Holdings The holdings field is an efficient way for actors to optionally store some data referring to each other account on the system. For example, it might be used to store a balance of actor-defined tokens that are held by the account. The holdings field MUST be a Index representing a mapping of Address to holding values. Values MAY be any CVM value. Holding values SHOULD be meaningfully defined by the address that sets them. ### Controller The controller field MAY be any CVM value, including `nil` If set to a specific Address, the CVM MUST regard that address as a controller. The CVM MUST regard any non-`nil` value in this field as defining a trust monitor, and check as if called with `(call controller (check-trusted? :control ))` to determine whether another account should be regarded as a controller. If another account is regarded as a controller, it MUST be able to control the account in its entirely, including use of `eval-as`. ### Environment The environment field MUST be a map of Symbols to defined values in the account. The environment field MUST be `{}` when initially created. ### Metadata The metadata field MUST be a map of Symbols to defined metadata in the account. The metadata field MUST be `{}` when initially created. CVM operations MAY set values in the environment without setting equivalent values in the metadata. This is primarily for efficiency purposes, as most environment values do not require metadata. ### Parent The parent field MUST be `nil` when the account is initially created --- # CAD005: CVM Execution ## Overview The Convex Virtual Machine (CVM) execution operates as decentralised virtual machine. The CVM implements a pure, deterministic state transition function which can be executed and validated by all peers. Conceptually this can be viewed as: ``` State' = f (State, Block) ``` Under this model, the latest consensus state can always be reconstructed given both: - A initial State - All Blocks in the CPoS ordering between the initial State and the current consensus point Normally, Peers maintain the current Consensus State, and update this accordingly whenever one or more new Blocks are confirmed by the CPoS Consensus Algorithm. However, a new Peer can reliably reconstruct the Consensus State from any preceding State as long it it also holds the necessary Blocks from that state onwards. This enables a new Peer to efficiently synchronise with the Convex Network without having to process all preceding Blocks. ## The State The State is a global, decentralised data structure which contains all of the currently active on-chain information. Logically, the purpose of transactions is to cause changes in the State. Some useful notes about the State: - It may be large - typically larger than RAM on many machines - There is only one "consensus" state at any one time, but it is also possible to refer to previous (or potential future) States using the State Hash. ### State Representation The State is regarded by the CVM as an immutable value, a special type of Record. See CAD-002 for more details on the specifics of Record types. Since it is a CVM value, the state is internally implemented as a Merkle Tree / DAG allowing for full cryptographic verification of the entire global state given a single 32-byte root hash. This model allows for immutable snapshots of the entire state to be analysed and stored for future reference. It also allows for the entire tree to be considered as content-addressable storage. ## State Transition The State Transition Function performs the following steps, in order: - Block Preparation - For each Transaction in the Block - Prepare an execution Context for the Transaction - Execute the Transaction - Complete the Transaction - Record transaction result (outside the State) - Block Completion ### Block preparation #### Timestamp update At the start of Block Preparation, the Timestamp of the Block is examined. If and only if the timestamp is later than the State timestamp, the State Timestamp is updated to be equal to the Block Timestamp. This procedure ensures that the State Timestamp never goes backwards (i.e. is monotonically increasing). #### Scheduled Execution As the next step of Block Preparation, the CVM examines the Schedule data structure in the State, and identifies if any transactions are scheduled to be executed before or at the State Timestamp. If any scheduled transactions exist, then the CVM selects a number of transaction up to the defined constant `MAX_SCHEDULED_TRANSACTIONS_PER_BLOCK` (in scheduled order, i.e. the earliest scheduled transactions are prioritised). The reason for this maximum limit is to prevent an excessive number of transactions scheduled at the same time from holding up progress on transactions in the current Blocks being processed (TODO: needs revisiting) For each selected scheduled transaction, the CVM executes the scheduled transaction as if it had been submitted at the beginning of the Block, with the following minor modifications: - There is no need to perfrom a full digital signature check, since the scheduled transactions were provably issued internally on the CVM - Transaction results not need to be reported back to Clients, since the scheduled transaction was not submitted by a Client ### Transaction Execution For each transaction executed, the CVM first checks the Account in the State for which the transaction is submitted. If the Account does not exist, the transaction is aborted. Assuming the account exists, the verifies the digital signature of the transaction against the current public key associated with the Account . If verification fails, the transaction is aborted. Otherwise, the CVM creates a Context for the Account and proceeds to execute the transaction in the given Context. ## Context Regular CVM Execution of operations occurs in a Context. A Context is required to keep track of relevant variables during execution, including: - The current State - The `*origin*` Account for the transaction - The `*address*` of the Account for which the context is currently executing - The `*caller*` of the current account (if current execution is happening within an Actor call) - The `*depth*` of the CVM execution stack - The CVM execution log - Any variables locally bound in the execution context - The latest operation Result Register - An Exception value, if an Exception has been thrown For Performance reasons, Contexts are implemented as mutable Objects on the JVM. A complete copy of a Context can however be created cheaply with `Context.fork()`, since the immutable values that the Context refers to can be safely shared by multiple threads / Contexts. ## Ops CVM operations are referred to as "Ops", which represent the fundamental executable code on the CVM. These can be considered as the "bytecode" of the CVM, and are typically produced by compilation of CVM code (which may be performed by either an on-chain compiler or an off-chain tool). CVM Ops are language agnostic - while they might typically be compiled from Convex Lisp source code, alternative language frontends such as Convex Scrypt exist which can produce equivalent Ops. Adventurous hackers are encoraged to experiment with compiling different languages to the CVM. All Ops are defined with a one-byte OpCode that identifies the type of Op, and defines what additional data is associated with the Op. ### Constant ``` Logical Structure: 0xe0 ``` The `Constant` Op loads a single CVM value into the Context's Result Register. ### Invoke ``` Logical Structure: 0xe1 [ ....] ``` The `Invoke` Op recursively executes a sequence of child Ops, and if all these execute successfully invokes the Function provided by the Result of the first child Op, with the results of the following child Ops passed as arguments. The `Invoke` Op must throw a `:CAST` error if the first Op does not return a valid Function. Otherwise, the resulting Context will be the Context produced by execution of the Function. ### Cond ``` Logical Structure: 0xe2 [ .... (optional ElseOp)] ``` The `Cond` Op implements conditional evaluation of child Ops, expreseed as `TestOpX ResultOpX` pairs followed by an option `ElseOp`. For each pair in sequence the `TestOp` is evaluated. If this evaluates to true value, then the result of `Cond` is produced by the corresponding `ResultOp` and no further Ops are executed. If false, execution proceeds immediately to the next pair of Ops. In the case that no test returns true then the result of `Cond` is the result of executing `ElseOp` if it is provided, otherwise a constant result of `nil` is returned. ### Do ``` Logical Structure: 0xe3 [ ....] ``` The `Do` Op implements sequential execution of multiple child Ops. Each child Op is executed in turn. It if succeeds, then execution continues to the next Op. The final result of `Do` is the result of executong the last child Op. In case no child Ops are provided, then `Do` returns a constant result of `nil`. ### Let ``` Logical Structure: 0xe4 [ ....] ``` The `Let` Op allows execution of a sequence of Ops with local bindings ### Loop ``` Logical Structure: 0xe5 [ ....] ``` The `Loop` Op allows execution of a sequence of Ops with local bindings similat to `Let`, except that it additionally serves as a target for `recur` allowing the construction of efficient looping constructs. ### Def ``` Logical Structure: 0xe6 [ ] ``` The `Def` Op defines the value of a Symbol in the current Context's Environment. The parameter (`SymOrSyntax`) MUST be either a Symbol or a Syntax Object containing a Symbol value. This restriction is enforced by Op validation. If a Syntax Object is provided for `SymOrStnax`, metadata from the Syntax Object is stored for the contained Symbol in current Context's Environment Metadata. Otherwise, any existing Metadata is unchanged. If `ValueOp` is `nil`, the definition MUST be created or updated in the environment but the existing value in the environment (if any) will be unchanged. Note that in the compiler, `def` takes metadata from its value argument in the compiler and adds it to the Symbol if provided, hence the subtle difference: ``` ;; defines a Syntax value (def a (syntax 1 {:foo true})) ;; defines the value 1 (with metadata on a) (def b ^{:foo true} 1) ``` The compiler also interprets a `def` with only on argument as having a `ValueOp` equal to `nil`. This is is useful for forward definitions (e.g. as used in the core macro `declare`) ### Lookup ``` Logical Structure: 0xe7 [ ] ``` The `Lookup` Op performs lookup of a value for a Symbol in the current Context's Environment. ### Special ``` Logical Structure: 0xef ``` Where: `` is a byte indicating the special symbol as defined below. Special Ops allow fast access to key values in the current Context, loading these into the Result Register. Special Ops are high performance ways to make certain information in the Context available to CVM Code. #### `0x00 - *juice*` Gets the current Juice count in the Context. #### `0x01 - *caller*` Gets the Caller for the current context, defined as the address of the account that made the enclosing `(call ...)` invocation. `*caller*` is `nil` for top level execution of a user transaction (i.e. there was no enclosing caller). Normally, `*caller*` SHOULD be used to perform access control checks within an actor or smart contract, since it determines which account made the request. #### `0x02 - *address*` `*address*` returns the address of the currently executing account. The Address of the currently executing Account. `*address*` MAY vary within a single transaction in the case where execution control is transferred between accounts, e.g. with `call` or `eval-as`. Normally, `*address*` should be passed as an argument to function that check for access control rights #### `0x03 - *memory*` Gets the current memory allowance (in bytes) for the currently executing Account. #### `0x04 - *balance*` Gets the current CVM coin balance (in copper) for the currently executing Account. #### `0x05 - *origin*` Gets the Address of the origin Account for the current transaction, i.e. the Account that signed and submitted the transaction. Unlike `*caller*`, `*origin*` remains constant throughout the entire transaction regardless of any `call` or `eval-as` invocations. `*origin*` SHOULD generally be avoided for access control (prefer `*caller*`), but is useful for determining the original transaction submitter. #### `0x06 - *result*` Gets the current value of the Result Register. This is the result of the most recently executed operation in the current Context. Useful in macro expansions and advanced metaprogramming. #### `0x07 - *timestamp*` Gets the current State Timestamp as a Long value (milliseconds since Unix epoch). The timestamp is set during Block Preparation and is guaranteed to be monotonically increasing. #### `0x08 - *depth*` Gets the current execution depth of the CVM stack. The depth is `0` at the top level of a transaction and increases with each nested `call` or function invocation. This can be used to guard against excessive recursion. #### `0x09 - *offer*` Gets the current coin offer amount (in copper) available in the Context. The offer is the amount of coins made available by the caller via the `call` form for potential acceptance by the called Actor. #### `0x0a - *state*` Gets the entire current CVM State as a value. This is a large data structure and SHOULD be used with care. Primarily useful for advanced introspection and debugging. #### `0x0b - *holdings*` Gets the holdings map for the currently executing Account. Holdings represent assets or token balances held by the Account in various Actors. #### `0x0c - *sequence*` Gets the current sequence number for the currently executing Account. The sequence number is incremented with each transaction and is used to prevent replay attacks. #### `0x0d - *key*` Gets the public key (Account Key) associated with the currently executing Account, or `nil` if no key is set (e.g. for Actor accounts). #### `0x0e - *juice-price*` Gets the current juice price from the State. The juice price determines the cost of CVM execution in terms of coins per unit of juice consumed. #### `0x0f - *scope*` Gets the current scope value in the Context. The scope is set by `set-scope` within an Actor and provides a mechanism for Actors to pass contextual information across internal function calls. #### `0x10 - *juice-limit*` Gets the juice limit for the current transaction. This is the maximum amount of juice that can be consumed before the transaction fails with a `:JUICE` error. #### `0x11 - *controller*` Gets the controller Address for the currently executing Account. The controller is an Account that has the authority to manage the Account (e.g. update its key or transfer ownership). May be `nil` for self-sovereign Accounts. #### `0x12 - *env*` Gets the current environment map for the executing Account. The environment is a map of Symbols to their defined values, representing the Account's namespace. #### `0x13 - *parent*` Gets the parent Address for the currently executing Account. The parent is the Account from which this Account was created, if applicable. #### `0x14 - *nop*` A no-operation Special that returns the Context unchanged, propagating the current value of the Result Register. This is functionally equivalent to `*result*` but exists as a distinct opcode. #### `0x15 - *memory-price*` Gets the current memory price from the State as a Double value. The memory price determines the cost per byte of on-chain memory allocation. #### `0x16 - *signer*` Gets the Address of the signer for the current transaction. In most cases this is the same as `*origin*`, but may differ in contexts where signing authority is delegated. #### `0x17 - *peer*` Gets the Address of the Peer that submitted the current Block containing this transaction. Returns `nil` if not available in the current execution context. #### `0x18 - *location*` Gets the current location value in the Context. The location provides information about the source position of the currently executing code, useful for debugging and error reporting. ## Op Execution When executed in a given Context, every Op MUST do exactly one of the following: - Complete normally with some resulting value loaded into the Context's Result Register - Throw an Error, which is never caught and results in the failure of the whole transaction - Throw a special Exceptional value, which is handled by the CVM in special ways to implement control flow (`recur`, `return` etc.) ## Memory Management Memory management is a critical aspect of any scalable computational system. The CVM memory management works on the following principles: - On-chain developers never have to worry about memory management. It is fully automatic and transparent. - Memory management costs are properly accounted for in the transaction fees paid by users of the network (either for juice execution costs or via memory accounting). The CVM therefore implements full automatic garbage collection - values which are no longer referenced are automatically discarded from memory without the need for any programmer intervention. ### A note on the efficiency of GC We note that GC is an important prerequisite for high performance in an execution that depends heavily on immutable, persistent data structures. Some reasons for this: - It allows safe structural sharing of values without the need to resort to cumbersome and computationally expensive approaches such as reference counting. - Approaches that are dependent on "ownership" of memory (RAII, Rust-style borrowing) are not effective when there is a need to make multiple, cheap `O(1)` copies of references. - Modern generational GCs are extremely efficient - in may cases better than traditional heap-based allocators ### Further implementation notes - While the CVM specification does not require persistent storage, it is expected that Peers will rely upon persistent storage for CVM Objects. To the extend that CVM values are written to persistent storage in a database, Peers may need to perform a separate garbage collection phase on the database - The current CVM implementation makes use of JVM `SoftReference`s and lazy loading, which allows the host JVM to garbage collect values in many cases even if they are still potentially reachable. This is safe provided that the values can be recovered from storage on demand if required. The advantage of this approach is that it allows the processing of large CVM data structures (such as the State itself) even if these structures exceed the size of available Peer memory. ## See Also - [CAD002: CVM Values](../002_values/index.md) — Value types operated on by the CVM - [CAD003: Encoding](../003_encoding/index.md) — Binary encoding of Ops and values - [CAD004: Accounts](../004_accounts/index.md) — Account model referenced by execution contexts - [CAD006: Memory Accounting](../006_memory/index.md) — Memory management cost model - [CAD007: Juice Accounting](../007_juice/index.md) — Execution cost accounting - [CAD008: Compiler](../008_compiler/index.md) — Compilation of source code to Ops - [CAD010: Transactions](../010_transactions/index.md) — Transaction structure and submission - [CAD011: Errors](../011_errors/index.md) — Error types and handling - [CAD027: Event Logging](../027_log/index.md) — CVM execution log --- # CAD006: Memory Accounting ## Solving The State Growth Problem Decentralised ledgers often encounter an serious issue with *state growth*, defined as the increasing requirement for peers to store information that accumulates over time. Because on-chain data must be retained, potentially indefinitely, in order to satisfy future on-chain queries or smart contract operation, there is no option to discard data arbitrarily: a correct peer cannot do so and maintain correct participation in the consensus protocol. This growing demand for storage space presents a significant problem. - It creates a requirement for peers to incur increasing storage costs over time, for data that must potentially be retained indefinitely if the peer is to be able to fulfil its commitments to provide access to this data to any future transactions. - There are perverse incentives at work: a user might pay a one-off code to store data on-chain, but does not bear the cost of indefinite future storage (which falls on peer operators) - Over time, careless use of on-chain storage may make it impractical for a typical individual to operate a peer with normal computer hardware and storage capabilities - This problem might be manageable for low-transaction-volume platforms, but is clearly unaccaptable for systems such as Convex that are designed to handle high volumes of transactions for sustained periods of time Convex implements a novel solution of Memory Accounting to help manage the problem. - Each user is given a Memory Allowance - Memory Allowance is consumed when on-chain storage is allocated, and released when stored objects are deleted - A common "Pool" of memory is available which limits the maximum size of the on-chain state. - A user may buy additional memory at any time from the Pool, or sell memory back to the pool for Convex Coins ## Memory Accounting Design ### Storage Size Each CVM object is defined to have a "Storage Size" which approximates the actual storage requirement (in bytes) for the object The Storage Size includes: - The size of the encoding of the Cell in bytes - The total size of all child Cells, (e.g. if the object is a data structure) - An allowance for indexing and storage overheads (currently set to a fixed estimate of 64 bytes per complete non-embedded cell) ### Consumption Whenever a transaction is executed on the CVM, Memory Consumption is calculated based on the total impact of the transaction on the Storage Size of the CVM State. Memory Consumption is computed at the end of each transaction, and is defined as: ``` Memory Consumption = [Size of CVM state at end of transaction] - [Size of CVM state at start of transaction] ``` If a transaction has zero Memory Consumption, it will complete normally with no effect from the Memory Accounting subsystem If a transaction would complete normally, but has a positive Memory Consumption, the following resolutions are attempted, in this order: 1. If the user has sufficient allowance, the additional memory requirement will be deducted from the allowance, and the transaction will complete normally 2. If the transaction execution context has remaining juice, and attempt will be made to automatically purchase sufficient memory from the Pool. The maximum amount paid will be the current juice price multiplied by the remaining juice for the transaction. If this succeeds, the transaction will complete successfully with the additional memory purchase included in the total juice cost. 3. The transaction will fail with a MEMORY condition, and any state changes will be rolled back. The User will still be charged the juice cost of the transaction If a Transaction has negative Memory Consumption, the Memory Allowance of the user will be increased by the absolute size of this value. In effect, this is a refund granted for releasing storage. ### Allowance transfers It is permissible to make an allowance transfer directly between Accounts. This is a practical decision for the following reasons: - It enables Actors to automate management of allowances more effectively - It enables Accounts controlled by the same user to shift allowances appropriately - It avoids any need for resource-consuming "tricks" such as allocating Memory from one Account, and deallocating it from another to make an allowance transfer ### Actor Memory allowances All Accounts, including Actors, have a Memory Allowance. However, in most cases Actors have no need for a memory allowance because the allowance utilised will be that of a User account that was the Origin of a transaction. The exception to this is with scheduled execution, where an Actor itself may be the Origin for a transaction. Actor developers may include a capability to reclaim Memory allowances from an Actor (e.g. transferring it to a nominated User Account). This is optional, but without this there may be no way to ever utilise an allowance held within an Actor (either because a scheduled transaction obtained a Memory refund, or because an allowance transfer was made to the Actor). ### Pool trading The Memory Pool employs a simple Automated Market Maker, allowing users to buy and sell Memory Allowances at any time. The Memory Pool liquidity is initially set at network genesis to be: - 1,000,000 bytes of Memory Allowance (~1 MB) - 1,000 Convex Gold Giving an initial memory price of approx. 1 Convex Gold / KB ### Pool Growth The memory trading Pool is automated to grow at a fixed rate (currently 1mb per day). This is for the following reasons: - Creates additional memory supply over time, reflecting improved storage and peer capabilities - Ensures there is no hard memory supply limit that might otherwise prevent future growth - Reduces the incentive for memory hoarding, since it will tend to penalise holders of excess memory allowances ## Incentives ### Incentive to use memory carefully Well designed Actors should avoid allocating too much Memory when called by Users - this will be expected of Actors that wish to offer an efficient service with low transaction costs. ### Incentive to clean up old data Any user who executes a transaction that releases memory will get a refund, so the system creates an incentive to remove data that is no longer useful or important. Most trivially, users have an incentive to clean up their own data in their environment. If there is no longer any requirement for such data to be on-chain, it can be deleted at the user's discretion. This is a relatively safe thing to do if the user has appropriate backups: Data in the user's own environment can always be restored at a later date. More subtly, this system creates a incentive for users to call Actors in ways that cause these actors to release memory in the Actor's environment, and thus gain the benefit of the refund. Some examples: - A messaging system can allow users to delete old messages from their inbox that they have already read / responded to - Tokens could offer a facility to delete unused (zero balance) account records - Users may wish to de-register themselves from registries or subscriptions that they no longer wish to use ### Cross-party incentives There is a slight incentive to being the receiving party in interactions that affect memory: under normal circumstances, the party initiating an on-chain request will have to consume some Memory in order to do so (e.g. creating request data within an appropriate Actor), and the responding party may have the opportunity to perform clean-up when making an appropriate response and thus claim a small Memory allowance refund. We consider this a design advantage: - It can help discourage spamming of unnecessary requests - It rewards parties that respond diligently The ability of receiving parties to do this will of course depend on the facilities that Actors offer to perform clean-up, but we anticipate that this will become an expected feature of well-designed Actors and smart contracts. ### De-duplication The immutable object storage model of Convex means that data objects with the exact same encoding (and therefore the same hashed Cryptographic ID) means that objects in storage need only be stored once, even if the same data value is allocated by multiple users. This provides an incentive for Peers to make efficient use of de-duplication, which is in any case a built-in feature of Convex. Users do not have a particular incentive to ensure that their data is de-duplicated, since they will be required to cover the Memory allowance in all cases. This seems appropriate: users are paying to ensure the retention of the data that they allocate, and the actions of other users should not affect this. Alternatives to this design that were considered include techniques such as reference counting which would allow Users to share the cost of storage that is de-duplicated. However this presents some significant problems: - On-chain reference counts would be computationally expensive to track and maintain - It would mean that the cost of Memory allocations would be unpredictable, based on Memory usage of other users - Accounting for refunds of memory become highly complicated, and can create perverse incentives: If only the last User to release the object gets the refund, there is no incentive to be the first one to do do. ## Technical implementation ### Definition of Storage Size A storage size is defined for each cell (branch or root) is calculated as: `64 + [Size in bytes of encoded cell representation] + [Memory Size of any child Cells]` The constant 64 is chosen to approximate the expected number of bytes of overhead involved in the storage of each root or branch cell. This more accurately reflects the costs to peers to store each cell. This creates a strong incentive to minimise the total number of cells and use embedded values wherever possible (apart from memory size, this is valuable because they are handled more efficiently in general on the CVM). ### Definition of Memory Size In the above definition of storage size, **memory size** for child cells is defined as: - The storage size of any branch cells (i.e. non-embedded references) - Embedded values do not require storage so are considered to have a memory size of zero, however the size of embedded objects will be included in the encoded cell representation of the parent. Since this is a mutually recursive definition, computation of storage / memory size requires the ability to access the memory sizes of any child branches, which in means that the complete cell tree must ultimately be available in the Peer's storage in order to fully validate the complete size. This means that the cell must in general have a status of PERSISTED at minimum, to preclude the possibility of any missing values in nested child cells. ### Lazy computation Memory requirements for a cell are only calculated when required (usually at the point that the state resulting from a transaction is persisted to storage). Implementations SHOULD cache the memory size to avoid re-computing memory sizes on large trees. This minimises the computational costs associated with memory accounting for transient in-memory objects. ### Memory Size Caching and Persistence Implementations SHOULD cache memory size for cells, and persist cached values in storage. In particular Peers MUST cache memory sizes in order to meet CVM performance requirements. Caching and persisting memory size is important to ensure that memory sizes can be computed incrementally without re-visiting the complete tree of cells. This ensures that memory size computation is at worst `O(n)` for a Transaction, where `n` is the number of new cells constructed during the transaction. Implementation note: we cache memory size rather than storage size because storage size is only relevant for root and branch cells, whereas memory size is needed for all cells. ### Memory Accounting impact The memory accounting subsystem is designed so that it always has a minimal effect on CVM state size, even though it causes changes in the CVM state (consumption of allowances etc.). This limits any risk of state growth size from the memory accounting itself. This is achieved mainly by ensuring that state changes due to memory accounting cause no net Cell allocations, at most small embedded fields within existing Cells are updated (specifically balances and allowances stored within Accounts). ### Performance characteristics Assuming correct caching and persistence, memory accounting cost is `O(1)` for each cell allocated, with a very small constant. Benchmarks suggest the overall impact is negligible on CVM performance, and well justified by the upside in creating good economic incentives for efficient storage use (which is itself a net positive performance gain). This would appear to be asymptotically optimal for any system that performs exact memory accounting at a fine-grained level. This achievement is possible because: - The memory size is computed incrementally and cached for each cell. - The number of child cells for each cell is itself bounded by a small constant - Memory size computation is usually lazy, i.e. it is not performed until required - The immutable nature of Convex cell values means that there is never a need to update memory sizes once cached The computational cost of performing this memory accounting is factored in to the juice cost of operations that perform new cell allocations. The storage cost is, of course, factored in to the general economics of the memory accounting model. ## Open Design Questions - It likely that memory accounting will be used to add a per-transaction cost to submitted transactions based on the size of the transaction data. This would incentivise submitting smaller transactions. - There are options regarding on-chain procedures for opening up new storage allowances (which might depend on advances in underlying storage technology). Initial assumption is that this is a Foundation network governance responsibility - there is potentially a need to monitor medium-term state growth and release new allowances accordingly over time. - There is a potential for memory allowance hoarding and speculation, in anticipated of high prices driven by shortages. It may be necessary to set an clear expectation that holding allowances is risky, as new allowances may be added to the Memory Pool at any time which would devalue large allowance holdings. - There is a risk that if memory price become too low, participants may become careless with memory usage. This is mitigated by the fact that on average we expect the Convex state to grow, so large falls in price precipitated by selling allowances is unlikely. This probably requires ongoing monitoring. - Computing allowances at the end of each transaction might cause extraneous storage activity - it would be more efficient to persist CVM state changes in their entirety at the end of each block. Need to investigate whether memory accounting can be efficiently performed in-memory before hitting storage? This is related to the problem of estimating maximum memory requirement per block. - It would be possible for state size changes caused by Memory Accounting itself (i.e. outside the scope of specific transaction effects) to be charged to the Memory Pool. This is unlikely to have a significant impact, so is probably unnecessary. - There is a potential to charge "rent" for total storage allocation over time. This would add significant complexity and potentially cause issues for Actors unable to pay sufficient rent, but might be a useful additional incentive to keep long-term storage requirements low. --- # CAD007: Juice Accounting ## Overview Juice accounting refers to the system of accounting and pricing for transaction processing costs on the Convex network (CPU and networking). We need this system for cryptoeconomic security: without it an attacker could cause the network to execute large amount of pointless work at zero cost and deny service to others. The CVM implements a system of Juice with the following reasons: - Control execution costs on the CVM so that e.g. infinite loops are terminated - Prevent bad actors from flooding the network with pointless transactions (or at least, making it very expensive to do so). - Reward peers with a share of transaction fees to compensate them fairly for the valuable resources they provide to the network ## Juice Fees The Juice Fees are the execution cost component of transaction fees and are computed as: ``` Juice Consumed * Juice Price where: Juice Consumed = Transaction Size Cost + sum(Juice cost of each operation executed) Transaction Size Cost = TRANSACTION_PER_BYTE * Storage Size of Transaction ``` `Juice Price` is a variable defined in the CVM State that varies according to the level of network demand, and is available to CVM code via accessing the special Symbol `*juice-price*`. `TRANSACTION_PER_BYTE` is a global constant pricing the relative cost of transaction storage in the ordering. The value is `20` as of September 2024 but may be updated in future Convex versions if required. The rationale for this component of juice fees is to improve incentives regarding efficient transaction size: we want users and application developers to submit small transactions as much as possible to minimise storage requirements for peers. ## Juice Allowance Each Transaction MUST have a **juice allowance** The juice allowance MAY by specified by the user in the transaction, otherwise the maximum juice allowance shall be assumed (subject to available coins). The juice allowance MUST NOT exceed the ability of the transaction origin account to pay the juice fees for the transaction The juice allowance MUST be limited to a maximum value in order to place an upper bound on computation costs for any single Transaction (currently `10,000,000`) ## Juice Consumed During CVM execution of a transaction, the CVM MUST track juice consumed, by incrementally adding juice cost for each CVM op or runtime function executed. Execution of any CVM OP or runtime function MUST have a fixed positive Juice Cost. Execution of certain CVM ops or Runtime Functions MAY additionally have a variable Juice Cost, where the cost is scaled by the size or the computation requested (typically the size of the largest data structure(s) used in the computation). Juice cost SHOULD be scaled according to the estimated computation time required to execute the corresponding CVM code on a typical Peer. This is not expected to be precise, as real world execution times may vary based on different Peer technology and operational environments. Juice cost MUST be scaled according to an upper bound on the maximum of compute time, storage size or bandwidth cost. This is essentially to ensure that an attacker with `O(n)` resources cannot consume a higher level of resources e.g. `O(n^2)` (which would enable a DoS attack given sufficient scale in terms of `n`). ## Juice Constraints The CVM MUST NOT execute expensive computation with `O(n)` or greater cost without first checking sufficient Juice is available, and throwing a `:JUICE` Error if available juice is insufficient. This prevents attackers from causing `O(n)` computations with less than `O(n)` proven juice committed to pay for it. If a `:JUICE` Error occurs in a Transaction: - The CVM MUST charge the Origin Account of the transaction the cost of the full Juice Allowance - The CVM MUST NOT allow any other changes to the CVM State (other than charging fees related to the Transaction), i.e. all effects are rolled back ## Juice Price calculation The network MUST define a governance constant INITIAL_JUICE_PRICE as a Long value (currently `2`), which is used as the juice price in the genesis state The network MUST define a governance constant JUICE_SCALE_FACTOR as a Double value (currently `1.125`) The network MUST define a governance constant JUICE_PER_SECOND as a Long value (currently `100,000,000`, calculated according to a reasonable lower bound estimate of steady-state CVM execution speed for an average peer). This typically translates to between 10-100,000 transactions per second, depending on transaction complexity, which is comfortable given that the CVM has been benchmarked processing over 1 million TPS. The Network MAY update JUICE_SCALE_FACTOR and JUICE_PER_SECOND as part of a governance update, in which case peers MUST utilise the new values for any State updates on or after the governance update timestamp. Juice price MUST be updated after every block of transactions. Combined with the bounded size of blocks and bounded execution costs on transactions, this ensure a bounded amount of computation can occur between juice price updates. For every `JUICE_PER_SECOND` amount of total juice consumed, the update MUST increase the juice price according to `Juice Price = Juice Price / JUICE_SCALE_FACTOR (rounded up)` For every `1000ms` elapsed the juice update MUST reduce Juice Price according to `Juice Price = Juice Price / JUICE_SCALE_FACTOR (rounded down)` Juice Price MUST have a minimum value of `1`, i.e. Juice consumption is never free. ## Cryptoeconomics The juice accounting system is designed to give effective cryptoeconomics for transaction costs, where the cost of transactions is proportional to the amount of network resources consumed (compute, bandwidth and storage). The variation of juice prices implies: - Juice Price will increase if the network is loaded by more than JUICE_PER_SECOND worth of transactions, which will increasingly discourage Users from submitting lower value transactions - Juice Price will decrease if the network is loaded by less than JUICE_PER_SECOND worth of transactions - An equilibrium level of JUICE_PER_SECOND amount of Juice Consumed per second may be reached, where the Juice Price remains stable. - If the Network has zero load, the Juice Price will decline exponentially towards `1` (with a "half life" of around 6 seconds) Since we assume good peers are able to handle over JUICE_PER_SECOND worth of computation, an attacker would be unable to put excessive load on the network for long periods of time without incurring exponentially rising costs, which makes such sustained attacks infeasible. Alternatively, an attacker could attempt temporary DoS attacks when the Juice Price is low. However, since these cannot be sustained, they would only cause temporary delays to confirmation of final State updates, and would not stop legitimate Transactions from being included in consensus and ultimately executed. --- # CAD008: Compiler ## Overview Convex includes an on-chain compiler as part of the CVM. The compiler is responsible for taking Convex Lisp source code forms and compiling these down to low level CVM Ops. The compiler is designed for Convex Lisp, which is a natural fit the Lambda Calculus features of the CVM. However alternative language front-ends are possible for Convex providing that these are able to compile down to either Convex Lisp (as an intermediate language) or CVM Ops (the basic operations of the CVM) ## Phases The Compiler operates in two phases, expansion and compilation. These phases can be accessed in several ways: - The `eval` runtime function function can be used to perform both phases of the Compiler and execute the result. - The `expand` runtime function can be used to perform expansion alone, typically using the `*initial-expander*`. - The `compile` runtime function can be used to execute the expansion and compilation phase (i.e. without execution). ## Expansion Expansion takes a form (usually defined in Convex Lisp, but can be an alternate language or DSL) and fully expands this to an expanded Form suitable for compilation. Expansion follows the general mechanism outlined in *Dybvig, R.K., Friedman, D.P. & Haynes, C.T. Expansion-passing style: A general macro mechanism* which provides a powerful framework for composable expanders, more powerful and flexible than traditional macro mechanisms. An expanded form may include: - Convex Lisp forms excluding macros and expanders (these should already have been fully expanded) - Syntax Objects, which may be used to annotate certain values with metadata - Raw CVM Ops (already compiled) ## Compilation Compilation takes an Expanded form and converts it into a CVM Op (which typically includes a tree of child Ops), according to the compilation rules specified below. NOTE: Compilation of already compiled CVM Ops is simply the identity function: The pass-though of CVM Ops in this way is useful for implementations of custom languages on the CVM - for example, arbitrary languages can be compiled and executed within an `eval` operation through the use of custom Expanders. ### Overall Form handling - If the input is already a CVM Op, it is returned directly as the output - If the input is a symbol, the symbol is compiled as a symbolic lookup in the current environment (see below) - If the input is an empty data structure, the Output is a Constant Op for the data structure - If the input is a List: - Lists starting with special symbols denoting CVM Ops (e.g. `do`, `def`) are compiled as the appropriate Op, with remaining list elements handled according to the special op definition - List starting with special symbol relating to quoting (e.g. `quote`, `unquote`) are handled according to quoting semantics - Otherwise, the output is an Invoke Op, invoking the compiled first element as a function, with the remaining elements compiled as arguments - If the input is `Map`, `Set` or `Vector`, the output is an Invoke Op calling one of the following constructor functions, with each argument formed by compiling the corresponding element of the data structure in sequential order: - `(vector arg1 arg2 ... argN)` - `(hash-map k1 v1 k2 v2 ... kN vN)` - `(hash-set arg1 arg2 ... argN)` - Otherwise, a Constant Op is produced which returns the Input unchanged NOTE: If construction of a List is required, users should either quote the list or use the constructor `(list arg1 arg2 ... argN)` explicitly to avoid interpretation as a function application. ### Symbolic lookup When compiling a symbol, the following possibilities are checked in turn: 1. If the symbol references a lexically defined value on the stack (e.g. from `let`) then it is compiled as a direct `Local` lookup Op 2. If the symbol references a special op e.g. `*balance*` it is compiled to the appropriate `Special` CVM op (see below) 3. If the symbol starts with `#` a check is performed to see if it matches a core definition such as `#%count`, in which case that value is used as a constant 4. If the symbol references an existing value in the current environment, then: 1. If that symbol is defined with `^:static` metadata flag it is compiled as a constant equal to that defined value 2. Otherwise, a `Lookup` op is created to refer to the exiting definition 5. Otherwise, a dynamic `Lookup` op is constructed to resolve the symbol at runtime ### Special Ops Special Ops are conventionally named with "earmuffs" like `*address*` are are intended to provide efficient access to CVM execution information from the current convex. Developers SHOULD prefer Special Ops over alternate methods of accessing such information, as it is likely to be benefit from special optimisations in the CVM and correspondingly lower juice costs. The Special Ops available are defined in [CAD005](../005_cvmex/index.md) ### Quoting The special symbol `quote` leaves its argument unexpanded and unevaluated. Quote is most useful where developers wish to express CVM data structures (which may include Convex forms) without evaluating them. The compiler MUST compile `quote` forms to a `Constant` CVM op ### Compiler Errors If the Compiler encounters any unexpected input, it MUST return an Error (often `:COMPILE` or `:SYNTAX`). However, Clients SHOULD NOT rely on a specific Error Code, because: - Expanders may produce arbitrary Error Codes - More specific CVM Errors such as `:JUICE` or `:DEPTH` may occur in some cases - Compilation Error Codes may be changed in future Network Upgrades Users making use of the Compiler SHOULD ensure that they avoid error cases as far as possible by ensuring that they only provide correct input. ### Compiler Costs The Expansion Phase incurs a Juice Cost according to the cost of any expanders used (which may be user defined in regular CVM code). The `*initial-expander*` incurs small constant costs for each element expanded. In normal usage, expansion costs are usually `O(n)` in the size of the Form being expanded, but can be arbitrarily large if custom macros or expanders are used. The Compilation Phase incurs a constant Juice Cost for each node of the Expanded Form processed. Compilation costs are therefore never more than `O(n)` in the size of the Expanded Form. Juice Consumed is accounted for in the normal way (i.e. it will be charged for as part of the transaction executing the compiler, and the transaction will fail if any limits are exceeded). --- # CAD009: Expanders and Macros ## Overview The CVM provides the facility for advanced macro capabilities for on-chain code generation. Macros are run at compile time, and can perform arbitrary code transformations. As such, they offer a number of advantages over regular functional code: - They can perform optimisations, e.g. pre-computing certain values at compile time - They can be used to extend the Convex Lisp language with new language constructs that would not be expressible as a regular function ## Key Components ### Forms Forms are code represented as data, in the manner of Lisp, e.g. a List `(+ 1 2)` Where the list contains the Symbol `+` followed by the Long values `1` and `2`. ### Expanders Expanders are functions that transform code as a Convex Lisp form into a replacement form. Expander functions have the signature: `(fn [x e] ....)` Where: - `x` is the form to be transformed - `e` is a continuation expander, i.e. another expander which may be called recursively to expand further forms. Expanders stored in the environment MUST have the metadata `{:expander true}` set in order to be recognised as expanders by the compiler Expansion Phase. Typically an Expander will: - Perform some specialised code expansion - Call the Continuation Expander on the result and / or parts of the result with the pattern `(e z e)` However, Expanders are extremely flexible. It is perfectly possible to create Expanders which vary this logic. e.g. - Changing the Continuation Expander call to `(e z identity)` will cause the continuation expander to expand only once (useful e.g. for debugging and tests) - Omitting any calls to the continuation expander gives this Expander the final say on the resulting form - Using itself as the continuation expander, an expander can implement a custom language. For more information, it is worth referring to the 1988 paper "Expansion-passing style: A general macro mechanism" (Dybvig, Friedman & Haynes) which describes this elegant approach in more detail. ### Macros A Macro is a specialised instanceof an Expander, with the following additional restrictions: - It is declared using the pattern `(macro [arg1 arg2 arg3 ....])`, effectivly destructuring the form passed to the Exapnder into individual arguments. This is often more convenient and intuitive than defining Expanders directly. - A Macro always calls the Continuation Expander as its final step on the result of the Macro. In normal usage, this implements the standard expecation that Lisp macros will be expanded recursively until a final non-macro form is produced. This also means that if a custom language is defined by the continution expander, the macro will continue expansion logic using this same custom langauge. ### Initial Expander The core library defines `*initial-expander*` as an Expander that expands Convex Lisp. Unless explicitly overriden, this will also be the Continuation Expander for all Convex Lisp forms. The standard Convex Lisp Expansion Phase is therefore implemented by: `(*initial-expander* form *initial-expander*)` The initial expander executes the following logic when passed the parameters `[x e]`: - If the Form `x` is a List which starts with an Expander `m` defined in the current Environment, return the result of calling `(m x e)` - Otherwise, if the form is a data structure (List, Vector, Map or Set) then expand each element `y` of the data structure with `(e y e)` and return the resulting data structure. - Otherwise, return the form unchanged ### Implementation Notes - `*initial-expander*` is implemented in optimised Java for performance reasons. It is however possible to implement custom expanders in pure Convex Lisp. - Expanders MAY make use of tail recursion to avoid consuming CVM stack depth on multiple expansions, since the call to the Continuation Expander often occurs in tail position. ## Process In the expansion phase of compilation, a source code form MUST be expanded according to the relevant expander (`*initial-expander*` by default) If the relevant expander is `e` and the source form is `x`, the expanded form is the result of calling `(e x e)` The expanded form SHOULD be a fully expanded form that will successfully compile in the current context, i.e. is valid code that requires no further expansion. ## Examples ### Defining macros Macros can be defined with `defmacro` and are simple expanders that return a transformed form (which may then be subject to further expansion). The example below is a macro that converts a tree-like source form into an arithmetic operation that returns the size of the tree. ```clojure ;; A macro that counts the number of nodes in a nested tree data structure (including the parent) (defmacro tree-size [c] (cond (coll? c) (cons '+ 1 (map (fn [x] `(tree-size ~x) ) c)) :else 1)) ;; Using the macro is simple: just put in operation position (first element of list) (tree-size [1 2 [3 4]]) => 6 ;; this is the full macro expansion of the above example (expand '(tree-size [1 2 [3 4]])) => (+ 1 1 1 (+ 1 1 1)) ;; This is a single stage of expansion (expand-1 '(tree-size [1 2 [3 4]])) => (+ 1 (tree-size 1) (tree-size 2) (tree-size [3 4])) ;; Note this works totally fine on source code forms, which are not executed. ;; This wouldn't be possible with a regular function (you would need to quote the source code) (tree-size (count [1 2 3])) => 6 ``` --- # CAD010: Transactions ## Overview Transactions are instructions to the Convex Network submitted by users. Transactions SHOULD be instructions that a user wishes to have executed and reflected in the global state. Typical examples might be: - Transfer of a digital asset from one account to another - Executing a smart contract - Updating an on-chain database record - Deploying or upgrading CVM code - Voting in a decentralised governance process - Registering a hash that can be used to identify and validate off-chain content (e.g. on the [Data Lattice](../024_data_lattice/index.md) ) The general lifecycle of a transaction is as follows: 1. Client constructs a transaction containing the desired instruction to the network 2. Client signs the transaction using a private Ed25519 key 3. The signed transaction is submitted to a peer of the client's choosing 4. The peer incorporates the transaction into a Belief, which is propagated to the network 5. The transaction is confirmed in consensus according to the CPoS algorithm 6. The peer computes the effect of the transaction on the CVM state, and any result(s) 7. Peer returns a confirmed transaction result to the client ## Transaction Types All signed transactions MUST contain at least the following fields: - An Ed25519 **digital signature**. This field is critical to ensure that the transaction is authorised by the holder of the correct private key - An Address specifying the **origin** account for the transaction. This is the account that will pay any transaction fees, and against which the digital signature will be checked for cryptographic security purposes. - A *sequence number* specifying the order in which the transaction must be run for the origin account. This MUST be *one more* than the total number of transactions executed for the origin account so far. i.e. the first sequence number accepted will be `1`, the next `2` etc. This field is critical to prevent replay attacks. ### Transfer A `Transfer` is a transaction requesting the transfer of Convex Coins from a user (origin) account to some other (target) account. A transfer transaction MUST specify an amount to transfer, as an integer. The source Account MUST be the origin account for the transaction, i.e. transfers can only occur from the account which has the correct digital signature Both accounts MUST be valid, otherwise the transaction MUST fail The transaction MUST fail if any of the following are true: - The source Account has insufficient balance to pay for Transfer Transaction fees. - The transferred Amount is negative - The transferred Amount is greater than the Convex Coin Balance of the source Account (after subtracting any Transfer Transaction Fees) If the transfer transaction does not fail for any reason, then: - The Amount MUST be subtracted from the Source Account's Balance - The Amount MUST be added to the Destination Account's balance A transfer amount of zero will succeed, though this is relatively pointless. Users SHOULD avoid submitting such transfers, unless there is a good reason (e.g. public proving the ability to transact with a given account). ### Invoke An `Invoke` transaction is a request to execute some CVM code by a user account. This is the most general type of transaction: any CVM code may be executed. An Invoke transaction causes the execution of CVM Code when successfully signed and submitted to the Convex network An Invoke Transaction MUST include a payload of CVM Code. This may be either: - A pre-compiled CVM Op - A source code form that will be compiled and executed, as if using `eval` High volume users MAY consider pre-compilation of CVM code to avoid additional compilation juice fees. An Invoke transaction MUST fail if: - The CVM Code is not valid for execution (e.g. a syntax error in compilation) - The Origin Account has insufficient balance to pay for Juice required by the code execution - The execution of CVM Code causes any Error (e.g. a `:TRUST` Error caused by attempting an unauthorised operation on an actor) Otherwise, the CVM State MUST be updated by the result of executing the CVM Code for the Origin Account ### Call A `Call` is a transaction requesting the execution of a callable function (typically a smart contract entry point) from a user account. Semantically, this is broadly equivalent to using an `Invoke` transaction to do the following: `(call target-address (function-name arg1 arg2 .... argN)` `Call` transaction types are mainly intended as an efficient way for user applications to invoke smart contract calls on behalf of the User. A Call Transaction causes the invocation of an Actor function. Apart from lower transaction fees, the Call instruction MUST be functionally equivalent to invoking CVM Code of the form: `(call actor offer (actor-function arg1 arg2 .....))` Call Transaction are primarily intended for efficient execution of Smart Contract functionality by Clients that need to utilise the functionality of a specific Actor. ## General Handling ### Construction Clients MUST ensure the transaction is correctly constructed according to one of the transaction types defined in this CAD Clients MAY delegate transaction construction to another system. If this is done, care should be taken to ensure the transaction is correctly constructed (e.g. does not contain malicious code that has been inserted by a hacker) Clients MUST ensure that the sequence number on the transaction is correct. Failure to do so is likely to result in immediate transaction rejection, and possible blacklisting by Peers. Clients MUST ensure that they sign the transaction with the correct private key for the origin account. Failure to do so is likely to result in immediate transaction rejection, and possible blacklisting by Peers. ### Results Transaction results MUST be returned in a `Result` record which contains the following fields: - `:id` - the message ID of the transaction to correlate with the client - `:result` - the final result of the transaction (will be the error message if an error occurred) - `:error` - the error code (MUST `nil` if no error occurred, otherwise can be any Keyword) - `:log` - a vector of log entries created (may be omitted if no logs events occurred) - `:info` - a Map of information reported by the peer to the client, which SHOULD include: - `:tx` - the 32-byte SHA3-256 hash of the signed transaction - `:loc` - the location of the transaction in consensus, as a vector `[block-index transaction-index]` - `:trace` - an error trace, which is a vector of stack messages if an error occurred - `:eaddr` - the Address of execution where the error was raised - `:mem` - Integer amount of memory consumed by the transaction (may be omitted if zero, may be negative for a refund) - `:juice` - Execution juice for the transaction - `:fees` - Total fee paid in Convex coppers, including memory cost - `:source` - The source location at which the Result was generated An an optimisation, peers MAY avoid creating `Result` records if they have no requirement to report results back to clients. ### Fees Total fees for a successful transaction are calculated as: ``` (juice used + transaction base cost) * juice price + memory costs ``` See [CAD007](../007_juice/index.md) for more details on juice cost calculation. See [CAD006](../006_memory/index.md) for more details on memory cost calculation. Memory costs MUST be zero if no memory was used, or if the origin account had sufficient memory allowance to cover the increase in state size caused by the transaction. In the case of a failed transaction, memory fees MUST be zero (since state changes are rolled back) If a transaction failed signature or sequence verification, the base transaction cost is paid by the peer that submitted the erroneous transaction. ### Verification If the client trusts the peer, the returned result may be assumed as evidence that the transaction has succeeded. If there are doubts about the integrity of the peer, further verification may be performed in several ways: - Checking the consensus ordering to ensure that the transaction occurred when the peer claimed - Querying the CVM state to ensure transaction effects have been carried out - Confirming the result with one or more independent peers It is generally the responsibility of the user / app developer to choose an appropriate level of verification and ensure connection to trusted peers. ## Peer Responsibilities Peers are generally expected to be responsible for validating and submitting legitimate transactions for consensus on behalf of their clients. Peers MAY define their own terms for allowing a client to use their services. Peers SHOULD submit legitimate transaction for consensus, unless they have a reason to believe the transaction is invalid or illegitimate. Peers SHOULD submit transactions in the order that they are received from any single client. Failure to do so is likely to result in sequence errors and potential economic cost for the peers. Peers SHOULD validate the digital signature of transactions they include in a block. Failure to do so is likely to result in penalties (at a minimum, paying the fees for the invalid transaction) Peers MAY reject transactions that do not appear to be legitimate, in which case the Peer MUST return a Result to the Client submitting the transaction indicating the reason for rejection. Some examples where this may be appropriate: - Any transaction that has an obviously invalid sequence number (less than that required for the current Consensus State) - A transaction that has a future Sequence number (greater than would be valid for the current consensus), and the Peer is unaware of any previous in-flight transactions from the Client that would make this valid. - A transaction that appears too expensive for its origin account to execute (a very large transaction size or a large transfer that would be likely to fail) - An account or client has been blacklisted by the Peer for previous bad behaviour ## Signatures All valid transactions MUST be signed by the account key of the account for which the user submits the transaction. The relevant account key is the one that is set in the CVM state at the time the transaction is executed. --- # CAD011: Error Handling Error handling is a critical feature of all good code, and therefore requires special consideration on the CVM. ## General Design ### Expression behaviour Every expression MUST do one of three things: - Succeed with a valid CVM value as the `*result*`. - Fail with an error as described in ths CAD. - Perform an exceptional exit, e.g. because of a nested CVM `return` function. In these cases, control is always returned to some higher level expression. ### Error Codes All errors are defined to have a non-nil error code that describes the general nature of the error. The error code SHOULD provide information regarding the type or cause of the error, in a way that may be interpreted appropriately by clients. Error codes SHOULD be upper case keywords (e.g. `:ASSERT`) by convention. The CVM itself MUST follow this convention, though this is not enforced in user code and alternative CVM types MAY be used. ### Error Messages All errors are accompanied with a message, which may be any CVM value (including `nil`). Error messages are returned alongside the error code when an error is thrown, with the intention that this can be relayed to clients if the error is not otherwise caught and handled. Messages SHOULD be meaningful and human readable to facilitate debugging or appropriate notification to users. If not otherwise handled, the contents of the message SHOULD be returned to clients by peers for informational purposes. ### Try / Catch Most errors can be caught and handled within CVM code (`:JUICE` errors are a notable exception, because they are unrecoverable by any code). Errors can be caught and handled with the `try` construct: ```clojure (try (do-something-that-might-fail) (do-something-else-in-failure-case)) ``` This construct has several notable features: - Sub-expressions are executed in turn until the *first one that succeeds* (completes without an error). - Each sub-expression is *atomic* - either it succeeds, or in the case of an error, the whole sub-expression is rolled back. This is important protection to ensure that code causing an error does not result in inconsistent state from partially completed operations. - The whole `try` expression can only fail if the last sub-expression fails (or an uncatchable error like `:JUICE` is thrown) NOTE: Originally catching errors was not allowed in the CVM, because of the fears that the security and integrity of smart contracts would be at risk if error recovery was mishandled. This risk is largely mitigated by the "rollback" behaviour implemented in the `try` construct. ### Errors vs. other exceptional exit An error represents a failure in code execution that could either be handled or reported back to the user. Other exceptional exits are where an expression never completes normally but there is no error e.g. `return` from a function or `rollback` of state changes in an actor call. These special cases are generally caught and handled at appropriate points in the CVM implementation, and are not normally visible to external users. ## Mechanics ### Throwing errors An error is said to be "thrown" when the execution of a CVM operation produces and error rather than a normal result. There are two possibilities for this to occur: - The error is thrown according to CVM runtime execution rules, for example during the execution of `convex.core` runtime functions. - The error is thrown by user code explicitly calling the `fail` Runtime function. If an error is thrown, normal execution is terminated and no normal result is produced by the CVM operation. ### Error handling ## Standard Error Codes The following are standard Error Codes that are recommended for use in the CVM. User code SHOULD follow these conventions where possible. ### `:ARGUMENT` An `:ARGUMENT` error SHOULD be thrown whenever a function is passed an argument that is of an allowable type but for some reason is invalid (perhaps in relation to other arguments). Examples: - Attempting to put `assoc` a non-Blob value into a Index (which only accepts Blob-like values as keys). - Attempting to cast a value that is out of the allowable range (e.g. `(long 1e100)`) If the argument is definitely of the wrong type (i.e. would never be valid in any situation) then a `:CAST` error should be thrown instead. ### `:ARITY` An `:ARITY` error SHOULD be thrown whenever an attempt is made to call a Function with an illegal number of arguments. Note that a function may allow a variable number of arguments with a parameter declaration such as `[a & more]`. In such cases, code SHOULD still throw an `:ARITY` Error if the number of variable arguments is impermissible for any reason (e.g. requiring an odd number of arguments) ### `:ASSERT` An `:ASSERT` error MAY be thrown whenever a precondition for some code is not satisfied. In many cases, a more specific error message may be appropriate or informative (e.g. `:CAST` or `:STATE`). The Core function `assert` throws an `:ASSERT` error if any of its conditions evaluates to `false` ### `:BOUNDS` An `:BOUNDS` error SHOULD be thrown whenever an attempt is made to access an indexed element of a countable collection when the index used does not exist. This Error is useful because it is more specific than `:ARGUMENT` when working with indexed collection. ### `:CAST` A `:CAST` error SHOULD be thrown whenever a function is passed an argument that is of the wrong type. In particular, a `:CAST` Error MUST be thrown whenever an attempt is made to convert a value to a different type using a CVM runtime function, but the conversion is not permitted for any member of the type. ### `:NOBODY` A `:NOBODY` Error should be thrown whenever an attempt is made to access an Account that does not exist. ### `:SYNTAX` A `:SYNTAX` error indicates a syntax error in code, typically during expansion or compilation. ### `:STATE` A `:STATE` error SHOULD be thrown when an operation is attempted that would possibly be legal, but fails in the current situation because of some information in the current CVM State not permitting it. ### `:TODO` A `:TODO` error SHOULD be thrown by code that is not yet complete, but may be later upgraded to full functionality. This error is probably most appropriate during development and testing, but could plausibly be used in production code that is designed to be upgraded at a later date. ### `:TRUST` A `:TRUST` error SHOULD be thrown when an operation is attempted that is not permitted due to security or access control conditions. Typically, this would indicate an attempt to perform an action that the user is not allowed to perform. ### `:FUNDS` A `:FUNDS` error SHOULD be thrown when an operation is attempted that fails because an Account has insufficient balance of Convex Coins (or another digital asset) to afford the operation. ### `:MEMORY` A `:MEMORY` error SHOULD be thrown when an operation is attempted that fails because an Account has insufficient Memory Allowance to complete the operation. ### `:JUICE` A `:JUICE` error MUST be thrown with the source `:CVM` if the `*origin*` account of the currently executing code has insufficient Convex Coin balance to pay the required Juice costs. A `:JUICE` error from the CVM MUST NOT caught: it would be pointless because any error handling code would not be able to execute due to lack of juice. User code MAY throw a `:JUICE` error to indicate that an infeasibly expensive CVM operation was attempted. ### `:UNDECLARED` An `:UNDECLARED` error SHOULD be thrown whenever an attempt is made to lookup a symbol in an account's environment that is not defined. ## CVM Behaviour The CVM itself is subject to additional rules on how and when it creates errors. Clients MAY assume that the CVM behaves consistently according to these rules, But SHOULD NOT assume that a particular error has been generated by the CVM itself, since user code may produce similar errors. ### Core Runtime functions Core runtime functions MUST throw an `:ARITY` error if an invalid number of arguments is passed to a function. Otherwise, core runtime functions MUST throw a `:CAST` error when an argument of the wrong Type is provided, or if an explicit cast function such as `blob` fails. Otherwise, core runtime functions MUST throw a `:NOBODY` error when an attempt is made to access an Account that does not exist. Otherwise, core runtime functions MUST throw an `:ARGUMENT` error when an argument of an allowable Type is provided, but the specific value is not permitted. Otherwise, core runtime functions MUST throw an error if their execution causes any CVM code to be executed that in turn causes an error. CVM functions MAY, in certain defined cases, alter the error code or message to provide additional information. Otherwise, core runtime functions MUST NOT throw an error. ### Fatal Failures If the CVM encounters any condition that should not be legally possible during CVM execution (typically caused by a host runtime exception), it should regard this condition as a Fatal Failure The CVM MUST report a `:FATAL` error with a source of `:CVM` if any fatal failure occurs. The CVM MUST NOT interpret an error thrown by user code as a fatal failure, but such errors MAY still have the `:FATAL` error code - however the source MUST be `:CODE` in such cases. A peer that encounters a fatal error has a serious problem. Hardware failure, bugs in the CVM implementation or resource limitations of the host environment are all possibilities, all of which may cause the peer to fail to correctly compute the updated CVM state in consensus. The peer MAY attempt the following resolutions: - Retry the CVM execution, to see if it can recover from a transient error - Re-sync with other peers that may not have encountered the failure Otherwise, peers SHOULD shut attempt to shut down gracefully to prevent risk of loss (e.g. stake slashing) from failing to maintain consensus. Peers MAY choose to act as a client of another peer to submit transactions as part of this process, for example to withdraw peer stake. ### Transaction handling When attempting to execute an externally submitted transaction, the CVM MUST throw one of the following errors if a failure to execute the transaction occurs. #### `:NOBODY` If the transaction is submitted with an address that does not refer to an account in the current CVM State, the CVM MUST throw a `:NOBODY` error. In this case, the CVM MUST NOT execute the transaction. This is necessary to prevent the risk of DoS attacks using non-existent accounts. #### `:SEQUENCE` The CVM throws a `:SEQUENCE` error in the case that the sequence number of a transaction is invalid for the origin account that it is submitted for. The only correct sequence value is the integer which is the current `*sequence*` plus one. This protection is necessary to stop replay attacks (multiple executions of the same transaction cause by an adversary re-submitting it). If the transaction is submitted for a valid account but has the wrong sequence number for the account (i.e. it is not the next Sequence Number), the CVM MUST return a `:SEQUENCE` Error with the source code `:CVM` In this case, the CVM MUST NOT commence execution of the transaction. #### `:STATE` The CVM MUST throw a `:STATE` error in the case that the origin account does not have a `*key*` defined, i.e. it is a library or actor account. No external transaction can be executed for this account unless a `*key*` is subsequently assigned. #### `:SIGNATURE` The CVM MUST throw a `:SIGNATURE` error if the digital signature for the transaction is not valid given the public `*key*` of the origin account. Clients MAY consider this error as a hint that the user has used the wrong key pair, and take action accordingly, e.g. prompting for the correct key pair. ### General Advice #### Consider not catching errors Even with rollback of failed expressions, error handling adds complexity that may not be necessary: the default behaviour of the CVM is to fail and roll back the entire transaction if an error is not caught. This is often the safest thing to do: attempting to catch and recover from an error may create security risks. This is especially true if the expression contains a `call` to potentially untrusted code: attackers may be able to get that code to throw an error in order to exploit a vulnerability. #### Use precondition checks It is always worth considering checks on preconditions before executing expressions that might fail. If a pre-condition is not met, it may then be appropriate to `fail` immediately with an informative reason: this often allows better error messages to the user relevant to the context. ## Error Sources Error sources indicate the region in the network where an error occurred. These are important mainly because they can indicate responsibility for failure and/or or how to diagnose the problem. All error results SHOULD include a source code to indicate the source of the error. | Source Code | Location of error | Example(s) | ------------ | -------------------------------- | ----------- | `:CLIENT` | Client library code | Failed input validation | `:COMM` | Client-Server communications | IO failure, connection failure, timeout | `:SERVER` | Server handling of request | Bad request format, server error, server load | `:PEER` | Peer handling of user request | Rejected for bad signature detected by peer | `:NET` | Consensus network | Transaction failed to get into consensus | `:CVM` | CVM state transition handling | Invalid sequence number, `:JUICE` error | `:CODE` | CVM code execution | `:CAST` error in user code Error sources are not formally part of the Convex Network / CVM specification, but are important additional information normally returned alongside transaction results. Be aware that a malicious peer could fabricate the error source, so it may be useful to independently validate results. --- # CAD012: Numerics ## Overview The CVM implements numerical operations similar to those available in typical programming languages. Most basic numerical operations are provided as runtime functions such as `+`, `sqrt`, `max` etc. CVM numerical capabilities are designed to support the kind of calculations likely to be required in on-chain decentralised applications. Floating point vales are fully supported on the CVM. This is because we believe floating point numerics are often important for functionality such as: - Probability calculations in prediction markets - Exponential growth rates - Dividing up shares in asset pools - Computing statistics such as moving averages However, compute intensive numerical operations should normally be performed off-chain. Hence we do not offer any special support in the CVM for operations such as tensor multiplication. You can do these on your GPU, and summarise or hash the result to put on chain if required. ## Numeric Tower The numeric tower is defined as follows: - Double (64 bit double precision floating point) - Integer, which can be divided into: - Big Integer (65-32768 bits) - Long (64 bits or less) ### Double Doubles are 64-bit double precision floating point values as defined in the IEEE754 standard. Certain special values are supported: - `##Inf` is positive infinity (hex 0x7ff0000000000000) - `##-Inf` is negative infinity (hex 0xfff0000000000000) - `##NaN` is a unique "not a number" value (hex 0x7ff8000000000000) - `-0.0` is the IEEE754 negative zero (hex 0x8000000000000000) - `1.7976931348623157e+308` is the maximum value (hex 0x7fefffffffffffff) ### Integers Integers are signed integer values. The CVM natively supports big integer values: Currently the maximum size of an Integer is 4096 bytes (32,768 bits). This MAY be extended in the future. Longs are 64-bit signed integer values. They are the subset of Integers that fit within 64 bits, which is convenient for implementation on most modern hardware. Usually, developers don't need to worry about whether they are using Longs or "big" Integers. The CVM converts between the two automatically as required (preferring the more efficient Long values where possible). However, staying within Long range is recommended because it usually results in lower juice costs / transactions fees. ## Numeric Equality Numerical equality can be checked with the `==` Function. This differs from value equality (`=`) in several ways, and is consistent with the IEEE754 specification. In particular, Users should note that: - `(== ##NaN ##NaN) => false` - `(== 0.0 -0.0) => true` - `(== 5 5.0) => true` In general: prefer `==` if you want to compare values on a numerical basis, `=` if you want exact value / representational equality (e.g. for consistency of hash values). ## Numerical functions The CVM has built-in support for a wide range of functions which operate on numerical values. Some examples: ```clojure ;; Regular addition. Supports any number of arguments (+ 2 3 4) => 9 ;; Subtraction (- 100 30) => 70 ;; Negation (subtraction with just one argument) (- 17) => -17 ;; Integer division (div 1234 100) => 12 ;; Integer modulus (mod 1234 100) => 34 ``` Some numerical functions are defined to always result in `Double` values: ```clojure ;; Note: Numerical division results in doubles. Use `div` or `quot` if you want integer division! (/ 1 2) => 0.5 ;; Square root (sqrt 16) => 4.0 ;; Exponential function (exp 1) => 2.7182818284590455 ``` There are also several predicates designed to work with numerical values: ```clojure ;; zero? is equivalent to numerical comparison with any zero value (zero? -0.0) => true ;; long? tests if a value is an integer in the long range (64 bits) (long? 6786) => true ;; number? tests for any numeric type (number? 0.12) => true ``` Where an equivalent IEEE764 function exists using double-precision floating point values, the CVM behaves the same as the IEEE754 standard. ## Numeric Comparison Numerical comparisons can be performed with `<`, `>`, `<=`, `>=`. These generally behave in the expected way, support variadic arguments and are consistent with `==`. Examples are given below: - `(< 1 2 3) => true` - `(< 1 3 2) => false` - `(>= 3 3) => true` ## Numeric promotion Many numerical functions promote to the largest number type used in the sequence Integer -> Double, e.g. the `+` function: ```clojure ;; Using Integer arguments results in an Integer (+ 10 20) => 30 ;; Any Double argument causes the result to be Double (+ 1 1.2) => 1.2 ;; Functions using Long values that overflow Long range automatically promote to full (big) Integers: (* 9999999999999 9999999999999) => 99999999999980000000000001 ``` Use of a non-numeric type in a numerical function usually results in a `:CAST` error: ```clojure (+ 1 0x00) ``` ## Conversion to Numeric Types Certain types can be explicitly cast to numeric values the using runtime casting functions `(long x)`, `(int x)` and `(double x)` If these functions succeed, it is guaranteed that the result will be a value of the expected numeric type. It is worth noting some of the more esoteric conversions to Integers possible: ```clojure ;; Characters convert to their Unicode code point (int \c) => 99 ;; Hex values are treated as two's complement integers (sign extended if necessary) (int 0x00ffff) => 65535 ;; Booleans convert to 1 / 0 (int true) => 1 ;; Doubles convert to integers (rounding towards zero, like Java casts) (int -2.7) => -2 ;; Can get the numerical value of an address with `int` or `long` (int #1234) => 1234 ``` ## General Recommendations Users SHOULD perform checks on the validity of numeric arguments before executing numerical operations. In particular: - Be aware of the possibility of `##NaN`, `##Inf` or `##-Inf` as a result from Double operations - Check for risk of 64-bit overflow if relying upon Long values. - Be careful accepting arguments from untrusted sources and casting them directly to numeric values. You might get unexpected results. If in doubt, check arguments with predicates like `int?` first. Users SHOULD NOT use the CVM for complex or intensive numerical computations, e.g. modelling fluid dynamics. Such computations are better performed off-chain. --- # CAD013: Metadata ## Overview The CVM provides a powerful facility for attaching a metadata map to a defined symbol. The purpose of metadata is to provide any information on a defined symbol independent of the value this symbol holds. Some metadata information, if specified, MUST follow a particular structure for external purposes. For instance, this CAD describes how to document a symbol by following an expected structure. Other information in metadata CAN follow any arbitrary structure. Creators of such SHOULD collaborate with the community to establish useful standards where appropriate (which may include updates to this CAD) ## Usage ### Declaring metadata Metadata is a map containing any arbitrary set of key-values. It is specified after the defined symbol, prefixed with `^`: ```clojure (def some-symbol ^{:my ["meta" :data]} 42) ``` ### Retrieving metadata The core library defines a function for retrieving metadata: ```clojure (lookup-meta 'some-symbol) ;; Following previous example, returns `{:my ["meta" :data]}` ``` Each account stores a map of `symbol` to `metadata map` under `:metadata`: ```clojure (get (account *address*) :metadata) ;; Following previous examples, returns `{'some-symbol {:my ["meta" :data]}}` ``` ## Standard structures For some purposes, metadata must follow at least a set of contraints so that the CVM or any external consumer has access to relevant information. ### Documentation Attaching documentation to a symbol allows any user to gain insight about the purpose of that symbol. Notably, it can provide information regarding how a function is intended to be used and why. User MAY specify any information as described below. None is mandatory. When specified, structure MUST conform as expected by any external consumer. A documentation map MAY be attached to metadata under `:doc`. Reference example: ```clojure (defn add ^{:doc {:description "Adds 2 numbers together." :errors {:CAST "If an argument cannot be cast to a number"} :examples [{:code "(add 2 3)"}] :signature [{:params [a b] :returns Number}] :type :function}} :implementation...) ``` Documentation map MAY contain any of the key-values described in the following subsections. #### `:description` String describing the symbol in human language . TODO. Also a vector of strings where each string is a paragraph for easier formatting? #### `:errors` Assuming the symbol is callable, map of `error code` to `string` describing in human language how and why calling this function might fail. See also [CAD 011](../011_errors/index.md) about errors. #### `:examples` Assuming the symbol is callable, vector of examples where an example is a map which MAY contain: | Key | Value | |---|---| | `:code` | String, excerpt of code demonstrating a function call | TODO. Should it be a string? Is it for memory issues? TODO. Should it also have a `:return` key-value? #### `:signature` Assuming the symbol is callable, vector of signatures where a signature is a map which MAY contain: | Key | Value | |---|---| | `:params` | Vector of parameters | | `:returns` | Type of returned value | TODO. Is there a list of supported types for `:returns`? More future-proof to have a human-readable string? #### `:type` Keyword designating what category the symbol belongs to: | Keyword | Meaning | |---|---| | `:function` | Symbol is a regular function | | `:macro` | Symbol is a macro | TODO. What about other values? ### Expanders As described in [CAD 009](../009_expanders/index.md), expanders MUST have at least `{:expander true}` in their metadata. TODO. Use `:expander?` for consistency. TODO. Example here? Or rather in CAD009? TODO. Other sections besides documentation and expanders? --- # CAD014: Convex Name System The Convex Name System (CNS) is a globally accessible naming service available on the CVM. It performs the important function of providing trusted names for user accounts and other services on the CVM, and acts as a root of trust for the broader ecosystem of lattice technology services. CNS provides trusted mutable references to resources via user-friendly names. When you resolve `convex.trust`, you get the CURRENT trusted implementation, not a frozen historical version. ## Motivation Similar to DNS on the Internet, there is a need for a naming system which provides: - Meaningful human readable names - A mapping to resources on the network - Trusted and secure facilities for managing this information This is intentional: it allows ecosystem-wide upgrades without requiring modifications to source code or changing stored identifiers. ## User API The CNS user API is a library implemented as standard by the `*registry*` (account `#9`, or `@convex.registry`). Users need not use this library to access CNS (alternative implementations are possible, as is direct usage of the CNS SPI), but it is the canonical way to do so and is recommended in most cases. ### `resolve` The resolve function gets the value referred to by a CNS name. This is the most fundamental CNS function. ```clojure (@convex.cns/resolve 'convex.asset) => #18 ``` This is equivalent to using the core macro `resolve`: ```clojure (resolve convex.asset) => #18 ``` And is also equivalent to just using the resolution symbol `@` directly: ```clojure @convex.asset => #18 ``` The resolve function returns `nil` if the CNS name referred to does not exist. ### `read` The `read` function is similar to `resolve`, but returns the entire CNS record as a `[value controller metadata child]` vector (or `nil` if the record does not exist): ```clojure (@convex.cns/read 'convex.asset) => [#18 #6 nil nil] ``` ### `create` The `create` function creates or overwrites a CNS record at the given path, subject to authorisation (see Authority model below): ```clojure (@convex.cns/create 'my.actor.name target-value) (@convex.cns/create 'my.actor.name target-value controller metadata) ``` Any intermediate CNS nodes required by the path are created automatically. Where record values are not specified, existing values are preserved (or defaulted for a new record, with the controller defaulting to `*address*`). ### `update` The `update` function changes the value of an existing CNS record, preserving its controller, metadata and child node. It fails with a `:STATE` error if the record does not exist. ```clojure (@convex.cns/update 'my.actor.name new-value) ``` ### `control` The `control` function changes the controller of an existing CNS record: ```clojure (@convex.cns/control 'my.actor.name new-controller) ``` Note: this changes the *record controller* only. It does not transfer ownership of any child CNS node associated with the record — see Authority model below. ## Specification ### CNS Base Address CNS is defined to have a specific base address, the account `#9` in the genesis state, which is also specified by `*registry*`. This is the actor from which all standard CNS lookups begin. ### CNS Records A CNS record is a logical entry in CNS. A CNS record MUST contain the following logical values: - **Value** : The entity referred to by the CNS record, which SHOULD be an address or a scoped reference e.g. `[#45 :some-key]` - **Controller** : The controller that enables access to update the CNS record, which SHOULD be a valid trust monitor - **Metadata** : Optional metadata attached to the CNS record - **Child** : Optional CNS node that handles child namespaces (typically a scoped address) Any of the logical values MAY be `nil`. The standard registry represents a record as a 4-element vector `[value controller metadata child]`. Alternative node implementations SHOULD use the same representation for interoperability. ### CNS Paths Records in CNS MUST be specified by a path. Paths can be represented in at least two forms: - A dot delimited symbol e.g. `convex.asset` - A vector of strings e.g. `["convex" "asset"]` Normally the user API makes use of symbols, while the CNS internal SPI uses vectors of strings. The rationale for this is: - Dot delimited symbols are preferred for conciseness and human readability - A vector of strings is more convenient and efficient for programmatic manipulation and internal database representation. #### Segment syntax Path segments are strings. Constraints on segment names are **node-implementation-defined**: each CNS node determines what names it accepts (e.g. a node selling user names may enforce a restricted charset and minimum length; the standard registry currently accepts any string). However, since the canonical user API addresses records via dot-delimited symbols: - Segment names SHOULD be non-empty strings that round-trip through the symbol representation (i.e. contain no `.` characters and form a valid symbol when joined with `.`). - Node implementations SHOULD reject segment names that cannot be expressed as part of a dot-delimited symbol, since such records are unreachable via the standard user API and standard tooling. ### CNS Nodes CNS nodes are elements of the CNS tree that maintain mappings of names to records. Typically, a CNS node stores a map of names to records, but different implementations MAY implement such mappings in any way they choose, including creating mappings dynamically. A node is identified by a scoped reference, e.g. `[#9 ["convex"]]` — an actor address plus an implementation-defined key (the standard registry uses the full path vector as the key). Note: there is a correspondence between CNS records and nodes: each CNS record may optionally specify the CNS node (Child) that represents any children. ### Authority model CNS separates two distinct authorities, **by design**: - The **record controller** (element 1 of a record) — a trust monitor governing changes to *that record*: updating its value, metadata or child link, and transferring record control. - The **node owner** — a trust monitor governing the *namespace structure* of a node: creating and deleting entries within it, creating child nodes, and transferring node ownership. The two usually start out identical (creating a path sets both to the same controller) but MAY legitimately diverge. This supports **delegation**: a namespace owner can hand out records (names) to third parties, who control their own record values, while the namespace owner retains authority over the namespace itself — including the ability to revoke (delete) entries. Consequences that users MUST understand: - Transferring a name with `control` transfers the record only. The namespace *under* that name (the child node, if any) remains with its current owner. A complete handover of a name and its subtree is therefore **two** capability transfers: 1. `(@convex.cns/control 'my.name new-controller)` — transfers the record; 2. `(trust/change-control [#9 ["my" "name"]] new-owner)` — transfers ownership of the child node. - Conversely, receiving control of a record does NOT grant authority to manage names beneath it. The node owner retains that, including deletion rights over records in the node. - Subtree nodes are independently owned. A node MAY be referenced as the child of more than one record (shared subtrees), so deleting a record or node MUST NOT recursively delete nodes beneath it: those belong to their respective owners, who are responsible for deleting them. Trust monitor actions used by the standard registry: | Action | Checked against | Meaning | |--------|----------------|---------| | `:update` | record controller | update an existing record | | `:create` | node owner | create a new entry (record or child node) in a node, with the segment name as object | | `:delete` | node owner | delete an entry in a node, with the segment name as object | | `:control` | node owner | transfer node ownership | ### CNS Root The CNS root is the CNS node indicated by the empty path `[]`. As such, it acts as the parent for all root namespaces (e.g. `convex`). The CNS root SHOULD NOT be modifiable by any accounts other than those controlled by the governance body. Initially both root node ownership and root record creation are restricted to the network governance account `#6`. ### Root namespaces The following root namespaces exist in the genesis state: - `convex` - reserved for approved Convex libraries and tools. Typically these SHOULD be fully specified with a CAD (e.g. the `convex.asset` API as specified in CAD19). Governance approval required. - `asset` - reserved for digital asset implementations. Names will be allocated subject to community / governance approval and/or audit. - `torus` - reserved for validated open source marketplaces and trading contracts such as the Torus DEX. Community approval required. - `currency` - reserved for fungible currency tokens (initially the genesis test currencies, managed under network governance). - `init` - a root record referring to the network initialisation account `#1`. The following root namespaces are planned but not yet provisioned: - `user` - intended for developer user accounts. Open access, names / subdomains are assets obtainable for a fixed Convex Coin cost. - `id` - reserved for decentralised IDs. Open access, names are assets obtainable for a small Convex Coin cost. - `app` - reserved for decentralised application actors and contracts. Community approval required. - `lab` - open access, for on-chain testing of actors and smart contracts. - `peer` - reserved for established peer operators. Community or governance approval required, which will normally be granted to peer operators with a proven record of correctly maintaining at least 10,000 Convex Gold stake. New root namespaces require Foundation governance approval. Purely numeric top-level names (e.g. `42`) MUST NOT be created. This is a permanent governance invariant: it guarantees that `did:convex` identifiers are unambiguous, since numeric identifiers always denote account addresses (see CAD043). ### The `convex.cns` alias The genesis state defines the record `convex.cns` with value `#9` (the standard registry) and child node `[#9 []]` — the CNS root itself. This makes `convex.cns` a named alias for the registry and the root namespace: `@convex.cns` resolves to the registry account, and `convex.cns.X` resolves identically to `X` for any name `X`. This provides a stable, named entry point for tooling that prefers not to hardcode `#9`. ### Node SPI Any actor MAY act as a CNS node by implementing the following callable functions, invoked via scoped call on the node reference (the scope carries the node key). Alternative implementations MUST honour these signatures to interoperate with the standard user API: - `(cns-read name)` — returns the record for the given segment name in this node, or `nil` if absent. SHOULD be freely readable. - `(cns-write name record)` — creates or replaces the record for the segment name. MUST enforce the authority model: `:update` trust from the record controller for existing records; `:create` trust from the node owner for new records. - `(cns-create-node name owner)` — creates (or returns, if already linked) a child node for the given segment name, owned by `owner`, returning its scoped reference. MUST enforce `:create` trust from the node owner. - `(cns-delete-node name)` — deletes the entry (and node, where hosted by the same actor) for the given segment name. MUST enforce `:delete` trust from the node owner. MUST NOT recursively delete independently-owned descendant nodes. - `(change-control controller)` — transfers node ownership. MUST enforce `:control` trust from the current node owner. CNS records also function as trust monitors: the standard registry implements `check-trusted?` on scoped references, delegating trust decisions to the record controller (or the root controller for the root node). This allows a CNS path itself to be used wherever a trust monitor is expected. ## Alternatives Nothing in Convex requires the use of CNS. It MUST be possible to create working systems on the CVM without relying on CNS. ### Direct addressing Instead of CNS, it is perfectly legitimate to directly use addresses that are known and trusted: ```clojure (call #16789 (known-actor-function :foo)) ``` Commonly, actor code might `import` a library from CNS at compile/deploy time into the local environment, then use it in subsequent code without further CNS lookups: ```clojure (import convex.asset :as asset) (defn my-balance [token] (asset/balance token *address*)) ``` ### Static compilation It is reasonable to statically compile resolved CNS addresses so that future uses do not actually perform CNS lookups (which can be a relatively expensive operation). This can be done by compile time resolution as follows: ```clojure (defn my-function [x] (call ~(resolve my.actor) (actor-function x))) ``` Examination of the resulting CVM ops will confirm that the result of `~(resolve my.actor)` has been statically compiled into the function. ### Alternative CNS roots Users and application developers SHOULD adopt canonical CNS names, as other systems may not be able to resolve against an unrecognised CNS root. Alternative CNS implementations MAY however be deployed with a different CNS root. Users will need to refer to the correct entry point for the alternative implementation. ## Security Considerations ### Deploy-time binding It is important to note that `import` typically resolves at deployment time (i.e. when the `deploy` command is executed). This is intended behaviour, as it ensures the current on-chain CNS value is used. However it does imply that the same source code deployed at different times MAY resolve to different implementations, if the CNS value is updated in the interim. Developers SHOULD consider the following for production usage: - Pinning to a specific address e.g. `(import #5678 :as foolib)` - Asserting a particular version e.g. `(assert (= 0xa412bce98da16e4b6790069a108d2d902a9929df76af3bcabaf7cf22df470108 (hash (encoding (account foolib)))))` - Checking resolution is as expected post deployment e.g. `foolib` as defined in the account refers to the intended library. ### Controller Trust CNS records are as secure as their controllers. For mission-critical actor code handling significant value, users SHOULD verify: - Controller addresses match expectations and are trusted - Controllers of parent nodes are similarly trusted - **Owners of parent nodes** are similarly trusted: the node owner can delete or replace records within the node, so trusting a name means trusting the owner of every node on its path - Accounts referenced via CNS resolution are similarly secure ### Delegated authority Because record control and node ownership are separate capabilities (see Authority model), acquiring a name via `control` does NOT remove the previous owner's authority over the surrounding namespace. Parties acquiring names SHOULD verify node ownership (and obtain it, where full control of a subtree is required) as part of any transfer. ### Root Control CNS root namespace and some key libraries are controlled by secure governance accounts, at least for the Protonet stage (typically `#6`). It is likely some of these will be made completely immutable once fully battle-tested. ## Implementation status and open issues The standard registry (`convex-core/src/main/cvx/convex/core/registry.cvx`, account `#9`) implements this specification with the following known deviations, scheduled to be addressed via a network upgrade (the registry account is genesis state, so in-place source fixes are not possible for live networks): 1. **Record creation authority**: the registry currently checks the *parent record's controller* (rather than the node owner) when creating a new record in a node. Besides deviating from the authority model above, this means a node with no corresponding parent record (possible via direct `cns-create-node`, or after record deletion) has no authority able to create records in it. 2. **Orphaned nodes cannot be deleted**: deleting a node entry requires `:delete` trust from the *parent* node's owner. Once a parent node is deleted, its ownership entry is gone and any surviving descendant nodes become permanently undeletable — even by their own owners. A node owner SHOULD be able to delete their own node. 3. **No user-level `delete`**: records can currently only be removed via a direct SPI call. 4. Minor: segment names are not validated against the round-trip recommendation above; `cns-delete-node` takes an unused `owner` parameter. See `CNS.md` in the `convex-core` repository documentation for the detailed design analysis and the proposed upgrade. --- # CAD015: Peer Communications ## Overview Peers in Convex need to communicate messages to ensure the effective running of the protocol and communication with clients and other peers. We need highly efficient messaging that is well suited for distributed systems dealing with advanced lattice data structures and operations. This CAD describes the Convex / lattice messaging model. Objectives: - Resilient to network failures and deliberate attacks - Low latency communications - Messaging efficiency - Asynchronous model supported by default - Flexibility to adapt to different transport protocols - Consistency with lattice data principles - Usage of CAD3 data for message payloads ## Messages Peers communicate via messages. A message is an atomic, asynchronous piece of data passed from a sender to a receiver. Each message has a CAD3 payload. ### Message Encoding The Message encoding format is designed to efficiently encode a CAD3 payload. Since CAD3 structures can contain multiple branches, each with their own encoding, the key idea is to send the top level encoding first, then follow this with any required child branches. A Message is normally encoded as a Blob of bytes consisting of the following: - Encoded top level message payload (according to CAD3 Cell encoding rules) - Optional: One or more additional branch cell encodings - A VLC encoded length - Branch cell encoding (according to Cell encoding rules) Each individual cell encoding MUST fit within a fixed size buffer (currently 16383 bytes). However, by including the additional branch cell encodings, it is possible to include branch cells referenced by the payload. In this way: - Large data structures can be passed in a single message - Branch cells can be omitted, in which case the message is regarded as **partial**. Partial messages are appropriate for values such as lattice deltas where the recipient is expected to already be in possession of the omitted branches. A partial message is valid, however the receiver may not be able to access the full payload immediately. The overall size of the message is not part of the message itself, but will typically be provided by the transport mechanism e.g.: - For binary protocol messages, the message length precedes the message - For HTTP messages of type `application/cvx-raw` the message length is specified in the HTTP `Content-Length` header. - For messages passed as an octet stream, the message length is naturally delineated by the end of the stream ### Message Types The type of the message can be inferred from the payload. Any CAD3 value may form a valid Message, so the interpretation of these values is part of the peer protocol (i.e. application specific). Currently recognised message types follow: #### BELIEF ``` CAD3 Payload: Belief ``` This message specifies a belief from an other peer that is being shared as part of the CPoS consensus algorithm. Receiving peers SHOULD validate this belief message, and if valid perform a belief merge with their current belief. Receiving peers MAY ignore beliefs if they are experiencing high demand and need to throttle the number of belief merges being performed. Receiving peers SHOULD ignore and/or minimise processing for beliefs that have already been received and merged. This is safe because belief merges are idempotent. #### QUERY ``` CAD3 Payload: [:QR msg-id form address?] ``` This message represents a request for a peer to compute the results of a query (considered as a read-only transaction). Peers SHOULD make a best effort attempt to respond to queries from authorised clients. Peers MAY reject queries if they are experiencing high demand. In such cases peers MUST attempt to return a result message with an error code indicating temporary failure due to load. #### TRANSACT ``` CAD3 Payload: [:TX msg-id signed-transaction] ``` This message represents a request for a Peer to process a transaction by incorporating it into a subsequent Block that the Peer proposes to the Network. Peers MUST reject transactions that are not correctly signed. Failure to do so may result in slashing. Peers MUST reject transactions that have a previously used sequence number. #### RESULT ``` CAD3 Payload: Result ``` This message represents the result of another message request (usually a transaction or query). A Result message MUST reference the message ID of the original message. Results that do not correspond to a pending outgoing request ID should normally be ignored. #### STATUS ``` CAD3 Payload: [:SR msg-id] ``` This message represents a request for a Peer to provide information about its current status, including data about latest consensus state. Peers SHOULD respond to the status request immediately if able. Peers MAY cache their most recent status response for efficiency reasons. #### DATA_REQUEST ``` CAD3 Payload: [:DR msg-id hash0 hash1 .....] ``` This message represents a request for missing data. Usually, this is sent by a peer when it is attempting to process a partial message but some data is missing locally, and it needs to acquire the missing data before proceeding. The Missing Data request must include the Value IDs (hash of encoding) for the missing data branches. Note: It is guaranteed that if a peer has received a partial message, it must be able to determine the hashes of any directly missing data (since they will be encoded as branch refs in the partial message). Peer that send this message MAY suspend processing of a message pending receipt of missing data from the original sender. If the original sender is unable to satisfy this request in a timely manner, the suspended message SHOULD be discarded. Receiving Peers SHOULD respond by sending a `RESULT` message containing the missing data specified. ### Trust Peers in general SHOULD only trust outbound connections to other peers where the other peer is able to prove their authenticity by signing a unique challenge with the peer's private key. Peers SHOULD reject messages that appear to be malicious, incorrectly formed or too large for reasonable handling. Peers MAY accept messages from any source, but if they do, they SHOULD prioritise messages from trusted sources. ## Transport ### TCP Connections The standard mechanism for message passing is TCP connections established between peers. Messages are sent as: - a VLQ encoded message length N - N bytes representing the Message encoding ### UDP Connections UDP will be explored as a potential future transport protocol for efficiency and performance reasons. --- # CAD016: Peer Staking ## Overview Staking is the process by which peers in the network and other participants lock up economic value (stake) to support the security of the network and earn economic rewards from participating in the CPoS consensus. Peers must place a peer stake to participate in consensus. This is at risk if the Peer provably misbehaves, and may be lost through a process of Slashing, but is safe as long as the Peer continues to operate correctly and securely. Other participants may also place a delegated stake on a peer they wish to vouch for. It is in the interests of large coin holders to support the security of the Network by placing stake on good peer operators that they trust, as well as to earn additional rewards on their holdings. The total Stake of a seer determines its voting weight in the CPoS consensus algorithm. ## Meaning of Stake Stake involves taking a risk and performing useful work for the network to earn rewards. A peer operator that stakes on its own peer ("peer stake") is warranting that it has **fully secured its peer key used for operational participation in consensus**. The work they do is ensuring this peer is properly managed, secured and maintains network consensus correctly. It may lose its stake if this key is compromised (typically this would mean that the peer server is itself compromised). It may also lose its stake if the controller account is compromised. Delegated stakers are warranting that they **trust the peer operator to maintain consensus and earn rewards while properly protecting the peer controller account**. The work they do is in evaluating peer operators and betting their coins that the peer operators performs their role honestly and effectively. Their delegated stake is not at risk if the peer itself is compromised or crashes, but *is* at risk if the controller account is compromised. It should be observed that the most important thing from a security perspective is the private key used to control the peer controller account: all stake is at risk if this is lost. For this reason it is STRONGLY RECOMMENDED that important peer controller keys are kept secure in offline storage / air-gapped systems. This is a good incentive since the network as a whole could go offline if sufficient numbers of peers are simultaneously compromised. ## Rewards Stakers are rewarded with a share of Convex Coins earned from: - Transaction fees executed on the network affecting the CVM global stake - Reward Pools build up over time set by the Convex Foundation ### Transaction fee handling Transaction fees are accumulated for all transactions executed within a complete block submitted by a peer. When a block of transactions is successfully executed: - 50% are given to the peer and its stakers immediately - The remainder is added to the the common reward pool ### Reward pool distribution Rewards are delivered from the reward pools based on peer stake and active time elapsed. Active time elapsed is calculated when a block is successfully executed in consensus for a given peer. Active time is calculated as: - The difference between the timestamp of CVM state for this Block and the last such timestamp for the same peer - Subject to a maximum of 6000ms * total coin supply / total peer stake i.e. if the peer's total stake is 1% of the coin supply, it must submit a block at least once every ten minutes to maintain 100% active time. This mechanism is designed to ensure that peers must participate regularly and correctly in order to obtain full rewards. ### Staking split When rewards are given to a peer, they are divided as follows: - The total reward is divided over all Peers according to Peer Stake - For each Peer: - 50% is allocated to the Peer itself (added to peer stake) - 50% is divided across delegated stakers on the peer (according to their relative stake) - If there are no delegated stakers, the full reward goes to the Peer ## Stake pools It is possible to establish a stake pool where an actor places stake on behalf of others. Examples: - A public stake pool which issues a token that entitles stake pool members to a share of returns gained from peer rewards - A private stake pool run by a large peer operator to manage stake across its own peers - A charitable stake pool which distributes returns to good causes Stake pools are made possible by peer staking and CVM actor code, but are outside the scope of CAD016. Innovation is encouraged in designing effective stake pool implementations. ## Effective stake decay Peer stakes are temporarily discounted if the peer is inactive. This enables the network to progress even in the event of major peers going offline for an amount of time. Stake decay occurs at the following rate by default: - 3 minutes grace period with no decay - A fall by a factor of `1/e` every 5 minutes thereafter Stake decay does not effect the actual peer's stake, but does affect: - The effectiveness and voting weight of the stake in consensus - The ability of other network participants to evict the peer ## Slashing Slashing is the penalisation of peers for bad behaviour. Any slashing will result in a deduction of stake, which will be transferred to the overall peer reward pool for properly behaving peers to collect in the future. There will be **no stake slashing in Protonet**, although stake decay is active so inactive or misbehaving peers will become quickly irrelevant to consensus (and probably be evicted). Slashing conditions for main network will be evaluated during Protonet phase. Questions to be considered: - Under what conditions might slashing occur? - Is delegated stake subject to slashing or not? ## Changing Peer Stake Peer operators may add or remove peer stake from their owns peers with the following command: ```clojure ;; note: stake is denominated in coppers (set-peer-stake 0x42272E789B7a3D57f8267c15c2d9B8BeD9b0E2035b3a8AE9A0eb9A024B7FADe5 10000000000000) ``` Removing all peer stake can be done by setting stake to `0`, though typically it is better to use the `evict-peer` command to remove the peer record entirely and get a memory refund. ## Changing Delegated Stake Changing delegated stake on any peer can be done with the `set-stake` command: ```clojure ;; note: stake is denominated in coppers (set-stake 0x42272E789B7a3D57f8267c15c2d9B8BeD9b0E2035b3a8AE9A0eb9A024B7FADe5 10000000000000) ``` ## Peer Eviction Peers can be evicted from the global state in two situations: - The peer's effective stake is less than the minimum effective stake (currently 1000 Convex Gold) - The peer controller can always evict it's own peer(s) ```clojure (evict-peer 0x42272E789B7a3D57f8267c15c2d9B8BeD9b0E2035b3a8AE9A0eb9A024B7FADe5) ``` When evicted: - The peer's stake is returned to the controller account - Any delegated stakes are returned to the staking accounts - The peer record is removed from the Global State There is an incentive to evict peers because deletion of the peer record will result in a memory refund to the account that performs the `evict-peer` operation. Anyone can do this for insufficiently staked peers and it reduces the CVM state size, so it is good for the ecosystem! --- # CAD017: Peer Operations ## Overview Operating a peer is an important responsibility in the Convex Network. Anyone can run a peer, and they are responsible for maintaining the Consensus of the Network. They must place a minimum stake of 1000 Convex Coins. Most users of the Convex Network do not need to run a peer - they connect to peers via client software that submits transactions and queries information on their behalf. The peer API is open to client requests by default: Peer operators who wish to restrict this may need to set up additional configuration or infrastructure. This document primarily contains recommendations for peer operators ## Requirements Running a Peer requires: - An Internet-connected Server - At least 100 MBits/sec continuous network bi-directional bandwidth - A modern processor with at least 8 dedicated Cores - At least 8 GB RAM (32 GB Recommended) - At least 1 TB fast Storage (NVMe Recommended) - A secure modern operating system (Linux recommended) with good support for memory mapped files - Java 21 or above The network should be configured with: - a publicly accessible IP address (IPv4 or IPv6). Dual stack networking support is required in the OS. - firewall access to the server via TCP on a chosen port (the Convex protocol default `18888` is recommended) - a trusted DNS entry (e.g. `peer.mycompany.com`) is recommended - HTTPS certificates recommended for the HTTPS REST API The DNS entry is optional, but it will help significantly with discoverability / user access to your peer. ## Configuration ### Accounts In order to operate a peer you will need a Peer Controller account. This can be any account on the Convex network, e.g. `#1678` with at least 1000 Convex Coins. ### Peer Config Peers can be configured at launch in various ways. #### Outgoing connections Peers MAY configure the number of concurrent outgoing Peer connections according to their available bandwidth. 20 (the default) recommended for Peers with sufficient outgoing bandwidth. There are trade-offs here: - With more outgoing connections, your transactions will reach consensus faster - You must weight this up against bandwidth costs - If the number is too low your published blocks may get lost if the destinations do not relay them. TODO: describe mechanism to set connection count controls ## Startup ## Syncing Your peer will need to synchronise with the network by connecting to at least one existing peer. The following peers are available at time of writing for synchronisation: ``` peer.convex.live:18888 ``` TODO: CLI commend top start peer with target host ## Shutdown ## Upgrade ## Recovery ### Network Partitions It may occur that a peer becomes temporarily disconnected from the peer network. Peers are designed to automatically recover from temporary network failure and re-establish connections when possible. Peers with normal configuration SHOULD periodically re-attempt to connect with other peers randomly until connection with the Network is re-established. Peer Operators SHOULD provide for an alternative way to connect to the main network, if only for the purposes of withdrawing the Peer's Stake. For example, a peer operator may monitor the connectivity of their peer and use Convex Desktop to de-stake the peer if it loses connections. TODO: describe best way to monitor this. Perhaps API peer health endpoint? ### Security Breach If a security breach is detected, the Peer SHOULD be immediately shut down to minimise potential risks. Peer Operators SHOULD attempt to withdraw their Stake immediately, possibly through a separate Client with access to an independent Good Peer, although in the case of severe security breach compromising private keys this may already be too late. ## Staking Peers are required to post a Peer Stake to participate in consensus. ### Setting a stake ### Withdrawing stake ### Stake penalties In Protonet, there is no slashing of stake (i.e. peers are not formally penalised for incorrect behaviour). In the future peers may be automatically penalised for provably incorrect behaviour. ## Key Management ## Connection Management A Convex Peer is designed to automatically manage P2P connections to the Network during normal operations. In most cases, assuming good network connectivity, a Peer should require no manual intervention to control connections to other Peers. ### Incoming Connections Peers treat incoming connections as regular Clients, i.e. they afford no particular special privileges to incoming connections from other Peers. The purpose of this is to ensure that Bad Peers have no particular ability to influence a Peer that they connect to. ### Outgoing connections A Peer maintains a managed list of outgoing connections (i.e. connections to which they broadcast their Beliefs). Outgoing connections follow the following rules: - **Validated hosts**: Peers MUST only connect to Peers accessible on the network via the host address specified for the destination Peer in the current consensus, **or** if they are explicitly instructed to connect to a specific host address by the Peer Operator (e.g. when joining the Network). This minimises the chance of connecting to Bad Peers. - **Random elimination**: Peers SHOULD eliminate connections at random for low-staked Peers. This allows the Peer network to stay dynamic, and give an opportunity for new Peers to be connected to - **Stake-weighted selection** Peers MUST connect to other Peers preferentially according to stake, so that Bad Peers do not have a significant chance of isolating a Peer - **Target connection count**: Peers should attempt to maintain a number of outgoing connections to Peers as configured by the Peer Operator. This allows Peer Operators to control their bandwidth usage. Peers SHOULD NOT reveal their current outgoing connection list to external parties, since this opens up some risk of the Peer being vulnerable to attacks in situations where it could be isolated from the rest of the Network (e.g. censorship). ## Storage Management Storage management is at the discretion of the Peer Operator. Peer Operators SHOULD arrange for periodic garbage collection of the Etch Store, if they do not have another reason to keep historical data (e.g. for analytics). ## Logging and Analytics In General, logging and analytics is at the discretion of the Peer Operator. ## Security Risks ### Key compromise The greatest risk to Peer Operators is the compromise of their Peer's private key. A compromise of this nature could allow an attacker to sign messages an impersonate the Peer, potentially deliberately causing the Peer to be slashed or (it the Peer is highly staked) disrupting consensus. Peer Operators SHOULD maintain high level security procedures for their Peer's environment. Peer Operators SHOULD maintain offline backups for their Peer's private key. ### Control Account compromise A compromise of the Peer control Account(s) could allow an attacker to withdraw a Peer's staked coins and steal these. Peer Operators SHOULD ensure maximum security for their control Accounts. Peer Operators MAY keep private keys for control Accounts in separate environments from the Peer private key. While this may add operational complexity and risk, it mitigates against the risk of a Control Account compromise happening at the same time as a Peer private key compromise. ### Complete Network Partition A complete network partition could cause a Peer to be isolated and excluded from consensus, e.g. network failures at a data centre. Peer Operators SHOULD establish an alternative means of submitting a transaction to the remainder of the Network to withdraw their Peer's stake if this partition cannot be resolved in a reasonable timeframe (e.g. minutes). Failure to do so may result in partial slashing. Peer Operators SHOULD signal to clients if their Peer is unable to participate in Consensus. The recommended approach is returning a Result with the Error Code `:NETWORK` to indicate that network connectivity is unavailable. This can signal to clients that they should try again later, or alternatively attempt to connect to a different Peer in the main network assuming this is still live. --- # CAD018: Scheduler The scheduler provides a on-chain facility for code execution to be deferred until a later time. Users (and Actors) can make use of this to reliably ensure code execution at a future time, secured by the network protocol. Once scheduled, such operations are effectively "unstoppable". The primary motivation of this capability is for users to trigger actions at a future time without requiring further external interaction. ## General Design Principles ### Concept of State timestamps Scheduled operation MUST specify a timestamp ### Guaranteed execution Peers MUST execute scheduled operations as part of the first state update after the scheduled timstamp. The first state update after the scheduled timestamp is defined as the one caused by the first Block that has a timestamp greater than or equal ### Ordering Peers MUST execute scheduled transactions before any regular user transactions in the same block. This is logical in the sense that they are considered to be known to the network before any user transactions. Peers MUST execute scheduled transactions in timestamp order. If two scheduled transactions have the same timestamp, Peers MUST execute scheduled transactions in the order that they were scheduled. ## Costs and Fees As with all Convex operations, The account performing a schedule operation MUST be charges fair costs for execution and memory usage. This is necessary to prevent DoS attacks and ensure Peers are fairly compensated for their resources used to maintain the network. The general principles are that: - The Account performing the scheduling must pay for the execution cost of the scheduled operation, as if it were a transaction executed at a future time. - The Account performing the scheduling must pay for temporary on-chain memory usage for data stored in the Schedule, but will be refunded for this after the scheduled operation is executed. ### Scheduler costs Scheduler costs are paid at the time that the requested operation is scheduled, i.e. as part of the transaction that calls the `schedule*` core function (the "Scheduling Transaction") The Scheduler MUST impose a low overhead per schedule operation, maximum O(log N) with the size of the schedule. This is necessary to prevent DoS attacks through excessive scheduling operations. The Scheduler MUST ensure pre-payment by the Account performing the schedule operation to pay for the initiation of the schedules transaction, when this occurs (transaction flat fee) The Scheduler MUST ensure that memory required for the schedule data structure is paid for by the Scheduling Transaction. ### Execution costs Execution costs for the scheduled operation are incurred by the Account for which the operation is scheduled, as if these were executed in a regular transaction. The Scheduler MUST impose juice limits in accordance with the available balance of the Account. ### Memory costs The Scheduler MUST ensure that the Account is refunded for memory used by the schedule data structure when it is released. This effect MUST apply at the end of the scheduled operation, as if it was a regular transaction releasing memory. After the scheduled operation is complete, the net impact on state memory size through the use of the scheduler MUST be zero. This is necessary to prevent state growth over time from scheduled operations. This MUST happen regardless of whether or not the scheduled operation succeeds. ### Memo The scheduler MUST reclaim memory by deallocating any state associated with any given scheduled operation. This is necessary to avoid long term state growth problems. ## Security Scheduled operations have some security implications ### Front running Scheduled operations are publicly visible, so user code SHOULD ensure that it cannot be exploited via attacks that are executed prior to the scheduled operation itself. Achieving this can be done by ensuring that only authorised accounts are permitted to take actions which may affect the outcome of the scheduled operation. Use of a scheduled operation that performs value exchange on digital assets (for example purchasing a digital asset from a marketplace) should be aware that 3rd parties may take actions that influence the price of the asset. The scheduled operation SHOULD be designed so that this cannot be easily exploited via market manipulation. ### Account transfers If a user takes over an account that has previously been in the control of another party, care should be taken to ensure that the account does not have previous scheduled operations that may present a security risk (e.g. taking control of the account again). Proposed solution: A timestamp can be stored so that all scheduled operations prior to the time of transfer can be automatically blocked. Setting this timestamp appropriately should prevent previous scheduled operations from presenting a risk. ## Operations ### Schedule ### Query until ### Drop --- # CAD019: Asset Model The Convex Asset Model is a universal system for expressing and controlling digital assets in an on-chain environment. The key motivation for the Asset Model is to enable economic value transfer using digital assets: digital assets can be securely owned, traded and used as part of contractual agreements just like real world assets. Of course, the security of the digital assets is enforced by the security guarantees of the Convex network. ## Objectives ### Universal and Extensible A key design goal is therefore that the API is universal and extensible, in the sense that it can be used to handle a diverse ecosystem of digital assets, including asset types that have not yet been invented. By way of example, it should be possible to use the same API to transfer a quantity of a fungible token: `(asset/transfer destination-address [currency.USD 10000] )` As it is to transfer a set of numbered NFTs: `(asset/transfer destination-address [asset.nfts #{101 102 105}] )` So... why is this important? - It reduces the number of APIs that developers must learn: instead of having separate APIs for fungible token, NFTs, game items etc. we offer a single unified API. - It makes it possible to write *generic* smart contracts that can work with any type of asset. - It makes it possible to innovate with new types of asset without having to redesign user code We therefore propose the following objectives: - The Asset Model should be able to express all types of on-chain digital assets (fungible tokens, security tokens, stablecoins, NFTs, voting rights etc.) - There should be a standard API for users that works with all types of assets in a generic way - There should be a standard SPI that allows for flexibility in underlying asset implementations - in particular it should be possible to create new kinds of digital assets and new rules / behaviours without changing the user level API ### Efficiency The Asset Model should allow efficient and simple implementations to minimise transaction costs and memory usage. If digital assets are to be widely accepted as a part of economic value exchange, it is essential that they are efficient and offer low transaction costs compared to alternatives. ## Definitions and Key Concepts ### Assets An **asset** is a logical entity that supports ownership and transfer according to the requirements in this CAD. Examples: - A non-fungible token (NFT) - A fungible token representing shares in a company - A fungible token representing units of derivative "put" contract ### Asset Implementation Asset logic MUST be implemented by an actor on Convex. This actor may be referred to as the "asset implementation" or "asset actor". An asset MAY map one-to-one to an actor, however a single actor MAY implement multiple assets. This allowance is primarily for efficiency reasons: if many assets share the same on-chain logic, it makes sense for a single actor to implement them all rather than deploying new actors for each one. The use of an actor to provide the asset implementation is important for two reasons: - It allows for the development of new types of compatible assets: these simply need to provide a new implementation and they can be used according to the standard CAD19 asset model, often without needing to change existing code. - Actors allow for trusted code execution and governance, providing assurance that digital assets will behave correctly and not present unacceptable security risks ### Asset Path An asset path is a descriptor that identifies an asset. Asset paths are important because they enable a stable reference to a specific asset under consideration. An asset path MUST be either: - The address of the actor that provides the asset implementation - e.g. - `#1234` is a valid asset path referring to the asset implemented by the actor at the address `#1234` - A vector where the first element is the address of the actor, and the second element of the vector is a scoped value interpreted by that actor on an implementation defined basis. e.g. `[#2345 :foo]`. The asset actor will see the scoped value as `*scope*` when called. A Vector-based asset path SHOULD be used to allow a single actor to implement many different digital assets, e.g. - Currencies might be designated by an asset path like `[#123456 :USD]` - Derivative contracts such as put options might have an asset path that includes the underlying asset, strike price and expiry time e.g. `[#98765 [[#12345 :USD] 12500 1741948885345]]` - Bets on a football match might specify the match date and selected winner `[#8978 ["2023-6-06" "Manchester United"]]` (note that a bet on a different outcome of the same match would be a different fungible asset since they are not mutually fungible) ### Quantities All Assets MUST define a notion of "quantity" that can be used to represent an amount or subdivision of an asset. Because we want to enable innovation in the types and representations of assets, we do not restrict the definition of quantity to a specific type (e.g. integer amounts) - instead we define the mathematical properties that quantities must obey. Quantities for any asset MUST be a **commutative monoid** (in the mathematical sense). This requirement is necessary in order for addition, subtraction and comparison of asset quantities to behave in well-defined ways. Assets MUST define an **addition** function enable quantities to be additively combined, i.e. given any two valid quantities of an asset it MUST be possible to use the addition function to compute the total quantity. This is equivalent to the addition function of the commutative monoid. Assets MUST define a **comparison** function enabling quantities to compared, i.e. given any two quantities of an asset it MUST be possible to use the comparison function to determine if one quantity is a subset of the other. This is equivalent to the algebraic pre-ordering of the monoid. Assets MUST define a **subtraction** function enabling quantities to be subtracted, i.e. given two quantities of an asset where the first is "larger" than the second (as defined by the comparison function), it MUST be possible to subtract the second value from the first and get a result that is also a valid quantity. Assets MUST define a **zero** quantity that logically represents an empty holding of an asset. This zero value is the identity element of the commutative monoid. The zero quantity will usually be an "empty" value such as `#{}` or `0`. The zero quantity SHOULD be the default balance of all accounts. Logically, an account should have a zero holding of an asset until some quantity of the asset is otherwise obtained. An example of an exception to this might be an asset implementation that gives a free non-zero quantity of the asset to all accounts, though this is probably unwise given the obvious potential for abuse. For any given Asset, it MUST be possible to identify a Holding of the Asset for a given Account, where the Holding is the Quantity of the Asset that the Account owns. An asset implementation MAY permit quantities to be specified using non-canonical values, provided that it MUST behave as if the equivalent canonical quantity was provided. For example, a NFT actor could reasonably consider a non-set value such as `123` to refer to the singleton set `#{123}` representing a single numbered NFT. An asset implementation MAY attempt to produce reasonable results where quantity operations are passed arguments that are not normally valid, e.g. subtracting `3000` from `1000` would reasonably produce the result `-2000`, even though this is not a valid balance. However users of assets SHOULD NOT rely on this behaviour (in this case, subtraction is invalid because the first argument is "less than" the amount subtracted). When passed as an argument to an asset implementation, the value `nil` MUST be treated as the zero quantity. This requirement ensures that a zero-equivalent value is known for all implementations, and can be used by generic code without having incurring the cost of explicitly querying the zero value. #### Quantity examples - Fungible tokens typically use a quantity expressed as non-negative integers e.g. `0`, `1000`, `987654321` - Non-fungible tokens typically use a quantity expressed as sets of NFT IDs e.g. `#{101 1002 1003}` - An asset representing a voting right may use a boolean quantity `true` and `false` ### Ownership Quantities of digital assets are owned by accounts in Convex. Assets may be owned by either user accounts or actors. In the latter case, it should be expected that the actor implements code able to manage the assets that it owns. Assets SHOULD be transferable, i.e. it should be possible for a quantity of an asset to be transferred from one owned to another. ### Offers and Acceptance It is often necessary for a smart contract to ensure that it receives another asset before it takes some action: for example a contract for sale would expect payment to be made before allowing the purchased assets to be released. A typical process would be something like: - Account `A` **offers** a quantity of asset `F` to account `B` (where `B` is a smart contract) - `A` calls a callable smart contract function on `B` to request a transaction - `B` checks preconditions for the transaction as necessary - `B` **accepts** the quantity of `F` from the caller (`A`) to pay for the transaction - Assuming all is successful, `B` completes the transaction - `B` returns to caller with transaction complete - Optional: `A` closes down the offer (only relevant if some non-accepted quantity remains) Assets SHOULD implement a system of offers, whereby an owner may offer a quantity of an asset to another account, which can subsequently be accepted by that account. Assets SHOULD implement a system of acceptance, whereby an account that has been offered a quantity of the asset may accept that quantity (or a partial quantity thereof). Offers SHOULD remain open at the discretion of the offering account. However, closing of offers by a trusted 3rd party (e.g. to mitigate against security risks) MAY be acceptable in some cases. ## User API The user API for the Asset Model is provided by the library `convex.asset`. Examples below assume the user has imported an alias for `convex.asset` library e.g. ```clojure (import convex.asset :as asset) ``` Users do not need to use the `convex.asset` library to work with Convex digital assets - they are free to access the underlying actor functions directly. However the user API presents a convenient, well-tested interface that should be suitable for most purposes. :::warning `convex.asset` is a library, which means that functions run in the context of the account that invokes them. Only do this with a trusted library, i.e. be sure you have imported the correct library. The address of the standard `convex.asset` is `#65` on Protonet. ::: ### `balance` The `balance` function gets the total quantity of an asset currently held by an account. ```clojure (asset/balance some-fungible-asset *address*) => 1500 ;; Omitting the address argument is equivalent to using *address* (asset/balance some-fungible-asset) => 1500 ``` The returned value should always be a valid quantity for the specified asset. In particular, the owner should be able to `transfer` this amount to send their entire holdings of the specified asset to another account. If the address argument is omitted, the balance for the current account is queried (i.e. an implicit `*address*` argument is used to specify the current account). ### `transfer` The `transfer` function transfers a quantity of an asset to a recipient. ```clojure ;; This transfers to account #13 1000 units of MY_TOKEN (asset/transfer #13 [MY-TOKEN 1000]) ``` As an alternative form, `transfer` may be called with an asset path and quantity as separate arguments: ```clojure ;; This transfers to account #13 1000 units of MY_TOKEN (asset/transfer #13 MY-TOKEN 1000) ``` The receiving account MAY implement a `^:callable receive-asset` function, in which case instead of directly transferring the asset, the asset will be *offered* and `receive-asset` will be called. Actors could, for example, use this so that they can automatically reject assets that they are not supposed to receive, and `accept` assets that they can handle. `transfer` MUST fail if the caller does not have the quantity of assets specified. This SHOULD be a `:FUNDS` error, since this is conventionally used to indicate the lack of a sufficient asset quantity. ### `offer` Making an `offer` makes a quantity of an asset available for a subsequent `accept` by a nominated receiver. ```clojure (asset/offer receiver [fungible-token-address 1000]) ``` :::warning If executed, an offer stays open until explicitly closed (by setting the quantity to zero). Users are advised to be cautious about making offers unless they expect the offer to be immediately accepted on terms they approve of (or rolled back in case of error) since the receiver may accept the offer at some arbitrary point in the future. ::: Asset implementations SHOULD reclaim memory by removing any offer records if the offer is set to the zero quantity. Failure to do so may result in wasted memory allowances. ### `get-offer` The current offer of an asset from any sender to any receiver can be obtained with the `get-offer` function: ```clojure (asset/get-offer fungible-token-address sender receiver) ``` Asset implementations MUST return the current offer, i.e. the quantity last specified via `offer` from the sender to the receiver minus any quantities subsequently accepted with `accept`. Asset implementations MUST return the zero value if an offer does not exist. ### `accept` An offer may be accepted by a receiver as follows: ```clojure (asset/accept sender [fungible-token-address 1000]) ``` Often, the sender will be the `*caller*` to an actor function which needs to take a quantity of an asset from the caller, e.g. as part of an atomic value exchange or as a payment of fees. Asset acceptance MUST fail with a `:FUNDS` error if the quantity could not be accepted because it exceeds the sender's current balance. If accepted, the asset implementation MUST: - Transfer the specified quantity of the asset from the sender to the receiver - Subtract the accepted quantity from the offer (which may result in it becoming zero) ### `owns?` Tests whether a given address owns the specified quantity of an asset: ```clojure (asset/owns? #1456 fungible-token-address 1000) ;; alternative argument layout with an asset vector (asset/owns? #1456 [fungible-token-address 1000]) ``` The `owns?` function MUST return true if and only if the balance of the owner is at least as large as the specified quantity, i.e. the owner would normally be able to `transfer` this amount. ### `total-supply` The `total-supply` function obtains the current total supply of an asset, if available. ```clojure (asset/total-supply my-token) => 1000000000000000000 ``` The result (if available) MUST equal the sum of all current balances of the asset. The `total-supply` function MAY return `nil` if the asset does not support computation of the total supply. The total supply of an asset MAY change during the lifetime of an asset, e.g. because new quantities are minted or burned. Asset implementations SHOULD cache the total supply by keeping track of the total quantity of an asset created or destroyed, e.g. via mint and burn operations. This allows the total supply to be provided without adding up all current balances, which is likely to be unreasonably expensive for callers. ## Security considerations ### Untrusted assets Assets are implemented by actors in the Convex asset model, and as such there are a number of issues that may arise if untrusted assets are used. The general recommendation is that users SHOULD NOT interact with untrusted assets. However, in some circumstances, it may be necessary to write code that may interact with untrusted assets. An example of this could be an "Auction House" actor that enables users to post lots containing other digital assets for auction. There is no way, in advance, for the auction house actor to know whether assets that may be sold in the future will be trusted or not. Hence the auction house must be written in a way that is robust to the inclusion of untrusted assets. #### Arbitrary code execution When an actor implementing an asset is interacted with, it may execute arbitrary code. Unless the actor is trusted, caution must be taken to mitigate these risks (as with a call to an untrusted actor). While the Convex security model prevents the actor from directly taking actions on behalf of the caller (e.g. it cannot steal arbitrary assets) it may take other actions during the scope of the transaction. Developers interacting with untrusted assets should be particularly aware of: - Re-entrancy attacks where a malicious asset calls back into the same smart contract - Possibility that a malicious asset implementation may call other smart contracts, e.g. DEX exchange calls #### Unspecified behaviour If an asset implementation is untrusted, it is possible that the asset may not behave correctly according to the requirements in this CAD. Some examples: - A transfer may appear to have succeeded, when it has in fact failed - A malicious asset may lie about balances or the success or otherwise of transfers - A 3rd party may have the ability to update holdings unilaterally, without the authorisation of owners. - A previously well-behaved asset may be "upgraded" to become malicious ### Inaccessible Accounts It is possible to transfer assets to an account that may be locked or otherwise inaccessible (e.g. a user account with a lost key pair). In such cases, the quantity of asset transferred may be irretrievably lost. Some mitigations for this risk: - Prefer solutions where the destination account explicitly calls `accept` to obtain the asset. By requiring the active participation of the receiver, this minimises the risk of the asset being locked in an inaccessible account. - Only implement `receive-asset` for an actor if the actor provides mechanisms for extracting assets at a later time. Otherwise,a `transfer` to an actor risks losing access to the asset. - Perform off-chain validation that the destination user has access to their account (e.g. requiring a signature of a random number to prove possession of the appropriate key pair). - Use appropriate governance mechanisms (e.g. `set-controller`) to enable account recovery as a last resort. ### Quantity Overflow Asset implementations should ensure that they do not allow bugs resulting from numerical overflow or other issues relating to calculations of quantities. In general: - Limits MUST be placed on total quantities available where overflow might otherwise be possible, e.g. a maximum supply for a fungible token. This limit should normally be enforced when minting / issuing new quantities. - Functions involving quantities as arguments MUST check whether the quantity is valid, e.g. a non-negative integer less than or equal to the balance of the account requesting the transfer for a fungible token. Typically a check such as `(<= 0 amount balance)` is appropriate. An asset implementation which is vulnerable to quantity overflow issues should be considered as broken (and therefore untrusted). ### Non-compliant asset implementations It is possible that some asset implementations (either by mistake or ill intention) are not compliant with the Convex Asset Model. Users SHOULD NOT interact with non-compliant assets. Some specific reasons for non-comliance are listed below: #### Volatile balances Some assets may have balances / quantities that may vary independently of usage via the asset model. Some examples: - An interest-bearing asset that pays out some form of periodic rewards that increase balance - A utility token that has quantity deducted automatically through usage of services Such assets are NOT COMPLIANT with the Convex Asset model and SHOULD NOT be implemented or used without extreme caution. If applications nevertheless choose to make use of assets with volatile balances, applications SHOULD always check the balance at the start of any transaction rather than relying on any previously stored balances which may no longer be correct. Such assets SHOULD NOT be used in general purpose smart contracts, since accounting for held balances may be invalidated. If functionality is desired that requires balances to be varied, better approaches include: - Requiring a user to explicitly claim interest or rewards - Requiring balances to be deposited in a pre-payment account, so that usage quantities can be deducted and accounted for properly. #### Aliasing It is possible that multiple assets may refer to the same underlying quantity or resources. As such, code referencing multiple assets MUST NOT assume that operations on the assets are independent. For example, transferring a quantity of an asset `A` may affect the balance available of asset `B`. This behaviour is NOT COMPLIANT for reasons similar to the volatile balances mentioned above In general, it is safest to operate on each asset in turn - avoid interleaving actions API calls on multiple assets. --- # CAD020: Tokenomics ## IMPORTANT LEGAL NOTICE Yeah, we need a mandatory disclaimer here :-) This repository discusses and describes tokenomics concepts in relation to the Convex network. It does not in any way represent a commitment on behalf of any party to implement the Convex Network in precisely the manner described, and may be subject to change based on learnings from Protonet or elsewhere. The information contained in this repository is provided on an "as is" basis for informational and discussion purposes only, with no guarantees of completeness, accuracy, usefulness or timeliness. This does not in any way constitute financial or legal advice. Participants in the ecosystem must take their own professional advice regarding any such issues. ## Overview Convex serves as a public utility network, where participants are free to transact on a decentralised basis. As such, there is a requirement for and economic protocol whereby users of the network can fairly compensate the providers of infrastructure for their services. ![Convex High Level Tokenomics](tokenomics.png) Convex Coins are initially issued in two ways: - 75% are available for purchase on the **release curve**. This is is a mathematically defined mechanism that releases coins as and when demanded by economic participation in the ecosystem. Funds raised are reinvested in the ecosystem to create a virtuous cycle. - 25% are available as **awards** to contributors who add value to the ecosystem in various ways (can be software engineering, open source contributions, marketing, building great uses cases etc.). Contributions must benefit the ecosystem as a whole. Once issued, coins are fully transferable and can circulate freely according to the wishes of their holders (e.g. traded on a private basis, used in smart contracts etc.). Coins used for transaction fees (or deliberately burned) are removed from the coin supply and a placed in a special "Reward Pool" which is released back to peer operators and stakers over time as a reward for maintaining the network. This model strikes the right balance between enabling long term sustainable growth and recognising those who bring value to the Convex ecosystem (financially or otherwise). There is a maximum supply cap of 1,000,000,000 Convex coins, though it will take a long time to get there. The total Coin supply at Protonet launch is 1m Convex Coins, initially held by Convex Foundation account(s) prior to onward distribution. ## Rationale / Discussion All digital currencies must have a mechanism for initial issuance, and Convex is no exception. However, many existing mechanisms have significant flaws: - **Protocol based issuance** - it is possible to issue tokens purely through the operation of the protocol, such as Bitcoin mining. Due to the mechanical nature, this is relatively predictable, automatic and trustworthy (at least to the extent that forks / upgrades to not fundamentally change the protocol). However, such approaches disproportionately incentivise infrastructure provision to the exclusion of all else: huge amounts of resources get consumed in Bitcoin mining, for example. This is undesirable because, at least for Convex, infrastructure provision is relatively cheap and efficient and can be well compensated through transaction fees alone - we want the majority of rewards to flow to people who add value to the ecosystem. - **Pre-mining** - tokens can be generated "as if" they were mined and distributed to select parties (typically founding teams and early investors). The problem with this approach is that it creates a significant incentives for these parties to "cash out" at the earliest opportunity, often at the expense of later ecosystem entrants, and it reduces the incentives for insiders to continue to build the project. There is a significant danger of a negative "pump and dump" dynamic. - **ICOs** - tokens can be generated and sold in a large public sale event. Such an event may generate a significant treasury for a foundation, and create broad public ownership. However, the price of such ICOs is hard to get right, and likely to attract significant speculative activity. Similar to pre-mining (which might occur alongside an ICO), there is a significant risk of "pump and dump" dynamics, often at the expense of less sophisticated retail investors. - **Airdrops** - tokens can be distributed for free in large quantities according to a variety of eligibility criteria (in-person events, community membership, holding some other token etc.). While airdrops can create publicity and temporary excitement, giving out tokens for free risks devaluing the token by arbitrarily giving a windfall to individuals without them having to contribute anything. This is a disservice to those who truly do add value. It also incentivises negative behaviour in terms of attempts to benefit from future airdrops (signing up with multiple fake accounts etc.) What we really need is a fair way to distribute coins that: - is **proportionate** to contribution / value add to the ecosystem - continues to **maintain good incentives** for contribution long after initial launch - aligns **incentives** of participants (e.g. avoiding "pump and dump" by large holders or insiders) - grows with the economic demand of the ecosystem on a **sustainable** basis Hence we have defined a new approach of **market driven release**: new coins are issued as and when the economic growth of the ecosystem requires it. Purchasable coins are issued on demand if and only if they are purchased from the release curve that pre-defines a schedule of rising prices. If demand for coins is strong (i.e. market cap growth causes prices to rise above the current release curve price) then there is an economic arbitrage opportunity for purchasers to buy new coins from the release curve. This continues until equilibrium is reached, i.e the newly increased supply and new higher price equal the new market cap. This can be considered, in some ways, analogous to a company issuing new shares to raise capital. Existing shareholders will be diluted, but the extra injection of capital and resources increases the market value of the company so that they are still better off (hopefully, assuming the capital is well invested). There are however some notable differences with traditional equity: - The Convex ecosystem is not a company, but a decentralised network of participants - The release curve guarantees that previous purchasers will never be diluted by new coin issuance at a lower price: additional issuance must occur at a higher price (or at worst equal, if purchased from the same tranche). - There is a fixed maximum supply cap ## Design Objectives The tokenomic model for Convex has been designed with the following objectives: ### Utility Token The Convex Coin is a utility token - it represents the right to use the coins to access services provided by or on the Convex Network. As such, it should be broadly available as a decentralised asset to anyone wishing to use these facilities. ### Means of exchange The Convex Coin serves as a means of exchange within the ecosystem, which can be used to facilitate efficient transactions between parties on a decentralised basis. As such, it must be convenient to use both by network users and autonomous actors / smart contracts as a digital currency. ### Ecosystem driven supply The Convex coin supply is driven by ecosystem growth: new issuance occurs primarily due to ecosystem demand. In the longer term, we would expect the Convex Coin to achieve a relatively stable value representing the utility of lattice technology powering a diverse ecosystem. ### Anti-dilution We wish to provide a degree of assurance to coin holders that their holdings will not be diluted by large issuance of coins at a lower prices. Hence, the release curve will not offer coins for sale at a lower price than previous purchases. ### Equitable rewards for contribution We must ensure that coins are distributed fairly, on the basis of making contributions to the Convex ecosystem. While there are many ways to contribute, we consider three categories of participants essential to reward fairly: - **Contributors** : Those who help build the ecosystem, whether through contributions to Convex itself of building value in the broader ecosystem. - **Coin Purchasers** : Those who help fund the development of Convex, by purchasing newly issued Convex Coins - **Peer Operators**: Those who provide the infrastructure to operate the network on a decentralised basis ### DoS prevention It is necessary to protect the network against the risk of DoS attacks enabled by flooding the network with unnecessary transactions, hence impeding the ability of regular users to transact normally and imposing unfair costs on peer operators. ### Sustainability In the long term, the tokenomics should converge towards a stable steady state equilibrium, where the pricing of transactions is fair and balances the cost of providing services and infrastructure for the network. ## Specification ### Overview The following overall tokenomic flows are possible: - **Genesis** : The complete coin supply is generated and placed into governance accounts which are initially considered "unissued" - **Issuance** : Coins are issued to either **Contributors** or **Coin Purchasers** - **Transfers** : Holders of coins may freely transfer coins between accounts. These accounts may include autonomous actor accounts as well as external users. - **Memory Accounting** : Coins may be exchanged for memory on the Global State. The memory accounting pool is considered a special governance account. See the Memory Accounting CAD for more details. - **Transaction Fees** : Coins are taken from users who commit transactions to the network, and held in a peer reward pool for subsequent distribution to Peers. See the Transactions and Juice Accounting CAD for more details. - **Peer Rewards** : **Peer Operators** may claim a share of the peer reward pool as they participate in running the network. See the Peer Staking CAD for more details. ### Coin Supply The issued coin supply is VARIABLE based on coin issuance via the Release Curve or contributor awards. It is denominated in Convex Coins. Each Convex Coin MUST be sub-divided into 1,000,000,000 base units, referred to informally as "coppers" The Network must treat Convex Coins and coppers identically, i.e. the implementation should consider the range of possible coin values to be a value from `0` to `10^18`, where `10^9` is a Convex Coin. Coins that are used for transaction fees (or deliberately burned) are removed from the coin supply and placed in a special Reward Pool that is used to pay rewards to peer operators and stakers. In this way, the coin economy becomes fully circular after initial issuance. The Network MUST implement a technical fixed maximum coin supply cap of 1,000,000,000 Convex Coins. The number of issued coins at any time may be less than this amount, but can never exceed this amount. Note: The maximum supply cap is chosen so that all valid coin balances can be expressed within a 64-bit long value, which allows for efficient implementation on most modern CPU architectures. ### Genesis The genesis process in Convex includes the process of creating the initial Global State and establishing the first peer on the network, to which others can then connect. This genesis state is important for tokenomics because it established the initial coin allocation and the rules by which future coin allocations may be made. #### Top Level Coin Allocation The Network MUST divide the total initial supply of Convex Coins into two quantities: - 75% for Coin Purchasers via the Release Curve - 25% for contributor and ecosystem awards #### Reserve accounts The genesis MUST create a set of reserve accounts (`#1` to `#7`) which represent unissued coins. Such coins MUST NOT be considered part of the current coin supply. By reserving these amounts, we maintain the technical invariant that the total maximum supply cap of 1,000,000,000 Convex Gold is constant and coins cannot be created or destroyed - however the majority of these may not yet part of the current coin supply. Any cryptographic keys for reserve accounts MUST be kept securely and governed according to the release tokenomics described in this CAD. The Convex Foundation will use air-gapped systems initially for this purpose. #### Genesis account The genesis of the network MUST include a genesis account (currently `#11`) that is responsible for initial distribution of Convex Coins to start up the network. The genesis account MUST receive funds from one or more reserve accounts to handle initial coin issuance for the following purposes: - Performing any network setup transactions - Establishing and staking the first peer on the network - Making initial coin distributions to early purchasers and contributors The genesis account MUST NOT have access to the majority of the reserve account funds, beyond what is necessary for the above. #### Distribution account(s) The genesis process SHOULD define one or more secondary distribution accounts that will hold Convex coins temporarily before distribution to award recipients or purchasers. The distribution accounts SHOULD NOT hold large balances of coins, and are only intended for short term holdings of coins that are already allocated to recipients (e.g. purchasers who have purchased coins, but not yet provided a public key or account into which the coins can be delivered). These balances are considered as issued (i.e. part of the current coin supply) but not yet distributed, i.e. still in the control of the governance body. The governance body MUST ensure these accounts are securely controlled by authorised individuals to ensure legitimate distributions are made. ### Memory Exchange Pool A certain amount of Convex Coins are placed in an AMM exchange for CVM memory allowances. Such coins are in effect locked under a smart contract, though should still be considered part of the overall coin supply as they are technically available for use (e.g. people selling back memory allowances). The memory allowances themselves are a secondary native token used purely for memory accounting purposes. See [CAD006 Memory Accounting](../006_memory/index.md) for more details. ### Release Curve The release curve determines the price at which new coins are issued. There is an important economic principle behind this: more **coins only get released when prices go up** (i.e. hit new highs). This gives coin purchasers the assurance that they will never get diluted by new coin releases at lower prices, while still allowing for the coin supply to be increased as ecosystem demand grows. Coin purchases MUST be priced in fiat currency or equivalent, consistent with the Release Curve defined in this section. The price of a Coin on the release curve is defined as `$100 * x / (1-x)` where `x` is the proportion of coins released out of the total allocation for coin purchasers, and `$` represents United States dollars or equivalent currency. ![Convex Coin Release Curve](release-curve.png) Note: The constant value `$100` is chosen so that once `50%` of all coins are issued, the market cap of Convex Coins would be equal to `$50bn`. At this stage, one would expect the Convex Foundation to have a significant treasury available to give strong economic support to the Convex ecosystem in perpetuity. The Release Curve formula MAY be adjusted in the event of significant economic events affecting the relative value of fiat currencies used (e.g. sustained high rates of inflation). The Foundation MUST consult with the ecosystem and provide a robust rationale for any such changes. To account for transaction costs, effective financial management or purchaser convenience, the Foundation MAY group the release of some coins into rounds, provided that such rounds MUST be broadly consistent with the overall Release Curve. ### Coin Purchases The 75% allocation for Coin Purchasers MUST be distributed on the basis of purchases of coins in a manner consistent with the release curve. The distribution MUST be administered by the Convex Foundation, or an entity acting on its behalf in accordance with relevant local regulations. Coins purchases MUST be delivered to purchasers on demand after the release of the main network. Coin distribution MAY start early (during Protonet phase) subject to purchasers accepting the risk of doing so. Coin purchasers MUST meet all relevant regulatory, legal and other requirements that may be defined to ensure that they are permitted to purchase Convex coins in their respective jurisdictions. The Convex Foundation MUST sell coins on an equal, non-discriminatory basis to any purchaser that meets any relevant criteria. The Convex Foundation MAY place restrictions on the rate that coins are distributed to early purchasers, giving consideration to the potential impact of large numbers of coins being issued at the same time. In the long term, distribution of coins from coin purchasers SHOULD be operated on a fully decentralised basis, once technology, financial and decentralised governance capabilities are sufficiently robust to enable such a system (for example, having nation state currencies directly available on Convex). The funds raised from coin purchases MUST be used in accordance with the objectives of the Foundation. For example, these may be used to fund ongoing development of the network, invest in key ecosystem initiatives, and as an incentive to reward those who contribute to the Convex ecosystem. The funds raised SHOULD be managed as a long term fund by the Foundation to ensure the development of the network can be funded on a sustainable ongoing basis. ### Coin Awards The 25% allocation for awards MUST be distributed by the Convex Foundation, in its role as the governance body of the Convex Network. The Convex Foundation MUST make awards in the interest of developing the ecosystem, with a focus on awards to: - Contributors who have made positive contributions to the development of Convex core technology - Contributors who make positive contributions the the development of the community and ecosystem - Infrastructure providers (such as peer operators) who help to ensure the security and reliability of the network - Organisations or startups that help build the ecosystem by developing effective use cases The Convex Foundation MAY require contributors to sign a Contributor's Agreement before award or distribution of coins. The Convex Foundation MAY require contributors to verify their legal identity (KYC requirement) The Convex Foundation SHOULD aim to ensure that the rate of awards remains broadly consistent with the ratio 25% : 75% relative to purchases from the release curve, with the understanding that this ratio may deviate from target in the short term. The Convex Foundation SHOULD explore options for decentralised governance of awards. In the long term, decentralised governance SHOULD apply to all awards. earn future awards. ### Vesting Early coin purchases via the FCPA (up to and during Protonet phase) are subject to a vesting schedule, reflecting the desire that early purchasers should remain committed to the ecosystem for a period of time, and to mitigate the potential destabilising effect of large simultaneous sales of coins. Coin awards will not be subject to any vesting schedule as they are considered already "earned" by contributors. However, contributors are likely to wish to remain involved for other reasons e.g. building applications on top of Convex or wishing to stake their coins in various ways. ### Transaction Fees Transactions executed on the Convex network are subject to fees that reflect the cost of providing underlying network infrastructure to maintain the consensus network and global state. Transaction fees are intended to be small, to encourage adoption and use of the Convex network. Transaction fees MUST NOT be zero to mitigate against denial of service (DoS) attacks on the network. Transaction fees MUST be collected at the point of transaction execution, and placed in the Reward Pool for subsequent distribution to peer operators. This process MUST occur automatically as part of the network protocol. ### Reward Pool The Peer Reward Pool is stored in the special account `#0`. Transactions Fees for execution of transactions are deposited in the Reward Pool (this occurs at the end of each block of transactions successfully submitted by a peer and confirmed in consensus). Over time, this Reward Pool is used to make payments for peers that are participating actively and correctly in maintaining network consensus, thus giving a return to stakers. Account `#0` is also an address to which users can optionally "burn" coins. Such coins are removed from the circulating coin supply, but will be available for future distribution as peer rewards. The Convex Foundation MAY, at its discretion, issue coins and immediately "burn" them in order to increase the incentives for peer operators to participate. Such coins will be considered to come out of the 25% available for contributor awards. ## Other considerations ### Secondary sale Once issued, Convex Coins may be traded on secondary markets, e.g. via private sale or on digital asset exchanges. Such exchanges are beyond the scope of CAD020 and do not effect the overall coin supply (since they represent transfers between users and/or actors with coins that are already part of the coin supply). For example: It is entirely legitimate business model to purchase coins from the Release Curve and offer them for sale to retail users of the network. This is good for the ecosystem because it enables innovation and diversity in the ways that Convex Coins are made available to end users. ### Wrapped / Locked Coins It is possible to lock Convex Coins in an actor account / smart contract. Examples: - Holding a deposit in escrow for a smart contract - Liquidity pools on the Torus DEX - "Wrapping" Convex Coins as CAD029 fungible token Such coins are considered part of the coin supply, even though they are not immediately available for users: the rationale for this is that they can still be redeemed or withdrawn under various circumstances. ### Lost coins It is possible to put coins beyond any possible future use in certain ways, e.g.: - Transferring them to an actor account which has no way to return them - Permanently losing the private key to a user account Such coins are considered "lost". While technically still part of the coin supply, for practical purposes they are gone forever. ### Fully Diluted Value Some tokenomic analysis depend on the concept of "Fully Diluted Value", by multiplying the current price of tokens by some maximum supply cap. Such analysis is generally invalid and SHOULD NOT be applied in the case of Convex, because full dilution cannot occur under the release curve model at the current price and such a number is effectively meaningless. Analysts SHOULD use the market capitalisation of issued coins (i.e. the current coin supply multiplied by the market price) as the overall valuation for the Convex Coin, which more accurately represents the coin ecosystem. This is also consistent with valuation approaches for other tokens that do not have a fixed supply. ### Testnets Testnets (i.e. any Convex based network other than the official main network) MAY alter or ignore any aspect of the tokenomics for testing, development or experimental purposes. Indeed, this is encouraged for the purposes of research and learning. Test networks MAY implement a system of distributing free coins for testing purposes, e.g. a "faucet" which automatically issues testnet coins to users. Participants in the ecosystem SHOULD NOT conduct economically significant activity on test networks, with the understanding that they have no assurance of the tokenomic model or governance over such networks, and may face significant security or legal risks if they do so. ### Disaster Recovery In the event of a high severity event that substantially affects overall tokenomics, the governance body MAY take remedial action, up to and including requiring a mandatory network update to fix the issue. Such action will only be taken as a last resort, and MUST be done in a manner that minimises the impact on legitimate coin holders. The governance body MUST NOT take any action which results in the modification of the coin balances of any 3rd party user or actor accounts, except where necessary to remediate the effects of a security breach as above. --- # CAD021: Observability ## Overview There are many cases where it is valuable to be able to observe events on the Convex network. Examples are: - Peer operators wish to monitor performance of their peer(s) and the service levels they are offering to their clients - dApp developers looking to observe events in the smart contracts they have deployed - Traders looking to monitor price movements to inform their trading decisions - Centralised exchanges monitoring for fraud and other suspicious activity ## Design Objectives ### Pluggable Architecture Different observers can be plugged in to support different use cases. Examples might be: - Sending events to a Kafka topic where they can subsequently be filtered, processed and analysed by various monitoring tools - Visualising events in a realtime monitoring tool ### Opt In The observability model is opt-in from the perspective of peer operators. Peer operators may choose which events they wish to observe, and how they want to consume these on a per-peer basis. ## Solution Specification ### Overview The following topics will be available as observability hooks: - Client Transactions ### Client Transactions Client transactions are transactions that are handled on behalf of a client from a specific peer. The following events related to client transactions will be available - Request (when the peer accepts a transaction from a client) - Response (when the peer retuens the result to the client) The primary key for transactions should be regarded as (Peer Key, Transaction ID, Event type) ### Ordering events Ordering events are any time a peer recieves an updated ordering, directly or indirectly, from another Peer. Important fields: - `peer` = Peer public key - `hash` = Hash of ordering (this can deduplicate identical orderings) - `cps` = Consensus points (including ordering length) - `ts` = Timestamp (at which Ordering validated by peer) ### Consenus Events Events related to consenus update ### Peer status change Peer status changes are any time a Peer has an update in the global state: - Peer added - Peer removed - Peer stake change - Peer metadata change ### CVM Log Events CVM log events are events emitted via the CVM `log` instruction. As this is an application specific `log` capability, it is likely that observers will want to filter this event stream for events of particular interest. Examples might be: - DAO votes - Token trades - Auction bids ### Performance Stats Peer performance stats detail live performance metrics from a peer. Fields include: - Thread-based CPU utilisation (time not blocked) - Internal queue sizes (and total capacity) - Rates of events per second Default is to produce one performance record every second during peer operations ### Message Events Message events are low level peer protocol messages. Key fields: - Event type "SND" or "RCV" - Timestamp - Remote address / port - Message type - Length ## Observability Consumers ### Kafka / Strimzi The Kafka consumer provides a general purpose observability framework This Consumer works on the following basis: - Events are serialised as JSON - Events are posted to a Kafka queue via Strimzi (over https:) - There is some buffering at the client side - Different events are normally routed to different Kafka topics (configurable) ### NOP Consumer The NOP consumer ignores events, and produces no output. It has the advantage of near-zero overhead for Peer operators. The NOP Consumer is the default, unless come other Consumer is specifcied. ## Other considerations ### End user access It is assumed that the observability service will not usually be directly accessible to end users, e.g. clients using a Convex dApp. Peer operators may of course choose to make data from their observability service publicly visible, e.g. performance graphs. --- # CAD022: Trust Monitors ## Overview Trust Monitors are composable, secure, on-chain authorisation modules that can be used to define arbitrary access rules for smart contracts, digital assets and other capabilities needed for open economic systems. The core idea is that a Trust Monitor can grant or deny access based on three parameters: - **Subject** - The entity attempting to perform an action (almost always an Account on Convex) - **Action** - The specific action being performed (usually denoted by a human readable keyword e.g. `:update`) - **Object** - The target object affected by the action (depends on context, but often the ID of an entity being accessed) Examples of possible Trust Monitor implementations: - A list of authorised users who may approve a smart contract action - A time delay before an action may be performed - Authorisation based on possession of a specific type of NFT Trust Monitors are based on the reference monitor model, which was developed for the United States military. It continues to be the case that systems evaluated at level B3 and above under the Trusted Computer System Evaluation Criteria (TCSEC) are required to use the reference monitor model to enforce access controls. ## Design Objectives ### Pluggable Architecture Trust monitors are designed to be pluggable so that they can be re-used in different contexts. It is common that different systems may want to grant the same access rights, and hence the monitors themselves should be easily re-usable. ### User controlled Trust monitors should normally be specified by the person who has control rights over some resource. For example, if a user is running an auction, they might wish to provide a trust monitor to define who has the right to cancel the auction. ### Sandboxing It is frequently desirable that access to trust monitors is fully sandboxed, in particular they should not be able to perform any on-chain action that might harm the caller (e.g. a re-entrancy attack). This danger is particularly acute given that: - Trust monitors represent delegation to potentially untrusted code - We want to be able to allow people to provide their own trust monitors As such, we want to be able to use trust monitors in `query` mode. ### Efficiency We wish to allow trust monitors to be widely and cheaply used. As such, they should be efficient in terms of both memory and juice costs. ## Specification ### Trust Monitor Reference A trust monitor SHOULD be referenced as an account address with an optional scope. Examples: ``` - #45 ;; A trust monitor as a account - [#78 1467476] ;; A trust monitor as a scoped account ``` The target account SHOULD exist. Attempts to use a trust monitor with a non-existent account are possible but likely to error. The target account MAY be a user account. The reference MAY be `nil`, which behaves as a trust monitor that never provides any authorisation. ### `check-trusted?` SPI A Trust Monitor MUST implement the callable function `check-trusted?` as a common SPI to support its use according to this standard. The `check-trusted?` function must accept three arguments corresponding to `subject`, `action` and `object`. Example: ``` (defn ^:callable check-trusted? [subject action object] (boolean (and (= subject object) (= action :examine-self)))) ``` The Trust Monitor MUST return `true` or `false` for all possible argument values. Callers may expect these specific values, so returning anything else risks breaking compatibility with some applications. Checkers however MUST NOT rely on this contract being honoured — see the defensive checking requirements under "Trust checks" below. The Trust Monitor MAY implement arbitrary access control logic on the basis of the arguments provided and the current CVM state. This might include looking up values in an on-chain database or calling other actors for confirmatory information. The Trust Monitor SHOULD avoid excessive computation and stack depth usage. Ideally it SHOULD be guaranteed `O(1)` in both, with a small constant. Use of pre-computed Sets or Maps for lookups in data structures is recommended: it is unsafe to perform scans of arbitrary data structures in a trust monitor context, since this could enable DoS attacks. The Trust Monitor MUST NOT rely on side effects, and MUST operate correctly when called within a `query` context. This requirement is to ensure that callers can wrap a trust monitor call in a `(query ...)` or similar construct to avoid potential security risks (e.g. re-entrancy attacks from some malicious nested code). ### Trust checks A trust monitor MUST return `true` or `false` for any `[subject action object]` combination, and SHOULD never fail (except for hitting resource constraints like `:JUICE`). Checkers however MUST NOT rely on external monitor implementations honouring this contract: a defective or malicious monitor may throw an error or return a non-boolean value. A checker MUST be **defensive and fail closed** — a monitor error or non-`true` result MUST be treated as a denial, and MUST NOT be allowed to propagate as either a grant or an error that disrupts the caller. (An error-propagating checker also creates a denial-of-service shape: an actor holding an attacker-supplied monitor reference would throw on every check.) When checking a `[subject action object]` combination against a trust monitor the following procedure SHOULD be performed: 1. If the trust monitor is an account implementing a callable function `check-trusted?` then call that function protected by a query (against re-entrancy), catching any error as denial and coercing the result to a strict boolean: `(boolean (try (query (call trust-monitor (check-trusted? subject action object))) false))` 2. If the trust monitor is an unscoped Address, and is precisely equal to the `subject`, then return `true` 3. Return `false` This is the behaviour of the reference `trust/trusted?` from protocol version 1: monitor errors are caught and yield `false`, and the result is always a strict `true` or `false` regardless of what the monitor returned. A monitor MAY still fail internally on malformed input (e.g. a bad scope) — the standard genesis monitors do — since the checker contains such failures as denials. If the trust monitor is known and trusted, then the caller MAY simplify the above procedure and call the SPI directly: ``` (call trust-monitor (check-trusted? subject action object)) ``` ### Subjects Subjects of Trust Monitor checks SHOULD be valid unscoped Convex account addresses. In most cases, the subject will be the `*caller*` of some actor code which needs to perform authorisation checks on the caller. ### Actions Actions SHOULD be short human readable keywords, e.g. `:update` Actions MAY be any CVM value, which could include a data structure describing the action in more detail. We urge caution on making actions too complex: it would be easy to introduce tricky security bugs. Actions SHOULD NOT depend on information provided by or subject to influence by untrusted users. In most cases, the action should be hard-coded to a specific value relevant to the context of the authorisation check being performed. ### Objects Objects SHOULD be the canonical identifier of the object being acted upon. Typically this is some information or resource which is protected by the authorisation check. Most common object types are likely to be: - A Convex address (possibly scoped) - An integer ID ## Reference Implementation notes ### `convex.trust` library The `convex.trust` library provides a canonical CAD22 compatible interface to trust monitors via the `trusted?` function: ```clojure (import convex.trust :as trust) (trust/trusted? monitor subject) (trust/trusted? monitor subject action) (trust/trusted? monitor subject action object) ``` Action and object are optional: if omitted, they are passed to the Trust Monitor as the value `nil`. From protocol version 1, `trusted?` is defensive as specified under "Trust checks": it always returns a strict `true` or `false`, treating any monitor error or non-`true` result as a denial. ### `convex.trust.monitors` library The `convex.trust.monitors` library provides a set of lightweight standard trust monitor implementations. These are designed to be *composed*, i.e. they are building blocks which can be used to create more sophisticated trust monitors and associated governance functionality. ```clojure (import convex.trust.monitors :as mon) ;; Permit a specific set of subjects (effectively a fixed whitelist) (trust/trusted? (mon/permit-subjects #3 #14 #17) #14) => true ;; Permit a specific set of actions only (trust/trusted? (mon/permit-actions :open :close) #14 :delete :some-target) => false ;; Permit something that satisfies ALL of the given trust monitors ;; This is an example of composing trust monitors (trust/trusted? (mon/all (mon/permit-actions :open :close) (mon/permit-subjects #13 #17)) #13 :open :some-target) => true ;; Permit based on calling a function on (subject, action, object) ;; This can be a good way to allow subjects to control resources that they logically "own" (trust/trusted? (mon/rule (fn [s a o] (= s o))) #16 :foo #16) => true ``` --- # CAD023: Keystore ## Overview Strong security of private keys is essential to many decentralised systems, including Convex In many circumstances, it is useful for users to keep a local keystore for their hot / warm wallets on their device. Common usage scenarios include: - Generating and using keys on the CLI - Providing a persistent store of keys for running a peer - Making keys available to local applications (e.g. the Convex GUI) This CAD outlines standards and principles for the use of such key stores. ## Design Objectives ### Use of existing standards We avoid creating new standards for key storage and security, and prefer to use existing, proven methods. Key reasons for this include: - Security / reliability of well designed existing standards - Compatibility with external tools - Minimise risk that important key stores may be unreadable opr inaccessible in the future - No point wasting effort reinventing wheels ### Convenient use Cryptographic keys need to be easy to use for users, while providing sufficient security. In particular, key security should not be hard for users to do correctly. We do not want users to compromise security for "convenience". ### Sensible defaults Default security should be strong, and certainly "good enough" for most plausible use cases. Any user decision that weakens security should come with clear warnings and require explicit decision on the part of the user. ### Multiple levels of security As far as possible, security should include multiple levels of protection so that a mistake or vulnerability in one level does not immediately lead to a major security compromise. Levels of security anticipated include: - Key stores should be private files inaccessible to aunauthorised users - Key stores should be encrypted, so that obtaining the file does not provide access to the key - Decryption passwords should not be stored anywhere on the machine - After decryption, private keys are only kept in memory for immediate use ## Specification ### Peer Usage Peers SHOULD obtain a key from a secure key store at startup, and use this key for the duration of live operations where signatures using the peer's private key are required (for example signing their latest CPoS ordering). Peers SHOULD NOT persist or otherwise export the obtained key in any way to external files or storage. Any unencrypted copies of the private key should be considered as a major security risk. ### PKS #12 A key store for Convex MAY use the PKS #12 standard for key storage. This standard is well established, is supported by many tools, and provides effective security if used correctly. Assuming that PKS #12 is chosen as a key store: Implementations SHOULD use the hex string representation of the public key as an alias to each key in the store, to enable easy user identification and automatic lookup as required. Implementations SHOULD require both a key store password and a password for each private key Users SHOULD be advised to use strong passwords Users SHOULD be warned if they use an empty / blank passwords. Implementations MAY prohibit this for extra security. ### Password input Password input SHOULD be via appropriate mechanisms that protect password privacy , e.g. via a secure password TTY prompt that disables echoing. Implementations SHOULD clear memory for any passwords immediately after they are used. For example, zeroing memory in a character array. ### PEM Export Convex implemetations SHOULD support Ed25519 private key export in encyrpted PEM format. Implementations should use at least 4096 iterations for encryptions, as well as advising users to choose a strong password The encryped format in ASN.1 form SHOULD resemble the following: ``` U.P.SEQUENCE { U.P.SEQUENCE { U.P.OBJECTIDENTIFIER 1.2.840.113549.1.12.1.5 (PKCS #12 Password Based Encryption With SHA-1 and 128-bit RC2-CBC) U.P.SEQUENCE { U.P.OCTETSTRING 98d43621377dbe5377fc5cf424f0da80b38c0966 U.P.INTEGER 0x0400 (1024 decimal) } } U.P.OCTETSTRING 773a34486cbd07b92dc597e9b96b8e556ea44eaaff4e167333552543ec18a66738362e699fc5655f3b8a26ad17fbd011186f2589bc316e32 } ``` --- # CAD024: Data Lattice ## Overview The Data Lattice is the data substrate of the Convex ecosystem, designed for high volume storage, high performance and verifiable content delivery at scale. Convex maintains data as part of the operation of the CVM global state (on-chain). It is possible to build decentralised solutions using these capabilities alone which we call a "pure dApp", which is sufficient for many use cases. However, more sophisticated decentralised applications are likely to require additional data sources (off-chain). We call such applications "hybrid dApps". While off-chain data can be provided in many different ways, including via traditional Web 2.0 server infrastructure, there is a compelling case for a more powerful decentralised storage solution to support hybrid dApps on Convex ## Rationale Here are some of the key reasons data storage and processing extrinsic to Convex is useful for decentralised applications: 1. Scalability: Global-state DLTs inherently have limits on computational capacity and data throughput. Storing and analysing large data sets is inefficient on-chain. Off-chain resources that do not require the same degree of synchronisation via consensus can horizontally scale to demand. 2. Privacy: Sensitive data as with health records or personal financial information requires restricted access. Public blockchains provide transparency but not confidentiality. Off-chain systems support encryption and fine-grained access control. 3. Speed & Responsiveness: DLTs feature lengthy settlement finality and limited transactions per second. This hinders uses needing real-time data flow like messaging or IoT monitoring. Off-chain networks have can react faster to data changes. 4. Cost Efficiency: Persisting every small update on-chain becomes expensive long term. Offloading data which does not require use of the global on-chain state curtails unnecessary fees and bloat. Subnets handle localised logic. 5. Regulatory Compliance: Chains are transnational making compliance difficult. Off-chain systems can enforce jurisdiction specific rules around permissible data types, storage locations, access restrictions and lifecycles. 6. Rich Data Formats: Chains restrict data schemas to basic key-value pairs and byte strings. Document databases, media assets, and complex analytic jobs operate off-chain. In summary, a hybrid decentralised model using blockchains for consensus/settlement alongside purpose-built off-chain coordination handles data-intensive tasks aligned to use case needs more effectively. This unlocks wider adoption. ## Capabilities The Data Lattice provides the following capabilities: - **Content Addressability**: all data is addressed and indexed by a cryptographic hash. The near impossibility of hash collisions ensures that having the correct hash always allows location of the correct data. - **Lazy loading**: data can be partially loaded on demand, allowing many processes to proceeded without downloading complete data structures. This enables many capabilities such a streaming media, and storage volumes beyond the memory size of individual nodes in the network. - **Verifiability**: all data can be verified in its entirety from the cryptographic hash. Because the data takes the form of a Merkle Tree, this verifiability extends to multiple levels - **Structural Sharing**: all common subtrees (i.e. sharing the same content and cryptographic hash) are automatically shared and de-duplicated. This property arises naturally from content addressability and the Merkle Tree structure, which in turn allows for efficient operations such as cloning and storing modified copies of any data. - **Rich Data Types**: The Lattice supports a wide variety of data structures, including all data types available on the CVM such as maps, lists, vectors, numbers, strings and arbitrary blobs of byte data. In particular, the data lattice types support a superset of JSON, so JSON objects can be naturally represented with ease. - **CRDT support**: The data lattice forms a natural CRDT, where arbitrary sets of data can be merged to create a union of all data. This process is aided by automatic de-duplication to reduce storage and transmission costs. - **Self healing** - The CRDT also makes the data lattice "self-healing": nodes which lose access to some data (e.g. due to disk corruption) may obtain it again on subsequent merges as long as at least one copy survives somewhere in the network. - **Garbage Collection**: Stores can be garbage collected to reduce resource requirements at any time, simply by specifying which data is required to be maintained ("pinning"). This facilitates better operational management and allows flexible control by data lattice providers regarding what data they are interested in preserving or hosting. - **Access Control**: data lattice hosts may optionally impose whatever access controls they require for governance, security or privacy purposes. Typically, these would involve authentications against a decentralised ID (DID) and a digital signature (Ed25519 as standard, though other systems can also be adopted) ## Specification ### Fundamental Principles The Data Lattice is built on a set of core principles that together enable self-sovereign, peer-to-peer data structures that scale to internet level. #### Self-Sovereign P2P Data Data in the Lattice is inherently self-sovereign. Each participant owns and controls their own data, signed with their Ed25519 key pair. There is no central authority that mediates access or dictates structure. Participants publish data into the Lattice and other participants may choose to replicate, merge and act upon it — but the original author retains cryptographic proof of authorship and control over their own slice of the data structure. The Lattice MUST partition data by owner public key. Each owner's data MUST be wrapped in a cryptographic signing boundary that enforces signing at the point of mutation. Only the holder of the corresponding private key may create or update values within their partition. Any reader MUST be able to independently verify the authenticity and integrity of data they receive without trusting the peer that delivered it. #### Replicated, Self-Healing Structures The Lattice is designed for replication across an open network of peers. Any peer may hold a partial or complete copy of the Lattice state. Because merge operations are commutative, associative and idempotent (see CRDT Properties below), peers can synchronise in any order, at any time, and always converge to the same result. This makes the Lattice inherently self-healing. If a peer loses data due to disk corruption, network partitioning or any other failure, it MUST recover that data on the next successful merge with any peer that still holds it. No co-ordination protocol is required — the algebraic properties of the merge guarantee convergence. As long as at least one copy of any datum survives somewhere in the network, it can be recovered by all peers through normal replication. #### Atomic Updates All updates to the Lattice MUST be atomic. Observers MUST never see partially applied changes. A fork/sync model provides transactional semantics: `fork()` creates an isolated working copy where local changes accumulate without affecting the parent. When `sync()` is called, all changes are applied to the parent atomically via lattice merge. Because merge is always well-defined (see CRDT Properties below), merge-aware atomic updates never fail or require retry — the merge function deterministically combines the local changes with whatever concurrent state the parent holds. #### Time Travel and Snapshots Because all Lattice values are immutable persistent data structures (Merkle Trees), the Lattice naturally supports snapshot and time-travel capabilities analogous to Git. Every state is identified by its cryptographic hash. Forking creates a snapshot at that point in time. Previous states remain accessible as long as the underlying cells are retained in storage (i.e. not garbage collected). This enables powerful patterns: historical queries, audit trails, rollback, branching workflows, and diff-based synchronisation between peers. The structural sharing property ensures that snapshots are storage-efficient — only the cells that differ between versions occupy additional space. #### Fully Mergeable CRDTs at Internet Scale The Data Lattice forms a Conflict-free Replicated Data Type (CRDT). Every lattice type MUST define a `merge` function with the following algebraic properties: - **Commutativity**: `merge(a, b) = merge(b, a)` — merge order does not matter - **Associativity**: `merge(merge(a, b), c) = merge(a, merge(b, c))` — grouping does not matter - **Idempotence**: `merge(a, a) = a` — re-merging the same data is a no-op These properties guarantee that any number of peers, merging in any order, with any degree of message duplication or reordering, will always converge to the same final state. This eliminates the need for complex consensus protocols for off-chain data — eventual consistency is guaranteed by the mathematics of the lattice. The CRDT merge extends hierarchically through the entire Lattice structure. The root lattice merges by delegating to child lattices at each key, which in turn delegate to their children, and so on. This compositional design means the full Lattice — from the root down to individual data entries — is a single, coherent CRDT that can scale to internet level. #### Cursor-Based Application Interface Applications interact with the Lattice through cursors — lightweight handles that provide a uniform interface for reading, writing, navigating and synchronising Lattice data. Cursors abstract away the details of atomic updates, lattice merging, cryptographic signing and hierarchical navigation, allowing application developers to focus on their domain logic. ### Lattice Types A Lattice Type defines the merge semantics for a particular kind of value. Every lattice type MUST define: - `merge(ownValue, otherValue)` — the core merge function, which MUST satisfy the CRDT properties (commutativity, associativity, idempotence) - `zero` — the identity element of the lattice (i.e. `merge(zero, x) = x` for all `x`) - `validate(value)` — validation that a received value is well-formed for this lattice type - `child(key)` — returns the child lattice type for a given key, or nil if the lattice has no defined child structure at that key Lattice types compose hierarchically. A parent lattice delegates merge of child values to the appropriate child lattice type. This enables construction of arbitrarily deep, well-typed lattice structures from simple building blocks. #### Standard Lattice Types The following standard lattice types are defined: **Set Lattice** — Merges by set union. The result of merging two sets is a set containing all elements from both. Zero is the empty set. **Max / Min Lattice** — Merges by taking the maximum (or minimum) value. Useful for monotonically increasing counters, timestamps, or version numbers. **LWW Lattice** (Last-Write-Wins) — Merges by selecting the **whole value** with the most recent timestamp; it never recurses into the inner structure. In the case of equal timestamps, a deterministic tiebreaker MUST be applied to ensure a stable result. Because the newer value replaces the old wholesale, whole-value LWW makes **deletions durable**: a smaller value carrying a newer timestamp replaces the larger old one, and removed entries do not reappear on merge. This is what lets applications delete directly rather than accumulate tombstones. **JSON Lattice** — A structural navigation layer for JSON-shaped data (maps, vectors, strings, numbers, booleans and nil). It defines how to navigate and how to build intermediate containers from the shape of a key, but deliberately defines **no merge of its own** — it is composed under a merge layer. **Stamping Lattice** — A stamp-on-write layer that stamps values with the write timestamp from the merge context as they are written, without altering merge or navigation. Composed as `StampingLattice(LWWLattice(JSONLattice))`, these three single-concern layers together give a navigable, whole-value-LWW region with durable deletes — the standard shape for mutable application state. **Map Lattice** — Merges maps by applying a value lattice to each key independently. Keys present in only one map are included directly. Keys present in both maps have their values merged using the configured child lattice. Zero is the empty map. **Index Lattice** — Similar to Map Lattice but operates on sorted indexes with blob keys. Suitable for ordered data such as time-series feeds. **Keyed Lattice** — A map-like lattice where each key has a distinct, statically configured child lattice type. Used for the root Lattice structure where different sections (e.g. data, filesystem, key-value, queue) each require different merge semantics. **Owner Lattice** — Partitions data by owner public key. Each owner's slot is wrapped in a Signed Lattice. Merge operates independently per owner. This is the foundation for self-sovereign data — each owner controls their own partition via their signing key. **Signed Lattice** — Wraps a child lattice in a cryptographic signing boundary. Values MUST be stored as signed data and verified on merge. Signing on write is enforced at the boundary by a `SignedCursor` (see [CAD035](../035_cursors/index.md)), which signs values with the merge context's key as they are written — rather than by re-signing during merge. This ensures data integrity and provenance throughout the lattice hierarchy. **Data Lattice** — A general-purpose content store indexed by hash. Merge is set union over the hash-indexed cells. Useful for bulk data storage where individual items are referenced by content hash. **Local Lattice** — Stores peer-local data that is not replicated. Useful for configuration, caches, or other node-specific state. ### Root Lattice Structure The system-wide Lattice root is a Keyed Lattice with the following standard sections: - **Data** — General-purpose content-addressable data store. Stores arbitrary cells indexed by hash. - **Filesystem** — Distributed Lattice File System. An Owner Lattice where each owner may maintain named drives, each containing a tree of directories and files. - **Key-Value** — Key-value databases. An Owner Lattice where each owner may maintain named key-value stores with per-entry merge semantics. - **Queue** — Message queues. An Owner Lattice where each owner may maintain named topics with partitioned, append-only message streams. - **P2P** — Peer-to-peer network state, shared between peers to co-ordinate replication. - **Local** — Peer-local storage that is not replicated across the network. Applications MAY extend the root structure with additional sections by registering new keys and lattice types. For example, a social networking application could register a section with an Owner Lattice containing per-user social state (feeds, profiles, follow graphs), each with appropriate merge semantics. ### Cursors Cursors are the primary interface for interacting with Lattice data. A cursor is a lightweight handle to a position within the Lattice hierarchy, providing atomic operations for reading, writing, navigating and merging values. In particular, lattice-aware cursors extend the base cursor model with fork/sync semantics: forking creates an isolated working copy, and syncing pushes changes back to the parent via lattice merge — which always succeeds without retry. The full cursor specification is defined in [CAD035: Lattice Cursors](../035_cursors/index.md). When used with the Data Lattice, cursor path navigation MUST be lattice-aware: at each level, the cursor consults the lattice type to determine the child lattice for the next key, and when a Signed Lattice boundary is crossed, cryptographic signing MUST be enforced transparently. ### Merge Context Merge and write operations may require contextual information beyond the two values being merged. The merge context (`LatticeContext`) MUST provide: - **Timestamp** — the single write clock. Used by LWW Lattice for conflict resolution and by stamp-on-write regions to stamp values as they are written (the same clock DLFS uses for node update times) - **Signing key** — used by a Signed Lattice boundary to sign values on write - **Owner verifier** — used by Owner Lattice to verify that a signer is authorised to write to an owner's partition The context MUST be propagated through the lattice hierarchy and made available to merge and write functions as needed. ### Cursor Sync Guarantees The fork/sync model has specific guarantees that applications MUST observe: 1. **`sync()` fires the cursor's `onSync` callback** — this is the mechanism by which persistence layers are notified of changes 2. **`ForkedLatticeCursor.sync()` merges into the parent but does NOT fire the parent's `onSync`** — the fork deliberately does not propagate the sync signal upward. Applications requiring full persistence MUST call `sync()` on both the fork and the root cursor. 3. **Path-derived cursors are transparent views** — writes through a path cursor land directly at the parent's storage. They are not independent snapshots. 4. **Broken cursor chains throw `IllegalStateException`** — cursor chain integrity is validated; invalid chains are rejected rather than silently producing incorrect results. This asymmetry is intentional. It allows applications to batch writes in a fork and control exactly when the persistence layer is notified. For example, a venue may accumulate many writes in a forked cursor and periodically sync both the fork (to merge into root) and the root (to trigger persistence). ### P2P Replication Peers replicate Lattice data using a propagation model: 1. **Announce** — When a peer produces new data (via local writes or received merges), it computes the delta (novel cells not previously present). 2. **Persist** — The merged root value is persisted to durable storage, enabling recovery after restart. 3. **Broadcast** — The delta is broadcast to connected peers. 4. **Receive and Merge** — When a peer receives a broadcast, it merges the received value into its local state. The CRDT properties guarantee convergence regardless of message ordering, duplication or partial delivery. This model requires no central co-ordinator. Peers form an open mesh and replicate data transitively. The commutativity and idempotence of merge mean that redundant messages are harmless and message ordering is irrelevant. ### Data Types The data lattice supports the full set of decentralised data values used in the Convex CVM. This enables the construction of arbitrary data structures. In practice, applications are likely to rely primarily upon composing data structures from the following types: Maps, Indexes, Vectors, Sets, Strings, Keywords, Integers, Booleans, Doubles and Signed Data. Applications SHOULD consider whether there is an advantage to limiting usage to the subset of these that represents JSON (Numbers, Maps, Vectors, Strings, Booleans and `nil`). This enables easy one-to-one mapping to JSON representations. ## Reference Implementation The reference implementation is provided in the `convex-core` and `convex-peer` Java modules. Key implementation notes: - **Etch** is the storage subsystem, specialised for efficient storage of content addressable Merkle Trees. - The Data Lattice operates using the same efficient **binary protocol** used by Convex peer-to-peer communication. Peers SHOULD support hosting Data Lattice access on a different port from CPoS / peer communication. - A **REST API** provides access to key data lattice capabilities: insert data, retrieve data, pin/unpin data, access controls and replication. - The implementation is **lock-free**, using atomic references containing immutable values. Merge-aware atomic updates ensure progress without retry. Multiple threads may concurrently read, write, fork and sync without external synchronisation. - Cursor types in the implementation include root cursors, forked cursors, descended cursors (for sub-path navigation), and update-on-write cursors — the shared `AUpdateCursor` funnel with its two instances: signed cursors (inserted at Signed Lattice boundaries) and stamped cursors (inserted at stamp-on-write regions). ## See Also - [CAD002: CVM Values](../002_values/index.md) — Immutable value types used in the Lattice - [CAD003: Encoding](../003_encoding/index.md) — Binary encoding and content addressing - [CAD035: Lattice Cursors](../035_cursors/index.md) — Full cursor specification - [CAD036: Lattice Node](../036_lattice_node/index.md) — Networked lattice replication --- # CAD025: Convex HD Wallets ## Overview Convex requires Ed25519 key pairs for users to sign transactions and provide account security. Any valid Ed25519 key pair will work with Convex, however it is helpful for wallets to generate key pairs in a way that is: - Able to produce a large number of hierarchical, deterministic key pairs from a single root key - Easy for users to understand and manage - Compatible with existing wallet standards ## Seed Phrases Wallets SHOULD allow for seed generation accounting to the [BIP39 standard](https://github.com/bitcoin/bips/blob/master/bip-0039.mediawiki) As such, wallets can be restored completely using the combination of: - A mnemonic seed phrase such as `share resource quantum rely train chicken they plug amazing` - A passphrase (considered optional, if not provided acts as the empty string `""`) A BIP39 seed, consisting of 64 bytes, SHOULD be created from the seed phase and passphrase according to BIP39 ## Heirarchical Key Organisation (BIP44 compatibility) There is no strict requirement for HD wallets to organise keys in any particular way, however compatibility with the BIP44 standard is strongly recommended. ### Option 1: Deterministic from Convex Accounts Wallets MAY generate Ed25519 key pairs in accordance with the BIP43 / BIP44 standard, in the format: `m/44/864//0/` Where: - `` is the address of the Convex account for which the key is specified - `` is the sequence number of the transaction used to set the public key - `864` is the registered coin type for the Convex Coin (CVM) in [SLIP-0044](https://raw.githubusercontent.com/satoshilabs/slips/master/slip-0044.md) If the address is not known at the time of key generation, wallets SHOULD generate a temporary key pair, and rotate the public key of the account to the above once the address is known. The advantage of this method is that if the user knows their Convex address, it is possible to attempt a relatively small number of lookups to find the correct keypair from the wallet (possibly just one, if the sequence number of the last key rotation is known). ### Option 2: Conventional BIP44 Alternatively wallets MAY generate Ed25519 key pairs using the conventional BIP44 approach: `m/44/864//0/` In this case, wallets / apps bear a greater responsibility for handing the mapping from wallet accounts to Convex accounts / addresses. ## Key Derivation (SLIP-0010) Keys SHOULD be derived according to [SLIP-0010](https://github.com/satoshilabs/slips/blob/master/slip-0010.md), which produces Ed25519 key pairs in a manner compatible with BIP32 / BIP39 / BIP43 / BIP44 etc. Note: Ed25519 key generation with SLIP-0010 is assumed to use hardened key generation by default, as this is the only option supported for Ed25519. ## Key Rotation Wallets SHOULD periodically rotate the public key to provide security against key compromise Wallets SHOULD use the sequence number of the transaction used to rotate the key as the ``. This ensures that: 1. the rotated key can be re-derived by observing transaction / state history 2. there is no risk of re-using the same key because sequence numbers increase monotonically e.g. assuming the key is rotated in the first transaction for account `#1234`, the derivation path is `m/44/864/1234/0/1` ## Account linkage An app SHOULD remember the user's addresses and which key pair it is associated with. An app MAY query the known addresses and attempt to find a compatible key pair from the user's wallet. ## Test Vectors Seed Phrase: `hold round save brand meat deposit armed idea taste reunion silent pair estate ladder copper` Passphrase: `test` BIP39 Seed: `d46c4e60d0137e7ee0acc8b836d76d9a0458705caa128899709f576bade690b3c7cba49ece50a211193b9eb7803be49d02c8ddae02c3b88790ac17fa72f219a6` Derivation path: `m/44/864/1234/0/1` Ed25519 Private key seed: `23a4270c0336913c75ce391e6ec69d50d0a1b87fb1760dfd6bbc789b51905ddd` --- # CAD026: Convex Lisp ## Overview Convex Lisp is a general purpose, high level programming language for the Convex Virtual Machine (CVM), designed to facilitate effective construction of smart contracts, digital assets and open economic systems. This document outlines the key elements of Convex Lisp. It is intended primarily as an **introduction and programmer's guide** for experienced developers wishing to understand the language, its implementation and key features: more detailed specifications for specific aspects of the language and underlying runtime are provided in other CADs. ### Key language features - Pure **immutable data structures** with highly optimised implementations for usage in decentralised systems based on Convex - Emphasis on **functional programming** with support for the **lambda calculus** - Powerful **macro capabilities**, following the expansion-passing style developed by Dybvig, Friedman & Haynes - Automatic **memory management**, including [memory accounting](../006_memory/index.md) - Elegant **Lisp syntax** largely inspired by Clojure - **On-chain compiler** (smart contracts writing smart contracts....) - **Strong dynamic typing** - for well defined, consistent behaviour at runtime ![XCCD Cartoon](https://imgs.xkcd.com/comics/lisp_cycles.png) (Image courtesy of [XKCD](https://xkcd.com/)) ### Motivation As a platform for decentralised, open economic systems Convex requires a powerful and flexible language for developers to build the next generation of digital assets, smart contracts and services. While the CVM itself is language agnostic, we chose a Lisp dialect as the first language for Convex for a number of key reasons: - Productivity: general recognition of Lisp as a productive and flexible language - The advantages of a homoiconic language ("code is data") for generating code with powerful macros and DSL capabilities - The ability to create a small and efficient [compiler](../008_compiler/index.md) suitable for on-chain code generation and compilation - Familiarity for developers of existing Lisp-based languages such as **Clojure**, **Scheme**, **Racket** or **Common Lisp** Designing a new language is no easy task, so we naturally considered adopting an existing language for Convex. Unfortunately, none of the available options proved attractive for a variety of reasons: - High level general purpose programming languages (Python, Java, JavaScript, Clojure, C# etc.) are not generally designed for decentralised VM operation. In particular, execution must be deterministic to allow independent execution and validation of the same exact computation by peers in a decentralised network. Cutting down such languages to a consistent deterministic subset (no IO, no randomness, no observable differences across platforms etc.) would itself be a massive task and break compatibility with the majority of existing code, negating most of the value of existing library ecosystems. - Low level languages (WASM etc.) do not provide the high level capabilities and abstractions needed for productive development of decentralised economic systems. The amount of library support required to provide this would be a significant development and performance overhead. Typically, such languages also lack good support for automatic memory management which is important for developer productivity and essential for the kinds of immutable data structures used in Convex. These languages would also imply complex toolchains and execution infrastructure that would add complexity and inhibit development and maintenance efficiency. - Existing smart contract languages (Solidity etc.) have significant limitations / design flaws and would not allow us to take full advantage of the power of the CVM (e.g. the improved account model with key rotation, memory accounting, extra CVM data structures etc.). - Convex performance at global scale depends heavily on very efficient immutable data structures that are not easy to represent in existing languages that were not designed with these in mind. No existing language would be a good fit for these natively, and translating to / from these structures would imply an unacceptable performance penalty: we need to use them directly. ### Discussion and contributions Questions or discussions on Convex Lisp including design choices and potential improvements are encouraged on the [Convex Discord](https://discord.com/invite/xfYGq4CT7v) in the `#language-design` channel. ## Interactive development Convex Lisp is designed to support *interactive usage*. By compressing the traditional write / compile / test / debug cycle, developers can write code more efficiently and work directly with (including modifying) a running program. Typically, developers will make use of a REPL (Read-Eval-Print-Loop) which executes expressions directly to return a result (and possibly also modifying the CVM state). REPLs can be used in local test environments, or directly on a deployed Convex network. It is possible (though *not usually recommended*, for obvious security and risk reasons) to enable REPL capabilities on live production code. This capability can be used to hot-fix and upgrade running CVM code. Examples given in this CAD are suitable for execution at a Convex Lisp REPL. ## Expressions and Forms Convex Lisp code operates through the evaluation of expressions. Expressions are represented as CVM data structured as a "form" - a data value that represents code. In this sense "code is data" because the language is represented in its own data structures. This property is sometimes termed *homoiconicity*. Typically, a form is a List where the first element represents the operation and the following elements represent arguments, e.g.: ```clojure (operation arg1 arg2 .... argN) ``` Each element is itself a form - so forms can be nested to construct more complex expressions. At the lowest level, forms will be a single (non-compound) data value that does not contain any further elements. These are sometimes called "atoms" in Lisp literature: ```clojure ;; the following are all atomic forms 1 hello "This is a string" ``` ## Data types Convex Lisp provides a rich set of data types suitable for general purpose development. These include: - A *superset of JSON* for easy interoperability with web based systems - Immutable persistent data structures with automatic structural sharing - Binary Blobs for arbitrary user-defined data and interoperability with external systems - Indexes for database implementations with efficient sorted keys Convex Lisp directly uses the data types provided natively by the CVM, for maximum efficiency. For more detailed specification of CVM data types see [CAD002](../002_values/index.md) ### Basic Literal values Literal values are expressions that evaluate to themselves (as a constant). These include numbers, strings, booleans etc. #### Integers Integers are positive or negative integer values. Most typical mathematical operations are supported. ```clojure 1 => 1 (+ 2 3) => 5 ``` The CVM supports "Big Integer" mathematics natively, with values up to 2^32768 (4096 bytes). Typically, most integers require less than 64 bits, so are stored efficiently as 64-bit long integers in the CVM implementation, switching to a Big Integer representation only when required. To the user, there is no visible difference apart from paying somewhat higher transactions fees to fairly reflect the additional computation requirement. ```clojure ;; A large negative big integer -9999999999999999999999999999999999999999 => -9999999999999999999999999999999999999999 ;; Big integer mathematics (this would overflow 64 bits) (* 100000000000000000000001 987654321) => 98765432100000000000000987654321 ``` Key motivations for including big integer support in Convex Lisp include: - Avoiding the risk of numerical overflow when performing computations on large numbers such as asset balances - Supporting 256-bit integers commonly found in other decentralised systems - Supporting cryptographic applications which rely on large integers (e.g. 4096 bits) #### Doubles Doubles are 64-bit double precision floating point vales, as specified in IEEE754 ```clojure ;; Double values are literals, just like Integers 12.45 => 12.45 ``` See below for more details on floating point support in Convex Lisp #### Booleans Booleans are either `true` or `false` ```clojure true => true false => false ``` Booleans are primarily used in conditional expressions, or as return values from predicates. #### Characters Characters are Unicode code points (equivalent to 1-4 bytes in UTF-8 representation) ```clojure ;; character literals can be specified with a leading \ \a => \a ;; characters can also be specified by numerical Unicode code points (char 65) => \A ``` Characters are mostly useful for constructing Strings. Characters may also be used as efficient small keys in Maps or Sets. The first 256 code points (equivalent to ISO-8859-1 characters) encode to just 2 bytes and are efficiently cached, though care should be taken that some of these values are not always correctly printable. #### Strings Strings are arbitrary length, immutable UTF-8 strings ```clojure ;; A a literal String evaluates to itself "Hello" => "Hello" ;; Constructing a String (str "A" "BCD" "EF") => "ABCDEF" ``` Strings are provided to enable human readable output, for programmer convenience and for compatibility with JSON. Typically, string processing should be avoided on the CVM - while possible this is usually best done on the client side or on a separate server backend. #### Blobs Blobs are arbitrary length sequences of byte data. These are particularly useful in Convex for: - Representing cryptographic values such as hashes and Ed25519 public keys - Allowing applications to store custom data in their own encodings Blobs can be easily constructed as literals by prefixing `0x` to a hexadecimal representation of the byte data ```clojure ;; A small Blob literal 0x1234ee99 => 0x1234ee99 ;; Can use functions like `count` to get the length of a Blob (count 0x12e5) => 2 ;; `cat` concatenates the raw bytes of Blobs (and other Blob-like values) (cat 0x1234 0xee99) => 0x1234ee99 ;; `slice` is the inverse, extracting a range of bytes (slice 0x1234ee99 2) => 0xee99 ``` #### Addresses Addresses are identifiers for accounts on Convex (either user or actor accounts). They are expressed as positive integers preceded by `#` ```clojure ;; An address literal #123678 ``` ### Data Structures Data structures are composite values containing other values as elements. In Convex Lisp, all data structures are immutable, in the sense that once an instance is constructed, it cannot be modified: however a new instance can be constructed with any desired modifications extremely efficiently (most importantly this does not require copying the entire data structure - the unchanged elements are shared with the original instance) #### Vectors Vectors are the most common data structure, representing an indexed sequence of elements. They can be constructed by square brackets `[ ... ]` or with the core function `vector` ```clojure [1 2 3] => [1 2 3] (vector true 0x1234 "Hello") => [true 0x1234 "Hello"] ;; concatenate two vectors (concat [1 2 3] [4 5 6]) => [1 2 3 4 5 6] ;; append a new element (conj [1 2 3] 4) => [1 2 3 4] ``` You should use Vectors in most cases when you would use an "array" or "list" as defined in other languages. Vectors are particularly efficient for operations that add/remove at the end of the Vector. #### Lists Lists are sequential data structures most commonly used for expressing Convex Lisp code. They are represented by surrounding zero or more elements with regular parentheses `( )`. Because they are interpreted as code, if you want to construct a List literal you must quote it to suppress evaluations with `'( )` or use the constructor function `list` ```clojure ;; Note single quote symbol needed to produce a literal List '(1 2 3) => (1 2 3) ;; This also works (quote (1 2 3)) => (1 2 3) ;; This fails, because it gets interpreted as a function application, and "1" isn't a function! (1 2 3) => Exception: :CAST Can't convert value of type Integer to type Function ;; List constructor is also useful, especially if you want to compute elements to be the result of some expression (list 1 2 (+ 1 2)) => (1 2 3) ;; Construct a list by adding a new element to the front (cons 1 '(2 3 4)) => (1 2 3 4) ``` Lists are primarily used to express Lisp forms / expressions. They are also efficient for operations that add/remove at the front of the list. For most other sequential data, you should consider using a Vector. #### Maps Maps are data structures that map arbitrary keys to values, similar to an immutable `HashMap` in Java. ```clojure ;; A map of keywords to integers {:red 1 :blue 2} ;; accessing a value with `get` (get {:red 1 :blue 2} :red) => 1 ;; associating a new value with a key (creates a new immutable map) (assoc {:red 1 :blue 2} :green 3) => {:blue 2,:red 1,:green 3} ;; constructing a map from keys and values (hash-map :foo 2 :bar 4) => {:foo 2,:bar 4} ``` With functions that expect a sequential data structure, maps operate as if they are a sequence of entries, where each entry is a `[key value]` vector ```clojure (def m {:blue 2,:red 1,:green 3}) (first m) => [:blue 2] (count m) => 3 ``` `nil` may be used as a map key or value, however this is usually not recommended since `nil` also signal the absence of a value and this can be ambiguous. e.g. all of the following evaluate to `nil` ```clojure (get {nil nil} nil) ;; nil key present, but value is nil (get {} nil) ;; nil key not present (get {:foo nil} :foo) ;; :foo key present, but value is nil (get {:foo nil} :bar) ;; :bar key not present ``` Internally maps are implemented as radix trees based on the hash value of keys. This means that ordering is deterministic (since hashes are deterministic) but will appear random. Code using maps SHOULD NOT make any assumptions about map order. #### Sets Sets represent an unordered collection of distinct values, equivalent to a finite set on mathematics. ```clojure ;; A set of integers #{1 2 3 4} ;; The empty set #{} ``` Sets can be tested for membership with the `get` function, which returns `true` or `false` based on whether the element is present: ```clojure (get #{1 2 3} 2) => true (get #{1 2 3} 5) => false ``` Sets are particularly useful in cases where logic is required to compute intersections, unions and differences between sets. These operations have optimised support in the CVM, which makes such operations much cheaper than accessing or comparing elements individually. ```clojure (union #{1 2 3} #{3 4 5}) => #{5,4,2,3,1} (intersection #{1 2 3} #{3 4 5}) => #{3} (difference #{1 2 3} #{3 4 5}) => #{2,1} ``` A key motivation for the inclusion of Sets in the CVM (besides their mathematical elegance) is to efficiently support systems that must keep track of a variable number of distinct values - Trusted users authorised via an access control list - Which users are still eligible to vote in an election, or have already voted - Flags describing which optional terms apply to a smart contract Internally sets are implemented as radix trees based on the hash value of elements. This means that ordering is deterministic (since hashes are deterministic) but will appear random. Code using sets SHOULD NOT make any assumptions about set order. #### Indexes Indexes are specialised ordered maps that support "Blob-Like" keys only (Blobs, Strings, Addresses, Keywords and Symbols). Entries are sorted based on the byte values of the keys, up to a maximum of 32 bytes. ```clojure ;; Construct an index, note that the map is sorted in order (index 0x1234 :foo 0x3456 :bar :5678 :baz) => {0x1234 :foo,0x3456 :bar,0x5678 :baz} ;; Keys with identical byte values in their content will overwrite previous entries (even if a different type!) (assoc (index :foo 567) "foo" 789) => {"foo" 789} ``` Use an Index instead of a Map if both of the following are true: a) you need a sorted map b) you can strictly control the type of keys ### Keywords Keywords are symbolic names preceded by a colon (`:`) which are typically used to represent: - short human-readable keys in data structures - possible values for a set of flags, similar to an "enum" in many languages - error codes such as `:TRUST` - metadata keys such as `:callable` ```clojure ;; Keywords are literals that evaluate to themselves :hello => :hello ;; Keywords can be converted to and from Strings (name :hello) => "hello" (keyword "hello") => :hello ``` Keywords of this form have been popularised in the Clojure language. The CVM implementation may perform some optimisations to make use of common keywords more efficient. Technically, CVM Keywords can contain any UTF-8 characters, but for compatibility with the Reader, consistency with Clojure, for use in text files and by convention it is RECOMMENDED to limit character usage to the following: - Alphabetic characters (lowercase or uppercase, case sensitive) - Numerical digits `0` to `9` (disallowed as the first character by the Reader, but OK in other positions e.g. `:level7`) - The symbols `*`, `+`, `!`, `_`, `?`, `<`, `>` and `=` - The hyphen `-` (preferred as a word separator) A Keyword MUST have a length of between 1 and 128 UTF-8 bytes (inclusive). This size is motivated by the following considerations: - Large enough for most sensible human readable names - Small enough that Keywords are always embedded values in encodings - Large enough to contain a 32-byte hex string with a prefix (note restriction on numerical digits at start of keywords) - Discourage the use of Keywords for arbitrary content (use a Blob or String instead) ### Symbols Symbols are symbolic names used to refer to other values. They are similar to Keywords, however they are treated specially by the compiler as they are used to look up values in the current context / environment. ```clojure ;; You can define a symbol to refer to any value you like, e.g. a Vector. This will consume some CVM memory. (def bar [1 2 3]) ;; Once a symbol is defined in the environment, it will evaluate to the value itself bar => [1 2 3] ;; Trying to evaluate a symbol that is undefined will result in an :UNDECLARED error foo => :UNDECLARED error ;; `count` refers to the core runtime function. This prints as `count` but is NOT the symbol: the result is the `count` core function count => count ;; This is how code works! the symbol refers to a function, which is looked up then applied to the arguments (count [0 1 2 3]) => 4 ``` It is sometimes useful to use a Symbol as a value in itself (without performing any lookup). In this case, it is possible to **quote** the Symbol, so that the symbol itself will be returned (rather than the value that it refers to). This can be done in two ways: ```clojure ;; Quoting with the single quotation mark 'foo => foo ;; Quoting with the `quote` special form: (quote foo) => foo ``` ### Records Records are CVM data structures that behave like maps with a fixed set of keys. They are primarily used for internal data structures supporting the CVM. You cannot currently construct Records directly, but can access and read them: ```clojure ;; Get the account status record for an account (account #14) => {:sequence 1,:key 0x168e11d2512217576c30ec305ed672147125c9e20636fac29f4fca46cda0f003,:balance 132933333327304,:allowance 9999977,:holdings {},:controller nil,:environment {},:metadata {},:parent nil} ``` For practical purposes, Records behave as an immutable Map, and can be used in similar ways. ## Functions Functions are values in Convex Lisp that can be applied to zero or more arguments. Key properties: - They are full first class values, i.e. can be stored in data structure or passed as arguments to other functions - They can optionally support variable arity (i.e. variable numbers of arguments) - They operate as closures over lexically defined values ### Applying functions Function application is performed by evaluating a List where the function to be applied is the first element of the list, and the following elements are the arguments: ```clojure ;; The `+` function adds together any number of numerical values (+ 1 2 3 4) => 10 ;; The `count` function counts the number of elements in a data structure: (count [1 2 3 5 8]) => 5 ``` If the number of arguments is variable, you can use `apply` to apply a function to a sequence of arguments. ```clojure (def numbers [1 2 3 4 5]) (apply * numbers) => 60 ``` ### Defining functions Functions are typically defined with the `defn` macro, which creates a function and stores it in the current environment. ```clojure (defn square [x] (* x x)) (square 12) => 144 ``` ### Variable arities Functions can support multiple arities by specifying different combinations of parameter lists: ```clojure ;; A function with arity 1 and 2 specified (defn greet ([a] (str "Hello " a)) ([a b] (str "Hello " a " and b))) (greet "Bob") => "Hello Bob" (greet "Bill" "Ben") => "Hello Bill and Ben" ``` Functions can also support fully variadic arguments using the `&` symbol preceding a variadic argument ```clojure (defn average [& nums] (/ (apply + nums) (count nums))) (average 1.0 2.0 3.0 4.0) => 2.5 ``` Variadic arguments may be included at any position in the parameter list, but to avoid ambiguity a maximum of one `&` may be used in any single binding list. Variadic arguments are not required to be placed in the final position, however it is recommended to do so by convention. An example where a non-termial variadic argument might be useful is where the arguments represent a "stack" and it is helpful to bind the last argument representing the top element of the stack for special treatment. ### Higher order functions Higher order functions are functions that themselves take functions as arguments. This is fully supported on the CVM, and often results in cleaner, more robust code than would be achieved with equivalent imperative programming style. A good example is `map`, which applies a function to all elements of a collection, avoiding the need for an explicit loop: ```clojure ;; The inc function simply increments an Integer value (inc 7) => 8 ;; map can be used with inc to increment all elements of a collection (map inc [1 2 3 4]) => [2 3 4 5] ``` Another very useful higher order function is `reduce`, which can be used to sequentially apply a function to create an accumulated result: ```clojre (defn square [x] (* x x)) (defn sum-of-squares [coll] (reduce (fn [acc x] (+ acc (square x))) ; Function to add squares to an accumulator 0.0 ; initial accumulator value coll)) ; the collection argument to reduce over (sum-of-squares [1 2 3 4 5]) => 55.0 ``` ### Anonymous functions It can sometimes be convenient to create a function without storing it against a symbol in the environment. This can be done with the `(fn [...] ...)` anonymous function constructor. ```clojure (map (fn [x] (* x x x)) ; Anonymous function to cube a number [0 1 2 3 4]) => [0 1 8 27 64] ``` ### Returning values A `return` expression can be used to return early from a function with a given result: ``` (defn foo [x] (return (str x)) ; Early return with a result (fail "Shouldn't happen")) ; This line never gets executed (foo 678) => "678" ``` In the absence of a explicit `return`, the result of a function will be the result of the final expression executed. ## Floating point Convex Lisp supports IEE754 double precision floating point mathematics with the built-in `Double` type. This is important for many domains where Integer values may be inconvenient or introduce inaccuracies, e.g. in statistical or pricing applications. `Double` support is also important so that Convex Lisp can express a superset of JSON. ```clojure ;; Double values are literals, just like Integers 12.45 => 12.45 ;; Mathematical operations are supported for Doubles (+ 1.2 3.4) => 4.6 ;; Some mathematical operations always produce double results (sqrt 16) => 4.0 ;; You can convert any other number to a Double (including big integers) (double 100000000000000000000000000000000000) => 1.0E35 ;; Truncation to IEE754 Double precision is automatic when required (this is done by the Reader) 12.6678347835634781562349785632948756 => 12.667834783563478 ``` In general, operations that mix Integer and Double values will return a Double value. ```clojure (+ 1 2.0) => 3.0 (+ 1 2) => 3 ``` There are special Double literals for NaN and +/- Infinity. Applications using Double values should be aware of expressions that might produce such results and handle them accordingly. ```clojure (sqrt -1) => ##NaN (/ 1 0) => ##Inf (- 4 ##Inf) => ##-Inf ``` ## Equality and comparisons ### Value equality The `=` function tests for equality between any values, returning a Boolean that is `true` if and only if all values are equal: ```clojure (= 123 123) => true (= :foo "foo") => false ``` Value equality in Convex Lisp is **strict** and corresponds exactly with identity of CVM values (i.e. they must have the same encoding and Value Id) ### Numerical equality The `==` function tests for numerical equality. This is *less strict* than `=`. In particular it should be noted that Integers and Doubles can be numerically equal while not being identical CVM values (i.e. equality according to `=`) ```clojure (= 1 1.0) => false (== 1 1.0) => true ``` ### Numerical comparison The `<`, `>`, `<=` and `>=` symbols perform numerical comparison in the conventional fashion. Note that they support variable arities and mixtures of numerical types: ```clojure (< 1 3) => true (>= 2.0 2) => true (> 10 2 4.3) => false ``` ## The `nil` value The value `nil` is an important special value. While usage may depend on context, it is typically used to mean "no value" or "not found". Frequently, it is used to indicate when something is not found in a data structure, e.g. ``` ;; Trying to `get` a value from a map for a key that does not exist. (get {:foo 1, :bar 2} :baz) => nil ``` When passed to functions that expect a data structure, `nil` is interpreted as an empty data structure: ```clojure ;; Concatenating vectors with `nil` (concat [1 2] nil [3 4]) => [1 2 3 4] ;; Intersecting sets with `nil` (intersection nil #{1 2 3}) => #{} ;; Merging maps with `nil` leaves them unchanged (merge {:foo 1} nil) => {:foo 1} ;; `nil` is considered to be `empty?` (empty? nil) => true ``` NOTE: while `nil` may behave like an empty data structure in many contexts, it is a distinct value from the empty data structures (`[]` `()` `{}` and `#{}`). None of these values are considered equal to each other. In particular, functions that are expected to return a data structure should normally produce an empty data structure rather than `nil` if they succeed. When used in conditional expressions, `nil` is considered as `false` (see section on conditional expressions for more details) When used in JSON-like data structures, `nil` maps to the JSON value `null`. ## Conditional Expressions ### `if` macro The most common form of conditional expression is the `if` macro, which evaluates the first (test) expression to determine whether the second (true) or third (false) expression should be evaluated to determine the final result. ```clojure (if true "This will be the result" "This will never happen") ``` The `false` branch may be omitted, in which case the result will be `nil` ### `cond` special form If multiple test expressions are required, the `cond` special form allows this, returning the result for the first conditional expression matched (or an optional default expression if none match). ```clojure (def a 13) (cond (< a 10) "a is too small") (> a 20) "a is too big") "a is just right") ``` NOTE: The `if` macro expands to a `cond` expression in the standard Convex Lisp implementation. Using `cond` may be mildly more efficient in performance sensitive code, as it avoids one additional step of macro expansion. ### Truth Values In conditional expressions, results are determined by whether the evaluation of a test expression is "truthy" (like `true`) or "falsey" (like `false`). The rule is simple: - `false` or the value `nil` are considered "falsey" - `true` or *any other value* are considered "truthy" NOTE: A key reason for this rule is for convenience and simplifying code: `(if (not (nil? (lookup-thing a b))) ...)` can often become `(if (lookup-thing a b) ...)`. This is consistent with behaviour in other Lisps, where it is frequently referred to as "nil-punning". ## Importing and referencing other accounts It is frequently useful to refer to symbols in the environment of a different account from the one currently being used. Examples where this is important: - Referring to functions in shared library code - Examining a data structure in an Actor account - Defining a value once and referring to it from many user accounts ### Namespace Lookup Referring to a value in another account is made convenient with the `/` lookup syntax: ```clojure ;; A symbol in the current account (i.e. *address*) foo ;; A symbol in a different account #42/foo ;; Any expression can be used to define the target account (def other #42) other/foo ``` Referring to functions and data structures in this way is usually more memory efficient, and recommended in most cases where there is no need for multiple accounts to keep a duplicate copy of the same value. A very common use case is importing a library of code or values. Fot this purpose, the `import` macro is provided that creates an alias to any account via the Convex Name System (CNS) ```clojure ;; Import a library (import convex.fungible :as fun) ;; Use the library alias for references (deploy (fun/build-token {:supply 1000000})) ``` ### Nested lookups It is possible to nest lookups, since the target account is defined by an expression, and a lookup is a valid expression in itself. ```clojure ;; This works providing that `other-alias` is defined in the account specified by `alias` alias/other-alias/target-symbol ;; This is equivalent to using a temporary intermediate alias: (let [intermediate-alias alias/other-alias] intermediate-alias/target-symbol) ``` While possible, nested lookups would be unusual. A possible use case would be adding a layer of indirection so that the intermediate account can switch to different versions of a final destination account, for purposes of version control or providing alternative implementations of a component. ### Security implications - IMPORTANT **IMPORTANT NOTE**: while namespaced lookups refer to a value in another account, they do not change the security context, therefore code from another account that is executed will still run within the current account (`*address*`). To avoid security risks users MUST ensure they do not execute untrusted code. It practice this means: - Always review whether you trust an account that you `import` - NEVER execute a function from an untrusted account directly - Wrap execution of potentially untrusted code in a `query` to eliminate the possibility of any adverse side effects ```clojure (import some.untrusted.account :as danger) ;; This is safe because you are just reading a value, but you can't trust the value of `do-something` danger/do-something ;; DON'T DO THIS - it will execute any code in the `do-something` function in your current account (danger/do-something) ;; This is safe from side effects, though still potentially unwise: an attacker could burn your juice, or cause an error (query (danger/do-something)) ``` Be aware that CNS references MAY change, i.e. `(import some.library.account as lib)` may result in a reference to a different underlying account in future executions. Ideally, the import should only be executed once, and the value of the alias `lib` should be examined afterwards to ensure it refers to the correct account (e.g. `#43567`). If a malicious change to CNS is considered a risk, it may be preferable to define the alias to a known trusted account directly e.g. `(def lib #43567)` ## Calling actors Within a set of accounts that a user controls, the CVM provides a complete general purpose programming language where any code can be executed and data can be modified. However, for meaningful decentralised systems to operate, it is necessary to interact with other accounts that the user does not control, and which may provide important functionality such as trusted smart contracts or shared digital asset implementations. This can be done with a `call` that transfers execution control to another account. ### Call Syntax A `call` should be regarded as an instruction to an actor to perform an action on the caller's behalf. The `call` takes the following arguments: - A destination Address, which can be any valid account address (*or* a scoped address of the form `[#1579 :some-value]`, see below) - An optional *offer* of Convex Coins, which the destination account may choose to accept from the caller - A function invocation (which may include any arguments) Typical usage: ```clojure (call #67 (some-function :arg1 :arg2)) ``` Usage with an offer of 1,000,000 copper: ```clojure (call #67 1000000 (some-function :arg1 :arg2)) ``` Executing a `call` expression will: - Check if the target account exists - Check if the function name exists in the target account and has the `:callable` metadata set to true - Check if the current account has enough Convex Coins to reserve for the offer (if set) - If and only if all checks pass, switch the execution context to the target account and run the specified function with the given arguments - Once complete, control will return back to the caller with a result value for the call (or an error if one is thrown) ### Callable functions A callable function is any function with the `:callable` metadata set to true. This instructs the CVM to allow the function to be a target of a `call`. An example definition of a callable function that might be defined in an actor account is as follows: ```clojure (def visitor-count 0) (defn ^:callable visit [name] (set! visitor-count (inc visitor-count)) (str "Hello " name " you are visitor number " visitor-count)) ``` This can be called from any other account as follows: ```clojure (def actor #456756) ;; refer to whatever the actor account is (call actor (visit "Bob")) => "Hello Bob you are visitor number 1" (call actor (visit "Mary")) => "Hello Mary you are visitor number 2" ``` Important points to note: - The callable function modifies a value `visitor-count` that is defined *within the actor*. Only the actor itself can adjust this value. This demonstrates how actors can have control over their own internal state, but still allow callers to interact in a way that modifies this state in a predictably defined manner. - Other users can *observe* `visitor-count` (but not modify it!) e.g. using a lookup `actor/visitor-count`. This demonstrates how actor state is publicly visible. However, users should exercise caution when referring to internal actor implementation details: it may be preferable to use a separate callable function to query state, especially if it is possible that implementation details may change. - Assuming that the actor is immutable (i.e. has no external access or upgrade functionality) then the visitor count will be correctly managed for all time. This demonstrates the use of an actor as an *unstoppable decentralised program* that serves a clear purpose. ### Scoped calls It is frequently useful for an actor to manage multiple instances of entities of a particular type (e.g. a large number of concurrently running auctions in an auction house). In such cases, we can refer to each entity with a *scoped address* which is a vector that includes both the actor address and an identifier for the specific entity e.g. `[#123 101]`. To support this usage, `call` may optionally support the provision of a *scope* specified as follows. ``` (call [#67 :scope-value] (some-function :arg1 :arg2)) ``` A scoped call operates in the same way as any other call, except that the special value `*scope*` will be set to the value passed in the scope vector (in this case `:scope-value`). `*scope*` will be `nil` if no such scope value is used. Values or identifies used as a `*scope*` are defined by the actor: any CVM values may be used. It is however STRONGLY RECOMMENDED to enforce unique IDs e.g. allocating IDs using an incrementing integer counter for each entity created. Usage of scoped calls is ultimately an interface design decision for creators of actors. It is possible to achieve the same functionality with an additional ID argument to the call, for example. Experience suggests however that using scoped addresses simplifies writing generic user code that must refer to multiple entities provided by multiple actors, so it is RECOMMENDED to do so if your actor manages multiple entities. In most cases, entities managed by an actor will have a limited lifecycle. It is STRONGLY RECOMMENDED that: - Live entities are stored in a data structure indexed by `*scope*` for efficient access and existence checks (this should normally be a hash-map or index) - Actors provide a facility to delete expired entities (this allows memory reclaim to the benefit of whoever does the cleanup) - Scoped calls that reference a deleted/non-existent entity should fail - There is no way for a new entity to be created with the same ID as a deleted entity (using an incrementing counter solves this) ### Security context The current account is `*address*`. Any code executing has full control over this account, including the ability to modify the account's environment with `def` or `set!`. The account for which the transaction was initiated is `*origin*`. This remains unchanged for the entire transaction, and is initially equal to `*address*` The account that transfered control to this account, if any, is `*caller*`. This is `nil` initially, but will be the address of the account that executed any `call` to this account. IMPORTANT: From a security perspective, `*caller*` should be regarded as the account to check for authorisation to perform any action within an actor, since that is the account that made the `call` and requested for the action to be performed. DO NOT rely on `*origin*` for security checks. ### Call security #### For callers Within the scope of the call the target actor executes within it's own account context. This **protects the caller** : the actor does not have the ability to modify the caller's account and cannot impersonate the caller for the purpose of interactions with other actors (it must act on its own behalf). As with all CVM code execution, **juice costs are paid by the account that initially executed the transaction** (`*origin*`). It is possible for an actor to burn all available juice (in which case the transaction will fail). While the downside is limited by available juice, users should be aware that malicious or badly written actors may consume more juice that desired, and avoid calling untrusted actors. #### For actor developers IMPORTANT SECURITY NOTES: - Actor creators should remember that *any* account may call a callable function: they are a **public API** - Actor code SHOULD always perform authorisation checks against `*caller*` to see if the caller has the right to perform the requested action, and fail with a `:TRUST` error otherwise. The only exceptions to this are operations that are truly intended for anyone to be able to perform (e.g. depositing in a public donation box) - The `:callable` metadata should only be set for functions that are intended to be part of a public actor interface. Each such function represents any entry point that increases the size of public actor API that must be security audited. ## Other syntax ### Whitespace The Convex Lisp reader does not distinguish between different types of whitespace. Any number of tabs, commas, spaces and new lines are all considered equivalent. Programmers may find this useful for formatting source code for better readability. ```clojure (+ 3,4) => 7 ( + 3 4 ) => 7 (+ 3 4) => 7 ``` In some cases, whitespace MAY be omitted, where the syntax is unambiguous to the reader. This is NOT RECOMMENDED, since it may harm legibility and does not result in any on-chain data savings. ```cloure (+(+ 1 2)(+ 3 4)) ``` ### Comments Line comments include any text after a semicolon `;` up to the end of the line. Comments are considered whitespace by the reader: ``` ; This is a comment ;;;;; So is this ``` The reader macro `#_` can be used to instruct the reader to ignore any single form. This can be useful for temporarily ignoring chunks of code: ```clojure (+ 1 #_(this is a block of code which will not compile and is ignored) 2 3) => 6 ``` ## Metadata Convex Lisp supports metadata on any value. When metadata is applied to a value, it creates a Syntax object, which wraps both the metadata and the annotated value. The `^` symbol may be used to add metadata to a value and create a Syntax object, or you can use the `syntax` function to construct one: ```clojure ;; A syntax object adding a metadata map to a vector (Note the quote: compilation would otherwise strip the metadata) (quote ^{:foo "This is a metadata value"} [1 2 3]) => ^{:foo "This is a metadata value"} [1 2 3] ;; The `syntax` core function can be used to construct the same syntax object as above (syntax [1 2 3] {:foo "This is a metadata value"}) => ^{:foo "This is a metadata value"} [1 2 3] ;; Metadata can be empty (syntax [1 2 3]) => ^{} [1 2 3] ;; A keyword can be used as a shortcut to set a single metadata flag to true (= ^:mark [1] ^{:mark true} [1]) => true ``` Syntax objects can be wrapped and unwrapped with `syntax`, `meta` and `unsyntax`: ```clojure ;; Unwrap the value from a syntax object (unsyntax (syntax [1 2 3] {:some :metadata}) => [1 2 3] ;; Unwrap the metadata from a syntax object (meta (syntax [1 2 3] {:some :metadata})) => {:some :metadata} ``` Metadata can be attached to any definition in the environment: ```clojure ;; Define a symbol with metadata (def myval ^{:level 12} [1 2 3]) ;; Lookup metadata for a symbol (lookup-meta 'myval) => {:level 12} ``` Key use cases for metadata: - Control **behaviour of definitions** in the environment e.g.: - The `:callable` metadata tag indicates a callable actor function - The `:static` metadata tag indicates a definition that should be inlined by the compiler - Provide **on-chain documentation** for key functions, conventionally stored under the `:doc` field of metadata - Allow custom expansion / compilation logic for DSLs, e.g. type annotations ```clojure ;; example of accessing documentation metadata via the `doc` macro (doc count) => {:description "Returns the number of elements in the given collection, blob, or string.",:signature [{:return Long,:params [coll]}],:errors {:CAST "If the argument is not a countable object."},:examples [{:code "(count [1 2 3])"}]} ``` ## The Reader The Convex Lisp Reader is a software component that converts UTF-8 strings into CVM data structures. E.g. the String "(+ 1 2 3)" is converted by the Reader into the list `(+ 1 2 3)`containing 4 elements where the first element is the Symbol `+` and the following elements are Integers. The Reader is **not available on chain** - it is intended for use in client code or tools that communicate with the Convex network, such as a REPL terminal. The reader syntax in the `convex-core` reference implementation is available as a [ANTLR Grammar](https://github.com/Convex-Dev/convex/blob/develop/convex-core/src/main/antlr4/convex/core/lang/reader/antlr/Convex.g4) NOTE: Use of the Reader is not mandatory: it is possible to construct Convex Lisp forms programmatically (or even directly build pre-compiled CVM Ops) rather than parsing a String via the Reader. This may offer marginal performance benefits in some applications, e.g. JVM based systems that need to construct a large number of transactions. ## Coding Conventions The following conventions are recommended and/or generally utilised in Convex Lisp libraries: ### Hyphenation Hyphens are generally preferred to separate symbol names, e.g. `do-something` (rather than `do_something` or `doSomething`). This makes no difference to the CVM, but is primarily done for consistency with other Lisp based languages. ### Constant naming Prefer capitalised names like `PRICE` for global configuration variables and constants. This is to differentiate clearly from local, temporary or dynamically changing values. ### Keywords vs. Symbols If in doubt whether to use Symbols or Keywords, the following may be helpful: - Symbols are best when referring to values defined in the current environment (e.g. using `def`) - Keywords are best as keys in data structures or literal / constant values since they do not require quoting for such usage ### Comma usage Commas are considered whitespace, so there is no functional difference between `[a b c]` and `[a,b,c]`. We recommend using spaces instead of commas, unless the comma helps with source code readability or compatibility. Examples where commas may be helpful: ```clojure ;; Commas may be helpful to visually group keys and values in maps {:a 1, :b 2, :c 3} ;; Commas may be used to make vectors compatible with JSON format. This is a valid JSON array: [1, 2, 3] ``` ### Clojure Consistency Where possible, coding style should be consistent with Clojure which shares a very similar syntax to Convex Lisp. The [Clojure Style Guide](https://github.com/bbatsov/clojure-style-guide) may be informative. ## Performance tips and tricks Efficiency is an important concern for decentralised systems, as all computation and storage comes with a cost. Users of your product will thank you for minimising their transaction fees. Here are some methods for developing more efficient code in Convex Lisp. ### Do complex processing elsewhere In many cases, there is no point doing computation on the CVM at all: consider carefully if the processing can be done on the client or a product backend server instead. Some common examples: #### String parsing Do not attempt to parse strings in Convex Lisp (or on the CVM generally). This is almost always a bad idea: it is computationally expensive and likely to be error prone. Formats are also likely to change which may cause maintenance headaches for on-chain code. Instead: parse strings on the client or server with well tested libraries (e.g. ANTLR) and send to the CVM as CVM data structures. This is the approach taken by the Convex Lisp Reader, for example. #### Human readable output Do not try to produce human readable output for users on the CVM. Code to generate such output almost certainly belongs on the client or backend server. Apart from the execution cost, there are additional practical problems with this approach: - Text is likely to change. You don't want to be updating CVM code or data every time marketing changes some copy or formatting rules. - You definitely don't want to be dealing with things like internationalisation on the CVM. Instead: return a well defined integer value, keyword or other data structure that represents the relevant information and can be converted to the right human readable message on the client and/or server. ### Don't store content The CVM is not the place for storing static content such as images, text or other large binary files. Instead: use the [Data Lattice](../024_data_lattice/index.md), IPFS or a traditional web server / CDN that clients can download content from. If you absolutely must validate content against an on-chain record, store a single 32-byte hash of the content. This can be the merkle root of a large tree of content if necessary. Clients can hash the content and check this for authenticity / integrity. ### Beware loops Loops will often be at least `O(n)` in the size of the data structure they are iterating over. This can include explicit `loop` constructs or a `map` or `reduce` which implicitly loop over elements of a data structure. Usually, looping in CVM code indicates a design problem that will get worse as data size grows. Instead: Design your data structures and actor APIs so that data can be accessed or updated directly without looping. Typically, this might involve accessing records directly via a key in a Map. If necessary, clients can do loops themselves and access the content via multiple queries / transactions. ### Minimise encoding lengths It will be more efficient (and save memory costs) if you use data values with shorter encodings. If you expect to store large numbers of similar data structures, it is definitely worth minimising the size of each instance. You can use the `encoding` function to see the byte representation of any value. As an example you can see that the encoding of `true` is actually more memory efficient than the integer `1` ```clojure (encoding 1) 0x1101 (encoding true) => 0xb1 ``` Other tips for shortening encodings: - Use a vector `[1 2]` instead of a map with fixed keys `{:field1 1 , :field2 2}` - Use shorter Keywords e.g. `:f` instead of `:failure` - Avoid having entries in maps where the value is a default value like `nil` or `0` e.g. `{:name "Bob" :ferraris 1 :bugattis 0 :lambos 0}` becomes `{:name "Bob" :ferraris 1}`. You code can provide a default value in `get` when reading the key e.g. `(get person-record :lambos 0)` - If the same code is going to be duplicated in multiple accounts, put it in one account (or a library) and refer to that from the other accounts - Use a Set rather than a Map with dummy values if the values don't matter - `nil` `0` `true` and `false` only require 1 byte of encoding. These are the smallest CVM values. ### Pre-compilation If you compile Convex Lisp code before sending code to the CVM, you avoid the cost of compilation. This may be significant for complex expressions, especially if they involve macros. While unimportant for small one-off transactions, this may be valuable if you are sending a lot of transactions to Convex. There is no strong reason to avoid pre-compilation unless the compilation depends on something that might change in the global state and you need compilation to happen atomically in the same transaction as it is executed. ## Notable Differences vs. other languages on decentralised VMs Many decentralised systems offer virtual machines that are Turing complete and can execute code using one or more general purpose programming languages. Ability to do this however does not mean that it is easy to build efficient, secure and capable decentralised economic systems. We believe the following features of Convex Lisp, among others, offer substantive improvements over typical existing solutions: - **Functional Programming**: full support for the lambda calculus and first class functions - **Code is data**: Convex Lisp is a fully homoiconic language, with code expressed in its own data structures - **On chain compiler**: Convex Lisp on-chain code can perform code generation, compile and deploy new Convex Lisp code - **Big Integer support**: arbitrary precisions integers are supported as standard, avoiding risks of overflow (e.g. 256-bit fixed words) - **Floating point support**: Full IEEE754 Double compatibility, which are more suitable than integer mathematics for many purposes - **Orthogonal Persistence**: storage is automatic, with no need to explicitly store data. Extremely large data structures are supported (including larger than machine memory) and are loaded when accessed on demand. - **Extra Types**: A full range of general purpose value types including: Sets, literal Keywords, sorted Indexes, UTF-8 Strings, Blobs etc. - **Memory accounting**: Economic system for memory management. See [CAD006](../006_memory/index.md) - **Powerful macro capability**: ability to fully customise the language with expanders. See also [CAD009](../009_expanders/index.md) We hope that developers will find the tools provided in Convex Lisp a compelling solution as we continue to build open economic systems. ## Future plans Convex Lisp will continue to develop alongside Convex. Key goals beyond Convex V1 include: - **Backwards compatibility**: we must never break the behaviour of existing correct code. New changes will be strictly additive. - **Type System**: The CVM already has a rich type system. We will explore options to make this more visible and useful to developers: in particular support for gradual typing may be appealing - **Cryptographic primitives**: support for cryptographic operations natively in Convex Lisp. We note that most cryptographic operations should be performed at the level of client or peer implementations rather than on the CVM, but support for some such operations on the CVM may be justified where they enable important use cases (e.g. homomorphic encryption) - **32-bit floats** may be important as a key additional data type, particularly given their prevalence in AI systems. We will support these if and when they are natively supported on the CVM. --- # CAD027: Event Logging ## Overview Convex provides a verifiable event log for on-chain events. The log is designed for events that may be consumed / observed by external observers interested in meaningful events in the CVM state. Typical use cases include: - Detecting transactions that represent transfers of assets to / from a specific account - Notifying observers of availability of smart contracts, e.g. opening of an auction - Alerting external observers to situations that may require action This CAD describes the CVM based system for event logging that allows messages to be explicitly emitted during execution of CVM code, and reported by peers in interested parties. Example use cases: - A trader monitoring price changes in a market - A wallet app detecting when fungible tokens are received by a user from a third party - An auditor examining the minting history of a particular digital asset - A user checking their personal transaction history ## Alternatives It is possible for a peer to be instrumented to detect and register events of interest as it runs computations in the CVM state transition function. While this can theoretically allow any events of interest to be observed, this has some key drawbacks: - It requires custom instrumentation of a Peer - a significant development task - To get a new analysis of historical events, state transitions must be replayed (possibly from the very start of the ordering!) - Correctly detecting and categorising events in arbitrary CVM code is hard to get right ## Specification ### The `log` function Events are emitted via builtin CVM `log` function which adds a log entry for the current transaction. ```clojure (log val1 val2 val3 ....) ``` As an example, a decentralised auction house might choose to log when items are sold with this following code ```clojure (log :SOLD asset-id buyer price) ``` Other than the juice cost for `log` instruction execution, logging has no effect on CVM State ### Log Record ### Log entries A log entry consists of: - The `*address*` that caused the `log` entry to be created (never nil) - The `*scope*` present at the time of logging (this will usually be `nil`) - The location of the transaction, as a `[block-number transaction-number]` pair - A vector of the values logged ```clojure [address scope location [val1 val2 val3 ...]] ``` #### Position 0 : Address The Address of the account that called the `log` function. Typically this will be an actor. #### Position 1: Scope The `*scope*` at the time the `log` function was called. This may, be `nil`, otherwise it enables #### Position 2: Block Index The position of the Block in which the transaction was executed. #### Position 3: Transaction Index The position of the Transaction in the block for which the transaction was executed. Scheduled transactions are assigned negative numbers. #### Position 4: Log Data The log data is a vector of values representing the arguments to the `log` function. Log data SHOULD be structured according to application requirements and consistency with relevant logging standards. Conventionally, the first element of the log data SHOULD be a Keyword that describes the type of event e.g. `:TRANSFER` ### Interaction with rollbacks The log MUST NOT be updated with any log entries created within code that was rolled back (either due to an explicit `rollback` or failure of some atomic expression or transaction). The reason for this is that the Log should only include *things that happened* rather than any operations that are rolled back. The log SHOULD NOT be used for error reporting or diagnostics. ### Conventional values Users of logging capabilities MAY log any values they wish. However by convention, and in order to facilitate standards in tool, the following conventions are recommended. The first log entry SHOULD by a short uppercase string value that describes the type of event. Common codes are: - "TR" = a transfer of an asset - "ALERT" = a warning that external action may be needed #### Transfers The standard values for a transfer "TR" log event are: ```clojure [:TR sender receiver quantity data] ``` Where: - `sender` is the account address of the asset sender - `receiver` is the account address of the receiver - `quantity` is the quantity of the asset transferred, as per CAD19 - `sender-bal` is the new sender balance - `receiver-bal` is the new receiver balance - `data` is any additional data attached to the transfer (e.g. a map containing a payment reference) Transfer events of this type SHOULD be emitted by the actor implementing the asset, with a `*scope*` set as appropriate. #### Minting The standard values for a minting "MINT" log event are: ```clojure ["MINT" minter amount new-supply] ``` Where: - `minter` is the account performing the minting operation - `amount` is the quantity minted (negative for burn) - `new-supply` is the new total supply of the token #### NFT Transfers When sending a uniquely identified asset such as an NFT, conventional log values are: ```clojure [:TR sender receiver quantity data] ``` Where: - `sender` is the account address of the asset sender - `receiver` is the account address of the receiver - `id` is the unique ID of the asset - `data` is any additional data attached to the transfer (e.g. a map containing a payment reference) Note: since NFT transfers may involve a set of NFTs, this can result in multiple log entries for a single transfer. This approach is preferred because it allows individual NFTs to be tracked. ### Log Indexing Peers SHOULD index log records for efficient access and query by interested parties. The exact structure of log indexes are implementation details left to the peer operator. By default peers SHOULD maintain the following indexes into the log, for all log entries that they retain: - block -> log start and end position (allows fast location of log entries for a given block) - [ address | first value | second value | third value ] -> vector of log positions - An index on the each of the first 4 fields of the log data, if these are Blob-like values that can be indexed Values are included in the index if present and bloblike, otherwise empty blob. Peers MAY maintain additional indexes as relevant for their users. The following lookup paths may also enable efficient access to relevant information: - Log entry -> block -> transaction -> transaction details (e.g. `*origin*`) - Log entry -> block -> historical state -> state after block completion (includes `*timestamp*` etc.) ### Execution fees The cost of a `log` operation is: - A fixed base cost for the `log` (currently 1000) - A cost per byte of the log record (currently 20) The outputs of `log` are not stored in the CVM State, so this has no direct effect on memory size ### Log Data The log is a flat Vector of all log entries of all blocks up to the current consensus point. ### Juice The `log` function consumes juice proportional to the memory size of the logged data. This juice cost represents the cost imposed on peers for maintaining log entries, and prevents DoS attacks by methods such as including extremely large data structures as log values. ### Retention Peers MAY determine their own retention policy for historical log records. Peers MUST maintain logs for at least one month, for the purposes of transaction confirmation by clients and recipients. It is RECOMMENDED that Peers retain at least 1 year of log records. Peers SHOULD maintain logs for as long as possible given resource availability (primarily storage and indexing costs). --- # CAD028: DLFS - Data Lattice File System ## Overview The Data Lattice File System (DLFS) is a decentralised virtual file system built on the [Data Lattice](../024_data_lattice/index.md). It provides automatic synchronisation, conflict-free replication, and cryptographic verification of file data across distributed networks. ## Motivation ### The Problem with Centralised Storage Today's cloud storage solutions (Dropbox, Google Drive, iCloud) require users to trust centralised providers with their data. This creates fundamental problems: - **Single points of failure** - Provider outages mean no access to your files - **Privacy concerns** - Providers can access, analyse, and monetise your data - **Vendor lock-in** - Switching providers is difficult; your data is held hostage - **Censorship risk** - Providers can delete your files or terminate your account - **Subscription dependency** - Stop paying and lose access to your own data ### Why DLFS is Powerful DLFS eliminates these problems through decentralised architecture: **Self-Sovereign Storage**: You own your data. No third party can deny access, delete files, or terminate your account. Your cryptographic keys are the only authority. **Automatic Conflict Resolution**: Unlike traditional distributed filesystems that fail on conflicts, DLFS uses CRDT-based merge semantics. Edit the same file on two offline devices? DLFS merges changes intelligently when they reconnect - no manual intervention required. **Cryptographic Integrity**: Every file, every directory, every version is cryptographically verified through Merkle trees. Tampering is mathematically impossible without detection. The entire drive state can be verified with a single 32-byte hash. **Efficient Synchronisation**: DLFS uses delta encoding and content-addressable storage. Sync only what changed, not the entire file. Identical content is automatically deduplicated across your entire filesystem. **Offline-First Design**: Work offline indefinitely. DLFS is designed for intermittent connectivity. Changes merge automatically when connections are restored. **Zero Infrastructure**: No servers to maintain, no databases to manage. DLFS drives replicate peer-to-peer using Lattice Nodes (CAD036). **Future-Proof**: Based on immutable data structures and content addressing. Your data format will work decades from now - no proprietary formats, no migration nightmares. ## Design Goals - Provide a decentralised alternative to cloud storage services - Support peer-to-peer data replication (BitTorrent-style efficiency) - Maintain compatibility with familiar personal computer file systems - Provide POSIX compatibility as far as possible - Leverage CRDT semantics for automatic conflict resolution - Enable off-chain personal and/or private file systems - Support cryptographic access control and optional encryption ## Overall Architecture DLFS operates over a **P2P network** of Lattice Nodes (CAD036) that manage **drives** controlled by independent users. ``` ┌─────────────────────────────────────────────────────────────────────┐ │ DLFS Architecture │ ├─────────────────────────────────────────────────────────────────────┤ │ │ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ │ │ Device A │ │ Device B │ │ Device C │ │ │ │ │ │ │ │ │ │ │ │ ┌────────┐ │ │ ┌────────┐ │ │ ┌────────┐ │ │ │ │ │ Cursor │ │ │ │ Cursor │ │ │ │ Cursor │ │ │ │ │ └────────┘ │ │ └────────┘ │ │ └────────┘ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ ┌────────┐ │ │ ┌────────┐ │ │ ┌────────┐ │ │ │ │ │ DLFS │ │ │ │ DLFS │ │ │ │ DLFS │ │ │ │ │ │ Drive │ │ │ │ Drive │ │ │ │ Drive │ │ │ │ │ └────────┘ │ │ └────────┘ │ │ └────────┘ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ ┌────────┐ │ │ ┌────────┐ │ │ ┌────────┐ │ │ │ │ │Lattice │◄─┼────┼─►│Lattice │◄─┼────┼─►│Lattice │ │ │ │ │ │ Node │ │ │ │ Node │ │ │ │ Node │ │ │ │ │ └────────┘ │ │ └────────┘ │ │ └────────┘ │ │ │ └──────────────┘ └──────────────┘ └──────────────┘ │ │ │ │ ◄──────────── P2P Replication ────────────► │ │ (Delta-encoded, automatic) │ └─────────────────────────────────────────────────────────────────────┘ ``` **Lattice Cursors** (CAD035) provide atomic access to drive state, enabling thread-safe concurrent operations. **Lattice Nodes** (CAD036) handle network replication, automatically propagating changes to peers with delta encoding. **DLFSLattice** defines rsync-like merge semantics ensuring drives converge to consistent state. ### Cryptographic Security - **Authentication**: Only authorised users can update drives using Ed25519 digital signatures - **Integrity**: Each drive state is a Merkle tree verifiable via SHA3-256 hashes - **Content Addressing**: All data is collision-resistant and deduplicated - **Optional Encryption**: Privacy through client-side encryption ### Decentralised Identity W3C-style DIDs identify and validate users. This can be anchored on-chain via the Convex network as a secure public root of trust. ## Specification ### DLFS Drives A DLFS drive manages and enables access to DLFS data, analogous to a mounted filesystem on a personal computer. Conceptually, a drive is a cursor (CAD035) pointing to a Drive State, where filesystem operations atomically update the state. #### Drive Implementations DLFS drives MAY have multiple implementations: - **In-memory drives**: Fast, volatile storage - **Local drives**: Persistent storage backed by Etch or other stores - **Remote drives**: Drawing data from the Data Lattice via Lattice Nodes - **Read-only drives**: Immutable snapshots of another drive Systems MAY determine custom access rules: - Restricting access to authenticated users - Read-only public access (e.g., web servers) - Group-based permissions via SignedLattice #### Drive State The Drive State is the DLFS Node at the root of a drive. The Drive State MUST be a valid DLFS directory node (the "root" directory). Implementations SHOULD support immutable snapshots of Drive State. #### Drive State Hash The Drive State Hash MUST be the SHA3-256 hash of the root node's encoding (its Value ID per CAD003). This single hash cryptographically verifies the entire drive contents as a Merkle tree. ### DLFS URIs DLFS operates as a fundamental Internet protocol with URI-based addressing. **Proposed URI scheme:** `dlfs` #### Global Addresses ``` dlfs:bob/shared/images/photo.png ``` Refers to drive `shared` controlled by user `bob`, file `photo.png` in directory `images`. Usernames MUST be registered on CNS with a public DID in the `user` namespace (e.g., `user.bob`). #### Hosted Drives ``` dlfs://dlfs.example.com/bob/shared/images/photo.png ``` Specifies a particular host for resolution. Implementations MAY support unregistered usernames. #### Local Drives ``` dlfs:local:bob/private/documents/report.pdf ``` Local drives need not be publicly resolvable. They MAY be replicas of public drives. #### HTTPS Mapping DLFS URIs map to HTTP/HTTPS for web access: ``` https://dlfs.example.com/dlfs/bob/shared/images/photo.png ``` ### DLFS Nodes Each object in a DLFS drive is represented as a Node. A Node is a Vector with the following structure: ``` [directory-contents file-contents metadata update-time tombstones?] ``` | Index | Field | Type | Description | |-------|-------|------|-------------| | 0 | directory-contents | Index or nil | Map of names to live child nodes | | 1 | file-contents | Blob or nil | File data | | 2 | metadata | Any or nil | Arbitrary metadata | | 3 | update-time | Long | Timestamp of last modification | | 4 | tombstones | Index or absent | Per-directory index of deleted child name → deletion timestamp; on directory nodes only, present only when non-empty | Nodes MUST contain the first four fields. Directory nodes MAY carry an optional fifth element — the tombstone index — subject to the canonical invariant that it is present **if and only if** it is non-empty: a directory with no deletions has length 4, and one with pending deletions has length 5. Future extensions MAY include additional fields. Implementations MUST preserve unrecognised fields. #### Directory Nodes A directory node has: - `directory-contents`: Index mapping String names to child DLFS nodes - `file-contents`: nil #### File Nodes A file node has: - `directory-contents`: nil - `file-contents`: Blob containing file data #### Tombstones A tombstone records that a child name has been deleted, so that the deletion propagates during replication rather than being reintroduced by a merge with a peer that still holds the old entry. Tombstones are **not nodes** and never appear among the live directory entries. Instead, each directory node records its deletions in a separate tombstone index — the optional fifth node element (`tombstones`) — mapping each deleted child name to its deletion timestamp. Deleting a child removes it from the live `directory-contents` and records `name → deletion-time` in this index; creating or updating a name again clears any tombstone for it. Because tombstones live outside the live entries, ordinary directory operations — listing, emptiness checks, and navigation — read only the live entries and never scan tombstones. The tombstone index participates only in merge (see Merge Semantics). Implementations MAY support tombstone cleanup after replication completes (e.g., after a configurable retention period). #### Metadata The metadata field MAY contain: - nil (no metadata) - Any valid CVM value Implementations MAY assign custom semantics to metadata. #### Update Time The timestamp of the last operation modifying this node. Implementations SHOULD update timestamps on all content or metadata changes. Implementations SHOULD use timestamps to resolve replication conflicts (newer wins). ### DLFS File Names File names MAY be any UTF-8 String of length 1 or more. File names are **case sensitive**. For Windows compatibility, avoid names differing only by case. **Recommended characters:** - Alphabetic: `A-Z`, `a-z` - Digits: `0-9` (except first position) - Hyphen: `-` - Dot: `.` - Underscore: `_` **Avoid** `-` and `.` as initial characters (special meanings in shells and Unix). ### Merge Semantics DLFS uses rsync-like merge semantics via DLFSLattice: #### Directory Merge When merging two directories, the live entries and the tombstone indexes are reconciled together: 1. Entries present in both are merged recursively 2. Entries present only in one are included in the result 3. The two tombstone indexes are merged by taking the later (max) deletion timestamp per name 4. A name that is live on one side and tombstoned on the other is reconciled by timestamp: the later operation wins, so a newer deletion removes a stale live entry, and a newer write resurrects a name over an older tombstone 5. The merged directory timestamp is `max(timestamp_a, timestamp_b)` Reconciliation is driven only from the set of names whose tombstones actually differ between the two sides; when neither side has tombstones the merge skips reconciliation entirely. #### File/Node Merge When merging non-directory nodes: - The node with the newer timestamp wins - Equal timestamps: first operand is favoured (deterministic) #### Conflict Resolution All conflicts resolve deterministically: - **Timestamp-based**: Newer modifications win - **Deterministic tie-break**: on equal timestamps the first operand is favoured (by design), so the outcome is stable for any given pair - **Idempotent**: merge(a, a) = a Replicas exchange state bidirectionally, so all replicas converge to identical state regardless of the order in which updates arrive. #### Robust Merge Because merge combines data from untrusted peers, it fails closed on malformed input. Incoming foreign nodes are validated before they are accepted — a node must be a vector with at least the four required elements, a Long update time, and (where a fifth element is present) a non-empty tombstone index. If a foreign node is malformed, or a merge step would otherwise throw, the merge falls back to the local ("own") value rather than propagating the error. A malicious or corrupt value can therefore neither crash the merge (a denial of service) nor corrupt merged state — it is simply ignored. ### Replication DLFS drives replicate via Lattice Nodes (CAD036). #### Automatic Propagation When drive state changes: 1. Cursor update triggers Lattice Node propagator 2. Delta-encoded LATTICE_VALUE broadcast to peers 3. Peers merge received state using DLFSLattice 4. Missing data automatically acquired via DATA_REQUEST #### Manual Synchronisation Applications MAY trigger explicit sync: 1. Send LATTICE_QUERY to peer for current state 2. Merge response into local drive 3. Propagate merged state to other peers #### Efficiency - **Delta encoding**: Only novel cells transmitted - **Content addressing**: Identical content deduplicated - **Structural sharing**: Common subtrees shared between versions - **Root sync**: Periodic lightweight divergence detection ### Path-Based Access Drives support hierarchical path access via cursors (CAD035). ``` cursor.path(:fs, owner, "drivename", "path", "to", "file.txt") ``` The standard lattice ROOT structure positions DLFS at: ``` :fs → OwnerLattice(MapLattice(DLFSLattice)) ``` This provides: - Per-owner isolation via SignedLattice - Multiple named drives per owner via MapLattice - rsync-like merge within each drive via DLFSLattice ## Encoding DLFS nodes use the standard encoding format specified in CAD003. Node encoding: - Vector tag followed by element encodings - Index (for directories) uses radix tree encoding - Blob (for files) uses chunked encoding for large files - Timestamps as VLQ-encoded Long Value identity is determined by encoding equality. Value ID is SHA3-256 hash of encoding. --- ## Reference Implementation A reference implementation is provided in the Convex `convex-core` module (Java). ### Classes | Specification Concept | Java Class | Package | |-----------------------|------------|---------| | DLFS Lattice | `DLFSLattice` | `convex.lattice.fs` | | DLFS Node utilities | `DLFSNode` | `convex.lattice.fs` | | Local Drive | `DLFSLocal` | `convex.lattice.fs.impl` | | Java NIO FileSystem | `DLFileSystem` | `convex.lattice.fs` | ### Node Structure Constants ```java public static final long NODE_LENGTH = 4; // Minimum node length (directory nodes may add POS_TOMBS) public static final int POS_DIR = 0; // Directory entries index (live children) public static final int POS_DATA = 1; // File data blob public static final int POS_METADATA = 2; // Arbitrary metadata public static final int POS_UTIME = 3; // Update timestamp public static final int POS_TOMBS = 4; // Optional per-directory tombstone index (present only when non-empty) ``` ### Example: Local Drive with Cursor ```java // Create local DLFS drive AStore store = EtchStore.createTemp(); DLFSLocal drive = DLFSLocal.create(store); // Get cursor to drive root ACursor> cursor = drive.getCursor(); // Create a file CVMLong timestamp = CVMLong.create(System.currentTimeMillis()); AVector fileNode = DLFSNode.createEmptyFile(timestamp); fileNode = fileNode.assoc(DLFSNode.POS_DATA, Blob.fromHex("48656c6c6f")); // "Hello" // Update drive state atomically cursor.updateAndGet(root -> DLFSNode.updateNode(root, DLPath.create("hello.txt"), fileNode, timestamp) ); ``` ### Example: Replicated Drive via Lattice Node ```java // Create lattice node for DLFS replication ALattice rootLattice = Lattice.ROOT; NodeServer node = new NodeServer<>(rootLattice, store, 19999); node.launch(); // Access DLFS via cursor path ACursor cursor = node.getCursor(); Address owner = Address.create(42); ACell driveState = cursor.get(Keywords.FS, owner, Strings.create("main")); // Update drive and trigger automatic replication cursor.set(newDriveState, Keywords.FS, owner, Strings.create("main")); // Connect to peer for replication Convex peer = Convex.connect(peerAddress); node.addPeer(peer); // Changes automatically propagate via LatticePropagator ``` ### Example: Java NIO Integration ```java // Mount DLFS as Java NIO FileSystem URI uri = URI.create("dlfs:local:myuser/drive1"); FileSystem fs = FileSystems.newFileSystem(uri, env); // Use standard Java file operations Path file = fs.getPath("/documents/report.txt"); Files.write(file, "Hello DLFS".getBytes()); byte[] content = Files.readAllBytes(file); // Changes reflect in underlying DLFS drive ``` ### Implementation Notes - `DLFSLattice` implements `ALattice>` for merge operations - `DLFSNode.merge()` provides the rsync-like recursive merge algorithm - `DLFSLocal` wraps a cursor and store for local drive operations - `DLFileSystem` provides Java NIO FileSystem SPI integration - Replication uses `NodeServer` from convex-peer module ## CLI Support DLFS drives can be started and managed from the command line via the `convex-cli` module: ```bash convex dlfs start --port 8080 --auth ed25519 ``` The CLI supports: - Starting a DLFS server with configurable authentication - Mounting drives for WebDAV access - Drive management operations ## WebDAV Integration DLFS drives are accessible via standard WebDAV at `/dlfs/{drive}/{path}`. The WebDAV layer supports GET, PUT, DELETE, MKCOL, PROPFIND, MOVE, and COPY operations. ### Sync After Mutations WebDAV mutation operations (PUT, DELETE, MKCOL, MOVE, COPY) explicitly call `sync()` on the underlying cursor after each operation. This ensures changes propagate through the cursor chain to the root for persistence and lattice replication. This is critical for durability: without the explicit sync, writes would remain in the cursor's local state and might be lost on restart. ## DID URL Resource Addressing DLFS drives are addressable via DID URL paths for capability-based access control: ``` did:key:z6MkAlice.../dlfs/docs/specs ``` This enables fine-grained capability delegation using UCAN tokens ([CAD038](../038_lattice_auth/index.md)). A capability `{with: "did:key:z6Mk.../dlfs/docs/", can: "crud/read"}` grants read access to the `docs` directory and all its contents. ## See Also - [CAD002: CVM Values](../002_values/index.md) - Value types - [CAD003: Encoding](../003_encoding/index.md) - Binary encoding format - [CAD024: Data Lattice](../024_data_lattice/index.md) - Theoretical foundation - [CAD035: Lattice Cursors](../035_cursors/index.md) - Cursor system for state access - [CAD036: Lattice Node](../036_lattice_node/index.md) - Network replication --- # CAD029: Fungible Token Standard Fungible tokens are an important class of digital assets, that share the property of being fully fungible, i.e. any unit of a balance of the asset is equivalent to any other unit. Fungible tokens are always represented with **Integer balances**. This allows applications to make consistent assumptions about asset balance behaviour, which may not be possible with assets that use custom definitions for balances. Typical use cases for fungible tokens might include: - Digital currencies - Utility tokens use for micropayments - Shares in a DAO or virtual enterprises - Quantities of some resource in a game or metaverse - Reputation points This CAD represents the Convex replacement for ERC20 / ERC777 on Ethereum ## Specification ### CAD19 Compliance A fungible token MUST meet all the specifications for a compatible CAD19 asset, as well as the mandatory requirements specified in this CAD. ### Balances A fungible token balance MUST be a non-negative integer. Big integers MAY be used, however this is not recommended. Care MUST be taken to ensure that minting or burning does not allow fungible quantities to exceed usable big integer sizes: a minting operation that would cause supply to exceed the maximum big integer value MUST fail immediately. ### Decimals A fungible token MUST specify a `decimals` callable function, which indicates the number of decimal places used when displaying natural units of the token. Typically this might be a callable function of the form: ```clojure (defn decimals ^:callable [] 2) ``` The decimals value SHOULD NOT be utilised or depended upon to affect behaviour by on-chain code: it is intended to allow user interfaces to produce meaningful human readable units. The decimals value MUST NOT change for the lifetime of a fungible token. A re-denomination to a different unit should logically be considered as the creation of a new fungible token. Example usage: ```clojure (@convex.fungible/decimals USD-TOKEN) => 2 ``` i.e. if `decimals` is `2` then an asset quantity of `1499` will be considered as `14.99` in the natural unit of the currency. This is common in national currencies, e.g. the US Dollar can be considered to have 2 decimal places, where 100 cents to one dollar. ### Total Supply A fungible token SHOULD be able to report its total supply. The total supply MAY change (e.g. when minting or burning occurs). If this is the case, then the total supply MUST always be consistent with the sum of all balances of the token. A fungible token MUST NOT have any mechanism to create or destroy net balances without simultaneously updating the total supply. Typically this might be implemented with a fixed constant definition of total supply, e.g.: ```clojure (define max-supply 12345678) (defn total-supply ^:callable [] max-supply) ``` ### CAIP-19 encoding CAD29 tokens MUST be possible to reference with a CAIP19 token ID. This enables CAD29 assets to be uniquely referenced as a CAIP cross-chain asset for interoperability. The token ID MUST be in the format: ``` // For CAD29 tokens represented by a single actor cad29: Example: "cad29:789" // For scoped CAD29 assets cad29:-> Examples: "cad29:890-56778" "cad29:891-%5B%2339%20%23789%5D" ``` Note: As per RFC3986: "For consistency, URI producers and normalizers should use uppercase hexadecimal digits for all percent-encodings." In order to be canonical, CAD29 token IDs expressed in CAIP-19 format MUST use uppercase hex URL encoding. ## convex.fungible library Fungible tokens MAY be constructed using any actor that presents a callable interface according to this specification For convenience and efficiency, a standard `convex.fungible` library has been created that provides library wrapped for working with fungible tokens. Alongside `convex.asset`, this library provides full access to any CAD29 fungible token. --- # CAD030: Torus DEX Torus is a decentralised exchange (DEX) built on Convex. It is an open source actor that operates by facilitating smart contract interactions between: - Liquidity providers, who earn commission on trades - Traders who wish to swap fungible tokens ## Specification ### Torus actor The Torus actor manages all Torus DEX markets and provides user functionality. Users typically import the Torus actor as a library to make Torus functions available in their environment: ```clojure (import exchange.torus :as torus) ``` ### Markets Torus maintains a market for each fungible token that users wish to exchange. Markets are created on demand when the first user requests creation: ```clojure (torus/create-market token-id) => #678 ``` A new market is a deployed actor that implements the Torus market SPI. Initially the market will have zero liquidity (this may be added by liquidity providers later). If a second user requests creation of a market after it is already created, the original market is returned and no new market is deployed. Subsequently, the address of a market can be accessed with the `get-market` function: ```clojure (torus/get-market token-id) => #678 ``` If no market exists for a fungible asset, `get-market` returns `nil`. ### Trades Any user may trade on a Torus market. ```clojure ;; Assume a CAD token identified by TOKEN ;; Buy a 100 token quantity of a token (torus/buy-tokens TOKEN 100) ;; Sell 100 tokens (torus/sell-tokens TOKEN 100) ;; Buy 1000 CVM using the token (torus/buy-cvx TOKEN 1000) ;; Sell 1000 CVM, receiving the token (torus/sell-cvx TOKEN 1000) ``` It is also possible to do swaps between any two tokens. These are atomic swaps that execute a Torus trade on both underlying markets. ```clojure ;; Assume we have a second token named USD ;; Buy 200 TOKEN using USD (torus/buy TOKEN 200 USD) ;; Sell 300 TOKEN and receive USD (torus/sell TOKEN 300 USD) ``` Trades will fail if any of the following are true: - An attempt is made to buy or sell a negative quantity - The liquidity pool has insufficient liquidity to complete the trade - The user has insufficient funds (CVM or token) to complete the trade ### Liquidity Pool Each Torus market holds a liquidity pool of two assets: - Convex Coins - The CAD29 fungible token that the market represents Each active Torus market MUST have a positive Convex Coins in the liquidity pool. Each active Torus market MUST own a positive balance of the CAD29 fungible token being traded. The Torus market assumes the two asset quantities have equal value. We use Convex Coins as the common pairing for each fungible token because: - This enables swaps between any two CAD29 tokens with just two swaps, avoiding the need to create a market for every possible fungible token pair. - Convex Coins generally make sense to hold as a common asset for ecosystem participants - It provides an additional opportunity to earn a return on Convex Coins ### Adding and withdrawing liquidity A prospective liquidity provide MAY add liquidity to a market at any time with the `torus/add-liquidity` function. ```clojure ;; Add initial liquidity with Convex quantity ;; This is necessary for the initial liquidity provided (torus/add-liquidity token-id token-quantity convex-quantity) ;; Add liquidity with token quantity only ;; (will infer Convex quantity from current price) (torus/add-liquidity token-id token-quantity) ``` If successful, `add-liquidity` will return an integer equal to the number of liquidity shares gained by the liquidity provider. The attempt to add liquidity will fail if the user has insufficient Convex coins or tokens to provide the liquidity. If the market does not already exist, one will be created as if `create-market` was used. ### Torus Market SPI The Torus market SPI is usually not directly accessed by users: Torus functionality should be accessed by the `exchange.torus` library. A Torus market MUST implement a CAD29 fungible token SPI so that its liquidity share function as a fungible token. The Torus market SPI MAY change due to updates of Torus --- # CAD031: NFT Metadata ## Overview NFTs are uniquely identifiable digital assets on Convex Each NFT may define metadata which is accessible on-chain on a per-NFT basis ## Metadata Format Metadata SHOULD be expressed as a Convex data structure value that is representable as a JSON Object This implies that the following values are permissible: - A Hashmap with Strings as Keys - A String - Boolean values - A Vector of values - Numbers (must be a small integer or double precision floating point value to fit within valid JSON ranges) - `nil` ```clojure { "name" "Bob" "image" "http://foo.com/image1.png" } ``` Which would be equivalently expressed in JSON as: ```json { "name": "Bob", "image": "http://foo.com/image1.png" } ``` ## Metadata API Metadata MUST be accessed using the following interface either directly or indirectly: ``` ;; Using scoped actor (call [nft-actor nft-id] (get-metadata)) ;; Using a argument (call nft-argument (get-metadata nft-id)) ``` ## Stored vs. Generated metadata NFT implementations MAY either generate metadata dynamically on demand or store metadata on-chain. --- # CAD032: CVX Reader ## Overview Convex presents users with a rich variety of decentralised data structures. There is a common requirement for such data structures to be presented in text format. The Reader is a software component that can read CVM Values from a standard `.CVX` file format. The Convex reader format is defined in an [ANTLR grammar](https://github.com/Convex-Dev/convex/blob/develop/convex-core/src/main/antlr4/convex/core/lang/reader/antlr/Convex.g4) ## Format ### Basic Literals Most basic values are available as literals in the reader. ```clojure ;; Integers - can be big integers 1 456576575675675676586787888 ;; Doubles 1.0 -1e89 ##Inf ; Positive Infinity ##NaN ; Not-a-number ;; Strings "Hello World" "My name is \"Shady\"" ; escaping works like Java ;; Booleans true false ;; Nil value nil ``` ### Data structures ```clojure ;; Vectors [1 2 3] ;; Lists (+ 2 3 4) ;; Sets #{:foo :bar} ;; Maps {:a 1, :b 2} ;; Index (note: uses tagged reader syntax) #Index {0x1234 :foo} ``` ### Symbols Symbols consist of alphanumeric characters plus any of: `.*+!-_?$%&=<>:#` Examples: ``` foo this-is-a-descriptive-name mysym1456757 *hello* +++<>+++ ``` Symbols cannot start with a number, `:` or `#` (since these map to other readable types). These characters *are* valid elsewhere in the symbol. Symbols MUST be 1-128 UTF-8 characters in length. This restriction has two purposes: - It ensure that symbols are *always* embedded values - It discourages excessive name lengths: symbols are intended for human readability! - Empty symbols would be problematic for many reasons Note: ### Keywords Keywords start with `:` and are followed by a symbolic name with the same rules as a Symbol. Examples: ``` :foo :name :special-key ``` ### Whitespace Whitespace is any combination of space, tab, comma and newline characters. Line comments are also considered whitespace. A line comment is the semicolon `;` to the end of the line ```clojure ;;;; This is a line comment [some symbols] ; This is also a comment that will be treated as whitespace Whitespace MAY be omitted in cases where there is no ambiguity, e.g.: ```clojure (+(+ 2 3)(+ 4 5)) ``` ### Syntax Objects A syntax object is a value with attached metadata. Conceptually, syntax objects can be considered as wrapped values with a metadata map. Syntax objects can be specified in the Reader with the `^` symbol preceding some metadata value and the value to wrap. The exact handling of the metadata depends on its type: ```clojure ;; A Syntax object with the value [1 2 3] and the metadata {:foo bar} ^{:foo bar} [1 2 3] ;; Special handling for Symbol - adds the metadata {:tag SomeSymbol} ^SomeSymbol 127 ;; Special handling for Keyword - adds the metadata {:test true} ^:test (fn [] (test-name)) ``` Note: The Convex Lisp compiler generally compiles syntax objects to and expression that returns the value, rather than the syntax object itself. So you may need to quote a syntax object if you wish to obtain one in CVM code: ```clojure ^:foo [1 2 3] => [1 2 3] (quote ^:foo [1 2 3]) => ^{:foo true} [1 2 3] ;; This also works, but may be confusing. '^{:foo true} [1 2 3] => ^{:foo true} [1 2 3] ``` ### Tagged values The reader supported **tagged values** where custom tags indicate how a value should be interpreted. ```clojure ;; You can tag with #Index to signify an Index rather than a regular map #Index {0x :empty-blob} => #Index {0x :empty-blob} ;; A CVM Result record #Result {:id 2,:result 2} ;; A signed value #Signed {:public-key 0x8556aba53e4de38cc4e8775b546261c803eb0b09bd9d8868ebefd469f6e93e37, :signature 0xb85f149acac95790d2bccdd8c3bf32ac603e8a34a9378684ba073bb2065455d56056a5240283b4d1d39f1d04c69cf53acc5236710fa8d76ecfe2e8e9795f9502, :value {:origin #12,:sequence 5,:command 1}} ``` Currently, this is mostly useful for CVM types such as records which do not have a direct data structure representation. In the future, this may be extended to allow more custom types. ## Usage ### Java API Most functions of the reader are available through the static `convex.core.lang.Reader` class in the `convex-core` module. The reader takes arbitrary character strings and returns an appropriate Convex data value. Example usage: ```java import convex.core.lang.Reader // These will throw a ClassCastException if the value is not of the correct type AVector myVector = Reader.read("[1 2 3]"); AInteger myVector = Reader.read("546456456"); Symbol sym = Reader.read("foobar"); ``` :::danger If receiving symbols as strings from untrusted sources, ensure these are passed through the reader and result in a valid symbol rather than used directly. This is to avoid possible code injection attacks. The string `"filler-symbol (malicious-code)"` is an instructive example. ::: ## Conventions ### `.cvx` File Extensions Files intended to be read by the reader conventionally utilise a `.cvx` extension. Applications SHOULD name readable files with the `.cvx` extension. ### Mime Types The mime type used for Convex reader compatible files is `application/cvx` Applications SHOULD return `application/cvx` as the `Content-Type` header in response to HTTP requests that return `.cvx` readable text files. --- # CAD033: Convex CVM Types ## Overview Convex uses the CAD3 format to represent data. CAD3 is an extensible format, so Convex data types are represented using CAD3 extensions. This CAD lists the extension types and specifications used. ## Format ### `0xB0` - `0xB1` Boolean The possible Boolean values are `true` and `false`, which are coded as 1-byte Byte Flags. ``` Encoded as: 0xB0 <=> false 0xB1 <=> true ``` The two Boolean Values `true` or `false` have the Encodings `0xb1` and `0xb0` respectively. Note: These Tags are chosen to aid human readability, such that the first hexadecimal digit `b` suggests "binary" or "boolean", and the second hexadecimal digit represents the bit value. ### `0xEA` Address Addresses are used to reference sequentially allocated accounts in Convex, conventionally written as `#14567`. As such, they are conveniently encoded as CAD3 extension values with the tag `0xEA` ``` 0xEA ``` An Address is encoded by the tag byte followed by a VLQ Encoding of the 64-bit value of the Address. The address number MUST be positive, i.e. a 63-bit positive integer. Since addresses are allocated sequentially from zero (and accounts can be re-used), this usually results in a short encoding. Addresses MAY be used by implementations outside the CVM for other types of sequentially allocated values. --- # CAD034: Curated Registry ## Overview In many contexts, it is necessary to have a curated registry of authorised entities. Example use cases: - A list of members in a DAO or membership organisation - A list of verified Tokens for use in end user wallets CAD34 defines a generic, extensible interface for such registries, in order to support flexible tools and administration of registries that follow the standard. Design goals: - Support for any kind of unique ID / entity - Integration with teh CAD22 trust monitor model - Support for flexible administrative control at a per-registry level - Support for a single Actor hosting many registries ## Key functions ### Create Registry ```clojure (call registry-actor (create-registry controller {:optional :metadata})) ``` Where: - `controller` is a trust monitor for registry actions - An arbitrary metadata map may be provided, which will be attached to the registry The registry actor MUST enforce that only a caller allowed to use the `:create-registry` action is able to execute this, otherwise MUST fail with a `:TRUST` error. If permitted, the registry actor MUST creates a registry with the given metadata, and returns a new registry ID. ### Delete Registry ```clojure (call registry-actor (delete-registry registry-id)) ``` The registry actor MUST enforce that only a caller allowed to use the `:delete-registry` action is able to execute this, otherwise MUST fail with a `:TRUST` error. If permitted, the registry actor MUST delete the entire registry. The registry actor MUST NOT re-issue the same registry ID for any newly created registry. An administrator of the registry actor MAY re-instate the registry, but will have to restore this from off-chain information. ### Register ```clojure (call registry-actor (register subject metadata)) ``` If successful, the call MUST add `subject` to the given registry with the given metadata. If the `subject` already exists, the call MUST replace the metadata for the given subject. --- # CAD035: Lattice Cursors ## Overview Lattice Cursors provide mutable access points into immutable CVM data structures. They bridge the gap between Convex's immutable value semantics and the practical need for stateful updates in applications working with the Data Lattice. A cursor can be thought of as a mutable pointer that references an immutable value. When the value is "updated", the cursor atomically switches to reference a new immutable value while the old value remains unchanged. This pattern preserves all benefits of immutable data (structural sharing, safe concurrency, Merkle verification) while providing a familiar mutable interface. ## Motivation The Data Lattice (CAD024) provides decentralised storage using immutable, content-addressable data structures. While immutability offers strong guarantees for verification and replication, applications need practical mechanisms to: 1. **Track mutable state** - Applications need to maintain and update their current view of lattice data 2. **Navigate nested structures** - Complex data often requires access to deeply nested values 3. **Perform atomic updates** - Concurrent access requires safe compare-and-swap semantics 4. **Transform and cache values** - Expensive computations benefit from caching and lazy evaluation 5. **Synchronise changes** - Distributed systems need mechanisms to merge divergent states Cursors solve these problems by providing a thin mutable layer over immutable data. ## Specification ### Cursor Concept A **cursor** is a mutable container that holds a reference to an immutable CVM value (see CAD002). Cursors provide atomic operations for reading and updating this reference. All CVM values are encoded according to the format specified in CAD003. Cursors operate on these values but do not affect their encoding or identity. ### Cursor Types #### Root Cursor A Root cursor is the fundamental cursor type providing atomic access to a CVM value. Root cursors MUST support the following operations: | Operation | Signature | Description | |-----------|-----------|-------------| | `get` | `() → V` | Returns current value | | `set` | `(V) → void` | Sets value atomically | | `getAndSet` | `(V) → V` | Sets value, returns previous | | `compareAndSet` | `(expected, new) → bool` | Sets if current equals expected | | `getAndUpdate` | `(fn: V→V) → V` | Applies function, returns old value | | `updateAndGet` | `(fn: V→V) → V` | Applies function, returns new value | | `getAndAccumulate` | `(x, fn: (V,V)→V) → V` | Combines with x, returns old | | `accumulateAndGet` | `(x, fn: (V,V)→V) → V` | Combines with x, returns new | All operations MUST be atomic with respect to concurrent access. Root cursors MUST track their initial value (the value at cursor creation time) to support synchronisation patterns. #### Path Cursor A Path cursor provides atomic access to a nested value within a parent cursor. Path cursors MUST: - Maintain a reference to a parent cursor - Store an array of path keys for navigation - Delegate all operations atomically to the parent cursor - Use standard CVM collection access semantics for navigation Path navigation uses these semantics: - Maps and Indexes: key lookup returns associated value - Vectors and Lists: integer index returns element at position - Other types: return nil for any path Path updates use associative (`assoc` / `assocIn`) semantics: - For maps: `assoc-in` style nested update - For vectors: index-based update - A missing intermediate is **not** silently promoted to a map: without a lattice, writing through a nil intermediate is an error; with a lattice (see [Lattice-Aware Cursors](#lattice-aware-cursors)) intermediates are created from the lattice's `zero()` value, so they take the correct type (e.g. an `Index`, not a hash map) #### Forkable Cursor A forkable cursor tracks its initial value, enabling optimistic concurrency and transactional update patterns. Forkable cursors MUST provide: - `getInitialValue() → V`: the value at fork time - `fork() → Cursor`: create an independent working copy - `merge(detached) → bool`: attempt to merge a detached working copy back by compare-and-set There are two ways to converge a working copy back to its parent: - **`merge(detached) → bool` (compare-and-set)** — succeeds only if the parent still holds the value it had at fork time; returns `false` if the parent moved concurrently, leaving the caller to retry. This is the plain optimistic-concurrency path. - **`sync() → V` (lattice merge)** — available on lattice-aware cursors (see [Lattice-Aware Cursors](#lattice-aware-cursors)). Instead of compare-and-set, it merges the working copy into the parent using lattice semantics, so it **always succeeds** and never needs a retry: concurrent forks converge by merge rather than one overwriting the other. The **fork-modify-sync** pattern: 1. `fork()` a working copy (records the initial value) 2. Make one or more updates to the working copy, in isolation 3. `sync()` the changes back — lattice merge folds them into the parent, combining with any concurrent changes ### View Types Views are read-only cursors that observe or transform underlying values. #### View Base Views MUST: - Provide a `get()` operation returning the current viewed value - Throw an error (or equivalent) for all mutation operations - Support path navigation (returning new path cursors over the view) #### Transformer A Transformer applies a function to values from a source cursor. Transformers MUST: - Store a reference to a source cursor - Store a transformation function - Apply the transformation lazily on each `get()` call - NOT cache results (see Time Cache for caching) Transformers MAY support function chaining for composing multiple transformations. #### Time Cache A Time Cache provides TTL-based caching over a source cursor. Time caches MUST: - Store a TTL (time-to-live) duration - Track the timestamp of the last cache update - Return cached values if within TTL - Fetch fresh values from source when cache expires - Provide `invalidate()` to force cache refresh Time caches SHOULD handle timestamp overflow gracefully (treat as never-expiring). ### Atomic Operation Semantics All atomic operations MUST behave as if executed instantaneously with respect to other atomic operations on the same cursor. **Compare-and-Set:** Returns true and updates if and only if the current value equals the expected value (using CVM value equality as defined in CAD002). **Update operations:** The update function receives the current value and returns the new value. The function MAY be called multiple times if concurrent updates occur (implementations using CAS loops). ### Path Navigation The `path` operation creates a new cursor focused on a nested location. ``` path(key1, key2, ...) → Cursor ``` Path semantics: - Empty path returns the same cursor (or equivalent) - Single key navigates one level into the structure - Multiple keys navigate to arbitrary depth Path keys are CVM values (see CAD002). Common key types: - Keywords (e.g., `:data`, `:fs`) - Integers (for indexed collections) - Strings (for string-keyed maps) - Any valid CVM value (maps support arbitrary keys) ### Value Conversion Implementations SHOULD support automatic conversion from host language types to CVM values where unambiguous: - Strings → CVM String - Integers → CVM Long - Floating point → CVM Double - Booleans → CVM Boolean - Null/nil → CVM nil For complex types (maps, vectors, etc.), implementations SHOULD preserve CVM values and convert host equivalents using the encoding rules in CAD003. ### Thread Safety **Root cursors:** MUST be thread-safe. Concurrent operations MUST be atomic. **Path cursors:** Inherit thread-safety from their parent cursor's atomic operations. **Views:** NOT required to be thread-safe. Implementations MAY require external synchronisation for concurrent access to cached values. ### Cursor Hierarchy ``` ACursor ├── AForkableCursor (fork; tracks initial value; merge/CAS) │ ├── Root (atomic CAS value holder) │ ├── PathCursor (non-lattice path navigation) │ └── ALatticeCursor (lattice-aware: fork/sync, path/resolve) │ ├── RootLatticeCursor (root of a lattice tree) │ ├── ForkedLatticeCursor (independent working copy) │ ├── DescendedCursor (navigated into a sub-path) │ └── AUpdateCursor (update-on-write funnel) │ ├── StampedCursor (V → V: stamp on write, LWW) │ └── SignedCursor (V → SignedData: sign on write) └── AView (read-only base) ├── ACachedView (caching base) │ └── TimeCache (TTL caching) └── Transformer (lazy transformation) ``` ### Lattice-Aware Cursors Cursors that carry an `ALattice` — the `ALatticeCursor` family — understand lattice merge semantics. They add three capabilities over a plain cursor. **Fork and sync.** `fork()` produces an independent working copy; `sync()` merges it back into the parent via the lattice merge function, and therefore always succeeds (see [Forkable Cursor](#forkable-cursor)). `merge(value)` folds an external value into the cursor the same way. **Canonical vs logical keys — `path` vs `resolve`.** `path(keys...)` navigates using already-canonical keys and is the hot primitive; at each level it descends into the sub-lattice given by `ALattice.path(key)`. `resolve(keys...)` is the user-facing counterpart: it canonicalises external or logical keys (via the lattice's `resolveKey`) before navigating. Navigation copies on change only, reusing shared structure where keys are already canonical. **Auto-initialising writes.** `assoc(key, value)` and `assocIn(value, keys...)` are the write primitives. On a lattice cursor a missing intermediate is created from the value lattice's `zero()` — the correctly typed empty value — rather than a default hash map, and an update function that would otherwise receive a missing value receives `lattice.zero()` in place of nil. Writing through a missing intermediate with no lattice present is an error, not a silent promotion. **Write interception (update cursors).** Some lattice layers must transform a value on the way out — stamping it with a write time, or wrapping it in a signature. Rather than special-casing these in the navigation code, a lattice declares its own write boundary through generic hooks (`isWriteBoundary`, `createPathCursor`, `consumesPathKey`); crossing that boundary inserts an **update cursor** (`AUpdateCursor`), a shared funnel whose update-on-write step runs on every write. There are two instances: - **`StampedCursor`** (`V → V`) — stamps the value with the context write clock on write; used by stamp-on-write regions such as DLFS node times. - **`SignedCursor`** (`V → SignedData`) — signs the value on write using the key from the merge context, so a `SignedLattice` boundary yields correctly signed data. Deferred through a fork, the value is signed once at `sync()` rather than on every intermediate edit. Because signing and stamping are enforced by the cursor at the boundary — not by `instanceof` checks in the navigation code — new boundary behaviours can be added as new lattice layers without touching the cursor machinery. ## Lattice Integration Cursors integrate with the Data Lattice (CAD024) for distributed state management. ### Lattice Merge Operations When cursors are used with lattice-aware systems (see CAD036), merge operations combine values according to lattice semantics. Each lattice type defines its merge function satisfying: - **Commutativity**: `merge(a, b) = merge(b, a)` - **Associativity**: `merge(merge(a, b), c) = merge(a, merge(b, c))` - **Idempotency**: `merge(a, a) = a` ### Lattice Context Lattice-aware cursors carry a `LatticeContext` — the ambient information a merge or write needs: - **Timestamp**: the single write clock. Stamp-on-write regions read it to stamp values (the same clock DLFS uses for node update times); a `StampedCursor` with no context timestamp is an error - **Signing Key**: used by a `SignedCursor` to sign values at a signing boundary - **Owner Verifier**: used to authorise owners during merge (see [CAD038](../038_lattice_auth/index.md)) A cursor is given a context with `withContext(ctx)`, and context-aware merge takes the form `merge(context, own, other) → merged`. ## Encoding Cursor values use the standard encoding format specified in CAD003: - All CVM values have a unique canonical encoding - Value identity is determined by encoding equality - Value ID is the SHA3-256 hash of the encoding Cursors do not have their own encoding format; they are runtime constructs that hold references to encoded values. --- ## Reference Implementation A reference implementation is provided in the Convex `convex-core` module (Java). ### Classes | Specification Concept | Java Class | Package | |-----------------------|------------|---------| | Cursor (abstract) | `ACursor` | `convex.lattice.cursor` | | Forkable cursor (fork/merge/CAS) | `AForkableCursor` | `convex.lattice.cursor` | | Root cursor (atomic CAS holder) | `Root` | `convex.lattice.cursor` | | Path cursor (non-lattice navigation) | `PathCursor` | `convex.lattice.cursor` | | Lattice-aware cursor | `ALatticeCursor` | `convex.lattice.cursor` | | Root of a lattice tree | `RootLatticeCursor` | `convex.lattice.cursor` | | Forked working copy | `ForkedLatticeCursor` | `convex.lattice.cursor` | | Descended sub-path cursor | `DescendedCursor` | `convex.lattice.cursor` | | Update-on-write funnel | `AUpdateCursor` | `convex.lattice.cursor` | | Stamp-on-write cursor | `StampedCursor` | `convex.lattice.cursor` | | Sign-on-write cursor | `SignedCursor` | `convex.lattice.cursor` | | View | `AView` | `convex.lattice.cursor` | | Cached View | `ACachedView` | `convex.lattice.cursor` | | Time Cache | `TimeCache` | `convex.lattice.cursor` | | Transformer | `Transformer` | `convex.lattice.cursor` | | Merge context (timestamp, key, verifier) | `LatticeContext` | `convex.lattice` | | Lattice base + write-boundary hooks | `ALattice` | `convex.lattice` | | Factory | `Cursors` | `convex.lattice.cursor` | ### Example (Java) ```java // Plain root cursor (atomic CAS value holder) Root counter = Root.create(CVMLong.ZERO); counter.set(CVMLong.ONE); CVMLong old = counter.getAndUpdate(v -> v.inc()); // Lattice-aware cursor: fork / modify / sync — always succeeds, merges concurrently RootLatticeCursor> root = Cursors.createLattice(SetLattice.create(), Sets.empty()); ALatticeCursor> fork = root.fork(); fork.updateAndGet(s -> s.include(item1)); fork.updateAndGet(s -> s.include(item2)); fork.sync(); // both items merged into root by set union // Navigate to a sub-lattice with path(); merge folds a value in via lattice semantics ALatticeCursor> foo = mapRoot.path(Keywords.FOO); foo.merge(Sets.of(CVMLong.create(2))); // Cross a signing boundary — deferred signing via SignedCursor ALatticeCursor> drive = root2.path( Keywords.FS, // KeyedLattice → OwnerLattice ownerKey, // OwnerLattice → SignedLattice Keywords.VALUE, // SignedLattice → SignedCursor (signing enforced here) driveName); // MapLattice → DLFSLattice ALatticeCursor> dfork = drive.fork(); dfork.updateAndGet(state -> addFile(state, "a.txt")); // local, unsigned dfork.sync(); // signs once, merges into parent // Transformer (lazy, no caching) and a 5-second time cache Transformer doubled = Transformer.create(counter, v -> CVMLong.create(v.longValue() * 2)); TimeCache cached = new TimeCache<>(expensiveCursor, 5000); ``` ### Implementation Notes The Java implementation uses `java.util.concurrent.atomic.AtomicReference` for thread-safe root cursors. Other languages should use equivalent atomic primitives. ## See Also - [CAD002: CVM Values](../002_values/index.md) - Immutable value types - [CAD003: Encoding](../003_encoding/index.md) - Binary encoding format - [CAD024: Data Lattice](../024_data_lattice/index.md) - Distributed storage - [CAD028: DLFS](../028_dlfs/index.md) - Distributed filesystem using lattice - [CAD036: Lattice Node](../036_lattice_node/index.md) - Networked lattice replication --- # CAD036: Lattice Node ## Overview A Lattice Node is a networked server that participates in Data Lattice replication. Unlike full Convex Peers that implement Convergent Proof of Stake consensus for global state, Lattice Nodes focus exclusively on CRDT-based synchronisation of lattice values across a distributed network. Lattice Nodes enable decentralised applications to share and replicate data without blockchain consensus overhead, while still benefiting from content-addressable storage, Merkle verification, and automatic conflict resolution through lattice merge semantics. ## Motivation The Data Lattice (CAD024) provides the theoretical foundation for decentralised data storage. However, applications need practical infrastructure to: 1. **Host and serve lattice data** - Nodes that store and make data available 2. **Synchronise across networks** - Automatic replication between distributed nodes 3. **Handle partial data** - Recover missing data from peers during merges 4. **Propagate updates efficiently** - Minimise bandwidth with delta encoding 5. **Detect divergence** - Lightweight mechanisms to identify out-of-sync nodes Lattice Nodes provide this infrastructure as a lightweight alternative to full blockchain peers, suitable for: - Distributed file systems (DLFS - see CAD028) - Content delivery networks - Collaborative applications - Federated data sharing - Off-chain data for hybrid dApps ## Specification ### Node Architecture A Lattice Node consists of the following logical components: ``` ┌─────────────────────────────────────────────────────────────┐ │ Lattice Node │ ├─────────────────────────────────────────────────────────────┤ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────┐ │ │ │ Cursor │ │ Lattice │ │ Store │ │ │ │ (State) │ │ (Merge) │ │ (Persistence) │ │ │ └─────────────┘ └─────────────┘ └─────────────────────┘ │ ├─────────────────────────────────────────────────────────────┤ │ ┌─────────────────────────────────────────────────────────┐│ │ │ Network Server ││ │ │ • Message handling (PING, QUERY, VALUE, DATA_REQUEST) ││ │ │ • Peer connections ││ │ └─────────────────────────────────────────────────────────┘│ ├─────────────────────────────────────────────────────────────┤ │ ┌─────────────────────────────────────────────────────────┐│ │ │ Update Propagator ││ │ │ • Delta broadcasting ││ │ │ • Root sync ││ │ │ • Novelty detection ││ │ └─────────────────────────────────────────────────────────┘│ └─────────────────────────────────────────────────────────────┘ ``` #### Cursor (State) Each node maintains a cursor (see CAD035) pointing to its current lattice value. The cursor MUST: - Provide atomic read/write operations - Support path-based access into nested structures - Enable thread-safe concurrent updates #### Lattice (Merge) The lattice defines merge semantics for conflict resolution. The lattice MUST: - Satisfy CRDT properties (commutative, associative, idempotent) - Provide a `zero` identity value - Validate foreign values before merging via `checkForeign()` - Support hierarchical path navigation to sub-lattices #### Store (Persistence) The store provides content-addressable storage for lattice values. The store MUST: - Support retrieval by cryptographic hash (SHA3-256) - Track which data has been announced (for delta encoding) - Handle persistence of Merkle tree structures - Use the encoding format specified in CAD003 #### Commit Durability A node's primary propagator commits **synchronously**. When an application calls `sync()`, the primary propagator runs announce, root persistence and broadcast on the caller's own thread, and the call returns only once the new root is durably persisted in the primary store. Persistence errors propagate to the caller rather than being swallowed, so a successful return is a durability guarantee. Any secondary propagators remain asynchronous. To keep this guarantee consistent under concurrency, the propagator is the **sole writer** of the store's root pointer: snapshot and persist pipelines are serialised, so an older snapshot can never demote the root pointer after a newer snapshot's sync has already returned. ### Network Protocol Lattice Nodes communicate using a binary protocol. Messages use the encoding format specified in CAD003. #### Message Framing Messages are framed as: - VLQ-encoded length prefix - Message payload (encoded per CAD003) #### Message Types Message types are identified by a Keyword tag at the start of the message payload (see CAD015). There are no numeric message type IDs. | Tag | Name | Description | |-----|------|-------------| | `:DR` | DATA_REQUEST | Request missing data cells | | `:PING` | PING | Connectivity test | | `:LV` | LATTICE_VALUE | Announce lattice value update | | `:LQ` | LATTICE_QUERY | Request lattice value at path | #### PING (`:PING`) Connectivity test message. **Request format:** ``` [:PING id] ``` - `id`: Request identifier (any CVM value) **Response:** Result message containing `(id, "PONG")` Nodes MUST respond to PING messages with a Result containing the same ID. #### LATTICE_QUERY (`:LQ`) Request lattice value at a path. **Request format:** ``` [:LQ id path] ``` - `id`: Request identifier (for correlating response) - `path`: Vector of keys specifying path into lattice (empty vector `[]` = root) **Response:** Result message containing `(id, value)` - `value`: Lattice value at the requested path (encoded per CAD003) Nodes MUST respond with the current value at the specified path, or an error Result if the path is invalid. #### LATTICE_VALUE (`:LV`) Announce a lattice value update. **Message format:** ``` [:LV path value] ``` - `path`: Vector of keys specifying path (empty vector `[]` = root) - `value`: New lattice value to merge (encoded per CAD003) Messages MAY include delta-encoded data for bandwidth efficiency (see Delta Encoding below). Upon receiving a LATTICE_VALUE message, nodes MUST: 1. Extract the path from the message 2. Validate the path against the lattice structure 3. Validate the foreign value using `checkForeign()` 4. Merge the value using lattice semantics 5. Handle missing data by acquiring from peers (see Missing Data Recovery) 6. Trigger propagation of merged changes #### DATA_REQUEST (`:DR`) Request missing data cells by hash. **Request format:** ``` [:DR id hash1 hash2 ...] ``` - `id`: Request identifier - `hash1, hash2, ...`: SHA3-256 hashes of missing data cells (32-byte blobs) **Response:** Message containing requested cells Nodes SHOULD respond with available data from their store. Unavailable cells SHOULD be encoded as nil values. ### Value Encoding All values exchanged between nodes MUST use the canonical encoding format specified in CAD003. Key encoding properties: - All CVM values have a unique canonical encoding - Value ID = SHA3-256 hash of encoding - Merkle tree structure enables incremental verification - Multi-cell encoding supports complex nested values ### Missing Data Recovery Lattice values are Merkle trees that may reference data not yet available locally. Nodes MUST implement missing data recovery. #### Speculative Merge Pattern 1. Attempt merge with received value 2. If merge fails due to missing data, capture the missing hash 3. Acquire missing data from connected peers via DATA_REQUEST 4. Retry merge (up to configurable limit) 5. Commit merged value on success **Recommended parameters:** - Maximum retry attempts: 3 - Acquisition timeout per peer: 5 seconds #### Data Acquisition Algorithm ``` function acquire(missingHash): for each connectedPeer: response = sendDataRequest(peer, missingHash) if response contains data: store data locally return data return null (acquisition failed) ``` ### Delta Encoding To minimise bandwidth, nodes SHOULD implement delta encoding for LATTICE_VALUE messages. #### Novelty Detection Nodes track which cells have been "announced" to the store. When broadcasting: 1. Walk the value's Merkle tree 2. Collect cells not previously announced 3. Mark collected cells as announced 4. Encode only novel cells #### Delta Message Format LATTICE_VALUE messages include encoded message data: - Standard encoding: Only the value's top cell - Delta encoding: All novel cells in dependency order Receivers reconstruct the full value by: 1. Decoding received cells into local store 2. Resolving the top cell's references 3. Acquiring any still-missing data from peers ### Update Propagation Nodes SHOULD implement automatic propagation of updates to peers. #### Broadcast Triggers Propagation SHOULD be triggered when: - Local value changes (via cursor update) - Merge produces a new value - Periodic root sync interval elapses #### Timing Parameters | Parameter | Recommended Value | Description | |-----------|-------------------|-------------| | MIN_BROADCAST_DELAY | 50ms | Minimum time between broadcasts | | ROOT_SYNC_INTERVAL | 30 seconds | Interval for root-only sync | #### Delta Broadcasting When local values change: 1. Detect novel cells (not previously announced) 2. Create delta-encoded LATTICE_VALUE message 3. Broadcast to all connected peers 4. Track announced values to avoid redundant transmission #### Root Sync Nodes SHOULD periodically broadcast root-only messages containing only the top cell's encoding. Root sync provides: - Lightweight divergence detection (typically 50-200 bytes) - Trigger for pull-based recovery via DATA_REQUEST - Bandwidth-efficient heartbeat mechanism Receivers detecting divergence (merge triggers missing data) will automatically acquire missing cells. ### Peer Management Nodes maintain connections to peer nodes for synchronisation. #### Connection Operations Nodes MUST support: - `addPeer(connection)`: Add a peer connection - `removePeer(connection)`: Remove a peer connection - `getPeers()`: List current peer connections Nodes SHOULD maintain persistent connections for efficient messaging. Nodes MAY support automatic peer discovery (implementation-defined). #### Synchronisation Operations - **Manual sync**: Request root value from peer via LATTICE_QUERY, merge response - **Automatic sync**: Propagator broadcasts to all connected peers on changes ### Path-Based Operations Lattice Nodes support hierarchical access into nested lattice structures. #### Path Semantics - Empty path (`[]`) refers to root value - Single key path (`[:key]`) accesses first level - Deep path (`[:a :b :c]`) navigates nested structures Path operations MUST: - Use the sub-lattice at the path for merge semantics - Apply updates atomically to the root cursor - Support arbitrary nesting depth #### Sub-Lattice Resolution Given a path, nodes resolve the applicable sub-lattice: 1. Start with root lattice 2. For each path key, navigate to child lattice 3. Use child lattice's merge semantics for that path #### JSON Key Compatibility Lattice paths internally use CVM-native key types (Keywords, AccountKeys, etc.), but JSON-based APIs address paths using JSON-native types (strings, integers). Each lattice level provides a `resolveKey` function that translates an external key to the canonical CVM key for that level. On a cursor this is exposed as `resolve(keys...)` — the user-facing counterpart to `path(keys...)`, which navigates with already-canonical keys (see [CAD035](../035_cursors/index.md)): | Lattice Level | JSON Key | Canonical CVM Key | |---------------|----------|-------------------| | KeyedLattice (root) | `"kv"` | Keyword `:kv` | | OwnerLattice | `"0x49b44c..."` | ABlob (parsed hex) | | SignedLattice | `"value"` | Keyword `:value` | | Vector-valued (e.g. DLFSLattice) | `"0"` | AInteger `0` (parsed numeric string for vector position) | | MapLattice / IndexLattice | `"mykey"` | AString (identity) | The reverse mapping (`toJSONKey`) converts canonical keys back to JSON representations: Keywords become their name strings, blobs become hex strings, and other types pass through unchanged. JSON-based callers MUST resolve each path element through `resolveKey` before using standard lattice operations. CVM-native code that already uses canonical key types does not need resolution. The `SignedLattice` `:value` step is a virtual key consumed at the signing boundary (`consumesPathKey`) — it selects the `SignedCursor` enforcement point rather than indexing an ordinary map entry. ### Thread Safety Lattice Nodes MUST be thread-safe: - Cursor operations MUST be atomic - Multiple concurrent merges MUST produce consistent results - Propagation MUST not interfere with local operations Implementations SHOULD use appropriate synchronisation primitives (atomic references, locks, etc.) for the target language/platform. ## Standard Lattice Types Nodes MAY implement these standard lattice types: | Type | Merge Behaviour | Zero Value | |------|-----------------|------------| | SetLattice | Set union | Empty set | | MaxLattice | Maximum value | 0 (or min value) | | MapLattice | Recursive merge of entries by key | Empty map | | IndexLattice | Union of keys, per-entry child merge | Empty index | | KeyedLattice | Per-key statically-typed child lattices | Per-key zeros | | LWWLattice | Whole value with the newer timestamp wins (durable deletes) | nil | | JSONLattice | Structural navigation only (defines no merge) | Empty container by key shape | | StampingLattice | Stamp-on-write layer; delegates merge and navigation | Delegated | | DataLattice | Union of hash-indexed values | Empty index | | SignedLattice | Validates signatures on merge; signs on write at the boundary | nil | | OwnerLattice | Per-owner signed data maps | Empty map | The standard ROOT lattice structure uses a `KeyedLattice` backed by an `Index`, which provides lexicographic key ordering and compatibility with both CVM keyword paths and JSON string paths (keywords and strings share the same blob representation in the Index): ``` KeyedLattice (Index) { :data → DataLattice :fs → OwnerLattice(MapLattice(DLFSLattice)) :kv → OwnerLattice(MapLattice(KVStoreLattice)) :queue → OwnerLattice(MapLattice(TopicLattice)) :p2p → P2PLattice :local → LocalLattice } ``` ## Comparison with CPoS Peers | Aspect | Lattice Node | CPoS Peer | |--------|--------------|-----------| | Purpose | Data replication | Global consensus | | Consensus | None (CRDT merge) | Convergent Proof of Stake | | State | Single lattice value | Full blockchain state | | Messages | 4 types | Many (BELIEF, TRANSACT, etc.) | | Authentication | Optional (per lattice type) | Ed25519 required | | Ordering | Unordered (commutative) | Ordered transactions | ## Security Considerations ### Foreign Value Validation Nodes MUST validate all received values using `checkForeign()` before merging. This prevents: - Malformed data injection - Type confusion attacks - Invalid lattice structures ### Signature Verification When using SignedLattice or OwnerLattice types, nodes MUST verify Ed25519 signatures on received data before accepting merges. ### Denial of Service Mitigation Implementations SHOULD: - Rate-limit incoming connections - Limit maximum message sizes - Timeout slow peers - Bound retry attempts for data acquisition - Limit concurrent pending requests ## Configuration Lattice Nodes require the following configuration: | Parameter | Required | Description | |-----------|----------|-------------| | Lattice | Yes | Lattice type defining merge semantics | | Store | Yes | Storage backend for persistence | | Port | No | Network port (default: implementation-defined) | Optional configuration: - Initial peer list - Propagation timing parameters - Retry limits - Maximum message size --- ## Reference Implementation A reference implementation is provided in the Convex `convex-peer` module (Java). ### Classes | Specification Concept | Java Class | Package | |-----------------------|------------|---------| | Lattice Node | `NodeServer` | `convex.node` | | Update Propagator | `LatticePropagator` | `convex.node` | ### Example (Java) ```java // Create lattice node with SetLattice semantics ALattice> lattice = SetLattice.INSTANCE; AStore store = EtchStore.createTemp(); Integer port = 19999; NodeServer> node = new NodeServer<>(lattice, store, port); node.launch(); // Update local value ASet data = Sets.of("item1", "item2"); node.updateLocal(data); // Connect to peer Convex peer = Convex.connect(new InetSocketAddress("peer.example.com", 19999)); node.addPeer(peer); // Manual sync with peer node.sync(peer).join(); // Get current value ASet current = node.getLocalValue(); // Access via cursor ACursor> cursor = node.getCursor(); cursor.updateAndGet(set -> set.include(Strings.create("item3"))); // Clean up node.close(); ``` ### Implementation Notes - The Java implementation uses Netty for network I/O - Thread safety is provided via `java.util.concurrent.atomic.AtomicReference` - Delta encoding uses the `Cells.announce()` mechanism for novelty detection - Other language implementations should use equivalent networking and concurrency primitives ## See Also - [CAD002: CVM Values](../002_values/index.md) - Value types - [CAD003: Encoding](../003_encoding/index.md) - Binary encoding format - [CAD024: Data Lattice](../024_data_lattice/index.md) - Theoretical foundation - [CAD028: DLFS](../028_dlfs/index.md) - Distributed filesystem - [CAD035: Lattice Cursors](../035_cursors/index.md) - Cursor system - [CAD037: KV Database](../037_kv_database/index.md) - Replicated key-value store --- # CAD037: KV Database ## Overview The KV Database is a replicated key-value store built on the [Lattice](../024_data_lattice/index.md). It provides Redis-like data structure operations with CRDT merge semantics, cryptographic signing of replicas, and automatic network replication via [Lattice Nodes](../036_lattice_node/index.md). Each KV Database is a named, multi-writer store where independent nodes maintain signed replicas. Replicas converge through lattice merge without coordination, enabling offline-first distributed applications with rich data types. ## Motivation Decentralised applications frequently need shared mutable state beyond what a global blockchain provides. Common requirements include: - **Session data, caches, and configuration** that must be shared across nodes - **Counters, sets, and sorted sets** that multiple writers update concurrently - **Per-user or per-organisation databases** with cryptographic ownership - **Offline-capable writes** that merge when connectivity is restored Traditional distributed databases solve these problems with consensus protocols, leader election, or conflict resolution callbacks. The KV Database instead uses the mathematical properties of lattice merge to guarantee convergence without coordination. ### Design Goals - Provide a familiar key-value API (GET, SET, DEL, HSET, SADD, INCR, etc.) - Support multiple data types with type-appropriate CRDT merge strategies - Enable per-database, per-node signed replicas for authentication - Integrate with the standard lattice ROOT structure for network replication - Maintain compatibility with the [Lattice Cursor](../035_cursors/index.md) system ## Specification ### Lattice Path The KV Database occupies the `:kv` path in the standard lattice ROOT: ``` ROOT { :data → DataLattice :fs → OwnerLattice → MapLattice → DLFSLattice :kv → OwnerLattice → MapLattice → KVStoreLattice } ``` The full path to a specific replica is: ``` :kv / → Signed({ → {key → KVEntry, ...}, ...}) ``` Where: - **owner-key** — the owner identity (see Owner Types below) - **Signed(...)** — the owner's signed map of database names to KV store states - **db-name** — a string database name, scoped per owner This structure means each owner has their own namespace of databases within their signed state, matching the `:fs` pattern. Different owners can independently create databases with the same name without conflict. ### Owner Types The OwnerLattice supports multiple owner identity types, each with its own verification scheme. See [CAD038: Lattice Authentication](../038_lattice_auth/index.md) for the full specification. | Owner Type | Key Format | Verification | Use Case | |-----------|------------|-------------|----------| | Public Key | 32-byte Ed25519 public key | Direct equality with signer | Individual nodes, simple deployments | | Convex Address | Address (#0, #1337, etc.) | Account lookup for authorised keys | Organisations, multi-key accounts | | DID Identifier | String ("did:key:...", "did:convex:...") | DID resolution | Cross-system identity, standards compliance | All three types ultimately verify against the Ed25519 public key embedded in the signed data. The OwnerLattice performs verification at O(delta) cost during merge — only entries that differ between the local and incoming maps are checked. See CAD038 for the two-layer verification model (owner authorisation + signature validity). ### Lattice Composition ``` OwnerLattice ← per-owner merge with auth (CAD038) └── SignedLattice ← Ed25519 signature verification └── MapLattice ← per-database-name merge └── KVStoreLattice └── per-key merge └── per-type merge (LWW, structural, or PN-counter) ``` **OwnerLattice** at the top level merges per-owner, verifying that the signer is authorised for the owner identity and that Ed25519 signatures are valid before accepting values ([CAD038](../038_lattice_auth/index.md)). **MapLattice** inside the signed value merges per-database-name, allowing each owner to maintain multiple named databases. **KVStoreLattice** merges per-key using type-specific merge strategies for each KV entry. ### KV Entries Each owner signs a map of `{db-name → {key → KVEntry, ...}}`, where each key in a KV store maps to a **KV Entry**, a positional vector: ``` [value, type, utime, expire] ``` | Index | Field | Type | Description | |-------|-------|------|-------------| | 0 | value | any | The stored value (structure depends on type) | | 1 | type | integer | Type tag (see Data Types below) | | 2 | utime | integer | Last modification timestamp (epoch millis) | | 3 | expire | integer / nil | Expiry timestamp (nil = no expiry) | #### Tombstones A tombstone is an entry with `nil` value and `nil` type. The timestamp is preserved. Tombstones are required for lattice-compatible deletes: since lattice values can only grow monotonically, a delete is represented as a tombstone that wins over older live entries during merge. Implementations SHOULD support garbage collection of expired entries and old tombstones. ### Data Types | Type Tag | Name | Value Structure | Merge Strategy | |----------|------|-----------------|----------------| | 0 | Value | Any value | LWW by timestamp | | 1 | Hash | `{field → [value, timestamp], ...}` | Per-field LWW | | 2 | Set | `{member-hash → [member, addTime, removeTime], ...}` | Max timestamps per member | | 3 | Sorted Set | `{member-hash → [member, score, addTime, removeTime], ...}` | Max timestamps; score from latest add | | 4 | List | Vector of values | LWW by timestamp | | 5 | Counter | `{replica-id → [positive, negative], ...}` | PN-Counter (max per replica per column) | ### Merge Semantics KV entry merge follows these rules, evaluated in order: 1. **Equal entries** — return own (identity) 2. **One side nil** — return the other (with foreign value check) 3. **Same type, mergeable** (hash, set, sorted set, counter) — structural merge with max timestamp 4. **Otherwise** (value, list, or type conflict) — newer timestamp wins (LWW) 5. **Tombstone vs live** — newer timestamp wins These rules satisfy the lattice properties: - **Commutative**: merge(a, b) = merge(b, a) - **Associative**: merge(merge(a, b), c) = merge(a, merge(b, c)) - **Idempotent**: merge(a, a) = a #### Value/List Merge (LWW) The entry with the greater timestamp wins. Equal timestamps favour the first operand for determinism. #### Hash Merge Each field is independently merged by LWW on its per-field timestamp. Fields present in only one side are included. Field tombstones (nil value with timestamp) propagate deletes. #### Set Merge Each member is tracked with add and remove timestamps. A member is present when `addTime > removeTime`. Merge takes the maximum of each timestamp independently, ensuring adds and removes from different replicas combine correctly (OR-Set semantics). #### Counter Merge (PN-Counter) Each replica maintains independent positive and negative accumulators identified by a replica ID. Merge takes the maximum of each accumulator per replica. The counter value is `sum(positive) - sum(negative)` across all replicas. ``` Replica "node-0": [positive=3, negative=1] Replica "node-1": [positive=5, negative=0] Counter value = (3 + 5) - (1 + 0) = 7 ``` #### Sorted Set Merge Combines set membership semantics with scores. Each member tracks add/remove timestamps and a score. The score from the entry with the latest add timestamp is used. Membership follows the same rule as sets. ### TTL and Expiry Entries MAY have an expiry timestamp at position 3. - **nil** means no expiry - An integer value is the absolute epoch millis at which the entry expires Implementations SHOULD check expiry on read and return nil for expired entries. Implementations SHOULD provide a garbage collection operation to remove expired entries. ### Ownership and Authentication Each owner's data is signed with an Ed25519 key pair and verified during lattice merge by the OwnerLattice ([CAD038](../038_lattice_auth/index.md)). The signed state per owner is: ``` Signed({db-name → {key → KVEntry, ...}, ...}) ``` The signed value is a map of database names to KV store states, allowing each owner to maintain multiple databases under a single signed envelope. Verification during merge provides: - **Authentication** — only authorised signers can produce accepted values for an owner - **Integrity** — any tampering invalidates the signature - **Flexible ownership** — owners may be public keys, Convex addresses, or DID identifiers - **Multi-key support** — address and DID owners may authorise multiple signing keys (e.g. organisational accounts) ### Replication Model The KV Database uses a **merge-on-write** replication model: 1. Each node maintains its own signed replica (a map of database names to KV stores) 2. The node publishes its replica into the lattice at `:kv` (the OwnerLattice level) 3. Lattice Nodes (CAD036) automatically propagate signed replicas to peers 4. On the receiving side, the OwnerLattice merge combines signed entries from all owners 5. The application reads the merged owner map and absorbs remote data into the local KV store, extracting the specific database by name from each owner's signed map ``` ┌──────────┐ ┌──────────┐ ┌──────────┐ │ Node A │ │ Node B │ │ Node C │ │ │ │ │ │ │ │ KVDatabase │ KVDatabase │ KVDatabase│ │ key-a │ │ key-b │ │ key-c │ │ │ │ │ │ │ │ export() │ │ export() │ │ export() │ │ ↓ │ │ ↓ │ │ ↓ │ │ :kv/A │◄───────►│ :kv/B │◄───────►│ :kv/C │ │ │ Lattice │ │ Lattice │ │ │ │ Merge │ │ Merge │ │ └──────────┘ └──────────┘ └──────────┘ │ │ │ └────────────────────┼────────────────────┘ │ All converge to same owner map: { A→signed({db→storeA}), B→signed({db→storeB}), C→signed({db→storeC}) } ``` #### Selective Merge Applications MAY filter which replicas to merge using a predicate on the owner identity. This enables: - Trusting only known peers - Ignoring revoked or untrusted keys - Implementing access control lists #### Authentication Lattice merge MUST validate both owner authorisation and cryptographic signatures before accepting incoming values ([CAD038](../038_lattice_auth/index.md)). Entries where the signer is not authorised for the claimed owner, or where the signature is invalid, MUST be rejected. ## Operations ### Core KV | Operation | Description | |-----------|-------------| | `get(key)` | Get value for key (nil if absent or expired) | | `set(key, value)` | Set value | | `set(key, value, ttl)` | Set with TTL in milliseconds | | `del(key)` | Delete key (creates tombstone) | | `exists(key)` | Check if key exists and is not expired | | `keys()` | Return all live keys | | `type(key)` | Return type name of key's value | | `expire(key, ttl)` | Set expiry on existing key | | `ttl(key)` | Get remaining TTL (-1 = no expiry, -2 = not found) | ### Hash | Operation | Description | |-----------|-------------| | `hset(key, field, value)` | Set hash field | | `hget(key, field)` | Get hash field value | | `hdel(key, field)` | Delete hash field | | `hexists(key, field)` | Check if hash field exists | | `hgetall(key)` | Get all fields and values | | `hlen(key)` | Get number of fields | ### Set | Operation | Description | |-----------|-------------| | `sadd(key, members...)` | Add members to set | | `srem(key, members...)` | Remove members from set | | `sismember(key, member)` | Check membership | | `smembers(key)` | Get all members | | `scard(key)` | Get set cardinality | ### Counter (PN-Counter) | Operation | Description | |-----------|-------------| | `incr(key)` | Increment by 1 | | `incrby(key, amount)` | Increment by amount | | `decr(key)` | Decrement by 1 | | `decrby(key, amount)` | Decrement by amount | Counter operations require a **replica ID** to identify the calling node. Each replica maintains independent accumulators. ### Sorted Set | Operation | Description | |-----------|-------------| | `zadd(key, score, member)` | Add member with score | | `zrem(key, members...)` | Remove members | | `zscore(key, member)` | Get member's score | | `zrange(key, start, stop)` | Get members by score range | | `zcard(key)` | Get cardinality | ### List | Operation | Description | |-----------|-------------| | `lpush(key, values...)` | Prepend values | | `rpush(key, values...)` | Append values | | `lpop(key)` | Remove and return first element | | `rpop(key)` | Remove and return last element | | `lrange(key, start, stop)` | Get range of elements | | `llen(key)` | Get list length | Lists use LWW merge on the entire list. They are not CRDT-friendly for concurrent modification; applications requiring concurrent list operations SHOULD use sets or sorted sets instead. ### Maintenance | Operation | Description | |-----------|-------------| | `gc()` | Remove expired entries and old tombstones | --- ## Reference Implementation A reference implementation is provided in the Convex `convex-core` and `convex-peer` modules (Java). KV entries use the `AVector` type for positional vectors, `CVMLong` for integer fields, and `Index` for sorted string-keyed maps. Owner keys are represented as `AccountKey`, `Address`, or `AString` instances in the `ACell` hierarchy. After deserialisation, owner keys may appear as raw `ABlob` instances; the implementation resolves these to `AccountKey` via `OwnerLattice.resolveKey()`. ### Classes | Specification Concept | Java Class | Package | |-----------------------|------------|---------| | KV Store Lattice | `KVStoreLattice` | `convex.lattice.kv` | | KV Entry utilities | `KVEntry` | `convex.lattice.kv` | | KV Entry merge | `KVEntryLattice` | `convex.lattice.kv` | | KV API facade | `LatticeKV` | `convex.lattice.kv` | | Database wrapper | `KVDatabase` | `convex.lattice.kv` | | Hash operations | `KVHash` | `convex.lattice.kv` | | Set operations | `KVSet` | `convex.lattice.kv` | | Counter operations | `KVCounter` | `convex.lattice.kv` | | Sorted set operations | `KVSortedSet` | `convex.lattice.kv` | | List operations | `KVList` | `convex.lattice.kv` | | Index lattice (generic) | `IndexLattice` | `convex.lattice.generic` | The `KVDatabaseTest` and `LatticeKVTest` classes provide comprehensive test coverage for database operations, replication, and merge semantics. ### Example: Local KV Operations ```java // Create a KV database with signing key AKeyPair keyPair = AKeyPair.generate(); KVDatabase db = KVDatabase.create("mydb", keyPair, "node-1"); // Value operations db.kv().set("user:alice", Strings.create("Alice")); db.kv().set("config:timeout", CVMLong.create(30000)); // Hash operations db.kv().hset("user:1", "name", Strings.create("Alice")); db.kv().hset("user:1", "email", Strings.create("alice@example.com")); // Set operations db.kv().sadd("tags", Strings.create("alpha"), Strings.create("beta")); boolean isMember = db.kv().sismember("tags", Strings.create("alpha")); // Counter (PN-Counter with replica ID) db.kv().incr("page-views"); db.kv().incrby("page-views", 10); long views = db.kv().incrby("page-views", 0); // read current value // TTL db.kv().set("session:abc", Strings.create("data"), 3600000); // 1 hour TTL long remaining = db.kv().ttl("session:abc"); ``` ### Example: Multi-Node Replication ```java // Create two nodes with different keys AKeyPair keyA = AKeyPair.generate(); AKeyPair keyB = AKeyPair.generate(); KVDatabase dbA = KVDatabase.create("shared", keyA, "node-a"); KVDatabase dbB = KVDatabase.create("shared", keyB, "node-b"); // Each writes different data dbA.kv().set("from-a", Strings.create("hello")); dbA.kv().incr("counter"); dbB.kv().set("from-b", Strings.create("world")); dbB.kv().incr("counter"); // Exchange signed replicas dbA.mergeReplicas(dbB.exportReplica()); dbB.mergeReplicas(dbA.exportReplica()); // Both now see all data dbA.kv().get("from-b"); // "world" dbA.kv().incrby("counter", 0); // 2 (PN-counter merged) ``` ### Example: Network Replication via Lattice Nodes ```java // Create NodeServers with Lattice.ROOT NodeServer server1 = new NodeServer<>(Lattice.ROOT, store1, 19800); NodeServer server2 = new NodeServer<>(Lattice.ROOT, store2, 19801); server1.launch(); server2.launch(); // Connect peers server1.addPeer(ConvexRemote.connect(addr(19801))); server2.addPeer(ConvexRemote.connect(addr(19800))); // Create KV databases and write data KVDatabase db1 = KVDatabase.create("shared", key1, "node-1"); db1.kv().set("key", Strings.create("value")); // Publish signed replica to lattice at :kv (OwnerLattice level) // exportReplica returns {ownerKey → signed({dbName → kvStore})} AHashMap replica = (AHashMap) db1.exportReplica(); server1.updateLocalPath(replica, Keywords.KV); // Sync — LatticePropagator broadcasts automatically server1.sync(); // Read merged owner map from lattice at :kv AHashMap ownerMap = (AHashMap) server1.getCursor().get(Keywords.KV); db1.mergeReplicas(ownerMap); ``` ## See Also - [CAD002: CVM Values](../002_values/index.md) - Value types used in KV entries - [CAD003: Encoding](../003_encoding/index.md) - Binary encoding format - [CAD024: Lattice](../024_data_lattice/index.md) - Theoretical foundation - [CAD035: Lattice Cursors](../035_cursors/index.md) - Cursor system for atomic state access - [CAD036: Lattice Node](../036_lattice_node/index.md) - Network replication infrastructure - [CAD028: DLFS](../028_dlfs/index.md) - Distributed filesystem (similar lattice pattern) - [CAD038: Lattice Authentication](../038_lattice_auth/index.md) - Owner verification during merge --- # CAD038: Lattice Authentication ## Overview Lattice Authentication defines how the [Lattice](../024_data_lattice/index.md) validates ownership and authenticity of incoming values during merge operations. Since lattice merges combine data from untrusted sources, every incoming signed value must be verified before it is accepted into the local state. The authentication model is built into the `OwnerLattice` merge path, operates at O(delta) cost (only verifying entries that differ), and is safe by default: if ownership cannot be verified, the merge is rejected. ## Motivation The Lattice is a decentralised, multi-writer system. Any node can produce signed values and propagate them to peers. Without authentication, an attacker could: - **Impersonate an owner** by placing forged data under another owner's key - **Inject unauthorised data** into an organisation's namespace - **Corrupt replicated state** by submitting values signed with unrelated keys Traditional systems solve this with access control at the network boundary. The lattice instead validates at the **merge boundary** — every incoming value is checked against the owner key it claims to belong to, regardless of how it arrived. This is a fundamental property of the lattice: **merge is the security boundary**. ### Design Principles 1. **Verify on merge** — authentication happens inside the merge operation, not at the transport layer 2. **O(delta) verification** — only changed entries are verified, via the keyed `MergeFunction` in `mergeDifferences` 3. **Safe by default** — if verification is not possible (no verifier, unknown owner type), the merge is rejected for typed owners, and lenient only for raw public key owners where equality is self-evident 4. **Flexible ownership** — the same lattice structure supports multiple owner identity types without changes to the merge infrastructure ## Specification ### Owner Identity Types The OwnerLattice maps owner identities to signed values. The owner key determines the verification scheme. Three owner types are supported: | Owner Type | Key Format | Verification | State Required | |-----------|------------|-------------|----------------| | Public Key | 32-byte Ed25519 public key | Direct equality with signer | No | | Convex Address | Address (#0, #1337, etc.) | Account lookup for authorised keys | Yes (CVM State) | | DID Identifier | String ("did:key:...", "did:convex:...") | DID resolution | Depends on method | All three types ultimately verify against the Ed25519 public key embedded in the signed data. ### Verification Flow For each incoming entry during merge: ``` 1. Extract owner key (map key) and signer key (from signed data) 2. Verify signer is authorised for this owner: - Public key: signer == owner (identity check) - Address: delegate to owner verifier (state lookup) - DID string: delegate to owner verifier (DID resolution) 3. If verification fails → reject (keep own value) 4. If verification passes → proceed with SignedLattice merge 5. SignedLattice validates Ed25519 signature on the data ``` Both checks must pass: the signer must be authorised for the owner **and** the cryptographic signature must be valid. ### Keyed Merge Verification requires access to the map key during merge. The merge infrastructure provides a keyed merge method that receives `(key, ownValue, otherValue)` for each entry that differs between the two maps. This ensures: - **O(delta) cost** — identical entries are skipped entirely - **Key availability** — the owner key is passed to the merge function - **Backward compatibility** — callers not using keyed merge are unaffected ### Owner Verifier The merge context carries an optional owner verifier — a predicate that takes `(ownerKey, signerKey)` and returns `true` if the signer is authorised for that owner. This supports: - **Single-key owners** — one public key per owner - **Multi-key owners** — organisations with multiple authorised signers - **Key rotation** — verification against current state, not historical keys - **Revocation** — removed keys immediately fail verification #### Public Key Owners For 32-byte public key owners, verification is a direct equality check: ``` ownerKey == signerKey ``` This is handled inline without consulting the verifier, as it requires no external state. #### Address Owners For Convex Address owners, the verifier looks up the account in CVM state. An account may have multiple authorised keys (e.g. an organisation with several administrators). Any authorised key can sign data for that address. #### DID Owners For string owners containing DID identifiers, the verifier resolves the DID: | DID Method | Resolution | |-----------|-----------| | `did:key:z6Mk...` | Extract Ed25519 public key from multibase encoding | | `did:convex:#1337` | Resolve to Convex address, then lookup account key | | `did:web:example.com` | Fetch DID document, extract verification methods | The `did:key:` method is a pure computation (no external state). Other methods may require network access or state lookups. ### Safety Properties The authentication model is safe by default: 1. **No verifier, blob owner** — direct equality check, always deterministic 2. **No verifier, non-blob owner** — lenient mode accepts (for backward compatibility); production deployments SHOULD always set a verifier 3. **Verifier present, verification fails** — merge rejected, own value preserved 4. **Verifier present, verification passes** — proceed to signature check 5. **Signature invalid** — merge rejected by SignedLattice 6. **Both pass** — value accepted into lattice merge The two-layer check (owner authorisation + signature validity) means: - A valid signature alone is not sufficient — it must be from an authorised signer - Owner authorisation alone is not sufficient — the data must actually be signed ### Merge Direction Only **incoming** (foreign) values are verified. The local node's own values are trusted: ``` merge(ownValue, otherValue) ↑ ↑ trusted verified ``` This is consistent with the lattice merge model where `ownValue` represents the node's current state and `otherValue` is received from the network. ## Lattice Composition The authentication layer sits within the standard lattice composition: ``` ROOT └── :kv → OwnerLattice ← owner key verification here └── SignedLattice ← Ed25519 signature verification here └── MapLattice └── KVStoreLattice ``` The same pattern applies to any lattice path using OwnerLattice: ``` :fs → OwnerLattice → SignedLattice → MapLattice → DLFSLattice ``` ## UCAN Capability Tokens Lattice Authentication provides the foundation for **UCAN (User Controlled Authorisation Network)** tokens — signed JWTs that encode delegated capabilities from an issuer to an audience. ### Token Structure A UCAN token is a JWT with the following claims: | Claim | Description | |-------|-------------| | `iss` | Issuer DID (who is delegating) | | `aud` | Audience DID (who receives the delegation) | | `exp` | Expiry timestamp (token invalid once `exp <= now`) | | `nbf` | Not-before timestamp (token inactive until `nbf <= now`); optional | | `att` | Attenuations — vector of capabilities | | `prf` | Proof chain — vector of parent UCAN tokens | | `nnc` | Nonce for uniqueness; optional | ### JWT Encoding UCAN tokens can be encoded as standard JWTs with EdDSA signatures: ``` Header: {"alg": "EdDSA", "typ": "JWT"} Payload: {"iss": "did:key:z6Mk...", "aud": "did:key:z6Mk...", "exp": 1718000000, "att": [...], "prf": [...]} Signature: Ed25519 signature over header.payload ``` The `toJWT()` and `fromJWT()` methods convert between UCAN objects and JWT strings. The signature is **always verified against the Ed25519 public key bound in the `iss` DID** — the `did:key` in the payload encodes the issuer's key directly, so the key that must validate a token is fixed by the token's own issuer claim. The sender-controlled JWT `kid` header MUST NOT be used to select the verification key: trusting `kid` would let an attacker sign with their own key, name any issuer in `kid`, and thereby forge that issuer's identity. ### Capabilities Each attenuation in the `att` array is a capability with two fields: ```json { "with": "did:key:z6Mk.../kv/mydb/", "can": "crud/read" } ``` - **`with`** — a DID-scoped resource path (see below) - **`can`** — an ability in a slash-delimited hierarchy Per [UCAN], `with` is a resource pointer in URI form and the scheme set is **open** — a validator MUST NOT reject a token merely because it cannot interpret a resource. Convex **profiles** this rather than restricting it: resources that Convex itself anchors and delegates are named as **DID-scoped paths** (§[Resource Ownership](#resource-ownership-and-root-authority)). Utility code tolerates any UCAN-legal `with`; the profile governs which resources Convex can anchor to a trusted root, not which tokens are well-formed. A resource whose scheme yields no derivable owner is not rejected — it simply grants nothing, fail-closed at the capability rather than the token. A capability covers a request only if **both** its `with` covers the request's resource **and** its `can` covers the request's ability. **Resource matching (`with`)** is path-prefix matching at path-segment boundaries — never a raw string prefix: - **Exact match** — `w/decisions` covers `w/decisions` - **Prefix at a segment boundary** — `w/decisions` covers `w/decisions/INV-123`, because the match ends on a `/` boundary. It does **not** cover the sibling `w/decisions-secret`, even though that shares the textual prefix - **Trailing slash** — `w/decisions/` covers both its children and the bare parent `w/decisions` **Fail-closed:** a null or empty `with` (or an empty request resource) covers **nothing**. Absence is never a resource wildcard — a resource wildcard must be stated explicitly, never inferred from a missing or truncated capability. **Ability matching (`can`)** follows the same segment-boundary rule: the wildcard ability `*` covers any ability; otherwise the grant matches on an exact ability, or a prefix that ends on a `/` boundary (`crud` covers `crud/read`, but not a hypothetical `crudX`). ### Standard Abilities ``` crud (any CRUD operation) ├── crud/read ├── crud/write └── crud/delete convex (Convex-specific) ├── convex/transfer ├── convex/call └── convex/deploy * (wildcard — covers any ability) ``` ### Attenuation Rule Delegated capabilities can only narrow the grantor's authority — the delegate's resource MUST be covered by the grantor's under the segment-boundary rule above (a genuine sub-path, not merely a textual prefix), and the delegate's ability MUST be a sub-ability. This ensures capability chains are monotonically decreasing in scope. ### Validation UCAN tokens are validated by checking: 1. **Signature** — the EdDSA signature is valid against the public key bound in the token's `iss` DID (never the `kid` header). Signatures are verified once, at the transport boundary. 2. **Temporal bounds** — the token is within its validity window: `exp > now` (not expired) and, if present, `nbf <= now` (already active). Temporal bounds are re-checked at the point of use, after the boundary signature check, so a token that expires in flight is rejected on use. 3. **Capabilities** — each capability in `att` is covered by the issuer's own authority, under the resource and ability matching rules above. 4. **Proof chain** — every parent token in `prf` is validated recursively, and each link is well-formed: `proof.aud == token.iss` (the parent delegated to this issuer) and `token.exp <= proof.exp` (a child may not outlive its parent). Audience and issuer matching (e.g. `aud == caller DID`) is an **application policy** decision layered on top of these mandatory cryptographic, temporal, and chain-linkage checks — the core validator enforces proof-chain linkage but leaves audience acceptance to the caller. ### Resource Ownership and Root Authority Resource URIs use DID paths for cross-user scoping. The **DID prefix names the resource owner**; the path scopes within it: ``` did:key:z6MkAlice.../dlfs/docs/specs ← owned by did:key:z6MkAlice... did:key:z6MkAlice.../w/vendor-records ``` This enables fine-grained delegation across user boundaries while maintaining the lattice's per-owner signing model. **Why a DID-scoped profile.** A delegation chain is only trustworthy if its *root* is signed by an authority entitled to grant the resource. Establishing that authority requires knowing who owns the resource — which an opaque URI (`https:`, `mailto:`, application-custom) does not reveal. Naming the owner *inside* the resource lets any verifier check the root without a shared registry: offline for self-sovereign owners, and via a pluggable trust policy for custodial ones. Schemes that do not name an owner remain UCAN-legal and are not rejected, but Convex cannot anchor them to a root on its own. **How ownership is anchored.** For a DID-scoped `with`, the owner is the DID part of the resource (path and fragment stripped; DIDs compared canonically, not by raw string). The **root** of any chain granting that resource (the token with an empty `prf`) MUST be signed by the owner's controlling authority: - **self-sovereign owner** (`did:key`, self-certifying; `did:web`, domain-resolved) — the owner signs their own root, so the root issuer's DID equals the owner's DID. Verifiable offline; no policy required. - **custodial owner** (an identity a hosting node or venue controls) — the controlling authority signs an attestation on the owner's behalf, accepted only if the verifier's trust policy accepts that authority for that owner. The verifier resolves the root issuer's key through a DID resolver (`did:key` computed inline; other methods pluggable and expected to return only cryptographically-authenticated keys) and checks the root signature against it. The **mechanism** — chain walk, signature verification, key resolution, and the self-sovereign base case (root issuer == owner) — is fixed and shared. The **policy** — owner-derivation for non-`did:` schemes, and which custodial authorities are trusted — is supplied by the caller. No application re-implements chain-walking, signature verification, or resolution. **Fail-closed granularity.** A capability whose resource cannot be anchored to an accepted root grants nothing. This is per-capability: the token stays valid and its other, anchorable capabilities remain usable — satisfying the UCAN requirement not to reject uninterpretable resources while never conferring unanchored authority. ## Security Considerations The two-layer verification model defends against several attack vectors: | Attack | Defence | |--------|---------| | **Impersonation** — attacker signs data and places under victim's owner key | Owner verification rejects: signer not authorised for owner | | **Replay** — attacker replays victim's signed data under attacker's owner key | Owner verification rejects: victim's key not authorised for attacker's owner | | **Signature forgery** — attacker creates data with invalid signature | SignedLattice rejects: Ed25519 signature check fails | | **Key confusion** — valid signature but embedded key differs from owner | Owner verification rejects: embedded signer key != owner key (for public key owners) | | **Issuer spoofing** — attacker signs a UCAN with their own key and names a victim in the `kid` header | Verification key is bound to the `iss` DID's embedded public key; the `kid` header is ignored, so a token only validates for the key that actually signed it | | **Capability prefix escape** — a grant on `w/notes` is abused to reach the sibling `w/notesSECRET` | Resource matching is path-segment-boundary aware; a shared textual prefix does not cover a sibling resource | | **Fail-open delegation** — a truncated or empty `with` is treated as an implicit wildcard | Empty or absent resources fail closed and cover nothing; wildcards must be explicit | | **Rogue root** — attacker roots a delegation chain over a victim's resource, signed with their own key | Root authority: the root issuer MUST be the resource owner (self-sovereign) or an authority the trust policy accepts for that owner (custodial); a resource that cannot be anchored to an accepted root grants nothing | Production deployments SHOULD always configure an owner verifier for Address and DID owners. Without a verifier, these owner types fall back to lenient mode (accept all), which is suitable only for development and testing. ## Reference Implementation The reference implementation is in the Convex `convex-core` module (Java). ### Classes | Concept | Class | Package | |---------|-------|---------| | Keyed merge function | `MergeFunction` | `convex.core.util` | | Merge context with verifier | `LatticeContext` | `convex.lattice` | | Owner-based signed map | `OwnerLattice` | `convex.lattice.generic` | | Signed value merge | `SignedLattice` | `convex.lattice.generic` | | O(delta) map merge | `AHashMap.mergeDifferences` | `convex.core.data` | | UCAN capability & resource/ability matching | `Capability` | `convex.auth.ucan` | | UCAN token (JWT encode/decode, `iss`-bound signature) | `UCAN` | `convex.auth.ucan` | | UCAN validation (temporal bounds, proof chain) | `UCANValidator` | `convex.auth.ucan` | The `OwnerLatticeTest` class provides comprehensive test coverage including adversarial scenarios (impersonation, replay attacks, signature forgery). ### Example: Setting Up Owner Verification ```java // Create a verifier that checks Address owners against CVM state // and supports multiple authorised keys per account BiPredicate verifier = (owner, signerKey) -> { if (owner instanceof Address addr) { AccountStatus as = state.getAccount(addr); return as != null && as.isAuthorised(signerKey); } if (owner instanceof AString s) { return verifyDID(s.toString(), signerKey); } return false; }; // Create context with verifier LatticeContext ctx = LatticeContext.create(timestamp, signingKey, verifier); // Merge with verification AHashMap> result = ownerLattice.merge(ctx, ownMap, incomingMap); ``` ## See Also - [CAD002: CVM Values](../002_values/index.md) — Value types including AccountKey and Address - [CAD024: Lattice](../024_data_lattice/index.md) — Lattice merge foundations - [CAD035: Lattice Cursors](../035_cursors/index.md) — Cursor system and LatticeContext - [CAD036: Lattice Node](../036_lattice_node/index.md) — Network replication where authentication is applied - [CAD037: KV Database](../037_kv_database/index.md) — KV store using OwnerLattice authentication --- # CAD039: Convex SQL ## Overview Convex SQL is a SQL database layer built on the [Lattice](../024_data_lattice/index.md). It provides relational table storage with CRDT merge semantics, enabling SQL-like operations over distributed, replicated data structures. Tables support schema definitions, primary key indexing, and row-level merge replication via [Lattice Nodes](../036_lattice_node/index.md). Convex SQL enables standard SQL query capabilities over lattice data through Apache Calcite, bridging familiar database paradigms with the convergent properties of lattice technology. ## Motivation While the [KV Database](../037_kv_database/index.md) provides excellent support for key-value workloads, many applications require: - **Relational data models** with defined schemas and typed columns - **SQL query language** for complex queries, joins, and aggregations - **Table-based organisation** for structured business data - **Primary key indexing** for efficient row lookups - **Schema evolution** with controlled migrations Traditional distributed databases solve these problems with consensus protocols or conflict resolution callbacks. Convex SQL instead uses lattice merge properties to guarantee convergence while maintaining relational semantics. ### Design Goals - Provide a familiar table-based API (CREATE TABLE, INSERT, SELECT, DELETE) - Support schema definitions with typed columns - Enable primary key indexing with ordered retrieval - Integrate with Apache Calcite for SQL parsing and query planning - Maintain compatibility with the lattice cursor system and owner authentication - Support per-database, per-table signed replicas for authentication ## Specification ### Lattice Structure :::note Provisional structure Convex SQL does not yet occupy a region in the standard lattice ROOT (which currently registers `:data`, `:fs`, `:kv`, `:queue`, `:p2p` and `:local`). A dedicated region (e.g. `:sql`) is anticipated, but its name and structure are still open design questions and may change. This section documents the current implementation. ::: Convex SQL currently operates as a **standalone lattice**, managed by `ConvexDB`: ``` ConvexDB (MapLattice: db-name → database state) └── SQLDatabase (KeyedLattice: :tables → table store) └── TableStoreLattice (table-name → TableEntry) ``` Each database is a keyword-keyed structure: the `:tables` key holds the table store, leaving room for further per-database sections in future. For replication between nodes, the database map is wrapped in a signed owner envelope: ``` { → Signed({ → {:tables → { → TableEntry, ...}}, ...})} ``` Where: - **owner-key** — the owner identity (see [CAD038](../038_lattice_auth/index.md)) - **Signed(...)** — the owner's signed map of database names to database state - **db-name** — a string database name, scoped per owner - **table-name** — a string table name within the database This structure mirrors the KV Database pattern: each owner has their own namespace of databases, and each database contains named tables. ### Lattice Composition The full lattice hierarchy for a replicated Convex SQL owner map: ``` OwnerLattice ← per-owner merge with auth (CAD038) └── SignedLattice ← Ed25519 signature verification └── MapLattice ← per-database-name merge └── KeyedLattice ← per-database sections (:tables) └── TableStoreLattice └── per-table-name merge └── SQLTableLattice └── schema + rows merge └── TableLattice └── per-row merge (by primary key) └── SQLRowLattice (LWW) ``` A standalone (non-replicated) ConvexDB uses the same hierarchy from `MapLattice` down, without the owner/signature envelope. #### Layer Descriptions | Layer | Lattice | Merge Behaviour | |-------|---------|-----------------| | Owner | `OwnerLattice` | Per-owner with authentication ([CAD038](../038_lattice_auth/index.md)) | | Signature | `SignedLattice` | Ed25519 signature verification | | Database Map | `MapLattice` | Per-database-name merge | | Database | `KeyedLattice` | Per-section merge (`:tables`) | | Table Store | `TableStoreLattice` | Per-table-name merge using `SQLTableLattice` | | Table | `SQLTableLattice` | Schema LWW + row-level merge via `TableLattice` | | Rows | `TableLattice` | Per-primary-key merge using `SQLRowLattice` | | Row | `SQLRowLattice` | LWW by timestamp; equal timestamps favour deletions | ### Table Entries Each table is stored as a **Table Entry**, a positional vector: ``` [schema, rows, utime] ``` | Index | Field | Type | Description | |-------|-------|------|-------------| | 0 | schema | vector | Column definitions: `[[name, type], ...]` | | 1 | rows | Index | Row data: `{primary-key → RowEntry, ...}` | | 2 | utime | integer | Schema update timestamp (epoch millis) | #### Schema Format The schema is a vector of column definitions, each a vector of `[name, type]`: ``` [[name₁, type₁], [name₂, type₂], ...] ``` Where: - **name** — column name (string) - **type** — column type (nil = any, or a type identifier) Example: ``` [["id", nil], ["name", nil], ["email", nil]] ``` Column types are advisory for now; all values accept any CVM data type. #### Table Tombstones A tombstone is a table entry with `nil` schema and `nil` rows. The timestamp is preserved. Tombstones are required for lattice-compatible DROP TABLE: a tombstone wins over older live tables during merge. ### Row Entries Each row within a table is stored as a **Row Entry**, a positional vector: ``` [values, utime, deleted] ``` | Index | Field | Type | Description | |-------|-------|------|-------------| | 0 | values | vector | Column values for this row | | 1 | utime | integer | Update timestamp (epoch millis) | | 2 | deleted | integer / nil | Deletion timestamp (nil = live) | #### Row Tombstones A row tombstone has `nil` values and a non-nil deleted timestamp. The deletion timestamp represents when the row was deleted. ### Primary Keys Primary keys MUST be blob-like types (Blob, String, AccountKey, etc.) as required by the Index data structure for lexicographic ordering. Implementations SHOULD provide automatic key conversion: - **Integer** → 8-byte big-endian encoding - **String** → UTF-8 byte encoding - **Blob** → direct use This allows natural primary key usage while maintaining Index compatibility. ### Merge Semantics Merge operates at multiple levels: #### Table Store Merge Per-table-name merge using `SQLTableLattice.merge()` for each table entry. #### Table Entry Merge For each table: 1. **Schema**: LWW by timestamp — latest schema wins 2. **Rows**: Merge using `TableLattice` (per-primary-key merge) 3. **Tombstone handling**: If the schema winner is a tombstone, the tombstone wins (table dropped) This means schema changes from a later timestamp override earlier schemas, while row data merges independently regardless of schema changes. #### Row Entry Merge For each row (by primary key): 1. **Equal entries** — return own (identity) 2. **One side nil** — return the other 3. **Compare timestamps** — newer wins (LWW) 4. **Equal timestamps** — deletion wins (tombstone takes precedence) These rules satisfy the lattice properties: - **Commutative**: merge(a, b) = merge(b, a) - **Associative**: merge(merge(a, b), c) = merge(a, merge(b, c)) - **Idempotent**: merge(a, a) = a ### Ownership and Authentication Like KV Database, Convex SQL uses OwnerLattice for per-owner authentication. Each owner's data is signed with an Ed25519 key pair and verified during merge ([CAD038](../038_lattice_auth/index.md)). The signed state per owner is: ``` Signed({db-name → {:tables → {table-name → TableEntry, ...}}, ...}) ``` Authentication provides: - **Data integrity** — tampering invalidates the signature - **Owner verification** — only authorised signers can update an owner's tables - **Multi-key support** — address and DID owners may authorise multiple signing keys ### Replication Model Convex SQL uses the same **merge-on-write** replication model as KV Database: 1. Each node maintains its own signed replica (databases containing tables) 2. The node exports its replica as a signed owner map (`exportReplica()`) 3. Replicas are exchanged between nodes (a standard global lattice region for automatic propagation is anticipated but not yet defined — see the provisional note above) 4. On receive, OwnerLattice merge verifies signatures and combines entries from all owners 5. Applications read the merged owner map and absorb remote data ``` ┌──────────┐ ┌──────────┐ ┌──────────┐ │ Node A │ │ Node B │ │ Node C │ │ │ │ │ │ │ │ SQLDatabase │ SQLDatabase │ SQLDatabase│ │ tables │ │ tables │ │ tables │ │ │ │ │ │ │ │ export() │ │ export() │ │ export() │ │ ↓ │ │ ↓ │ │ ↓ │ │ replica A│◄───────►│ replica B│◄───────►│ replica C│ │ │ Lattice │ │ Lattice │ │ │ │ Merge │ │ Merge │ │ └──────────┘ └──────────┘ └──────────┘ ``` #### Concurrent Schema Changes When two nodes change a table's schema concurrently, LWW applies: the schema with the later timestamp wins. Row data merges independently, so rows from both nodes are preserved even if the schema changed. Applications SHOULD coordinate schema changes through external mechanisms (e.g. governance) to avoid unexpected schema conflicts. #### Row-Level Convergence Row merges are deterministic and converge across all replicas: - Rows with different primary keys merge independently - Rows with the same primary key use LWW - Deletions propagate correctly via tombstones ## Operations ### Table Operations | Operation | Description | |-----------|-------------| | `createTable(name, columns)` | Create table with column names | | `dropTable(name)` | Drop table (creates tombstone) | | `tableExists(name)` | Check if table exists | | `getSchema(name)` | Get column definitions | | `getColumnNames(name)` | Get column names as array | | `getRowCount(name)` | Get count of live rows | | `getTableNames()` | Get all table names | ### Row Operations | Operation | Description | |-----------|-------------| | `insert(table, key, values)` | Insert or update row | | `selectByKey(table, key)` | Get row by primary key | | `selectAll(table)` | Get all live rows | | `deleteByKey(table, key)` | Delete row (creates tombstone) | ### Replication Operations | Operation | Description | |-----------|-------------| | `exportReplica()` | Export signed replica for lattice publication | | `mergeReplicas(ownerMap)` | Merge remote replicas into local store | | `mergeReplicas(ownerMap, filter)` | Merge with owner filter predicate | ## SQL Integration Convex SQL integrates with Apache Calcite to provide full SQL query capabilities over lattice data. ### Query Execution Pipeline ``` SQL Query → Calcite Parser → Calcite Planner → Convex Relational Operators → Results ``` The Calcite integration provides: - **SQL parsing** — standard SQL syntax via Calcite's SQL parser - **Query planning** — Calcite planner with lattice-optimised rules - **Cost-based planning** — table statistics (such as row counts) feed Calcite's cost model to inform join ordering and plan selection - **Schema discovery** — automatic Calcite schema bridge from table definitions - **DDL** — CREATE TABLE and DROP TABLE via SQL - **DML** — SELECT, INSERT, UPDATE, DELETE - **Joins** — cross-table queries including merge joins - **Aggregations** — COUNT, SUM, AVG, etc. via Calcite aggregate operators - **Sorting and projection** — ORDER BY, column selection ### Primary Key Filter Pushdown When a query includes a primary key equality predicate (`WHERE id = ?`), the planner pushes the filter into the table scan, converting a full table scan into an O(log n) index lookup via `selectByKey()`. This is a critical optimisation for point queries. ### Relational Operators Custom Convex relational operators integrate with Calcite's planner: | Operator | Description | |----------|-------------| | `ConvexTableScan` | Full or filtered table scan | | `ConvexFilter` | Filter with PK pushdown support | | `ConvexProject` | Column projection | | `ConvexSort` | Ordering | | `ConvexAggregate` | Aggregation functions | | `ConvexJoin` | Cross-table joins | | `ConvexMergeJoin` | Merge join for sorted data | ### Row Count Optimisation Row count queries avoid materialising rows — the count is derived directly from the lattice index structure. ### PreparedStatement Caching Calcite query plans are cached for PreparedStatements, avoiding repeated parsing and planning for repeated queries with different parameters. ## JDBC Driver Convex SQL provides a standard JDBC driver for direct integration with any JDBC-compatible tool or application. ### Connection URLs ``` jdbc:convex:mem:mydb In-memory database (non-persistent) jdbc:convex:file:/path/to/db.etch Persistent database backed by Etch store ``` The driver manages database instances automatically — no manual registration or server process required. ### Transaction Support The JDBC driver supports transaction isolation via the lattice fork/sync model: | Operation | Lattice Effect | |-----------|----------------| | `setAutoCommit(false)` | Forks the cursor (snapshot isolation) | | `commit()` | Syncs the fork back to parent (lattice merge) | | `rollback()` | Discards the fork | | `setAutoCommit(true)` | Direct writes, no isolation | Because lattice merge is always well-defined, commits never fail due to conflicts — the merge function deterministically combines the transaction's writes with concurrent state. ### Example: JDBC Usage ```java Connection conn = DriverManager.getConnection("jdbc:convex:mem:mydb"); // DDL Statement stmt = conn.createStatement(); stmt.execute("CREATE TABLE users (id INTEGER, name VARCHAR, email VARCHAR)"); // DML PreparedStatement ps = conn.prepareStatement("INSERT INTO users VALUES (?, ?, ?)"); ps.setInt(1, 1); ps.setString(2, "Alice"); ps.setString(3, "alice@example.com"); ps.execute(); // Query ResultSet rs = stmt.executeQuery("SELECT * FROM users WHERE id = 1"); while (rs.next()) { System.out.println(rs.getString("name")); } // Transactions conn.setAutoCommit(false); stmt.execute("INSERT INTO users VALUES (2, 'Bob', 'bob@example.com')"); conn.commit(); // Lattice merge — always succeeds conn.close(); ``` ## PostgreSQL Wire Protocol Convex SQL includes a PostgreSQL wire protocol server (`PgServer`), enabling any PostgreSQL-compatible client to connect: ```bash psql -h localhost -p 5432 -d mydb ``` The PG server reuses the Calcite pipeline and ConvexSchema, providing the same SQL capabilities as the JDBC driver. A `pg_catalog` virtual schema provides compatibility with PostgreSQL client tooling. Write operations (INSERT, UPDATE, DELETE) modify the local replica, which is then signed and propagated via the lattice. ## Reference Implementation A reference implementation is provided in the `convex-db` module (Java). ### Classes | Specification Concept | Java Class | Package | |-----------------------|------------|---------| | Root database manager | `ConvexDB` | `convex.db` | | SQL Database wrapper | `SQLDatabase` | `convex.db.lattice` | | Table operations facade | `SQLSchema` | `convex.db.lattice` | | Table store lattice | `TableStoreLattice` | `convex.db.lattice` | | Table entry merge | `SQLTableLattice` | `convex.db.lattice` | | Table utilities | `SQLTable` | `convex.db.lattice` | | Row index lattice | `TableLattice` | `convex.db.lattice` | | Row entry merge | `SQLRowLattice` | `convex.db.lattice` | | Row utilities | `SQLRow` | `convex.db.lattice` | | JDBC Driver | `ConvexDriver` | `convex.db.jdbc` | | Calcite schema bridge | `ConvexSchema` | `convex.db.calcite` | | Calcite table | `ConvexTable` | `convex.db.calcite` | | DDL executor | `ConvexDdlExecutor` | `convex.db.calcite` | | Relational operators (scan, filter, join, ...) | `ConvexTableScan` etc. | `convex.db.calcite.rel` | | Planner rules | `ConvexRules` | `convex.db.calcite.rules` | | PostgreSQL server | `PgServer` | `convex.db.psql` | The `SQLDatabaseTest` and `ConvexDBTest` classes provide test coverage for table operations, JDBC, replication, and merge semantics. ### Example: Basic Table Operations ```java // Create a database with signing key AKeyPair keyPair = AKeyPair.generate(); SQLDatabase db = SQLDatabase.create("mydb", keyPair); // Create a table db.tables().createTable("users", new String[]{"id", "name", "email"}); // Insert rows (primary key can be CVMLong, AString, or ABlob) db.tables().insert("users", CVMLong.create(1), Vectors.of(CVMLong.create(1), Strings.create("Alice"), Strings.create("alice@example.com"))); // Query by primary key AVector row = db.tables().selectByKey("users", CVMLong.create(1)); // Query all rows Index> allRows = db.tables().selectAll("users"); // Delete a row db.tables().deleteByKey("users", CVMLong.create(1)); // Drop a table db.tables().dropTable("users"); ``` ### Example: Multi-Node Replication ```java // Create two nodes with different keys AKeyPair keyA = AKeyPair.generate(); AKeyPair keyB = AKeyPair.generate(); SQLDatabase dbA = SQLDatabase.create("shared", keyA); SQLDatabase dbB = SQLDatabase.create("shared", keyB); // Both create the same table dbA.tables().createTable("data", new String[]{"id", "value"}); dbB.tables().createTable("data", new String[]{"id", "value"}); // Each writes different rows dbA.tables().insert("data", CVMLong.create(1), Vectors.of(CVMLong.create(1), Strings.create("from-a"))); dbB.tables().insert("data", CVMLong.create(2), Vectors.of(CVMLong.create(2), Strings.create("from-b"))); // Exchange signed replicas dbA.mergeReplicas(dbB.exportReplica()); dbB.mergeReplicas(dbA.exportReplica()); // Both now see all rows dbA.tables().getRowCount("data"); // 2 dbB.tables().getRowCount("data"); // 2 ``` ## Comparison with KV Database | Aspect | KV Database (CAD037) | Convex SQL (CAD039) | |--------|---------------------|---------------------| | Data Model | Key-value | Relational tables | | Schema | Implicit (type tag per entry) | Explicit column definitions | | Key Types | String keys | Primary key (blob-like) | | Value Types | Multiple (string, hash, set, counter, etc.) | Row vectors | | Merge Granularity | Per-key, type-specific | Per-row (LWW) | | Query Language | API methods | SQL (via Calcite) | | Use Case | Caches, sessions, counters | Structured business data | Both share the same OwnerLattice authentication model and lattice replication infrastructure. ## See Also - [CAD002: CVM Values](../002_values/index.md) — Value types used in table entries - [CAD003: Encoding](../003_encoding/index.md) — Binary encoding format - [CAD024: Lattice](../024_data_lattice/index.md) — Theoretical foundation - [CAD035: Lattice Cursors](../035_cursors/index.md) — Cursor system for atomic state access - [CAD036: Lattice Node](../036_lattice_node/index.md) — Network replication infrastructure - [CAD037: KV Database](../037_kv_database/index.md) — Key-value store with similar lattice pattern - [CAD038: Lattice Authentication](../038_lattice_auth/index.md) — Owner verification during merge --- # CAD040: Lattice Queue ## Overview The Lattice Queue is a replicated append-only log built on the [Data Lattice](../024_data_lattice/index.md). It provides Kafka-style streaming semantics — ordered records, offset-based access, independent consumer positions, and log truncation — with CRDT merge for automatic replication via [Lattice Nodes](../036_lattice_node/index.md). Queues are organised into **topics**: each topic contains a set of numbered **partitions** (each an independent queue) plus topic-level metadata. The standard `:queue` region of the global lattice holds a map of topics per owner. Where the [KV Database](../037_kv_database/index.md) models shared mutable state, the Lattice Queue models an ordered stream of events. Together they cover the two fundamental patterns of distributed data: state and logs. ## Motivation Distributed systems frequently need ordered, durable message streams between producers and consumers. Common requirements include: - **Task queues** where work items are dispatched to processing agents - **Event logs** recording actions for audit, replay, or downstream consumption - **Message channels** enabling asynchronous communication between services - **Change feeds** propagating updates across organisational or network boundaries Traditional message brokers (Kafka, RabbitMQ, NATS) solve these problems with centralised infrastructure. The Lattice Queue instead uses lattice merge to provide convergent replication without brokers, enabling offline-capable, peer-to-peer streaming. ### Design Goals - Provide a familiar offset-based log abstraction (append, read-by-offset, range queries) - Support Kafka-compatible record structures (key, value, timestamp, headers) - Enable log truncation while preserving offset continuity - Maintain queue-level metadata for naming, configuration, and provenance - Integrate with the [Lattice Cursor](../035_cursors/index.md) system for atomic state access - Support lattice fork/sync for conflict-free distributed replication ## Key Concepts ### Append-Only Log A Lattice Queue is an ordered sequence of records, each assigned a monotonically increasing **offset** — an integer starting from zero. Records are appended at the tail and are never modified in place. This is the same fundamental abstraction as a Kafka partition or a database write-ahead log. The append-only property maps naturally to lattice semantics: the log only grows, making it monotonic by construction. Merging two copies of the same log produces the longer (more complete) version. ### Offsets Every record in the queue has an **absolute offset** — its permanent position in the log's history. Offsets are stable: once a record is assigned offset `n`, that assignment never changes, even after truncation or replication. ``` offset: 0 1 2 3 4 5 ┌─────┬─────┬─────┬─────┬─────┬─────┐ │ r0 │ r1 │ r2 │ r3 │ r4 │ r5 │ └─────┴─────┴─────┴─────┴─────┴─────┘ ↑ ↑ startOffset endOffset (after truncation of 0-2) ``` The queue tracks a **start offset** — the absolute offset of the first physically present record. When leading records are truncated, the start offset advances. The **end offset** is the next offset to be assigned (exclusive). The number of physically present records is always `endOffset - startOffset`. ### Consumer Model Unlike traditional queues where consuming a message removes it, Lattice Queue records persist until explicitly truncated. Multiple consumers read independently by maintaining their own **consumer offset** — the next offset they wish to read. This decouples production from consumption. A fast consumer reads records as they arrive; a slow consumer catches up at its own pace; a new consumer can start from any offset. Producers and consumers need not be online at the same time. Consumer offsets are external to the queue itself. They MAY be stored in a [KV Database](../037_kv_database/index.md), in application state, or in any other persistent store. ### Single-Leader Append A Lattice Queue follows a **single-leader** model: one writer appends records, and replicas converge by adopting the longer log. This mirrors Kafka's partition model where each partition has exactly one leader that accepts writes. This design ensures that offset assignment is deterministic and conflict-free. Multiple replicas of the same queue always agree on which record occupies each offset. Applications requiring multiple independent writers SHOULD use separate queues per writer, or coordinate writes through a single leader node. ### Topics and Partitions Following the Kafka model, queues are grouped into **topics**. A topic is a named collection of partitions, where each partition is an independent Lattice Queue with its own offsets and single-leader append semantics. Partitioning enables: - **Parallelism** — producers and consumers work on different partitions concurrently - **Scaling writes** — each partition has its own leader, so a topic as a whole can accept writes from multiple nodes - **Key-based routing** — records with the same key are routed to the same partition, preserving per-key ordering When a record is offered to a topic with a key, the partition is selected deterministically from the key hash: `partition = floorMod(hash(key), numPartitions)`. (`floorMod` rather than `abs(...) % n`, so the result is always in range even for extreme hash values.) Topic metadata (such as `:num-partitions`) is carried in a separate metadata map alongside the partitions. ### Truncation Truncation advances the start offset and discards records before it. This reclaims storage for records that all consumers have processed. Truncation is monotonic: the start offset can only increase. In a lattice merge, the higher start offset wins, ensuring truncation decisions propagate consistently across replicas. ## Specification ### Global Lattice Region :::note Provisional structure The naming and structure of the global `:queue` region is still being refined and may change in future revisions. This section documents the current implementation. ::: The standard lattice ROOT registers the `:queue` region as: ``` :queue → OwnerLattice → MapLattice → TopicLattice ``` That is: each owner has a signed map of topic names to topics. The full path to a specific partition's queue state is: ``` :queue / → Signed({ → TopicState, ...}) ``` ### Topic State The topic state is an Index with two keys: | Key | Type | Description | |-----|------|-------------| | `:partitions` | Map | Partition map: `{partition-id → queue state, ...}` | | `:meta` | Map | Topic metadata (e.g. `:num-partitions`) | Topic merge semantics: 1. **Partitions** — per-partition merge, delegating each partition to the queue merge below 2. **Metadata** — map union (own entry wins on key conflict) Foreign topic values are never accepted wholesale: each partition is validated and merged through the queue lattice individually, so a malformed partition map cannot wedge subsequent merges. ### Queue State The queue state is a positional vector with four slots: | Index | Field | Type | Description | |-------|-------|------|-------------| | 0 | entries | Vector | Append-only vector of entry records | | 1 | metadata | Map | Queue metadata (name, configuration, provenance) | | 2 | timestamp | Integer | Last update timestamp (epoch milliseconds) | | 3 | startOffset | Integer | Absolute offset of the first entry in the vector | The absolute offset of `entries[i]` is `startOffset + i`. ### Queue Entry (Record) Each record is a positional vector with four slots, designed for Kafka compatibility: | Index | Field | Type | Description | |-------|-------|------|-------------| | 0 | key | any / nil | Record key (for partitioning, compaction, or routing) | | 1 | value | any | Record payload | | 2 | timestamp | Integer | Producer timestamp (epoch milliseconds) | | 3 | headers | Map / nil | Arbitrary key-value metadata | The **key** field is optional. When present, it enables downstream processors to partition, group, or compact records by key — the same role keys play in Kafka. The **headers** field carries out-of-band metadata (correlation IDs, tracing context, content types) without modifying the payload. ### Merge Semantics Queue merge follows a single-leader convergence model. Given two copies of the same queue, the merge produces the version with the most information: 1. **Start offset** — take the maximum (truncation is monotonic and irreversible) 2. **Timestamp** — take the maximum (most recent update wins) 3. **Metadata** — map union (include keys from both sides; own entry wins on key conflict) 4. **Entries** — align both vectors to the merged start offset by trimming any prefix that falls below it, then take the longer vector These rules satisfy the lattice properties: - **Commutative**: merge(a, b) produces the same result as merge(b, a) for all convergent state - **Associative**: merge(merge(a, b), c) = merge(a, merge(b, c)) - **Idempotent**: merge(a, a) = a Under the single-leader model, entries at the same offset are always identical across replicas, so the "take longer vector" rule is sufficient. If entries were to differ (a violation of the single-leader assumption), the merge is still well-defined: the first operand's entries take precedence. ### Metadata The metadata map carries queue-level configuration and identity information. Metadata keys SHOULD be keywords. Example keys: | Key | Description | |-----|-------------| | `:name` | Human-readable queue name | | `:owner` | Owner identity (DID, address, or public key) | | `:created` | Creation timestamp | | `:retention` | Retention policy hint (e.g. max age or max records) | Metadata merges by map union: keys from both sides are included, with the local entry winning on conflict. Applications SHOULD treat metadata as configuration that is set once or updated infrequently. ## Operations ### Producer | Operation | Description | |-----------|-------------| | `offer(value)` | Append a record with the given value; auto-assigns timestamp | | `offer(key, value)` | Append a keyed record | | `offer(key, value, headers)` | Append a full record with headers | All `offer` variants return the absolute offset assigned to the new record. ### Consumer | Operation | Description | |-----------|-------------| | `peek(offset)` | Read the value at an absolute offset (nil if out of range) | | `peekEntry(offset)` | Read the full record at an absolute offset | | `peekFirst()` / `peekLast()` | Read the first / last value | | `peekFirstEntry()` / `peekLastEntry()` | Read the first / last full record | | `range(from, to)` | Read values in an offset range (inclusive) | Consumer operations are read-only and do not modify the queue. ### Queue Info | Operation | Description | |-----------|-------------| | `startOffset()` | First valid absolute offset | | `endOffset()` | Next offset to be written (exclusive) | | `size()` | Number of records physically present | | `isEmpty()` | Whether the queue contains any records | ### Metadata | Operation | Description | |-----------|-------------| | `getMeta(key)` | Read a metadata value | | `setMeta(key, value)` | Write a metadata value | ### Truncation | Operation | Description | |-----------|-------------| | `truncate(newStartOffset)` | Advance start offset and discard earlier records | Truncation only advances forward. A `newStartOffset` less than or equal to the current start offset is a no-op. ### Lattice Operations | Operation | Description | |-----------|-------------| | `fork()` | Create an independent copy for isolated work | | `sync()` | Merge a forked copy back to its parent | Fork and sync follow the standard [Lattice Cursor](../035_cursors/index.md) semantics. ## Reference Implementation A reference implementation is provided in the Convex `convex-core` module (Java), in the `convex.lattice.queue` package. | Concept | Class | |---------|-------| | Queue state lattice | `QueueLattice` | | Record structure utilities | `QueueEntry` | | Queue API | `LatticeQueue` | | Topic state lattice | `TopicLattice` | | Topic API (partitions, metadata, keyed routing) | `LatticeTopic` | | Concurrent topic wrapper (per-partition locking) | `ConcurrentTopic` | | Message queue system (topics map) | `LatticeMQ` | ## See Also - [CAD024: Data Lattice](../024_data_lattice/index.md) — Theoretical foundation for lattice data structures - [CAD035: Lattice Cursors](../035_cursors/index.md) — Cursor system for atomic state access, fork, and sync - [CAD036: Lattice Node](../036_lattice_node/index.md) — Network replication infrastructure - [CAD037: KV Database](../037_kv_database/index.md) — Complementary state-oriented lattice data structure - [CAD038: Lattice Authentication](../038_lattice_auth/index.md) — Owner verification for signed lattice values --- # CAD041: Model Context Protocol (MCP) ## Overview This CAD describes the Convex implementation of the [Model Context Protocol (MCP)](https://modelcontextprotocol.io/), a standardised protocol that enables AI agents to discover, understand, and invoke tools — including executing economically significant operations — through a structured interface. MCP provides a universal language for AI agents to interact with Convex, allowing them to query data, execute transactions, manage digital assets, and receive real-time state change notifications without requiring custom integrations. ## `/mcp` Endpoint Convex peers MAY implement MCP endpoint support through a dedicated `/mcp` endpoint that exposes Convex network capabilities to AI agents and other MCP-compatible clients. The endpoint supports three HTTP methods: - **POST `/mcp`** — JSON-RPC requests (tools, prompts, queries, transactions). Supports both standard JSON responses and Server-Sent Events (SSE) streaming when the client sends `Accept: text/event-stream`. - **GET `/mcp`** — Opens an SSE stream for receiving asynchronous notifications (state change events, keep-alive pings). Requires a valid `Mcp-Session-Id` header. - **DELETE `/mcp`** — Terminates a session and cleans up associated resources (watches, connections). A discovery document is published at `/.well-known/mcp` for auto-detection by MCP clients. ## Protocol Version The Convex MCP server implements MCP specification version **2025-11-25** with the Streamable HTTP transport. On `initialize`, the server negotiates the protocol version with the client: if the client requests a version the server supports (currently `2025-11-25`, `2025-06-18`, or `2025-03-26`), the server echoes that version back. Otherwise the server responds with its latest supported version and the client decides whether to proceed. This keeps older MCP clients working whilst allowing newer clients to use the latest spec features. ## Sessions Stateful sessions are created on `initialize` and identified by the `Mcp-Session-Id` response header. Sessions track: - SSE connections for notification delivery - State watches registered via `watchState` - Session-scoped resources Sessions are terminated explicitly via `DELETE /mcp` or cleaned up on connection timeout. ## Tools The MCP server exposes tools organised into categories. Clients discover available tools via `tools/list`. The full set depends on whether the peer's signing service is configured. ### Core tools (always available) 22 tools covering queries, transactions, cryptography, encoding, and state watching: - **Queries and state** — `query` (execute Convex Lisp), `queryState` (navigate the global state tree), `describeAccount`, `getBalance`, `lookup`, `resolveCNS`, `peerStatus`, `getTransaction` - **Transactions** — `transact` (direct execution with seed), `prepare` / `submit` (two-step with external signing), `signAndSubmit`, `transfer` (convenience for coin/token transfers) - **Accounts** — `createAccount` (with optional faucet funding) - **Cryptography** — `keyGen`, `sign`, `validate`, `hash` (Ed25519 and SHA-256/SHA3) - **Encoding** — `encode` / `decode` (CVM literals ↔ CAD3 binary) - **State watching** — `watchState` / `unwatchState` (register watches on state paths; changes delivered as SSE notifications) ### Signing service tools (conditional) When a peer is configured with a signing service, 13 additional tools are registered for server-side key management. Private keys are stored encrypted and never leave the server. - **Key management** — `signingCreateKey`, `signingListKeys`, `signingSign`, `signingGetJWT` - **Convenience** — `signingTransact` (execute using a stored key), `signingCreateAccount`, `signingListAccounts` - **Delegation** — `signingDelegate` (issue a UCAN delegation token signed by a stored key, without exposing the key) - **Elevated** — `signingImportKey`, `signingExportKey`, `signingDeleteKey`, `signingChangePassphrase` — these require a two-step browser confirmation flow to prevent programmatic abuse - **Discovery** — `signingServiceInfo` (check availability, no authentication required) ## Prompts The MCP server exposes guided workflow prompts via `prompts/list` and `prompts/get`. Prompts are user-controlled templates (triggered by slash commands or explicit selection) that guide LLMs through multi-step tasks using available tools. Each prompt follows a three-message pattern: a **persona** message that teaches the LLM relevant Convex domain knowledge, a **request** message with `${argName}` placeholders substituted from user-supplied arguments, and an **assistant prefill** that anchors the response direction. Some prompts are always available (e.g. account exploration, network status, Convex Lisp guidance); others are registered only when the signing service is configured (e.g. account creation, contract deployment, fund transfers). ## Server-Sent Events (SSE) The MCP server supports real-time notifications via SSE for state watching. When a client registers a watch with `watchState`, the server monitors the specified path in the global CVM state and sends `notifications/stateChanged` events over the SSE stream whenever the value changes. SSE is also used for streaming responses on POST when the client includes `Accept: text/event-stream`. ## Extensibility The `McpServer` base class is designed for extension. Subclasses can override protected methods to customise tool discovery, tool dispatch, and protocol negotiation: | Method | Purpose | |--------|---------| | `buildInitializeResult(params)` | Customise the `initialize` response (capabilities, server info) | | `listTools()` | Return a custom set of available tools | | `toolCall(params)` | Dispatch tool calls with custom logic | | `handlePost(ctx)` | Override full POST request handling | | `createResponse(id, result)` | Customise JSON-RPC response construction | This enables: - **Adapter-based tool discovery** — dynamically register tools from loaded adapters (used by Covia venues) - **Custom tool dispatch** — route tool calls to domain-specific handlers - **Protocol extensions** — add custom JSON-RPC methods beyond the MCP specification - **Batch request handling** — the base implementation supports up to 20 requests per batch ### Tool Registration External modules register tools via the tool registry: ```java mcpServer.registerTool("my-tool", schema, handler); ``` Registered tools appear in `tools/list` responses and are dispatched by `toolCall()`. ## Security Considerations - **Seed-based tools** (`transact`, `signAndSubmit`, `transfer`) transmit the Ed25519 seed over the network — peers MUST use HTTPS. Servers refuse seed-carrying requests over cleartext HTTP from non-loopback clients (HTTPS or `X-Forwarded-Proto: https` accepted; an explicit configuration option opts out for trusted private networks) - **Origin allow-list** — servers MAY be configured with a set of allowed `Origin` values; requests carrying a different Origin are rejected with 403. This provides the DNS-rebinding protection the MCP spec requires for localhost and private deployments. Public peers default to allowing all origins, and requests without an Origin header (non-browser clients) always pass - **Signing service tools** keep private keys server-side, encrypted at rest with a user-chosen passphrase - **Elevated operations** (import, export, delete) require interactive browser confirmation to prevent programmatic abuse - **Queries** are read-only and require no authentication - **Sessions** are server-side and scoped to a single client connection ## Reference Implementations - **Server** — [`convex-restapi`](https://github.com/Convex-Dev/convex/tree/develop/convex-restapi) ships the `/mcp` endpoint as part of every Convex peer, covering the full tool and prompt set described above. - **Claude Code plugin** — [`Convex-Dev/convex-plugin`](https://github.com/Convex-Dev/convex-plugin) is a one-line install that binds the peer's MCP endpoint, injects Convex terminology and tool-selection guidance, and ships convenience slash commands (`/convex:faucet`, `/convex:explore`) along with a Convex-Lisp-aware subagent. ## References - [Model Context Protocol Specification](https://modelcontextprotocol.io/specification/2025-11-25) - [MCP Streamable HTTP Transport](https://modelcontextprotocol.io/specification/2025-11-25/basic/transports) - [CAD004: Accounts](../004_accounts/index.md) - [CAD026: Convex Lisp](../026_lisp/index.md) - [CAD019: Asset Model](../019_assets/index.md) - [Convex MCP Product Page](/docs/products/convex-mcp) --- # CAD042: x402 Protocol ## Overview This CAD describes the Convex implementation of the [x402 protocol](https://www.x402.org/), an open protocol for internet-native payments built around the HTTP 402 status code. x402 enables users to pay for resources via API without registration, emails, OAuth, or complex signatures. The protocol activates the dormant HTTP 402 status code and works with any HTTP stack through headers and status codes. :::note This CAD is currently under development. We implementing the x402 protocol for Convex, which will enable frictionless micropayments and payment-gated API access using Convex's native payment infrastructure, with support for CVM and any CAD29 fungible tokens. ::: ## References - [x402 Protocol Specification](https://www.x402.org/) --- # CAD043: Decentralised Identity ## Overview This CAD specifies Decentralised Identity (DID) support for the Convex network, covering identity methods, on-chain representation, and interoperability with the wider DID ecosystem. ## DID Methods ### `did:convex` The native Convex DID method. A `did:convex` identifier is either a numeric account address or a CNS name: ``` did:convex:13 did:convex:id.foo did:convex:user.mike ``` The method-specific identifier is self-describing: - **Purely numeric identifiers** resolve to account addresses (`did:convex:13` → account `#13`). These are **account DIDs**: the DID subject is the account itself, stable for the lifetime of the account, with key rotation under the account's own authority. - **All other identifiers** resolve as Convex Name System names (CAD014). These are **named DIDs**: human-readable identifiers managed through CNS records. No prefix is needed to distinguish the two forms. This is unambiguous by construction: purely numeric top-level CNS names MUST NOT be created (a permanent governance invariant, stated in CAD014), so no valid CNS name can be mistaken for an account address. The `id` root namespace is the primary home for named DIDs: open access, with names obtainable for a small Convex Coin cost per CAD014. Names in other namespaces (e.g. `user.mike`) are equally valid `did:convex` identifiers. Convex models a single universal global state, so `did:convex` identifiers contain no network qualifier. Resolvers are configured to point at the appropriate network — by default the Convex main network. For testing, a resolver can be pointed at a testnet or local peer without changing the identifiers themselves. Key properties: - Cryptographic authentication via Ed25519 account keys - On-chain resolution through the CVM - Controller-based delegation and recovery - Human-readable identifiers via the Convex Name System (CNS) #### Resolution of named DIDs A named DID is resolved by looking up its CNS record. The record's **value** determines the DID subject: - An **account address** — the DID document is derived from that account (see DID Documents below), with the account DID listed in `alsoKnownAs` as the canonical stable identifier. - A **scoped reference to a DID registry record** (e.g. `[ ]`) — the DID document is constructed from the on-chain registry record (see On-Chain Representation below), including any stored DID document content and authorised accounts. Resolvers MUST return `deactivated` metadata where the underlying registry record is deactivated, and SHOULD fail resolution (not found) where the CNS name does not exist or its value is not resolvable to a DID subject. Named DIDs are **mutable aliases**: the controller of the CNS record (or the owner of a parent CNS node — see the CAD014 authority model) can repoint the name, changing the resolved subject, keys and controller. This is intended behaviour — it is what makes recovery and delegation possible — but verifiers requiring a stable subject MUST pin the canonical account DID from `alsoKnownAs` rather than the name. ### `did:key` Support for the `did:key` method, enabling self-certifying identifiers derived from public keys without requiring on-chain registration. `did:key` identifiers in the Convex ecosystem SHOULD use Ed25519 keys for maximum compatibility with Convex account authentication: ``` did:key:z6Mkf5rGMoatrSj1f4CyvuHBeXJELe9RPdzo2PKGNCKVtZxP ``` The Ed25519 public key in a `did:key` identifier can be used directly as a Convex account key, enabling direct bridging between off-chain `did:key` identities and on-chain accounts. Useful for: - Ephemeral agent identities - Off-chain verification prior to on-chain account creation - Interoperability with external DID ecosystems See the [did:key Method Specification](https://w3c-ccg.github.io/did-method-key/) for the full method definition. ### `did:web` Clients in the Convex ecosystem SHOULD support the `did:web` method for resolving identities hosted at web domains. When a `did:web` identifier points to a Convex peer, it works equivalently to `did:convex` — the peer resolves the identity against on-chain state and returns a standard DID document. The path component carries the same self-describing identifier as `did:convex`: ``` did:web:peer.convex.live:13 did:web:peer.convex.live:id.foo did:web:peer.convex.live:user.mike ``` This enables: - Domain-anchored identities with human-readable names - Resolution via standard HTTPS without specialised Convex tooling - Equivalent semantics to `did:convex` when the host is a Convex peer - Compatibility with the broader `did:web` ecosystem for non-Convex hosts Convex peers SHOULD serve DID documents at the standard `did:web` resolution paths: - `GET /.well-known/did.json` — the peer's own DID document - `GET /{identifier}/did.json` — DID document for an account or CNS name Per the [did:web method specification](https://w3c-ccg.github.io/did-method-web/), colons in the method-specific identifier map to `/` in the resolution URL, and a non-default port MUST be percent-encoded (`did:web:example.com%3A8080:13`). Peers serving on non-default ports MUST construct document `id` values accordingly, so that the served `id` matches the DID being dereferenced. ## DID Documents A Convex DID document follows the [W3C DID Core](https://www.w3.org/TR/did-core/) structure. For user accounts with an Ed25519 key, the document includes a verification method and authentication relationship: ```json { "@context": "https://www.w3.org/ns/did/v1", "id": "did:web:peer.convex.live:13", "controller": "did:web:peer.convex.live:13", "verificationMethod": [{ "id": "did:web:peer.convex.live:13#key-1", "type": "Ed25519VerificationMethod2020", "controller": "did:web:peer.convex.live:13", "publicKeyMultibase": "z6Mkf5rGMoatrSj1..." }], "authentication": ["did:web:peer.convex.live:13#key-1"], "alsoKnownAs": [ "did:convex:13", "did:key:z6Mkf5rGMoatrSj1..." ] } ``` ### `alsoKnownAs` DID documents SHOULD include an [`alsoKnownAs`](https://www.w3.org/TR/did-core/#also-known-as) property linking equivalent identifiers across methods: - **`did:convex`** — always included, mapping to the canonical on-chain account address - **`did:key`** — included when the account has an Ed25519 public key, using the [multicodec](https://github.com/multiformats/multicodec) prefix `0xed01` for Ed25519 This enables verifiers to confirm that a `did:web`, `did:convex`, and `did:key` identifier all refer to the same subject. ### Named DID documents For a named DID (e.g. `did:convex:id.foo`): - `id` is the named form (`did:convex:id.foo`, or the `did:web` equivalent) - `alsoKnownAs` MUST include the canonical account DID currently resolved to (e.g. `did:convex:13`), plus `did:key` where an Ed25519 key exists - `controller` SHOULD reflect the party able to change the resolution — the CNS record controller — expressed as a `did:convex` account DID where the controller is a plain account. Where the controller is a complex trust monitor, `controller` MAY default to the document's own `id`. - Where the CNS record resolves to a DID registry record, document content (services, additional verification methods) is drawn from the stored record, and resolution metadata includes `created`, `updated` and `deactivated` from the record. ### Actor Accounts Actor accounts (smart contracts) do not have public keys. Their DID documents contain `id`, `controller`, and `alsoKnownAs` (with `did:convex`) but omit `verificationMethod` and `authentication`. ## On-Chain Representation On-chain DID state uses two complementary mechanisms: 1. **Accounts** — every account is implicitly a DID subject (`did:convex:
`), with its DID document derived from account state (public key, controller). No registration is required. 2. **The DID registry actor** (`convex.did` in CNS) — an on-chain registry storing DID records for named identities that need more than account-derived documents: stored DID document content, authorised account sets, controller-based management and deactivation. A registry record logically contains: - **document** — stored DID document content (or `nil` when deactivated) - **controller** — trust monitor authorising updates (`:update`) and control transfer (`:control`) - **created** / **updated** — timestamps maintained by the registry - **authorised accounts** — a set of accounts entitled to act on behalf of the identity Registry records implement the W3C required operations: Create, Read, Update and Deactivate. Deactivation is terminal: a deactivated DID MUST NOT be reactivated, and resolvers MUST report `deactivated: true` for it. Registry records also function as trust monitors: a scoped reference `[ ]` is trusted for a subject when the subject is in the record's authorised account set and the record is active. This allows a DID to be used directly wherever a CAD022 trust monitor is expected. Registry record identifiers are internal. Human-readable naming is provided by CNS: a name in the `id` namespace holds a scoped registry reference as its CNS record value, per the resolution rules above. This enables: - Decentralised resolution without external registries - Atomic updates via CVM transactions - Verifiable credential anchoring - Integration with CAD019 assets and CAD014 CNS names ## Security Considerations ### Named DIDs are mutable aliases The resolution of a named DID can change whenever its CNS record is updated — by the record controller, or by the owner of any parent CNS node (see CAD014 authority model and its security considerations). Verifiers MUST pin the canonical account DID from `alsoKnownAs` where a stable subject is required, and SHOULD assess the trustworthiness of the CNS path's controllers and node owners as part of accepting a named DID. ### Deactivation Deactivation of a registry-backed DID is permanent. Loss of a controller does not deactivate a DID, but renders it immutable in practice; controllers SHOULD be recoverable trust monitors (e.g. with a backup controller) for long-lived identities. ### Controller validity Registry operations MUST validate that a new controller is a plausible trust monitor (an existing account or scoped reference). In particular, setting a controller to a non-existent account address is dangerous: account addresses are allocated sequentially, so control would silently pass to whoever later creates that account. :::note This CAD is under development. The specification will be extended to cover authentication flows and verifiable credential anchoring. The current DID registry implementation (`convex/lab/did.cvx`) predates parts of this specification and is being upgraded to match (controller validation, terminal deactivation, resolver metadata). ::: ## References - [W3C DID Core Specification](https://www.w3.org/TR/did-core/) - [did:key Method](https://w3c-ccg.github.io/did-method-key/) - [did:web Method](https://w3c-ccg.github.io/did-method-web/) - [CAD004: Accounts](/docs/cad/accounts) - [CAD014: Convex Name System](/docs/cad/cns) - [CAD022: Trust Monitors](/docs/cad/trustmon) --- # CAD044: JSON on the Lattice ## Overview JSON (JavaScript Object Notation) is a widely-used data interchange format for web applications and APIs. Convex provides native support for JSON-compatible data structures, enabling direct interoperability between the CVM, the Lattice and external systems. **Key Highlight**: JSON is a *strict subset* of CVM data types, meaning any valid JSON structure can be represented directly in Convex without difficulty. ## JSON as a Subset of CVM Types The CVM data type system includes all types necessary to represent JSON: - **Numbers**: JSON numbers map to CVM `Integer` (integers) or `Double` (floating point) values - **Strings**: JSON strings map directly to CVM `String` values - **Booleans**: JSON `true`/`false` map to CVM `Boolean` values - **Null**: JSON `null` maps to CVM `nil` - **Arrays**: JSON arrays map to CVM `Vector` values - **Objects**: JSON objects map to CVM `Map` values with `Keyword` or `String` keys This means that any valid JSON structure can be represented as a native CVM value without any loss of information or structural changes. ## Conversion: Convex → JSON Converting CVM values to JSON format is straightforward for JSON-compatible data structures: ### Directly equivalent Types The following CVM types convert directly to JSON and can be reliably round-tripped: - `Long` and `Double` → JSON numbers - `String` → JSON strings - `Boolean` → JSON booleans - `nil` → JSON `null` - `Vector` → JSON arrays - `Map` with `Keyword` or `String` keys → JSON objects ### CVX to JSON Conversion CVX to JSON translation is: - **complete** - any CVX value can be mapped to JSON - **lossy** - some type information is lost When mapping CVX format to JSON, implementations SHOULD adhere to the following general rules: - **Keywords**: CVM `Keyword` values (e.g. `:foo`) are converted to JSON strings by removing the leading colon (e.g. `"foo"`) - **Map Keys**: `Keyword`, `Symbol` and `String` keys in Maps are JSON-compatible. Other types are printed as String keys. - **Blobs**: Converted to hex strings. A leading "0x" is recommended. - **Address**: Converted to the integer value of the Address - **Lists/Vectors/Sets**: Converted to JSON arrays. The type distinction between them is lost. - **Numbers**: Large integers beyond JavaScript's safe integer range can be printed as valid JSON but may lose precision when read by other applications - **Ordering**: JSON object key ordering is not guaranteed to match the original CVM Map ordering, as Maps use hash-based ordering ### JSON to CVX conversion JSON conversion to CVX is simple as all JSON types can be represented directly on the CVM according to their equivalent types. Applications utilising such conversions SHOULD be prepared to recognise possible JSON representations. For example, if an argument required an Address, then `"#13"`, `"13"` and `13` should all be accepted. Utility functions in `convex.core` such as `Address.parse(...)` are designed to handle such cases. ### Example ```clojure ;; A CVM value that is JSON-compatible {:name "Alice" :age 30 :active true :tags ["developer" "convex"] :metadata {:level 5 :score 100.5} } ``` Maps to: ```json { "name": "Alice", "age": 30, "active": true, "tags": ["developer", "convex"], "metadata": { "level": 5, "score": 100.5 } } ``` ## Serialisation: Strict JSON and JSON5 Convex serialises CVM values to text in two output modes: **Strict JSON** (`JSON.appendJSON` / `JSON.printJSON`) — RFC 8259-compliant output. Since JSON has no representation for non-finite floating-point values, a non-finite `Double` (`NaN`, `Infinity`, `-Infinity`) is emitted as `null`, matching the behaviour of JavaScript's `JSON.stringify`. **JSON5** (`JSON.appendJSON5` / `JSON.printJSON5`) — a superset of strict JSON. It differs only in its treatment of non-finite doubles, which render as the unquoted literals `NaN`, `Infinity` and `-Infinity` rather than `null`, so numeric values round-trip without loss. JSON5 is the format used for human- and agent-readable rendering of lattice data (see [CAD046: Cell Explorer](../046_cell_explorer/index.md)). Both modes follow the type mappings above; they differ only in how non-finite numbers are handled. ## Usage Notes When designing CVM data structures intended for JSON conversion: - Prefer `Keyword` or `String` keys in Maps for JSON compatibility - Use `Vector` for ordered sequences (JSON arrays) - Beware of CVM-specific types like `Address`, `Blob`, `Symbol`, or `Set` if JSON conversion is required - Consider using `nil` instead of omitting optional fields, as JSON supports `null` values ## See Also - [CAD002: CVM Values](../002_values/index.md) - Complete specification of CVM data types - [CAD026: Convex Lisp](../026_lisp/index.md) - Language features including JSON superset support - [CAD003: Encoding](../003_encoding/index.md) - CAD3 encoding format and JSON compatibility --- # CAD045: Lattice Applications ## Overview Lattice Applications are decentralised, self-sovereign data applications built on the [Data Lattice](../024_data_lattice/index.md). They use [Lattice Cursors](../035_cursors/index.md) as their primary interface for reading, writing, navigating and synchronising state, and compose [Lattice Types](../024_data_lattice/index.md) to define merge semantics for their domain data. Unlike traditional client-server applications where a central authority mediates access, lattice applications operate in a peer-to-peer environment where each participant owns and controls their own data. Conflict resolution is automatic — the algebraic properties of the lattice (commutativity, associativity, idempotence) guarantee that peers always converge to the same state without coordination. This CAD defines the architecture, composition rules and best practices for building lattice applications, using a social networking application (`convex-social`) as a running example. ## Motivation The Data Lattice provides a powerful substrate for decentralised applications, but the raw lattice primitives (merge functions, lattice types, cursors) leave significant design decisions to application developers. Without clear guidance, common mistakes include: - **Incorrect merge semantics** — choosing merge strategies that violate CRDT properties or lose data - **Type mismatches** — writing through uninitialised paths that create wrong container types - **Missing signing boundaries** — forgetting that self-sovereign data requires cryptographic ownership enforcement - **Monolithic state** — failing to decompose state into independently mergeable components This CAD establishes patterns that ensure lattice applications are correct, composable and secure by construction. ## Specification ### Application Architecture A lattice application MUST be structured in four layers: | Layer | Responsibility | Example | |-------|---------------|---------| | **Application API** | Domain-specific operations | `Social.post()`, `Feed.delete()` | | **Cursor Chain** | Navigation, atomic writes, signing | `path()`, `updateAndGet()`, `fork()`/`sync()` | | **Lattice Hierarchy** | Merge semantics, type information | `SocialLattice`, `IndexLattice`, `LWWLattice` | | **Node Infrastructure** | Networking, persistence, replication | `NodeServer`, `LatticePropagator` | The application layer MUST NOT call lattice merge functions directly. All reads and writes MUST go through cursors, which handle atomic updates, lattice-aware writes and transparent signing. The lattice hierarchy MUST NOT have knowledge of cursors. Lattice types define merge semantics only — they are pure functions over immutable values. ### Lattice Design #### Composition from Primitives Applications SHOULD compose their lattice hierarchy from the standard lattice types defined in [CAD024](../024_data_lattice/index.md): | Primitive | Merge Semantics | Zero | Use Case | |-----------|----------------|------|----------| | **LWW Lattice** | Higher timestamp wins | `nil` | Mutable registers (profiles, individual records) | | **Index Lattice** | Union of keys, child merge per entry | Empty Index | Ordered collections (feeds, logs, time-series) | | **Map Lattice** | Union of keys, child merge per entry | Empty Map | Unordered collections (follow lists, metadata) | | **Set Lattice** | Set union | Empty Set | Grow-only memberships (tags, permissions) | | **Owner Lattice** | Per-key merge with signature verification | Empty Map | Self-sovereign per-owner namespaces | | **Signed Lattice** | Signature verification, delegate to child | `nil` | Cryptographic ownership enforcement | | **Keyed Lattice** | Per-key dispatch to distinct child lattices | Empty Index | Root-level sections with heterogeneous children | Composition reads bottom-up from the data model. For example, a social feed is an ordered collection of posts where each post uses last-write-wins for edits: ``` Feed = IndexLattice(LWWLattice) ↑ union of posts ↑ per-post conflict resolution ``` #### Custom Lattice Types When a data structure has multiple named children with different merge strategies, applications SHOULD define a custom lattice type. A custom lattice type MUST implement: - **`merge(own, other)`** — the core merge function, satisfying commutativity, associativity and idempotence as specified in [CAD024](../024_data_lattice/index.md) - **`zero()`** — the identity element, used by cursors for auto-initialisation of empty paths - **`path(key)`** — returns the child lattice type for a given key, enabling lattice-aware cursor navigation The `path()` method is essential for the cursor system. It tells cursors what merge semantics exist at each level of the data hierarchy, and enables `assocIn` to create correctly-typed containers when writing through uninitialised paths (see [CAD035 § Lattice Integration](../035_cursors/index.md#lattice-integration)). Example: a social user record with feed, profile and follows — each with distinct merge semantics: ``` SocialLattice path(:feed) → IndexLattice(LWWLattice) ordered posts, LWW per entry path(:profile) → LWWLattice single register path(:follows) → MapLattice(LWWLattice) unordered map, LWW per entry zero() → Empty Index ``` #### CRDT Properties Every custom merge function MUST satisfy the three CRDT properties: - **Commutativity**: `merge(a, b) = merge(b, a)` - **Associativity**: `merge(merge(a, b), c) = merge(a, merge(b, c))` - **Idempotence**: `merge(a, a) = a` Violation of these properties causes divergence between peers — nodes that have received the same data in different orders will hold different state, with no mechanism for convergence. Applications SHOULD use deterministic tiebreakers (e.g. hash comparison) when timestamps are equal, to preserve commutativity. A merge function that picks "first argument on tie" is not commutative. Applications MUST handle null values in merge: `merge(nil, x) = x` and `merge(x, nil) = x`. This is required for the lattice identity law and for correct initialisation of new entries. #### Zero Values and Container Types The `zero()` method MUST return the correct container type for the lattice level. Cursors use `zero()` to auto-initialise empty paths during writes. If `zero()` returns the wrong type (e.g. a hash map where an index is expected), subsequent merge operations will fail with type mismatches. For container lattices (Index Lattice, Map Lattice, Keyed Lattice), `zero()` MUST return the appropriate empty container. For leaf lattices (LWW Lattice), `zero()` SHOULD return `nil`. #### Lattice Continuity The lattice hierarchy MUST be a continuous tree. Once `path(key)` returns `nil` at some level, no child lattice exists at or below that key. There is no mechanism for lattice semantics to resume after a gap, because there is no lattice object to call `path()` on. See [CAD035 § Lattice Integration](../035_cursors/index.md#lattice-integration) for how cursors handle navigation beyond the lattice boundary. If an application requires lattice merge semantics at a deeper level, the lattice hierarchy MUST be extended to be continuous through the intervening path. ### Self-Sovereign Data #### Owner Lattice Applications that manage per-user data SHOULD wrap their application lattice with an Owner Lattice. This partitions the state by owner public key, with each owner's data wrapped in signed data ([CAD038](../038_lattice_auth/index.md)). The resulting structure is: ``` Application Section (e.g. :social) └── OwnerLattice └── → SignedData └── Application Lattice (e.g. SocialLattice) ``` This structure enforces that only the holder of the owner's private key can create or update values within their partition. During network merge, the Owner Lattice verifies that the signer key matches the owner key and rejects forgeries. #### Signing Boundary Applications MUST NOT attempt to handle signing directly. The cursor system inserts a Signed Cursor automatically when `path()` crosses a Signed Lattice boundary. All writes through the cursor chain are signed transparently using the key pair from the Lattice Context. Applications MUST set a Lattice Context with a valid signing key pair before performing writes that cross a signing boundary. Failure to do so will result in an error at the Signed Cursor. #### Local Trust vs Network Verification Applications SHOULD understand the two-level trust model: - **Local writes** are always trusted. A cursor allows any write to any path — the signing boundary signs whatever it is given. This is correct because local state is under the operator's own control. - **Network merges** are verified. When data is received from another peer, the Owner Lattice checks that the signer is authorised for the owner key. Forgeries are silently rejected. This means an application can write data under any owner's key locally, but that data will be rejected by every other peer if the signing key doesn't match. Applications SHOULD NOT rely on local-only state for security guarantees — security is enforced at the merge boundary. ### Cursor Patterns #### The Wrapper Pattern Each level of the data model SHOULD have a corresponding wrapper class that holds a cursor and exposes domain-specific operations: ``` Social → cursor at OwnerLattice level SocialUser → cursor at SocialLattice level (through signing boundary) Feed → cursor at IndexLattice level Follows → cursor at MapLattice level ``` Each wrapper navigates one level deeper via `cursor.path(key)`. Signing, type management and merge semantics are handled transparently by the cursor chain. Wrapper classes SHOULD: - Hold a single cursor field (the entry point for their level) - Expose domain verbs (`post()`, `follow()`, `delete()`) rather than cursor primitives - Return domain types (post records, follow sets) rather than raw CVM values - Provide a `cursor()` accessor for escape-hatch access to the underlying cursor #### Writing Data Applications SHOULD use `updateAndGet` (or `getAndUpdate`) for read-modify-write operations. When a lattice is present, the update lambda receives the lattice's zero value instead of null for uninitialised paths, eliminating the need for null guards: ``` cursor.updateAndGet(feed -> feed.assoc(key, post)) // feed is auto-initialised to Index.EMPTY if it was null ``` For simple key-value writes, applications MAY use `assoc(key, value)` or `assocIn(value, keys...)` on the cursor. Applications MUST NOT use raw `RT.assocIn` or equivalent non-lattice-aware write utilities to update lattice state. These may create containers of the wrong type (e.g. hash maps where indexes are expected), causing merge failures. All writes through cursors are lattice-aware by construction. #### Reading Data `get()` returns `nil` for uninitialised paths — the zero-substitution only applies within update lambdas. Read-side code SHOULD handle null appropriately. #### Fork and Sync Applications that perform batch operations SHOULD use the fork/sync pattern: 1. **Fork** — `cursor.fork()` creates an isolated working copy 2. **Modify** — perform multiple writes on the fork 3. **Sync** — `cursor.sync()` merges changes back to the parent atomically Sync always succeeds when a lattice is present (see [CAD035](../035_cursors/index.md)). For the signing use case, forking from below a Signed Cursor gives unsigned local storage — signing is deferred to the single `sync()` call, reducing cryptographic overhead for batch operations. Application wrappers SHOULD expose `fork()` and `sync()` on top-level wrapper classes where batch operations are a use case. #### Connecting to Node Infrastructure Applications SHOULD provide two factory methods: - **Standalone** — creates its own root cursor, useful for testing and isolated operation - **Connected** — navigates from a node's root cursor via `path()`, useful for participation in the lattice network The connected pattern is how applications participate in peer-to-peer replication. Writes propagate up through the cursor chain to the node's root, where the propagator broadcasts deltas to peers (see [CAD036](../036_lattice_node/index.md)). ### Data Model #### Record Design Applications using LWW Lattice for conflict resolution MUST include a `:timestamp` field in every record. The default LWW merge function extracts the timestamp from this field to determine the winner. Applications SHOULD use Keyword keys for record fields. Keywords are interned, compact, and provide fast comparison. #### Ordered Collections Applications that need chronological ordering (feeds, logs, event streams) SHOULD use Index with blob keys encoding the timestamp in big-endian format. Big-endian encoding ensures lexicographic blob ordering matches chronological ordering. #### Tombstone Deletion Lattice merge is union-based — merged collections contain the union of all entries from both sides. This means removing an entry from one replica does not remove it from others; the next merge would restore it. Applications MUST use tombstone deletion rather than entry removal. A tombstone is a marker (e.g. a `:deleted` field) added to the existing entry. The entry remains in the collection but is filtered out in read paths. When tombstoning, applications MUST update the `:timestamp` field so that the tombstoned version wins over older non-deleted versions via LWW merge. #### Container Choice | Need | Container | Reason | |------|-----------|--------| | Lattice-level containers (where `path()` and `zero()` matter) | `Index` | Sorted, JSON-compatible key resolution, correct `zero()` type | | Leaf records (post data, profile fields) | Hash Map | Efficient for small unordered maps | | Dynamic key sets (follow lists) | Hash Map | Hash-based, unordered | | Ordered sequences (feeds, logs) | `Index` with blob keys | Lexicographic ordering | | Single mutable values | Direct value with LWW Lattice | Simple register semantics | Applications SHOULD prefer `Index` for lattice-level containers and `AHashMap` for leaf data records. `Index` resolves `Keyword` and `AString` identically (same blob comparison), which matters for JSON interoperability. ### Registration Applications SHOULD register their lattice under a Keyword in the root Keyed Lattice, enabling nodes to opt in to hosting the application's data: ``` root = Lattice.ROOT.addLattice(:social, socialLattice) ``` The keyword becomes the first path element when navigating from the node's root cursor. Different nodes may host different combinations of applications by composing different sets of lattice sections. ### Testing #### Unit Tests Applications SHOULD test domain operations in standalone mode (own cursor, no node infrastructure). This verifies application logic in isolation. #### Integration Tests Applications SHOULD test that writes propagate to the root cursor when connected to a node. This verifies the cursor chain is correctly wired. #### Fork/Sync Tests Applications SHOULD test that forked changes are invisible until sync, and that sync correctly merges changes back. #### Multi-User Tests Applications with per-user data SHOULD test that different users' state is independent and does not interfere. #### Adversarial Tests Applications using Owner Lattice MUST include adversarial tests that verify forgery rejection. These tests SHOULD: 1. Construct forged state at the raw data level (data signed by key A placed under key B's owner slot) 2. Merge the forged state with legitimate state using `OwnerLattice.merge(context, own, other)` 3. Assert that the forgery is rejected and the legitimate data survives This tests the [authentication boundary](../038_lattice_auth/index.md) that protects self-sovereign data in production. ## Reference Implementation The reference implementation is provided in the Convex `convex-core` and `convex-social` Java modules. ### Core Classes | Concept | Class | Package | |---------|-------|---------| | Abstract cursor | `ACursor` | `convex.lattice.cursor` | | Lattice-aware cursor | `ALatticeCursor` | `convex.lattice.cursor` | | Lattice-aware write utility | `LatticeOps` | `convex.lattice` | | Abstract lattice type | `ALattice` | `convex.lattice` | | Merge context | `LatticeContext` | `convex.lattice` | | Standard lattice types | `IndexLattice`, `MapLattice`, `LWWLattice`, `SetLattice`, `OwnerLattice`, `SignedLattice`, `KeyedLattice` | `convex.lattice.generic` | | Root lattice definition | `Lattice` | `convex.lattice` | ### Application Classes (convex-social) | Concept | Class | Package | |---------|-------|---------| | Top-level application wrapper | `Social` | `convex.social` | | Per-user wrapper | `SocialUser` | `convex.social` | | Feed operations wrapper | `Feed` | `convex.social` | | Follow list wrapper | `Follows` | `convex.social` | | Custom lattice type | `SocialLattice` | `convex.social` | | Record construction helpers | `SocialPost` | `convex.social` | | Timeline and query helpers | `SocialHelpers` | `convex.social` | ### Lattice Hierarchy (convex-social) ``` :social → OwnerLattice └── → SignedLattice └── SocialLattice (Index) ├── :feed → IndexLattice(LWWLattice) ├── :profile → LWWLattice └── :follows → MapLattice(LWWLattice) ``` ### Cursor Chain (navigating to a user's feed) ``` RootLatticeCursor [KeyedLattice] → DescendedCursor([:social]) [OwnerLattice] → DescendedCursor([ownerKey]) [SignedLattice] → SignedCursor ← signing enforcement point → DescendedCursor([:feed]) [IndexLattice(LWWLattice)] ``` Design documents for cursor internals (path collapsing, auto-initialisation, fork/sync mechanics) are in `convex-core/docs/LATTICE_CURSOR_DESIGN.md` and `convex-core/docs/LATTICE_APPLICATIONS.md`. ## See Also - [CAD024: Data Lattice](../024_data_lattice/index.md) — Lattice merge foundations and standard lattice types - [CAD035: Lattice Cursors](../035_cursors/index.md) — Cursor specification and lattice integration - [CAD036: Lattice Node](../036_lattice_node/index.md) — Networked lattice replication - [CAD038: Lattice Authentication](../038_lattice_auth/index.md) — Owner verification and signing boundaries - [CAD037: KV Database](../037_kv_database/index.md) — Example lattice application (key-value store) - [CAD028: DLFS](../028_dlfs/index.md) — Example lattice application (distributed filesystem) --- # CAD046: CellExplorer ## Overview CellExplorer is a budget-bounded renderer that produces JSON5 representations of CVM cell values. It enables progressive exploration of arbitrarily large lattice structures within a fixed output budget, making it suitable for LLM context windows, debugging tools, and data previews. ## Motivation CVM values can be arbitrarily large — a lattice root may contain gigabytes of nested data. Rendering such values fully is impractical for: - **LLM context** — language models have finite context windows; injecting a full lattice value would overflow the budget - **Debugging** — developers need a quick preview of a value's structure without materialising the entire tree - **API responses** — REST endpoints need bounded previews of lattice state CellExplorer solves this by rendering values within a caller-specified byte budget. Large values are progressively truncated with human-readable annotations showing what was omitted. ## Specification ### Budget Unit The budget is measured in **CAD3 storage bytes** — the size of a cell's binary encoding plus the encoding of any embedded children. This is computed via `Cells.storageSize(cell)`, which is O(1) for any cell (uses cached memory size from the Etch store). The storage size correlates well with LLM token count and wire size, making it a practical proxy for output cost. ### Output Format CellExplorer produces valid **JSON5** output. All truncation metadata appears in `/* */` comments, which are unambiguous from data values. The output can be parsed by any JSON5 parser; the comments provide additional context but are not required for parsing. ### Rendering Rules #### Leaf Values If a cell fits within the remaining budget, it is rendered fully using standard JSON encoding: | Type | Rendered Form | |------|---------------| | `nil` | `null` | | Boolean | `true` / `false` | | Integer | `42` | | Double | `3.14` | | String | `"hello"` | | Blob | `"0xabcdef"` | | Keyword | `:name` (unquoted if valid identifier) | | Address | `"#42"` | If a leaf does not fit, it is partially rendered: - **Strings**: truncated with annotation: `"hello wo..." /* 4.2KB */` - **Blobs**: truncated hex with annotation: `"0x4865..." /* Blob, 12MB */` - **Other**: `null /* truncated Type, Size */` #### Maps Maps use a **key-first rendering strategy**: show as many entries as possible, truncating values rather than omitting entries. 1. Scan forward, counting entries that fit (key cost + minimum value budget per entry) 2. Allocate remaining budget equally to visible values 3. Render each value; if it exceeds its allocation, give it half the remaining budget (geometric decay) 4. Overflow entries are summarised: `/* +99 more, 390KB */` Map keys follow JSON5 identifier rules: - Keywords render as unquoted identifiers when the name is valid: `name: "value"` (not `":name": "value"`) - Invalid identifiers are quoted: `"hello-world": "value"` #### Sequences (Vectors, Lists) Sequences use a **running-remainder strategy**: 1. For each item, check remaining budget 2. If the item fits, render fully; otherwise give it half the remaining budget 3. Stop when budget is exhausted 4. Overflow items are summarised: `/* +99 more, 390KB */` #### Sets Sets render as JSON5 arrays with a type marker: `[/* Set */]` (empty) or `[1, 2, /* Set: +99 more */]` (partial). #### Fully Truncated Containers When a container has zero budget for children: `{/* Map, 5 keys, 47.5MB */}` ### Size Annotations Size annotations appear when the value is ≥1KB: | Range | Format | Example | |-------|--------|---------| | <1KB | Bytes | `42B` | | <1MB | KB (1 decimal if <10) | `4.2KB`, `13KB` | | <1GB | MB | `3.5MB`, `24MB` | | ≥1GB | GB | `1.2GB` | ### Constants | Constant | Value | Purpose | |----------|-------|---------| | `ANNOTATION_RESERVE` | 30 | Per-container overhead for truncation annotation | | `ENTRY_OVERHEAD` | 10 | Per-entry cost (delimiters + slack) | | `MIN_VALUE_BUDGET` | 10 | Minimum budget to render any leaf | | `MIN_MAP_VALUE_BUDGET` | 20 | Minimum per-value in map entries | | `MIN_ITEM_BUDGET` | 10 | Minimum per-item in sequences | | `SIZE_ANNOTATION_THRESHOLD` | 1024 | Size shown only if ≥1KB | ## API CellExplorer uses an instance-based API: ```java CellExplorer explorer = new CellExplorer(2048); // 2KB budget AString result = explorer.explore(cell); ``` | Parameter | Type | Default | Description | |-----------|------|---------|-------------| | `budget` | int | *required* | Maximum output size in CAD3 storage bytes | | `compact` | boolean | `true` | Compact output (reserved for future pretty-print) | Instances are immutable and reusable across threads. ### Path Navigation CellExplorer does not perform path navigation. Callers use the lattice cursor infrastructure ([CAD035](../035_cursors/index.md)) to resolve a path to a cell, then pass the resolved cell to `explore()`. ## Performance | Property | Guarantee | |----------|-----------| | **Work** | O(output bytes) — proportional to rendered size, never input size | | **Budget check** | O(1) per cell — uses cached `Cells.storageSize()` | | **Memory** | O(output) — renders directly to `BlobBuilder`, no intermediate strings | | **Stack depth** | Bounded by `budget / ANNOTATION_RESERVE` (~budget/30) | A 1KB budget exploring a 1GB lattice value costs the same as exploring a 1KB value — work is always proportional to output, not input. ## Use Cases - **LLM agent context** — inject lattice state previews into agent system prompts at controlled token budgets - **`covia:inspect` operation** — the Covia venue's inspect operation uses CellExplorer for budget-controlled previews - **Debugging** — quick structural overview of complex lattice values - **API previews** — bounded responses for REST endpoints ## Reference Implementation The reference implementation is in the `convex-core` module. | Concept | Class | Package | |---------|-------|---------| | Explorer | `CellExplorer` | `convex.core.data.util` | | Storage size | `Cells.storageSize()` | `convex.core.data` | | JSON rendering | `JSON.appendJSON()` | `convex.core.util` | `CellExplorerTest` provides comprehensive coverage including leaf forms, key formatting, truncation semantics, and JSON5 round-trip verification. ## See Also - [CAD002: CVM Values](../002_values/index.md) — Value types rendered by CellExplorer - [CAD003: Encoding](../003_encoding/index.md) — CAD3 storage size computation - [CAD024: Lattice](../024_data_lattice/index.md) — Lattice structures explored by CellExplorer - [CAD035: Lattice Cursors](../035_cursors/index.md) — Path navigation before exploration - [CAD044: JSON](../044_json/index.md) — JSON encoding used for leaf rendering --- # CAD047: Etch Storage Format ## Overview Etch is the embedded, append-only, content-addressable store that persists Convex cells — the immutable Merkle-tree nodes that make up CVM state, lattice data and every other value in the system. It is described in its own source as "a stupid, fast database for immutable data you want carved in stone": once a value is written for a key it is never changed, which eliminates cache-invalidation concerns entirely. This CAD specifies the **on-disk file format** (Etch format version 1). Higher-level *store management* — the store abstraction, in-memory caching, garbage collection and pinning — is a separate concern and is not specified here. Keys are fixed 32-byte identifiers, normally the SHA3-256 [Value ID](../003_encoding/index.md) of the cell being stored. Values are the canonical cell [encodings](../003_encoding/index.md). Etch is indifferent to the meaning of keys, but assumes they are pseudo-random hashes so that data distributes evenly across the index. ## Motivation A decentralised system persists enormous quantities of immutable, content-addressed data. A store purpose-built for that workload can make strong simplifying assumptions that a general-purpose database cannot: - **Immutability removes invalidation** — because a key's value can never change, there is no cache-coherence problem and no need for update-in-place of stored content. - **Hash keys index cleanly** — pseudo-random 32-byte keys distribute uniformly, so a radix tree over the key bytes gives shallow, balanced lookups without rebalancing. - **Lazy loading** — cells reference their children by Value ID, so a reader can traverse and load only the parts of a structure it needs. - **Single file, memory-mapped** — the whole store is one file, mapped into memory for fast random access, with data appended as it is written. ## Design Principles - **Write-once data** — the content of a record (key, length and encoding) is fixed once written. - **Content addressing** — every key is a 32-byte identifier of the value it locates. - **Append-only growth** — new records and index blocks are appended; existing content is not relocated. - **Single writer, many readers** — one writer appends and updates the root; readers map the file concurrently. - **One mutable field** — a record's persistence-status label may be upgraded in place (see [Immutability](#immutability-and-in-place-label-updates)); nothing else changes. ## Specification All multi-byte integers are stored **big-endian**. Offsets in pointers are absolute byte positions within the file. ### File Layout An Etch file consists of a fixed-size **header**, followed by a region of **index blocks** and **data records** that grows as values are written. The first (root) index block begins immediately after the header. ### Header The header is **44 bytes**: | Offset | Size | Field | Description | |--------|------|-------|-------------| | 0 | 2 | Magic number | Always `0xE7C6` | | 2 | 2 | Version | Format version (currently `1`) | | 4 | 8 | Database length | Logical length in bytes of database content (the append position); the file MAY be slightly longer owing to the write margin | | 12 | 32 | Root hash | Value ID of the current root cell, or a zero marker when no root is set | The root index block starts at offset **44**. The **root hash** is the store's single durable entry point. After the cells for a new root have been appended, the writer updates the database length and root hash in the header; on open, a store recovers the last committed root by reading the header. This is the on-disk basis for the synchronous-commit durability guarantee described in [CAD036](../036_lattice_node/index.md). ### Keys and Content Addressing Every key is exactly **32 bytes (256 bits)**. In normal use a key is the SHA3-256 Value ID of the stored cell's encoding, so a value is located by its own content hash. Etch does not interpret keys beyond using their bytes to navigate the index, and relies on their pseudo-random distribution for balance. ### Index Blocks The store is indexed by a **radix tree** over the key bytes. Each index block is a contiguous array of 8-byte **slots** (pointers). The number of slots — the radix — depends on the level in the tree, so that the most significant bytes fan out widely near the root and narrowly deeper down: | Level | Slots | Key material used | |-------|-------|-------------------| | 0 | 65536 | Bytes 0–1 (16 bits) | | 1 | 256 | Byte 2 (8 bits) | | 2–59 | 16 | Successive hex nibbles of bytes 3–31 (level 2 = high nibble of byte 3, level 3 = low nibble of byte 3, and so on) | The root index block (level 0) sits at offset 44. To locate a key, a reader extracts the digit for the current level, reads that slot, and follows the pointer according to its type. The maximum depth is 60 levels, which exhausts all 32 key bytes. ### Slot and Pointer Encoding Each slot is a big-endian **64-bit** value. A slot value of `0` means **empty**. Otherwise the top two bits give the pointer **type** and the low 62 bits give the absolute file offset of the target: | Top 2 bits | Type | Target | |------------|------|--------| | `00` | Data pointer | A data record | | `01` | Index pointer | A child index block (next level) | | `10` | Chain start | A data record that begins a chained entry list | | `11` | Chain continuation | A data record continuing a chained entry list | ### Data Records A data record stores one cell: | Size | Field | Description | |------|-------|-------------| | 32 | Key | The cell's Value ID | | 1 | Flags | Reference flags, including the persistence **status** | | 8 | Memory size | The cell's memory footprint ([CAD006](../006_memory/index.md)); `0` until the cell reaches PERSISTED status | | 2 | Length | Length `N` of the encoding, as a big-endian short (always non-zero) | | N | Encoding | The canonical [CAD003](../003_encoding/index.md) encoding of the cell | The 1-byte flags field together with the 8-byte memory size form the record's mutable **label**; the key, length and encoding are fixed. ### Collision Handling and Chaining When two distinct keys resolve to the same slot at some level, Etch first tries to store the newcomer in an **adjacent empty slot**, forming a short **chain**: the original slot is marked as a chain start (`10`) and each continuation is marked as a chain continuation (`11`). A lookup that lands on a chain start scans forward through the continuation slots, comparing keys. When a chain cannot be extended — because the neighbouring slots are occupied by unrelated data — it is **collapsed** into a new child index block at the next level: the colliding records are re-inserted one level deeper, and the slot becomes an index pointer (`01`). This keeps index-block allocation lazy (deeper blocks are created only when genuinely needed) while bounding the length of any chain scan. ### Immutability and In-Place Label Updates Once a record is written, its **key, length and encoding never change** — this is what makes the store content-addressable and safe for concurrent, lock-free reads. The single exception is the **label**. A cell's persistence status may be upgraded in place — for example from STORED to PERSISTED — by rewriting the 1-byte flags field via an idempotent, monotonic merge, and the 8-byte memory size is filled in the first time a cell reaches PERSISTED. Because these updates are monotonic and never touch the encoding, a reader always observes a consistent record regardless of concurrent status upgrades. ### Memory-Mapped Regions For access, the file is memory-mapped in **regions of up to 1 GiB** (with a small margin to accommodate writes near a region boundary), reflecting platform limits on individual mappings. This is a runtime access strategy only: it does not affect the logical byte layout, and all pointer offsets are absolute within the single file. ## Reference Implementation The reference implementation lives in the Convex `convex-core` module (Java), package `convex.etch`. | Concept | Class | |---------|-------| | File format, index navigation, read/write | `Etch` | | Store binding and durable root data | `EtchStore` | | Integrity-failure signal | `EtchCorruptionError` | | Index traversal callback | `IEtchIndexVisitor` | | Format utilities | `EtchUtils` | The store abstraction that wraps Etch (caching layers, garbage collection and pinning) is a separate concern, intended for its own CAD. ## See Also - [CAD002: CVM Values](../002_values/index.md) — Cells and references stored by Etch - [CAD003: Encoding Format](../003_encoding/index.md) — Value IDs and the cell encodings stored in data records - [CAD006: Memory Accounting](../006_memory/index.md) — The memory size recorded in each data record - [CAD024: Data Lattice](../024_data_lattice/index.md) — Lattice data persisted through Etch - [CAD036: Lattice Node](../036_lattice_node/index.md) — Durable-root commit built on the Etch root pointer