Urbit’s whitepaper is quite complex and challenging, especially for people without technical knowledge.
Therefore, we will try to explain it in a simple and uncomplicated way, step-by-step, respecting the order in which the whitepaper was built.
As there are many terms that will be new to you, the beginning of reading may seem difficult, but I guarantee that after reading two or three paragraphs things will start to become clearer, as several terms are repeated and used again in different contexts. In other words, we will explain the same concept in different ways throughout the text. So, if any point has been unclear, it will probably be clarified throughout the text.
Are you ready? So let’s start!
Sections
Abstract
The Whitepaper is entitled “Urbit: A Solid-State Interpreter” and this term is repeated in the Abstract:
A “solid-state interpreter” (SSI) is an interpreter which is also an ACID database. An “operating function” (OF) is a general purpose OS defined as a pure function of its input stream. Urbit is an SSI defined as an OF. Functionally, Urbit is a full execution stack. VM, compiler, OS, network, web server, and core apps are 30kloc; the system is alpha-grade with a live, fairly stable network.
Let’s break down step-by-step what this all means:
“Solid-state interpreter” (SSI): In Urbit, a SSI is like a mix between a program that can understand and run code (an interpreter) and a very secure type of database (an ACID database) that can keep data safe and correct even when things go wrong. ACID stands for Atomicity, Consistency, Isolation, and Durability, key features for maintaining data integrity. The term “solid-state” is a metaphor here, saying the whole system works in a predictable, trustworthy way, much like electronic devices with no moving parts. Atomicity, one part of ACID, means that in a series of actions, everything must go right or the whole transaction is cancelled. This keeps the database in a consistent state even if something goes wrong during the transaction.
“Operating function” (OF): This means that the entire behavior of the system can be predicted from its inputs. The system does not have any hidden state or side-effects: it behaves in a deterministic way.
“Urbit is an SSI defined as an OF“: This sentence combines the above concepts. Urbit is a system that interprets and executes code, maintaining the robustness and determinism of an ACID database, and behaves as a deterministic function of its inputs.
“Urbit is a full execution stack“: This means that Urbit includes all the layers you need to go from raw code to running software. This typically includes a virtual machine (to provide an isolated environment for code to run in), a compiler (to translate high-level code into machine instructions), an operating system (to manage resources), a network (for communication), and a web server (to host and deliver web content).
“VM, compiler, OS, network, web server, and core apps are 30kloc“: This states that the entire system, including the virtual machine, compiler, operating system, network, web server, and core applications, is implemented in 30,000 lines of code (“kloc” stands for “thousands of lines of code”). This is surprisingly small for such a complete system, which is noteworthy.
“The system is alpha-grade with a live, fairly stable network“: The term “alpha-grade” refers to an early stage of development where software might be feature-complete but still contains bugs or lacks optimization. This is generally not stable enough for general use, but the authors also mention that the network is “live” and “fairly stable”, suggesting that it is running and operational, albeit potentially with some issues still being worked out. It is important to consider that the whitepaper was written in 2016 and this sentence refers to the stage of the project at that time.
Uniting all the concepts we have just learned, we can summarize the Abstract as: Urbit is a highly deterministic, compact system that combines the robustness of an ACID database with the predictability of a pure function, and includes all the tools you need to go from code to execution.
Don’t worry if some points are still unclear, as we’ll be revisiting several of these concepts throughout the text.
Introduction to Urbit
After this brief summary, the whitepaper introduces “what Urbit is” using 6 different approaches:
Functionally:
This refers to what Urbit does and the components it’s made up of. Here are the main parts:
Operating System (Arvo): This is like the boss of the computer. It manages the computer’s memory, processes, and all of its software and hardware. It also allows you to communicate with the computer without knowing how to speak the computer’s language.
Functional Language (Hoon): This is a programming language that Urbit uses. In functional languages, the output value of a function depends only on the arguments that are input to the function. This makes it different from languages that depend on a local or global state, which may change over time.
Virtual Machine (Nock): This is a program that acts like a computer that you can run other programs on. It’s “virtual” because it operates only in software running on an actual physical computer.
Packet Network (Ames): This is the system that Urbit uses to send data across the network. Data is sent in small packets that are reassembled at their destination.
Version-Control System (Clay): This is like a time machine. It keeps track of all the changes that have been made to a system’s source code. If you make a mistake, you can turn back time and compare earlier versions of the code to help fix the mistake while minimizing disruption to all team members.
Formally:
This refers to how Urbit is defined in a mathematical or logical sense. Urbit is described as an “operating function”. In simple terms, you can think of a function as a machine that takes in certain things (input), does something to them, and then gives back something else (output). An “operating function” for Urbit means that the entire life of the Urbit system can be described as a function that takes in an input stream (a sequence of data) and produces an output. This function is an interpreter, which means it translates high-level instructions into a format that can be executed or run. The interpreter starts with an initial language, which then starts up an initial kernel (the core of the computer system). The interesting thing about Urbit is that everything except the lifecycle function can upgrade itself from source code in the input stream. This means that most parts of Urbit can improve or update themselves over time.
Securely:
Urbit aims to be simpler and more rigorous than current systems, learning from past mistakes in software design. It aims to avoid the complexity and security issues associated with current computing semantics.
Structurally:
Urbit runs on a normal OS (like Linux or macOS) server and interacts with the user, other Urbit nodes, and the Internet.
Productively:
Urbit is designed to work as a “personal server”, replacing multiple web services with self-hosted applications on one personal server. It’s designed to be simpler to use than the current web model. In the current model, users make requests to centralized servers. In Urbit, every user is a server.
Practically:
As of the time of writing, Urbit was in a semi-closed alpha test. This is no longer the case.
Section 2: Highlights and novelties
The section 2 of the whitepaper provides an overview of the various components and features of the Urbit platform. It dives into the inner workings of Urbit’s architecture, specifically discussing four main components: Nock, Hoon, Arvo, and Ames. So let’s break each one of these down into simpler terms:
Nock (combinator interpreter): Nock is the foundational level of Urbit, working as its virtual machine. Think of it as the base language that everything in Urbit is built on. It’s designed to be super simple and unchanging. A “Nock value” or “noun” is either a number of any size or a pair of these “nouns”. The key concept here is that Nock is like a bare-bones computing system that only knows how to do the simplest things, like counting up. Read this article for more details.
Hoon (typed functional language): Hoon is a programming language that is built on top of Nock. It’s like the translator that allows more complex programming to be translated down into the simple operations that Nock understands. Hoon is designed to be more concrete and less abstract than many programming languages, making it more straightforward to learn and use. It also handles things like validating data received from an untrusted network, ensuring it fits the expected format.
Arvo (nonpreemptive kernel): Arvo is the operating system of Urbit, like Windows or macOS for a PC. It’s the part of Urbit that manages everything else, including the record of past events (akin to transaction logs in traditional computing). It can update any part of the system in real-time and saves changes to a ‘snapshot’ or a record of changes. Arvo is also capable of managing ‘events’ to prevent complexity from escalating uncontrollably. One of its features is Clay, a data storage system that’s like a typed version of Git, a popular version control system.
Ames (P2P packet protocol): Ames is the networking layer of Urbit. It provides a way for different parts of Urbit to communicate securely with each other, much like how the Internet allows computers to exchange information. It assigns a unique identity to each entity and enables encrypted communication, ensuring security and privacy.
Section 3: The solid-state interpreter (SSI)
We are already on page 8 of the whitepaper. Here, we will start to detail some concepts that we superficially learned at the beginning of the whitepaper, like “solid-state interpreter”.
In essence, an SSI is a new design for system software that aims to be more reliable and predictable, while still being able to handle a broad range of computations.
The SSI is considered a “stateful packet transceiver”. A packet transceiver is a device or component that receives and sends packets of information. In this context, the SSI acts as a virtual “chip” that performs the functions of receiving and sending packets within the Urbit network.
The SSI’s state refers to its internal configuration or condition at a given point in time. As the SSI receives and sends packets, it processes and executes instructions, which can cause its internal state to change. These state changes reflect the SSI’s progression through different tasks or operations within the Urbit system.
Uniform Persistence: This section talks about the storage system of an SSI. Unlike traditional systems that differentiate between “memory” (fast but volatile) and “disk” (slow but stable), an SSI’s storage is “uniformly persistent,” meaning it’s all one stable data structure. The paper likens this to a hardware implementation of uniform persistence that could use technologies like NVDIMMs, but mentions that Urbit currently uses a log-snapshot mechanism like a standard database on typical hardware. The system never “reboots” in the traditional sense, meaning it doesn’t unpredictably erase data.
Source-Independent Packet Networking: An SSI only interacts with the outside world through “unreliable, insecure packet networking.” It receives packets (units of data sent over the internet) and processes them, but it doesn’t care about where they came from (their “source”) — the only thing that matters is the content of the packet. This section makes an important distinction between a bus, which transports commands, and a network, which transports packets. Unlike commands, packets are like “facts” that can be dropped or repeated without changing the information they convey.
High-Level Determinism: This means that the SSI always behaves in a predictable way based on its current state and the inputs it receives. It’s deterministic at the “semantic level,” meaning that the entire operation is more like a function call, as opposed to being based on individual CPU cycles and memory images. It ensures that interrupting the SSI doesn’t produce an unpredictable state.
Decidability, Determinism and Interruption: The paper talks about a balance between Turing completeness (the ability to solve any computation problem) and deterministic/repeatable behavior. While an SSI is Turing complete, it can still be interrupted safely. The text also mentions some techniques for managing long-running computations and dealing with infinite loops.
In summary, an SSI can be thought of as a “chip” that receives and sends packets of data, sometimes changing its state as a result. It never loses data and doesn’t conceptually “reboot”. It operates as an operating system and logs its event stream (chronological sequence of events or actions performed by the SSI) to a cluster for low-latency determinism, periodically saving its state to a snapshot image.
Logging refers to the process of recording or capturing events or data for future reference or analysis.
A cluster refers to a group of interconnected computers or servers working together to perform a specific task. In this case, the SSI logs its event stream to a cluster, indicating that it sends the recorded event data to a collection of servers or computing resources.
Low-latency refers to minimal delays or short response times. Determinism, in this context, refers to the property of being predictable or having a cause-and-effect relationship. By logging its event stream to a cluster, the SSI aims to achieve low-latency determinism. This means that the recorded events can be quickly and reliably accessed and analyzed, enabling predictable and deterministic behavior within the Urbit system.
Section 4: The operating function (OP)
An operating function is a single, unchanging function that dictates how a computer operates. It’s like a set of rules or instructions that the computer follows to process information. In the context of Urbit, this function is a “lifecycle function” that uses a history of inputs to determine the current state of the system.
Lifecycle function:
The lifecycle function in Urbit, denoted as L, is essentially a mapping from the history of inputs received by the system to the current state of the system. In practice, this means that the state of a node in the Urbit network at any given time is determined entirely by the inputs it has received in the past.
In the context of a stateful system (a system that keeps track of its state), a transition function (T) would typically be used. A transition function is a rule that takes the current state and an input, and produces a new state and output. However, in Urbit, output is not part of the fundamental definition of the system, as it’s merely a suggestion that a packet might be sent, not a guarantee. Also, because the rules of the system are frozen (they don’t change), Urbit uses a lifecycle function (L) that uses the entire history of inputs to determine the current state.
Larval stage:
When a new node is created in the Urbit network, it goes through a “larval stage” where it’s fed a series of “larval” packets that prepare it for life in the network. This stage allows the system to establish the node’s identity and to upload any necessary code to it. While this stage is essential, it’s not considered cheating because the lifecycle function (L) remains the only non-updatable part of the system. Once the node’s identity is established, it can’t be updated. If a new identity is needed, a new node must be created.
Step model:
In Urbit, the system doesn’t only process packets (pieces of data sent over the network), it also processes commands from the host operating system (OS). These pieces of input are referred to as “steps”. While packets can be ignored, commands must be processed. If a step can’t be processed or is interrupted, it can be replaced with an error command, which allows the system to track errors and handle them appropriately.
Symmetry breaking and secrets:
In order to differentiate different nodes (ships) in the network, the system introduces entropy or secrets. These are unique to each node and break the symmetry.
“Symmetry breaking” and “Secrets” have to do with establishing the unique identity of a node (referred to as a “ship”) in the network.
“Symmetry breaking” refers to the process of differentiating each node from all others. In a perfectly symmetrical network, all nodes would be identical and indistinguishable. But for a network to function properly, each node needs a unique identity—there needs to be a way to tell them apart. This is where “symmetry breaking” comes in: it’s the process of assigning a unique identity to each node.
The “entropy” or “secrets” mentioned are related to this process. In information theory and computer science, “entropy” often refers to the degree of randomness or uncertainty in a set of data. When the whitepaper talks about using “entropy” to break symmetry, it’s referring to the use of random data to generate a unique identity for each node. This could involve, for example, generating a random number or string that serves as the node’s identifier.
These “secrets” are unique pieces of information used to establish a node’s identity. The text suggests that these secrets need to be delivered as packets to the node, but not over the network (since they wouldn’t be secrets anymore if they were publicly transmitted). This is done during the “larval stage,” a phase where the node is set up and prepared for participating in the network.
The secrets play a vital role because they help to ensure that each node has a unique identity. This is crucial for the proper functioning of the network: you need to be able to address specific nodes, authenticate their identities, and ensure secure communication between nodes. The secrets, therefore, are the foundation of a node’s identity and security within the network.
These secrets exist because of the need for security and unique identification in a network environment. By assigning a unique secret to each node, Urbit can ensure that nodes can authenticate each other and communicate securely. Once established, this identity cannot be updated or changed, preserving the integrity of the node within the network.
Interactions with the host OS:
Despite its isolation from the host OS, the interpreter (the component of the system that processes inputs and produces outputs) can process commands from the OS. In response, it can suggest actions. This interaction does not entangle the semantic definitions (meanings) of the interpreter and the OS.
The Urbit model is complex, as it involves a novel approach to designing and operating a peer-to-peer network. It aims to minimize the complexity of its core function (the lifecycle function) and instead delivers more advanced functionality (like a high-level programming language or an operating system) through packets over the network. This design ensures that each node is isolated and its state can only be altered by its owner, enhancing the security and privacy of the network.
Section 5: The Urbit lifecycle function
Continuing on the subject of operating function, the whitepaper introduces two important concepts in Urbit: Nouns and Nock.
Here’s a simplified analogy: imagine Urbit as a game, where nouns are the game pieces, and Nock is the rulebook that determines how you can move the pieces.
In the context of the lifecycle function, nouns are the states and inputs, and Nock provides the rules for transitioning from one state to another based on an input.
The larval stage is the initial stage of the lifecycle function in which the system sets up the basic semantics and identity of a node. This is done by feeding the node a sequence of specially crafted inputs (the “larval” packets), which are processed according to the rules defined by Nock to produce the initial state of the node.
The transition function T embedded within the lifecycle function L is the specific rule set that describes how to update the state based on a given input. The transition function is not explicitly defined here, but it’s understood to be part of the definition of Nock.
To summarize: the lifecycle function L in Urbit uses the history of inputs received by a node to determine its current state. This is done using Nock, which provides the fundamental computation rules of the system, and the concept of nouns, which are the basic units of data in Urbit.
Let’s now get a little more specific about the definition of each of these terms.
Nouns:
In Urbit, the term “noun” is used to refer to the most basic unit of data. There are two types of nouns: atoms and cells.
An atom is an unsigned integer, meaning it can represent any non-negative whole number.
A cell is a pair of nouns. It’s akin to a small data structure that can hold two pieces of data. These pieces can be any two nouns, so they could be two atoms, or two cells, or an atom and a cell.
Nouns do not carry any type information with them because Urbit expects a static type system at a higher level. A statically typed language is one where the type of a variable (i.e., the kind of data it can hold) is checked at compile time, before the program is run. This means that the type of a variable is fixed once it’s declared and can’t be changed. Common statically typed languages include Java, C, and C++. In contrast, dynamically typed languages, such as Python or JavaScript, allow you to change the type of a variable even after it’s been defined.
So an atom is just a number, and a cell is just a structure containing two nouns. The system doesn’t inherently know what these numbers or structures represent (for example, whether a certain atom represents a user ID, a timestamp, or something else entirely).
This is where the static typing system comes into play. In the higher-level layers of the Urbit system, these nouns are given meaning through static typing. For example, in the Urbit high-level language, Hoon, you might declare that a certain atom is to be used as a timestamp, and from that point on, the system knows to treat that atom as a timestamp.
To summarize, Urbit’s basic units of data (nouns) are simple and do not carry inherent meaning. They can represent any data but need a static typing system to give them context and meaning. The static typing is expected to be done at a higher level in the Urbit system, in the Hoon programming language.
Nock:
Nock is essentially the “machine language” of Urbit. It’s a simple interpreter defined by a set of reduction rules. These rules describe how to evaluate or “reduce” expressions in Nock. The purpose of Nock is to define the fundamental computation rules of the system.
The nock syntax looks like this: [ a 7 [[7 [0 1] b ] 0 1] c ]
Finishing the Whitepaper
Sections 6 and 7 talk about details of the nock and hoon languages, such as operators, functions, and statements. In short, nock is the machine language and hoon is the programming language used in practice by developers who create applications on Urbit.
Likewise, sections 8, 9 and 10 provide technical details about the Arvo kernel and one of its vanes: Ames. We recommend reading these articles: “What is Arvo” and “What is Ames” for a simplified understanding of these elements.
Let’s end this article with a direct quote from the end of the whitepaper, which directly compares Urbit with other projects::
“Many historical OSes and interpreters have approached the SSI ideal, but fail on persistence, determinism, or both. In the OS department, the classic single-level store is the IBM AS/400. NewtonOS was a shipping product with language-level persistence. Many image-oriented interpreters (e.g., Lisps and Smalltalks) are also SSI-ish, but usually not transactional or deterministic. And of course, many databases are transactional and deterministic, but their lifecycle function is not a general-purpose interpreter.”
This makes it clear that Urbit is a unique project.
Today, almost 10 years after the release of the whitepaper, Urbit is already a functional platform with hundreds of applications and thousands of users. The protocol continues to improve and gain adoption. We hope that this article has been useful in unveiling a little more about this fascinating technology. You can delve into more details about each component of the Urbit network in this free course.