So what exactly have you been working on lately?
As many of you may have realized, I have been working on something called Gemer for the past few years, and made the first few commits of its proof of concept on a public GitHub repository a few months ago. Yes this is currently a part-time project, and that's why I couldn't really update it as frequently as I first hoped.
Since the GitHub repo and website went up, people kept on asking me what's up with the project, since there weren't any new updates lately. Some also asked me why I don't put up an easier introduction on my blog for non-techies to understand. Well to be honest, I was coping with other things in life, but also have been finalizing some internal specs to integrate all of this into actual applications and businesses.
Working on the concepts for Gemer has been an incredibly challenging and adventurous task for me all this time. The reason why it was "adventurous": nothing like this have existed before, and thus I have nothing to reference or to compare against while working on this thing. This is not even a distributed ledger of any kind, which means no native cryptocurrencies. But it does share a lot of features from a blockchain, and takes advantage of existing blockchains, like Ethereum, for certain tasks and purposes. And yes, decentralization all the way, but I also wanted to provide a stable, proven development environment for developers to work on actual products that have real use cases and a solid consumer base.
So here it is, as requested: a brief, not-so-technical introduction to one of the most adventurous tech journeys in my life.
A background on blockchains and decentralized applications
I mentioned this multiple times on my blog and through Facebook: the blockchain isn't working as many enthusiasts have originally hoped for. The blockchain industry is slowly being accepted as a serious mainstream business nowadays, including Samsung's native support for public blockchains, Facebook's Libra and China's digital currency. But it is still missing one key component in order to be truely accepted into the mainstream: a significant userbase that are not enthusiasts or cryptocurrency traders. Blockchain people often call that missing component "a killer DApp".
People have had numerous theories on why a "killer DApp" is not emerging, especially at a point when it has been more than 5 years since Ethereum became a thing. Some suspected it is the performance constraints and bottlenecks of a blockchain; others blame the poor user experience. A significant portion also blame other factors, including regulatatory issues and even a "lack of creativity". But I have another theory: that blockchains - especially public blockchains - are not really suitable as an application layer that directly interacts with the end user.
Most platform blockchains capable of running Smart Contracts rely on a very clever solution. Bitcoin and other "1st gen" blockchains defined coins as a set of unspent transactions, or "unspent transaction outputs" (UTXOs). While I still believe this is still the best solution for verifying payments (as all coins can be easily verified as a chain of cryptographic proofs), it fell short on one key area: programmability.
The concept of automatically executing contracts written in code, or Smart Contracts, have already been proposed back in the 90s by Nick Szabo. But Bitcoin and other cryptocurrencies before Ethereum mostly focused more on the "money" part than the "program" in "programmable money". While attempts like Namecoin did generalize the concept of this a bit further by assigning features other than payments on a blockchain, "blockchain" and "cryptocurrency" were essentially the same concept before Vitalk propsed Ethereum.
Ethereum generalizes a blockchain to run a Smart Contract, which is code running on a blockchain's runtime environment (such as the EVM). This enabled general purpose applications, and not just money, to live on the blockchain as well. The way Vitalik implemented this concept was very simple, but powerful: on Ethereum, each state transition for a Smart Contract is also defined as a transaction, combined with Ethereum's new "account-based" transaction model. This is both a huge advantage and a drawback, especially from a traditional CS standpoint.
What exactly is a state?
Before I proceed, let me briefly explain what a "state" exactly is. Simply defined, if a given system is designed to be aware of and/or store any previous interactions or events with it, the system is defined to be stateful. This information stored by the system is known as a state. When a program runs on a computer, it writes information and data, such as variables, on memory (whether it is temporary, such as RAM, or permanent, such as a hard disk). The contents of this memory space used by the program is normally defined as a program's state.
For a better comparison, imagine two dolls - one a wind-up toy, and another powered by electric motors and a small microcomputer. A wind-up doll will always behave the same way regardless of how the person interacted with it previously; it remembers nothing. A doll powered by a microcomputer, on the other hand, could be programmed to remember the recordings of a person's voice and play it back afterwards, for example. This greatly changes the way the doll operates and interacts with people, as it can be programmed to dynamically change its behavior based on the doll's previous interactions.
The second doll would be stateful, while the first one (without a concept of state) is often known to be stateless. This concept of state can also be expanded to protocols as well. When a protocol is designed to change its behavior based on the previous stream of data that is processed, it is a stateful protocol - and "state" here is defined as all previous streams of data the protocol has interacted with, impacting its current behavior. Stateless protocols do not get affected by any of the data that it have previously processed, and do not remember them in any way.
For instance, HTTP, the protocol that powers the World Wide Web, is a completely stateless protocol. Like a wind-up doll, every data stream transmitted over HTTP is treated independently; they do not affect each other at all. Because of this, every time a web browser files a request to a server via HTTP, it has to send over the same request traffic and receive a response. Each request is completely independent, and do not get stored; but other techniques, such as cookies, are used on top of HTTP within web browsers to create the concept of state (saved logins and shopping carts, for instance).
FTP, on the other hand, is a stateful protocol. This is because the user first has to authenticate via its credentials in order to gain access to a remote server, and the server keeps track of that data during the session in order to ensure the user's credentials.
State processing on the blockchain
Assuming that a blockchain is also a "protocol", platform blockchains capable of running Smart Contracts, such as Ethereum, are always "stateful" protocols. But the way blockchain platforms define and treat state are very different from the concept of state for applications running on your own computer, mobile device or a remote server.
Traditional computer science views state as a co-requisite of an executing program. This means that state can be modified in a number of ways, and program execution can be completely independent from state data. For example, there can be multiple applications or threads accessing the same data (state) simultaneously, processing them on a processor, and writing the results back to its state (assuming that issues with multithreading, such as deadlocks, are properly resolved). Execution flow control can be optimized based on the level of available system resources. Intense execution optimization and improvements can take place. This is the standard architecture of executing programs on a modern computer, and is the reason why high-performing systems and applications are possible today.
There is another approach to this execution-based approach, however. State-machine based approaches view program execution as a set of modifications to the program state. This method of system interpretation allows a machine model to be greatly simplified, but comes at the cost of limiting the views of the execution process itself. State modification within this model is called a state transition, and program execution is considered equal to a state transition.
Unlike every other machine, Ethereum (and every other platform blockchain) is a purely state-oriented platform. What this means is that on Ethereum, everything works the other way around. Unlike standard computer programs, where state is the co-requisite of a program execution, Ethereum defines program execution itself as a state transition, and a state transition is defined as a transaction on the Ethereum blockchain.
This is the reason why Ethereum's initial introduction was "the world computer". Every state transition generated by the execution of a contract is considered to be a modification to a single machine state model: the world state. The machine running on this world state (by definition of a state-oriented machine) is the EVM, and because of this the EVM always assumes a single-threaded approach. There is no threading or any other parallelization technique. System resource optimization is done solely through economic incentives - gas. There is also no execution flow control available to the application; execution ordering is solely determined based on the order of transactions (state transitions) being stored within a block. When two different transactions try to access the same state, Ethereum either rejects the transaction or assumes one is a priority of the other (defined by block and transaction ordering mechanisms with Proof-of-Work). This makes the EVM much more deterministic and thus more reliable than traditional VMs, but also makes it much less flexible.
That is the fundamental reason why the blockchain cannot be a "decentralized" drop-in replacement of the current web. Ethereum and platform blockchains are simply a very different system from what developers are used to deal with. They are also not effective by design for intensive processing tasks that are required for modern applications, as blockchains follow a state-oriented execution model.
Because of this architectural difference, simply improving "scalability" - or transaction processing/finalization performance - of a decentralized ledger will not help with mass adoption. It's common sense, really. Running 1+1 on your local computer or on a remote server is a very simple and quick task - it is just a matter of execution flow. On a blockchain with a specific transaction ordering system*, though, even running simple operations like those need to go through consensus. Even if consensus is lightning fast, it will simply be much slower compared to just running an operation locally, as it does not require the extra overhead. Blockchains can never be as fast and effective as a server in those kinds of operations.
(*More commonly referred to as consensus. Under Proof-of-Work, transaction ordering is never "finalized"; transactions just get less likely to be reverted as more blocks accumulate on the blockchain. Under Proof-of-Stake systems, though, there is the concept of finalization and irreversability - which greatly reduces uncertainty and improves finalization speed compared to PoW systems.)
Does that mean everyone is wrong, and blockchains are a fundamentally inferior system?
Not quite. Blockchains can do amazing things when paired with the right job. The problem here is that people are trying to squeeze tasks into blockchains that they were simply not designed to deal with effectively. There are tasks that only blockchains can perform, and people are just not getting that yet. Getting blockchains to replace the entire web and calling it "not effective" is like evaluating a fish based on its ability to climb a tree and calling it inferior. They are just... different, like a monkey and a fish. Attempting to make a fish climb a tree well by adding arms simply is not the right solution. That fish will never beat a monkey on climbing trees, and nor will it be good at swimming. That's how I view current blockchain platforms attempting to replace the entire web.
So what is the right job for a blockchain? What can a fish do well?
There are obvious examples out there that only a blockchain can do well. Publicly verifying signatures. Contract automation. Transmission of value. Effective payment clearance. DeFi. Uniswap. Stablecoins like MakerDAO. Building DAOs to improve on the situation of current remote organizations, such as Aragon.
But also obviously, those are very specific tasks that cannot be generalized and/or applied for all applications out there. This means that blockchains still cannot effectively replace the entire backend, except for particular applications.
This is why blockchains are becoming less of a "Web 3.0" on its own, and slowly turning into a modular component within the traditional web stack. Let's take a look at some of Ethereum's recent use cases within the mainstream web infrastructure industry.
- Cloudflare's Ethereum Gateway. Cloudflare is one of the world's top CDN and DNS routing infrastructure providers. In fact, this website runs on Cloudflare's DNS and CDN caching services on top of a normal VPS. Back in June, Cloudflare announced that they will be providing a gateway service to Ethereum using their global infrastructure network, which provides developers a much simpler way to access the Ethereum blockchain. Cloudflare's Ethereum gateway, cloudflare-eth.com, accepts standard Ethereum JSON RPC calls and relays requests through its network. What's cool is that this gateway uses the same worker techniques used with their lightning fast CDN, and the service itself is a SSL for SaaS provider - which means that custom domain support for their gateway is also available. Alongside with their IPFS gateway service, devs can build an entire website out of Ethereum and IPFS while taking advantage of Cloudflare's CDN services. Pretty sweet.
- Microsoft Azure. This may be surprising to some people, but Microsoft has been a consistant supporter of Ethereum and its ecosystem starting back in 2015. They even gave multiple talks at Devcon and was a proud long-time sponsor. Microsoft first announced their Ethereum Blockchain as a Service on Azure alongside with ConsenSys in 2015, and gradually introduced partial support for the public Ethereum chain as well. They also now provide Azure Blockchain Workbench, which basically maps blockchain identities with existing enterprise entities on Active Directory and allows Smart Contracts to run based on that (although this is mostly for enterprise private chains, it is still legitimately an awesome feature). Native Solidity & Smart Contract deployment support on Visual Studio Code (which they also demonstrated at this year's Devcon5) and support to the ecosystem overall, such as the Enclave Ready EVM (eEVM) written in C++, also should be mentioned.
- Samsung. Honestly Samsung was the only tech giant that was very serious about supporting the public chain ecosystem, among all those corps that I have met. Samsung Blockchain Keystore, preloaded on all Galaxy phones starting from the Galaxy S10, is powered by Knox - which is by far the best widely-available mobile security solution on Android. Knox is the same component used to secure Samsung Pay. Some Samsung Exynos processors (starting from the Exynos 9820) are also equipped with a special circuit for running hardware wallet-like signing functionallity with a Knox container, known as a PUF. But it is pretty clear that Samsung is using this wallet feature to gain control over their own ecosystem - Samsung controls which DApp and cryptocurrency is preloaded on their own wallet app.
So the direction of how the mainstream tech market is going seems pretty simple: use blockchain as a pipeline or a component within existing cloud and web infrastructure configurations, especially for Smart Contracts, signatures, and DIDs. Nothing else will change. The user experience, the existing web stack, everything that we already know and used to - will firmly remain in place. Blockchain will just become one component powering services with traditional architecures, enabling new things like public signatures, contract automation and identification - and will most likely become a component provided by cloud services, like AWS or Azure. Another application would be central bank digital currencies (CBDCs) or stablecoins used to process payments and/or value exchanges on such existing services, assuming that MasterCard and Visa will start to show their limitations fairly soon.
The reason for this: there is no reason to switch from proven infrastructures that already provide high performance. A monkey should climb a tree, and a fish should swim in water. It is just way too inefficient to ask a fish to climb a tree instead.
But I know some people will still complain: "but... decentralization! privacy! self-soverignty! freedom of speech! censorship resistance! I don't want big tech gobbling up my data and controlling my daily digital lives!"
I'm glad you asked - because that is exactly what I am trying to achieve.
Gemer - not a blockchain, nor a distributed ledger
Remember what I said at the beginning about this being something completely different? I meant it - Gemer is a better, decentralized monkey. It is not a fish trying to climb a tree.
So what makes Gemer a monkey, and not a fish?
State processing on a Gemer container
As mentioned earlier, state processing on a blockchain occurs in a different way compared to a standard computer system. Execution is defined as a state transition, and state transitions are processed as a transaction on the blockchain. While this approach does have a significant hit on performance, it has an important advantage - public verifiability. This is what makes the fish a fish.
Because every execution is treated as a state transition, every time a Smart Contract is called and modifies the world state, the call - or execution - is recorded permanently and publicly on the blockchain. This is what makes the blockchain so great at processing tasks like token transfers and votes. Extreme transparency on execution gives unique capabilities to the blockchain, but makes it simply unsuitable for applications that do not need that extra execution verifiability and transparency.
State processing with Gemer, on the other hand, uses a hybrid approach between the stateful and stateless model of programming. Much like the traditional execution-oriented model, state is managed independently from program execution; this means that developers have nearly full access to the performance of raw hardware.
But there is a catch: internally, the code for Gemer applications should be treated as purely functional. Let me quickly explain what this is, and the reason for this decision.
Gemer's State Management & Execution Cycle
As mentioned with the paper, a Gemer application is consisted of three main components: the code graph (or the bare codex), the state chain, and the keychain. The entire data structure consisted of all three components is referred to as the codex.
This is because state information needs to be completely separated from its corresponding codebase, in order to ensure full portability and abstraction across different worker groups, or machines consisting the network.
On a traditional computer system, there is the assumption that state will always be available to a running application. It's pretty obvious; if a programmer wants to access a file, for example, the program just needs to create a FileStream object and perform disk IO. But on a distributed system like this one, it is impossible to assume as such, as code needs to run on multiple machines and return the same result. What if a file that exists on this computer does not exist on that one? The program will break.
Blockchains solved this problem by defining a linear ledger, and defining that as a single machine with "world state", which every program running on it should only refer to. However, as mentioned earlier, this created performance constraints for general-purpose applications other than specific use cases that require such verifiability. This means that a single state model, like those of a blockchain, cannot apply anymore for solutions targeting markets that require high performance.
So what are we going to do?
Purely functional programming is a form of the functional programming paradigm. With functional programming, all computation is treated as evaluating a mathematical function. A function only has an input and an output; no other assumptions can be made about its evaluation while running it as a program.
Because of this, functional programs are usually not allowed to modify its state directly. All assumptions should be passed through as function arguments, and when the passed arguments and the function match, it is always expected to return the same values as well - because there cannot be other assumptions other than its arguments. (If we know that \(f(x)=g(x)\) and \(a=b\), it is trivial that \(f(a)=g(b)\); same thing.)
With programming languages that enforce the pure functional programming paradigm, including Haskell, external state is usually expected to be controlled through a special type of monad, such as
Because of this property, functional programming is incredibly useful for parallelization and distributed systems, as it eliminates state assumptions by design. This means that once we agree on the actual function code and its assumptions (parameters), we can easily reach an agreement on its results by simply computing them.
With Gemer, the code graph and its hash are always fixed to ensure this (unless an agreed container upgrade occurs). But what about its state? How is the state chain used?
Take a look at this:
There are two worlds with Gemer's execution cycle: the stateless world and the stateful world. The stateless world is where no state access or storage is allowed, and everything, including the worker processes themselves, are destroyed immediately after computation is complete. The stateful world manages and stores all committed state, including passing parameters, the code graph and the state chain.
When an application is first called, it first calls the root function. It is then assigned a worker process - which is essentially just a randomly selected group of nodes runnint that particular function. Required information is passed through from the stateful world to the stateless world through a state wrapper, which allows access to prerequisite information using
State-type monads or any other
context type switcher.
When computation is complete, the updated state is passed through the wrapper and recorded as a block on the state chain.
This is then passed to the next function, following the callstack of this session. Another worker group is created, and evaluates the following function based on the information provided, as shown above. The previous worker process is destroyed (unless there is a midfunction call that resulted in the next worker process being invoked).
Multiple function calls done in parallel within the callstack is also evaluated in parallel, with multiple workers.
When execution is complete, all functions existing on the callstack now has their state chains updated, forming a full callstack graph.
Consensus within a worker group
Node selection occurs via ProgVRF. When a node - or multiple nodes - within a worker group returns an invalid computation result, a failure rate of less than 33% is ignored and the results of the majority is recorded on the state chain. If the failure rate is more than 33% and less than 50%, the corresponding functions above the failure point within the callstack graph (done through a tree search) is recomputed, and the majority is selected as the truthful state.
Penalization records are updated on the device keychain, as all state verification commits must be signed with a unique device key. For larger, permissionless application groups with many nodes, rewards for computation and - yes, staking for improved security - can be done on an external public chain with an application-specific Smart Contract (an external world state provider). It is recommended that value transfers occur with a trusted stablecoin solution. More on this will be covered with a separate article.
Where from here?
I only covered the way Gemer treats application state, how that affects its technical implementation, and why Gemer is different from other blockchains and DLTs. This means that other aspects of a distributed system, including consensus, security and economics, should also be completely redesigned. I will be giving occasional updates later on, including:
- More details on Gemer's BFT implementation (including ProgVRF) and node penalization
- How Gemer processes payments on an external chain, payment gateways and its service subscription model
- Gemer's security guarantees & its Ethereum bridge (ensuring better security, and enabling interactions with existing Ethereum DApps)
- How Gemer's WebAssembly implementation was modified to fit its state model
- Other miscellaneous details
This was honestly one of the longest articles that I have ever written. If you are still here and reading this, I really appreciate it - thanks for your support.
To sum up everything I covered in this article:
- The reason behind why blockchain is not gaining enough "mainstream adoption" and "support" is because blockchains are not built for such for tasks in the first place - their state processing model is inefficient from a traditional infrastructure perspective.
- Blockchains treat state transitions as program execution that requires consensus, whereas in traditional computer science it's the other way around. This is the reason for blockchain application bottlenecks.
- However, this doesn't mean that blockchains are useless - it may already have reached "mainstream adoption". It's just that they are not designed to do what crypto people usually expect. Blockchains are amazing as a component, but not so great at tasks requiring user interaction - which is why they are moving from a "Web. 3.0" on its own to a modular component within the traditional web stack. And it's working really well. They are just different, like a monkey and a fish. Making blockchains run commercial apps directly is like trying to make a fish climb a tree. Fish have its own strengths, and so does blockchains.
- Gemer is built to act as another component within this web stack, but replacing the user-facing computation part instead of being a blockchain.
- This is possible because Gemer's state processing is done in a different way from distributed ledgers - they rely on functional programming and a stateless execution model for state verification.