Introducing rosetta-bitcoin: Coinbase’s Bitcoin implementation of the Rosetta API

Posted on September 23, 2020 by admin

By Patrick O’Grady

In June, we launched Rosetta as an open-source specification that makes integrating with blockchains simpler, faster, and more reliable. There are now 20+ blockchain projects working on a Rosetta implementation (Near, Cardano, Celo, Coda, Neo, Tron, Handshake, Oasis, Cosmos, Decred, Filecoin, Ontology, Sia, Zilliqa, Digibyte, Harmony, Kadena, Nervos, and Blockstack), five in-progress SDKs (Golang, JavaScript, TypeScript, Java, and Rust), and eight teams have made contributions to at least one of the Rosetta repositories on GitHub (rosetta-specifications, rosetta-sdk-go, and rosetta-cli).

Today, we are sharing a key contribution to this growing collection of implementations: rosetta-bitcoin.

Why Bitcoin?

Bitcoin is the bellwether for all of crypto, is the most popular blockchain, has the largest market capitalization, and most blockchain developers know how it works (so it is easier to understand how Rosetta can be implemented for other blockchains).

On another note, the reference implementation for Bitcoin (known as Bitcoin Core) doesn’t provide native support for many of the features integrators want. It is not possible to query account balances and/or UTXOs for all accounts, serve preprocessed blocks to callers so they don’t need to fetch all inputs to parse a transaction, nor to construct transactions without importing private keys onto the node (which isn’t practical for users that never bring private keys online). Often, these missing features drive integrators to run some sort of additional “indexing” software and implement their own libraries to handle transaction construction.

rosetta-bitcoin provides access to all these features, requires no configuration by default, and can be started with a single command. Furthermore, rosetta-bitcoin enables these features exclusively through RPC interaction with Bitcoin Core so we don’t need to maintain a fork of Bitcoin Core to enable this new functionality and easy configuration!

Rosetta API Refresher

rosetta-bitcoin implements both of the Rosetta API core components: the Data API and the Construction API. Together, these components provide universal read and write access to Bitcoin. We’ve included several diagrams below that outline the specific endpoints that any Rosetta API implementation supports. If you are interested in building on top of an implementation, we recommend using rosetta-sdk-go (which abstracts away these flows behind Golang functions).

The Data API consists of all the endpoints used to “get information” about a blockchain. We can get the networks supported by an implementation (which may be > 1 if a blockchain supports sharding or if it is a gateway to multiple networks), the supported operation types on each network, and the status of each network.

The Data API also allows for getting the contents of any block, getting a particular transaction in a block, and fetching the balance of any account present in a block. Rosetta validation tooling ensures that the balance computed for any account from operations in blocks is equal to the balance returned by the node (often called “reconciliation”).

Lastly, the Data API allows for fetching all mempool transactions and for fetching any particular mempool transaction. This is useful for integrators that want to monitor the status of their broadcasts and to inspect any incoming deposits before they are confirmed on-chain.

While the Data API provides the ability to read data from a blockchain in a standard format, the Construction API enables developers to write to a blockchain (i.e. construct transactions) in a standard format. To meet strict security standards, implementations are expected to be stateless, operate entirely offline, and support detached key generation and signing. We can derive an address from a public key (on blockchains that don’t require on-chain origination).

When constructing a transaction generically, it is often not possible to fully specify the result or what may appear on-chain (ex: constructing a transaction that attempts to use a “flash loan”). We call the collection of operations we can specify the transaction “intent” (which is usually a subset of all operations in the on-chain transaction). At a high-level, constructing a transaction with the Construction API entails creating an “intent”, gathering the metadata required to create a transaction with the “intent”, signing payloads from accounts responsible for the “intent”, and broadcasting the transaction created. Before attempting to sign or broadcast a transaction, we confirm that the transaction we constructed has the same “intent” we originally provided when kicking off the construction flow. You can see this entire construction flow in the diagram below:

Once we have a signed transaction (that performs the “intent” of our choosing), we can calculate its network-specific hash and broadcast it.

How it Works

We optimized for package re-use when developing rosetta-bitcoin. If it could be done with an existing package from rosetta-sdk-go, we used it. This has led to upstreaming a few significant performance improvements as we benchmarked and optimized rosetta-bitcoin.

We use Bitcoin Core to sync blocks/broadcast transactions, ingest those blocks using the syncer package, store processed blocks using the storage package, and serve Rosetta API requests using the server package from data cached using the storage package. You can find a high-level view of this architecture below:

To implement the Rosetta API /account/balance endpoint, we had to build a UTXO indexer that provides atomic balance lookups. “Atomic” in this sense means that we can get the balance of an account with the block index and block hash where it was valid in a single RPC call. With our Rosetta Bitcoin implementation, you don’t need to run a separate indexer anymore!

We implemented concurrent block ingestion to speed up block syncing and automatic pruning to remove blocks from Bitcoin Core after we ingest a block to save on space. Concurrent block ingestion allows us to populate multiple blocks ahead of the currently processing block while we wait for the most recently populated block to save (keeping our storage resources busy). Because we store all ingested blocks in our own storage cache, we don’t need to keep duplicate data around in Bitcoin Core’s database.

Last but not least, we implemented stateless, offline, curve-based transaction construction for sending from any SegWit-Bech32 Address. We opted to only support sending from SegWit-Bech32 addresses to minimize complexity in the first release (there are a lot of new moving pieces here). We look forward to reviewing community contributions that add MultiSig, Lightning, and other address support.

Try it Out

Enough with the talk, show me the code! This section will walk you through building rosetta-bitcoin, starting rosetta-bitcoin, interacting with rosetta-bitcoin, and testing rosetta-bitcoin. To complete the following steps, you need to be on a computer that meets the rosetta-bitcoin system requirements and you must install Docker.

First, we need to download the pre-built rosetta-bitcoin Docker image (saved with the tag rosetta-bitcoin:latest):

curl -sSfL https://raw.githubusercontent.com/coinbase/rosetta-bitcoin/master/install.sh | sh -s

Next, we need to start a container using our downloaded image (the container is started in detached mode):

docker run -d --rm --ulimit "nofile=100000:100000" -v "$(pwd)/bitcoin-data:/data" -e "MODE=ONLINE" -e "NETWORK=TESTNET" -e "PORT=8080" -p 8080:8080 -p 18333:18333 rosetta-bitcoin:latest

After starting the container, you will see an identifier printed in your terminal (that’s the Docker container ID). To view logs from this running container, you should run:

docker logs --tail 100 -f <container_id>

To make sure things are working, let’s make a cURL request for the current network status (you may need to wait a few minutes for the node to start syncing):

curl --request POST 'http://localhost:8080/network/status'

--header 'Accept: application/json'

--header 'Content-Type: application/json'

--data-raw '{

  "network_identifier": {

    "blockchain": "Bitcoin",

    "network": "Testnet3"

}' | jq

Now that rosetta-bitcoin is running, the fun can really begin! Next, we install rosetta-cli, our CLI tool for interacting with and testing Rosetta API implementations (this will be installed at ./bin/rosetta-cli):

curl -sSfL https://raw.githubusercontent.com/coinbase/rosetta-cli/master/scripts/install.sh | sh -s

We recommend moving this downloaded rosetta-cli binary into your bin folder so that it can be run by calling rosetta-cli instead of ./bin/rosetta-cli). The rest of this walkthrough assumes that you’ve done this.

We also need to download the configuration file for interacting with rosetta-bitcoin:

curl -sSfL https://raw.githubusercontent.com/coinbase/rosetta-bitcoin/master/rosetta-cli-conf/bitcoin_testnet.json -o bitcoin_testnet.json

We can lookup the current sync status:

rosetta-cli view:networks --configuration-file bitcoin_testnet.json

We can lookup the contents of any synced block (make sure the index you lookup is less than the index returned by the current index returned in sync status):

rosetta-cli view:block <block index> --configuration-file bitcoin_testnet.json

We can validate the Data API endpoints using the the `check:data` command:

rosetta-cli check:data --configuration-file bitcoin_testnet.json

This test will sync all blocks and confirm that the balance for each account returned by the `/account/balance` endpoint matches the computed balance using Rosetta operations.

Lastly, we can validate the Construction API endpoints using the `check:construction` command:

rosetta-cli check:construction --configuration-file bitcoin_testnet.json

This test will create, broadcast, and confirm testnet transactions until we reach our specified exit conditions (# of successful transactions of each type). This test automatically adjusts fees based on the estimated size of the transactions it creates and returns all funds to a faucet address at the end of the test.

When you are done playing around with rosetta-bitcoin, run the following command to shut it down:

docker kill --signal=2 <container_id>

Future Work

Publish benchmarks for sync speed, storage usage, and load testing on both testnet and mainnet
Implement Rosetta API /mempool/transaction endpoint
Add CI test to repository using rosetta-cli (likely on a regtest network)
Support Multi-Sig transactions and multi-phase transaction construction
Write a wallet package (using rosetta-sdk-go primitives) to orchestrate transaction construction for any Rosetta implementation (you can find some early work on this effort here)

If you are interested in any of these items, reach out on the community site.

Work at Coinbase

We are actively hiring passionate developers to join the Crypto team and a developer relations lead to work on the Rosetta project. If you are interested in helping to build this common language for interacting with blockchains, Coinbase is hiring.

This website contains links to third-party websites or other content for information purposes only (“Third-Party Sites”). The Third-Party Sites are not under the control of Coinbase, Inc., and its affiliates (“Coinbase”), and Coinbase is not responsible for the content of any Third-Party Site, including without limitation any link contained in a Third-Party Site, or any changes or updates to a Third-Party Site. Coinbase is not responsible for webcasting or any other form of transmission received from any Third-Party Site. Coinbase is providing these links to you only as a convenience, and the inclusion of any link does not imply endorsement, approval or recommendation by Coinbase of the site or any association with its operators.

All images provided herein are by Coinbase.

Introducing rosetta-bitcoin: Coinbase’s Bitcoin implementation of the Rosetta API was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.