What’s Coming To The Bitcoin Core Wallet in 0.21

Bitcoin Core 0.21 brings about a turning point to the Bitcoin Core wallet with the introduction of two major features and an eventual breaking of compatibility. The landmark wallet features of this release are Descriptor Wallets and the SQLite database backend. In this article, I will be explaining what these features are, what else is changing, and what the plan for the future is.

Upcoming Features

Descriptor Wallets

The headlining wallet feature for Bitcoin Core 0.21 is Descriptor Wallets. Descriptor Wallets store Output Script Descriptors in the wallet and use these to generate the addresses that users can use. Legacy Wallets (the non-descriptor wallet type, and the only type of wallet previous versions would create) instead used private keys to generate addresses. For even more detailed information on Descriptor Wallets, please read the tentative release notes.

The Problem with Legacy Wallets

Legacy Wallets were initially designed by Satoshi himself. As this was when Bitcoin was first created, the understanding of what Bitcoin could do and how Bitcoin can be used was not as well understood as it is today. As such, the wallet was built around the private keys and everything based around private keys even though Bitcoin has a scripting language that supports a lot more than just private keys. So Legacy Wallets primarily contain private keys, and from these private keys, addresses are made.

Even with just private keys, the Legacy Wallet was designed long before BIP 32 Hierarchical Deterministic Wallets were invented. It was designed to use a random number generator to create private keys, not deriving them from a seed. While the Legacy Wallet does support BIP 32 now, it is still not up to the level of support that other wallets have. This support is done by replacing the RNG with BIP 32 derivation. But there is no support for watching extended public keys (xpubs) and pubkeys cannot be derived, nor are pubkeys stored by the wallet.

Since Bitcoin was first created, we have learned that there are many more things that output scripts can do than just single keys. The scripting system in Bitcoin is powerful, but Bitcoin Core’s wallet cannot make use of it because more than just keys are involved in scripts. Even just the introduction of P2SH introduced some issues for the Bitcoin Core wallet, and supporting the watching of scripts is largely bolted on and not integrated very well. Watch-only in Legacy Wallets does not quite work as one would expect, and having a mix of watch-only and non-watch-only things in a Legacy Wallet only makes things more confusing. Ultimately, extending the Legacy Wallet has primarily been hacking in new features rather than having a truly well designed wallet that can make the full use of Bitcoin.

Benefits of Descriptor Wallets

In contrast with Legacy Wallets, Descriptor Wallets are designed to support the Bitcoin scripting system through the use of descriptors. Descriptors explicitly give an output script (and thus address) as well as all of the keys and scripts necessary to sign them. This essentially means that Descriptor Wallets are a script based wallet, while Legacy Wallets are key based.

A Descriptor Wallet will then be able to support any kind of descriptor. Newly introduced descriptors for new script types can be easily added to the wallet by adding a new descriptor. For example, the Taproot proposal introduces a new address type and output scripts. This can be easily added to the Bitcoin Core wallet by implementing a new descriptor.

Alternative ways of deriving keys and scripts can be added by implementing them in the simpler descriptor module. For example, Miniscript will eventually be added to the descriptor module. This will allow descriptors to describe arbitrary scripts, which means that the wallet will be able to create addresses for arbitrary scripts.

Descriptor Wallets also bring a simpler and saner way to do watch-only wallets in Bitcoin Core. The confusing behavior of having mixed watch-only and non-watch-only things in Legacy Wallet is not present in Descriptor Wallets. Descriptor Wallets can only have descriptors imported into them as well, so the scripts that it will watch for is explicit and can be computed. There will no longer be the Legacy Wallet behavior of watching for something related to an import but was not explicitly imported. Notably, because descriptors support xpub derivation, watch-only Descriptor Wallets can watch for BIP 32 wallets such as hardware wallets. This paves the way for full hardware wallet support in a future release of Bitcoin Core.

SQLite Database Backend

As Descriptor Wallets are a wholly new type of wallet which is backwards incompatible, it made sense to introduce a new database backend at the same time. So newly created Descriptor Wallets will now use SQLite as the database backend, instead of Berkeley DB that is used for Legacy Wallets. This will eventually lead to the outright removal of Berkeley DB. The introduction of a new database backend is motivated by the fact that Berkeley DB is not suitable for our use case, and the age of the version of Berkeley DB that we use.

The problem with Berkeley DB

Berkeley DB is not very suitable for our use case as an application data file; it was not designed to be used in this way. Instead it was designed for large, multithreaded databases, which the wallet is not. Because of this, the Legacy Wallet has several hacks to force Berkeley DB to behave like an application data file database, and these hacks can result in undesirable behavior. Notably, Berkeley DB wallet files can be easily corrupted which can result in the loss of private keys, which is not good.

Furthermore, Berkeley DB requires a database environment and it produces extra files which typically need to be moved with the database file if it is moved elsewhere. Without those extra environment files, data can be lost. This means that Berkeley DB is not portable, but that is unsuitable for a wallet file. Many users expect it to be a singular file that can be transferred to other computers or to other directories for backups. The Berkeley DB environment requirement also forces us to have directories for each wallet. This is confusing and unintuitive to users.

Lastly, the version of Berkeley DB used by Bitcoin Core is more than 10 years old. We use Berkeley DB 4.8 which was released in 2010. The requirement on this aging version of Berkeley DB is because backwards incompatible changes were introduced to the database environment files. This means that wallets loaded with a newer version of Berkeley DB cannot be loaded with an older version of Berkeley DB, and this breaks downgrade compatibility. This age has thus introduced additional issues for us. For example, to compile Berkeley DB using C++11, a patch is required. Berkeley DB 4.8 also cannot currently be compiled on the macOS Big Sur beta which is problematic for users who use (and will use) that OS and want to self compile Bitcoin Core. So a new database backend is desired.

Why SQLite

SQLite was chosen to be the new database backend because the SQLite developers provide several guarantees with regards to compatibility, support, and testing. The SQLite developers advertise that SQLite is suitable as an application data file, which is what we want for the wallet. SQLite is also very well tested and very widely used, so bugs and corruption issues tend to be found and fixed quickly. Furthermore, SQLite makes few file format changes, and all of these file format changes are feature dependent and backwards compatible. This means that if we don’t use a particular feature, then the file format will not change and it will be possible to upgrade and downgrade SQLite versions. Even though we currently use the latest version of SQLite, we have found that it is possible to use a version of SQLite from 2013.

SQLite is also designed to be portable and does not require a database environment. While it does temporarily produce extraneous files, those files are removed as soon as it is done with them, and everything is written to the database file when a database write completes. This is in contrast to Berkeley DB where a completed write does not guarantee that the data was written to the database file, it may have been written to an environment file. The lack of the environment will also allow us to move to single wallet files instead of wallet directories.

Other databases, such as LMDB, were also in consideration however none of the databases investigated had all of the same guarantees of SQLite. For example, LMDB is not portable across different CPU architectures, so this was determined to not be suitable. Many other databases have database environments and environment files, so these were not suitable for use either. SQLite was the only widely used database system that had all of the features that we wanted for a wallet file.

No Longer Creating The Default Wallet

The Bitcoin Core wallet used to always create a wallet for the user when one did not already exist. With the introduction of the multiwallet feature and the ability to create and open specifically named wallets, the default created wallet started to become more of an issue rather than a useful feature. In particular, users would sometimes find that their wallet appeared to disappear when some configuration changed. Of course, the wallet would not actually disappear, just that the data directory changed and a new default wallet was created. This can be very confusing and extra stressful to the user as a new blank wallet is created. This situation can be hard to debug, hard to explain, and difficult for non-technical users to fix.

So The default wallet will no longer be created if it doesn’t exist. Instead users will be prompted to create a new wallet or open an existing wallet. In this way, when a configuration mistake occurs, a new empty wallet won’t be created so users won’t think that all of their Bitcoin has disappeared. Furthermore, not creating a default wallet means that disk space is not unnecessarily used. Users can create a wallet of the type and name that they want instead of having an unnamed default wallet taking up space on disk.

Since users expect that their wallets be loaded when they start, there have been changes to the configuration system as well. These changes, while not wallet related, allow for Bitcoin Core to save which wallets to load again on the next startup. So the wallets that users create and load from the GUI will automatically be loaded again on the next startup unless they were closed during the session. For RPC users, a load_on_startup argument has been added to the createwallet, loadwallet, and unloadwallet RPCs to specify whether a wallet should be loaded on startup.

Compatibility

An important change to the wallet is that compatibility will be breaking. Backwards compatibility has always been a key pillar in the Bitcoin Core wallet. Changes have always had to consider users who decided to downgrade after an upgrade and users who bring in old wallets to a new release. But maintaining backwards compatibility is difficult, and almost impossible to do when making large sweeping changes.

Unfortunately, to make progress, sometimes compatibility needs to be broken. Sometimes the past needs to be left behind. As the Bitcoin Core wallet moves to making Descriptor Wallets the default, and later, only, wallet type, a future release of Bitcoin Core will no longer be able to open Legacy Wallets.

Of course this does not mean that the change will be overnight and occur in one release. Nor does this mean that Bitcoin Core will begin to make gratuitous backwards compatibility breaking changes. Instead we will be making it easy to move from Legacy Wallets to Descriptor Wallets. We will be making every effort to make Descriptor Wallets backwards compatible with previous versions that have them.

As such, there will be a long deprecation cycle for Legacy Wallets and Berkeley DB. The proposed timeline is to only have the final removal occur in late 2023. In the meantime, we will be introducing ways to migrate a Legacy Wallet to a Descriptor Wallet. It is, and will be, possible to take an existing Legacy Wallet, and make a Descriptor Wallet which exactly matches it. There will be tooling that allows users to do this. Even after the removal of Legacy Wallets and Berkeley DB, it will be possible to migrate Legacy Wallets to Descriptor Wallets through the use of a minimally dependent migration tool.

As we get closer to the actual removal date, there will also be warnings and instructions on how to migrate a wallet. So even users who don’t read the release notes and don’t pay attention to the Bitcoin Core news will know that they should migrate, and how to do so. Even though compatibility will be breaking, we are not leaving users out to dry and we are doing everything that we can to make sure that the change to Descriptor Wallets only is as smooth as possible.