“SPV mining” is the solution, not the problem

Is was Wednesday, March 26, 2014. It was a cold winter outside, but not inside an old building of the Princeton university campus. The room, full of people, was warm. I was standing in front of some of the brightest minds in Bitcoin: Andresen, Bonneau, Eyal, Maxwell, Miller, Gün Sirer, Szabo, Todd and Zooko, to name only a few. My talk was about “SPV Mining”, but at that time that misleading name had not been coined.

Mining in the dark

My point was simple, I had to convince everyone in the room that SPV Mining was not only inevitable (if not prevented by a softfork or hardfork) but was the solution to many of the Bitcoin protocol problems, including a future hard fork to scale Bitcoin. One of the problems with Bitcoin at that time was the monetary bias towards bigger miners, because bigger miners have a lower rate of orphan blocks (but not bigger mining pools, as the block withholding attack later proved).

The core developers were at least very skeptic: miners not validating transactions sounded like a crazy and dangerous idea. A heresy. My thesis was that either we programmed SPV mining right into the heart of Bitcoin Core and do it right, or the miners will do it anyway, but wrong.

Miners have always been lazy or afraid to change the bitcoin client due to the danger of breaking consensus code, then create invalid blocks and lose the rewards. Even if the default mining strategy that results in using Bitcoin Core with most mining pool management applications is still sub-optimal and makes them loose about 0.6% of their revenue (as some data that G. Maxwell has recently gathered from mining pool clients proves) they haven’t attempted to improve it.

However, that was not the case with SPV Mining: on July 4^th, 2015,  BIP 66 (which requires the block version to be >= 3) became enforced. But about 5% of the network was still solving version 2 blocks. One of the miners in this 5% solved a block with version 2 (which was therefore invalid), and then the “BTC Nuggets” pool mined a block that referenced the invalid block header without verifying the transactions (aka “SPV Mining”). Afterward, 6 additional invalid blocks were mined in a chain by different pools without any transaction verification. About 67K USD was lost (at today’s rate). Many mining pools had coded the “SPV Mining” strategy after all.

The problem is that they had coded it wrong.

Now you may be asking why should we allow SPV mining, why not modify Bitcoin to prevent it?

Preventing SPV mining

SPV mining can be prevented easily by including in a block coinbase field some data gathered from the transactions in the previous block. One minimalistic approach suggests adding to the coinbase field a single byte (the “proof of knowledge” field or PoN) from the previous block coinbase hash. That method prevents SPV mining when it is performed by listening to other pools Stratum API, such as what happened in the BIP66 event. But it does not prevent coordinated SPV Mining: if all miners agree to broadcast the missing PoN byte along the header, it can still be performed, with minimum extra overhead.

A better solution would be that the PoN field is a hash of the current coinbase scriptPub concatenated by all the transactions of the previous block. This way no additional data smaller than the transaction data can serve that purpose for everyone. Even better, the PoN could consist of the hash of a signature of the coinbase scriptpub (using the same key that redeems the coinbase scriptPub), followed by all previous transactions. That forces an SPV miner to perform an interactive protocol with a full node. But both proposals do not prevent a miner from having the data, and even then refuse to validate it.

An even a better solution is that the PoN consist of the hash of the signature (as before), all previous transactions and the heights of the blocks of all transaction outputs consumed in the previous block. At least this way the miner is proving it has performed one of the steps of transaction verification (UTXO lookups).

Why SPV mining should not be prevented but encouraged

Two years ago Matt Corallo (TheBlueMatt) started coding the Bitcoin Relay Network (BRN, http://bitcoinrelaynetwork.org/). The BRN is a private owned, funded by donations, centrally controlled, and possibly non-DoS protected nor load balanced network of servers that relay blocks and transactions between three continents at the lowest possible latency. As Matt describes it:

“The Bitcoin Relay Network is a system of peering between nodes in the network by creating a system of high-speed relay nodes for miners and merchants/exchanges. This system a) acts as a fallback in the case that the public Bitcoin network encounters issues and b) decreases block propagation times between miners.”

Sounds good? So instead of a block requiring 10 seconds to reach more than 50% of the miners, now we can do it in just 150 msec. Matt did some miners a good favor. The problem, of course, is that the BRN disadvantages all miners not using it (as Aaron van Wirdum pointed out). So currently I suspect most mining pools are using it, since they have nothing to lose by doing so. But the BRN does not benefit all miners equally: the bigger pools benefit less. Now if somebody anonymously attack the BRN, the biggest miners will have their advantage back. In the extreme case (irony here) that group of miners that represents more than 50% of the hashing power are physically clustered in a close area and therefore have low latency between them, then by doing a DDoS on the BRN they could increase their revenue by 2%, by forcing the remaining 50% to lose 2%.

So the main network, the original peer-to-peer network for all we mortals, must be almost as fast as the BRN. The community should aim to reduce the latency in the main network and expect that one day the BRN is not needed anymore.

And the way to do it is, among other improvements, to allow SPV Mining. The other immediately evident modification is to allow header-only propagation, in a flood-fill mode, just before full block propagates in the normal announce-request mode. SPV Mining can be coded (as a policy) to be allowed for some very short time, such as 20 seconds. That gives plenty of time to download block transactions and validate them. My old NimbleCoin cryptocurrency was heavily based on SPV Mining and had a 5 seconds block interval with a very low orphan rate. Also NimbleCoin implemented (and later G. Maxwell re-invented) that SPV mined blocks are tagged so that SPV clients can wait a little longer on those blocks. Any website that accept 1 confirmation could request a second confirmation if the first block is tagged. That does not mean it has to wait more: if a block is tagged is because it was solved too fast. Another way to prevent miners from doing SPV mining all the time it with a soft-fork (a consensus change). Similar to how PoN that prevents SPV mining, if the DoN field corresponds to grandparent block data instead of to the parent block data, it limits SPV mining to a single block, instead of to zero blocks.

Basically SPV Mining + header-first propagation provides O(1) propagation (as long as propagating the full block requires much less than 10 minutes). That was the aim of Gavin’s IBLT proposal, which involves much complex changes to the code-base.

One third change than improves the solution is adding a local route optimization to the Main net. This optimization dynamically embeds into the gossip network a DAG, like a miner’s minimum spanning tree, that reduces even more the hops and propagation latency between miners.

The Rootstock smart-contract platform (a new Bitcoin side-chain) makes use of SPV mining, so soon everyone will benefit from it, and see how it works in practice.

And I hope that after two years, I made myself clear enough to convince you about the benefits of it. Don’t hesitate in writing to me if you believe my thesis it wrong.

(Mining image courtesy of FreeDigitalPhotos.net)