A couple corrections:  

In the description for Section C, drop the word "addresses" as there is no address to represent a complex output script (other than P2SH).  You also need to add an element to represent "m" (the number of signatures), "n" can be implicitly determined by the number of PubKeys.   This is pretty easy to achieve as native multisig (not be confused w/ P2SH) is limited to m<=2 and n <= 3 resulting in only five combinations possible (1 of 1, 1 of 2, 1 of 3, 2 of 2, 2 of 3).

Pay2PubKey and native multisig outputs use PubKeys not PubKeyHashes and they are 33 or 65 bytes not 20 bytes.  Only ScriptHashes and PubKeyHashes are 20 bytes.  You could either handle this by treating compressed (33 bytes ea) and uncompressed (65 bytes ea) PubKeys as a separate type ("tag" in your format) or by just parsing the first byte (0x2 or 0x3 = compressed = 32 bytes follow, 0x4 = uncompressed = 64 bytes follow).  The good news is that 99.9%+ of spendable outputs fit one of 7 standard templates.  By also having an unknown type and standard for storing arbitrary scripts (type 0 = unknown) you can store all possible claimants.

Something like this:


A couple notes
1) Yes 1 of 1 "multisig" is a valid output.  It can probably be combined with Pay2PubKey entries as the input format of both are identical.
2) It may be better to break the types down to separate uncompressed and compressed.  With multisig they can be mixed so there are multiple combinations (i.e. for x of 3: UUU, UUC, UCC, CCC).
3) Scripts are limited to 10,000 bytes so for type 0 that puts an upper bound on the length of unknown scripts.  In practice scripts are much smaller on average.
4) Unknown (type 0) could be broken out to identify more discrete templates.

All unspendable outputs can be dropped.  Outputs matching the following templates are unspendable (either by design or error) and can be dropped from the snapshot.bin.  There is no reason to keep them anymore than spent output but it doesn't make much difference as there are less than 2K outputs currently in the blockchain.


An alternative idea is to recognize that complex scripts (including native multisig) have different properties and thus treat them differently.  Pay2PubKeyHash, Pay2PubKey, Pay2ScriptHash make up the majority of the outputs (>99.97% by total created outputs lets see how it holds up by value) and they have a single identifying characteristic (as hash or key).  To normalize them and reduce the size, a hash could be taken of the PubKeys for formats 2 & 4.  This would make all 4 formats (1-4)  representable by a single hash and value.  

All other outputs could just be stored as raw scripts.  While it may not be the most efficient, remember they make up 0.03% of outputs and that will probably shrink due to P2SH, so how you handle the top 4 templates is going to determine the bulk of the snapshot ledger size.



As a side note it would have been useful if Bitcoin (or any derived coin) had moved all scripting to the tx inputs and kept all outputs as just script hashes (i.e. <ScriptHash:20> vs OP_HASH160 <ScriptHash:20> OP_EQUAL).  It would make transactions smaller by removing excess opcodes, and more importantly by shifting the "weight" of the tx to the input it would reduce the size of the UXTO (which records the set of outputs.  As a side benefit it would make things like snapshot.bin easier and lighter.  Case in point the current UXTO spends ~35 bytes per output.  If it could be reduced to a output hash (20 bytes), and value stored as varint (4 bytes average) it would reduce the size of UXTO by roughly a third.  Bitcoin could move towards a system like that by soft fork (make non P2SH output non-standard after certain point) and the UXTO would be normalized over time as non-P2SH outputs are spent and replaced with P2SH ones.  Just something for future altcoin developers to consider.

Yes it would be possible, I think I just wasn't clear.  The PubKey is not recorded in the parent tx input, so for SCV some form of PubKey recovery will be needed.  ECDSA does support PubKey recovery, Bitcoin just doesn't use it in transactions (why Satoshi why? ).  Given a signed bitcoin transaction, the PubKey can be recovered, hashed and compared to the ledger.  The more I think about it there is probably no reason to not do this way (and just store other outputs as raw scripts).

I wouldn't separate out the entries which are a hash of the PubKey pulled from the UXTO from the entries which are a PubKeyHash from the UXTO (i.e. Section A & Section B).

For example the output script for tx 0e3e2357e806b6cdb1f70b54c3a3a17b6714ee1f0e68bebb44a74b1efd512098, output 0 is:
0496b538e853519c726a2c91e61ec11600ae1390813a627c66fb8be7947be63c52da7589379515d 4e0a604f8141781e62294721166bf621e73a82cbf2342c858ee OP_CHECKSIG

If we take the HASH160 (which is RIPEMD-160(SHA-256(PubKey)) we get
119b098e2e980a229e139a9ed01a469e518e6f26

If this is the only output for that PubKey it can be represented in the ledger under "section A" as
119b098e2e980a229e139a9ed01a469e518e6f26 000000012A05F200


In essence we have created a Pay2PubKeyHash entry from each Pay2PubKey output in the UXTO, making Section B redundant.

So two final questions.
1) Does it make sense to store PubKeyHashes and ScriptHashes separately over a more universal IdentifierHash?
2) Does it make sense to decode native multisig outputs into "templates" instead of storing them as raw scripts (Section E)?  They represent only 0.03% of all outputs (hope to get the time to parse and determine % of value this weekend).

There is no right answer just trying to think ahead to what makes sense and what is gained or lost over one option compared to another.  The answers of those questions determine how to update your format.

I'm not making an ethical judgement here, I'm just saying that people will view this largely the same as a premine, because it is.

The DRK example keeps coming up because of the alleged extreme instamine of supposedly 50%. You could in some sense do a spin-off and then instamine so many coins (say 50% of the remainder) that that the spun-off coins become insignificant, or at least much less significant.

This has little to do with the spin-off technique though. There are all kinds of crappy things you can do with a coin after the spin off.

Yes, I've been following.  This is great!  I agree that it's probably best for the parser to work off the UXTO set as well.  


You obviously have way more experience than me with the bitcoin technical details, but I thought the reason there were two forms for compressed public keys 02<x> and 03<x> was because there are two possible y values for a given x value.  Aren't you losing a small amount of information this way?  And would there be anything wrong with converting uncompressed public keys into the standard compressed format?

Need some sleep but here is a preview of UXTO Parser.

Distribution of unspent outputs (haven't combined entries to the same identity).  RawScript includes "native" multisig.  It hasn't been broken out yet.


As expected the PubKeyHash, PubKey, and ScriptHash outputs combined make up 99.98% of the outstanding coins.

It is somewhat surprising.  I would have imagined that the cold storage of major exchanges would be secured by m of n  P2SH scripts.  We know that none of them do that simply because the total value encumbered by all ScriptHashes & Raw scripts is <0.16% of all BTC.

Haven't had much free time to do more analysis of the blockchain but found an interesting test case in this tx:
https://blockchain.info/tx/5143cf232576ae53e8991ca389334563f14ea7a7c507a3e081fbef2538c84f6e

This tx has (or had as of the snapshot block) 3073 unspent outputs.  It is the largest tx when measured by the number of unspent outputs.

I removed zero value outputs, excluded 20,412 provably unspendable outputs, and consolidated outputs to the same signer into a single claim.  Here are some updated distributions (which if you are like me and like stats may find interesting beyond just this concept).  

All stats are as of  block 305,303 but I doubt it has changed significantly through the most recent block.  I would expect the number of non-standard scripts, native-multisig, and PubKey outputs to decline as they are spent and replaced with an increasing number of P2PkH and P2SH outputs which means that snapshots should become more efficient over time.




A large portion of the "custom scripts" can be refactored to be a standard template or they are invalid.  I now have only 20 scripts which are potentially valid and can't be reclassified.  Another interesting thing is there are a large number of native-multisig outputs which have one or more invalid PubKeys.  This allows them to be refactored.   I parsed all multisig outputs and removed invalid pubkeys.  If the number of valid keys is less than m in m-of-n then the output is unspendable.  If the number of valid keys is reduced to 1 and the script is 1-of-n then it can be refactored to a Pay2PubKeyHash script.  I haven't hashed PubKeys (to convert them to PubKeyHashes) yet.  Doing so may allow further consolidation of the output set (for keys where there are both PubKeyHash and PubKey outputs for the same key).

The size of the snapshot will depend heavily on the min claim value used. If we assume the average record is 24 bytes (20 byte identifier and 4 byte for value) that would make a full blockchain snapshot ~72MB.  It would be up to the spinoff developer but I believe to reduce size a min claim of 10, 100 or 1,000 bits is defendable.   That would result in a snapshot of 54 MB (25% reduction), 48 MB (33% reduction), and 38 MB (47% reduction) respectively.  Even with a minimum of 1,000 bits the included claims represents > 99.99% of the spendable BTC in the blockchain.

I'd like to refine the notion in the OP of "most efficient coin distribution," why Bitcoin has something vastly closer to it and why other coins can't hope to catch up anytime soon.

I think the answer requires a clear understanding of the crucial role investors play in society, so here I will attempt to explain that for those who've never considered it:

Simply put, a good investor can predict the future better than other people can. If, for example, you're a good investor and you predict a water shortage in a few years you can buy water rights now and have a good chance of selling them later for a profit. However, as a side effect, you bid up the price of water for everyone else, which incentivizes people to conserve water, incentivizes water producers to invest more into water production, motivates new companies to get into the water business, prompts more people to major in related fields in college, etc. The water shortage you correctly predicted will be mitigated by these actions. In this way, "greedy speculators" can even save lives.

As a good investor, you would likely sell when the water production infrastructure has become too big and society would benefit more from some of the resources/manpower being diverted elsewhere, and by pushing down the price you create exactly the right incentive structure for that. Being as most people aren't good at predicting the future and you are, you help society manage resources more efficiently than it would without you.

Not only do you help society manage resources more efficiently, even to the point of saving lives, but the better you are at that the more power you will gain to do it on a larger and larger scale since you keep getting wealthier and have more power to move markets. Think of how many lives Jim Rogers has probably saved in Africa by being such a successful and large-scale commodities investor! Conversely, if you're bad at predicting the future like most people are, you will tend to lose money and damage society - in the case of water by aggravating the shortages and bankrolling the oversupply gluts - but your ability to affect society will diminish as you lose your wealth and thereby effectively lose the "right" to divert resources in society.

In this way, the freer the market, the more it optimizes toward moving the ability to direct resources into the most capable hands: the most highly skilled investors (future prediction specialists) and the best entrepreneurs - and these people are exactly the people we want getting rich. Without them, we'd all be a lot less rich ourselves, or maybe even dead or never born because the resources for our parents to raise us weren't sufficient due to systemic mismanagement. Notice this is not centralized government management; every argument for "why we need central planners" is answered in the concepts touched on above: there is a huge distributed network of specialists at predicting the future and putting resources to the most efficient known uses. These are your planners, bringing all the benefits of having experts plan stuff, but not central planners in government offices.

Over time (not immediately), this process vests greater control over the societal ledger with those people who benefit society the most with their skills, and of course these skilled people are also motivated to do this because it directly benefits them by making them rich.  

Note that even short-term speculators and day traders - if they are good - also play a valuable role: they dampen short-term and even intraday volatility by buying the dips and selling the peaks. It's the unskilled speculators that create the volatility, but those people tend to lose their shirts pretty fast. That's a major reason why there's less volatility as a market matures, with the unskilled and therefore volatility-increasing speculators getting smaller and smaller while the skilled and therefore volatility-reducing speculators get bigger and bigger, thereby making the market ever smoother. (Note, however, that even though the Bitcoin market is maturing, it is not getting that much less volatile, simply because the market expands exponentially, so that the immature portion of it is always bigger than the older mature portion.)

Now Bitcoin investment is a little more abstract than water investments. We cannot say that investing in Bitcoin before it got scarce helped society avert a water shortage. However, it incentivized miners to get in and secure the network to a level commensurate with the market cap, it motivated people like Tony Gallippi to divert his talents and time toward starting BitPay, and it generally created the incentive structure to support many other similar infrastructural buildouts. Of course it also attracted media and other much-needed public attention, as well as that of academics and of course investors.

It is obvious from reading the speculation forum (and reddit) that only those with the conviction to hodl actually hodl, and those strongly tend to be people who both understand Bitcoin at a deep level (aren't spooked by FUD) and understand the necessary aspects of investing (aren't spooked by volatility because they understand the dynamics of the binary bet and exponential growth). When bitcoin went from under a dollar to over $30, then down to $2 over the course of a single year, the market was shaking out the weak hands - the poor speculators - in a spectacularly violent fashion. But what is this volatility optimizing for - that is, who's this market making rich? Here are some obvious ones:

1) Good hodlers, people who deeply grok Bitcoin and its investment aspects, who will likely dishoard in an organized fashion at peaks, dampening the bubbles and not contributing to the panic knifedowns nor to the FUD-based selloffs

2) Good traders who see the big picture, thereby decreasing the volatility

3) Perhaps warranting their own category are the people who have extremely deep or highly technical understanding of Bitcoin, to the level of what would almost seem to others like "insider information" (though there's nothing bad about this; in fact it's a good thing, if you follow the reasoning outlined above)

The main point is, besides some people who got lucky, the Bitcoin ledger is held mainly by people who most strongly believe in and most intimately understand Bitcoin (and understood it early!) and the relevant investing principles, or those who have actually contributed materially to the ecosystem. These people are far more likely than others to know where resources should be diverted to in the economy for maximum benefit to the community, and the maturity of the Bitcoin ledger ensures that they actually have the ability to command such resources (i.e., they have accumulated a lot of bitcoins to invest in infrastructure, and they have relatively little competition from unskilled investors who fund dead-end projects that waste dev time, etc.).

TL;DR: While the debate over whether the Bitcoin ledger is more or less "fair" than some other distribution may never be settled to everyone's satisfaction, we can clearly see that it is in a very useful sense the most efficient distribution for the general benefit of the community, thanks to the eminently helpful role that skilled investors play in society and how the extreme volatility over the years has continually shaken out the unskilled investors who would waste the community's resources.

If we imagine even a new amazing coin that is magically perfectly distributed to everyone, 100 coins for every person in the world, we can easily see that the initial investment decisions will be terribly skewed since all sorts of unskilled people will be throwing their coins at things and diverting resources and labor into useless, go-nowhere projects. Also, volatility would be way worse, and panic sell-offs due to FUD/misunderstandings would be horrendous. A ledger needs to grow organically through a market process, or else it will face challenges that it would only be ready to face some years later when the market is more solidified in strong hands by the above process.

By the way, are there any other things the Bitcoin ledger has optimized for (any other type of person the history of Bitcoin price movements has made especially rich (or especially shaken out))?

Correct.  It would be roughly 70 MB depending on the specific format used. The raw script and native multisig claims are a rounding error and won't significantly affect the size of the snapshot. There are ~3 million claims so the size will depend on the size of the P2SH and P2PkH claim records.  At a minimum this would be 20 bytes for the identifier plus 2 to 3 bytes avg (varint) to represent the value.  At most it would be 29 bytes per record (1 byte for type, 20 byte identifier, 8 bytes for value).  

One final optimization option would be to refactor all the native multisig outputs to their P2SH equivalents.  Users could still make claims in a trust free manner but the original scripts should probably be retained in a separate file (probably along with all other refactored outputs).  The refactor file wouldn't be needed by clients but it would be useful for web services which assist users in identifying if they have a claim and what steps they need to take to redeem that claim.

For the record here are the only "valid" unspent outputs which are still raw scripts.  This just means I haven't yet been able to provably show them as unspendable or refactor them to either a Pay2PubKeyHash or Native-Multisig claims.  It probably is possible to reduce this list of exceptions further.


To refactor valid but not non-standard outputs and to remove invalid outputs I first reduced the raw scripts in the UTXO to a set of templates.  Then updated the parser to handle these templates.  The remaining raw script outputs are one of the following templates which the parser handles as "unknown" and keeps as a raw script.





Some of these outputs are "insecure" in that once the true owner tries to spend them (or sign a claim), anyone else could modify the output and steal the coins and/or claim.  One example is  af32bb06f12f2ae5fdb7face7cd272be67c923e86b7a66a76ded02d954c2f94d:0.  The output is encumbered only by a hash (no signature required).  The ScriptSig needed to produce a valid txn is thus just some value which when hashed produces the hash value in the PkScript (output script).  The problem is that when the owner attempts to spend those outputs any other node (including a miner) could modify the txn substituting their own output instead.  Since the required input is just a value with no signature there is no way to prevent this (other than trusting a miner in secret).  I am not exactly sure how they should be handled but I am leaning towards truncating them for security reasons.


A large portion of the sub dust txns (especially the 1 satoshi variety) are spam from Satoshi's Dice.  The service used a 1 satoshi (effectively unspendable) output to notify betters of a failed bet.  They were a significant reason for the introduction of the dust output rules.  At one time the 1 satoshi outputs represented almost half of the UTXO.   Actual spendable outputs tend to grow rather slowly so without the dust rules it is entirely possible at some point the overwhelming majority of the UTXO would have been dust spam.  Despite the controversy at the time the dust rules are both logical (what purpose is there in creating outputs which are marked as spendable, stored by all nodes forever, and yet are uneconomical to spend) and necessary to prevent the UTXO from becoming utterly unmanageable.