@gmaxwell
Reviewing this thread, I found that you have done almost a complete rewrite of your post above and my reply no longer covers yours properly.
Good news is your edit has signs of
progress and I appreciate it, but I suppose it is not best practice, editing and re-editing, so ...
What you are describing here as "Proximity Premium" is normally called "progress". Normally we think of mining as working as a lottery: Your chance of finding the next block is just simply your proportion of the hashrate out of the total. But in a system with progress things work more like a race: the fastest party wins more often (or even always, if there is a lot of progress). Lottery vs race is a near perfect analogy-- in a lottery your chance of winning on a ticket doesn't change because of your prior ticket, but in a race your chance of winning in this step depends critically on all the steps before it.
Mining as a lottery (with single winner) suffers from variance as I have proved mathematically above-thread: winner-takes-all approach means Bernoulli distribution and for small
p (probability of winning all instead of nothing) it results in very high and intolerable variance. Although mining is based on probabilities it is not gambling, high variance is not tolerable when the stakes are high. For large
ps (>0.1) variance is very low and they can survive but for smaller shares it becomes too risky, they have just to back-off or join a pool.
Mining, the way you (and Satoshi was used to) put it, as a lottery literally died 8 years ago. It belongs to POB, Plain Old Bitcoin, when the stakes were low enough and it was a fun and an experiment.
As of proximity premium, what you call it 'progress', the mining scene doesn't change radically. Your race analogy is good but as an extreme case scenario. In the real network we have a very short window (< 1%) for miners to take advantage of this premium and as I have discussed extensively the total hashpower of miners increases when the premium decreases (only a minority are in the premium in each round).
So, we have a short window of time and a minority of hashpower that is taking advantage of their proximity to the previous block's miner. Suppose the miners in premium have 10% power, their chance to win the next block would be like 1*10
-3 After the window closes (without a block being found) the scene is reset to become a lottery again.
As I understand what you are saying is 'if we don't take care of this issue, progress, the 1% window gets larger and this phenomenon becomes more dangerous'.
My answer:
1- The whole purpose of this thread is to convince you and other community members that this flaw, progress/proximity premium is a logarithmic homomorphism of mining variance flaw. For instance if the window would be 1 minute (10%) the chance for the 10% miners in premium to mine the next block would be not more than 1%.
2- Bitcoin relay network has been improved since bip 152 and compact blocks (as of my knowledge you were a contributor, kudos) and since then with FIBRE and similar relay networks which efficiently connect miners, the propagation delay is under control.
3- Generally discussing, propagation delay is a technological problem, much simpler than game theory and mathematical challenges built into bitcoin. It is feasible to shorten propagation delay even more by improving relay protocol and using more advanced technologies but you can't do anything about variance as long as you keep winner-takes-all approach in Nakamoto's design unchanged.
Progress can be introduced into mining in many different ways, -- including from things like bad POW designs that try to reduce variance by making mining incremental, or blocks simply taking non-zero time to spread and validate across the network -- but the end result is the same: Progress creates an insidious centralization pressure where the bigger miner actually gets a better return on investment. Due to re-investment even small amounts of progress could conceivably completely centralize a system even absent other centralization pressures.
Progress is a lot more hard hitting than most people expect. You might look at 6 seconds of delay and wonder how could that matter against a 600 second block interval. But because of the nature of poisson processes most blocks are found much closer to each other than 600 seconds. We saw the direct effects of this in the network, when blocks started getting larger than 200k propagation times started getting upwards of 2 seconds and miners saw higher orphaning rates and consolidated onto fewer larger pools, even giving a single pool a super-majority share of the hash power for a little while. After we invented and deployed new technology that made blocks propagate faster, the consolidation reversed. Unfortunately, there is probably no amount of progress which is "safe" -- at best it's only small enough to be too small a centralization pressure to worry about compared to other issues, and the improved propagation requires cooperation -- if a large miner intentionally produces blocks with unrelayed transactions will immediately have the old slow propagation which will improve their own bottom line. When it comes to security you have to build for the worst case, not just hope people will cooperate instead of doing whatever is best for themselves. So long as the difference between the worst case and typical is small there isn't much incentive to exploit it, for the moment it seems to be working.
Concern about progress isn't new-- It's been part of the regular understanding of the engineers working on Bitcoin for a long time-- and bitcoin's creator probably understood it too considering that he chose a 10 minute block interval (smaller numbers _look_ fine until you start trying to work out the effects of progress).
You are simply exaggerating.
Firstly, as I have discussed above, progress/proximity premium is based on mining variance, without mining variance there would be no premium.
Secondly, we are not discussing selfish mining and invariant here, the miner who chooses to keep blocks or transactions private, puts himself in danger of producing orphan blocks, it has been extensively analyzed and rejected as an attack vector.
Thirdly, the push towards pools historically appeared not as a result of propagation delay and opportunity costs involved but as a result of variance.
Fourthly, there is always an amount of progress that is
practically safe.
You exaggerate
progress issue because you don't want to become face-to-face with the main issue: variance. Plus, you do so, to justify you and your colleagues history of resisting on-chain scaling and commitment to Lightning and side-chain improvements which I believe is not the job you have been assigned to and is a complete distraction.
That is why I find it disappointing that A
day ago you were claiming that there were no reasons to not radically increase block sizes and slandering all the engineers who've maintained Bitcoin its whole life. You have been more or less calling the people that built so much of what everyone uses malicious criminals, in post after post.
As I have made it clear, I'm not in favor of radically increasing the block size, but I'm against making a taboo of any increase proposal and using this taboo to suppress any on-chain scaling specially a moderate block interval decrease which is my favorite.
As of 'slandering' accusation:
I think when someone takes a responsibility s/he should be ready for criticism. Bitcoin is not a simple amateur project, when you make a mistake there are consequences. We are losing bitcoin to pools and exchanges and whales and we need more than "it is what it is" arguments.
I don't think I have
slandered anybody, but if you feel so, while I don't mind to express my regrets about it, I hope it wouldn't be interpreted as a free ticket for you guys to do whatever you want with bitcoin (mostly doing nothing!) and remain focused on LN.
What will you realize tomorrow? Maybe that progress and variance are not one in the same and in fact almost any variance reduction proposal increases the potential for progress once you realize that participants will behave strategically instead of honestly.
it is where you deserve to be
slandered Greg
You are literally ignoring all my arguments without refuting them! How is it possible to have premium without high variance? Instead of discussing theoretically you choose a 'proof by example' methodology:
For a simple example of where making variance lower makes progress worse, say that you require that a block present 10000 POWs of 1/10000th the difficulty instead of one (your proposal basically does that, in fact): That is a system with very low variance (1/10000th) and almost perfect progress: If you had two miners with 30% hashpower that didn't share or only shared work with each other and four miners with 10%, the 10% miners would almost _never_ find a block. Again, not a new result: Low variance hashcash using multiple POW was described in the hashcash paper ...
You are saying that selfish mining and %50+1 attacks are real! Good for you, but what does it have to do with mining variance? A big miner (or a cartel of such miners) can damage network with our without variance.
My concern is prohibiting such a centralization by eliminating pooling pressure here, is it that hard to understand?
Nakamoto's winner-takes-all approach put bitcoin in danger 10 years ago, it took just 2-3 years for this mistake to become the main centralization factor and now you, one of the lead developers in charge of keeping bitcoin safe and decentralized, instead of isolating this factor and contributing to fixing it try to distract us by mixing up everything.
My proposal has zero relation with hashcash. Hashcash is an old proposal for DDoS and spam prevention, 2 decades old. What's the point here? Who says hashcash is low variance? You mean low difficulty?
and people have re-proposed it once or twice a year on BCT since the start. Maybe that if you don't bother researching you'll just rehash old and broken ideas ...
So, your argument is because I advise lowering the difficulty (somehow) to reduce and control the variance, I'm rehashing old hashcash story, again and again? Is it how you
read?
I'm sure that in the depths of your hearts, guys like you, are confident that nobody is capable of suggesting any thing new about bitcoin because every possibility has been discussed already and noobs (like me) are just rehashing old stories. Right?
It is truly dangerous, and it is why I think you guys are no longer qualified to take the lead. You may be helpful or not as a developer but you are dangerous as a leader.
bitcoin is a system and as a system it needs evolution and improvement. We should observe and improve and no proposal should be cut-down by claiming it as a
re-hash of a "broken" idea.
I think that is just kind of sad and very frustrating. Even the idea of "fix"ing variance is somewhat confused on its face. Variance is utterly essential to the operation of the system, it _cannot_ converge without variance significantly greater than the communications diameter of the network:
You are right. It is really sad.
First of all, Of course variance can not be
fixed. What the heck? We are talking about mining variance
flaw, about high variance for average miners who are forced to join pools because of it! My concern is fixing this
flaw not the variance which is a mathematical index of how sharp (low variance) or flat (high variance) are distributed probabilities.
Plus, how fool you suppose your readers are?
This is it?
Variance, higher than network diameter(
WTF?), is essential to network to converge? WHY?
Again, your proof by example:
Imagine a toy example with a world wide network of nodes and two miners with equal hashrate, in a system with no variance they'll keep producing blocks at exactly the same time, so once the network forks once it'll never heal-- nodes will just follow the closest miner.
Kidding? It is not a zero variance situation, actually it is a system with a standard deviation of 1. A normal random distribution!.
You have to check the concept more deeply: By mining variance flaw we are discussing the problem of players with small relative hash power that the cardinality and distribution of blocks they mine in a typically long period (which follows a Bernoulli distribution) has a large
standard deviation that translates to high risk of hitting nothing at all (or a lot more than what they deserve). i.e. A high risk, high stake gamble.
Given above definition, what the heck are you talking about?
In spite of pages and pages of discussion no one has until now pointed any of this out to you, because almost no one technically competent will bother even reading your posts because you haven't bothered learning what you don't know and you spend a lot of energy making really nasty and unprofessional insults-- I only read any of it because your seemingly
incorrect claims of "vulnerabilities" caught the attention of journalists and I was asked if you were full of it or not.
So, this is it! You are proving to some journalist that I'm not competent!
We have trolls, scammers, shills, who instead of contributing in discussions and adding value, deliberately choose the other option.
Now we have a new phenomenon,
a guardian troll, who tries to prove everything is ok, bitcoin is great, no vulnerabilities, no flaws, no need to improve and these guys in cheap forums like btctalk are a bunch of noobs who have no clue and journalists can rest assured that everything is ok.
