This is similar to adaptive block sizing, which applies a penalty for generating blocks over the median size. The cost of the penalty is supposed to be mitigated by the value of the fees. My opinion is that this will not work well.

Such a mechanism doesn't work for this reason: miners generate their own blocks and include whatever tx they choose too, however the smaller the block, the less likely it is to ever be orphaned. If there was an overlay PoS network (like for MC2) you could try and enforce tx being included into blocks even at some level of reward penalty, but naturally miners will just include as many high fee transactions as possible without destroying any of their reward. Such a method of securing a blockchain is highly experimental and may not work for other incentives reasons (that's for the MC2 thread). As PoW stands now, miners are free to manipulate the transaction volume/fees volume in the network as they desire, and so subsidy penalties will not really work.