Bashing around with a hammer is fine... In the recent rains, millions of tons of rock was washed off the northern Fells of the lake district. There is a 'ban' on mineral collecting in this area. In one rainstorm, more 'damage' was done to the outcrop geology than the entire population of the world could achieve with rock hammers in 100 years!
The whole argument is silly - it comes down to jealousy (I've got a better bit than you) and money (how come you are selling rocks on ebay and making money - I don't so why should you?)...
There is certainly an argument for preserving some minerals in- situ. Places like the blue john mines preserve veins of the stuff for people to see. Other mines around the world that are commercial ventures fore tourism preserve cavities. It would be nice to see SOME cavities preserved in other mines such as the Nenthead area. I had one hidden for many years in Yorkshire that I used to take people down and show. It was huge, full of stunning specimens. I had a couple of bits from it, and it gave me a lot of pleasure to show people. One greedy collector tried to rape the lot. Such a pity - the cavity was worth more to look at than the specimens - pieces from there have never fetched much money.
This argument will always rage - mine explorers have an automatic accusation that all damage in mines is caused by collectors - it isn't. Most of the stacked deads in Smallclough flats for example, were collapsed by collectors, according to mine explorers. They have no evidence of this - they just say it. No collector is going to collapse stacked deads - you don't find good specimens in stacked waste rock - you find them in big cavities! I've SEEN mine explorers - local kids, armed with nothing but hand torches, scrambling over deads in Smallclough and sending them everywhere.
To summarise Ian's example - being able to collect most probably stimulated that person's interest and turned it into a career. It certainly did for me, and for a number of other people I know. That has to be a good thing. As is the look of wonderment on a kids face when they find something pretty - it sparks a lifelong interest in geology and the world around them - keeps them off a computer and silly war games.
The Meek Shall Inherit The Earth
... but not the Mineral Rights...