John,
It may not be as bleak as some are making out. I live just south of Salisbury (about 25 mins north of Southampton) and manage to get underground fairly frequently.
I travel up to Mendip regularly being a member of the Mendip Caving Group, and we have an ongoing project exploring 18th Century lead mines on Sandford Hill - you would be more than welcome to join in (it takes me between 1 amd 1.5 hours to get to the MCG cottage depending on traffic).
On Purbeck there are some easily accessible coastal stone mines at places such as Seacombe and Winspit, which can be combined with superb walking and the excellent Square and Compass pub at Worth Matravers. There are inland Quarrs as well, I have not been in any but I understand some are quite extensive.
As for Portland the caves there are a bit of an acquired taste (I was brought up in Weymouth so know the area well), but there are other underground features including stone mines and military structures if that sort of thing floats your boat.
There are also a couple of Karst areas in the Sallisbury area (one in Limestone the other in Chalk). Though no significant cave is known there is the potential. There are also the stone mines around Chilmark that have been mentioned, underground access is not possible but might be worth a surface wander.
I did have an unsuccessful look for the Oil Shale adits at Kimeridge many years ago. The only one I could find comprised some railway lines protruding from the cliff about 30ft up - given that the cliff is clay there did not appear to be any way of getting up to it. Finally there were Ball Clay mines at Norden near Corfe Castle, and there is now a museum there. I have yet to visit but it is free.