There should be a path (prob overgrown) which takes you the the centre of the Lane End site (includes remains of the Beam Engine House and later colliery building). The site on the northern end is very likely to be overgrown at this time of year, you can get a few across the fields from the back of the site.
Or if you know who's land the viaduct is on, you can always ask.
The tunnel (or gated entrance) is next to where the public footpath crosses the lane to the west of the prison site.
I went in a mine once.... it was dark and scary..... full of weirdos
When do I get my soapbox, I need to rant on about some b***cks