Most non-developed caves, described on, are very easy to visit. Mostly they are known for many (maybe many hundreds) of years. In many cases you can just walk through. Nevertheless, you should wear good shoes, maybe old and warm clothes and you should have at least two lights per person! And whatever you do: use your brain!

Wear good shoes. It depends on the cave, but you can use walking boot or rubber boots.
Wear warm clothes, because its rather cold in caves. Wear old clothes, because they will become dirty, and very often they will be cave coloured for the the rest of their existance.
You should alway have enough light, at least two (different) lamps per person.
  • Do not use candles, they are not safe enough!
  • Do not use torches, they will destroy the cave!
  • Do not use camping gas lamps, they are much too fragile!
Use electric or carbide lamps. If you use electric lamps, be aware that the batteries will discharge very fast, because of the high humidity and the low temperature. They will last between 60% and 80% the time they would last outside. Always prefer water resistant lamps!
It is always a good idea to wear a helmet inside caves. You will know why, when you hit your head for the first time....
Never cave alone. Always include experienced cavers in the party. The easiest way to do this is joining a caving club or grotto.
Ask local experts for advice and act on it. Pick a cave or pothole within everyone's capabilities. Leave a note of your trip and latest time of return with a responsible person, and check in on exit.
The weather is essential for river caves (caves with a river inside). Take notice of the local weather forecast and ground conditions of the previous few days. Flooding can happen very quickly and violently.
First Aid Kit:
It is always good to take a first aid kit with you!
And most important:
Exit takes more effort than entry - plan with the return in mind.