If you're arriving into Dunedin International Airport, there are regular shuttles and buses that will take you the 19 miles (30 km) into town. You can also take a taxi, as well as rent a car.
There are daily buses to and from Christchurch, Invercargill, Wanaka, and Queenstown, including overnight buses. While the train station is in the heart of the city, it is limited to the Gorge Railway, and thus may not be the best choice for regular transit.
Once you're in the city, the Dunedin Bus service is reliable and well-priced, and will get you around to see the city's main attractions. There is also a bus line that goes out to the peninsula, but be warned that they do not go to the tourist locations such as the Royal Albatross Centre. Most buses are handicap-friendly.
You can also rent bicycles to get around town, as the city center is fairly flat, despite the extremely steep hills. Taking a bike is great way to see the Otago Peninsula via Harington Point Road.