I asked a friend who is a military history expert who lives in Atlanta (a mile from me) about Civil War sites and he wrote back:
Chattanooga is good but the Cyclorama (at the Zoo) and the Historical Society (West Paces Ferry Road) are much closer. In Columbus, there is the Confederate Naval Museum and the revamped US Army Infantry Museum. At Big Shanty (near Marietta), there is a museum commemorating the Locomotive Chase (it began outside the museum with the snatch of the locomotive by Union commandos).
****
One of the cool things about Atlanta and environs is that practically every place you go is a Civil War site. I see plaques all over the place commemorating, for instance, a battle that took place right where you're now waiting in traffic for the light to change to green. 
