go down to

to travel to a place, especially one that is further south or in a smaller town.