grow away from

to slowly stop being close to a person or an interest as you get older or change.