gnaw away at

to slowly and steadily destroy something or make someone feel worried over a long time.