The brain likes to be efficient and so even as its strengthening the pathways you're exercising, it's pulling - it's weakening the connections in other ways between the cells that supported old ways of thinking or working or behaving, or whatever that you're not exercising so much.
