Was it Joel Madden that broke up with Hilary duff or the other way around?

like good charlotte (Joel's the singer of it) has songs about their break up and how he loved her but she didnt respect him but then when Hilary found out Nicole Ritchie was pregnant she called Nicole "skank"