Hi,
When you unplug the power to the IC the internal FET should be off. There will be the possibility of a diode drop between the output and the input so reverse biasing the output to the input is not recommended. This may be happening when you are connecting the switched-off AC/DC charger. Please note also the apps diagram in the datasheet which shows external switch control for USB and AC/DC charger operation.
G Roxburgh.