Shut down the panel first,
xfce4-panel --quit
Kill the xfce4 configuration daemon,
pkill xfconfd
First delete settings for the panel,
rm -rf ~/.config/xfce4/panel
Clear out the settings for xfconfd ,
rm -rf ~/.config/xfce4/xfconf/xfce-perchannel-xml/xfce4-panel.xml
Restart the panel, run
xfce4-panel