Unselecting on the Magic Wand Tool

Would be pretty nice and time saving at complex shapes to have an unselecting feature on the magic wand aswell, just like the one on the "Selection Tool", where you can simply Shift+RMB to unselect a selected area. :)