Now that you mention it, it would indeed make more sense to have the selection in the colour chooser window.
If you could make the movable window save it's location between calls, placing it in the middle of the screen would actually be quicker than having to move the mouse all the way to the toolbar.
A few minor nitpicks while we're on the subject. With the "red value" text box in focus when you open the colour window, it's not possible to use the key shortcut (X) to swap the primary/secondary colours. Not that big of a deal right now, but if you do implement the shading palette in there, it would be useful to be able to switch easily when this window is in focus.
Also, when using Tab or shift+Tab to jump between the different colour properties (RBG & HSL), it would be great to have the next value already highlighted to be able to enter a new value right away. Ctrl+A works, but it's not as straightforward.
Don't worry, I don't have anything else to criticize ;)
Pyxel edit is fantastic. Thanks!