Handle cancelled selection keys... "properly"?

This seems very needlessly complex for something which shouldn't even
have to be an issue, but I can't obviously see how else to handle it,
but also not really what the whole point of selectors "cancelled-sets"
is.