Yes, that appears to be the case, and I'm tempted to call that a bug, as it's quirky behavior. Note that clicking the Run button to test it is another way to make it go live, which is probably why it's not a common issue (since you typically want to test your change).
Remi what do you think?