If you are exercising regularly, I wouldn’t expect a huge change if you skipped one day. But for periods of no exercise of a week or so, this doesn’t seem too unexpected.
Not only does exercise increase your carb requirements and insulin sensitivity, but it also increases the the level of GLUT4 expression. After a week of no exercise, this can change dramatically.