Sadly, and mathematically, yes. However, you can look at it as a 'maintenance' three days where you eat your maintenance calories, and so won't loose anything over that time. (Does that make any sense?)
Maybe you should do a weekly deficit, that way you can have treats like your big breakfast but just cut back the next few days.
For example if you aim to eat 2000 calories a day, (that's including you're deficit of 500 calories) then that's 14,000 calories a week you allow yourself. So instead of making it the same allowance of calories per day, you could have say, 1300 on monday, 1200 on tuesday, 2100 on Saturday, 1800 on Sunday etc - aslong as you don't go over 14000 calories over the whole week. Worth a try!