It might not be working for you, because you have the boobs and he knows quite rightly that you have a better supply available than that stuff you are trying to give him. Babies know this stuff.
That isnt to say that when there isnt you as an alternative, he wont cope perfectly fine.
Feed him as close as possible to the moment you have to leave, and leave a bottle of expressed in the fridge for the babysitter to give. He may sleep through you being out, he may wake up and submit to the bottle because you arent there, but while you ARE there, he knows exactly what he wants and you arent giving him it, which is why he is refusing.
Please dont panic. He will come to no harm while you are out. The problem is now that you wont relax and enjoy yourself because you will be constantly worried about how he is doing. This is perfectly natural and normal and will get easier the more often you do it.
Apart from anything else, you and your OH need some couple time. To take a tiny break from being parents and just enjoy each others company again. This is a rarity when they are so little, but in all seriousness, you should really take advantage of those moments and put yourself and OH back in the "number 1 priority seat" for a couple of hours.
NOTHING terrible will happen in the time you are away. Your babysitter will cope, and the baby will cope too. You are the one least likely to, so take a deep breath, and relax, and go and enjoy yourself as much as you can.