It's just not worth the payback

I went shopping with my friend on Friday.  I shouldn't have. I should have remembered that this is the same woman who takes her 2 year old with her when she goes shopping and doesn't get home until after 10:00. We exist on different schedules and I should have remembered that before leaving at 2:00pm, but I didn't. I got home at 9:00pm.
Tom went golfing on Saturday morning.  He always golfs one weekend morning and then I get to sleep in the other morning. He told me that he had to make the painful decision to actually turn down golf (insert mock pain here) when some guy was looking to set up a game on Sunday and Tom had to say no, his wife only lets him golf once a week. I made the comment that I'd almost be willing to let him go again if he gave me some time to work on my quilt in return. Part of the joke was a sincere desire for free time to sew, and part was me just not liking the whole "My wife won't let me" comment.  Either way, we went back to watching TV.
Tommy went down for his nap at the usual time and I started working on something here at the desk, in the living room.  I printed out Ryan's name in huge font, cut the letters out, and was pinning them to fabric, to iron and sew as appliques, when Tom told me to just go ahead and go back to the bedroom (where my sewing machine is) and do it without the distraction of Danny reaching for the scissors and pins.  I probably got 2 hours to do stuff, with the boys banging on the bedroom door, before Danny started crying for me and I gave up and came out.
So last night, at 11:00pm right before bed, he says, "So since I gave you time for your quilt, I get to play golf in the morning again, right?"  I got 2 hours of listening to my children cry and bang on the door while their father ignored them and for that I have to give up my only day to sleep in all week?  But again, I'm not his mother and I kind of resent the implication that I have to "let him" do things.  I just told him to do whatever, that if he was gone when the boys woke up in the morning that I'd get up with them.  So sure enough, he left before 7:00 am, and the boys woke up not long after.
I'm not going shopping any more. Or working on quilts. Or doing just about anything else that's fun and just for me.  It's not worth the price.  Anything I get, or get to do, comes back to bite me in the ass because I have to pay for it, later, at a cost I never agreed to in the first place.  Tom has his golf, and his wood working, and his 12 hours a week of televised football all fall and winter, and I'd just like to be able to leave the house without screaming children, and piece together old t shirts and sheets for fun (can't afford to buy fabric), and it turns out that a hobby is too much to ask for.