I guess I have a different view...
I feel it all depends on your relationship with your kids.
We have no grandchildren; just college kids. And, we've never vacationed with our mothers and/or fathers (since we've been married). We use vacation to get away from family. We are both very busy and our mothers live so close, it's nice to have a vacation with just the two of us (or 4 if our kids come). Not that we don't appreciate and love our parents, we cannot spend more than a day with them. I have taken the kids to FL to see grandma, but DH won't even go stay with his own mom. I'll be sad the day comes that our kids would rather not come with us (but as long as we're footing the bill, that won't happen).
Have you vacationed with your son and DIL previously to know how well you get along? If your son and DIL do not live close is this an opportunity to spend some time with them? Being in different units is a plus as well as setting up guidelines before you go.