If one of the divorced parents has sole custody, all you need is what's called a "Sole Custody" document. If you don't have that, you will need a notarized permission form from the other parent to take the child out of the country. It would be easier to get the document so that you don't have to get permission every time you travel internationally.
Even if customs doesn't consistently ask for either of these documents, you can bet that the time you don't have one, they will ask for one! Don't risk it!