It depends; the mother is incarcerated, it is very unlikely she can retain custody even of a child who is 16 or 17 years old. However, there are several possibilities. In most states, the other parent, the father, is the default custodian when the custodial parent is unable to play that role. But if DCS or other agencies have investigated the father in the past, the court may look to a third party guardian for the child's care. Furthermore, if the child is old enough to seek emancipation from his parents, the court may honor his wishes, especially when the custodial parent is incarcerated.
The good news for the father is that of all the parties who come before the court, he is the only person with rights as a parent. These rights must be given precedence over the foster care, grandparents, or even emancipation. Thus a good father with a stable home, should have no problem in these circumstances. A man who has a substantial criminal history himself, on the other hand, is not likely to be granted custody. In short, every situation is different.