It could be argued the interest is a financial one stemming from a social connection. Were Lindsay Graham to be in possession of information that might implicate one of the Thomas’, and was compelled to share it, then such a sharing may hit or help the business interests of the Thomas’ family.
It may be that Graham has nothing to share - but if that’s the case, then self-recusal might be prudent. Should it later transpire that business interests were involved, justice Thomas is leaving himself open to accusations of obstructing the course of justice (or whatever the American equivalent is).