It is possible to add an IP into the CT exclusions list by prepending a *, as in: *

Possibly better: if you know the subject name that the SSL certificate uses, you can add that to the excluded list. Even when you connect with an IP, when CT gets the certificate and sees the name, it will match and then set the connection to excluded.