OF tries to open a TLS connection and ask for SASL-features. Two sasl features EXTERNAL and DIALBACK could be handled.
If the certificates are self-signed only DIALBACK is used. But if something is wrong with the certificates the whole TLS-stuff is skipped and OF retries with classic dialback over an unencrypted connection.
If you set 'xmpp.server.certificate.verify' or 'xmpp.server.certificate.verify.chain' to false the whole chain is not validated and no exception is raised. In this case I could create an full-encrypted S2S between my both OF servers.
I read two documents to get involved whats happend:
http://xmpp.org/extensions/xep-0178.html #3 S2S
http://xmpp.org/extensions/xep-0220.html
Yesterday I tried ejabberd and the response is a bit strange: There is no offered feature for SAML authorization. But I'm not sure that the server is conigured very well. (See the first document example 25 - this is empty)
This cause an exception too and the unencrypted fallback is used as well.
When I look into your logs your reportet issue is different. I will search in the next days for a different ejabberd-server and try it again.