{"id":4399,"date":"2021-06-27T22:48:08","date_gmt":"2021-06-27T22:48:08","guid":{"rendered":"https:\/\/dalelane.co.uk\/blog\/?p=4399"},"modified":"2021-06-27T22:48:09","modified_gmt":"2021-06-27T22:48:09","slug":"how-to-avoid-ssl-handshake-errors-in-your-kafka-client-because-of-a-self-signed-cluster-ca","status":"publish","type":"post","link":"https:\/\/dalelane.co.uk\/blog\/?p=4399","title":{"rendered":"How to avoid SSL handshake errors in your Kafka client because of a self-signed cluster CA"},"content":{"rendered":"

You’re trying to connect a Kafka client to a development Apache Kafka cluster which has been quickly set up using a self-signed CA certificate. You don’t have a copy of that CA certificate, and (because it’s not signed by a well-known CA) your Kafka client is failing because of SSL handshake errors.<\/strong><\/p>\n

The error contains messages like
\norg.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed<\/code>
\nand
\njavax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target<\/code>.<\/p>\n

For example:<\/p>\n

$ .\/bin\/kafka-console-consumer.sh \\\n --bootstrap-server dale-kafka-saslscram-bootstrap-strimzi.apps.eem-test-fest-6.cp.fyre.ibm.com:443 \\\n --topic DALE.TOPIC \\\n --group dalegrp \\\n --consumer-property 'security.protocol=SASL_SSL' \\\n --consumer-property 'sasl.mechanism=SCRAM-SHA-512' \\\n --consumer-property 'sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username=\"dale-user\" password=\"pSRtfwTMKNlz\";'\n\n[2021-06-27 23:19:06,048] ERROR [Consumer clientId=consumer-dalegrp-1, groupId=dalegrp] Connection to node -1 (dale-kafka-saslscram-bootstrap-strimzi.apps.eem-test-fest-6.cp.fyre.ibm.com\/9.46.199.58:443) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)\n[2021-06-27 23:19:06,049] WARN [Consumer clientId=consumer-dalegrp-1, groupId=dalegrp] Bootstrap broker dale-kafka-saslscram-bootstrap-strimzi.apps.eem-test-fest-6.cp.fyre.ibm.com:443 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)\n[2021-06-27 23:19:06,069] ERROR Error processing message, terminating consumer process:  (kafka.tools.ConsoleConsumer$)\norg.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed\nCaused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target\n\tat java.base\/sun.security.ssl.Alert.createSSLException(Alert.java:131)\n\tat java.base\/sun.security.ssl.TransportContext.fatal(TransportContext.java:326)\n\tat java.base\/sun.security.ssl.TransportContext.fatal(TransportContext.java:269)\n\tat java.base\/sun.security.ssl.TransportContext.fatal(TransportContext.java:264)\n\tat java.base\/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1339)\n\tat java.base\/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1214)\n\tat java.base\/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1157)\n\tat java.base\/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)\n\tat java.base\/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444)\n\tat java.base\/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1074)\n\tat java.base\/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061)\n\tat java.base\/java.security.AccessController.doPrivileged(AccessController.java:770)\n\tat java.base\/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1008)\n\tat org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:430)\n\tat org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:514)\n\tat org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:368)\n\tat org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:291)\n\tat org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:173)\n\tat org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543)\n\tat org.apache.kafka.common.network.Selector.poll(Selector.java:481)\n\tat org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:561)\n\tat org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:265)\n\tat org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236)\n\tat org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:215)\n\tat org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:244)\n\tat org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:480)\n\tat org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1257)\n\tat org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1226)\n\tat org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1206)\n\tat kafka.tools.ConsoleConsumer$ConsumerWrapper.receive(ConsoleConsumer.scala:444)\n\tat kafka.tools.ConsoleConsumer$.process(ConsoleConsumer.scala:103)\n\tat kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:77)\n\tat kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:54)\n\tat kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala)\nCaused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target\n\tat java.base\/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439)\n\tat java.base\/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306)\n\tat java.base\/sun.security.validator.Validator.validate(Validator.java:264)\n\tat java.base\/sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:313)\n\tat java.base\/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:276)\n\tat java.base\/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:141)\n\tat java.base\/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1317)\n\t... 29 more\nCaused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target\n\tat java.base\/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)\n\tat java.base\/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)\n\tat java.base\/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)\n\tat java.base\/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434)\n\t... 35 more\nProcessed a total of 0 messages<\/pre>\n
I’m assuming that this is just for development purposes, that you know it’s safe to trust the certificate that the Kafka cluster is presenting, and that you’d rather just workaround the error than ask the owner of the Kafka cluster for a copy of their CA.<\/p>\n
If all that is true, here is how you can quickly workaround this error.<\/p>\n
Step 1 – get a copy of the certificate that the Kafka cluster is presenting<\/strong><\/p>\n
$ openssl s_client \\\n  -connect YOUR-KAFKA-CLUSTER-BOOTSTRAP-ADDRESS:PORT \\\n  -servername YOUR-KAFKA-CLUSTER-BOOTSTRAP-ADDRESS<\/pre>\n
For example:<\/p>\n
$ openssl s_client \\\n  -connect dale-kafka-saslscram-bootstrap-strimzi.apps.eem-test-fest-6.cp.fyre.ibm.com:443 \\\n  -servername dale-kafka-saslscram-bootstrap-strimzi.apps.eem-test-fest-6.cp.fyre.ibm.com\n\nCONNECTED(00000005)\ndepth=1 O = io.strimzi, CN = cluster-ca v0\nverify error:num=19:self signed certificate in certificate chain\nverify return:0\n---\nCertificate chain\n 0 s:\/O=io.strimzi\/CN=dale-kafka\n   i:\/O=io.strimzi\/CN=cluster-ca v0\n 1 s:\/O=io.strimzi\/CN=cluster-ca v0\n   i:\/O=io.strimzi\/CN=cluster-ca v0\n---\nServer certificate\n-----BEGIN CERTIFICATE-----\nMIIE7zCCA9egAwIBAgIJAMakmFsl67WbMA0GCSqGSIb3DQEBCwUAMC0xEzARBgNV\nBAoMCmlvLnN0cmltemkxFjAUBgNVBAMMDWNsdXN0ZXItY2EgdjAwHhcNMjEwNjI3\nMjIwNDEwWhcNMjIwNjI3MjIwNDEwWjAqMRMwEQYDVQQKDAppby5zdHJpbXppMRMw\nEQYDVQQDDApkYWxlLWthZmthMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKC\nAQEApeiV0pelGA3TBVYcRMR3bGwbNeHt8tvRErblZapzuDtDLSjWwZpIh26hpMWC\nx4BNGmiT44IohnVqB87DADjpLlEs960sxgTdFcIn5OQAUsUODPHiXjtPL3nVyT4p\n4WqsElHEUFnNTt0WdlYxN0c73eyUMA3hb87ayLKP4TVyGLJqYsWy8IEdv42UUv51\n2\/L7t44z0fva39HsVUAOCCxTmb+CyCrC0suXeVuDfqGpHPXqXVHfg01f2l1Z85au\nl8BZUwyWmajTKSobwVEoPkw4oAR61HA5EUTYX4s0yWalN0O+k4GX9ck70vuj9Hi4\nbMHGZm5Spo+WdsNk4T68c7MzLwIDAQABo4ICEzCCAg8wggILBgNVHREEggICMIIB\n\/oIeZGFsZS1rYWZrYS1icm9rZXJzLnN0cmltemkuc3ZjgixkYWxlLWthZmthLWJy\nb2tlcnMuc3RyaW16aS5zdmMuY2x1c3Rlci5sb2NhbIISZGFsZS1rYWZrYS1icm9r\nZXJzghpkYWxlLWthZmthLWJyb2tlcnMuc3RyaW16aYI5ZGFsZS1rYWZrYS0wLmRh\nbGUta2Fma2EtYnJva2Vycy5zdHJpbXppLnN2Yy5jbHVzdGVyLmxvY2FsgiBkYWxl\nLWthZmthLWJvb3RzdHJhcC5zdHJpbXppLnN2Y4IuZGFsZS1rYWZrYS1ib290c3Ry\nYXAuc3RyaW16aS5zdmMuY2x1c3Rlci5sb2NhbIJDZGFsZS1rYWZrYS1zYXNsc2Ny\nYW0tMC1zdHJpbXppLmFwcHMuZWVtLXRlc3QtZmVzdC02LmNwLmZ5cmUuaWJtLmNv\nbYIUZGFsZS1rYWZrYS1ib290c3RyYXCCS2RhbGUta2Fma2Etc2FzbHNjcmFtLWJv\nb3RzdHJhcC1zdHJpbXppLmFwcHMuZWVtLXRlc3QtZmVzdC02LmNwLmZ5cmUuaWJt\nLmNvbYIcZGFsZS1rYWZrYS1ib290c3RyYXAuc3RyaW16aYIrZGFsZS1rYWZrYS0w\nLmRhbGUta2Fma2EtYnJva2Vycy5zdHJpbXppLnN2YzANBgkqhkiG9w0BAQsFAAOC\nAQEAsOZZZ1+caVBj9qy6qg\/TNdGaym8VOmQny0qe2hRoVAC0ljQvoKUoC0HusO25\nWVerYTxLroknH88NgwsugcKdLgTCd8hhG+kB6XWpGtU2uo7P4wh96yZi3eUkyxrj\nCnIJkKZvkRkstqt62iGaivamAsCLlU+XNxv9l+RQ++cy76JIuWMTenED4ZdZgcuq\nLooPbukLBvDD3Iw41fG0wMEJRfn8H7HPamBiF5fezQQZUeiYLf7PBe7rticY1ST\/\nVGm\/uimPqWtUig3MyJE9XawsRA\/vKF4PcV9MH6wPvBufAVpo6tt0+EZeecSzHWET\n9Dl+4TT0eBZNVZEf\/PXohChyUw==\n-----END CERTIFICATE-----\nsubject=\/O=io.strimzi\/CN=dale-kafka\nissuer=\/O=io.strimzi\/CN=cluster-ca v0\n---\nNo client certificate CA names sent\nServer Temp Key: ECDH, X25519, 253 bits\n---\nSSL handshake has read 2538 bytes and written 373 bytes\n---\nNew, TLSv1\/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384\nServer public key is 2048 bit\nSecure Renegotiation IS supported\nCompression: NONE\nExpansion: NONE\nNo ALPN negotiated\nSSL-Session:\n    Protocol  : TLSv1.2\n    Cipher    : ECDHE-RSA-AES256-GCM-SHA384\n    Session-ID: 91C40A5999CD12938B5C199321C925750261D9212976FD95C813F08DD05BE166\n    Session-ID-ctx:\n    Master-Key: 0F2D6AE8BBA0C7E88A5805BB884415D7906315E6D6B12E29643C455EBD16ABA2C4A6F52A2A7D999E0998EABFE79AC67F\n    Start Time: 1624832427\n    Timeout   : 7200 (sec)\n    Verify return code: 19 (self signed certificate in certificate chain)\n---<\/pre>\n
Get a copy of the certificate from that output (between the BEGIN CERTIFICATE<\/code> and END CERTIFICATE<\/code> lines) and put it in a file called ca.pem<\/code>.<\/p>\n
For example:<\/p>\n
$ cat ca.pem\n-----BEGIN CERTIFICATE-----\nMIIE7zCCA9egAwIBAgIJAMakmFsl67WbMA0GCSqGSIb3DQEBCwUAMC0xEzARBgNV\nBAoMCmlvLnN0cmltemkxFjAUBgNVBAMMDWNsdXN0ZXItY2EgdjAwHhcNMjEwNjI3\nMjIwNDEwWhcNMjIwNjI3MjIwNDEwWjAqMRMwEQYDVQQKDAppby5zdHJpbXppMRMw\nEQYDVQQDDApkYWxlLWthZmthMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKC\nAQEApeiV0pelGA3TBVYcRMR3bGwbNeHt8tvRErblZapzuDtDLSjWwZpIh26hpMWC\nx4BNGmiT44IohnVqB87DADjpLlEs960sxgTdFcIn5OQAUsUODPHiXjtPL3nVyT4p\n4WqsElHEUFnNTt0WdlYxN0c73eyUMA3hb87ayLKP4TVyGLJqYsWy8IEdv42UUv51\n2\/L7t44z0fva39HsVUAOCCxTmb+CyCrC0suXeVuDfqGpHPXqXVHfg01f2l1Z85au\nl8BZUwyWmajTKSobwVEoPkw4oAR61HA5EUTYX4s0yWalN0O+k4GX9ck70vuj9Hi4\nbMHGZm5Spo+WdsNk4T68c7MzLwIDAQABo4ICEzCCAg8wggILBgNVHREEggICMIIB\n\/oIeZGFsZS1rYWZrYS1icm9rZXJzLnN0cmltemkuc3ZjgixkYWxlLWthZmthLWJy\nb2tlcnMuc3RyaW16aS5zdmMuY2x1c3Rlci5sb2NhbIISZGFsZS1rYWZrYS1icm9r\nZXJzghpkYWxlLWthZmthLWJyb2tlcnMuc3RyaW16aYI5ZGFsZS1rYWZrYS0wLmRh\nbGUta2Fma2EtYnJva2Vycy5zdHJpbXppLnN2Yy5jbHVzdGVyLmxvY2FsgiBkYWxl\nLWthZmthLWJvb3RzdHJhcC5zdHJpbXppLnN2Y4IuZGFsZS1rYWZrYS1ib290c3Ry\nYXAuc3RyaW16aS5zdmMuY2x1c3Rlci5sb2NhbIJDZGFsZS1rYWZrYS1zYXNsc2Ny\nYW0tMC1zdHJpbXppLmFwcHMuZWVtLXRlc3QtZmVzdC02LmNwLmZ5cmUuaWJtLmNv\nbYIUZGFsZS1rYWZrYS1ib290c3RyYXCCS2RhbGUta2Fma2Etc2FzbHNjcmFtLWJv\nb3RzdHJhcC1zdHJpbXppLmFwcHMuZWVtLXRlc3QtZmVzdC02LmNwLmZ5cmUuaWJt\nLmNvbYIcZGFsZS1rYWZrYS1ib290c3RyYXAuc3RyaW16aYIrZGFsZS1rYWZrYS0w\nLmRhbGUta2Fma2EtYnJva2Vycy5zdHJpbXppLnN2YzANBgkqhkiG9w0BAQsFAAOC\nAQEAsOZZZ1+caVBj9qy6qg\/TNdGaym8VOmQny0qe2hRoVAC0ljQvoKUoC0HusO25\nWVerYTxLroknH88NgwsugcKdLgTCd8hhG+kB6XWpGtU2uo7P4wh96yZi3eUkyxrj\nCnIJkKZvkRkstqt62iGaivamAsCLlU+XNxv9l+RQ++cy76JIuWMTenED4ZdZgcuq\nLooPbukLBvDD3Iw41fG0wMEJRfn8H7HPamBiF5fezQQZUeiYLf7PBe7rticY1ST\/\nVGm\/uimPqWtUig3MyJE9XawsRA\/vKF4PcV9MH6wPvBufAVpo6tt0+EZeecSzHWET\n9Dl+4TT0eBZNVZEf\/PXohChyUw==\n-----END CERTIFICATE-----<\/pre>\n
Step 2 – put that in a new truststore<\/strong><\/p>\n
$ openssl x509 -outform der -in ca.pem -out ca.der\n\n$ keytool -import -noprompt \\\n\t-alias ca \\\n\t-file ca.der \\\n\t-keystore ca.p12 -storepass STOREPASSW0RD<\/pre>\n
Step 3 – use your new truststore with your Kafka client<\/strong><\/p>\n
For example:<\/p>\n
.\/bin\/kafka-console-consumer.sh \\\n --bootstrap-server dale-kafka-saslscram-bootstrap-strimzi.apps.eem-test-fest-6.cp.fyre.ibm.com:443 \\\n --topic DALE.TOPIC \\\n --group dalegrp \\\n --consumer-property 'security.protocol=SASL_SSL' \\\n --consumer-property 'sasl.mechanism=SCRAM-SHA-512' \\\n --consumer-property 'sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username=\"dale-user\" password=\"pSRtfwTMKNlz\";' \\\n --consumer-property 'ssl.truststore.location=ca.p12' \\\n --consumer-property 'ssl.truststore.type=PKCS12' \\\n --consumer-property 'ssl.truststore.password=STOREPASSW0RD'<\/pre>\n","protected":false},"excerpt":{"rendered":"
You’re trying to connect a Kafka client to a development Apache Kafka cluster which has been quickly set up using a self-signed CA certificate. You don’t have a copy of that CA certificate, and (because it’s not signed by a well-known CA) your Kafka client is failing because of SSL handshake errors. The error contains […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[7],"tags":[593,584],"class_list":["post-4399","post","type-post","status-publish","format-standard","hentry","category-code","tag-apachekafka","tag-kafka"],"_links":{"self":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts\/4399","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=4399"}],"version-history":[{"count":0,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=\/wp\/v2\/posts\/4399\/revisions"}],"wp:attachment":[{"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=4399"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=4399"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dalelane.co.uk\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=4399"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}