You’re trying to connect a Kafka client to a development Apache Kafka cluster which has been quickly set up using a self-signed CA certificate. You don’t have a copy of that CA certificate, and (because it’s not signed by a well-known CA) your Kafka client is failing because of SSL handshake errors.
The error contains messages like
org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
and
javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target.
For example:
$ ./bin/kafka-console-consumer.sh \ --bootstrap-server dale-kafka-saslscram-bootstrap-strimzi.apps.eem-test-fest-6.cp.fyre.ibm.com:443 \ --topic DALE.TOPIC \ --group dalegrp \ --consumer-property 'security.protocol=SASL_SSL' \ --consumer-property 'sasl.mechanism=SCRAM-SHA-512' \ --consumer-property 'sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="dale-user" password="pSRtfwTMKNlz";' [2021-06-27 23:19:06,048] ERROR [Consumer clientId=consumer-dalegrp-1, groupId=dalegrp] Connection to node -1 (dale-kafka-saslscram-bootstrap-strimzi.apps.eem-test-fest-6.cp.fyre.ibm.com/9.46.199.58:443) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient) [2021-06-27 23:19:06,049] WARN [Consumer clientId=consumer-dalegrp-1, groupId=dalegrp] Bootstrap broker dale-kafka-saslscram-bootstrap-strimzi.apps.eem-test-fest-6.cp.fyre.ibm.com:443 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient) [2021-06-27 23:19:06,069] ERROR Error processing message, terminating consumer process: (kafka.tools.ConsoleConsumer$) org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:326) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:269) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:264) at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1339) at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1214) at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1157) at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392) at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444) at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1074) at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061) at java.base/java.security.AccessController.doPrivileged(AccessController.java:770) at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1008) at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:430) at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:514) at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:368) at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:291) at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:173) at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543) at org.apache.kafka.common.network.Selector.poll(Selector.java:481) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:561) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:265) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:215) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:244) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:480) at org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1257) at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1226) at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1206) at kafka.tools.ConsoleConsumer$ConsumerWrapper.receive(ConsoleConsumer.scala:444) at kafka.tools.ConsoleConsumer$.process(ConsoleConsumer.scala:103) at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:77) at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:54) at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala) Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439) at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306) at java.base/sun.security.validator.Validator.validate(Validator.java:264) at java.base/sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:313) at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:276) at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:141) at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1317) ... 29 more Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297) at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434) ... 35 more Processed a total of 0 messages
I’m assuming that this is just for development purposes, that you know it’s safe to trust the certificate that the Kafka cluster is presenting, and that you’d rather just workaround the error than ask the owner of the Kafka cluster for a copy of their CA.
If all that is true, here is how you can quickly workaround this error.
Step 1 – get a copy of the certificate that the Kafka cluster is presenting
$ openssl s_client \ -connect YOUR-KAFKA-CLUSTER-BOOTSTRAP-ADDRESS:PORT \ -servername YOUR-KAFKA-CLUSTER-BOOTSTRAP-ADDRESS
For example:
$ openssl s_client \
-connect dale-kafka-saslscram-bootstrap-strimzi.apps.eem-test-fest-6.cp.fyre.ibm.com:443 \
-servername dale-kafka-saslscram-bootstrap-strimzi.apps.eem-test-fest-6.cp.fyre.ibm.com
CONNECTED(00000005)
depth=1 O = io.strimzi, CN = cluster-ca v0
verify error:num=19:self signed certificate in certificate chain
verify return:0
---
Certificate chain
0 s:/O=io.strimzi/CN=dale-kafka
i:/O=io.strimzi/CN=cluster-ca v0
1 s:/O=io.strimzi/CN=cluster-ca v0
i:/O=io.strimzi/CN=cluster-ca v0
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIE7zCCA9egAwIBAgIJAMakmFsl67WbMA0GCSqGSIb3DQEBCwUAMC0xEzARBgNV
BAoMCmlvLnN0cmltemkxFjAUBgNVBAMMDWNsdXN0ZXItY2EgdjAwHhcNMjEwNjI3
MjIwNDEwWhcNMjIwNjI3MjIwNDEwWjAqMRMwEQYDVQQKDAppby5zdHJpbXppMRMw
EQYDVQQDDApkYWxlLWthZmthMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKC
AQEApeiV0pelGA3TBVYcRMR3bGwbNeHt8tvRErblZapzuDtDLSjWwZpIh26hpMWC
x4BNGmiT44IohnVqB87DADjpLlEs960sxgTdFcIn5OQAUsUODPHiXjtPL3nVyT4p
4WqsElHEUFnNTt0WdlYxN0c73eyUMA3hb87ayLKP4TVyGLJqYsWy8IEdv42UUv51
2/L7t44z0fva39HsVUAOCCxTmb+CyCrC0suXeVuDfqGpHPXqXVHfg01f2l1Z85au
l8BZUwyWmajTKSobwVEoPkw4oAR61HA5EUTYX4s0yWalN0O+k4GX9ck70vuj9Hi4
bMHGZm5Spo+WdsNk4T68c7MzLwIDAQABo4ICEzCCAg8wggILBgNVHREEggICMIIB
/oIeZGFsZS1rYWZrYS1icm9rZXJzLnN0cmltemkuc3ZjgixkYWxlLWthZmthLWJy
b2tlcnMuc3RyaW16aS5zdmMuY2x1c3Rlci5sb2NhbIISZGFsZS1rYWZrYS1icm9r
ZXJzghpkYWxlLWthZmthLWJyb2tlcnMuc3RyaW16aYI5ZGFsZS1rYWZrYS0wLmRh
bGUta2Fma2EtYnJva2Vycy5zdHJpbXppLnN2Yy5jbHVzdGVyLmxvY2FsgiBkYWxl
LWthZmthLWJvb3RzdHJhcC5zdHJpbXppLnN2Y4IuZGFsZS1rYWZrYS1ib290c3Ry
YXAuc3RyaW16aS5zdmMuY2x1c3Rlci5sb2NhbIJDZGFsZS1rYWZrYS1zYXNsc2Ny
YW0tMC1zdHJpbXppLmFwcHMuZWVtLXRlc3QtZmVzdC02LmNwLmZ5cmUuaWJtLmNv
bYIUZGFsZS1rYWZrYS1ib290c3RyYXCCS2RhbGUta2Fma2Etc2FzbHNjcmFtLWJv
b3RzdHJhcC1zdHJpbXppLmFwcHMuZWVtLXRlc3QtZmVzdC02LmNwLmZ5cmUuaWJt
LmNvbYIcZGFsZS1rYWZrYS1ib290c3RyYXAuc3RyaW16aYIrZGFsZS1rYWZrYS0w
LmRhbGUta2Fma2EtYnJva2Vycy5zdHJpbXppLnN2YzANBgkqhkiG9w0BAQsFAAOC
AQEAsOZZZ1+caVBj9qy6qg/TNdGaym8VOmQny0qe2hRoVAC0ljQvoKUoC0HusO25
WVerYTxLroknH88NgwsugcKdLgTCd8hhG+kB6XWpGtU2uo7P4wh96yZi3eUkyxrj
CnIJkKZvkRkstqt62iGaivamAsCLlU+XNxv9l+RQ++cy76JIuWMTenED4ZdZgcuq
LooPbukLBvDD3Iw41fG0wMEJRfn8H7HPamBiF5fezQQZUeiYLf7PBe7rticY1ST/
VGm/uimPqWtUig3MyJE9XawsRA/vKF4PcV9MH6wPvBufAVpo6tt0+EZeecSzHWET
9Dl+4TT0eBZNVZEf/PXohChyUw==
-----END CERTIFICATE-----
subject=/O=io.strimzi/CN=dale-kafka
issuer=/O=io.strimzi/CN=cluster-ca v0
---
No client certificate CA names sent
Server Temp Key: ECDH, X25519, 253 bits
---
SSL handshake has read 2538 bytes and written 373 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: 91C40A5999CD12938B5C199321C925750261D9212976FD95C813F08DD05BE166
Session-ID-ctx:
Master-Key: 0F2D6AE8BBA0C7E88A5805BB884415D7906315E6D6B12E29643C455EBD16ABA2C4A6F52A2A7D999E0998EABFE79AC67F
Start Time: 1624832427
Timeout : 7200 (sec)
Verify return code: 19 (self signed certificate in certificate chain)
---
Get a copy of the certificate from that output (between the BEGIN CERTIFICATE and END CERTIFICATE lines) and put it in a file called ca.pem.
For example:
$ cat ca.pem -----BEGIN CERTIFICATE----- MIIE7zCCA9egAwIBAgIJAMakmFsl67WbMA0GCSqGSIb3DQEBCwUAMC0xEzARBgNV BAoMCmlvLnN0cmltemkxFjAUBgNVBAMMDWNsdXN0ZXItY2EgdjAwHhcNMjEwNjI3 MjIwNDEwWhcNMjIwNjI3MjIwNDEwWjAqMRMwEQYDVQQKDAppby5zdHJpbXppMRMw EQYDVQQDDApkYWxlLWthZmthMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKC AQEApeiV0pelGA3TBVYcRMR3bGwbNeHt8tvRErblZapzuDtDLSjWwZpIh26hpMWC x4BNGmiT44IohnVqB87DADjpLlEs960sxgTdFcIn5OQAUsUODPHiXjtPL3nVyT4p 4WqsElHEUFnNTt0WdlYxN0c73eyUMA3hb87ayLKP4TVyGLJqYsWy8IEdv42UUv51 2/L7t44z0fva39HsVUAOCCxTmb+CyCrC0suXeVuDfqGpHPXqXVHfg01f2l1Z85au l8BZUwyWmajTKSobwVEoPkw4oAR61HA5EUTYX4s0yWalN0O+k4GX9ck70vuj9Hi4 bMHGZm5Spo+WdsNk4T68c7MzLwIDAQABo4ICEzCCAg8wggILBgNVHREEggICMIIB /oIeZGFsZS1rYWZrYS1icm9rZXJzLnN0cmltemkuc3ZjgixkYWxlLWthZmthLWJy b2tlcnMuc3RyaW16aS5zdmMuY2x1c3Rlci5sb2NhbIISZGFsZS1rYWZrYS1icm9r ZXJzghpkYWxlLWthZmthLWJyb2tlcnMuc3RyaW16aYI5ZGFsZS1rYWZrYS0wLmRh bGUta2Fma2EtYnJva2Vycy5zdHJpbXppLnN2Yy5jbHVzdGVyLmxvY2FsgiBkYWxl LWthZmthLWJvb3RzdHJhcC5zdHJpbXppLnN2Y4IuZGFsZS1rYWZrYS1ib290c3Ry YXAuc3RyaW16aS5zdmMuY2x1c3Rlci5sb2NhbIJDZGFsZS1rYWZrYS1zYXNsc2Ny YW0tMC1zdHJpbXppLmFwcHMuZWVtLXRlc3QtZmVzdC02LmNwLmZ5cmUuaWJtLmNv bYIUZGFsZS1rYWZrYS1ib290c3RyYXCCS2RhbGUta2Fma2Etc2FzbHNjcmFtLWJv b3RzdHJhcC1zdHJpbXppLmFwcHMuZWVtLXRlc3QtZmVzdC02LmNwLmZ5cmUuaWJt LmNvbYIcZGFsZS1rYWZrYS1ib290c3RyYXAuc3RyaW16aYIrZGFsZS1rYWZrYS0w LmRhbGUta2Fma2EtYnJva2Vycy5zdHJpbXppLnN2YzANBgkqhkiG9w0BAQsFAAOC AQEAsOZZZ1+caVBj9qy6qg/TNdGaym8VOmQny0qe2hRoVAC0ljQvoKUoC0HusO25 WVerYTxLroknH88NgwsugcKdLgTCd8hhG+kB6XWpGtU2uo7P4wh96yZi3eUkyxrj CnIJkKZvkRkstqt62iGaivamAsCLlU+XNxv9l+RQ++cy76JIuWMTenED4ZdZgcuq LooPbukLBvDD3Iw41fG0wMEJRfn8H7HPamBiF5fezQQZUeiYLf7PBe7rticY1ST/ VGm/uimPqWtUig3MyJE9XawsRA/vKF4PcV9MH6wPvBufAVpo6tt0+EZeecSzHWET 9Dl+4TT0eBZNVZEf/PXohChyUw== -----END CERTIFICATE-----
Step 2 – put that in a new truststore
$ openssl x509 -outform der -in ca.pem -out ca.der $ keytool -import -noprompt \ -alias ca \ -file ca.der \ -keystore ca.p12 -storepass STOREPASSW0RD
Step 3 – use your new truststore with your Kafka client
For example:
./bin/kafka-console-consumer.sh \ --bootstrap-server dale-kafka-saslscram-bootstrap-strimzi.apps.eem-test-fest-6.cp.fyre.ibm.com:443 \ --topic DALE.TOPIC \ --group dalegrp \ --consumer-property 'security.protocol=SASL_SSL' \ --consumer-property 'sasl.mechanism=SCRAM-SHA-512' \ --consumer-property 'sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="dale-user" password="pSRtfwTMKNlz";' \ --consumer-property 'ssl.truststore.location=ca.p12' \ --consumer-property 'ssl.truststore.type=PKCS12' \ --consumer-property 'ssl.truststore.password=STOREPASSW0RD'
Tags: apachekafka, kafka
Very good article clearly explaining problem and solution. Thank you.