Kafka 4 Connect

Set up Kafka Connect from Ambari

When you install Kafka, ensure you select the appropriate nodes for installing the Kafka broker and Connect services.

Set up Kafka Connect with Kerberos, SSL, and Ranger

Since Kafka is already configured with SSL, Kerberos, and Ranger, the underlying topics maintain these settings during Kafka Connect operations. The following are the configurations for the Advanced kafka3-connect-distributed section in Kafka3's CONFIGS.

bootstrap.servers=<broker1-host>:7001,<broker2-host>:7001,<brokern-host>:7001
config.storage.replication.factor=3
config.storage.topic=connect-configs
group.id=connect-cluster
key.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=true
offset.flush.interval.ms=10000
offset.storage.replication.factor=3
offset.storage.topic=connect-offsets
plugin.path=/usr/odp/current/kafka3-broker/libs/*
 
listeners=localhost:8084
rest.advertised.listener=HTTPS
 
status.storage.replication.factor=3
status.storage.topic=connect-status
sun.security.krb5.debug=true
value.converter=org.apache.kafka.connect.json.JsonConverter
value.converter.schemas.enable=true
 
sasl.enabled.mechanisms=GSSAPI
sasl.kerberos.service.name=kafka
sasl.mechanism=GSSAPI
sasl.mechanism.inter.broker.protocol=GSSAPI
 
security.inter.broker.protocol=SASL_SSL
security.protocol=SASL_SSL
 
ssl.key.password=<password>
ssl.keystore.location=/opt/security/pki/keystore.jks
ssl.keystore.password=<password>
ssl.truststore.location=/opt/security/pki/truststore.jks
ssl.truststore.password=<password>
 
java.security.auth.login.config=/usr/odp/current/kafka-broker/config/kafka_jaas.conf
java.security.krb5.kdc=<KDC server hostname>
java.security.krb5.realm=<KDC server realm name>
javax.security.auth.useSubjectCredsOnly=true
 
consumer.bootstrap.servers=<broker1-host>:7001,<broker2-host>:7001,<brokern-host>:7001
consumer.security.protocol=SASL_SSL
consumer.ssl.truststore.location=/opt/security/pki/truststore.jks
consumer.ssl.truststore.password=<password>
 
producer.bootstrap.servers=<broker1-host>:7001,<broker2-host>:7001,<brokern-host>:7001
producer.security.protocol=SASL_SSL
producer.ssl.truststore.location=/opt/security/pki/truststore.jks

After you configure the settings above, restart the Kafka service, including the broker and Connect. Once the restart completes, Kafka Connect operates in Kerberos and SSL mode.

Kafka connector use cases

Acceldata ODP Kafka Connect supports the following connectors, included in this release:

1. Pubsub Connector
2. MongoDB Connector
3. JDBC Connector
4. S3 Connector
5. Amazon SQS Sink Connector
6. Amazon Kinesis Firehose Connector
7. camel-aws-sqs-sink

The following table describes the supported connector type and library version for each connector.