Kafka安装、配置、生产者及消费者验证（jdk、zk安装及配置）

本文详细介绍了Kafka的安装配置过程，包括JDK环境配置、Zookeeper（ZK）的安装与配置、Kafka的安装与配置。最后，通过kafka-topics.sh管理主题，kafka-console-consumer.sh和kafka-console-producer.sh进行消息的生产和消费验证。

文章目录

- - Kafka安装与配置

Kafka安装与配置

jdk环境配置

安装

[root@10 software]# rpm -ivh jdk-8u261-linux-x64.rpm
警告：jdk-8u261-linux-x64.rpm: 头V3 RSA/SHA256 Signature, 密钥 ID ec551f03: NOKEY
准备中...                          ################################# [100%]
正在升级/安装...
   1:jdk1.8-2000:1.8.0_261-fcs        ################################# [100%]
Unpacking JAR files...
	tools.jar...
	plugin.jar...
	javaws.jar...
	deploy.jar...
	rt.jar...
	jsse.jar...
	charsets.jar...
	localedata.jar...

配置环境变量

[root@10 software]# vim /etc/profile

export JAVA_HOME=/usr/java/jdk1.8.0_261-amd64
export PATH=$PATH:$JAVA_HOME/bin

验证是否安装成功

# 生效 
[root@10 software]# source /etc/profile
# 验证 
[root@10 software]# java -version
java version "1.8.0_261"
Java(TM) SE Runtime Environment (build 1.8.0_261-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.261-b12, mixed mode)
[root@10 software]#

zk安装与配置

上传zookeeper-3.4.14.tar.gz到服务器

解压zk到/opt目录

[root@10 software]# tar -zxf zookeeper-3.4.14.tar.gz -C /opt

配置环境变量

[root@10 software]# vim /etc/profile

export ZOOKEEPER_PREFIX=/opt/zookeeper-3.4.14
export PATH=$PATH:$ZOOKEEPER_PREFIX/bin
export ZOO_LOG_DIR=/home/laosan/software/zookeeper/log

#生效配置文件
[root@10 software]# . /etc/profile

配置zk配置文件

[root@10 conf]# pwd
/opt/zookeeper-3.4.14/conf
#复制zoo_sample.cfg命名为zoo.cfg
[root@10 conf]# cp zoo_sample.cfg zoo.cfg
[root@10 conf]# ll
总用量 16
-rw-rw-r--. 1 2002 2002  535 3月   7 2019 configuration.xsl
-rw-rw-r--. 1 2002 2002 2161 3月   7 2019 log4j.properties
-rw-r--r--. 1 root root  922 8月  23 17:45 zoo.cfg
-rw-rw-r--. 1 2002 2002  922 3月   7 2019 zoo_sample.cfg

# 编辑zoo.cfg文件
[root@10 conf]# vim zoo.cfg

#只修改这一处即可
dataDir=/home/laosan/software/zookeeper/log

验证是否安装成功

#启动zk
[root@10 conf]# zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper-3.4.14/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
#查看zk状态
[root@10 conf]# zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /opt/zookeeper-3.4.14/bin/../conf/zoo.cfg
Mode: standalone

kafka安装与配置

上传kafka_2.12-1.0.2.tgz到服务器
解压kafka到/opt目录

[root@10 software]# tar -zxf kafka_2.12-1.0.2.tgz -C /opt

配置环境变量

[root@10 software]# vim /etc/profile

export KAFKA_HOME=/opt/kafka_2.12-1.0.2
export PATH=$PATH:$KAFKA_HOME/bin

#生效配置文件
[root@10 software]# . /etc/profile

配置kafka配置文件

[root@10 config]# pwd
/opt/kafka_2.12-1.0.2/config
[root@10 config]# ll
总用量 64
-rw-r--r--. 1 root root  906 6月  30 2018 connect-console-sink.properties
-rw-r--r--. 1 root root  909 6月  30 2018 connect-console-source.properties
-rw-r--r--. 1 root root 5807 6月  30 2018 connect-distributed.properties
-rw-r--r--. 1 root root  883 6月  30 2018 connect-file-sink.properties
-rw-r--r--. 1 root root  881 6月  30 2018 connect-file-source.properties
-rw-r--r--. 1 root root 1111 6月  30 2018 connect-log4j.properties
-rw-r--r--. 1 root root 2730 6月  30 2018 connect-standalone.properties
-rw-r--r--. 1 root root 1221 6月  30 2018 consumer.properties
-rw-r--r--. 1 root root 4727 6月  30 2018 log4j.properties
-rw-r--r--. 1 root root 1919 6月  30 2018 producer.properties
-rw-r--r--. 1 root root 6877 8月  23 18:06 server.properties
-rw-r--r--. 1 root root 1032 6月  30 2018 tools-log4j.properties
-rw-r--r--. 1 root root 1023 6月  30 2018 zookeeper.properties

# 编辑server.properties文件
[root@10 config]# vim server.properties

#36行
advertised.listeners=PLAINTEXT://laosan1:9092
#60行，配置kafka存储持久化数据的目录
log.dirs=/home/laosan/software/kafka/log
#123行，多个路径用逗号分隔，端口号后面加上具体存放kafka的目录地址myKafka
zookeeper.connect=laosan1:2181/myKafka

启动验证是否安装成功

[root@10 bin]# pwd
/opt/kafka_2.12-1.0.2/bin
#启动kafka
[root@10 bin]# kafka-server-start.sh ../config/server.properties 
[2020-08-23 18:13:48,204] WARN No meta.properties file under dir /home/laosan/software/kafka/log/meta.properties (kafka.server.BrokerMetadataCheckpoint)
[2020-08-23 18:13:48,295] INFO [SocketServer brokerId=0] Started processors for 1 acceptors (kafka.network.SocketServer)
[2020-08-23 18:13:48,302] INFO Kafka version : 1.0.2 (org.apache.kafka.common.utils.AppInfoParser)
[2020-08-23 18:13:48,302] INFO Kafka commitId : 2a121f7b1d402825 (org.apache.kafka.common.utils.AppInfoParser)
#启动成功，且id为0
[2020-08-23 18:13:48,303] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)

#后台进程启动
[root@10 bin]# kafka-server-start.sh -daemon ../config/server.properties

查看zk的节点

[root@10 ~]# zkCli.sh
Welcome to ZooKeeper!
2020-08-23 18:27:37,711 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1025] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2020-08-23 18:27:37,721 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@879] - Socket connection established to localhost/127.0.0.1:2181, initiating session
JLine support is enabled
2020-08-23 18:27:37,753 [myid:] - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1299] - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x100001e64fc0002, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null

#查看kafka创建的节点
[zk: localhost:2181(CONNECTED) 5] ls /myKafka
[cluster, controller_epoch, brokers, admin, isr_change_notification, consumers, log_dir_event_notification, latest_producer_id_block, config]

生产与消费

kafka-topics.sh 用于管理主题

#创建主题
[root@10 bin]# kafka-topics.sh --zookeeper localhost:2181/myKafka --create --topic topic_laosan --partitions 1 --replication-factor 1
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic "topic_laosan".
#查看主题列表
[root@10 bin]# kafka-topics.sh --list --zookeeper localhost:2181/myKafka
topic_laosan
#查看指定主题的详细信息
[root@10 bin]# kafka-topics.sh --zookeeper localhost/myKafka --describe --topic topic_laosan
Topic:topic_laosan	PartitionCount:1	ReplicationFactor:1	Configs:
	Topic: topic_laosan	Partition: 0	Leader: 0	Replicas: 0	Isr: 0

kafka-console-consumer.sh用于消费消息

# 开启消费者
[root@10 config]# kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic topic_laosan
#接受消息
hello

# 开启消费者方式二，从头消费，不按照偏移量消费
[root@10 config]# kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic topic_laosan --from-beginning

kafka-console-producer.sh用于生产消息

[root@10 config]# kafka-console-producer.sh --broker-list localhost:9092 --topic topic_laosan
#发送消息
>hello