当前位置：首页 > news >正文

docker部署kafka实战

news 来源：原创 2025/4/29 5:31:41

一、部署kafaka、zookeeper

二、测试信息发送与接收

三、kafka进阶

一、部署kafaka、zookeeper

请提前安装docker、docker-compose

安装docker：docker--安装docker-ce-CSDN博客

安装docker-compose：安装docker-compose_安装 docker-compose-CSDN博客

docker-compose部署，建议用docker-compose部署而不是单容器部署，快速便捷。（yml文件：docker部署kafka的方法步骤_docker_脚本之家）

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    volumes:
      - ./data:/data
    ports:
      - "2181:2181"
       
  kafka:
    image: wurstmeister/kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME:127.0.0.1
      KAFKA_MESSAGE_MAX_BYTES: 2000000
      KAFKA_CREATE_TOPICS: "Topic1:1:3,Topic2:1:1:compact"
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    volumes:
      - ./kafka-logs:/kafka
      - /var/run/docker.sock:/var/run/docker.sock
 
  kafka-manager:
    image: sheepkiller/kafka-manager
    ports:
      - 9020:9000
    environment:
      ZK_HOSTS: zookeeper:2181

设置容器运行机制为自启动

#docker update --restart=always 容器名/容器ID
[root@kafka_50 ~]# docker update --restart=always e398ce29cbd2 f0ef33856875 e06af2230915

重启虚拟机kafka-manage无法启动，docker logs 容器名，查看容器日志，发现报错This application is already running (Or delete /kafka-manager-1.3.1.8/RUNNING_PID file).
，提示kafka已经运行或者需要删除RUNNING_PID文件需要，find命令找并删除RUNNING_PID文件。

[root@kafka_50 ~]# find / -name RUNNING_PID
/var/lib/docker/overlay2/1ac3b62babe69290c14730892564d317e4dc4c443d298d6804ab20290cb48c80/diff/kafka-manager-1.3.1.8/RUNNING_PID
[root@kafka_50 ~]# rm -rf /var/lib/docker/overlay2/1ac3b62babe69290c14730892564d317e4dc4c443d298d6804ab20290cb48c80/diff/kafka-manager-1.3.1.8/RUNNING_PID

二、测试信息发送与接收

查看容器信息，注意容器名

[root@kafka_50 ~]# docker ps
CONTAINER ID        IMAGE                       COMMAND                  CREATED             STATUS              PORTS                                                NAMES
e398ce29cbd2        wurstmeister/zookeeper      "/bin/sh -c '/usr/sb…"   7 days ago          Up About an hour    22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp   kafka-zookeeper-1
f0ef33856875        wurstmeister/kafka          "start-kafka.sh"         7 days ago          Up About an hour    0.0.0.0:9092->9092/tcp                               kafka-kafka-1
e06af2230915        sheepkiller/kafka-manager   "./start-kafka-manag…"   7 days ago          Up About an hour    0.0.0.0:9020->9000/tcp                               kafka-kafka-manager-1

创建名为test_topic的topic主题、生产者

# 进入kafka 容器后，创建topic
# kafka-topics.sh --create --topic <topic_name> --partitions 1 --replication-factor 1 --zookeeper zookeeper:2181
            
[root@kafka_50 ~]# docker exec -it kafka-kafka-1 /bin/bash      # 进入kafka 容器

bash-5.1# kafka-topics.sh --create --topic test_topic --partitions 1 --replication-factor     #创建名为test_topic的主题
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.        = #警告无需理会，关于".""_"的使用规范
Created topic test_topic.                                                                         
 
#在test_topic主题下创建生产者
bash-5.1# kafka-console-producer.sh --broker-list localhost:9092 --topic test_topic
>hello   #随便发送测试信息
>hello123

新建一个终端创建消费者

[root@kafka_50 ~]# docker exec -it kafka-kafka-1 /bin/bash

#创建消费者
bash-5.1# kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test_topic
hello    #成功接收到生成者发送的信息，接收实时数据，历史数据不会展示
hello123

#消费者接收历史信息参数--from-beginning
bash-5.1# kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test_topic --from-beginning