Pulsar 集群安装

本文涉及的产品
云原生网关 MSE Higress,422元/月
服务治理 MSE Sentinel/OpenSergo,Agent数量 不受限
注册配置 MSE Nacos/ZooKeeper,118元/月
简介:

0x1 摘要

本文记录Pulsar 2.2.0版本安装步骤,单机模式(standalone)比较简单,直接参考官网:http://pulsar.apache.org/docs/en/standalone/
按步骤执行就行,主要讲解集群模式安装,以及过程中遇到问题的解决。

0x2 环境要求

  • Linux
  • Java 8 及以上
  • 3 台ZooKeeper集群

0x3 安装顺序

  • 安装ZooKeeper集群
  • 初始化集群元数据信息
  • 安装BookKeeper集群
  • 安装Pulsar brokers

下面针对每一步进行详细介绍。

0x4 安装ZooKeeper集群

由于我本地环境已经有安装好的ZK集群,可以直接使用,此步省略。

0x5 初始化集群元数据信息

初始化元数据信息非常简单,只需一条命令就可以,具体参数的意义看官网更好理解:

bin/pulsar initialize-cluster-metadata \
  --cluster pulsar-cluster-1 \
  --zookeeper zk1.us-west.example.com:2181 \
  --configuration-store zk1.us-west.example.com:2181 \
  --web-service-url http://pulsar.us-west.example.com:8080 \
  --web-service-url-tls https://pulsar.us-west.example.com:8443 \
  --broker-service-url pulsar://pulsar.us-west.example.com:6650 \
  --broker-service-url-tls pulsar+ssl://pulsar.us-west.example.com:6651

初始化成功会看到以下日志信息:

10:36:09.876 [main] INFO org.apache.bookkeeper.discover.ZKRegistrationManager - Successfully formatted BookKeeper metadata
10:36:09.880 [main] INFO org.apache.zookeeper.ZooKeeper - Session: 0x16734464b360002 closed
10:36:09.880 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x16734464b360002
10:36:10.033 [main] INFO org.apache.pulsar.PulsarClusterMetadataSetup - Cluster metadata for 'pulsar-cluster-1' setup correctly

并且通过ZK客户端登录查看会看到以下节点信息:

[zk: localhost:2186(CONNECTED) 2] ls /
[zookeeper, counters, bookies, ledgers, managed-ledgers, schemas, namespace, admin, loadbalance]

此步非常重要,我在安装过程中忽略此步后启动BookKeeper直接报错,错误信息如下:
启动命令bin/bookkeeper bookie

10:26:17.185 [main] INFO org.apache.bookkeeper.proto.BookieNettyServer - Shutting down BookieNettyServer
10:26:17.196 [main] ERROR org.apache.bookkeeper.server.Main - Failed to build bookie server
org.apache.bookkeeper.bookie.BookieException$MetadataStoreException: Failed to get cluster instance id
 at org.apache.bookkeeper.discover.ZKRegistrationManager.getClusterInstanceId(ZKRegistrationManager.java:387) ~[org.apache.bookkeeper-bookkeeper-server-4.7.2.jar:4.7.2]
 at org.apache.bookkeeper.bookie.Bookie.checkEnvironmentWithStorageExpansion(Bookie.java:412) ~[org.apache.bookkeeper-bookkeeper-server-4.7.2.jar:4.7.2]
 at org.apache.bookkeeper.bookie.Bookie.checkEnvironment(Bookie.java:256) ~[org.apache.bookkeeper-bookkeeper-server-4.7.2.jar:4.7.2]
 at org.apache.bookkeeper.bookie.Bookie.<init>(Bookie.java:640) ~[org.apache.bookkeeper-bookkeeper-server-4.7.2.jar:4.7.2]
 at org.apache.bookkeeper.proto.BookieServer.newBookie(BookieServer.java:131) ~[org.apache.bookkeeper-bookkeeper-server-4.7.2.jar:4.7.2]
 at org.apache.bookkeeper.proto.BookieServer.<init>(BookieServer.java:100) ~[org.apache.bookkeeper-bookkeeper-server-4.7.2.jar:4.7.2]
 at org.apache.bookkeeper.server.service.BookieService.<init>(BookieService.java:43) ~[org.apache.bookkeeper-bookkeeper-server-4.7.2.jar:4.7.2]
 at org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:299) ~[org.apache.bookkeeper-bookkeeper-server-4.7.2.jar:4.7.2]
 at org.apache.bookkeeper.server.Main.doMain(Main.java:219) [org.apache.bookkeeper-bookkeeper-server-4.7.2.jar:4.7.2]
 at org.apache.bookkeeper.server.Main.main(Main.java:201) [org.apache.bookkeeper-bookkeeper-server-4.7.2.jar:4.7.2]
 at org.apache.bookkeeper.proto.BookieServer.main(BookieServer.java:280) [org.apache.bookkeeper-bookkeeper-server-4.7.2.jar:4.7.2]
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for BookKeeper metadata
 at org.apache.bookkeeper.discover.ZKRegistrationManager.getClusterInstanceId(ZKRegistrationManager.java:377) ~[org.apache.bookkeeper-bookkeeper-server-4.7.2.jar:4.7.2]
 ... 10 more

验证BookKeeper时报以下错误:
验证命令bin/bookkeeper shell bookiesanity

09:49:14.430 [main] INFO org.apache.bookkeeper.client.BookKeeper - Weighted ledger placement is not enabled
09:49:14.489 [main] ERROR org.apache.bookkeeper.client.BookieWatcher - Failed to get bookie list : 
org.apache.bookkeeper.client.BKException$ZKException: Error while using ZooKeeper
 at org.apache.bookkeeper.discover.ZKRegistrationClient.lambda$getChildren$0(ZKRegistrationClient.java:212) ~[org.apache.bookkeeper-bookkeeper-server-4.7.2.jar:4.7.2]
 at org.apache.bookkeeper.zookeeper.ZooKeeperClient$25$1.processResult(ZooKeeperClient.java:1174) ~[org.apache.bookkeeper-bookkeeper-server-4.7.2.jar:4.7.2]
 at org.apache.zookeeper.ClientCnxn$EventThread.processEvent_aroundBody0(ClientCnxn.java:604) ~[org.apache.pulsar-pulsar-broker-2.2.0.jar:2.2.0]
 at org.apache.zookeeper.ClientCnxn$EventThread$AjcClosure1.run(ClientCnxn.java:1) ~[org.apache.pulsar-pulsar-broker-2.2.0.jar:2.2.0]
 at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) ~[org.aspectj-aspectjrt-1.9.1.jar:?]
 at org.apache.pulsar.broker.zookeeper.aspectj.ClientCnxnAspect.timedProcessEvent(ClientCnxnAspect.java:72) ~[org.apache.pulsar-pulsar-broker-2.2.0.jar:2.2.0]
 at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:528) ~[org.apache.pulsar-pulsar-broker-2.2.0.jar:2.2.0]
 at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:508) ~[org.apache.pulsar-pulsar-broker-2.2.0.jar:2.2.0]
Exception in thread "main" org.apache.bookkeeper.client.BKException$ZKException: Error while using ZooKeeper
 at org.apache.bookkeeper.discover.ZKRegistrationClient.lambda$getChildren$0(ZKRegistrationClient.java:212)
 at org.apache.bookkeeper.zookeeper.ZooKeeperClient$25$1.processResult(ZooKeeperClient.java:1174)
 at org.apache.zookeeper.ClientCnxn$EventThread.processEvent_aroundBody0(ClientCnxn.java:604)
 at org.apache.zookeeper.ClientCnxn$EventThread$AjcClosure1.run(ClientCnxn.java:1)
 at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
 at org.apache.pulsar.broker.zookeeper.aspectj.ClientCnxnAspect.timedProcessEvent(ClientCnxnAspect.java:72)
 at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:528)
 at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:508)

0x6 安装BookKeeper集群

直接使用Pulsar内嵌的BK,单独安装未尝试。
修改$PULSAR_HOME/conf/bookkeeper.conf配置文件中的zkServers属性,如:

zkServers=zk1.us-west.example.com:2181,zk2.us-west.example.com:2181,zk3.us-west.example.com:2181

修改完成后直接命令启动,启动命令bin/pulsar-daemon start bookie
启动完成后使用命令bin/bookkeeper shell bookiesanity验证是否成功,看到以下信息说明启动成功

10:44:49.735 [main] INFO org.apache.zookeeper.ZooKeeper - Session: 0x46734464bc90004 closed
10:44:49.735 [main] INFO org.apache.bookkeeper.bookie.BookieShell - Bookie sanity test succeeded
10:44:49.735 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x46734464bc90004

0x7 安装Pulsar brokers

修改$PULSAR_HOME/conf/broker.conf配置文件

zookeeperServers=zk1.us-west.example.com:2181,zk2.us-west.example.com:2181,zk3.us-west.example.com:2181
configurationStoreServers=zk1.us-west.example.com:2181,zk2.us-west.example.com:2181,zk3.us-west.example.com:2181
clusterName=pulsar-cluster-1

修改完成后使用命令bin/pulsar-daemon start broker启动broker
启动完成后可以通过bin/pulsar-client produce命令发送消息测试,我自己通过Java Client代码测试。
Producer代码:

PulsarClient client = PulsarClient.builder().serviceUrl("pulsar://pulsar.us-west.example.com:6650").build();
Producer<String> producer = client.newProducer(Schema.STRING).topic("my-topic").create();
producer.send("aaaaaaaaaaaaaa");
producer.close();

Consumer代码:

PulsarClient client = PulsarClient.builder().serviceUrl("pulsar://pulsar.us-west.example.com:6650").build();
Consumer<String> consumer = client.newConsumer(Schema.STRING)
 .topic("my-topic")
 .subscriptionName("my-topic-group-1")
 .subscribe();
do {
 Message<String> receive = consumer.receive();
 System.out.println(receive.getValue());
} while (true);

参考:http://pulsar.apache.org/docs/latest/deployment/cluster/

相关实践学习
基于MSE实现微服务的全链路灰度
通过本场景的实验操作,您将了解并实现在线业务的微服务全链路灰度能力。
目录
相关文章
|
消息中间件 Java Linux
聊聊 Pulsar: 在 Linux 环境上搭建 Pulsar
聊聊 Pulsar: 在 Linux 环境上搭建 Pulsar
433 0
|
消息中间件 存储 Kubernetes
kafka/pulsar on k8s
kafka/pulsar on k8s
kafka/pulsar on k8s
|
存储 消息中间件 监控
Pulsar 介绍与部署
Pulsar 介绍与部署
4244 0
Pulsar 介绍与部署
|
7月前
|
Kubernetes 固态存储 容器
pulsar on k8s
pulsar on k8s
|
7月前
|
存储 Kubernetes Apache
pulsar on k8s 部署验证
pulsar on k8s 部署验证
|
存储 Cloud Native Java
聊聊 Pulsar: Pulsar 的核心概念与基础架构
聊聊 Pulsar: Pulsar 的核心概念与基础架构
1998 0
|
网络协议 算法 Java
聊聊 Pulsar: Pulsar 分布式集群搭建
聊聊 Pulsar: Pulsar 分布式集群搭建
941 0
|
存储 SQL 缓存
聊聊 Pulsar:编译 Pulsar 源码并搭建源码环境
聊聊 Pulsar:编译 Pulsar 源码并搭建源码环境
395 0
Pulsar常用命令记录
Pulsar常用命令记录
244 0
|
消息中间件 存储 Kafka
通过 KoP 将 Kafka 应用迁移到 Pulsar
KoP(Pulsar on Kafka)通过在 Pulsar Broker 上引入 Kafka 协议处理程序,为 Apache Pulsar 带来原生 Apache Kafka 协议支持。 通过将 KoP 协议处理程序添加到您现有的 Pulsar 集群,您可以将现有的 Kafka 应用程序和服务迁移到 Pulsar,而无需修改代码。 这使 Kafka 应用程序能够利用 Pulsar 的强大功能,
458 0