实操,ClickHouse高可用集群部署

clickhouse feko 3周前 (09-01) 24次浏览 已收录 0个评论 扫描二维码
文章目录[隐藏]

ClickHouse集群部署

实操,ClickHouse高可用集群部署
zookeeper + ReplicatedMergeTree(复制表) + Distributed(分布式表)

实例:

主机列表:

cat >> /etc/hosts<<EOF
192.168.4.9   vm-centos7-ck-0001  
192.168.4.10  vm-centos7-ck-0002    
192.168.4.11  vm-centos7-ck-0003    
192.168.4.12  vm-centos7-ck-0004    
EOF

ck角色划分(4台):

分片1:vm-centos7-ck-0001 vm-centos7-ck-0002 分片内两两互相复制,互为副本

分片2:vm-centos7-ck-0003 vm-centos7-ck-0004 分片内两两互相复制,互为副本

zookeeper集群(3台):最后单独分开,这里直接用

vm-centos7-ck-0001

vm-centos7-ck-0002

vm-centos7-ck-0003

zookeeper 集群进行了这样三种角色划分:leader、follower、observer分别对应着总统、议员和观察者。

总统(leader):负责进行投票的发起和决议,更新系统状态。

议员(follower):用于接收客户端请求并向客户端返回结果以及在选举过程中参与投票。

观察者(observer):也可以接收客户端连接,将写请求转发给leader节点,但是不参与投票过程,只同步leader的状态。通常对查询操作做负载。

安装zookeeper集群

  • 安装java环境
yum install java -y
  • 安装zk集群
leader下载二进制文件:
wget http://archive.apache.org/dist/zookeeper/zookeeper-3.6.0/apache-zookeeper-3.6.0-bin.tar.gz
rm -rf /usr/local/zookeeper
tar xf apache-zookeeper-3.6.0-bin.tar.gz  -C /usr/local/
mv /usr/local/apache-zookeeper-3.6.0-bin /usr/local/zookeeper

创建目录:
mkdir /data/zookeeper/data -p
mkdir /data/logs/zookeeper -p
机器1指定节点号1,以此类推:
echo '1' > /data/zookeeper/data/myid
zookeeper配置文件:
cat > /usr/local/zookeeper/conf/zoo.cfg <<EOF
# 基本时间单位, 毫秒值
tickTime=2000

# tickTime的倍数,follower和leader之间的最长心跳时间
initLimit=30000

# tickTime的倍数,leader和follower之间发送消息, 请求和应答的最大时间
syncLimit=10
# 数据目录
dataDir=/data/zookeeper/data

# 日志目录,如果没设定,默认和dataDir相同
dataLogDir=/data/logs/zookeeper

# 监听client连接的端口号
clientPort=2181

# zookeeper最大连接
maxClientCnxns=2000

# 最大的会话超时时间
maxSessionTimeout=60000000

# 保留的文件数目,默认3个
autopurge.snapRetainCount=10

# 自动清理snapshot和事务日志,清理频率,单位是小时
autopurge.purgeInterval=1

# 等待处理的最大请求数量
globalOutstandingLimit=200

# 日志文件大小Kb,切换快照生成日志
preAllocSize=131072

#两次事务快照之间可执行事务的次数,默认的配置值为100000
snapCount=3000000

# leader是否接受client请求,默认为yes即leader可以接受client的连接,当节点数为>3时,建议关闭
leaderServes=yes

# 2888 Leader选举的端口;3888 Zookeeper服务器之间的通信端口
server.1=192.168.4.9:2888:3888
server.2=192.168.4.10:2888:3888
server.3=192.168.4.11:2888:3888
EOF

三台机都要启动:
/usr/local/zookeeper/bin/zkServer.sh start

查看状态:
# /usr/local/zookeeper/bin/zkServer.sh       status

高可用说明:
1个leader + 2个follower  
leader机器宕机后,会跳到其他机器替换follower成leader 
恢复后自动同步为follower

安装clickhouse

curl -s https://packagecloud.io/install/repositories/altinity/clickhouse/script.rpm.sh | sudo os=centos dist=7 bash
  • 查看安装包
yum list "clickhouse*"
  • 安装服务
yum install clickhouse-server clickhouse-client  -y
  • 查看安装列表
yum list installed "clickhouse*"
  • 控制台输出
Installed Packages
clickhouse-client.noarch
clickhouse-common-static.x86_64
clickhouse-server.noarch
  • 查看配置
cd /etc/clickhouse-server/
vim config.xml
默认:
数据目录:/var/lib/clickhouse
临时目录:/var/lib/clickhouse/tmp
日志目录:/var/log/clickhouse-server
HTTP端口:8123
TCP 端口:9000
<http_port>8123</http_port>
<tcp_port>9000</tcp_port>
<mysql_port>9004</mysql_port>
<interserver_http_port>9009</interserver_http_port>

修改默认配置
mkdir /data/db/ck/clickhouse/tmp -p
mkdir /data/logs/clickhouse-server -p 
mkdir /data/db/ck/clickhouse/data_old -p

chown clickhouse.clickhouse /data/db/ck/clickhouse -R
chown clickhouse.clickhouse /data/logs/clickhouse-server -R

sed -i 's#/var/log/clickhouse-server#/data/logs/clickhouse-server#g' /etc/init.d/clickhouse-server
sed -i 's#/opt/clickhouse#/data/db/ck/clickhouse/data_old#g' /etc/init.d/clickhouse-server
sed -i 's#/var/lib/clickhouse#/data/db/ck/clickhouse/data#g' /etc/init.d/clickhouse-server
  • 配置文件

config.xml

cat > /etc/clickhouse-server/config.xml <<EOF
<?xml version="1.0"?>
    <yandex>
    <!--日志-->
    <logger>
    <level>error</level>
    <log>/data/logs/clickhouse-server/clickhouse.log</log>
    <errorlog>/data/logs/clickhouse-server/error.log</errorlog>
    <size>500M</size>
    <count>5</count>
    </logger>
    <!--本地节点信息-->
    <http_port>8123</http_port>
    <tcp_port>9000</tcp_port>
    <interserver_http_port>9009</interserver_http_port>
    <interserver_http_host>192.168.4.9</interserver_http_host>  <!--本机域名或IP ,需要修改-->
    <!--本地配置-->
    <listen_host>0.0.0.0</listen_host>
    <max_connections>4096</max_connections>

    <keep_alive_timeout>300</keep_alive_timeout>
    <max_concurrent_queries>1000</max_concurrent_queries>
    <max_server_memory_usage>0</max_server_memory_usage>
    <max_server_memory_usage_to_ram_ratio>0.8</max_server_memory_usage_to_ram_ratio>
    <uncompressed_cache_size>8589934592</uncompressed_cache_size>
    <mark_cache_size>5368709120</mark_cache_size>
    <path>/data/db/ck/clickhouse/</path>
    <tmp_path>/data/db/ck/clickhouse/tmp/</tmp_path>
    <users_config>users.xml</users_config>
    <default_profile>default</default_profile>

    <default_database>default</default_database>
    <!--集群相关配置-->
    <remote_servers incl="clickhouse_remote_servers"  />
    <zookeeper incl="zookeeper-servers"  optional="true"   />
    <macros incl="macros"  optional="true" />
    <builtin_dictionaries_reload_interval>3600</builtin_dictionaries_reload_interval>
    <max_session_timeout>3600</max_session_timeout>
    <default_session_timeout>300</default_session_timeout>
    <max_table_size_to_drop>0</max_table_size_to_drop>
    <merge_tree>
    <parts_to_delay_insert>300</parts_to_delay_insert>
    <parts_to_throw_insert>600</parts_to_throw_insert>
    <max_delay_to_insert>2</max_delay_to_insert>
    </merge_tree>
    <max_table_size_to_drop>0</max_table_size_to_drop>
     <max_partition_size_to_drop>0</max_partition_size_to_drop>
    <distributed_ddl>
    <!-- Path in ZooKeeper to queue with DDL queries -->
    <path>/clickhouse/task_queue/ddl</path>
    </distributed_ddl>
     <include_from>/etc/clickhouse-server/metrika.xml</include_from>
    </yandex>
EOF

user.xml

cat > /etc/clickhouse-server/users.xml << EOF
<?xml version="1.0"?>
    <yandex>
    <profiles>

    <default>
    <receive_timeout>800</receive_timeout>
    <send_timeout>800</send_timeout>
    <log_queries>1</log_queries>
    <max_memory_usage>10000000000</max_memory_usage>
    <use_uncompressed_cache>0</use_uncompressed_cache>
    <load_balancing>random</load_balancing>
    </default>
    <readonly>
    <max_memory_usage>10000000000</max_memory_usage>
    <use_uncompressed_cache>0</use_uncompressed_cache>
    <load_balancing>random</load_balancing>
    <readonly>1</readonly>
    </readonly>
    </profiles>
    <quotas>
    <!-- Name of quota. -->
    <default>
    <interval>
    <queries>0</queries>
    <errors>0</errors>
    <result_rows>0</result_rows>
    <read_rows>0</read_rows>
    <execution_time>0</execution_time>
    </interval>
    </default>
    </quotas>
    <users>
    <default>
    <!-- PASSWORD=$(base64 < /dev/urandom | head -c8); echo  "$PASSWORD"; echo -n  "$PASSWORD" | sha256sum | tr -d '-'   -->
    <!-- password M4XZ5biK -->
<password_sha256_hex>5c2dce318ffdb3ea379fc25af8dcee2f5b8d7b651748a90173437e7e199f208f</password_sha256_hex>
    <networks>
    <ip>::/0</ip>
    </networks>
    <profile>default</profile>
    <quota>default</quota>
    </default>
    <ck>
    <password_sha256_hex>5c2dce318ffdb3ea379fc25af8dcee2f5b8d7b651748a90173437e7e199f208f</password_sha256_hex>
    <networks>
    <ip>::/0</ip>
    </networks>
    <profile>readonly</profile>
    <quota>default</quota>
    </ck>
    </users>
    </yandex>
EOF

metrika.xml(2分片4副本4台机)

cat > /etc/clickhouse-server/metrika.xml << EOF
<yandex>
    <!--ck集群节点-->
    <clickhouse_remote_servers>
    <clickhouse_cluster_name>
    <!--分片1  两两互相复制-->
    <shard>
    <internal_replication>true</internal_replication>

    <replica>
    <host>vm-centos7-ck-0001</host>
    <port>9000</port>
    <user>default</user>
    <password>M4XZ5biK</password>
    </replica>

    <replica>
    <host>vm-centos7-ck-0002</host>
    <port>9000</port>
    <user>default</user>
    <password>M4XZ5biK</password>
    </replica>

    </shard>

    <!--分片2  两两互相复制-->
    <shard>
    <internal_replication>true</internal_replication>

    <replica>
    <host>vm-centos7-ck-0003</host>
    <port>9000</port>
    <user>default</user>
    <password>M4XZ5biK</password>
    </replica>

    <replica>
    <host>vm-centos7-ck-0004</host>
    <port>9000</port>
    <user>default</user>
    <password>M4XZ5biK</password>
    </replica>

    </shard>
    </clickhouse_cluster_name>
    </clickhouse_remote_servers>

    <!--zookeeper相关配置-->
    <zookeeper-servers>
    <node index="1">
    <host>vm-centos7-ck-0001</host>
    <port>2181</port>
    </node>
    <node index="2">
    <host>vm-centos7-ck-0002</host>
    <port>2181</port>
    </node>
    <node index="3">
    <host>vm-centos7-ck-0003</host>
    <port>2181</port>
    </node>
    </zookeeper-servers>
    <macros>
    <layer>01</layer>
    <shard>01</shard> <!--当前节点分片号,需要修改-->
    <replica>vm-centos7-ck-0001</replica> <!--当前节点IP,需要修改-->
    </macros>
    <networks>
    <ip>::/0</ip>
    </networks>
    <!--压缩相关配置-->
    <clickhouse_compression>
    <case>
    <min_part_size>10000000000</min_part_size>
    <min_part_size_ratio>0.01</min_part_size_ratio>
    <method>lz4</method> <!--压缩算法lz4压缩比zstd快, 更占磁盘-->
    </case>
    </clickhouse_compression>
    </yandex>
EOF
  • 启动服务
/etc/rc.d/init.d/clickhouse-server start
chkconfig   clickhouse-server  off   # 先启动zookeeper,再启动clickhouse
  • 查看服务
ps -aux |grep clickhouse
  • 查看日志,检查是否正常
cat /data/logs/clickhouse-server/error.log

clickhouse集群测试

  • 登陆集群
  clickhouse-client -h 127.0.0.1 -d default -m -u default --password M4XZ5biK
  use system;
  select * from clusters;
  select * from zookeeper where path = '/';
  • 集群测试
  # clickhouse-client -h 127.0.0.1 -d default -m -u default --password M4XZ5biK
  创建数据库和表
  :) create database test001; #所有节点执行
  :) CREATE TABLE test001.ontime (FlightDate Date,Year UInt16) ENGINE = ReplicatedMergeTree('/clickhouse/test001/ontime/shard1','1',FlightDate,(Year, FlightDate),8192); #分片1的第1个副本执行
  :) CREATE TABLE test001.ontime (FlightDate Date,Year UInt16) ENGINE = ReplicatedMergeTree('/clickhouse/test001/ontime/shard1','2',FlightDate,(Year, FlightDate),8192); #分片1的第2个副本执行
  :) CREATE TABLE test001.ontime (FlightDate Date,Year UInt16) ENGINE = ReplicatedMergeTree('/clickhouse/test001/ontime/shard2','1',FlightDate,(Year, FlightDate),8192); #分片2的第1个副本执行
  :) CREATE TABLE test001.ontime (FlightDate Date,Year UInt16) ENGINE = ReplicatedMergeTree('/clickhouse/test001/ontime/shard2','2',FlightDate,(Year, FlightDate),8192); #分片2的第2个副本执行
  :) CREATE TABLE test001.ontime_all (FlightDate Date,Year UInt16) ENGINE= Distributed(clickhouse_cluster_name, test001, ontime, rand()); #所有节点执行
  ####说明:ontime是本地表,存数据的;ontime_all是分布式表,不存数据

  插入本地表数据测试:
  说明:在分片1的第1个副本插入数据到表ontime,在分片1的第2个副本可以查,其他切片不可查
       在分片2的第1个副本插入数据到表ontime,在分片2的第2个副本可以查,其他切片不可查
       在任意切片插入到ontime表,都可以在分布式表ontime_all上查看
  在分片1的第1个副本插入数据
  vm-centos7-ck-0001 :) insert into test001.ontime (FlightDate,Year)values('2020-8-12',2020);
  在分片1的第1和第2个副本查询本地表
  vm-centos7-ck-0001 :) select * from test001.ontime;
  ┌─FlightDate─┬─Year─┐
  │ 2020-08-12 │ 2020 │
  └────────────┴──────┘
  vm-centos7-ck-0002 :) select * from test001.ontime;
  ┌─FlightDate─┬─Year─┐
  │ 2020-08-12 │ 2020 │
  └────────────┴──────┘

  插入分布式表数据测试:
  说明:在任意机器上插入数据到分布式表的时候,分布式表会把插入的信息轮询到分片1和分片2的机器上
  vm-centos7-ck-0001 :) insert into test001.ontime_all (FlightDate,Year)values('2020-8-12',2020);

  在每一个分片的每一个副本都可以查分布式表:
  vm-centos7-ck-0003 :) select * from test001.ontime_all;
  ┌─FlightDate─┬─Year─┐
  │ 2020-08-12 │ 2020 │
  └────────────┴──────┘

  删除更新本地表
  ALTER TABLE test001.ontime DELETE WHERE Year = '2020' ;
  ALTER TABLE test001.ontime update Year = 2019 WHERE Year = 2021 ;
  索引列不能进行更新
  分布式表不能进行更新
  ALTER TABLE UPDATE/DELETE不支持分布式DDL,因此需要在分布式环境中手动在每个节点上local的进行更新/删除数据。  
  • 分布式DDL
在任意一台机上创建数据库,会在每一个节点上执行
  CREATE database test002  ON CLUSTER '{cluster}';
在任意一台机上创建本地表,会在每一个节点上执行
  CREATE TABLE IF NOT EXISTS test001.ontime ON CLUSTER '{cluster}' (FlightDate Date,Year UInt16) ENGINE = ReplicatedMergeTree('/clickhouse/test001/ontime/{shard}','{replica}',FlightDate,(Year, FlightDate),8192);
在任意一台机上创建分布式表,会在每一个节点上执行
CREATE TABLE test001.ontime_all ON CLUSTER '{cluster}' (FlightDate Date,Year UInt16) ENGINE= Distributed('{cluster}', test001, ontime, rand());
在任意一台机,删除更新本地表
  ALTER TABLE test001.ontime ON CLUSTER  '{cluster}' DELETE WHERE FlightDate = '2021-01-13' ;
其中,ON CLUSTER语法表示分布式DDL,即执行一次就可在集群所有实例上创建同样的本地表。集群标识符{cluster}、分片标识符{shard}和副本标识符{replica}来自之前提到过的复制表宏配置,即config.xml中<macros>一节的内容

模拟单机宕机

  • 假设vm-centos7-ck-0001宕机(0001机器有clickhouse切片1的1副本,和zookeeper的一个集群)

    直接关机即可

  • 测试集群是否正常

  ## 测试clickhouse是否正常运行
  在任意没宕机的机器上插入数据到本地表或从分布式表中插入数据
  vm-centos7-ck-0002 :) insert into test001.ontime (FlightDate,Year)values('2022-2-2',2022);
  在没宕机任意节点查看分布式表可以查到
  vm-centos7-ck-0003 :) select * from test001.ontime_all;

  ##测试分片内的副本复制是否正常,即是测试zookeeper集群
  在vm-centos7-ck-0003机器上插入数据到本地表
  vm-centos7-ck-0003 :) insert into test001.ontime_all (FlightDate,Year)values('2022-3-3',2022);
  在vm-centos7-ck-0004机器上可以查看到数据
  • 启动宕机的clickhouse服务和zookeeper服务

    /usr/local/zookeeper/bin/zkServer.sh start
    /etc/rc.d/init.d/clickhouse-server start
    查看在vm-centos7-ck-0002机器上的内容是否自动同步过来到001机器,是同步的
    vm-centos7-ck-0001 :) select * from test001.ontime;
  • 小结:

    该集群,对与clickhouse集群服务有2个切片4个副本,宕掉任意一个切片的一个副本都可以正常运行,可以实现高可用。但是同时宕掉一个切片的的两个副本就不能正常运行。对于zookeeper集群,正常运行可以允许1台服务器挂掉。

扩容

扩容节点(clickhouse分片扩容)

  • 增加两台机器扩容

    ##增加的机器,做分片3
    192.168.4.13  vm-centos7-ck-0005
    192.168.4.14  vm-centos7-ck-0006
  • 每个节点上的新的hosts文件

    cat >/etc/hosts <
  • 在005/006新节点上安装clickhouse-server

  • 在005/006新节点的集群配置文件,新节点的权重设大点

    ##注意配置config.xml的192.168.4.13  
    ##配置metrika.xml需要修改的
    cat > /etc/clickhouse-server/metrika.xml << EOF
    
      
      
      
    
      
      
      true
    
      
      vm-centos7-ck-0001
      9000
      default
      M4XZ5biK
      
    
      
      vm-centos7-ck-0002
      9000
      default
      M4XZ5biK
      
      
    
      
      
      true
    
      
      vm-centos7-ck-0003
      9000
      default
      M4XZ5biK
      
    
      
      vm-centos7-ck-0004
      9000
      default
      M4XZ5biK
      
      
    
      
      
      3
      true
    
      
      vm-centos7-ck-0005
      9000
      default
      M4XZ5biK
      
    
      
      vm-centos7-ck-0006
      9000
      default
      M4XZ5biK
      
      
    
      
      
    
      
      
      
      vm-centos7-ck-0001
      2181
      
      
      vm-centos7-ck-0002
      2181
      
      
      vm-centos7-ck-0003
      2181
      
      
      
      01
      05 
      vm-centos7-ck-0005 
      
      
      ::/0
      
      
      
      
      10000000000
      0.01
      lz4 
      
      
      
    EOF
  • 启动新节点005/006的clickhouse-server

    /etc/rc.d/init.d/clickhouse-server start
  • 创建本地表和分布式表

    # clickhouse-client -h 127.0.0.1 -d default -m -u default --password M4XZ5biK
    :) create database test001; #005/006节点执行
    :) CREATE TABLE test001.ontime (FlightDate Date,Year UInt16) ENGINE = ReplicatedMergeTree('/clickhouse/test001/ontime/shard3','1',FlightDate,(Year, FlightDate),8192); #分片3的第1个副本执行
    :) CREATE TABLE test001.ontime (FlightDate Date,Year UInt16) ENGINE = ReplicatedMergeTree('/clickhouse/test001/ontime/shard3','2',FlightDate,(Year, FlightDate),8192); #分片3的第2个副本执行
    :) CREATE TABLE test001.ontime_all (FlightDate Date,Year UInt16) ENGINE= Distributed(clickhouse_cluster_name, test001, ontime, rand()); #005/006节点执行
    
    ##此时可以再005/006机器上查看分布式表,但是新节点还没有和其他旧节点打通
    select * from  test001.ontime_all;
    
    ##解析:这一步必须再配置旧节点之前否则会报错
  • 配置旧节点metrika.xml,配置完就直接生效的

    ##/etc/clickhouse-server/metrika.xml 加上新节点的分片
    
      
      true
    
      
      vm-centos7-ck-0005
      9000
      default
      M4XZ5biK
      
    
      
      vm-centos7-ck-0006
      9000
      default
      M4XZ5biK
      
      
  • 扩容测试

    在分片3的第1个本地表插入数据
    vm-centos7-ck-0005 :) insert into test001.ontime (FlightDate,Year)values('2022-8-12',2022);
    在分片3的第2个查看本地表
    vm-centos7-ck-0006 :)  select * from  test001.ontime;
    在每个节点上查询分布式表
    vm-centos7-ck-0001 :)  select * from  test001.ontime_all;
    
    插入分布式表数据测试:
    说明:在任意机器上插入数据到分布式表的时候,分布式表会把插入的信息轮询到分片1、分片2、分片3的机器的本地表
    vm-centos7-ck-0001 :) insert into test001.ontime_all (FlightDate,Year)values('2023-1-12',2023);
    

扩容目录

  • 直接用lvm扩容数据目录大小

笔记阁 , 版权所有丨如未注明 , 均为原创丨本网站采用BY-NC-SA协议进行授权
转载请注明原文链接:实操,ClickHouse高可用集群部署
喜欢 (2)
发表我的评论
取消评论
表情 加粗 删除线 居中 斜体 签到

Hi,您需要填写昵称和邮箱!

  • 昵称 (必填)
  • 邮箱 (必填)
  • 网址