一、前言
前几天有幸接触了MinIO,不知所以然,经过查询MinIO是一个对象存储。究竟有什么魅力,让这么多人对它情有独钟。
二、MinIO概述
MinIO 是在 Apache License v2.0 下发布的高性能对象存储。它是与 Amazon S3 云存储服务兼容的 API。使用 MinIO 构建 用于机器学习、分析和应用的高性能基础设施数据工作负载。MinIO 从根本上与众不同,专为企业和私有云设计。MinIO生产部署涵盖了全球。MinIO是全球使用最多和下载量最大的对象存储服务系统,还是全世界增长最快的对象存储系统。
官方文档:http://docs.minio.org.cn/docs/(中文文档)
主要有以下几个特点:
三、MinIO单机部署(docker 单节点单驱动)
MinIO强烈推荐生产集群由 minimum 4 个组成 minio server 节点在服务器池。
3.1 准备工作
$ mkdir -p /data/minio/data $ mkdir -p /data/minio/config $ docker run -d \ -p 9000:9000 \ -p 9090:9090 \ --name minio \ -v /data/data:/data \ -v /data/minio/config:/root/.minio \ -e "MINIO_ROOT_USER=minioadmin" \ -e "MINIO_ROOT_PASSWORD=minioadmin" \ quay.io/minio/minio:RELEASE.2022-09-25T15-44-53Z server /data --console-address ":9090
参数说明:
- docker run启动 MinIO 容器
- -p将本地端口绑定到容器端口
- -name为容器创建一个名称
- 9000表示MinIO服务地址,其上传调用的就是这个服务地址
- 9090表示MinIO的Web Console地址,Console监听的是一个动态的端口, 可以通过 --console-address ":port" 指定静态端口
- /mnt/data/minio/data:/data表示将MiniIO的数据挂载到宿主机上
- /mnt/data/minio/config:/root/.minio表示将MiniIO的配置文件挂载到宿主机上
- MINIO_ROOT_USER=minioadmin表示MinIO部署的root用户的用户名(accessKey),不写默认的用户名就是minioadmin
- MINIO_ROOT_PASSWORD=minioadmin表示MinIO部署的root用户的密码(secretKey),不写默认的密码就是minioadmin
3.2 部署
提前拉取镜像,docker pull quay.io/minio/minio:latest 这里采用最新稳定版镜像。
$ docker run \ > -p 9000:9000 \ > -p 9090:9090 \ > --name minio \ > -v /data/data:/data \ > -v /data/minio/config:/root/.minio \ > -e "MINIO_ROOT_USER=minioadmin" \ > -e "MINIO_ROOT_PASSWORD=minioadmin" \ > quay.io/minio/minio:RELEASE.2022-09-25T15-44-53Z server /data --console-address ":9090" Unable to find image 'quay.io/minio/minio:latest' locally latest: Pulling from minio/minio d5d2e87c6892: Pull complete 008dba906bf6: Pull complete e632dfd7f0e0: Pull complete bb2a22a09061: Pull complete 19b7bd2519a4: Pull complete a2d565af0f6d: Pull complete f0067029ce19: Downloading [============> ] 10.65MB/42.53MB
安装成功后如下:
$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 1aa0bf931300 quay.io/minio/minio "/usr/bin/docker-ent…" 6 seconds ago Up 5 seconds 0.0.0.0:9000->9000/tcp, :::9000->9000/tcp, 0.0.0.0:9090->9090/tcp, :::9090->9090/tcp minio [root@xiaohezi ~]#
3.3 登录Web Console
浏览器中输入http://ip:9090/,用户名密码登录系统。
四、MinIO单机部署(二进制单节点单驱动)
4.1 创建数据目录
$ mkdir /data/minio/data -p
4.2 下载二进制包
下载地址:https://www.minio.org.cn/download.shtml#/linux
$ wget https://dl.min.io/server/minio/release/linux-amd64/minio $ chmod +x minio $ mv minio /usr/local/bin/ $ minio --version minio version RELEASE.2022-09-25T15-44-53Z (commit-id=877bd95fa312c5282c3aa0b73c75af43af9c5914) Runtime: go1.18.6 linux/amd64 License: GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html> Copyright: 2015-2022 MinIO, Inc.
4.3 设置环境变量
配置用户名密码
$ export MINIO_ROOT_USER=minioadmin $ export MINIO_ROOT_PASSWORD=minioadmin
4.4 创建服务启动文件(可选)
vim /usr/lib/systemd/system/minio.service [Unit] Description=Minio service Documentation=https://docs.minio.io/ [Service] WorkingDirectory=/data/minio/run/ ExecStart=/usr/local/bin/minio --console-address ":9090" /data/minio/data Restart=on-failure RestartSec=5 [Install] WantedBy=multi-user.target
4.5 启动
$ minio server --console-address ":9090" /data/minio/data 或者: $ systemctl daemon-reload $ systemctl start minio.service && systemctl status minio.service $ systemctl enable minio.service
$ minio server --console-address ":9090" /data/minio/data WARNING: Detected Linux kernel version older than 4.0.0 release, there are some known potential performance problems with this kernel version. MinIO recommends a minimum of 4.x.x linux kernel version for best performance Formatting 1st pool, 1 set(s), 1 drives per set. WARNING: Host local has more than 0 drives of set. A host failure will result in data becoming unavailable. WARNING: Detected default credentials 'minioadmin:minioadmin', we recommend that you change these values with 'MINIO_ROOT_USER' and 'MINIO_ROOT_PASSWORD' environment variables MinIO Object Storage Server Copyright: 2015-2022 MinIO, Inc. License: GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html> Version: RELEASE.2022-09-25T15-44-53Z (go1.18.6 linux/amd64) Status: 1 Online, 0 Offline. API: http://192.168.0.4:9000 http://172.17.0.1:9000 http://127.0.0.1:9000 RootUser: minioadmin RootPass: minioadmin Console: http://192.168.0.4:9090 http://172.17.0.1:9090 http://127.0.0.1:9090 RootUser: minioadmin RootPass: minioadmin Command-line: https://docs.min.io/docs/minio-client-quickstart-guide $ mc alias set myminio http://192.168.0.4:9000 minioadmin minioadmin Documentation: https://docs.min.io
通过上面可以看到用户名密码为minioadmin,默认启动的API端口是9000,API 默认端口可通过--address IP:PORT来指定;还有一个Web Console的端口,并且Console监听的是一个动态的端口, 可以通过 --console-address ":port" 指定静态端口。
默认的配置目录是{HOME}/.minio,可以通过 --config-dir 命令自定义配置目录
$ minio server -config-dir /data/minio/config --console-address ":9090" --address ":9000" /data/minio/data
后台启动
nohup minio server -config-dir /data/minio/config --console-address ":9090" --address ":9000" /data/minio/data >/data/minio/minio.log 2>&1 &
五、MinIO单机部署(docker-compose
单节点单驱动)
5.1 准备工作
编写docker-compose-minio.yaml文件,内容如下:
version: '3.7' services: minio: image: quay.io/minio/minio:RELEASE.2022-09-25T15-44-53Z container_name: minio hostname: minio restart: always command: server --console-address ":9090" /data environment: MINIO_ROOT_USER: minioadmin MINIO_ROOT_PASSWORD: minioadmin TZ: Asia/Shanghai volumes: - /data/minio/data:/data - /data/minio/config:/root/.minio ports: - "9000:9000" - "9090:9090" networks: - pk_net healthcheck: test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"] interval: 30s timeout: 20s retries: 3 networks: pk_net: external: true
自定义docker网络
### 预先创建一个自定义的网络pk_net,此处的10.139可以自定义,不冲突即可 $ sudo docker network create --driver bridge --subnet 10.139.0.0/16 --gateway 10.139.0.1 pk_net
参数说明:healthcheck表示健康探测 server --console-address ":9090" /data 表示指定的minio服务下面挂载的目标磁盘为/data,并且指定Web Console的端口
5.2 部署
$ docker-compose -f docker-compose-minio.yaml up -d
六、MinIO单机部署(docker-compose
单节点多驱动)
6.1 准备工作
挂载的磁盘增加到四个,单机版部署也可挂载多个磁盘,单个服务挂载超过(等于)4个磁盘,自动启动纠删码模式,可以预防磁盘损坏的情况下,导致文件不丢失。
调整docker-compose-minio.yaml文件,内容如下:
version: '3.7' services: minio: image: quay.io/minio/minio:RELEASE.2022-09-25T15-44-53Z container_name: minio hostname: minio restart: always command: server --console-address ":9090" http://minio/data{1...4} environment: MINIO_ROOT_USER: minioadmin MINIO_ROOT_PASSWORD: minioadmin TZ: Asia/Shanghai volumes: - /data/minio/data1:/data1 - /data/minio/data2:/data2 - /data/minio/data3:/data3 - /data/minio/data4:/data4 - /data/minio/config:/root/.minio ports: - "9000:9000" - "9090:9090" networks: - pk_net healthcheck: test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"] interval: 30s timeout: 20s retries: 3 networks: pk_net: external: true
参数说明:healthcheck表示健康探测 http://minio/data{1...4} 表示指定的minio服务下面挂载的目标磁盘为/data1、/data2、/data3和/data4 单机版部署也可挂载多个磁盘,单个服务挂载超过(等于)4个磁盘,自动启动纠删码模式,可以预防磁盘损坏的情况下,导致文件不丢失。
6.2 部署
$ docker-compose -f docker-compose-minio.yaml up -d
七、MinIO分布式部署(docker-compose模拟
多节点多驱动)
分布式 Minio 可以让你将多块硬盘(甚至是在不同的机器上)组成一个对象存储服务。由于硬盘分布在不同的节点上,分布式 Minio 避免了单点故障。
数据保护:分布式 Minio 采用纠删码来防范多个节点宕机和位衰减 bit rot。分布式 Minio 至少需要四块硬盘,使用分布式 Minio 自动引入了纠删码功能。MinIO 使用纠删码 erasure code 和校验和 checksum 来保护数据免受硬件故障和无声数据损坏。即便您丢失一半数量 (N/2) 的硬盘,依然可以使用。纠删码是一种恢复丢失和损坏数据的数学算法,MinIO 采用 Reed-Solomon code 将对象拆分成 N/2 数据和 N/2 奇偶校验块。这就意味着如果是 12 块硬盘,一个对象会被分成 6 个数据块、6 个奇偶校验块,你可以丢失任意 6 块(不管其是否存放的数据还是奇偶校验块),你仍可以从剩下的盘中的数据进行恢复。
高可用:单机节点存在单点故障,相反,如果是一个有 N 块硬盘的分布式 MinIO,只要有 N/2 块硬盘在线,你的数据就是安全的。不过你需要有 N/2 + 1 个硬盘来创建新的对象。例如,一个 16 个节点的 MinIO 集群,每个节点 16 块硬盘,就算 8 台服务器宕机,这个集群仍然是可读的,不过需要有9台服务器才能写数据
一致性:MinIO 在分布式和单机模式下,所有读写操作都严格遵守 read-adter-write 一致性模型
7.1 准备工作
调整docker-compose-minio.yaml文件,内容如下:
version: '3.7' # Settings and configurations that are common for all containers x-minio-common: &minio-common image: quay.io/minio/minio:RELEASE.2022-09-25T15-44-53Z command: server --console-address ":9090" http://minio{1...4}/data{1...4} expose: - "9000" - "9090" environment: MINIO_ROOT_USER: minioadmin MINIO_ROOT_PASSWORD: minioadmin TZ: Asia/Shanghai healthcheck: test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"] interval: 30s timeout: 20s retries: 3 networks: - pk_net # 启动4个minio服务器实例的docker容器 # 使用nginx反向代理,负载均衡,通过端口9000可以访问 # it through port 9000. services: minio1: <<: *minio-common hostname: minio1 container_name: minio1 volumes: - /data1/data1:/data1 - /data1/data2:/data2 - /data1/data3:/data3 - /data1/data4:/data4 minio2: <<: *minio-common hostname: minio2 container_name: minio2 volumes: - /data2/data1:/data1 - /data2/data2:/data2 - /data2/data3:/data3 - /data2/data4:/data4 minio3: <<: *minio-common hostname: minio3 container_name: minio3 volumes: - /data3/data1:/data1 - /data3/data2:/data2 - /data3/data3:/data3 - /data3/data4:/data4 minio4: <<: *minio-common hostname: minio4 container_name: minio4 volumes: - /data4/data1:/data1 - /data4/data2:/data2 - /data4/data3:/data3 - /data4/data4:/data4 nginx: image: nginx:1.19.2-alpine hostname: nginx container_name: nginx volumes: - ./nginx.conf:/etc/nginx/nginx.conf:ro ports: - "9000:9000" - "9090:9090" depends_on: - minio1 - minio2 - minio3 - minio4 networks: pk_net: external: true
参数说明:x-minio-common表示所有容器的设置和配置 启动4个minio服务器实例的docker容器 使用nginx反向代理,负载均衡,通过端口9000可以访问
nginx.conf内容如下:
user nginx; worker_processes auto; error_log /var/log/nginx/error.log warn; pid /var/run/nginx.pid; events { worker_connections 4096; } http { include /etc/nginx/mime.types; default_type application/octet-stream; log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; access_log /var/log/nginx/access.log main; sendfile on; keepalive_timeout 65; # include /etc/nginx/conf.d/*.conf; upstream minio { server minio1:9000; server minio2:9000; server minio3:9000; server minio4:9000; } upstream console { ip_hash; server minio1:9090; server minio2:9090; server minio3:9090; server minio4:9090; } server { listen 9000; listen [::]:9000; server_name localhost; # To allow special characters in headers ignore_invalid_headers off; # Allow any size file to be uploaded. # Set to a value such as 1000m; to restrict file size to a specific value client_max_body_size 0; # To disable buffering proxy_buffering off; proxy_request_buffering off; location / { proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_connect_timeout 300; # Default is HTTP/1, keepalive is only enabled in HTTP/1.1 proxy_http_version 1.1; proxy_set_header Connection ""; chunked_transfer_encoding off; proxy_pass http://minio; } } server { listen 9090; listen [::]:9090; server_name localhost; # To allow special characters in headers ignore_invalid_headers off; # Allow any size file to be uploaded. # Set to a value such as 1000m; to restrict file size to a specific value client_max_body_size 0; # To disable buffering proxy_buffering off; proxy_request_buffering off; location / { proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-NginX-Proxy true; # This is necessary to pass the correct IP to be hashed real_ip_header X-Real-IP; proxy_connect_timeout 300; # To support websocket proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; chunked_transfer_encoding off; proxy_pass http://console; } } }
7.2 部署
$ docker-compose -f docker-compose-minio.yaml up -d Creating minio4 ... done Creating minio2 ... done Creating minio3 ... done Creating minio1 ... done Creating test_nginx_1 ... done $ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES a6bda0af2d24 nginx:1.19.2-alpine "/docker-entrypoint.…" 7 seconds ago Up 6 seconds 0.0.0.0:9000->9000/tcp, :::9000->9000/tcp, 80/tcp, 0.0.0.0:9090->9090/tcp, :::9090->9090/tcp test_nginx_1 239c4bdeb151 quay.io/minio/minio:RELEASE.2022-09-25T15-44-53Z "/usr/bin/docker-ent…" 9 seconds ago Up 6 seconds (health: starting) 9000/tcp, 9090/tcp minio1 0c5050be7fd3 quay.io/minio/minio:RELEASE.2022-09-25T15-44-53Z "/usr/bin/docker-ent…" 9 seconds ago Up 6 seconds (health: starting) 9000/tcp, 9090/tcp minio2 e65601ce11c0 quay.io/minio/minio:RELEASE.2022-09-25T15-44-53Z "/usr/bin/docker-ent…" 9 seconds ago Up 7 seconds (health: starting) 9000/tcp, 9090/tcp minio3 f75733c2da78 quay.io/minio/minio:RELEASE.2022-09-25T15-44-53Z "/usr/bin/docker-ent…" 9 seconds ago Up 7 seconds (health: starting) 9000/tcp, 9090/tcp minio4
八、MinIO分布式部署(多节点多驱动)--生产推荐
在单机上部署只能保证磁盘损坏的情况下,文件不丢失,并不能解决单点故障的问题,所以我们下面为了避免单点故障导致服务不可用,把minio服务改成真正分布式部署。MinIO强烈推荐生产集群由 minimum 4 个组成 minio server 节点在服务器池。可以采用docker-compose部署,也可采用二进制部署,最后使用nginx进行负载均衡就行了。这里我们还是采用docker-compose为例。
8.1 准备工作
服务器1-4如下:
IP | 部署服务 | 备注 | |
服务器1 | 192.168.0.1 | minio | 挂载4块磁盘 |
服务器2 | 192.168.0.2 | minio | 挂载4块磁盘 |
服务器3 | 192.168.0.3 | minio | 挂载4块磁盘 |
服务器4 | 192.168.0.4 | minio | 挂载4块磁盘 |
服务器5 | 192.168.0.5 | nginx | nginx进行负载均衡 |
docker-compose-minio.yaml文件,内容如下:
version: '3.7' services: minio: image: quay.io/minio/minio:RELEASE.2022-09-25T15-44-53Z container_name: minio hostname: minio restart: always command: server --console-address ":9090" http://192.168.0.1:9000/data{1...4} environment: MINIO_ROOT_USER: minioadmin MINIO_ROOT_PASSWORD: minioadmin TZ: Asia/Shanghai volumes: - /minio1/data1:/data1 - /minio2/data2:/data2 - /minio3/data3:/data3 - /minio4/data4:/data4 - /data/minio/config:/root/.minio ports: - "9000:9000" - "9090:9090" networks: - pk_net healthcheck: test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"] interval: 30s timeout: 20s retries: 3 networks: pk_net: external: true
服务器1-4都采用以上docker-compose文件部署。/minio1/data1、/minio2/data2、/minio3/data2、/minio4/data2表示分别挂载的四块磁盘
docker-compose-nginx.yaml文件,内容如下:
version: '3.7' services: nginx: image: nginx:1.19.2-alpine hostname: nginx container_name: nginx restart: always environment: TZ: Asia/Shanghai volumes: - ./nginx.conf:/etc/nginx/nginx.conf:ro ports: - "9000:9000" - "9090:9090" networks: - pk_net networks: pk_net: external: true
nginx.conf内容如下:
user nginx; worker_processes auto; error_log /var/log/nginx/error.log warn; pid /var/run/nginx.pid; events { worker_connections 4096; } http { include /etc/nginx/mime.types; default_type application/octet-stream; log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; access_log /var/log/nginx/access.log main; sendfile on; keepalive_timeout 65; # include /etc/nginx/conf.d/*.conf; upstream minio { server 192.168.0.1:9000; server 192.168.0.2:9000; server 192.168.0.3:9000; server 192.168.0.4:9000; } upstream console { ip_hash; server 192.168.0.1:9090; server 192.168.0.2:9090; server 192.168.0.3:9090; server 192.168.0.4:9090; } server { listen 9000; listen [::]:9000; server_name localhost; # To allow special characters in headers ignore_invalid_headers off; # Allow any size file to be uploaded. # Set to a value such as 1000m; to restrict file size to a specific value client_max_body_size 0; # To disable buffering proxy_buffering off; proxy_request_buffering off; location / { proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_connect_timeout 300; # Default is HTTP/1, keepalive is only enabled in HTTP/1.1 proxy_http_version 1.1; proxy_set_header Connection ""; chunked_transfer_encoding off; proxy_pass http://minio; } } server { listen 9090; listen [::]:9090; server_name localhost; # To allow special characters in headers ignore_invalid_headers off; # Allow any size file to be uploaded. # Set to a value such as 1000m; to restrict file size to a specific value client_max_body_size 0; # To disable buffering proxy_buffering off; proxy_request_buffering off; location / { proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-NginX-Proxy true; # This is necessary to pass the correct IP to be hashed real_ip_header X-Real-IP; proxy_connect_timeout 300; # To support websocket proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; chunked_transfer_encoding off; proxy_pass http://console; } } }
8.2 部署
分别部署四台minio服务
$ docker-compose -f docker-compose-minio.yaml up -d
部署nginx服务
$ docker-compose -f docker-compose-nginx.yaml up -d
九、优化
nginx也存在单节点故障,可以进行高可用配置。
非云上:
- 使用 Keepalived 和 HAproxy 负载minio
- 使用 Keepalived 和 Nginx 负载minio
云上:
- 使用 Keepalived 和 HAproxy 负载minio
- 使用 Keepalived 和 Nginx 负载minio
- 可以直接使用云上的lb,比如阿里云slb,腾讯云elb,青云lb等负载minio(推荐)
ps:有些云可能不支持keepalived