通过tunasync搭建镜像仓库

当前以CentOS7系统为例，搭建一个基于内网的镜像仓库，建议硬盘至少2T以上，同步速度与网络带宽、磁盘读写速度有关！

1、安装依赖

yum install rsync wget

wget https://github.com/tuna/tunasync/releases/download/v0.3.7/tunasync-linux-bin.tar.gz
tar xf tunasync-linux-bin.tar.gz
mv tunasync /usr/bin/
mv tunasynctl /usr/bin/

mkdir -p /data/mirrors

mkdir -p /data/logs/tunasync

mkdir /etc/tunasync

vi /etc/tunasync/manager.conf

debug = false

[server]
addr = "127.0.0.1"
port = 12345
ssl_cert = ""
ssl_key = ""

[files]
db_type = "bolt"
db_file = "/etc/tunasync/manager.db"
ca_cert = ""

addr：manager服务监听地址
port：manager服务监听端口
ssl_cert和ssl_key：证书配置
db_file：数据库文件位置

7、创建worker.conf配置文件

vi /etc/tunasync/worker.conf

[global]
name = "test_worker"
log_dir = "/data/logs/tunasync/{{.Name}}"
mirror_dir = "/data/mirrors"
concurrent = 10
interval = 240

[manager]
api_base = "http://127.0.0.1:12345"
token = ""
ca_cert = ""

[cgroup]
enable = false
base_path = "/sys/fs/cgroup"
group = "tunasync"

[server]
hostname = "localhost"
listen_addr = "127.0.0.1"
listen_port = 6000
ssl_cert = ""
ssl_key = ""

[[mirrors]]
name = "centos"
provider = "rsync"
upstream = "rsync://mirrors.tuna.tsinghua.edu.cn/centos/"
use_ipv6 = false

[[mirrors]]
name = "epel"
provider = "rsync"
upstream = "rsync://mirrors.tuna.tsinghua.edu.cn/epel/"
use_ipv6 = false

log_dir：tunasync日志路径
mirror_dir：镜像的路径
concurrent：线程数
interval：同步周期，单位分钟
api_base：manager地址
name：镜像名称
upstream：upstream同步地址

8、下载tunasync启动停止脚本

wget -P /etc/init.d/ https://raw.githubusercontent.com/whsir/tunasync-bin/master/tunasync-manager
wget -P /etc/init.d/ https://raw.githubusercontent.com/whsir/tunasync-bin/master/tunasync-worker
chmod +x /etc/init.d/tunasync-manager
chmod +x /etc/init.d/tunasync-worker

/etc/init.d/tunasync-manager start
/etc/init.d/tunasync-worker start

此处12345为manager的端口

tunasynctl list -p 12345 --all

/1    * wget -q http://127.0.0.1:12345/jobs -O /data/mirrors/jobs.json

rpm -ivh http://mirrors.wlnmp.com/centos/wlnmp-release-centos.noarch.rpm
yum install wnginx

vi /usr/local/nginx/conf/vhost/demo.conf

server

{
    listen 80 default_server;
    server_name _;
root /data/mirrors;
location ~ ^/(centos|epel) {
      index  index.html index.htm;
      autoindex on;
      autoindex_exact_size off;
      autoindex_localtime on;
    }

    error_page   500 502 503 504  /50x.html;

    #include enable-php56.conf;
    #include enable-php70.conf;
    #include enable-php71.conf;
    #include enable-php72.conf;
    #include enable-php73.conf;
#include enable-php74.conf;

    location ~ .*\.(gif|jpg|jpeg|png|bmp|swf)$
    {
        expires      30d;
    }

    location ~ .*\.(js|css)?$
    {
        expires      12h;
    }

    access_log  /data/logs/nginx/demo_access.log;
error_log /data/logs/nginx/demo_error.log;
}</pre>

14、设置前端web

cd /data/mirrors
wget http://down.whsir.com/downloads/mirrors_web.tar.gz

/etc/init.d/nginx restart

访问URL即可看到以下内容，由于这两个源很大，同步时间会非常长

附：我这里再举个例子，基于docker-ce镜像同步

镜像同步脚本：https://github.com/tuna/tunasync-scripts

1、创建镜像同步脚本目录

mkdir /etc/tunasync/scripts/
cd /etc/tunasync/scripts/

wget https://raw.githubusercontent.com/tuna/tunasync-scripts/master/docker-ce.py
chmod +x docker-ce.py

vi /etc/tunasync/worker.conf

[[mirrors]]
name = "docker-ce"
provider = "command"
upstream = "https://download.docker.com/"
command = "/etc/tunasync/scripts/docker-ce.py"
docker_image = "tunathu/tunasync-scripts:latest"

yum install python3 python-pip
pip3 install requests pyquery

/etc/init.d/tunasync-manager start
/etc/init.d/tunasync-worker start

在location中继续增加一个docker-ce

vi /usr/local/nginx/conf/vhost/demo.conf

server

{
    listen 80 default_server;
    server_name _;
root /data/mirrors;
location ~ ^/(centos|epel|docker-ce) {
      index  index.html index.htm;
      autoindex on;
      autoindex_exact_size off;
      autoindex_localtime on;
    }

    error_page   500 502 503 504  /50x.html;

    #include enable-php56.conf;
    #include enable-php70.conf;
    #include enable-php71.conf;
    #include enable-php72.conf;
    #include enable-php73.conf;
#include enable-php74.conf;

    location ~ .*\.(gif|jpg|jpeg|png|bmp|swf)$
    {
        expires      30d;
    }

    location ~ .*\.(js|css)?$
    {
        expires      12h;
    }

    access_log  /data/logs/nginx/demo_access.log;
error_log /data/logs/nginx/demo_error.log;
}</pre>

7、重启nginx

/etc/init.d/nginx restart

前端web都是基于html写的，如果继续添加其他的源，可以自行更改

写在最后

手动同步方式

rsync -aHvh --no-o --no-g --stats --exclude .~tmp~/ --delete --delete-after --delay-updates --safe-links --timeout=120 --contimeout=120 rsync://mirrors.tuna.tsinghua.edu.cn/epel/ /data/mirrors/epel

rsync -aHvh --no-o --no-g --stats --exclude .~tmp~/ --delete --delete-after --delay-updates --safe-links --timeout=120 --contimeout=120 rsync://mirrors.tuna.tsinghua.edu.cn/centos/ /data/mirrors/centos

python3 /etc/tunasync/scripts/docker-ce.py --working-dir /data/mirrors/docker-ce/

rsync: readlink_stat("7.7.1908/isos/x86_64/.CentOS-7-x86_64-Everything-1908.iso.RjFDl5" (in centos)) failed: Permission denied (13)

可以考虑将centos源更换香港这个源：rsync://mirror.hostlink.com.hk/centos/