[#37] 测试通过swarm跨机部署节点;更新文档
This commit is contained in:
parent
b9611c2dd2
commit
38f8ee9301
@ -68,20 +68,23 @@ out_name="argus-metric_$(echo "$VERSION" | sed 's/\./_/g').tar.gz"
|
|||||||
log "Client package ready: $PKG_DIR/$out_name"
|
log "Client package ready: $PKG_DIR/$out_name"
|
||||||
echo "$VERSION" > "$PKG_DIR/LATEST_VERSION"
|
echo "$VERSION" > "$PKG_DIR/LATEST_VERSION"
|
||||||
|
|
||||||
# include publish helper and setup.sh for convenience
|
# include publish helper and setup.sh for convenience (place first)
|
||||||
PUBLISH_TPL="$BUILD_DIR/templates/client/publish.sh"
|
PUBLISH_TPL="$BUILD_DIR/templates/client/publish.sh"
|
||||||
if [[ -f "$PUBLISH_TPL" ]]; then
|
if [[ -f "$PUBLISH_TPL" ]]; then
|
||||||
cp "$PUBLISH_TPL" "$PKG_DIR/publish.sh" && chmod +x "$PKG_DIR/publish.sh"
|
cp "$PUBLISH_TPL" "$PKG_DIR/publish.sh"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
# also place a copy of setup.sh alongside
|
# also place a copy of setup.sh alongside
|
||||||
SETUP_SRC="$ROOT_DIR/src/metric/client-plugins/all-in-one-full/scripts/setup.sh"
|
SETUP_SRC="$ROOT_DIR/src/metric/client-plugins/all-in-one-full/scripts/setup.sh"
|
||||||
[[ -f "$SETUP_SRC" ]] && cp "$SETUP_SRC" "$PKG_DIR/setup.sh" || true
|
[[ -f "$SETUP_SRC" ]] && cp "$SETUP_SRC" "$PKG_DIR/setup.sh" || true
|
||||||
|
|
||||||
# docs for end users
|
# docs for end users (this may overwrite file modes), then fix execute bits
|
||||||
CLIENT_DOC_DIR="$BUILD_DIR/templates/client"
|
CLIENT_DOC_DIR="$BUILD_DIR/templates/client"
|
||||||
if [[ -d "$CLIENT_DOC_DIR" ]]; then
|
if [[ -d "$CLIENT_DOC_DIR" ]]; then
|
||||||
rsync -a "$CLIENT_DOC_DIR/" "$PKG_DIR/" >/dev/null 2>&1 || cp -r "$CLIENT_DOC_DIR/." "$PKG_DIR/"
|
rsync -a "$CLIENT_DOC_DIR/" "$PKG_DIR/" >/dev/null 2>&1 || cp -r "$CLIENT_DOC_DIR/." "$PKG_DIR/"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
# ensure helpers are executable
|
||||||
|
chmod +x "$PKG_DIR/publish.sh" "$PKG_DIR/setup.sh" 2>/dev/null || true
|
||||||
|
|
||||||
exit 0
|
exit 0
|
||||||
|
|||||||
@ -25,6 +25,13 @@
|
|||||||
- `sudo /tmp/setup.sh --server <FTP_IP> --user ftpuser --password 'ZGClab1234!' --port 21`
|
- `sudo /tmp/setup.sh --server <FTP_IP> --user ftpuser --password 'ZGClab1234!' --port 21`
|
||||||
- 如需自定义安装根目录:`--install-dir /opt/argus-metric`
|
- 如需自定义安装根目录:`--install-dir /opt/argus-metric`
|
||||||
|
|
||||||
|
提示(容器接入 overlay 网络时):
|
||||||
|
- 在执行 setup 前,先将容器内 DNS 指向 Bind9 的 overlay IP:
|
||||||
|
- `echo "nameserver <BIND_OVERLAY_IP>" > /etc/resolv.conf`
|
||||||
|
- 这样 `master.argus.com`、`es.log.argus.com` 等域名即可解析;首次下载 `setup.sh` 仍建议使用 FTP 的 overlay IP。
|
||||||
|
|
||||||
|
更多快速步骤请参考:`QUICK_NODE_DEPLOY_zh.md`。
|
||||||
|
|
||||||
## 安装后自检(setup 自动执行)
|
## 安装后自检(setup 自动执行)
|
||||||
- setup 会等待最多 5 分钟,确认以下条件后才报告完成:
|
- setup 会等待最多 5 分钟,确认以下条件后才报告完成:
|
||||||
- `/private/argus/agent/<hostname>/node.json` 已生成;
|
- `/private/argus/agent/<hostname>/node.json` 已生成;
|
||||||
|
|||||||
57
deployment/build/templates/client/PUBLISH_CLIENT_zh.md
Normal file
57
deployment/build/templates/client/PUBLISH_CLIENT_zh.md
Normal file
@ -0,0 +1,57 @@
|
|||||||
|
# Argus Metric 客户端发布说明(FTP)
|
||||||
|
|
||||||
|
本说明面向“发布人员”,讲清楚如何把客户端离线包发布到 FTP,供各节点通过 `curl` 自动安装。
|
||||||
|
|
||||||
|
## 目录结构(构建后)
|
||||||
|
- `client-YYYYMMDD/`
|
||||||
|
- `argus-metric_YYYYMMDD.tar.gz` 客户端离线包
|
||||||
|
- `setup.sh` 客户端安装入口脚本(提供给节点用 curl 下载)
|
||||||
|
- `publish.sh` 发布脚本(将上述两项与 `LATEST_VERSION` 上传到 FTP)
|
||||||
|
- `LATEST_VERSION` 文本(内容为 `YYYYMMDD`,或 `YYYYMMDD-rN`)
|
||||||
|
- `INSTALL_CLIENT_zh.md` 本地安装指南(给使用者看,不会上载到 FTP)
|
||||||
|
- `PUBLISH_CLIENT_zh.md` 本说明
|
||||||
|
|
||||||
|
> 注意:`publish.sh`/`setup.sh` 为可执行脚本;构建脚本已保证二者具有执行权限。
|
||||||
|
|
||||||
|
## 前置条件
|
||||||
|
- FTP 服务已运行(默认容器:`argus-ftp`),并打开端口:21、20、21100–21110(被动模式)。
|
||||||
|
- FTP 账号:默认 `ftpuser / ZGClab1234!`(如有更改,以实际为准)。
|
||||||
|
|
||||||
|
## 发布步骤(在 server 机器或能直连 FTP 的任意机器上)
|
||||||
|
1) 进入发布目录:
|
||||||
|
- `cd client-YYYYMMDD`
|
||||||
|
|
||||||
|
2) 执行发布:
|
||||||
|
- `./publish.sh --server <FTP_HOST> --user <USER> --password '<PASS>' [--port 21]`
|
||||||
|
- 例如在服务端本机:`./publish.sh --server localhost --user ftpuser --password 'ZGClab1234!' --port 21`
|
||||||
|
|
||||||
|
脚本会上传三类文件到 FTP 根:
|
||||||
|
- `setup.sh`
|
||||||
|
- `argus-metric_YYYYMMDD[ -rN ].tar.gz`
|
||||||
|
- `LATEST_VERSION`(内容为当前版本号)
|
||||||
|
|
||||||
|
3) 发布后验证:
|
||||||
|
- `curl -u ftpuser:****** -I ftp://<FTP_HOST>:21/LATEST_VERSION` 应返回 200
|
||||||
|
- `curl -u ftpuser:****** -fsSL ftp://<FTP_HOST>:21/LATEST_VERSION` 内容为版本号(如 `20251104`)
|
||||||
|
- `curl -u ftpuser:****** -I ftp://<FTP_HOST>:21/argus-metric_YYYYMMDD.tar.gz` 返回 200
|
||||||
|
|
||||||
|
## 节点侧使用方式(摘要)
|
||||||
|
- 首次下载用 FTP 的“IP 地址”:
|
||||||
|
- `curl -u ftpuser:****** -fsSL ftp://<FTP_IP>:21/setup.sh -o /tmp/setup.sh && chmod +x /tmp/setup.sh`
|
||||||
|
- 执行安装:
|
||||||
|
- 必需元数据:`AGENT_ENV/AGENT_USER/AGENT_INSTANCE`,以及 `MASTER_ENDPOINT=http://master.argus.com:3000`
|
||||||
|
- `sudo /tmp/setup.sh --server <FTP_IP> --user ftpuser --password '******' --port 21`
|
||||||
|
- overlay 容器场景:
|
||||||
|
- 先将容器内 DNS 指向 Bind9 的 overlay IP:`echo "nameserver <BIND_OVERLAY_IP>" > /etc/resolv.conf`
|
||||||
|
- 然后再执行上述安装;安装后约 1–2 分钟内 DNS 即可解析 `*.argus.com` 域名。
|
||||||
|
|
||||||
|
## 常见问题
|
||||||
|
- `530 Access denied`:用户名/密码错误或 FTP 目录无权限;请核对账号与 FTP 容器状态。
|
||||||
|
- `Permission denied` 执行 `publish.sh`:为脚本权限问题;`chmod +x publish.sh`。构建脚本已修复默认权限。
|
||||||
|
- 被动端口不通导致失败:请开放 21100–21110。
|
||||||
|
- 客户端安装后短时 `curl http://master.argus.com:3000` 为 000:服务冷启动或 DNS 同步延迟,等待 1–2 分钟再试。
|
||||||
|
|
||||||
|
## 版本与回滚
|
||||||
|
- `LATEST_VERSION` 决定客户端默认安装的版本号。
|
||||||
|
- 如需回滚:将旧版本号写回 `LATEST_VERSION` 并重新发布(或手动指定 `--version` 安装)。
|
||||||
|
|
||||||
58
deployment/build/templates/client/QUICK_NODE_DEPLOY_zh.md
Normal file
58
deployment/build/templates/client/QUICK_NODE_DEPLOY_zh.md
Normal file
@ -0,0 +1,58 @@
|
|||||||
|
# Argus Metric 节点快速部署(Overlay 网络容器)
|
||||||
|
|
||||||
|
本文档给出在 Docker Swarm external overlay 网络中,快速拉起一个测试节点并完成注册的最小可行步骤。
|
||||||
|
|
||||||
|
## 前提
|
||||||
|
- 服务端已在 Manager 机安装完成并运行良好(`server-selfcheck` 通过)。
|
||||||
|
- Overlay 网络名称:`argus-sys-net`(默认)。
|
||||||
|
- 已通过 FTP 发布 `setup.sh` 与客户端包,且能从 FTP 获取 `LATEST_VERSION`。
|
||||||
|
- 用于测试的镜像:`argus-sys-metric-test-node:latest` 已存在于目标机器。
|
||||||
|
|
||||||
|
## 步骤
|
||||||
|
|
||||||
|
- 获取 FTP 和 Bind 的 overlay IP(在 Manager 上执行)
|
||||||
|
- `FTPIP=$(docker inspect -f '{{ (index .NetworkSettings.Networks "argus-sys-net").IPAddress }}' argus-ftp)`
|
||||||
|
- `BINDIP=$(docker inspect -f '{{ (index .NetworkSettings.Networks "argus-sys-net").IPAddress }}' argus-bind-sys)`
|
||||||
|
- `echo "FTP=$FTPIP BIND=$BINDIP"`
|
||||||
|
|
||||||
|
- 准备宿主挂载目录(以 s4 为例)
|
||||||
|
- `mkdir -p /home2/yuyr/deploy/test-metric-node/s4`
|
||||||
|
|
||||||
|
- 启动测试节点容器(接入 overlay)
|
||||||
|
- `docker run -d --name argus-metric-test-node-s4 \
|
||||||
|
--hostname dev2-yuyr-node002s4 \
|
||||||
|
--network argus-sys-net \
|
||||||
|
-v /home2/yuyr/deploy/test-metric-node/s4:/private/argus/agent \
|
||||||
|
argus-sys-metric-test-node:latest sleep infinity`
|
||||||
|
|
||||||
|
- 在容器内执行安装(先用 FTP IP 引导,DNS 指向 Bind)
|
||||||
|
- `docker exec -it argus-metric-test-node-s4 bash`
|
||||||
|
- `echo "nameserver $BINDIP" > /etc/resolv.conf`
|
||||||
|
- `curl --ftp-method nocwd -u ftpuser:ZGClab1234! -fsSL "ftp://$FTPIP:21/setup.sh" -o /tmp/setup.sh`
|
||||||
|
- `chmod +x /tmp/setup.sh`
|
||||||
|
- `export AGENT_ENV=dev2 AGENT_USER=yuyr AGENT_INSTANCE=node002s4`
|
||||||
|
- `export MASTER_ENDPOINT=http://master.argus.com:3000`
|
||||||
|
- `/tmp/setup.sh --server "$FTPIP" --user ftpuser --password 'ZGClab1234!' --port 21`
|
||||||
|
- 说明:setup 会自动执行安装后自检(最多 5 分钟),无需手动轮询。
|
||||||
|
|
||||||
|
## 验证(推荐在容器内执行,避免宿主权限问题)
|
||||||
|
|
||||||
|
- 查看 node.json 关键字段
|
||||||
|
- `cat /private/argus/agent/dev2-yuyr-node002s4/node.json | jq '{last_report, health}'`
|
||||||
|
- 期望:四个 health 全部 healthy;等待 ≥70s 再查看,`last_report` 持续更新。
|
||||||
|
|
||||||
|
- 指标端口
|
||||||
|
- `curl -s -o /dev/null -w '%{http_code}\n' http://localhost:9100/metrics`(期望 200)
|
||||||
|
- (如测试 GPU)`curl -s -o /dev/null -w '%{http_code}\n' http://localhost:9400/metrics`(有 GPU 时 200)
|
||||||
|
|
||||||
|
- 与服务端连通(域名经 Bind 解析)
|
||||||
|
- `curl -s -o /dev/null -w '%{http_code}\n' http://master.argus.com:3000/readyz`(期望 200)
|
||||||
|
- `curl -s -o /dev/null -w '%{http_code}\n' http://es.log.argus.com:9200/_cluster/health`(期望 200)
|
||||||
|
|
||||||
|
## (可选)在服务器主机侧观察 Prometheus 目标更新
|
||||||
|
- `cat /home2/yuyr/deploy/versions/<VERSION>/private/argus/metric/prometheus/nodes.json | jq '.'`
|
||||||
|
|
||||||
|
## 常见提示
|
||||||
|
- 初次安装后短时 `curl` 域名返回 000/超时属正常,多等待 1–2 分钟 DNS 同步/组件冷启动完成。
|
||||||
|
- 如在宿主直接读取挂载的 node.json 报 Permission denied,请使用 `docker exec` 在容器内查看。
|
||||||
|
- MASTER_ENDPOINT 固定使用域名 `http://master.argus.com:3000`,客户端无需固定 IP。
|
||||||
@ -7,11 +7,10 @@
|
|||||||
|
|
||||||
## Quick Start
|
## Quick Start
|
||||||
1. Extract to a target dir, e.g. `/opt/argus-deploy/versions/<YYYYMMDD>`
|
1. Extract to a target dir, e.g. `/opt/argus-deploy/versions/<YYYYMMDD>`
|
||||||
2. `cd scripts && sudo ./server-prepare-dirs.sh` (recommended)
|
2. `./server-install.sh` (non‑root is supported: it will precreate minimal dirs and auto-fix Kibana/ES/Bind in containers)
|
||||||
3. `./server-install.sh` (non‑root is supported: it will precreate minimal dirs and auto-fix Kibana/ES/Bind in containers)
|
3. `./server-status.sh`
|
||||||
4. `./server-status.sh`
|
4. `./server-selfcheck.sh` (on failure it auto-runs diagnose and writes logs under `logs/`)
|
||||||
5. `./server-selfcheck.sh` (on failure it auto-runs diagnose and writes logs under `logs/`)
|
5. `./server-uninstall.sh` to tear down
|
||||||
6. `./server-uninstall.sh` to tear down
|
|
||||||
|
|
||||||
## What the Installer Does
|
## What the Installer Does
|
||||||
- Loads local images (`images/all-images.tar.gz`)
|
- Loads local images (`images/all-images.tar.gz`)
|
||||||
@ -29,7 +28,7 @@
|
|||||||
- If you cannot use sudo, the installer will:
|
- If you cannot use sudo, the installer will:
|
||||||
- create minimal data dirs (incl. `private/argus/log/{elasticsearch,kibana}`) with permissive perms when possible;
|
- create minimal data dirs (incl. `private/argus/log/{elasticsearch,kibana}`) with permissive perms when possible;
|
||||||
- ensure inside containers: Kibana `data` → `/private/argus/log/kibana`, Elasticsearch `data` → `/private/argus/log/elasticsearch`, and Bind `rndc.key` is generated.
|
- ensure inside containers: Kibana `data` → `/private/argus/log/kibana`, Elasticsearch `data` → `/private/argus/log/elasticsearch`, and Bind `rndc.key` is generated.
|
||||||
You can still run `sudo ./server-prepare-dirs.sh` later to normalize ownership.
|
(Manual pre-creation scripts are no longer required.)
|
||||||
|
|
||||||
## Files & Layout
|
## Files & Layout
|
||||||
- `compose/` (docker-compose.yml, .env)
|
- `compose/` (docker-compose.yml, .env)
|
||||||
|
|||||||
@ -7,11 +7,10 @@
|
|||||||
|
|
||||||
## 快速开始
|
## 快速开始
|
||||||
1. 解压到目标目录(例如 `/opt/argus-deploy/versions/<YYYYMMDD>`)
|
1. 解压到目标目录(例如 `/opt/argus-deploy/versions/<YYYYMMDD>`)
|
||||||
2. 进入 `scripts/`:`sudo ./server-prepare-dirs.sh`(推荐)
|
2. 安装:`./server-install.sh`(支持普通用户:会自动创建最小目录并在容器内修复 Kibana/ES/Bind)
|
||||||
3. 安装:`./server-install.sh`(支持普通用户:会自动创建最小目录并在容器内修复 Kibana/ES/Bind)
|
3. 状态:`./server-status.sh`
|
||||||
4. 状态:`./server-status.sh`
|
4. 自检:`./server-selfcheck.sh`(失败会自动采集诊断)
|
||||||
5. 自检:`./server-selfcheck.sh`(失败会自动采集诊断)
|
5. 卸载:`./server-uninstall.sh`
|
||||||
6. 卸载:`./server-uninstall.sh`
|
|
||||||
|
|
||||||
## 安装流程要点
|
## 安装流程要点
|
||||||
- 仅启动 10 个服务端组件(不包含测试节点);
|
- 仅启动 10 个服务端组件(不包含测试节点);
|
||||||
@ -20,11 +19,10 @@
|
|||||||
|
|
||||||
## 兼容说明(NixOS 等)
|
## 兼容说明(NixOS 等)
|
||||||
- 使用 `security_opt: ["label=disable"]` 与 `userns_mode: host`。
|
- 使用 `security_opt: ["label=disable"]` 与 `userns_mode: host`。
|
||||||
- 若不能使用 sudo:安装器会创建最小目录(含 `private/argus/log/{elasticsearch,kibana}`),并在容器内完成:
|
- 非 root 场景:安装器会创建最小目录(含 `private/argus/log/{elasticsearch,kibana}`),并在容器内完成:
|
||||||
- Kibana 的 `data` 软链到 `/private/argus/log/kibana`
|
- Kibana 的 `data` 软链到 `/private/argus/log/kibana`
|
||||||
- Elasticsearch 的 `data` 软链到 `/private/argus/log/elasticsearch`
|
- Elasticsearch 的 `data` 软链到 `/private/argus/log/elasticsearch`
|
||||||
- Bind 生成 `/etc/bind/rndc.key`
|
- Bind 生成 `/etc/bind/rndc.key`
|
||||||
安装后也可再执行 `sudo ./server-prepare-dirs.sh` 统一目录属主。
|
|
||||||
|
|
||||||
## 故障排查(见下文 Troubleshooting_zh)
|
## 故障排查(见下文 Troubleshooting_zh)
|
||||||
- `./server-selfcheck.sh` → `logs/selfcheck.json`
|
- `./server-selfcheck.sh` → `logs/selfcheck.json`
|
||||||
|
|||||||
@ -7,7 +7,6 @@
|
|||||||
- 初始化 Swarm:`docker swarm init --advertise-addr <manager_ip>`
|
- 初始化 Swarm:`docker swarm init --advertise-addr <manager_ip>`
|
||||||
- 创建 overlay:`docker network create --driver overlay --attachable argus-sys-net`
|
- 创建 overlay:`docker network create --driver overlay --attachable argus-sys-net`
|
||||||
- 解压离线包后执行:
|
- 解压离线包后执行:
|
||||||
- `cd scripts && sudo ./server-prepare-dirs.sh`
|
|
||||||
- `./server-install.sh`(会验证 Swarm 状态、确保 overlay 存在、写入 dns.conf)
|
- `./server-install.sh`(会验证 Swarm 状态、确保 overlay 存在、写入 dns.conf)
|
||||||
- `./server-selfcheck.sh`(失败会自动触发诊断)
|
- `./server-selfcheck.sh`(失败会自动触发诊断)
|
||||||
|
|
||||||
@ -24,3 +23,28 @@
|
|||||||
- MASTER_ENDPOINT 永远使用域名:`http://master.argus.com:3000`
|
- MASTER_ENDPOINT 永远使用域名:`http://master.argus.com:3000`
|
||||||
- docker compose 改为 external overlay;容器内不使用 Docker 服务名;web-proxy 与组件上游统一用域名
|
- docker compose 改为 external overlay;容器内不使用 Docker 服务名;web-proxy 与组件上游统一用域名
|
||||||
|
|
||||||
|
## 找回/轮换 Swarm 加入令牌与解锁密钥
|
||||||
|
|
||||||
|
在任意一个 Manager 节点上执行以下命令即可查看或轮换加入令牌(join token):
|
||||||
|
|
||||||
|
- 查看加入 Worker 的命令:
|
||||||
|
- `docker swarm join-token worker`
|
||||||
|
- 只打印 Worker 的 token:
|
||||||
|
- `docker swarm join-token -q worker`
|
||||||
|
- 查看加入 Manager 的命令:
|
||||||
|
- `docker swarm join-token manager`
|
||||||
|
- 只打印 Manager 的 token:
|
||||||
|
- `docker swarm join-token -q manager`
|
||||||
|
|
||||||
|
在待加入节点执行(示例,替换 Manager_IP):
|
||||||
|
- `docker swarm join --token <上面查到的token> <Manager_IP>:2377`
|
||||||
|
|
||||||
|
轮换 token(怀疑泄露或需要更新时):
|
||||||
|
- 轮换 Worker:`docker swarm join-token --rotate worker`
|
||||||
|
- 轮换 Manager:`docker swarm join-token --rotate manager`
|
||||||
|
|
||||||
|
如果你指的是“解锁密钥”(autolock 的 unlock key),在 Manager 上:
|
||||||
|
- 查看:`docker swarm unlock-key`
|
||||||
|
- 轮换:`docker swarm unlock-key --rotate`
|
||||||
|
|
||||||
|
提示:当看到 “This node is not a swarm manager.” 时,说明当前节点不是 Manager,需要到 Manager 节点执行,或在现有 Manager 上 `docker node promote <NODE-ID>` 将其提升为 Manager。
|
||||||
|
|||||||
@ -17,5 +17,4 @@ Kibana/ES:
|
|||||||
- Verify `es.log.argus.com` resolves inside Kibana
|
- Verify `es.log.argus.com` resolves inside Kibana
|
||||||
|
|
||||||
Permissions:
|
Permissions:
|
||||||
- Ensure `sudo ./server-prepare-dirs.sh` ran and ownership matches runtime UID:GID
|
- The installer auto-creates minimal dirs and applies container-side fixes (Kibana/ES/Bind). If you still see EACCES/lock errors, rerun `./server-install.sh` and review diagnose logs.
|
||||||
|
|
||||||
|
|||||||
@ -12,5 +12,5 @@
|
|||||||
Web‑Proxy:8083=200/302/403;8084/8085 需包含 CORS
|
Web‑Proxy:8083=200/302/403;8084/8085 需包含 CORS
|
||||||
Kibana:确认可解析 `es.log.argus.com`
|
Kibana:确认可解析 `es.log.argus.com`
|
||||||
权限:
|
权限:
|
||||||
- 非 root 安装时,安装器已创建最小目录并在容器内修复 Kibana/ES/Bind;
|
- 非 root 安装时,安装器会创建最小目录并在容器内修复 Kibana/ES/Bind;
|
||||||
- 如仍有 `EACCES`/锁文件报错,可再运行 `sudo ./server-prepare-dirs.sh` 统一目录属主。
|
- 如仍有 `EACCES`/锁文件报错,先重跑 `./server-install.sh`(会重复容器内修复),并查看诊断日志。
|
||||||
|
|||||||
Loading…
x
Reference in New Issue
Block a user