From f78bb9f5476efc1640502119f9453585a54aa195 Mon Sep 17 00:00:00 2001 From: wenyongda Date: Tue, 5 Aug 2025 17:11:29 +0800 Subject: [PATCH] commit --- source/_posts/Linux.md | 104 +++++++++++++++++++++++++++++++++++ source/_posts/大数据.md | 117 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 221 insertions(+) diff --git a/source/_posts/Linux.md b/source/_posts/Linux.md index 7807e9e..7ad81c7 100644 --- a/source/_posts/Linux.md +++ b/source/_posts/Linux.md @@ -443,6 +443,110 @@ chmod u-w /etc/sudoers 这样普通用户就可以使用sudo了. +### 修改目录权限 + +**查看当前目录权限** + +```bash +sudo ls -ld /OLAP +``` + +输出实例： + +`drwxr-xr-x 5 root root 4096 Aug 5 08:27 /OLAP` + +**修改目录权限** + +使当前用户（假设为 `user`）能够对 `/OLAP`目录进行读写操作，可以将目录权限修改为`775`（即`rwxrwxr-x`）： + +```bash +sudo chmod 775 /OLAP +``` + +这样，目录的所有者和所属组的用户都可以读写该目录，其他用户则由读取和执行权限。 + +**将当前用户加入目录所属组** + +如果目录所属组是`root`，可以将当前用户`user`加入`root`组（不推荐，因为`root`组权限过高）： + +```bash +sudo usermod -aG root user +``` + +然后，重新登录或重启系统以使组变更生效。 + +### 更改目录的所有者 + +**查看当前目录的所有者** + +查看`/OLAP`目录的当前所有者： + +```bash +sudo ls -ld /OLAP +``` + +**更改目录的所有者** + +将`/OLAP`目录的所有者更改为当前用户（假设为`user`）： + +```bash +sudo chown user:user /OLAP +``` + +这样，当前用户将拥有对该目录的完全控制权。 + +### 使用 ACL（访问控制列表） + +> ACL 提供了更细粒度的权限控制，允许为特定用户或组设置特定权限。 + +**安装 ACL 工具** + +在某些系统中，ACL 工具可能未默认安装，可以通过以下命令安装： + +```bash +sudo apt install acl -y +sudo dnf insyall acl -y +``` + +**设置 ACL 权限** + +为当前用户（假设为`user`）设置读写权限： + +```bash +sudo setfacl -m u:user:rwx /OLAP +``` + +这样，`user`用户将获得对`/OLAP`目录的读写权限，而不会影响其他用户的权限。 + +**验证 ACL 权限** + +查看当前目录的 ACL 权限： + +```bash +getfacl /OLAP +``` + +输出实例： +``` +getfacl: Removing leading '/' from absolute path names +# file: OLAP +# owner: user +# group: user +user::rwx +group::r-x +other::r-x +``` + +### 更改目录的默认权限（可选） + +如果要将新创建的文件和子目录自动继承特定权限，可以设置默认 ACL： + +```bash +sudo setfacl -dm u:user:rwx /OLAP +``` + +这样，新创建的文件和子目录将自动继承 `user` 用户的读写权限。 + ## Shell 脚本 ### 为什么大多数 shell 脚本都包含 #! /bin/bash 在 shell 脚本的开头？ diff --git a/source/_posts/大数据.md b/source/_posts/大数据.md index 8d39986..589a5fa 100644 --- a/source/_posts/大数据.md +++ b/source/_posts/大数据.md @@ -44,3 +44,120 @@ echo "* hard nproc 65536" >> /etc/security/limits.conf ulimit -n 65536 ``` +# Datax + +## MySQL离线同步至Apache Doris + +### 单表同步 + +新建配置文件`mysql_2_doris_t_base_material.json` + +```json +{ + "job": { + "setting": { + "speed": { + "channel": 1 + } + }, + "content": [ + { + "reader": { + "name": "mysqlreader", + "parameter": { + "username": "root", + "password": "123456", + "column": [ + "ID", "PID", "CODE", "SHORTCODE", "NAME", "FULLNAME", + "ISDETAIL", "CHILDCOUNT", "PYCODE", "LVL", "FLAG", "MODEL", + "ISSTANDARD", "AUXCLASSID", "ERPCLSID", "UNITGROUPID", "UNITID", + "DEFAULTLOC", "SPID", "QTYDECIMAL", "SECINV", "MTYPE", "MVER", + "MATERIAL", "FIRM", "FIRMORDER", "FACEDEAL", "SHAPESIZE", + "TECHDESC", "UNITWEIGHT", "MCODE", "MCAT", "COSTITEM", + "ORDERRECTOR", "POHIGHPRICE", "POHGHPRCMNYTYPE", "WWHGHPRC", + "WWHGHPRCMNYTYPE", "SOLOWPRC", "SOLOWPRCMNYTYPE", "TRACK", + "PRICEDECIMAL", "ACCTID", "SALEACCTID", "COSTACCTID", + "DEFAULTROUTINGID", "DEFAULTWORKTYPEID", "PRODUCTPRINCIPAL", + "PLANNER", "ISBACKFLUSH", "MRPCON", "MRPORDER", "CHARTNUMBER", + "INSPECTIONLEVEL", "PROCHKMDE", "WWCHKMDE", "SOCHKMDE", + "WTHDRWCHKMDE", "STKCHKMDE", "OTHERCHKMDE", "INSPECTIONPROJECT", + "NAMEEN", "MODELEN", "HSNUMBER", "IMPOSTTAXRATE", + "CONSUMETAXRATE", "STARTSERVICE", "MAKEFILE", "ISFIX", + "TTERMOFSERVICE", "TTERMOFUSEFULTIME", "PRODUCTNO", "PARAMVALUE", + "MEMO", "CDATE", "CUSERID", "CSTAFFNAME", "MDATE", "MUSERID", + "MSTAFFNAME", "K_ID", "K_PID", "DDATE", "DUSERID", "DSTAFFNAME", + "K_CODE", "AMODEL", "K_COSTITEM", "DWGSYMDESC", "SPDESC", + "CDRATE", "GWEIGHT", "NWEIGHT", "L", "W", "H", "MSIZE", + "K_COST", "BATCHAPPLICANT", "MATSLUGGISHSTATUS", + "SLUGGISHEFFECTIVEDATE", "K_AUTO_ID", "FRATE", "K3CLOUDID" + ], + "connection": [ + { + "table": ["t_base_material"], + "jdbcUrl": ["jdbc:mysql://192.168.6.35:3306/erp?useSSL=false&serverTimezone=Asia/Shanghai"], + "selectedDatabase": "erp" + } + ] + } + }, + "writer": { + "name": "doriswriter", + "parameter": { + "loadUrl": ["192.168.6.35:8040"], + "username": "root", + "password": "", + "column": [ + "ID", "PID", "CODE", "SHORTCODE", "NAME", "FULLNAME", + "ISDETAIL", "CHILDCOUNT", "PYCODE", "LVL", "FLAG", "MODEL", + "ISSTANDARD", "AUXCLASSID", "ERPCLSID", "UNITGROUPID", "UNITID", + "DEFAULTLOC", "SPID", "QTYDECIMAL", "SECINV", "MTYPE", "MVER", + "MATERIAL", "FIRM", "FIRMORDER", "FACEDEAL", "SHAPESIZE", + "TECHDESC", "UNITWEIGHT", "MCODE", "MCAT", "COSTITEM", + "ORDERRECTOR", "POHIGHPRICE", "POHGHPRCMNYTYPE", "WWHGHPRC", + "WWHGHPRCMNYTYPE", "SOLOWPRC", "SOLOWPRCMNYTYPE", "TRACK", + "PRICEDECIMAL", "ACCTID", "SALEACCTID", "COSTACCTID", + "DEFAULTROUTINGID", "DEFAULTWORKTYPEID", "PRODUCTPRINCIPAL", + "PLANNER", "ISBACKFLUSH", "MRPCON", "MRPORDER", "CHARTNUMBER", + "INSPECTIONLEVEL", "PROCHKMDE", "WWCHKMDE", "SOCHKMDE", + "WTHDRWCHKMDE", "STKCHKMDE", "OTHERCHKMDE", "INSPECTIONPROJECT", + "NAMEEN", "MODELEN", "HSNUMBER", "IMPOSTTAXRATE", + "CONSUMETAXRATE", "STARTSERVICE", "MAKEFILE", "ISFIX", + "TTERMOFSERVICE", "TTERMOFUSEFULTIME", "PRODUCTNO", "PARAMVALUE", + "MEMO", "CDATE", "CUSERID", "CSTAFFNAME", "MDATE", "MUSERID", + "MSTAFFNAME", "K_ID", "K_PID", "DDATE", "DUSERID", "DSTAFFNAME", + "K_CODE", "AMODEL", "K_COSTITEM", "DWGSYMDESC", "SPDESC", + "CDRATE", "GWEIGHT", "NWEIGHT", "L", "W", "H", "MSIZE", + "K_COST", "BATCHAPPLICANT", "MATSLUGGISHSTATUS", + "SLUGGISHEFFECTIVEDATE", "K_AUTO_ID", "FRATE", "K3CLOUDID" + ], + "postSql": ["select count(1) from t_base_material"], + "preSql": [], + "flushInterval":30000, + "connection": [ + { + "jdbcUrl": "jdbc:mysql://192.168.6.35:9030/erp", + "selectedDatabase": "erp", + "table": ["t_base_material"] + } + ], + "loadProps": { + "format": "json", + "strip_outer_array":"true", + "line_delimiter": "\\x02" + } + } + } + } + ] + } +} + +``` + +执行任务 + +```bash +cd datax +python ./bin/datax.py mysql_2_doris_t_base_material.json +``` +