如何用Rclone集成Linux与云存储?

Cloud platform

1 基础知识

1.1 Rclone的简介

Rclone是一个命令行程序,该命令用于云和文件系统(文件和目录)之间的数据同步,所有别称云存储的“rsync”。

1.2 Rclone的作用

云与文件系统之间的数据同步

1.3 Rclone与rsync的区别

– rclone用于文件系统与云之间的数据同步
– rsync用于文件系统之间的数据同步

1.4 Rclone的特点

– 确保文件的完整性(通过检查MD5或SHA-1哈希值实现)
– 保留文件的时间戳
– 支持文件的部分同步
– 复制模式为增量复制(复制新的或更改的)
– 单向同步保持目录一致性
– 支持两个云账户的网络同步
– 支持加密(可选,Crpt)
– 支持缓存
– 支持FUSE安装

1.5 Rclone支持的云存储

– Alibaba Cloud (Aliyun) Object Storage System (OSS) page_facing_up
– Amazon Drive page_facing_up (See note)
– Amazon S3 page_facing_up
– Backblaze B2 page_facing_up
– Box page_facing_up
– Ceph page_facing_up
– DigitalOcean Spaces page_facing_up
– Dreamhost page_facing_up
– Dropbox page_facing_up
– FTP page_facing_up
– Google Cloud Storage page_facing_up
– Google Drive page_facing_up
– HTTP page_facing_up
– Hubic page_facing_up
– Jottacloud page_facing_up
– IBM COS S3 page_facing_up
– Koofr page_facing_up
– Memset Memstore page_facing_up
– Mega page_facing_up
– Microsoft Azure Blob Storage page_facing_up
– Microsoft OneDrive page_facing_up
– Minio page_facing_up
– Nextcloud page_facing_up
– OVH page_facing_up
– OpenDrive page_facing_up
– OpenStack Swift page_facing_up
– Oracle Cloud Storage page_facing_up
– ownCloud page_facing_up
– pCloud page_facing_up
– put.io page_facing_up
– QingStor page_facing_up
– Rackspace Cloud Files page_facing_up
– Scaleway page_facing_up
– SFTP page_facing_up
– Wasabi page_facing_up
– WebDAV page_facing_up
– Yandex Disk page_facing_up
– The local filesystem page_facing_up
详细请参阅下表,
https://rclone.org/overview/

2 最佳实践

2.1 系统配置

2.1.1 环境信息

IP Addresses = 10.168.0.167
OS = CentOS 7.x x86_64
Host Name = Any

2.1.2 配置桌面环境

请参阅如下链接配置vnc环境,
https://www.cmdschool.org/archives/1884

2.2 基本的安装配置

2.2.1 安装软件包

yum install -y https://downloads.rclone.org/v1.46/rclone-v1.46-linux-amd64.rpm

2.2.2 启动云端配置服务

rclone config

向导如下,

2019/04/11 15:41:31 NOTICE: Config file "/root/.config/rclone/rclone.conf" not found - using defaults
No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> remote
Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
[...]
 6 / Box
   \ "box"
[...]
Storage> box
** See help for box backend at: https://rclone.org/box/ **

Box App Client Id.
Leave blank normally.
Enter a string value. Press Enter for the default ("").
client_id>
Box App Client Secret
Leave blank normally.
Enter a string value. Press Enter for the default ("").
client_secret>
Edit advanced config? (y/n)
y) Yes
n) No
y/n> n
Remote config
Use auto config?
 * Say Y if not sure
 * Say N if you are working on a remote or headless machine
y) Yes
n) No
y/n> y
If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth
Log in and authorize rclone for access
Waiting for code...

注:“[…]”表示省略

2.2.3 配置云端连接

如果没有自动跳转,请打开浏览器并手动输入如下地址,
http://127.0.0.1:53682/auth
可见如下登录界面,

如上图所示:
输入box服务端的账号和密码,单击【Authorize】->【授权访问Box】看到提示“Success”字样即可。

2.2.4 确认配置

Got code
--------------------
[remote]
type = box
token = {"access_token":"xxx","token_type":"bearer","refresh_token":"xxx","expiry":"xxx"}
--------------------
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y
Current remotes:

Name                 Type
====                 ====
remote               box

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q

另外,你可通过如下命令查看当前的配置,

cat /root/.config/rclone/rclone.conf

可见如下配置,

type = box
token = {"access_token":"xxx","token_type":"bearer","refresh_token":"xxx","expiry":"xxx"}

注:如果你不想使用向导生成配置,可通过直接修改配置文件实现。

2.2.5 测试配置

rclone lsd remote:
rclone ls remote:
cp /etc/redhat-release ~/redhat-release.txt
rclone copy ~/redhat-release.txt remote:backup

可见如下显示

       27 backup/redhat-release.txt
      817 My Box Notes/Untitled Note 2019-04-09 16.34.32.boxnote

2.3 使用范例

2.3.1 命令行帮助

rclone -h

可见如下显示,

Rclone syncs files to and from cloud storage providers as well as
mounting them, listing them in lots of different ways.

See the home page (https://rclone.org/) for installation, usage,
documentation, changelog and configuration walkthroughs.

Usage:
  rclone [flags]
  rclone [command]

Available Commands:
  about           Get quota information from the remote.
  authorize       Remote authorization.
  cachestats      Print cache stats for a remote
  cat             Concatenates any files and sends them to stdout.
  check           Checks the files in the source and destination match.
  cleanup         Clean up the remote if possible
  config          Enter an interactive configuration session.
  copy            Copy files from source to dest, skipping already copied
  copyto          Copy files from source to dest, skipping already copied
  copyurl         Copy url content to dest.
  cryptcheck      Cryptcheck checks the integrity of a crypted remote.
  cryptdecode     Cryptdecode returns unencrypted file names.
  dbhashsum       Produces a Dropbox hash file for all the objects in the path.
  dedupe          Interactively find duplicate files and delete/rename them.
  delete          Remove the contents of path.
  deletefile      Remove a single file from remote.
  genautocomplete Output completion script for a given shell.
  gendocs         Output markdown docs for rclone to the directory supplied.
  hashsum         Produces an hashsum file for all the objects in the path.
  help            Show help for rclone commands, flags and backends.
  link            Generate public link to file/folder.
  listremotes     List all the remotes in the config file.
  ls              List the objects in the path with size and path.
  lsd             List all directories/containers/buckets in the path.
  lsf             List directories and objects in remote:path formatted for parsing
  lsjson          List directories and objects in the path in JSON format.
  lsl             List the objects in path with modification time, size and path.
  md5sum          Produces an md5sum file for all the objects in the path.
  mkdir           Make the path if it doesn't already exist.
  mount           Mount the remote as file system on a mountpoint.
  move            Move files from source to dest.
  moveto          Move file or directory from source to dest.
  ncdu            Explore a remote with a text based user interface.
  obscure         Obscure password for use in the rclone.conf
  purge           Remove the path and all of its contents.
  rc              Run a command against a running rclone.
  rcat            Copies standard input to file on remote.
  rcd             Run rclone listening to remote control commands only.
  rmdir           Remove the path if empty.
  rmdirs          Remove empty directories under the path.
  serve           Serve a remote over a protocol.
  settier         Changes storage class/tier of objects in remote.
  sha1sum         Produces an sha1sum file for all the objects in the path.
  size            Prints the total size and number of objects in remote:path.
  sync            Make source and dest identical, modifying destination only.
  touch           Create new file or change file modification time.
  tree            List the contents of the remote in a tree like fashion.
  version         Show the version number.

Use "rclone [command] --help" for more information about a command.
Use "rclone help flags" for to see the global flags.
Use "rclone help backends" for a list of supported services.

2.3.2 挂载目录到本地

rclone mkdir remote:/storages
rclone mount remote:/storages /mnt

测试挂载,

touch /mnt/abx.txt
ls -l /mnt/
rclone ls remote:/storages/

2.3.3 永久挂载

vim /usr/local/bin/rclonefs

加入如下内容,

#!/bin/bash
remote=$1
mountpoint=$2
shift 2

# Process -o parameters
while getopts :o: opts; do
    case $opts in
        o)
            params=${OPTARG//,/ }
            for param in $params; do
                if [ "$param" == "rw"   ]; then continue; fi
                if [ "$param" == "ro"   ]; then continue; fi
                if [ "$param" == "dev"  ]; then continue; fi
                if [ "$param" == "suid" ]; then continue; fi
                if [ "$param" == "exec" ]; then continue; fi
                if [ "$param" == "auto" ]; then continue; fi
                if [ "$param" == "nodev" ]; then continue; fi
                if [ "$param" == "nosuid" ]; then continue; fi
                if [ "$param" == "noexec" ]; then continue; fi
                if [ "$param" == "noauto" ]; then continue; fi
                if [[ $param == x-systemd.* ]]; then continue; fi
                trans="$trans --$param"
            done
            ;;
        \?)
            echo "Invalid option: -$OPTARG"
            ;;
    esac
done

# exec rclone
trans="$trans $remote $mountpoint"
PATH=$PATH rclone mount $trans &

out=`ls -l $dst`
until [ "$out" != 'total 0' ]; do
    out=`ls -l $dst`
    sleep 1
done

增加执行权限,

chmod +x /usr/local/bin/rclonefs

确认配置文件是否存在,

ls -l /root/.config/rclone/rclone.conf

创建挂载信息,

vim /etc/fstab

增加如下行,

rclonefs#remote:/storages /data fuse config=/root/.config/rclone/rclone.conf,allow-other,default-permissions,read-only,max-read-ahead=16M 0 0

注意:
– rclonefs,为负责挂载的脚本
– remote:/storages,为远程挂载的设备(之前创建的)
– /data,为挂载到本地的目录
– config,变量后面的路径代表连接云存储的配置路径
以上设置完成后,使用如下命令测试挂载,

mount -a

然后使用以下命令确认已经挂载,

df -h

参阅文章
================

rclone config for box
———————-
https://rclone.org/box/

rclone config for fstab
———————-
https://github.com/ncw/rclone/wiki/rclone-fstab-mount-helper-script

rclone的下载
——————-
https://downloads.rclone.org/

Linux OS 集成Box
——————-
https://uisapp2.iu.edu/confluence-prd/display/SOICKB/Using+Box+under+Linux

Box企业版开启FTP服务
——————-
https://community.box.com/t5/Upload-and-Download-Files-and/Using-Box-with-FTP-or-FTPS/ta-p/26050

盒子产品中心
——————-
https://community.box.com/t5/Box-Product-Central/ct-p/ProductCentral

没有评论

发表评论

Cloud platform
如何认识云计算服务化分层?

1 云计算的分层 1.1 IaaS 1.1.1 IaaS的概念 即Infrastructure-as …

Cloud platform
如何使用应答文件部署openStack?

1 理论知识 1.1 openStack的简介 OpenStack是由网络主机服务商Rackspac …