1. 下載&安裝

從下面的地址下載對應的二進制版本即可，之后安裝在系統的/usr/local/bin目錄下,或者其他用戶指定的路徑即可。
二進制//rclone.org/downloads/
源碼//github.com/rclone/rclone

2. rclone簡介

rclone是一個golang寫的開源遠程數據同步工具，后端支持文件系統以及多種類型的對象存儲產品，是一個跨平臺的數據同步工具，相比于s3cmd性能更加強大，同時還支持sync數據同步等功能實現差異數據的跨存儲后臺同步。并且還提供mount操作，可以直接將遠端對象存儲掛載到本地文件系統進行訪問。官方的Features介紹如下

MD5/SHA-1 hashes checked at all times for file integrity
Timestamps preserved on files
Partial syncs supported on a whole file basis
Copy mode to just copy new/changed files
Sync (one way) mode to make a directory identical
Check mode to check for file hash equality
Can sync to and from network, e.g. two different cloud accounts
Optional large file chunking (Chunker)
Optional transparent compression (Compress)
Optional encryption (Crypt)
Optional FUSE mount (rclone mount)
Multi-threaded downloads to local disk
Can serve local or remote files over HTTP/WebDav/FTP/SFTP/dlna

3. 初始配置與檢查

查看軟件版本

╭─user@Mac~
╰─$rcloneversion1?
rclonev1.58.0
-os/version:darwin10.15.7(64bit)
-os/kernel:19.6.0(x86_64)
-os/type:darwin
-os/arch:amd64
-go/version:go1.17.8
-go/linking:dynamic
-go/tags:cmount

通過命令rclone config file獲取默認的配置文件路徑，之后在對應的配置文件中填寫對應的配置

╭─user@Mac~
╰─$rcloneconfigfile
Configurationfileisstoredat:
/Users/user/.config/rclone/rclone.conf
╭─user@Mac~
╰─$vi/Users/user/.config/rclone/rclone.conf
╭─user@Mac~
╰─$cat/Users/user/.config/rclone/rclone.conf
[s3v2]
type=s3
provider=Ceph
access_key_id=xxx
secret_access_key=sss
endpoint=https://s3.demo.com
location_constraint=
force_path_style=false

╭─user@Mac~/SourceCode/rclone?master?
╰─$rcloneconfigshow
[s3v2]
type=s3
provider=Ceph
access_key_id=xxx
secret_access_key=sss
endpoint=http://s3.demo.com
location_constraint=
force_path_style=false

╭─user@Mac~/SourceCode/rclone?master?
╰─$rcloneconfigdump
{
"s3v2":{
"access_key_id":"xxx",
"endpoint":"http://s3.demo.com",
"force_path_style":"false",
"location_constraint":"",
"provider":"Ceph",
"secret_access_key":"sss",
"type":"s3"
}
}%

通過rclone listremotes查看已經配置好的遠端服務列表

╭─user@Mac~
╰─$rclonelistremotes
s3v2:

通過rclone lsd查看bucket列表或者是遠端的目錄列表

╭─user@Mac~
╰─$rclonelsds3v2:
-12022-01-1011:11:52-1demo-storage
╭─user@Mac~
╰─$rclonelsds3v2:demo-storage
02022-04-1110:42:45-1demo-storage
02022-04-1110:42:45-1wallpapers-zelda

通過rclone backend features獲取遠端服務支持的功能特性列表

╭─user@Mac~/SourceCode/rclone?master*?
╰─$rclonebackendfeaturess3v2:
{
"Name":"s3v2",
"Root":"",
"String":"S3root",
"Precision":1,
"Hashes":[
"md5"
],
"Features":{
"About":false,
"BucketBased":true,
"BucketBasedRootOK":true,
"CanHaveEmptyDirectories":false,
"CaseInsensitive":false,
"ChangeNotify":false,
"CleanUp":true,
"Command":true,
"Copy":true,
"DirCacheFlush":false,
"DirMove":false,
"Disconnect":false,
"DuplicateFiles":false,
"GetTier":true,
"IsLocal":false,
"ListR":true,
"MergeDirs":false,
"Move":false,
"OpenWriterAt":false,
"PublicLink":true,
"Purge":false,
"PutStream":true,
"PutUnchecked":false,
"ReadMimeType":true,
"ServerSideAcrossConfigs":false,
"SetTier":true,
"SetWrapper":false,
"Shutdown":false,
"SlowHash":false,
"SlowModTime":true,
"UnWrap":false,
"UserInfo":false,
"WrapFs":false,
"WriteMimeType":true
}
}

4. 數據上傳下載與遍歷

以上傳本地bin目錄到遠端S3為例

╭─user@Mac~/SourceCode/rclone?master?
╰─$lsbin
bisect-go-rclone.shcheck-merged.godecrypt_names.pymake_changelog.pynfpm.yamltest-repeat-vfs.shtest_proxy.pyupdate-authors.py
bisect-rclone.shconfig.pyget-github-release.gomake_manual.pynot-in-stable.gotest-repeat.shtidy-betaupload-github
build-xgo-cgofuse.shcross-compile.gomake_backend_docs.pymake_rc_docs.shtest-all-commits-compile.shtest_independence.gotravis.rclone.confwin-build.bat


╭─user@Mac~/SourceCode/rclone?master?
╰─$rclonecopybins3v2:demo-storage/bin-vv
2022/04/1314:50:25DEBUG:rclone:Version"v1.58.0"startingwithparameters["rclone""copy""bin""s3v2:demo-storage/bin""-vv"]
2022/04/1314:50:25DEBUG:Creatingbackendwithremote"bin"
2022/04/1314:50:25DEBUG:Usingconfigfilefrom"/Users/user/.config/rclone/rclone.conf"
2022/04/1314:50:25DEBUG:fscache:renamingcacheitem"bin"tobecanonical"/Users/user/SourceCode/rclone/bin"
2022/04/1314:50:25DEBUG:Creatingbackendwithremote"s3v2:demo-storage/bin"
2022/04/1314:50:25DEBUG:S3bucketdemo-storagepathbin:Waitingforcheckstofinish
2022/04/1314:50:25DEBUG:S3bucketdemo-storagepathbin:Waitingfortransferstofinish
2022/04/1314:50:25DEBUG:bisect-rclone.sh:md5=5476e61f31e1e2d697ce06a0f0100829OK
2022/04/1314:50:25INFO:bisect-rclone.sh:Copied(new)
...
2022/04/1314:50:26DEBUG:upload-github:md5=a26b267a5c784152a6378457b217c3b9OK
2022/04/1314:50:26INFO:upload-github:Copied(new)
2022/04/1314:50:26INFO:
Transferred:62.573KiB/62.573KiB,100%,0B/s,ETA-
Transferred:25/25,100%
Elapsedtime:0.8s

2022/04/1314:50:26DEBUG:13goroutinesactive
╭─user@Mac~/SourceCode/rclone?master?
╰─$rclonelsds3v2:demo-storage
02022-04-1314:50:32-1bin
02022-04-1314:50:32-1demo
02022-04-1314:50:32-1demo-storage
02022-04-1314:50:32-1wallpapers-zelda

通過lsd目錄查看指定路徑下的目錄列表

╭─user@Mac~/SourceCode/rclone?master?
╰─$rclonelsds3v2:demo-storage
02022-04-1315:02:41-1bin
02022-04-1315:02:41-1demo
02022-04-1315:02:41-1demo-storage
02022-04-1315:02:41-1wallpapers-zelda

通過ls命令查看指定目錄下的文件列表

╭─user@Mac~/SourceCode/rclone?master?
╰─$rclonelss3v2:demo-storage/bin
253.ignore-emails
382bisect-go-rclone.sh
902bisect-rclone.sh
...
370travis.rclone.conf
1224update-authors.py
1034upload-github
364win-build.bat

通過指定max-depth參數來設定ls命令所需遍歷的目錄層級深度

╭─user@Mac~/SourceCode/rclone?master?
╰─$rclonelss3v2:demo-storage--max-depth1?
382bisect-go-rclone.sh
902bisect-rclone.sh
142build-xgo-cgofuse.sh
...
370travis.rclone.conf
1224update-authors.py
1034upload-github
364win-build.bat

同時支持以json方式輸出，并輸出對應文件的md5值信息。

╭─user@Mac~/SourceCode/rclone?master?
╰─$rclonelsjsons3v2:demo-storage/bin--hash1?
[
{"Path":".ignore-emails","Name":".ignore-emails","Size":253,"MimeType":"application/octet-stream","ModTime":"2021-12-16T1033.005446571+08:00","IsDir":false,"Hashes":{"md5":"8155885487321df7790cf63a9db7ba46"},"Tier":"STANDARD"},
{"Path":"upload-github","Name":"upload-github","Size":1034,"MimeType":"application/octet-stream","ModTime":"2021-07-08T1834.487717787+08:00","IsDir":false,"Hashes":{"md5":"a26b267a5c784152a6378457b217c3b9"},"Tier":"STANDARD"},
{"Path":"win-build.bat","Name":"win-build.bat","Size":364,"MimeType":"application/x-msdownload","ModTime":"2021-07-08T1834.487814575+08:00","IsDir":false,"Hashes":{"md5":"bf695fba05eb1ec8972eac29a59b21b4"},"Tier":"STANDARD"}
]

同時支持按文件files-only或者目錄dirs-only進行內容篩選

╭─user@Mac~/SourceCode/rclone?master?
╰─$rclonelsjsons3v2:demo-storage--dirs-only1?
[
{"Path":"bin","Name":"bin","Size":0,"MimeType":"inode/directory","ModTime":"2022-04-13T1532.917088000+08:00","IsDir":true},
{"Path":"demo","Name":"demo","Size":0,"MimeType":"inode/directory","ModTime":"2022-04-13T1532.917140000+08:00","IsDir":true},
{"Path":"demo-storage","Name":"demo-storage","Size":0,"MimeType":"inode/directory","ModTime":"2022-04-13T1532.917145000+08:00","IsDir":true},
{"Path":"wallpapers-zelda","Name":"wallpapers-zelda","Size":0,"MimeType":"inode/directory","ModTime":"2022-04-13T1532.917149000+08:00","IsDir":true}
]

數據下載

╭─user@Mac/tmp/demo
╰─$ls
╭─user@Mac/tmp/demo
╰─$pwd
/tmp/demo
╭─user@Mac/tmp/demo
╰─$rclonecopys3v2:demo-storage/binbin
╭─user@Mac/tmp/demo
╰─$lsbin
bisect-go-rclone.shcheck-merged.godecrypt_names.pymake_changelog.pynfpm.yamltest-repeat-vfs.shtest_proxy.pyupdate-authors.py
bisect-rclone.shconfig.pyget-github-release.gomake_manual.pynot-in-stable.gotest-repeat.shtidy-betaupload-github
build-xgo-cgofuse.shcross-compile.gomake_backend_docs.pymake_rc_docs.shtest-all-commits-compile.shtest_independence.gotravis.rclone.confwin-build.bat

5. 使用sync數據同步

將遠端數據同步到本地

╭─user@Mac/tmp/demo
╰─$rclonesyncs3v2:demo-storage/binbin1?
╭─user@Mac/tmp/demo
╰─$ls
bin
╭─user@Mac/tmp/demo
╰─$lsbin
bisect-go-rclone.shcheck-merged.godecrypt_names.pymake_changelog.pynfpm.yamltest-repeat-vfs.shtest_proxy.pyupdate-authors.py
bisect-rclone.shconfig.pyget-github-release.gomake_manual.pynot-in-stable.gotest-repeat.shtidy-betaupload-github
build-xgo-cgofuse.shcross-compile.gomake_backend_docs.pymake_rc_docs.shtest-all-commits-compile.shtest_independence.gotravis.rclone.confwin-build.bat

模擬新建一個文件，同步到遠端

╭─user@Mac/tmp/demo
╰─$echonewfile1>bin/newfile1
╭─user@Mac/tmp/demo
╰─$rclonesyncbins3v2:demo-storage/bin
╭─user@Mac/tmp/demo
╰─$rclonelss3v2:demo-storage/bin
253.ignore-emails
382bisect-go-rclone.sh
...
370travis.rclone.conf
1224update-authors.py
1034upload-github
364win-build.bat

模擬修改一個文件，同步到遠端，可以看到只同步了差異部分

╭─user@Mac/tmp/demo
╰─$echonew123>bin/newfile1
╭─user@Mac/tmp/demo
╰─$rclonesyncbins3v2:demo-storage/bin-vv
2022/04/1315:39:01DEBUG:rclone:Version"v1.58.0"startingwithparameters["rclone""sync""bin""s3v2:demo-storage/bin""-vv"]
2022/04/1315:39:01DEBUG:Creatingbackendwithremote"bin"
2022/04/1315:39:01DEBUG:Usingconfigfilefrom"/Users/user/.config/rclone/rclone.conf"
2022/04/1315:39:01DEBUG:fscache:renamingcacheitem"bin"tobecanonical"/tmp/demo/bin"
2022/04/1315:39:01DEBUG:Creatingbackendwithremote"s3v2:demo-storage/bin"
2022/04/1315:39:01DEBUG:S3bucketdemo-storagepathbin:Waitingforcheckstofinish
...
2022/04/1315:39:02DEBUG:upload-github:Unchangedskipping
2022/04/1315:39:02DEBUG:win-build.bat:Sizeandmodificationtimethesame(differby0s,withintolerance1ns)
2022/04/1315:39:02DEBUG:win-build.bat:Unchangedskipping
2022/04/1315:39:02DEBUG:S3bucketdemo-storagepathbin:Waitingfortransferstofinish
2022/04/1315:39:02DEBUG:newfile1:md5=f9f9a6b3363a401f566becf83a5fad1bOK
2022/04/1315:39:02INFO:newfile1:Copied(replacedexisting)
2022/04/1315:39:02DEBUG:Waitingfordeletionstofinish
2022/04/1315:39:02INFO:
Transferred:7B/7B,100%,0B/s,ETA-
Checks:26/26,100%
Transferred:1/1,100%
Elapsedtime:0.3s

2022/04/1315:39:02DEBUG:25goroutinesactive

6. 掛載S3目錄到本地文件系統

rclone支持按指定目錄作為根目錄進行掛載,比如指定bucket名稱為demo-storage下的bin目錄作為根目錄

╭─user@Mac/tmp
╰─$rclonemounts3v2:demo-storage/bin~/mnt?

之后就可以在本地文件系統上訪問對應的目錄數據了

╭─user@Mac~
╰─$cd~/mnt
╭─user@Mac~/mnt
╰─$ls
bisect-go-rclone.shcross-compile.gomake_manual.pytest-all-commits-compile.shtidy-beta
bisect-rclone.shdecrypt_names.pymake_rc_docs.shtest-repeat-vfs.shtravis.rclone.conf
build-xgo-cgofuse.shget-github-release.gonewfile1test-repeat.shupdate-authors.py
check-merged.gomake_backend_docs.pynfpm.yamltest_independence.goupload-github
config.pymake_changelog.pynot-in-stable.gotest_proxy.pywin-build.bat

7. 其他操作

通過link命令生成文件的訪問URL鏈接，支持通過expire設置URL有效時長。

╭─user@Mac~/mnt
╰─$rclonelinks3v2:demo-storage/bin/win-build.bat--expire1h4?
http://demo-storage.s3.demo.com/bin/win-build.bat?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ZK4XIPVNW9PMOADEZMSR%2F20220413%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220413T080748Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=b4d60414a5d13662ac7026821e0aa7ffa10f4654142b87ea8629d2ba98b05e8c

通過size命令，獲取文件或者目錄的大小

─user@Mac~/mnt
╰─$rclonesizes3v2:demo-storage/bin
Totalobjects:26(26)
Totalsize:62.580KiB(64082Byte)

通過delete命令，刪除對應的文件

╭─user@Mac~/mnt
╰─$rclonedeletes3v2:demo-storage/bin/win-build.bat

使用rmdir只能刪除空目錄，如果目錄殘留文件，需要使用purge命令刪除數據，之后再執行rmdir才能徹底刪除目錄數據。

╭─user@Mac~/mnt
╰─$rclonermdirs3v2:demo-storage/bin-v
2022/04/131625INFO:S3bucketdemo-storagepathbin:Removingdirectory
╭─user@Mac~/mnt
╰─$rclonelss3v2:demo-storage/bin
253.ignore-emails
...
1034upload-github
╭─user@Mac~/mnt
╰─$rclonepurges3v2:demo-storage/bin-v
2022/04/1316:12:55INFO:build-xgo-cgofuse.sh:Deleted
...
2022/04/1316:12:55INFO:update-authors.py:Deleted
2022/04/1316:12:55INFO:S3bucketdemo-storagepathbin:Removingdirectory
╭─user@Mac~/mnt
╰─$rclonelss3v2:demo-storage/bin
╭─user@Mac~/mnt
╰─$rclonermdirs3v2:demo-storage/bin
╭─user@Mac~/mnt
╰─$rclonelsds3v2:demo-storage
02022-04-1316:13:16-1demo
02022-04-1316:13:16-1demo-storage
02022-04-1316:13:16-1wallpapers-zelda