混合雲大概是不可避免的趨勢，有時候主機放在地端會比較便宜且方便，若遇到要串接雲端的時候，這又變成了一門學問，這次要測試的主題是在地端的Linux主機mount AWS的S3，過程有一點小繁瑣，必須記錄一下以免之後會用得到。流程是先新建一個S3 Bucket、建立傳遞的user與role，最後在地端主機上mount S3 Bucket。
一．建立S3 bucket，並測試上傳物件
還沒有建立S3的經驗，就先自建一個S3 Standard的Bucket。 |
| 點選「建立儲存貯體」 |
 |
| 自定義一個主體名稱 |
 |
| 只是一般或測試用的話，就不用特別設定ACL，除非是較機敏的資料，不然就不用那麼麻煩 |
 |
| 控制版本啟用會比較好，因為可以避免被誤刪，預設的加密方式就用便宜的SSE-S3，並啟用它 |
 |
| 物件鎖定是用來防止刪除的，因為這邊只是測試用而已，就選「停用」，然後按「建立儲存貯體」 |
 |
| 檢視已建好的bucket |
 |
| 點進去後會自建資料夾 |
 |
| 資料夾名稱也是自定義 |
 |
| 檢視已建好的資料夾 |
 |
| 點選「上傳」，準備把資料上傳到此處 |
 |
| 按「新增檔案」 |
 |
| 把要上傳的資料都傳上來，然後按「上傳」 |
 |
| 上傳成功！ |
二．建立IAM user並賦予權限，產出連線金鑰
連結的傳遞必須靠AWS上面的user和role，並且產出一組金鑰，將該金鑰用於之後主機掛載用。
 |
| 到IAM>使用者，建立一個給S3用的user，名稱自定義 |
 |
| 因為是新建的，所以勾選「在群組中新增使用者」 |
 |
| 按「建立使用者」 |
 |
| 使用者建立完以後，點進去選「安全憑證」>「建立存取金鑰」 |
 |
| 存與金鑰實務和替代方案，選「本機代碼」 |
 |
| 自定義一個金鑰名稱 |
 |
| 金鑰發布之後，可以檢視明碼，然後點選下載成csv |
 |
| 回到IAM Dashborad，點選「使用者群組」 |
 |
| 自定義群組名稱，然後勾選剛剛建立的s3 user，並賦予S3 FullAccess權限 |
金鑰下載成csv好了以後要保存好，它是用來提供給用戶端連接S3用的。
 |
| csv裡面是ACCESS_KEY與SECRET_ACCESS_KEY |
三．Linux主機掛載S3資料夾
隨便開一台AlmaLinux主機，測試如何掛載S3資料夾，並且新增檔案。主要流程為：
1.下載並安裝s3fs。
2.將下載好的accessKeys複製到本機上。
3.建立一個資料夾並掛載S3。
[root@lab-s3test ~]# sudo yum -y install epel-release
[root@lab-s3test ~]# sudo yum -y install s3fs-fuse
[root@lab-s3test ~]# echo "AKIAVGEBRIPHFFZ6B3FO:RrY4RjgU96A2ayHTWOLmA95S4YGr0FcRC9Z7S3wZ" | sudo tee /etc/passwd-s3fs
[root@lab-s3test ~]# sudo yum -y install s3fs-fuse
[root@lab-s3test ~]# echo "AKIAVGEBRIPHFFZ6B3FO:RrY4RjgU96A2ayHTWOLmA95S4YGr0FcRC9Z7S3wZ" | sudo tee /etc/passwd-s3fs
\\製作一個金鑰檔，規則就用ACCESS_KEY:SECRET_ACCESS_KEY 的方式
[root@lab-s3test ~]# sudo chmod 600 /etc/passwd-s3fs
[root@lab-s3test ~]# sudo mkdir /mnt/s3bucket-ian
[root@lab-s3test ~]# sudo s3fs s3bucket-ian /mnt/s3bucket-ian
[root@lab-s3test ~]# sudo mkdir /mnt/s3bucket-ian
[root@lab-s3test ~]# sudo s3fs s3bucket-ian /mnt/s3bucket-ian
 |
| 用指令df -h檢視資料夾 |
[root@lab-s3test ~]# vim /etc/fstab
s3fs#s3bucket-ian /mnt/s3bucket-ian fuse _netdev,allow_other,url=https://s3.amazonaws.com 0 0
\\在最下面新增這一行
[root@lab-s3test ~]# mount -a
 |
| 永久性掛載的方式就把指令輸入在/etc/fstab底下，然後mount -a掛載即可 |
四．Windows主機掛載S3資料夾
Windows要mount S3比Linux複雜一點，因為步驟較多，要安裝套件以外，設定流程也比較長。
1.下載rclone
下載點有很多，Google一下就很多可以選擇：
2.choco安裝winfsp
PS C:\>choco install winfsp -y \\Choco套件已經用很久了就不多說
3.用Powershell安裝
PS C:\> mkdir rclone \\建立rclone資料夾在C槽
PS C:\rclone> cd C:\rclone\
PS C:\rclone> mkdir s3bucket-ian \\建立一個同s3 bucket名稱的資料夾
PS C:\rclone> mv C:\Users\Administrator\Downloads\rclone-v1.67.0-windows-amd64\* C:\rclone\ \\把下載好的rclone套件全部複製到rclone資料夾底下
PS C:\rclone> dir
Mode LastWriteTime Length Name
---- ------------- ------ ----
-a---- 2024/6/14 下午 11:26 89107 git-log.txt
-a---- 2024/6/14 下午 11:18 2590363 rclone.1
-a---- 2024/6/14 下午 11:29 61242880 rclone.exe
-a---- 2024/6/14 下午 11:18 2805365 README.html
-a---- 2024/6/14 下午 11:18 2328346 README.txt
PS C:\rclone> .\rclone.exe config \\用指令開始進行配置
Current remotes:
Name Type
==== ====
s3bucket-ian s3
e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> n \\建立一個新的remote
Enter name for new remote.
name> s3bucket-ian \\新的remote名稱跟s3 bucket一樣比較好
Option Storage.
Type of storage to configure.
Choose a number from below, or type in your own value.
1 / 1Fichier
\ (fichier)
2 / Akamai NetStorage
\ (netstorage)
3 / Alias for an existing remote
\ (alias)
4 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, ArvanCloud, Ceph, ChinaMobile, Cloudflare, DigitalOcean, Dreamhost, GCS, HuaweiOBS, IBMCOS, IDrive, IONOS, LyveCloud, Leviia, Liara, Linode, Magalu, Minio, Netease, Petabox, RackCorp, Rclone, Scaleway, SeaweedFS, StackPath, Storj, Synology, TencentCOS, Wasabi, Qiniu and others
\ (s3)
5 / Backblaze B2
\ (b2)
6 / Better checksums for other remotes
\ (hasher)
7 / Box
\ (box)
8 / Cache a remote
\ (cache)
9 / Citrix Sharefile
\ (sharefile)
10 / Combine several remotes into one
\ (combine)
11 / Compress a remote
\ (compress)
12 / Dropbox
\ (dropbox)
13 / Encrypt/Decrypt a remote
\ (crypt)
14 / Enterprise File Fabric
\ (filefabric)
15 / FTP
\ (ftp)
16 / Google Cloud Storage (this is not Google Drive)
\ (google cloud storage)
17 / Google Drive
\ (drive)
18 / Google Photos
\ (google photos)
19 / HTTP
\ (http)
20 / Hadoop distributed file system
\ (hdfs)
21 / HiDrive
\ (hidrive)
22 / ImageKit.io
\ (imagekit)
23 / In memory object storage system.
\ (memory)
24 / Internet Archive
\ (internetarchive)
25 / Jottacloud
\ (jottacloud)
26 / Koofr, Digi Storage and other Koofr-compatible storage providers
\ (koofr)
27 / Linkbox
\ (linkbox)
28 / Local Disk
\ (local)
29 / Mail.ru Cloud
\ (mailru)
30 / Mega
\ (mega)
31 / Microsoft Azure Blob Storage
\ (azureblob)
32 / Microsoft Azure Files
\ (azurefiles)
33 / Microsoft OneDrive
\ (onedrive)
34 / OpenDrive
\ (opendrive)
35 / OpenStack Swift (Rackspace Cloud Files, Blomp Cloud Storage, Memset Memstore, OVH)
\ (swift)
36 / Oracle Cloud Infrastructure Object Storage
\ (oracleobjectstorage)
37 / Pcloud
\ (pcloud)
38 / PikPak
\ (pikpak)
39 / Proton Drive
\ (protondrive)
40 / Put.io
\ (putio)
41 / QingCloud Object Storage
\ (qingstor)
42 / Quatrix by Maytech
\ (quatrix)
43 / SMB / CIFS
\ (smb)
44 / SSH/SFTP
\ (sftp)
45 / Sia Decentralized Cloud
\ (sia)
46 / Storj Decentralized Cloud Storage
\ (storj)
47 / Sugarsync
\ (sugarsync)
48 / Transparently chunk/split large files
\ (chunker)
49 / Uloz.to
\ (ulozto)
50 / Union merges the contents of several upstream fs
\ (union)
51 / Uptobox
\ (uptobox)
52 / WebDAV
\ (webdav)
53 / Yandex Disk
\ (yandex)
54 / Zoho
\ (zoho)
55 / premiumize.me
\ (premiumizeme)
56 / seafile
\ (seafile)
Storage> 4 \\Storage Tpye就選 AWS S3
Option provider.
Choose your S3 provider.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
1 / Amazon Web Services (AWS) S3
\ (AWS)
2 / Alibaba Cloud Object Storage System (OSS) formerly Aliyun
\ (Alibaba)
3 / Arvan Cloud Object Storage (AOS)
\ (ArvanCloud)
4 / Ceph Object Storage
\ (Ceph)
5 / China Mobile Ecloud Elastic Object Storage (EOS)
\ (ChinaMobile)
6 / Cloudflare R2 Storage
\ (Cloudflare)
7 / DigitalOcean Spaces
\ (DigitalOcean)
8 / Dreamhost DreamObjects
\ (Dreamhost)
9 / Google Cloud Storage
\ (GCS)
10 / Huawei Object Storage Service
\ (HuaweiOBS)
11 / IBM COS S3
\ (IBMCOS)
12 / IDrive e2
\ (IDrive)
13 / IONOS Cloud
\ (IONOS)
14 / Seagate Lyve Cloud
\ (LyveCloud)
15 / Leviia Object Storage
\ (Leviia)
16 / Liara Object Storage
\ (Liara)
17 / Linode Object Storage
\ (Linode)
18 / Magalu Object Storage
\ (Magalu)
19 / Minio Object Storage
\ (Minio)
20 / Netease Object Storage (NOS)
\ (Netease)
21 / Petabox Object Storage
\ (Petabox)
22 / RackCorp Object Storage
\ (RackCorp)
23 / Rclone S3 Server
\ (Rclone)
24 / Scaleway Object Storage
\ (Scaleway)
25 / SeaweedFS S3
\ (SeaweedFS)
26 / StackPath Object Storage
\ (StackPath)
27 / Storj (S3 Compatible Gateway)
\ (Storj)
28 / Synology C2 Object Storage
\ (Synology)
29 / Tencent Cloud Object Storage (COS)
\ (TencentCOS)
30 / Wasabi Object Storage
\ (Wasabi)
31 / Qiniu Object Storage (Kodo)
\ (Qiniu)
32 / Any other S3 compatible provider
\ (Other)
provider> 1 \\Provider就選 AWS
Option env_auth.
Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
Only applies if access_key_id and secret_access_key is blank.
Choose a number from below, or type in your own boolean value (true or false).
Press Enter for the default (false).
1 / Enter AWS credentials in the next step.
\ (false)
2 / Get AWS credentials from the environment (env vars or IAM).
\ (true)
env_auth> 1 \\驗證方式選AWS credentials
Option access_key_id.
AWS Access Key ID.
Leave blank for anonymous access or runtime credentials.
Enter a value. Press Enter to leave empty.
access_key_id> AKIAVGEBRIPHFFZ6B3FO \\access key id就用前面下載的key
Option secret_access_key.
AWS Secret Access Key (password).
Leave blank for anonymous access or runtime credentials.
Enter a value. Press Enter to leave empty.
secret_access_key> RrY4RjgU96A2ayHTWOLmA95S4YGr0FcRC9Z7S3wZ \\secret access key也一樣
Option region.
Region to connect to.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
/ The default endpoint - a good choice if you are unsure.
1 | US Region, Northern Virginia, or Pacific Northwest.
| Leave location constraint empty.
\ (us-east-1)
/ US East (Ohio) Region.
2 | Needs location constraint us-east-2.
\ (us-east-2)
/ US West (Northern California) Region.
3 | Needs location constraint us-west-1.
\ (us-west-1)
/ US West (Oregon) Region.
4 | Needs location constraint us-west-2.
\ (us-west-2)
/ Canada (Central) Region.
5 | Needs location constraint ca-central-1.
\ (ca-central-1)
/ EU (Ireland) Region.
6 | Needs location constraint EU or eu-west-1.
\ (eu-west-1)
/ EU (London) Region.
7 | Needs location constraint eu-west-2.
\ (eu-west-2)
/ EU (Paris) Region.
8 | Needs location constraint eu-west-3.
\ (eu-west-3)
/ EU (Stockholm) Region.
9 | Needs location constraint eu-north-1.
\ (eu-north-1)
/ EU (Milan) Region.
10 | Needs location constraint eu-south-1.
\ (eu-south-1)
/ EU (Frankfurt) Region.
11 | Needs location constraint eu-central-1.
\ (eu-central-1)
/ Asia Pacific (Singapore) Region.
12 | Needs location constraint ap-southeast-1.
\ (ap-southeast-1)
/ Asia Pacific (Sydney) Region.
13 | Needs location constraint ap-southeast-2.
\ (ap-southeast-2)
/ Asia Pacific (Tokyo) Region.
14 | Needs location constraint ap-northeast-1.
\ (ap-northeast-1)
/ Asia Pacific (Seoul).
15 | Needs location constraint ap-northeast-2.
\ (ap-northeast-2)
/ Asia Pacific (Osaka-Local).
16 | Needs location constraint ap-northeast-3.
\ (ap-northeast-3)
/ Asia Pacific (Mumbai).
17 | Needs location constraint ap-south-1.
\ (ap-south-1)
/ Asia Pacific (Hong Kong) Region.
18 | Needs location constraint ap-east-1.
\ (ap-east-1)
/ South America (Sao Paulo) Region.
19 | Needs location constraint sa-east-1.
\ (sa-east-1)
/ Israel (Tel Aviv) Region.
20 | Needs location constraint il-central-1.
\ (il-central-1)
/ Middle East (Bahrain) Region.
21 | Needs location constraint me-south-1.
\ (me-south-1)
/ Africa (Cape Town) Region.
22 | Needs location constraint af-south-1.
\ (af-south-1)
/ China (Beijing) Region.
23 | Needs location constraint cn-north-1.
\ (cn-north-1)
/ China (Ningxia) Region.
24 | Needs location constraint cn-northwest-1.
\ (cn-northwest-1)
/ AWS GovCloud (US-East) Region.
25 | Needs location constraint us-gov-east-1.
\ (us-gov-east-1)
/ AWS GovCloud (US) Region.
26 | Needs location constraint us-gov-west-1.
\ (us-gov-west-1)
region> 14 \\時區很重要，要選跟前面設定的一致，這邊是東京
Option endpoint.
Endpoint for S3 API.
Leave blank if using AWS to use the default endpoint for the region.
Enter a value. Press Enter to leave empty.
endpoint>
Option location_constraint.
Location constraint - must be set to match the Region.
Used when creating buckets only.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
1 / Empty for US Region, Northern Virginia, or Pacific Northwest
\ ()
2 / US East (Ohio) Region
\ (us-east-2)
3 / US West (Northern California) Region
\ (us-west-1)
4 / US West (Oregon) Region
\ (us-west-2)
5 / Canada (Central) Region
\ (ca-central-1)
6 / EU (Ireland) Region
\ (eu-west-1)
7 / EU (London) Region
\ (eu-west-2)
8 / EU (Paris) Region
\ (eu-west-3)
9 / EU (Stockholm) Region
\ (eu-north-1)
10 / EU (Milan) Region
\ (eu-south-1)
11 / EU Region
\ (EU)
12 / Asia Pacific (Singapore) Region
\ (ap-southeast-1)
13 / Asia Pacific (Sydney) Region
\ (ap-southeast-2)
14 / Asia Pacific (Tokyo) Region
\ (ap-northeast-1)
15 / Asia Pacific (Seoul) Region
\ (ap-northeast-2)
16 / Asia Pacific (Osaka-Local) Region
\ (ap-northeast-3)
17 / Asia Pacific (Mumbai) Region
\ (ap-south-1)
18 / Asia Pacific (Hong Kong) Region
\ (ap-east-1)
19 / South America (Sao Paulo) Region
\ (sa-east-1)
20 / Israel (Tel Aviv) Region
\ (il-central-1)
21 / Middle East (Bahrain) Region
\ (me-south-1)
22 / Africa (Cape Town) Region
\ (af-south-1)
23 / China (Beijing) Region
\ (cn-north-1)
24 / China (Ningxia) Region
\ (cn-northwest-1)
25 / AWS GovCloud (US-East) Region
\ (us-gov-east-1)
26 / AWS GovCloud (US) Region
\ (us-gov-west-1)
location_constraint> 14 \\location也一樣選東京
Option acl.
Canned ACL used when creating buckets and storing or copying objects.
This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
Note that this ACL is applied when server-side copying objects as S3
doesn't copy the ACL from the source but rather writes a fresh one.
If the acl is an empty string then no X-Amz-Acl: header is added and
the default (private) will be used.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
/ Owner gets FULL_CONTROL.
1 | No one else has access rights (default).
\ (private)
/ Owner gets FULL_CONTROL.
2 | The AllUsers group gets READ access.
\ (public-read)
/ Owner gets FULL_CONTROL.
3 | The AllUsers group gets READ and WRITE access.
| Granting this on a bucket is generally not recommended.
\ (public-read-write)
/ Owner gets FULL_CONTROL.
4 | The AuthenticatedUsers group gets READ access.
\ (authenticated-read)
/ Object owner gets FULL_CONTROL.
5 | Bucket owner gets READ access.
| If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
\ (bucket-owner-read)
/ Both the object owner and the bucket owner get FULL_CONTROL over the object.
6 | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
\ (bucket-owner-full-control)
acl> \\acl不用的就留空白，然後Enter
Option server_side_encryption.
The server-side encryption algorithm used when storing this object in S3.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
1 / None
\ ()
2 / AES256
\ (AES256)
3 / aws:kms
\ (aws:kms)
server_side_encryption> 1 \\若資料不重要，Server side encryption選 None
Option sse_kms_key_id.
If using KMS ID you must provide the ARN of Key.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
1 / None
\ ()
2 / arn:aws:kms:*
\ (arn:aws:kms:us-east-1:*)
sse_kms_key_id>
PS C:\rclone> 1 \\因為沒有用KMS，所以sse kms key就選None
Option storage_class.
The storage class to use when storing new objects in S3.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
1 / Default
\ ()
2 / Standard storage class
\ (STANDARD)
3 / Reduced redundancy storage class
\ (REDUCED_REDUNDANCY)
4 / Standard Infrequent Access storage class
\ (STANDARD_IA)
5 / One Zone Infrequent Access storage class
\ (ONEZONE_IA)
6 / Glacier storage class
\ (GLACIER)
7 / Glacier Deep Archive storage class
\ (DEEP_ARCHIVE)
8 / Intelligent-Tiering storage class
\ (INTELLIGENT_TIERING)
9 / Glacier Instant Retrieval storage class
\ (GLACIER_IR)
storage_class> 2 \\Storage Class也要跟前面設定的一樣，這裡選S3 Standard
Edit advanced config?
y) Yes
n) No (default)
y/n> n \\沒有進階設定，選n
Remote config
--------------------
[s3bucket-ian]
type = s3
provider = AWS
env_auth = false
access_key_id = AKIAVGEBRIPHFFZ6B3FO
secret_access_key = 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
region = ap-northeast-1
location_constraint = ap-northeast-1
storage_class = STANDARD
--------------------
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y \\確定好所有設定後就按y
Current remotes:
Name Type
==== ====
s3bucket-ian s3
e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q \\完成後按q退出rclone設定
PS C:\rclone> .\rclone.exe lsd s3bucket-ian:
PS C:\rclone> .\rclone mount s3bucket-ian:s3bucket-ian S: --vfs-cache-mode full
The service rclone has been started.
 |
| 到檔案總管上就會看到掛載的S磁區 |
這個只是臨時掛載S3，一但在Powershell用Ctrl+C強制中止，或者是Windows重開機就會卸除掛載，想要永久性掛載就得要將整段指令做成一個exe檔，並放在開機啟動程式下的資料夾，重開機後指令自己啟動便掛載S3磁區。
 |
| 到C:\ProgramData\Microsoft\Windows\Start Menu\Programs\Startup路徑下，建立一個檔名rclone-S3.cmd的檔案 |
 |
| 檔案內容貼這串指令：C:\rclone\rclone.exe mount s3bucket-ian:s3bucket-ian S: --vfs-cache-mode full |
S3還有很多可以掛載到各種系統的方式，我只挑選Linux和Windows比較常用的兩套系統，若之後還有更多系統要測試就到時候再說吧！
沒有留言:
張貼留言