Migrate file S3 AWS to S3 NIPA

Migrating Data from Amazon S3 to NIPA Cloud Space S3 Buckets Using s3fs and rsync

For customers who wish to move data from Amazon S3 to a NIPA Cloud Space S3 Bucket, the process can be accomplished using s3fs and rsync by following the steps below.

Restriction

1. You must have S3 buckets, Access IDs, and Secret Keys ready for both your AWS and NIPA Cloud Space accounts. (For help with AWS credentials, see the official AWS documentation).

2. A Compute Instance for mounting s3fs. (This guide uses Ubuntu 22.04 as an example).

Instruction

The process for syncing files from an AWS S3 bucket to a NIPA Cloud Space Object Storage bucket is as follows:

1. Create a Bucket on NIPA Cloud Space

Create your S3 bucket on the NIPA Cloud Space platform by following the guides below:

• Create an Object Storage Bucket

• Create an Object Storage Sub-User (for Access ID and Secret Key)

• Create a Bucket Policy

2. Secure Shell

Once your NIPA Cloud Space S3 bucket, Access ID, and Secret Key are ready, SSH into your prepared Compute Instance.

3. Install s3fs Dependencies

Install the required dependencies for using s3fs.

apt update
apt-get -y install automake autotools-dev fuse g++ git libcurl4-gnutls-dev libfuse-dev libssl-dev libxml2-dev make pkg-config

# install s3fs-fuse
git clone https://github.com/s3fs-fuse/s3fs-fuse.git
cd s3fs-fuse/
./autogen.sh
./configure --prefix=/usr --with-openssl
make
make install

#check s3fs
which s3fs

4. Create Access Key File

Create a file to store your Access ID and Secret Key credentials.

# Access ID for mount S3 AWS
echo "ACCESS_KEY:SECRET_KEY" | sudo tee /etc/passwd-s3fs-aws
# Access ID for mount S3 NIPA
echo "ACCESS_KEY:SECRET_KEY" | sudo tee /etc/passwd-s3fs-nipa

# Change File's Permission
chmod 600 /etc/passwd-s3fs-aws
chmod 600 /etc/passwd-s3fs-nipa

5. Create Paths

Create the directory paths that will serve as mount points for your AWS S3 bucket and your NIPA Cloud Space S3 bucket.

mkdir -p /mnt/s3aws
mkdir -p /mnt/s3nipacloud

6. Mount the AWS S3 Bucket

Mount the AWS S3 bucket to the /mnt/s3aws directory. The options are detailed below:

sudo s3fs {bucketname} {/mountpoint} -o passwd_file=/etc/passwd-s3fs-aws

• {bucketname} = The name of your bucket (e.g., my-aws-bucket).

• {/mountpoint} = The path where you want to mount the files (e.g., /mnt/s3aws).

# Example Command
sudo s3fs my-bucket /mnt/s3aws -o passwd_file=/etc/passwd-s3fs-aws

7. Mount the NCS Bucket

Mount the NIPA Cloud Space S3 bucket to the /mnt/s3nipacloud directory.

sudo s3fs {bucketname} {/mountpoint} -o passwd_file=/etc/passwd-s3fs-nipa -o allow_other -o url=https://s3-bkk.nipa.cloud

• {bucketname} = The name of your bucket (e.g., my-nipa-bucket).

• {/mountpoint} = The path where you want to mount the files (e.g., /mnt/s3nipacloud).

# Example Command
sudo s3fs my-bucket /mnt/s3nipacloud -o passwd_file=/etc/passwd-s3fs-nipa -o allow_other -o url=https://s3-bkk.nipa.cloud

8. Check Mounting

Verify that the S3 buckets were mounted successfully by running the appropriate command.

df -h

9. Syncing Buckets

Sync the files from the AWS S3 bucket mount point to the NIPA Cloud Space S3 bucket mount point.

rsync -av --progress {/path/to/source/} {/path/to/destination/}

# Example Command
rsync -av --progress /mnt/s3aws/ /mnt/s3nipacloud/

10. Checking

Once the sync is complete, verify that all files have been transferred to the NIPA Cloud Space S3 bucket by running the appropriate command.

diff -r {/path/to/source} {/path/to/destination}
# The diff command displays the differences between files in the source and destination directories. If no output is shown, it means the source and destination are identical.

#Example Command
diff -r /mnt/s3aws /mnt/s3nipacloud