In this guide I will show you how to sync ICOS bucket objects between accounts using Code Engine. Code Engine provides a platform to unify the deployment of all of your container-based applications on a Kubernetes-based infrastructure. The Code Engine experience is designed so that you can focus on writing code without the need for you to learn, or even know about, Kubernetes.
Code Engine is currently an experimental offering and all resources are deleted every 7 days.
We will be using Cloud Shell to generate Service IDs and Object Storage credentials for both the source and destination accounts.
Source Account
We will create a service ID on the source account. A service ID identifies a service or application similar to how a user ID identifies a user. We can assign specific access policies to the service ID that restrict permissions for using specific services: in this case it gets read-only access to an IBM Cloud Object Storage bucket.
Create Service ID
1
$ ibmcloud iam service-id-create <name-of-your-service-id> --description "Service ID for read-only access to bucket" --output json
Copied!
Service ID Creation
Create Reader access policy for newly created service id
Now we will limit the scope of this service ID to have read only access to our source Object Storage bucket.
Service Instance GUID - This is the GUID of the Cloud Object Storage instance. You can retrieve this with the command: ibmcloud resource service-instance <name of icos instance>
Expected Output Example
Generate HMAC credentials tied to our service ID
In order for the Minio client to talk to each Object Storage instance it will need HMAC credentials (Access Key and Secret Key in S3 parlance).
Save the access_key_id and secret_access_key as we will be using these with our Kubernetes job.
Create HMAC Credentials
Destination Account
We will create a service ID on the destination account. A service ID identifies a service or application similar to how a user ID identifies a user. We can assign specific access policies to the service ID that restrict permissions for using specific services: in this case it gets write access to an IBM Cloud Object Storage bucket.
Create Service ID
1
$ ibmcloud iam service-id-create <name-of-your-service-id> --description "Service ID for write access to bucket" --output json
Copied!
Expected Output Example
Create Reader access policy for newly created service id
Now we will limit the scope of this service ID to have read only access to our source Object Storage bucket.
Service Instance GUID - This is the GUID of the Cloud Object Storage instance. You can retrieve this with the command: ibmcloud resource service-instance <name of icos instance>
Generate HMAC credentials tied to our service ID
We'll follow the same procedure as last time to generate the HMAC credentials, but this time on the destination account.
Save the access_key_id and secret_access_key as we will be using these with our Kubernetes job.
Create free IBM Cloud Kubernetes cluster
You can have 1 free cluster at a time in IBM Cloud Kubernetes Service, and each free cluster expires in 30 days. This will allow is to run the Minio sync as a Kubernetes Job.
1
$ ibmcloud ks cluster create classic --name <name of cluster>
Copied!
The cluster should be ready in about 10-15 minutes. You can verify the status of the creation by running the ibmcloud ks clusters and seeing when State returns as normal:
Name ID State Created Workers Location Version Resource Group Name Provider
4
freeikscluster bstcs2nf0bkrt2968nm0 normal 14 minutes ago 1 par01 1.17.9_1534 default classic
Copied!
Create Kubernetes secret for use with job
In order for our container to speak to the Cloud Object storage buckets we'll need to specify some environmental variables to be stored as Kubernetes secrets. The first step is to target our free cluster:
1
$ ibmcloud ks cluster config --cluster <name of cluster>
Copied!
With kubectl now configured to talk to our cluster we can create our Kubernetes secrets.
With the Job spec downloaded we can now submit the job to Kubernetes
1
$ kubectl create -f job.yaml
Copied!
In my testing I am only syncing a few times so by the time I check the Kubernetes pods, the job has already completed. Looking at the logs I am able to verify that the contents have been synced between the Object Storage buckets.