Using GitLab to deploy app to Google App Engine:
1.
Create a service account from GCP’s iam-admim:
This will generate johnzeng-nodejs-70db00dcac20.json file
which contains the private key of service account ‘gitlab-service-account’.
2.
Setup Gitlab CI
SJCMACJ15JHTD8:action-service-johnzeng-nodejs
jzeng$ gcloud auth application-default login
The
environment variable [GOOGLE_APPLICATION_CREDENTIALS] is set to:
[/Users/jzeng/Documents/johnzeng-nodejs-70db00dcac20.json]
Credentials
will still be generated to the default location:
[/Users/jzeng/.config/gcloud/application_default_credentials.json]
To
use these credentials, unset this environment variable before
running
your application.
Use above generated key (a json file) to access fileStore
from app (GAE flexible):
"fireStore": {
"projectID": "johnzeng-nodejs",
"keyFile": "./credentials/key.json"
},
"projectID": "johnzeng-nodejs",
"keyFile": "./credentials/key.json"
},
(above is from config/dev.json)
Google Cloud KMS
a cloud-hosted key
management service that lets you manage cryptographic keys for your cloud
services. You can generate, use, rotate, and destroy cryptographic keys. Cloud
KMS is integrated with Cloud IAM and Cloud Audit Logging so that you can manage
permissions on individual keys and monitor how these are used.
// create key-ring (think of this
as grouping)
gcloud kms keyrings create [KEYRING_NAME] \
--location [LOCATION] \
--project live-project-id
gcloud kms keyrings create [KEYRING_NAME] \
--location [LOCATION] \
--project live-project-id
// create the encryption key
gcloud kms keys create [KEY_NAME] \
--location [LOCATION] \
--keyring [KEYRING_NAME] \
--purpose encryption \
--project live-project-id
gcloud kms keys create [KEY_NAME] \
--location [LOCATION] \
--keyring [KEYRING_NAME] \
--purpose encryption \
--project live-project-id
Create key in nodejs:
encrypt/decrypt secrets in nodejs:
Dealing with permission for the key:
Encrypt/Decrypt data with asymmetric key:
Asymmetric
encryption uses the public key portion of the asymmetric key and decryption
uses the private key portion of the key. Cloud KMS provides functionality to
retrieve the public key and functionality to decrypt ciphertext that was
encrypted with the public key. Cloud KMS does not allow direct access to the
private key.
gcloud kms encrypt --location 'global' --keyring
action-service-token --key as-jwt-encrypt-key --plaintext-file=./key.pem.clear
--ciphertext-file=./encrypted_key.pem
More about Client libraries:
KMS REST API:
Need to enable KMS API first:
Then, you can create a key ring:
curl --request POST \
'https://cloudkms.googleapis.com/v1/projects/johnzeng-nodejs/locations/global/keyRings?keyRingId=johnzeng-keyring'
\
--header
'Authorization: Bearer [YOUR_ACCESS_TOKEN]' \
--header 'Accept:
application/json' \
--header
'Content-Type: application/json' \
--data '{}' \
--compressed
create a key:
curl --request POST \
'https://cloudkms.googleapis.com/v1/projects/johnzeng-nodejs/locations/global/keyRings/johnzeng-keyring/cryptoKeys?cryptoKeyId=johnzeng-key1'
\
--header
'Authorization: Bearer [YOUR_ACCESS_TOKEN]' \
--header 'Accept:
application/json' \
--header
'Content-Type: application/json' \
--data
'{"purpose":"ENCRYPT_DECRYPT"}' \
--compressed
decrypt:
curl --request POST \
'https://cloudkms.googleapis.com/v1/projects/johnzeng-nodejs/locations/global/keyRings/johnzeng-keyring/cryptoKeys/johnzeng-key1:decrypt'
\
--header
'Authorization: Bearer [YOUR_ACCESS_TOKEN]' \
--header 'Accept:
application/json' \
--header
'Content-Type: application/json' \
--data
'{"ciphertext":"CiQAa0s+mU+dhVvgVSHMka2d2oj09aEL/x835LTELldFBmoSmakSMACzrBlgDT/1tK0gUP+GMWxHHOxNHzaPw4WoTLmiKBUvzAJLEIJKeB+QgYN0Q5l+/w=="}'
\
--compressed
Response from above (which is the base64 encoded “ABCDEFG”):
{
"plaintext": "QUJDREVGRw=="
}
Get ACCESS_TOKEN for default project:
gcloud
auth application-default print-access-token
SJCMACJ15JHTD8:~
jzeng$ gcloud auth print-access-token
ya29.Gl3mBtmxGWXpfqMpwv3Xr2dJHeXbwBP1Fyq3LKfATujYssphUuRINEE0oL3kytv2Fj89sItz6zclzrxZq8SmTTSLJL_t487XaYwkd3e8qRtF3DxNQQDJ4HdJrXSXdj0
SJCMACJ15JHTD8:~
jzeng$ gcloud auth application-default print-access-token
ya29.c.ElrmBtdEwxU3KOtKRjTKJbub3gq2XYBpXHExbuBKl5FbN8ZCkExW33UEonNqFMKzc_idzbie4_EiB7QhL0wEwlgAZzEHoSXBDbeExZWBS2OpS1XOmmS6r_CbQkE
Granting
permissions to use keys:
gcloud
kms keys add-iam-policy-binding \
golden-egg --location global --keyring golden-goose \
--member serviceAccount:my-service-account@my-project.iam.gserviceaccount.com \
--role roles/cloudkms.cryptoKeyEncrypterDecrypter
golden-egg --location global --keyring golden-goose \
--member serviceAccount:my-service-account@my-project.iam.gserviceaccount.com \
--role roles/cloudkms.cryptoKeyEncrypterDecrypter
OAuth2:
Client obtains OAuth 2.0 credentials such as a client
ID. Then gets access token. Then use access token to call Google API.
Difference between “gcloud auth
application-default login” and “gcloud auth login”
As a developer, I want to
interact with GCP via gcloud.
gcloud auth login
This obtains your credentials and stores them in ~/.config/gcloud/. Now you can run gcloudcommands from your terminal and it will find your credentials automatically. Any code/SDK you want to use will not automatically pick up your creds in this case.
gcloud auth login
This obtains your credentials and stores them in ~/.config/gcloud/. Now you can run gcloudcommands from your terminal and it will find your credentials automatically. Any code/SDK you want to use will not automatically pick up your creds in this case.
As a developer, I want my code
to interact with GCP via SDK.
gcloud auth application-default login
This obtains you credentials and stores them in 'the well-known location for Application Default Credentials'. Now any code/SDK you run will be able to find the credentials automatically. This is a good stand-in when locally testing code which would normally use service account credentials (which normally would sit in a file on your server, not on your local computer).
gcloud auth application-default login
This obtains you credentials and stores them in 'the well-known location for Application Default Credentials'. Now any code/SDK you run will be able to find the credentials automatically. This is a good stand-in when locally testing code which would normally use service account credentials (which normally would sit in a file on your server, not on your local computer).
This
command is useful when you are developing code that would normally use a
service account but need to run the code in a local development environment
where it's easier to provide user credentials.
Service account for App Engine
After you create an App Engine application, the App Engine
default service account is created and used as the identity of
the App Engine service. The App Engine default service account is associated
with your GCP project and executes tasks on behalf of your apps running in App
Engine.
By default, the App Engine default service account has the
Editor role in the project. This means that any user account with sufficient
permissions to deploy changes to the GCP project can also run code with
read/write access to all resources within that project.
The App Engine flexible environment service account is a
separate service account from the App Engine default service account. The App Engine flexible environment service account is not
listed on the Service Accounts page of the GCP Console and has the following
restrictions:
·
Do not modify the
permissions of the App Engine flexible environment service account.
·
Avoid using the
related App Engine Flexible Environment Service Agent role with any user account. You cannot rely on the role because
it can change without notice.
The App Engine flexible environment service account is not
listed on the Service Accounts page of the GCP Console. So, to verify that the
App Engine flexible environment service account exists in your GCP project, you
must view the Permissions page in the GCP Console:
1.
Open the GCP Console:
2.
In the Members list, locate the ID of the App Engine flexible
environment service account.
The App Engine flexible
environment service account uses the member ID:
service-[YOUR_PROJECT_ID]@gae-api-prod.google.com.iam.gserviceaccount.com
service-[YOUR_PROJECT_ID]@gae-api-prod.google.com.iam.gserviceaccount.com
3. The App Engine flexible
environment service account should have the App Engine Flexible
Environment Service Agent role.
Creating and
managing service account:
Create service account
key:
gcloud iam
service-accounts keys create ~/key.json
--iam-account [SA-NAME]@xyz.gserviceaccount.com
--iam-account [SA-NAME]@xyz.gserviceaccount.com
Proposed permission
for accessing KMS:
App engine default service account: “Cloud KMS
CryptoKey Decrypter”
Samples
for giving ‘decryption’ permission to service account (App Engine flexible environment service account):
(need to give permission to both keyring and the key)
SJCMACJ15JHTD8:~
jzeng$ gcloud kms keyrings add-iam-policy-binding action-service-token --location
'global' --member serviceAccount:
serviceAccount:service-195685343774@gae-api-prod.google.com.iam.gserviceaccount.com
--role roles/cloudkms.cryptoKeyDecrypter
Updated
IAM policy for keyring [action-service-token].
bindings:
-
members:
- serviceAccount:serviceAccount:service-195685343774@gae-api-prod.google.com.iam.gserviceaccount.com
role: roles/cloudkms.cryptoKeyDecrypter
etag:
BwWGNPWT30U=
SJCMACJ15JHTD8:action-service
jzeng$ gcloud kms keys add-iam-policy-binding as-jwt-encrypt-key --keyring
action-service-token --location 'global' --member
serviceAccount:service-195685343774@gae-api-prod.google.com.iam.gserviceaccount.com
--role roles/cloudkms.cryptoKeyDecrypter
Updated
IAM policy for key [as-jwt-encrypt-key].
bindings:
-
members:
-
serviceAccount:service-195685343774@gae-api-prod.google.com.iam.gserviceaccount.com
role: roles/cloudkms.cryptoKeyDecrypter
etag:
BwWGRgFXpu4=
Samples for getting permission info about key ring:
SJCMACJ15JHTD8:action-service
jzeng$ gcloud kms keyrings get-iam-policy action-service-token --location
'global'
bindings:
-
members:
-
serviceAccount:johnzeng-nodejs@appspot.gserviceaccount.com
-
serviceAccount:service-195685343774@gae-api-prod.google.com.iam.gserviceaccount.com
role: roles/cloudkms.cryptoKeyDecrypter
etag:
BwWGRdF99WY=
SJCMACJ15JHTD8:action-service
jzeng$ gcloud kms keys get-iam-policy as-jwt-encrypt-key --keyring
action-service-token --location 'global'
bindings:
- members:
-
serviceAccount:service-195685343774@gae-api-prod.google.com.iam.gserviceaccount.com
role: roles/cloudkms.cryptoKeyDecrypter
etag:
BwWGRgFXpu4=
Enable Stackdriver debugger for nodejs:
Access the docker running your app under App Engine
Flexible:
gcloud
app --project [PROJECT-ID] instances enable-debug
visit
GCP Console instance page for your project and choose SSH.
sudo
docker exec -it [CONTAINER-NAME] /bin/bash
Appendix: Notes from Google Cloud training:
GCP SDK:
gcloud
gsutil
Google Cloud Console
gcloud init
cloud based source repository:
bitbucket or github
cloud diagnostic tool
computing service:
Google app engine (GAE):
container engine (GKE): docker
compute engine (GCE): virtual machine
cloud functions (serverless environment)
Access to GCE:
ssh from console
ssh from cloudshell
ssh via SDK
Load Balancing
Global External Load Balancing:
HTTP(s) LB
Global Forwarding
Target Proxy
URL Map
backend service
Health check
SSL Load Balancer
TCP LB
Regional LB
Internal LB
Internal LB
External LB (Network LB)
HA
Auto Scaling
Stackdriver Monitoring metrics
KGE:
Kubernetes Master
Nodes
Node pools
Multi-zone cluster
Cluster autoscaling and balancing
Internal Load Balancing
kubectl
Kubernetes Cluster Federation
IAM
Role based access control
Demo for creating container cluster:
create docker image from local machine
create cluster using “gcloud container clusters {cluster_name} --num-nodes=3”
gcloude docker – push {docker_image}: push the local docker image to Google cloud container registry.
kubectl expose deployment
kubectl get service
kubectl get pods
kubectl scale deployment hello-web – replicas=3
Cloud Function (Lambda in AWS):
running on serverless environment.
Events & Trigger (i.e.. Firebase, etc.)
webhook
Access control of cloud function
Demo for Cloud Function:
Create function
GCP Storage & Database Service
Storage: GCS
Bucket
Object is immutable
Server side encryption: default; use default key or customer offered key
Client side encryption:
ACL:
Consistency:
transfer service: gsutil
Database:
Cloud SQL (my sql), Cloud Spanner, Cloud DataStore (mongo), BigTable(HBase), Firebase, BigQuery, etc.
