Access GCR images from different GCP projects

The manual way:

https://medium.com/google-cloud/using-single-docker-repository-with-multiple-gke-projects-1672689f780c

For each GCP project that will need to access the GCR images, you give storage.objectViewer permission to its service account in the GCP project where the GCR images will be accessed. 

The way through Terraform code:

resource "google_project_iam_member" "my-project-storage-access" {

  project = "gcp-project-of-gcr-images"

  role = "roles/storage.objectViewer"

  member = "serviceAccount:${google_service_account.my-cluster.email}"

}  

 For each GCP project that will need to access the GCR images, you use the above terraform code to give the service account the storage.objectViewer permission to that GCP project where the GCR images will be accessed. 

How to import IAM role to terraform state?

This sounds like a silly question.  Why IAM role is different?  We all know how to import resources to terraform state, right?

Actually, there are two tricks to this:

1. You have to know the resource id for each IAM role.

2. You have to use double quotes for it.

Let me give you more details on these.

Resource ID format for IAM role contains three components: project name, role, and member.

But the 'terraform import' has only two parameters.  How can you pass 3 components?

Actually, the first parameter is always the resource name so you have only one parameter to use to pass the above three components.  

That is the reason why double quotes are needed. Put all 3 components between double quotes and separate them by space.

Here is an example:

terraform import google_project_iam_memeber.my-cluster-autoscaling-metrics-writer "my-gcp-project roles/autoscaling.metricsWriter serviceAccount:my-cluster@my-gcp-project.iam.gserviceaccount.com"

Ingress Controller (continue)

On my March blog, I compared the community version ingress-nginx controller and the Nginx supported nginx-ingress controller.  My conclusion was the community version is better.  

Now my conclusion still stays.  But in reality, you do not always have the choice.  Sometimes, you have to depend on some features on Nginx plus or just your company decides to use nginx-ingress.  If that is the case, you will have to deal with some inconvenience from that. 

Today, I will share two things that are very easy in the community version but is hard in Nginx's nginx-ingress.

Rewrite Annotation

When we get the request from the customer, we need to route it to a certain URI (for example, /mycheck) to do something (such as a security check) regardless what its original URI is. 

 

The “rewrite” annotation worked very well for the community version of ingress-nginx controller using the annotation like this:

 

nginx.ingress.kubernetes.io/rewrite-target: /mycheck

 

The new way in Nginx plus ingress controller is something like this (they changed the annotation name and syntax):

 

nginx.org/rewrites: "serviceName=my-service rewrite=/mycheck"

 

On both cases, the original path is:

 

- path: /

 

But this does not work for us in Nginx plus ingress controller. 

After hours and hours working on this, I contacted the developer in Nginx team.  What I got from him is:

http://nginx.org/rewrites annotation doesn’t support a rewrite that you need, unfortunately.

The advice from him is:

1. To support it, you can use a custom template. 

Ingress template: https://github.com/nginxinc/kubernetes-ingress/blob/v1.8.0/internal/configs/version1/nginx-plus.ingress.tmpl

Main template: https://github.com/nginxinc/kubernetes-ingress/blob/v1.8.0/internal/configs/version1/nginx-plus.tmpl

2. You can also create a custom annotation 

https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/custom-annotations/

3. Alternatively, you can use a VirtualServer resource that supports a rewrite that you need:

  - action:

      proxy:

        rewritePath: /mycheck

        upstream: tea

    path: ~^/ # this is a regex path. for a rewrite that you need, it is necessary to use regex 

See an example here — https://github.com/nginxinc/kubernetes-ingress/tree/master/examples/rewrites

Multiple Ingresses for the same host

When using the community version ingress controller, we can create multiple ingresses (one for each path) for the same host.   But the Nginx plus ingress controller does not.   I have to put all paths into a single Ingress.  But we do not want to do that for multiple reasons (we may have lots of paths or we need to enable cache for some of the paths, but not all of the paths.  

To work around this in Nginx ingress controller, we need to define additional annotation:

https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/cross-namespace-configuration/

thisisunsafe trick

 When security is important, over-secure is stupid.  Currently, I spent lots of time to figure out an error message from Google Chrome when accessing a system which is set up by my coworker and is safe.  It ends up that I have to play this inconvenient 'thisisunsafe' trick documented by this person:

https://miguelpiedrafita.com/chrome-thisisunsafe

 

Redis running in Kubernetes

There is a need to run Redis inside of Kubernetes.  This will eliminate the egress costs.

Installation is straightforward if you use Bitnami:

Build a snapshot of redis-14.8.8 from https://github.com/bitnami/charts/tree/master/bitnami/redis

helm repo add bitnami https://charts.bitnami.com/bitnami

helm pull bitnami/redis

Then checkin the new tgz file (under 'charts' folder) and other files to keep the current version as a snapshot in git.  The minimal files include:

-rw-r--r--  1 user  1437522721   767 Aug 27 11:11 Chart.yaml

-rw-r--r--  1 user  1437522721   235 Aug 27 10:37 README.md

drwxr-xr-x  4 user  1437522721   128 Aug 27 10:37 charts

Install it to Kubernetes:

helm dependency update

helm upgrade -i --create-namespace redis .

To get the Redis default password: 

kubectl get secret --namespace "redis" redis -o jsonpath="{.data.redis-password}" | base64 --decode

Benchmark of the HMGET 

To prepare testing data, we need to create 10 millions entries in hash key.

If we want to use different key/value, we need to use lua to generate 10 million different key/values for ‘counterhash’ hash table:

@redis-master-0:/tmp$ cp /dev/stdin counterhash.lua

for i=1, 10000000 do

    redis.call('hset', 'counterhash',  'counter' .. i, i)

end

@redis-master-0:/tmp$ redis-cli -a cK56wtmkSg --eval counterhash.lua

Check the size of hashkey ‘counterhash’:

@redis-master-0:/$ redis-cli -a UpvoddWilx               

Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.

127.0.0.1:6379> HLEN counterhash

(integer) 10000000

Benchmark for HMGET when two fields are used:

@redis-master-0:/$ redis-benchmark -a UpvoddWilx -r 20000000 -n 20000000 HMGET counterhash "counter9679805" "counter9458449"

====== HMGET counterhash counter9679805 counter9458449 ======                                                   92)

  20000000 requests completed in 297.01 seconds

  50 parallel clients

  75 bytes payload

  keep alive: 1

  host configuration "save":

  host configuration "appendonly": yes

  multi-thread: no

Summary:

  throughput summary: 67338.93 requests per second

  latency summary (msec):

          avg       min       p50       p95       p99       max

        0.392     0.064     0.399     0.543     0.743     5.183

Benchmark for HMGET when ten fields are used:

@redis-master-0:/$ redis-benchmark -a UpvoddWilx -r 20000000 -n 20000000 HMGET counterhash "counter9679800" "counter9679801" "counter9679802" "counter9679803" "counter9679804" "counter9679805" "counter9679806" "counter9679807" "counter9679808" "counter9679809"

====== HMGET counterhash counter9679800 counter9679801 counter9679802 counter9679803 counter9679804 counter9679805 counter9679806 counter9679807 counter9679808 counter9679809 ======

  20000000 requests completed in 273.58 seconds

  50 parallel clients

  244 bytes payload

  keep alive: 1

  host configuration "save":

  host configuration "appendonly": yes

  multi-thread: no

Summary:

  throughput summary: 73103.42 requests per second

  latency summary (msec):

          avg       min       p50       p95       p99       max

        0.360     0.136     0.335     0.503     0.711     8.383

Rebuild Dockerfile from a docker image

alias dfimage="docker run -v /var/run/docker.sock:/var/run/docker.sock --rm alpine/dfimage"

dfimage -sV=1.36 nginx:latest

PromQL and Grafana

PromQL syntax is kind of non-intuitive and hard to remember.  If you use it everyday, you should be happy.  But if you use it occasionally, you might have to learn it each time when you are using.  In my case, I totally forgot its syntax even I used it two years ago.  

Now, I decided to document some of them so I will not have to learn it again next time.

1. Fiter in metrics

Without filter, you will get too many things you do not need.  For example, you will get graphs for all containers without filter in PromQL. That will be hard for you to find the container or pod you are trying to monitor. By applying filter in "metrics", you can display metrics for the ones you are interested. 

I will go through an example to explain some key syntax for filters:

{namespace="your_name_space",pod=~"main-api.+",container="main-api-container"}

• If you want to exact match, put your name insider the double quotes.  This is the easiest case.
• If you want to use wild card match, "*" will not work.  Instead, you need to use regex and put ~ (tilde) before the regex.  This syntax is weird, but that is how PromQL works.
• If you have multiple filter conditions, 'and' and '&&" will not work for it.  Instead, use comma (,) to concatenate them so they will work as 'and' logical operator.

2. Legend name

By default, the Legend name is super long so it will be hard for you to find the difference between them. 

You will basically use the similar way as what you did for filter to filter out the fields you do not need.  Instead of key-value pairs in filter field, you put the field name between "{{" and "}}".

For example, if you want to focus on pod name, just put {{pod}} into the 'Legend' field.  That will display only the pod name in Legend.  You can use comma (,) to concatenate multiple fields in Legend.  Here the comma has different meaning from what you see in filters. 

3. Sample PromQL 

For CPU usage per container:

rate(container_cpu_usage_seconds_total{namespace="my_name_space",pod=~"main-.+",container="main-api-container"}[100s]) * 100

For network traffic per pod:

rate(container_network_receive_bytes_total{namespace="my_name_space",pod=~"main.+"}[100s]) * 100

For memory usage per container:

container_memory_working_set_bytes{namespace="my_name_space",pod=~"detector-.+",container="detector-container"}

For network usage:

rate(container_network_receive_bytes_total{namespace="mynamespace",pod=~"main-.+"}[100s]) * 100