proxy_pass vs. rewrite in nginx ingress controller

 

	proxy_pass	rewrite
Return	200 with the content	302 with the URL to the redirected page
Content	To nginx, then the client	Second call skips nginx; returned through the redirect URL directly to client

Example of proxy_pass:

location /testing {

  proxy_pass https://news.yahoo.com/us;

}

Example of rewrite:

rewrite ^/(.*)$ https://www.newurl.com/$1-$http_x_header1 redirect

How to hook up them to nginx conf?

Add them as annotations to the Ingress. One example is:

apiVersion: networking.k8s.io/v1

kind: Ingress

metadata:

  name: my-ingress-name

  annotations:

    nginx.org/mergeable-ingress-type: "master"

    nginx.org/server-snippets: |

      rewrite ....

      location /testing {

        ....

      }

spec:

  ....

Annotation of nginx.org/rewriters:

nginx.org/rewriters: "serviceName=my-service-name rewrite=/my-new-endpoint" 

Ingress to support multiple hosts

The Nginx ingress controller allows you to specify ingress in the helm chart.  For each ingress, you define the ingress rules in the corresponding yaml file.  You can specify each host and its rules. 

But to make the ingress dynamically apply the same rules for multiple hosts, you will need to specify the list of hosts from values.yaml file.  In this case, you are not allowed to pass a list of hosts as comma-separated hosts or other delimiter-separated hosts.  Even all hosts follow a pattern, the wild card is not supported.

To allow you to pass a list of hosts in values.yaml file, there is a way to do this.

In your values.yaml file, you pass the hosts like this: 

ingress:
  dnshosts:
    - host-1
    - host-2

 

  In your ingress yaml file, you use 'range' to iterate each host:

{{- range .Values.ingress.dnshosts }}

---

apiVersion: networking.k8s.io/v1

kind: Ingress

metadata:

  name: my-ingress-service1-{{ . | quote | sha1sum | trunc 5}}

  namespace: {{ template "my.namespace" $ }}

  annotations:

    nginx.org/mergeable-ingress-type: "minion"

spec:

  ingressClassName: "nginx"

  rules:

  - host: {{ . | quote }}

    http:

      paths: 

      - path: /service1-endpoint

        pathType: Prefix

        backend:

          service:

            name: service1

            port:

              number: 80

{{- end }}

Reference:

https://devopslearners.com/how-to-use-with-and-range-flow-control-in-helm-731591d357c6

Access GCR images from different GCP projects

The manual way:

https://medium.com/google-cloud/using-single-docker-repository-with-multiple-gke-projects-1672689f780c

For each GCP project that will need to access the GCR images, you give storage.objectViewer permission to its service account in the GCP project where the GCR images will be accessed. 

The way through Terraform code:

resource "google_project_iam_member" "my-project-storage-access" {

  project = "gcp-project-of-gcr-images"

  role = "roles/storage.objectViewer"

  member = "serviceAccount:${google_service_account.my-cluster.email}"

}  

 For each GCP project that will need to access the GCR images, you use the above terraform code to give the service account the storage.objectViewer permission to that GCP project where the GCR images will be accessed. 

How to import IAM role to terraform state?

This sounds like a silly question.  Why IAM role is different?  We all know how to import resources to terraform state, right?

Actually, there are two tricks to this:

1. You have to know the resource id for each IAM role.

2. You have to use double quotes for it.

Let me give you more details on these.

Resource ID format for IAM role contains three components: project name, role, and member.

But the 'terraform import' has only two parameters.  How can you pass 3 components?

Actually, the first parameter is always the resource name so you have only one parameter to use to pass the above three components.  

That is the reason why double quotes are needed. Put all 3 components between double quotes and separate them by space.

Here is an example:

terraform import google_project_iam_memeber.my-cluster-autoscaling-metrics-writer "my-gcp-project roles/autoscaling.metricsWriter serviceAccount:my-cluster@my-gcp-project.iam.gserviceaccount.com"

Ingress Controller (continue)

On my March blog, I compared the community version ingress-nginx controller and the Nginx supported nginx-ingress controller.  My conclusion was the community version is better.  

Now my conclusion still stays.  But in reality, you do not always have the choice.  Sometimes, you have to depend on some features on Nginx plus or just your company decides to use nginx-ingress.  If that is the case, you will have to deal with some inconvenience from that. 

Today, I will share two things that are very easy in the community version but is hard in Nginx's nginx-ingress.

Rewrite Annotation

When we get the request from the customer, we need to route it to a certain URI (for example, /mycheck) to do something (such as a security check) regardless what its original URI is. 

 

The “rewrite” annotation worked very well for the community version of ingress-nginx controller using the annotation like this:

 

nginx.ingress.kubernetes.io/rewrite-target: /mycheck

 

The new way in Nginx plus ingress controller is something like this (they changed the annotation name and syntax):

 

nginx.org/rewrites: "serviceName=my-service rewrite=/mycheck"

 

On both cases, the original path is:

 

- path: /

 

But this does not work for us in Nginx plus ingress controller. 

After hours and hours working on this, I contacted the developer in Nginx team.  What I got from him is:

http://nginx.org/rewrites annotation doesn’t support a rewrite that you need, unfortunately.

The advice from him is:

1. To support it, you can use a custom template. 

Ingress template: https://github.com/nginxinc/kubernetes-ingress/blob/v1.8.0/internal/configs/version1/nginx-plus.ingress.tmpl

Main template: https://github.com/nginxinc/kubernetes-ingress/blob/v1.8.0/internal/configs/version1/nginx-plus.tmpl

2. You can also create a custom annotation 

https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/custom-annotations/

3. Alternatively, you can use a VirtualServer resource that supports a rewrite that you need:

  - action:

      proxy:

        rewritePath: /mycheck

        upstream: tea

    path: ~^/ # this is a regex path. for a rewrite that you need, it is necessary to use regex 

See an example here — https://github.com/nginxinc/kubernetes-ingress/tree/master/examples/rewrites

Multiple Ingresses for the same host

When using the community version ingress controller, we can create multiple ingresses (one for each path) for the same host.   But the Nginx plus ingress controller does not.   I have to put all paths into a single Ingress.  But we do not want to do that for multiple reasons (we may have lots of paths or we need to enable cache for some of the paths, but not all of the paths.  

To work around this in Nginx ingress controller, we need to define additional annotation:

https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/cross-namespace-configuration/

thisisunsafe trick

 When security is important, over-secure is stupid.  Currently, I spent lots of time to figure out an error message from Google Chrome when accessing a system which is set up by my coworker and is safe.  It ends up that I have to play this inconvenient 'thisisunsafe' trick documented by this person:

https://miguelpiedrafita.com/chrome-thisisunsafe

 

Redis running in Kubernetes

There is a need to run Redis inside of Kubernetes.  This will eliminate the egress costs.

Installation is straightforward if you use Bitnami:

Build a snapshot of redis-14.8.8 from https://github.com/bitnami/charts/tree/master/bitnami/redis

helm repo add bitnami https://charts.bitnami.com/bitnami

helm pull bitnami/redis

Then checkin the new tgz file (under 'charts' folder) and other files to keep the current version as a snapshot in git.  The minimal files include:

-rw-r--r--  1 user  1437522721   767 Aug 27 11:11 Chart.yaml

-rw-r--r--  1 user  1437522721   235 Aug 27 10:37 README.md

drwxr-xr-x  4 user  1437522721   128 Aug 27 10:37 charts

Install it to Kubernetes:

helm dependency update

helm upgrade -i --create-namespace redis .

To get the Redis default password: 

kubectl get secret --namespace "redis" redis -o jsonpath="{.data.redis-password}" | base64 --decode

Benchmark of the HMGET 

To prepare testing data, we need to create 10 millions entries in hash key.

If we want to use different key/value, we need to use lua to generate 10 million different key/values for ‘counterhash’ hash table:

@redis-master-0:/tmp$ cp /dev/stdin counterhash.lua

for i=1, 10000000 do

    redis.call('hset', 'counterhash',  'counter' .. i, i)

end

@redis-master-0:/tmp$ redis-cli -a cK56wtmkSg --eval counterhash.lua

Check the size of hashkey ‘counterhash’:

@redis-master-0:/$ redis-cli -a UpvoddWilx               

Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.

127.0.0.1:6379> HLEN counterhash

(integer) 10000000

Benchmark for HMGET when two fields are used:

@redis-master-0:/$ redis-benchmark -a UpvoddWilx -r 20000000 -n 20000000 HMGET counterhash "counter9679805" "counter9458449"

====== HMGET counterhash counter9679805 counter9458449 ======                                                   92)

  20000000 requests completed in 297.01 seconds

  50 parallel clients

  75 bytes payload

  keep alive: 1

  host configuration "save":

  host configuration "appendonly": yes

  multi-thread: no

Summary:

  throughput summary: 67338.93 requests per second

  latency summary (msec):

          avg       min       p50       p95       p99       max

        0.392     0.064     0.399     0.543     0.743     5.183

Benchmark for HMGET when ten fields are used:

@redis-master-0:/$ redis-benchmark -a UpvoddWilx -r 20000000 -n 20000000 HMGET counterhash "counter9679800" "counter9679801" "counter9679802" "counter9679803" "counter9679804" "counter9679805" "counter9679806" "counter9679807" "counter9679808" "counter9679809"

====== HMGET counterhash counter9679800 counter9679801 counter9679802 counter9679803 counter9679804 counter9679805 counter9679806 counter9679807 counter9679808 counter9679809 ======

  20000000 requests completed in 273.58 seconds

  50 parallel clients

  244 bytes payload

  keep alive: 1

  host configuration "save":

  host configuration "appendonly": yes

  multi-thread: no

Summary:

  throughput summary: 73103.42 requests per second

  latency summary (msec):

          avg       min       p50       p95       p99       max

        0.360     0.136     0.335     0.503     0.711     8.383