Sunday, November 21, 2021

Ingress Controller (continue)

On my March blog, I compared the community version ingress-nginx controller and the Nginx supported nginx-ingress controller.  My conclusion was the community version is better.  

Now my conclusion still stays.  But in reality, you do not always have the choice.  Sometimes, you have to depend on some features on Nginx plus or just your company decides to use nginx-ingress.  If that is the case, you will have to deal with some inconvenience from that. 

Today, I will share two things that are very easy in the community version but is hard in Nginx's nginx-ingress.


Rewrite Annotation


When we get the request from the customer, we need to route it to a certain URI (for example, /mycheck) to do something (such as a security check) regardless what its original URI is. 
 
The “rewrite” annotation worked very well for the community version of ingress-nginx controller using the annotation like this:
 
nginx.ingress.kubernetes.io/rewrite-target: /mycheck
 
The new way in Nginx plus ingress controller is something like this (they changed the annotation name and syntax):
 
nginx.org/rewrites: "serviceName=my-service rewrite=/mycheck"
 
On both cases, the original path is:
 
- path: /
 
But this does not work for us in Nginx plus ingress controller. 

After hours and hours working on this, I contacted the developer in Nginx team.  What I got from him is:

http://nginx.org/rewrites annotation doesn’t support a rewrite that you need, unfortunately.

The advice from him is:

1. To support it, you can use a custom template


2. You can also create a custom annotation 


3. Alternatively, you can use a VirtualServer resource that supports a rewrite that you need:

  - action:
      proxy:
        rewritePath: /mycheck
        upstream: tea
    path: ~^/ # this is a regex path. for a rewrite that you need, it is necessary to use regex 



Multiple Ingresses for the same host


When using the community version ingress controller, we can create multiple ingresses (one for each path) for the same host.   But the Nginx plus ingress controller does not.   I have to put all paths into a single Ingress.  But we do not want to do that for multiple reasons (we may have lots of paths or we need to enable cache for some of the paths, but not all of the paths.  

To work around this in Nginx ingress controller, we need to define additional annotation:



Wednesday, September 8, 2021

thisisunsafe trick

 When security is important, over-secure is stupid.  Currently, I spent lots of time to figure out an error message from Google Chrome when accessing a system which is set up by my coworker and is safe.  It ends up that I have to play this inconvenient 'thisisunsafe' trick documented by this person:

https://miguelpiedrafita.com/chrome-thisisunsafe



 

Friday, August 27, 2021

Redis running in Kubernetes

There is a need to run Redis inside of Kubernetes.  This will eliminate the egress costs.

Installation is straightforward if you use Bitnami:

Build a snapshot of redis-14.8.8 from https://github.com/bitnami/charts/tree/master/bitnami/redis


helm repo add bitnami https://charts.bitnami.com/bitnami

helm pull bitnami/redis


Then checkin the new tgz file (under 'charts' folder) and other files to keep the current version as a snapshot in git.  The minimal files include:


-rw-r--r--  1 user  1437522721   767 Aug 27 11:11 Chart.yaml

-rw-r--r--  1 user  1437522721   235 Aug 27 10:37 README.md

drwxr-xr-x  4 user  1437522721   128 Aug 27 10:37 charts


Install it to Kubernetes:


helm dependency update

helm upgrade -i --create-namespace redis .

To get the Redis default password: 

kubectl get secret --namespace "redis" redis -o jsonpath="{.data.redis-password}" | base64 --decode

Benchmark of the HMGET 

To prepare testing data, we need to create 10 millions entries in hash key.


If we want to use different key/value, we need to use lua to generate 10 million different key/values for ‘counterhash’ hash table:


@redis-master-0:/tmp$ cp /dev/stdin counterhash.lua

for i=1, 10000000 do

    redis.call('hset', 'counterhash',  'counter' .. i, i)

end


@redis-master-0:/tmp$ redis-cli -a cK56wtmkSg --eval counterhash.lua


Check the size of hashkey ‘counterhash’:


@redis-master-0:/$ redis-cli -a UpvoddWilx               

Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.

127.0.0.1:6379> HLEN counterhash

(integer) 10000000



Benchmark for HMGET when two fields are used:


@redis-master-0:/$ redis-benchmark -a UpvoddWilx -r 20000000 -n 20000000 HMGET counterhash "counter9679805" "counter9458449"

====== HMGET counterhash counter9679805 counter9458449 ======                                                   92)

  20000000 requests completed in 297.01 seconds

  50 parallel clients

  75 bytes payload

  keep alive: 1

  host configuration "save":

  host configuration "appendonly": yes

  multi-thread: no


Summary:

  throughput summary: 67338.93 requests per second

  latency summary (msec):

          avg       min       p50       p95       p99       max

        0.392     0.064     0.399     0.543     0.743     5.183


Benchmark for HMGET when ten fields are used:


@redis-master-0:/$ redis-benchmark -a UpvoddWilx -r 20000000 -n 20000000 HMGET counterhash "counter9679800" "counter9679801" "counter9679802" "counter9679803" "counter9679804" "counter9679805" "counter9679806" "counter9679807" "counter9679808" "counter9679809"

====== HMGET counterhash counter9679800 counter9679801 counter9679802 counter9679803 counter9679804 counter9679805 counter9679806 counter9679807 counter9679808 counter9679809 ======

  20000000 requests completed in 273.58 seconds

  50 parallel clients

  244 bytes payload

  keep alive: 1

  host configuration "save":

  host configuration "appendonly": yes

  multi-thread: no


Summary:

  throughput summary: 73103.42 requests per second

  latency summary (msec):

          avg       min       p50       p95       p99       max

        0.360     0.136     0.335     0.503     0.711     8.383

Monday, May 3, 2021

Rebuild Dockerfile from a docker image

alias dfimage="docker run -v /var/run/docker.sock:/var/run/docker.sock --rm alpine/dfimage"

dfimage -sV=1.36 nginx:latest


Thursday, April 15, 2021

PromQL and Grafana

PromQL syntax is kind of non-intuitive and hard to remember.  If you use it everyday, you should be happy.  But if you use it occasionally, you might have to learn it each time when you are using.  In my case, I totally forgot its syntax even I used it two years ago.  

Now, I decided to document some of them so I will not have to learn it again next time.

1. Fiter in metrics

Without filter, you will get too many things you do not need.  For example, you will get graphs for all containers without filter in PromQL. That will be hard for you to find the container or pod you are trying to monitor. By applying filter in "metrics", you can display metrics for the ones you are interested. 

I will go through an example to explain some key syntax for filters:

{namespace="your_name_space",pod=~"main-api.+",container="main-api-container"}

  • If you want to exact match, put your name insider the double quotes.  This is the easiest case.
  • If you want to use wild card match, "*" will not work.  Instead, you need to use regex and put ~ (tilde) before the regex.  This syntax is weird, but that is how PromQL works.
  • If you have multiple filter conditions, 'and' and '&&" will not work for it.  Instead, use comma (,) to concatenate them so they will work as 'and' logical operator.

2. Legend name

By default, the Legend name is super long so it will be hard for you to find the difference between them. 

You will basically use the similar way as what you did for filter to filter out the fields you do not need.  Instead of key-value pairs in filter field, you put the field name between "{{" and "}}".

For example, if you want to focus on pod name, just put {{pod}} into the 'Legend' field.  That will display only the pod name in Legend.  You can use comma (,) to concatenate multiple fields in Legend.  Here the comma has different meaning from what you see in filters. 

3. Sample PromQL 

For CPU usage per container:

rate(container_cpu_usage_seconds_total{namespace="my_name_space",pod=~"main-.+",container="main-api-container"}[100s]) * 100

For network traffic per pod:

rate(container_network_receive_bytes_total{namespace="my_name_space",pod=~"main.+"}[100s]) * 100

For memory usage per container:

container_memory_working_set_bytes{namespace="my_name_space",pod=~"detector-.+",container="detector-container"}

For network usage:

rate(container_network_receive_bytes_total{namespace="mynamespace",pod=~"main-.+"}[100s]) * 100

Wednesday, March 17, 2021

Protect your microservice by limiting which IPs can access it

If you are using ingress-nginx controller to route your web services, it is accessible from the whole internet.  This is definitely not what you want.  Among with other security protection, you might want to consider to set a white list for who can access your web services. 

The solution is pretty simple.  Just use the 'loadBalancerSourceRanges' to set these IPs for your white list.

Basically, your ingress controller service will be defined as something like this:

apiVersion: v1 kind: Service metadata: name: ingress-nginx spec: type: LoadBalancer ports: - protocol: TCP port: 80 targetPort: 80 loadBalancerSourceRanges: - x.x.x.x/32

To make it happen, add following to your values.yaml file (replace x.x.x.x with your white list IPs separated by comma) and apply such change to your ingress-nginx controller:

controller:
service:
loadBalancerSourceRanges: [x.x.x.x/32]

Sunday, March 7, 2021

Kubernetes Ingress: which implementation is better?

Three general approaches to exposing your web services running in Kubernetes:

  • NodePort: port 30000-32767 only; not good option for production.
  • LoadBalancer: simplest and safest; depends on cloud provider’s load balancer; each service needs one load balancer (expensive if you have multiple external services).
  • Ingress: an Ingress is an API object that manages external access to the services in a cluster (typically HTTP/S). still need one load balancer from cloud provider.
Given the cost saving from load balancer and the features in Ingress, it makes sense to explore the way for using Ingress.  Except the cost saving, one of the other benefits is to leverage cache ability from Ingress controller.

Nginx is popularly used in Ingress solution.  There are at least two nginx based ingress controller implementations:

  1. ingress-nginx: https://kubernetes.github.io/ingress-nginx/; repo in https://github.com/kubernetes/ingress-nginx/tree/master/charts/ingress-nginx
  2. NGINX kubernetes-ingress: https://www.nginx.com/products/nginx-ingress-controller/repo in https://github.com/nginxinc/kubernetes-ingress
The first one is maintained by kubernetes open source community; the second one is also open source, but maintained by nginx.com and offers a commercial version based on "Nginx Plus".

The ingress controller itself is a Kubernetes service.  It is responsible for fulfilling the ingress by generating/updating nginx.conf file based on all ingresses. 

Except above two, you can also find ingress solution from Google cloud load balancer, Istio, Envoy, Contour, etc.

We will mainly compare ingress-nginx and NGINX kubernetes-ingress in this blog. 

Ingress Controller Deployment and Ingress 


First of all, use the helm to install Ingress Controller:

  • ingress-nginx: https://github.com/kubernetes/ingress-nginx/tree/master/charts/ingress-nginx
  • NGINX Kubernetes-ingress: https://github.com/nginxinc/kubernetes-ingress/tree/master/deployments/helm-chart
Need to prepare a yaml file before you run 'helm install'.  Such yaml file should contains anything you want to set for customized values.

In my case, I put these cache related values in to my yaml file:

controller:
  config:
    http-snippet: |
      proxy_cache_path /tmp/mycachefile levels=1:2 keys_zone=mycache:32m use_temp_path=off max_size=10g inactive=12h;
      proxy_cache_key $scheme$proxy_host$request_uri;
      proxy_cache_lock on;
                      access_log off;  <<-- need this only you want to disable access log


Pass this file to 'helm install' through '--values' argument.  This will put above snippet to http block of nginx.conf.

** Use this to add ingress-nginx repo: 

    helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx


After you install the controller, you will need to create at least one Ingress resource to do the routing. Ingress resource can be created through a yaml file.  A sample Ingress yaml file is:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-for-my-k8s
  namespace: my-namespace
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-buffering: "on"
    nginx.ingress.kubernetes.io/server-snippet: |
      proxy_cache mycache;
      proxy_cache_valid 200 10m;
      proxy_cache_use_stale error timeout updating http_404 http_500 http_502 http_503 http_504;
      proxy_cache_bypass $http_x_purge;
      add_header X-Cache-Status $upstream_cache_status;
spec:
  rules:
  - host: my.hostname.com
    http:
      paths:
      - path: /mypath
        backend:
          serviceName: my-service

          servicePort: 80



In above sample, we are routing the traffic to my-service on port 80 from external load balancer in front of Ingress Controller.  We also add a server snippet in this case. 

Once your ingress resource is applied, ingress controller will take it automatically and update nginx.conf file accordingly.  

Overall, the relationship between Ingress Controller, Ingress and Load Balancer is:



Comparison between ingress-nginx and NGINX kubernetes-ingress


Now, Let us compare these two implementations.

Both have similar functionalities so there is no big mistake if you choose one or another one. But after using both, my recommendation is ingress-nginx.  The reasons are based on following 4 criteria:
  • Easy for troubleshooting
  • Support friendly
  • Contributor's activities
  • Lua enabled
I will go through an example to explain why ingress-nginx is better.  My recent work needs to support cache for some of our microservices.  Of course, I will not reinvent the wheel.  I will leverage the cache functionality through nginx so I tried to implement it through both ingress-nginx and NGINX kubernetes-ingress.  Following are what I observed.


Troubleshooting:

ingress-nginx is way better.  For example, if I create a cache zone by putting http-snippet to Ingress controller and I did it incorrectly, the cache zone is not created.  Now when trying to add cache to a host or a path, ingress-nginx will immediately show error:

Error: exit status 1

2021/03/04 01:18:39 [emerg] 186#186: "proxy_cache" zone "mycache" is unknown in /tmp/nginx-cfg216240277:863

nginx: [emerg] "proxy_cache" zone "mycache" is unknown in /tmp/nginx-cfg216240277:863

nginx: configuration file /tmp/nginx-cfg216240277 test failed

If this happens on NGINX kubernetes-ingress, it just silently drops the corresponding server-snippet without any error.  Then you will have to open nginx.conf file and realize the snippet is not there.  But you are still clueless for why it is not there.

Support friendly:

Some feature is really easy to add.  But even so many people voted on it, NGINX kubernetes-ingress's contributors just decided not to do anything.  Here is an example:

https://github.com/nginxinc/kubernetes-ingress/issues/209

The discussion was started in 2017. So many people expressed this is something we should have.  But they just decided to do nothing with all kinds of weird explanations.  I believe the request is reasonable and I actually also need it. ingress-nginx has had it already. 

Contributor's activities:

I was wondering why above issue was constantly refused by NGINX kubernetes-ingress contributors.  So, I did some searches on contributors activities for both projects.  And I found ingress-nginx has much higher activities than NGINX kubernetes-ingress in last 5 years.  Following graphs are from both project's official website (the first graph is scaled from 0-15 when the second graph is scaled from 0-40):


Contributions of NGINX-kubernetes-ingress


Contributions of ingress-nginx


Lua enabled, cache support, etc.:

By inspecting to the nginx.conf file created by ingress controller, I realized only ingress-nginx enabled it. Not sure why NGINX kubernetes-ingress does not enable it or if there is way to enable it. 

Also, the way to put http-snippet to nginx.conf works only for ingress-nginx. I used the same way for NGINX kubernetes-ingress, it never works for me.  

In summary, ingress-nginx works for me much better. This is just my personal opinion based my experience working on both.  

(Updated on December 2, 2021):

I am confirmed that nginx-ingress doesn’t support Lua and openresty. 

One of another difference between these two is:

When ingress-nginx put everything into /etc/nginx/nginx.conf, nginx-ingress creates a separate file for each ingress under /etc/nginx/conf.d/  

It’s hard to say which way is better  it’s just different