Protect your microservice by limiting which IPs can access it

If you are using ingress-nginx controller to route your web services, it is accessible from the whole internet.  This is definitely not what you want.  Among with other security protection, you might want to consider to set a white list for who can access your web services. 

The solution is pretty simple.  Just use the 'loadBalancerSourceRanges' to set these IPs for your white list.

Basically, your ingress controller service will be defined as something like this:

apiVersion: v1
kind: Service
metadata:
  name: ingress-nginx
spec:
  type: LoadBalancer
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
  loadBalancerSourceRanges:
  - x.x.x.x/32

To make it happen, add following to your values.yaml file (replace x.x.x.x with your white list IPs separated by comma) and apply such change to your ingress-nginx controller:

controller:
  service:
    loadBalancerSourceRanges: [x.x.x.x/32]

Kubernetes Ingress: which implementation is better?

Three general approaches to exposing your web services running in Kubernetes:

• NodePort: port 30000-32767 only; not good option for production.
• LoadBalancer: simplest and safest; depends on cloud provider’s load balancer; each service needs one load balancer (expensive if you have multiple external services).
• Ingress: an Ingress is an API object that manages external access to the services in a cluster (typically HTTP/S). still need one load balancer from cloud provider.

Given the cost saving from load balancer and the features in Ingress, it makes sense to explore the way for using Ingress.  Except the cost saving, one of the other benefits is to leverage cache ability from Ingress controller.

Nginx is popularly used in Ingress solution.  There are at least two nginx based ingress controller implementations:

1. ingress-nginx: https://kubernetes.github.io/ingress-nginx/; repo in https://github.com/kubernetes/ingress-nginx/tree/master/charts/ingress-nginx
2. NGINX kubernetes-ingress: https://www.nginx.com/products/nginx-ingress-controller/; repo in https://github.com/nginxinc/kubernetes-ingress

The first one is maintained by kubernetes open source community; the second one is also open source, but maintained by nginx.com and offers a commercial version based on "Nginx Plus".

The ingress controller itself is a Kubernetes service.  It is responsible for fulfilling the ingress by generating/updating nginx.conf file based on all ingresses. 

Except above two, you can also find ingress solution from Google cloud load balancer, Istio, Envoy, Contour, etc.

We will mainly compare ingress-nginx and NGINX kubernetes-ingress in this blog. 

Ingress Controller Deployment and Ingress 

First of all, use the helm to install Ingress Controller:

• ingress-nginx: https://github.com/kubernetes/ingress-nginx/tree/master/charts/ingress-nginx
• NGINX Kubernetes-ingress: https://github.com/nginxinc/kubernetes-ingress/tree/master/deployments/helm-chart

Need to prepare a yaml file before you run 'helm install'.  Such yaml file should contains anything you want to set for customized values.

In my case, I put these cache related values in to my yaml file:

controller:
  config:
    http-snippet: |
      proxy_cache_path /tmp/mycachefile levels=1:2 keys_zone=mycache:32m use_temp_path=off max_size=10g inactive=12h;
      proxy_cache_key $scheme$proxy_host$request_uri;

      proxy_cache_lock on;

                      access_log off;  <<-- need this only you want to disable access log

Pass this file to 'helm install' through '--values' argument.  This will put above snippet to http block of nginx.conf.

** Use this to add ingress-nginx repo: 

    helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx

After you install the controller, you will need to create at least one Ingress resource to do the routing. Ingress resource can be created through a yaml file.  A sample Ingress yaml file is:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-for-my-k8s
  namespace: my-namespace
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-buffering: "on"
    nginx.ingress.kubernetes.io/server-snippet: |
      proxy_cache mycache;
      proxy_cache_valid 200 10m;
      proxy_cache_use_stale error timeout updating http_404 http_500 http_502 http_503 http_504;
      proxy_cache_bypass $http_x_purge;
      add_header X-Cache-Status $upstream_cache_status;
spec:
  rules:
  - host: my.hostname.com
    http:
      paths:
      - path: /mypath
        backend:
          serviceName: my-service

          servicePort: 80

In above sample, we are routing the traffic to my-service on port 80 from external load balancer in front of Ingress Controller.  We also add a server snippet in this case. 

Once your ingress resource is applied, ingress controller will take it automatically and update nginx.conf file accordingly.  

Overall, the relationship between Ingress Controller, Ingress and Load Balancer is:

Comparison between ingress-nginx and NGINX kubernetes-ingress

Now, Let us compare these two implementations.

Both have similar functionalities so there is no big mistake if you choose one or another one. But after using both, my recommendation is ingress-nginx.  The reasons are based on following 4 criteria:

• Easy for troubleshooting
• Support friendly
• Contributor's activities
• Lua enabled

I will go through an example to explain why ingress-nginx is better.  My recent work needs to support cache for some of our microservices.  Of course, I will not reinvent the wheel.  I will leverage the cache functionality through nginx so I tried to implement it through both ingress-nginx and NGINX kubernetes-ingress.  Following are what I observed.

Troubleshooting:

ingress-nginx is way better.  For example, if I create a cache zone by putting http-snippet to Ingress controller and I did it incorrectly, the cache zone is not created.  Now when trying to add cache to a host or a path, ingress-nginx will immediately show error:

Error: exit status 1

2021/03/04 01:18:39 [emerg] 186#186: "proxy_cache" zone "mycache" is unknown in /tmp/nginx-cfg216240277:863

nginx: [emerg] "proxy_cache" zone "mycache" is unknown in /tmp/nginx-cfg216240277:863

nginx: configuration file /tmp/nginx-cfg216240277 test failed

If this happens on NGINX kubernetes-ingress, it just silently drops the corresponding server-snippet without any error.  Then you will have to open nginx.conf file and realize the snippet is not there.  But you are still clueless for why it is not there.

Support friendly:

Some feature is really easy to add.  But even so many people voted on it, NGINX kubernetes-ingress's contributors just decided not to do anything.  Here is an example:

https://github.com/nginxinc/kubernetes-ingress/issues/209

The discussion was started in 2017. So many people expressed this is something we should have.  But they just decided to do nothing with all kinds of weird explanations.  I believe the request is reasonable and I actually also need it. ingress-nginx has had it already. 

Contributor's activities:

I was wondering why above issue was constantly refused by NGINX kubernetes-ingress contributors.  So, I did some searches on contributors activities for both projects.  And I found ingress-nginx has much higher activities than NGINX kubernetes-ingress in last 5 years.  Following graphs are from both project's official website (the first graph is scaled from 0-15 when the second graph is scaled from 0-40):

Contributions of NGINX-kubernetes-ingress

Contributions of ingress-nginx

Lua enabled, cache support, etc.:

By inspecting to the nginx.conf file created by ingress controller, I realized only ingress-nginx enabled it. Not sure why NGINX kubernetes-ingress does not enable it or if there is way to enable it. 

Also, the way to put http-snippet to nginx.conf works only for ingress-nginx. I used the same way for NGINX kubernetes-ingress, it never works for me.  

In summary, ingress-nginx works for me much better. This is just my personal opinion based my experience working on both.  

(Updated on December 2, 2021):

I am confirmed that nginx-ingress doesn’t support Lua and openresty. 

One of another difference between these two is:

When ingress-nginx put everything into /etc/nginx/nginx.conf, nginx-ingress creates a separate file for each ingress under /etc/nginx/conf.d/  

It’s hard to say which way is better  it’s just different