Three general approaches to exposing your web services running in Kubernetes:
- NodePort: port 30000-32767 only; not good option for production.
- LoadBalancer: simplest and safest; depends on cloud provider’s load balancer; each service needs one load balancer (expensive if you have multiple external services).
- Ingress: an Ingress is an API object that manages external access to the services in a cluster (typically HTTP/S). still need one load balancer from cloud provider.
Given the cost saving from load balancer and the features in Ingress, it makes sense to explore the way for using Ingress. Except the cost saving, one of the other benefits is to leverage cache ability from Ingress controller.
Nginx is popularly used in Ingress solution. There are at least two nginx based ingress controller implementations:
- ingress-nginx: https://kubernetes.github.io/ingress-nginx/; repo in https://github.com/kubernetes/ingress-nginx/tree/master/charts/ingress-nginx
- NGINX kubernetes-ingress: https://www.nginx.com/products/nginx-ingress-controller/; repo in https://github.com/nginxinc/kubernetes-ingress
The first one is maintained by kubernetes open source community; the second one is also open source, but maintained by nginx.com and offers a commercial version based on "Nginx Plus".
The ingress controller itself is a Kubernetes service. It is responsible for fulfilling the ingress by generating/updating nginx.conf file based on all ingresses.
Except above two, you can also find ingress solution from Google cloud load balancer, Istio, Envoy, Contour, etc.
We will mainly compare ingress-nginx and NGINX kubernetes-ingress in this blog.
Ingress Controller Deployment and Ingress
First of all, use the helm to install Ingress Controller:
- ingress-nginx: https://github.com/kubernetes/ingress-nginx/tree/master/charts/ingress-nginx
- NGINX Kubernetes-ingress: https://github.com/nginxinc/kubernetes-ingress/tree/master/deployments/helm-chart
Need to prepare a yaml file before you run 'helm install'. Such yaml file should contains anything you want to set for customized values.
In my case, I put these cache related values in to my yaml file:
controller:
config:
http-snippet: |
proxy_cache_path /tmp/mycachefile levels=1:2 keys_zone=mycache:32m use_temp_path=off max_size=10g inactive=12h;
proxy_cache_key $scheme$proxy_host$request_uri;
access_log off; <<-- need this only you want to disable access log
Pass this file to 'helm install' through '--values' argument. This will put above snippet to http block of nginx.conf.
** Use this to add ingress-nginx repo:
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
After you install the controller, you will need to create at least one Ingress resource to do the routing. Ingress resource can be created through a yaml file. A sample Ingress yaml file is:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress-for-my-k8s
namespace: my-namespace
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/proxy-buffering: "on"
nginx.ingress.kubernetes.io/server-snippet: |
proxy_cache mycache;
proxy_cache_valid 200 10m;
proxy_cache_use_stale error timeout updating http_404 http_500 http_502 http_503 http_504;
proxy_cache_bypass $http_x_purge;
add_header X-Cache-Status $upstream_cache_status;
spec:
rules:
- host: my.hostname.com
http:
paths:
- path: /mypath
backend:
serviceName: my-service
In above sample, we are routing the traffic to my-service on port 80 from external load balancer in front of Ingress Controller. We also add a server snippet in this case.
Once your ingress resource is applied, ingress controller will take it automatically and update nginx.conf file accordingly.
Overall, the relationship between Ingress Controller, Ingress and Load Balancer is:
Comparison between ingress-nginx and NGINX kubernetes-ingress
Now, Let us compare these two implementations.
Both have similar functionalities so there is no big mistake if you choose one or another one. But after using both, my recommendation is ingress-nginx. The reasons are based on following 4 criteria:
- Easy for troubleshooting
- Support friendly
- Contributor's activities
- Lua enabled
I will go through an example to explain why ingress-nginx is better. My recent work needs to support cache for some of our microservices. Of course, I will not reinvent the wheel. I will leverage the cache functionality through nginx so I tried to implement it through both ingress-nginx and NGINX kubernetes-ingress. Following are what I observed.
Troubleshooting:
ingress-nginx is way better. For example, if I create a cache zone by putting http-snippet to Ingress controller and I did it incorrectly, the cache zone is not created. Now when trying to add cache to a host or a path, ingress-nginx will immediately show error:
Error: exit status 1
2021/03/04 01:18:39 [emerg] 186#186: "proxy_cache" zone "mycache" is unknown in /tmp/nginx-cfg216240277:863
nginx: [emerg] "proxy_cache" zone "mycache" is unknown in /tmp/nginx-cfg216240277:863
nginx: configuration file /tmp/nginx-cfg216240277 test failed
If this happens on NGINX kubernetes-ingress, it just silently drops the corresponding server-snippet without any error. Then you will have to open nginx.conf file and realize the snippet is not there. But you are still clueless for why it is not there.
Support friendly:
Some feature is really easy to add. But even so many people voted on it, NGINX kubernetes-ingress's contributors just decided not to do anything. Here is an example:
https://github.com/nginxinc/kubernetes-ingress/issues/209
The discussion was started in 2017. So many people expressed this is something we should have. But they just decided to do nothing with all kinds of weird explanations. I believe the request is reasonable and I actually also need it. ingress-nginx has had it already.
Contributor's activities:
I was wondering why above issue was constantly refused by NGINX kubernetes-ingress contributors. So, I did some searches on contributors activities for both projects. And I found ingress-nginx has much higher activities than NGINX kubernetes-ingress in last 5 years. Following graphs are from both project's official website (the first graph is scaled from 0-15 when the second graph is scaled from 0-40):
 |
Contributions of NGINX-kubernetes-ingress
|
 |
| Contributions of ingress-nginx |
Lua enabled, cache support, etc.:
By inspecting to the nginx.conf file created by ingress controller, I realized only ingress-nginx enabled it. Not sure why NGINX kubernetes-ingress does not enable it or if there is way to enable it.
Also, the way to put http-snippet to nginx.conf works only for ingress-nginx. I used the same way for NGINX kubernetes-ingress, it never works for me.
In summary, ingress-nginx works for me much better. This is just my personal opinion based my experience working on both.
(Updated on December 2, 2021):
I am confirmed that nginx-ingress doesn’t support Lua and openresty.
One of another difference between these two is:
When ingress-nginx put everything into /etc/nginx/nginx.conf, nginx-ingress creates a separate file for each ingress under /etc/nginx/conf.d/
It’s hard to say which way is better it’s just different
