This article is an Addendum to the Article on Creating secure services in Kubernetes. This is solely for debug/verification of the letsencrypt certificates that we are using in kubernetes .

Certificates Order creation:

The process of issuing certificate is as follows: Create Certificate, Create Order and Create Challenge . Each of these operations should complete successfully . The details are mentioned in the docs in the article’s reference section .

Lets check the order for confirmation — tls-secretd was the secret I used in the Ingress definitions while debugging

$ kubectl describe order tls-secretd-1812893479
Name: tls-secretd-1812893479-2396897813
Namespace: default
Labels: <none>
Annotations: cert-manager.io/certificate-name: tls-secretd
cert-manager.io/private-key-secret-name: tls-secretd
API Version: acme.cert-manager.io/v1alpha3
Kind: Order
Metadata:
Creation Timestamp: 2020-06-11T08:35:02Z
Generation: 1
......
......
......
QzVjMGxIc0lNOXNFTjMyNG0waFovd3YrclJYZDlBYnh0YnQrZ2xSbGdjWm94TQpVd2VuWHc9PQotLS0tLUVORCBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0K
Dns Names:
vetricars.xyz
Issuer Ref:
Group: cert-manager.io
Kind: Issuer
Name: letsencrypt-staging
Status:
Authorizations:
Challenges:
Token: vNOTp53BUzqcCOyb-5_AC00q_Hcn9CdELLCVYNkwQuA
Type: http-01
URL: https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/64054213/TqrTFQ
Identifier: vetricars.xyz
Initial State: valid
......
......
STNKMXRQUlRwMW5FdDlmeUdzcEJPTzA1Z2kxNDhRYXNwKzNOK3N2cUtvbW9RZ2xOb0F4VT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
Finalize URL: https://acme-staging-v02.api.letsencrypt.org/acme/finalize/14106643/100867263
State: valid
URL: https://acme-staging-v02.api.letsencrypt.org/acme/order/14106643/100867263
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Complete 20m cert-manager Order completed successfully

We can see form the above order the certificate was successfully issued after the challenge to the domain vetricars.xyz with challenge type a “http01” as mentioned in the “Issuer” yaml under the solvers . If you don’t get a successful completion of the challenge then dig deeper by “kubectl describe the challenge-id” , it will say the details why the challenge failed .

Ingress for host-based routing :

# ingress2.yaml : This is for the host based routing and the ingress definition is as shown below . We are still using the selfsigned certificates ( check the spec section of Ingress YAML in the tls entry we have “tls-secret” ; the one we generated in the first step above with openssl command) . Here we are doing a host based routing for “hello.info” and “world.info” similar to our effort in LoadBalancer configuration article

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress-nginx
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
spec:
tls:
- hosts:
- tls-secret
rules:
- host: hello.info
http:
paths:
- path: /
backend:
serviceName: web1-service
servicePort: 80
- host: world.info
http:
paths:
- path: /
backend:
serviceName: web2-service
servicePort: 80

Lets apply

$ kubectl apply -f ./ingress2.yaml 
ingress.extensions/ingress-nginx configured
$ kubectl get ing
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress-nginx <none> hello.info,world.info 35.238.236.76 80, 443 4d1h

After making the “/etc/hosts” entry on the node were the service is running lets do a curl with -k option since this is a selfsigned certificate we are asking curl not to do a validation check .

ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
169.254.169.254 metadata.google.internal metadata
35.238.236.76 hello.info
35.238.236.76 world.info
10.128.0.55 kubenode25.c.rosy-cache-200605.internal

Curl output:

$ kubectl get ing
NAME CLASS HOSTS ADDRESS PORTS AGE
ingress-nginx <none> hello.info,world.info 35.238.236.76 80, 443 4d1h
$ curl -k https://hello.info
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
$ curl -k https://world.info
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>

Bingo worked fine!

If we do a “http” request on the above hosts ; then we will get a redirection to “https” connection as shown below .

$ curl -k http://world.info
<html>
<head><title>308 Permanent Redirect</title></head>
<body>
<center><h1>308 Permanent Redirect</h1></center>
<hr><center>nginx/1.17.10</center>
</body>
</html>
$ curl -k http://hello.info
<html>
<head><title>308 Permanent Redirect</title></head>
<body>
<center><h1>308 Permanent Redirect</h1></center>
<hr><center>nginx/1.17.10</center>
</body>
</html>

Since in the ingress definition we have give a ssl-passthrough the decryption happens at the host level instead of the LoadBalancer level. Also since the certificate is enabled on the resource any http traffic will be redirected . If you do not want this to happen then include this in the annotation of the Ingress definitions : nginx.ingress.kubernetes.io/ssl-passthrough: “true”

Secrets Definition:

For a selfsigned certificate lets try debugging the curl outputs as shown below….

$ curl --insecure -v https://vetricars.xyz 2>&1 | awk 'BEGIN { cert=0 } /^\* SSL connection/ { cert=1 } /^\*/ { if (cert) print }'
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
* server certificate verification SKIPPED
* server certificate status verification SKIPPED
* error fetching CN from cert:The requested data were not available.
* common name: (matched)
* server certificate expiration date OK
* server certificate activation date OK
* certificate public key: RSA
* certificate version: #3
* subject:
* start date: Sun, 14 Jun 2020 08:32:46 GMT
* expire date: Sat, 12 Sep 2020 08:32:46 GMT
* issuer:
* compression: NULL
* ALPN, server accepted to use http/1.1
* Connection #0 to host vetricars.xyz left intact
rangapv08@kubenode251:~$ curl https://vetricars.xyz
curl: (60) server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
More details here: http://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.
rangapv08@kubenode251:~$ curl -k https://vetricars.xyz
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>

letsencrypt-staging : For the staging certificates the “curl” output is as shown below. You notice the curl gives the issuer as Fake LE Intermediate , which was empty for the selfsigned as highlighted ….

$ curl --insecure -v https://vetricars.xyz 2>&1 | awk 'BEGIN { cert=0 } /^\* SSL connection/ { cert=1 } /^\*/ { if (cert) print }'
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
* server certificate verification SKIPPED
* server certificate status verification SKIPPED
* common name: vetricars.xyz (matched)
* server certificate expiration date OK
* server certificate activation date OK
* certificate public key: RSA
* certificate version: #3
* subject: CN=vetricars.xyz
* start date: Mon, 15 Jun 2020 03:24:53 GMT
* expire date: Sun, 13 Sep 2020 03:24:53 GMT
* issuer: CN=Fake LE Intermediate X1
* compression: NULL
* ALPN, server accepted to use http/1.1
* Connection #0 to host vetricars.xyz left intact
$ curl https://vetricars.xyz
curl: (60) server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
More details here: http://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.
$ curl -k https://vetricars.xyz/
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>

letsencrypt-prod: For prod certificate the different curl output and the issuer is authenticated and a Valid authority .

$ curl --insecure -v https://vetricars.xyz 2>&1 | awk 'BEGIN { cert=0 } /^\* SSL connection/ { cert=1 } /^\*/ { if (cert) print }'
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
* server certificate verification SKIPPED
* server certificate status verification SKIPPED
* common name: vetricars.xyz (matched)
* server certificate expiration date OK
* server certificate activation date OK
* certificate public key: RSA
* certificate version: #3
* subject: CN=vetricars.xyz
* start date: Mon, 15 Jun 2020 03:37:37 GMT
* expire date: Sun, 13 Sep 2020 03:37:37 GMT
* issuer: C=US,O=Let's Encrypt,CN=Let's Encrypt Authority X3
* compression: NULL
* ALPN, server accepted to use http/1.1
* Connection #0 to host vetricars.xyz left intact
$ curl https://vetricars.xyz
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>

Works on Devops in Startups, reach me @rangapv on twitter or email: rangapv@gmail.com