Open in app

Sign in

Write

Sign in

Addendum1:

Rangaswamy P V
7 min readJun 15, 2020

This article is an Addendum to the Article on Creating secure services in Kubernetes. This is solely for debug/verification of the letsencrypt certificates that we are using in kubernetes .

Certificates Order creation:

The process of issuing certificate is as follows: Create Certificate, Create Order and Create Challenge . Each of these operations should complete successfully . The details are mentioned in the docs in the article’s reference section .

Lets check the order for confirmation — tls-secretd was the secret I used in the Ingress definitions while debugging

$ kubectl describe order tls-secretd-1812893479
Name:         tls-secretd-1812893479-2396897813
Namespace:    default
Labels:       <none>
Annotations:  cert-manager.io/certificate-name: tls-secretd
              cert-manager.io/private-key-secret-name: tls-secretd
API Version:  acme.cert-manager.io/v1alpha3
Kind:         Order
Metadata:
  Creation Timestamp:  2020-06-11T08:35:02Z
  Generation:          1
......
......
......QzVjMGxIc0lNOXNFTjMyNG0waFovd3YrclJYZDlBYnh0YnQrZ2xSbGdjWm94TQpVd2VuWHc9PQotLS0tLUVORCBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0K
  Dns Names:
    vetricars.xyz
  Issuer Ref:
    Group:  cert-manager.io
    Kind:   Issuer
    Name:   letsencrypt-staging
Status:
  Authorizations:
    Challenges:
      Token:        vNOTp53BUzqcCOyb-5_AC00q_Hcn9CdELLCVYNkwQuA
      Type:         http-01
      URL:          https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/64054213/TqrTFQ
    Identifier:     vetricars.xyz
    Initial State:  valid
......
......
STNKMXRQUlRwMW5FdDlmeUdzcEJPTzA1Z2kxNDhRYXNwKzNOK3N2cUtvbW9RZ2xOb0F4VT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
  Finalize URL:     https://acme-staging-v02.api.letsencrypt.org/acme/finalize/14106643/100867263
  State:            valid
  URL:              https://acme-staging-v02.api.letsencrypt.org/acme/order/14106643/100867263
Events:
  Type    Reason    Age   From          Message
  ----    ------    ----  ----          -------
  Normal  Complete  20m   cert-manager  Order completed successfully

We can see form the above order the certificate was successfully issued after the challenge to the domain vetricars.xyz with challenge type a “http01” as mentioned in the “Issuer” yaml under the solvers . If you don’t get a successful completion of the challenge then dig deeper by “kubectl describe the challenge-id” , it will say the details why the challenge failed .

Ingress for host-based routing :

# ingress2.yaml : This is for the host based routing and the ingress definition is as shown below . We are still using the selfsigned certificates ( check the spec section of Ingress YAML in the tls entry we have “tls-secret” ; the one we generated in the first step above with openssl command) . Here we are doing a host based routing for “hello.info” and “world.info” similar to our effort in LoadBalancer configuration article

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-nginx
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
spec:
  tls:
  - hosts:
      - tls-secret
  rules:
  - host: hello.info
    http:
      paths:
      - path: /
        backend:
          serviceName: web1-service
          servicePort: 80
  - host: world.info
    http:
      paths:
      - path: /
        backend:
          serviceName: web2-service
          servicePort: 80

Lets apply

$ kubectl apply -f ./ingress2.yaml 
ingress.extensions/ingress-nginx configured$ kubectl get ing
NAME            CLASS    HOSTS                   ADDRESS         PORTS     AGE
ingress-nginx   <none>   hello.info,world.info   35.238.236.76   80, 443   4d1h

After making the “/etc/hosts” entry on the node were the service is running lets do a curl with -k option since this is a selfsigned certificate we are asking curl not to do a validation check .

ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
169.254.169.254 metadata.google.internal metadata
35.238.236.76 hello.info
35.238.236.76 world.info
10.128.0.55 kubenode25.c.rosy-cache-200605.internal

Curl output:

$ kubectl get ing
NAME            CLASS    HOSTS                   ADDRESS         PORTS     AGE
ingress-nginx   <none>   hello.info,world.info   35.238.236.76   80, 443   4d1h$ curl -k https://hello.info
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p><p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p><p><em>Thank you for using nginx.</em></p>
</body>
</html>$ curl -k https://world.info
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p><p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p><p><em>Thank you for using nginx.</em></p>
</body>
</html>

Bingo worked fine!

If we do a “http” request on the above hosts ; then we will get a redirection to “https” connection as shown below .

$ curl -k http://world.info
<html>
<head><title>308 Permanent Redirect</title></head>
<body>
<center><h1>308 Permanent Redirect</h1></center>
<hr><center>nginx/1.17.10</center>
</body>
</html>$ curl -k http://hello.info
<html>
<head><title>308 Permanent Redirect</title></head>
<body>
<center><h1>308 Permanent Redirect</h1></center>
<hr><center>nginx/1.17.10</center>
</body>
</html>

Since in the ingress definition we have give a ssl-passthrough the decryption happens at the host level instead of the LoadBalancer level. Also since the certificate is enabled on the resource any http traffic will be redirected . If you do not want this to happen then include this in the annotation of the Ingress definitions : nginx.ingress.kubernetes.io/ssl-passthrough: “true”

Secrets Definition:

For a selfsigned certificate lets try debugging the curl outputs as shown below….

$ curl --insecure -v https://vetricars.xyz 2>&1 | awk 'BEGIN { cert=0 } /^\* SSL connection/ { cert=1 } /^\*/ { if (cert) print }'
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
*        server certificate verification SKIPPED
*        server certificate status verification SKIPPED
* error fetching CN from cert:The requested data were not available.
*        common name:  (matched)
*        server certificate expiration date OK
*        server certificate activation date OK
*        certificate public key: RSA
*        certificate version: #3
*        subject: 
*        start date: Sun, 14 Jun 2020 08:32:46 GMT
*        expire date: Sat, 12 Sep 2020 08:32:46 GMT
*        issuer: 
*        compression: NULL
* ALPN, server accepted to use http/1.1
* Connection #0 to host vetricars.xyz left intact
rangapv08@kubenode251:~$ curl https://vetricars.xyz
curl: (60) server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
More details here: http://curl.haxx.se/docs/sslcerts.htmlcurl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.
rangapv08@kubenode251:~$ curl -k https://vetricars.xyz
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p><p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p><p><em>Thank you for using nginx.</em></p>
</body>
</html>

letsencrypt-staging : For the staging certificates the “curl” output is as shown below. You notice the curl gives the issuer as Fake LE Intermediate , which was empty for the selfsigned as highlighted ….

$ curl --insecure -v https://vetricars.xyz 2>&1 | awk 'BEGIN { cert=0 } /^\* SSL connection/ { cert=1 } /^\*/ { if (cert) print }'
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
*        server certificate verification SKIPPED
*        server certificate status verification SKIPPED
*        common name: vetricars.xyz (matched)
*        server certificate expiration date OK
*        server certificate activation date OK
*        certificate public key: RSA
*        certificate version: #3
*        subject: CN=vetricars.xyz
*        start date: Mon, 15 Jun 2020 03:24:53 GMT
*        expire date: Sun, 13 Sep 2020 03:24:53 GMT
*        issuer: CN=Fake LE Intermediate X1
*        compression: NULL
* ALPN, server accepted to use http/1.1
* Connection #0 to host vetricars.xyz left intact$ curl https://vetricars.xyz
curl: (60) server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
More details here: http://curl.haxx.se/docs/sslcerts.htmlcurl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.$ curl -k https://vetricars.xyz/
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p><p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p><p><em>Thank you for using nginx.</em></p>
</body>
</html>

letsencrypt-prod: For prod certificate the different curl output and the issuer is authenticated and a Valid authority .

$ curl --insecure -v https://vetricars.xyz 2>&1 | awk 'BEGIN { cert=0 } /^\* SSL connection/ { cert=1 } /^\*/ { if (cert) print }'
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
*        server certificate verification SKIPPED
*        server certificate status verification SKIPPED
*        common name: vetricars.xyz (matched)
*        server certificate expiration date OK
*        server certificate activation date OK
*        certificate public key: RSA
*        certificate version: #3
*        subject: CN=vetricars.xyz
*        start date: Mon, 15 Jun 2020 03:37:37 GMT
*        expire date: Sun, 13 Sep 2020 03:37:37 GMT
*        issuer: C=US,O=Let's Encrypt,CN=Let's Encrypt Authority X3
*        compression: NULL
* ALPN, server accepted to use http/1.1
* Connection #0 to host vetricars.xyz left intact$ curl https://vetricars.xyz
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p><p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p><p><em>Thank you for using nginx.</em></p>
</body>
</html>
Tls Certificate
Httpscertificate
Lets Encrypt
Debug Tls

--

--

Rangaswamy P V

Written by Rangaswamy P V

47 Followers

Works on Devops in Startups, reach me @rangapv on twitter or email: rangapv@gmail.com

Help

Status

About

Careers

Blog

Privacy

Terms

Text to speech

Teams