It show How to auto-renewal “Let’s Encrypt” certificate in ESXi host. When I search, I can find a method in case that a site is received DNS service from AWS Route 53.¹⁾ But, there is no way to renewal Let’s Encrypt certificate in ESXi host itself.  So, I wrote this article.

If you find Korean lecture, please visit ‘ESXi Let’s Encrypt 자동 renewal‘

• Bold means that you have to type and run in ssh(shell).
• Italic means that you have to modify or change according to your case or situation.

Prerequisites

• ACME-tiny
  you can use diafygi’s acme-tiny(https://github.com/diafygi/acme-tiny), but when access https from esxi, it show certificate error([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed). So, I modified this to ignore certificate verify after fork it.²⁾
  ESXi dose not support git. So, download project from https://github.com/dmc31a42/acme-tiny by ‘Clone or download’->Download ZIP’, extract it and copy content into ‘/opt/acme-tiny/’ folder. Location of acme_tiny.py should be ‘/opt/acme-tiny/acme_tiny.py’.
• Firewall exception setting
  Set ‘httpClient’ rule to Enable(?) in ‘Networking’->’Firewall rule’ accessing VMWare Host Client(Web UI). In acme process, it must access some web page. In case of 80 port, It is already excepted by ‘dynamicruleset’, ‘Fault Tolerance’, ‘vCenter Update Manager’ rules, but, in case of 443, it isn’t. So, you should allow 443 port by enabling ‘httpClient’ rule.

Procedure

Make Let’s Encrypt key by manual.

(Procedure below follow https://github.com/diafygi/acme-tiny almost.)

Step 1: Create a Let’s Encrypt account private key (if you haven’t already)

As mentioned in github, if you know about account private key, acme-tiny is not required. But, I also does not understand. So, I’ll follow as github readme. I assume all certificate files are saved in ‘/etc/vmware/ssl’ when I write this article.

[root@esxi:~] cd /etc/vmware/ssl/
[root@esxi:/etc/vmware/ssl] openssl genrsa 4096 > account.key
Generating RSA private key, 4096 bit long modulus
…………………………………………………………………………………..++
……………………………….++
e is 65537 (0x10001)

If you have Let’s Encrypt key, please follow github yourself.

Step 2: Create a certificate signing request (CSR) for your domains.

create private key of HTTPS certificate and certificate signing request file.

[root@esxi:/etc/vmware/ssl] openssl genrsa 4096 > letsencrypt.key
Generating RSA private key, 4096 bit long modulus
………………………………….++
……………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………………..++
e is 65537 (0x10001)

If you create csr following github readme, error will be shown.

[root@esxi:/etc/vmware/ssl] openssl req -new -sha256 -key letsencrypt.key -subj “/CN=yourdomain.com” > letsencrypt.csr
unable to find ‘distinguished_name’ in config problems making Certificate Request 145401271976:error:0E06D06C:configuration file routines:NCONF_get_string:no value:conf_lib.c:324:group=req name=distinguished_name

it is because there is no information in openssl.cnf. you should add some information in ‘/etc/vmware/ssl/openssl.cnf’.³⁾

#
#  OpenSSL example configuration file.
#

# This definition stops the following lines choking if HOME isn’t
# defined.
HOME = .
RANDFILE = $ENV::HOME/.rnd

####################################################################
[ req ]
default_bits            = 4096
default_md              = sha256
default_keyfile         = letsencrypt.key
distinguished_name      = req_distinguished_name
extensions             = v3_ca
req_extensions = v3_ca 

[ v3_ca ]
basicConstraints       = critical, CA:TRUE, pathlen:0
subjectKeyIdentifier   = hash
##authorityKeyIdentifier = keyid:always, issuer:always
keyUsage               = keyCertSign, cRLSign
nsCertType             = sslCA, emailCA, objCA

[req_distinguished_name ]
countryName                     = KR
countryName_default             = KR
countryName_min                 = 2
countryName_max                 = 2 

# write company name
organizationName              = your name or company name
organizationName_default      = Company  

# write department
#organizationalUnitName          = department
#organizationalUnitName_default  = department

# write domain name which you service
commonName                      = yourdomain.com
commonName_default             = lesstif’s Self Signed CA
commonName_max                  = 64 

After modify openssl.cnf, if try to create csr, it should work without error.

Step 3: Make your website host challenge files

For checking domain which issue certificate from Let’s Encrypt to, make server provice challenge files related to the domain. Default location of ESXi web server is ‘/usr/lib/vmware/hostd/docroot’. you should make ‘.well-known/acme-challenge/’ folder under this location.

[root@esxi:/etc/vmware/ssl] mkdir /usr/lib/vmware/hostd/docroot/.well-known/acme-challenge/

You don’t need to location setting in github readme. I’m not sure what try_files is exactly and where can I set. So, I didn’t care about it, but it works well.

Step 4: Get a signed certificate!

Using acme_tiny.py and prepared files above, you can get signed certificate.

[root@esxi:/etc/vmware/ssl] /usr/bin/python /opt/acme-tiny/acme_tiny.py –account-key /etc/vmware/ssl/account.key –csr /etc/vmware/ssl/letsencrypt.csr –acme-dir /usr/lib/vmware/hostd/docroot/.well-known/acme-challenge > /etc/vmware/ssl/letsencrypt.crt
Parsing account key…
Parsing CSR…
Found domains: esxitest.nakwonelec.com
Getting directory…
Directory found!
Registering account…
Registered!
Creating new order…
Order created!
Verifying esxitest.nakwonelec.com…
esxitest.nakwonelec.com verified!
Signing certificate…
Certificate signed!

Step 5: Install the certificate

If you get signed certificate without error, backup old certificate and copy new certificate into old.

[root@esxi:/etc/vmware/ssl] cp -f /etc/vmware/ssl/rui.key /etc/vmware/ssl/orig.rui.key
[root@esxi:/etc/vmware/ssl] cp -f /etc/vmware/ssl/rui.crt /etc/vmware/ssl/orig.rui.crt
[root@esxi:/etc/vmware/ssl] cp -f /etc/vmware/ssl/letsencrypt.key /etc/vmware/ssl/rui.key
[root@esxi:/etc/vmware/ssl] cp -f /etc/vmware/ssl/letsencrypt.crt /etc/vmware/ssl/rui.crt

After copy new certificate into rui.crt(CA signed certificate) and rui.key(private key), apply certificates by restarting management agent.⁴⁾

[root@esxi:/etc/vmware/ssl] /etc/init.d/hostd restart
watchdog-hostd: Terminating watchdog process with PID 90365
hostd stopped.
hostd started.
[root@esxi:/etc/vmware/ssl] /etc/init.d/vpxa restart
watchdog-vpxa: Terminating watchdog process with PID 90405
vpxa stopped.
[root@esxi:/etc/vmware/ssl] /etc/init.d/vpxa start
vpxa is running

Make above process automatically by cron

you can make renewal CA signed certificate automatically by create script and add it to cron. Below script do renewal certificate and apply it including detect to fail to get certificate and backup previous certificate if it run first.⁵⁾ Create this script into location where you want. In my case, I saved it at ‘/etc/vmware/ssl/letsencrypt_renewal.sh’.

#!/usr/bin/sh
/usr/bin/python /opt/acme-tiny/acme_tiny.py –account-key /etc/vmware/ssl/account.key –csr /etc/vmware/ssl/letsencrypt.csr –acme-dir /usr/lib/vmware/hostd/docroot/.well-known/acme-challenge > /etc/vmware/ssl/letsencrypt.crt
if [ -s “/etc/vmware/ssl/orig.rui.key” ]; then
else
cp -f /etc/vmware/ssl/rui.key /etc/vmware/ssl/orig.rui.key
cp -f /etc/vmware/ssl/rui.crt /etc/vmware/ssl/orig.rui.crt
fi
if [ -s “/etc/vmware/ssl/letsencrypt.crt” ]; then
cp -f /etc/vmware/ssl/letsencrypt.key /etc/vmware/ssl/rui.key
cp -f /etc/vmware/ssl/letsencrypt.crt /etc/vmware/ssl/rui.crt
/etc/init.d/hostd restart
/etc/init.d/vpxa restart
/etc/init.d/vpxa start
else
echo “letsencrypt.crt file is not correct”
fi

And you should to add this in cron to run periodically. But, as there isn’t crontab in ESXi, you have to edit cron file yourself. below command means to run /etc/vmware/ssl/letsencrypt_renewal.sh at 1st day, 0 hour, 0 min, each month(UTC reference) and save output(texts shown in shell) into ‘/etc/vmware/ssl/letsencrypt.log’

#min  hour  day  mon  dow  command
0 0 1 * * /etc/vmware/ssl/letsencrypt_renewal.sh >> /etc/vmware/ssl/letsencrypt.log 2>&1

Write as below command for apply to cron.

[root@esxi:/etc/vmware/ssl] echo ‘0 0 1 * *
/etc/vmware/ssl/letsencrypt_renewal.sh >> /etc/vmware/ssl/letsencrypt.log 2>&1′ >> /var/spool/cron/crontabs/root

If you cannot modify ‘/var/spool/cron/crontabs/root’ due to write protect whether it has write permission or not, copy contents of ‘/var/spool/cron/crontabs/root’ to other(other file, windows notepad etc.), remove or rename root file, create new ‘root’ file, and restore previous contents. Then, you can modify cron.⁶⁾

And, restart cron service.

[root@esxi:/etc/vmware/ssl] /bin/kill $(cat /var/run/crond.pid)
[root@esxi:/etc/vmware/ssl] /usr/lib/vmware/busybox/bin/busybox crond

In this state, cron works. But, ESXi modify cron when booting. To add scheduler automatically when booting, you have to add this step into ‘/etc/rc.local.d/local.sh’.⁷⁾

Open ‘/etc/rc.local.d/local.sh’ by editor which you usally use and modify like below. Write what you type above(little bit modifiy) before ‘exit 0’ line. Modify and save.

#!/bin/sh

# local configuration options

# Note: modify at your own risk! If you do/use anything in this
# script that is not part of a stable API (relying on files to be in
# specific places, specific tools, specific output, etc) there is a
# possibility you will end up with a broken system after patching or
# upgrading. Changes are not supported unless under direction of
# VMware support.

# Note: This script will not be run when UEFI secure boot is enabled.

/bin/kill $(cat /var/run/crond.pid) # Gets the cron service pid and simply kills it.
/bin/echo ‘0 0 1 * * /etc/vmware/ssl/letsencrypt_renewal.sh >> /etc/vmware/ssl/letsencrypt.log 2>&1’ >> /var/spool/cron/crontabs/root
/usr/lib/vmware/busybox/bin/busybox crond
exit 0

Now, Let’s Encrypt certificate will be automatically renewal-ed each month.

 

Footnote

1. https://wiki.9r.com.au/display/9R/LetsEncrypt+Certificates+for+vCenter+and+PSC
2. http://multifrontgarden.tistory.com/219
3. https://www.lesstif.com/pages/viewpage.action?pageId=6979614
4. https://kb.vmware.com/s/article/1003490
5. http://foris.tistory.com/256 shell
6. https://communities.vmware.com/thread/332790
7. https://www.gotelsolutions.com/billing/ext.php?m=faq&h=public&i=1&hash=4a6e4ae063deaf91be0a772897fedcd9&&faq_product_id=

 

• ESXi
• Let's Encrypt