ESXi -Generate Self-signed Certificate using Open SSL and Certificate Authority

This post assumes you have an Active directory CA installed and web enrollment working as we will need it to complete the exercise

Generating a certificate key using OpenSSL

Download and install OpenSSL

Go to C:\Program Files\OpenSSL-Win64\bin and open openssl.cfg

Delete everything from this file and replace it as shown below

Here is my sample file, you will need copy this file as such and edit all the highlighted sections with your server info

[ req ]
default_bits = 2048
default_keyfile = rui.key
distinguished_name = req_distinguished_name
encrypt_key = no
prompt = no
string_mask = nombstr
req_extensions = v3_req

[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = DNS:sfo01-m01-esx02, IP:172.16.11.102, DNS:sfo01-m01-esx02.ash.local

[ req_distinguished_name ]
countryName = US
stateOrProvinceName = California
localityName = Palo Alto
0.organizationName = ash.local
organizationalUnitName = computers
commonName = sfo01-m01-esx02.ash.local

To generate a certificate request key for an ESXi server using OpenSSL

Go to C:\Program Files\OpenSSL-Win64\bin and run the command to create the keys

openssl req -new -nodes -out rui.crt -keyout rui-orig.key -config openssl.cfg 

To get the key in an RSA format, input the below command

openssl req -new -nodes -out rui.crt -keyout rui-orig.key -config openssl.cfg 

The output from the above command produces 2 files as shown.

Generating a certificate from Central CA dc.ash.local

Go to chrome browser and browse to your web enrollment page as shown

http://localhost/certsrv/

Click Request certificate

Submit an advanced certificate request.

Open the file C:\Program Files\OpenSSL-Win64\bin\rui.cert using notepad and paste the contents of notepad into saved request tab as shown

Click Download Certificate

Now we will be having two files – first one, the CA generated by dc1.ash.local & second, the key generated by Open SSL

Installing the Certificate on ESX

We will be copying these two files onto the /etc/vmware/ssl path on esx server to replace the certificates

Establish an ssh session to the esx host and navigate to /etc/vmware/ssl directory

The two files we are interested in are rui.crt and rui.key.

[root@sfo01-m01-esx02:~] ls -ltr /etc/vmware/ssl/rui*

-r——– 1 root root 1704 Apr 21 17:23 /etc/vmware/ssl/rui.key
-rw-r–r– 1 root root 1411 Apr 21 17:23 /etc/vmware/ssl/rui.crt
[root@sfo01-m01-esx02:~]

Rename the file certnew in the above picture to rui.crt and rui-orig.key to rui.key and copy these two files into a new folder

I’ve copied the files to a new folder

Overwrite rui.crt and rui.key on the esx from files our desktop to esx server using scp or filezila

During this copy, there is a chance that special characters could be added to the file rui.crt

Open rui.crt using vi editor as shown

Delete everything in file

Open rui.crt from our desktop in notepad and copy all contents to vi editor as shown

Restart hostd and vpxa services by executing the following command:

/etc/init.d/hostd restart && /etc/init.d/vpxa restart 
watchdog-hostd: Terminating watchdog process with PID 1050267
hostd stopped.
hostd started.
watchdog-vpxa: Terminating watchdog process with PID 1050602
vpxa stopped.
vpxa started.
[root@sfo01-m01-esx02:/etc/vmware/ssl]

Restart the host

reboot

At this stage you would expect vCF to pass validation but most likely it wont as you still will get the same error as shown above

Adding to Trust Root Certificate

There is one more step we need to do now to ensure the certificate we generated using OpenSSL is trusted by our Trust Root Certificate and this happens as below.

To fix that, we go back to the rui.crt and clck install certificate as shown

Place the certficate into Trusted Root Cert

Finish the wizard

Click Ok to finish

Now our certificate is valid and trusted.