How to Replace vCenter Machine SSL Certificate with a Custom Certificate

When managing a vCenter Server, it is important to ensure your certificates are properly configured for secure communication. This blog will walk you through the process of replacing the default machine SSL certificate with your own custom certificate in VMware vCenter.

Prerequisites:

1. Custom Certificate: You should already have your custom SSL certificate, private key, and root CA certificate ready.
2. Backup: Always take an offline snapshot of your vCenter Server Appliance (vCSA) before proceeding, especially if the vCenter is part of an Enhanced Linked Mode (ELM) configuration. This ensures you can revert back if anything goes wrong.

Steps to Replace the vCenter Machine SSL Certificate:

1. Take an Offline Snapshot of vCenter

If you’re working in a vCenter Cluster (ELM), make sure to take an offline snapshot of all vCenter Servers in the cluster before starting the process. This ensures you can roll back to a stable state if needed.

2. Log in to vCenter Appliance (vCSA)

Connect to your vCenter Server Appliance via SSH. If SSH is not enabled, enable it from the vCenter Appliance Management Interface (VAMI).

3. Launch the VMware Certificate Manager

Once logged in, navigate to the VMware Certificate Manager utility to replace the SSL certificate.

For vCenter Server 6.x/7.x/8.x Appliance, run the following command:

/usr/lib/vmware-vmca/bin/certificate-manager

4. Select Option 1: Replace Machine SSL Certificate with Custom Certificate

After launching the Certificate Manager, select Option 1 to replace the machine SSL certificate with your custom certificate.

1) Replace Machine SSL certificate with custom certificate

When prompted, enter the administrator@vsphere.local password to authenticate.

5. Select Option 2: Import Custom Certificate

Next, choose Option 2 to import your custom certificate since you already have the custom certificate prepared.

6. Enter Directories for Certificate Files

You will now be asked to provide the location of your certificate files. Make sure you have the following ready:

• Machine SSL Certificate (machine_name_ssl.cer)
• Private Key (machine_name_ssl.key)
• Root CA Certificate (Root64.cer)

Enter the paths of these files when prompted:

• Machine SSL Certificate (e.g., /tmp/ssl/machine_name_ssl.cer)
• Private Key (e.g., /tmp/ssl/machine_name_ssl.key)
• Signing Certificate (Root CA Certificate) (e.g., /tmp/ssl/Root64.cer)

For example:

Provide a valid custom certificate for Machine SSL:
/tmp/ssl/machine_name_ssl.cer

Provide a valid custom key for Machine SSL:
/tmp/ssl/machine_name_ssl.key

Provide the signing certificate of the Machine SSL certificate:
/tmp/ssl/Root64.cer

7. Confirm the Changes

After entering the file paths, the utility will prompt you to confirm the replacement of the existing certificates. Answer Yes (Y) to proceed.

8. Restart vCenter Services

The certificate replacement process will replace the old SSL certificate with your custom one. Once completed, vCenter services will automatically restart to apply the new certificate.

9. Verify the New SSL Certificate

After the vCenter services restart, verify that the new certificate is applied by logging into the vSphere Client (Web UI) and checking the SSL certificate in your browser. You should now see your custom certificate instead of the default self-signed VMware certificate.

---

Important Notes:

• If you have multiple vCenter servers in an Enhanced Linked Mode (ELM) configuration, ensure that you replace the machine SSL certificates on all vCenter instances in the ELM configuration.
• Make sure the root CA certificate is trusted by all clients and devices that will communicate with the vCenter Server.
• Always take a snapshot of your vCenter appliance before making any changes to the certificates.

By following these steps, you will replace the default SSL certificates with your custom ones, ensuring secure communications for your vCenter environment.

More details: you can also look in to VMware KB316601

Check SSH port connection using curl from VMware Appliances

Login to appliance using correct credential, once you logged in run below command and replace x.x.x.x with your target ip and Port with port number you want to check connectivity.

root@myserver [ ~ ]# curl -v telnet://x.x.x.x:Port
* Trying x.x.x.x:Port...
* Connected to x.x.x.x (x.x.x.x) port Port (#0)
^C

if port reachable you will get connected status. else it will keep trying but not able to connect.

Packet capture for VM on esxi host for vNic and pNic using pktcap-uw

To capture packet on ESXi host for particular VM and related uplink (pNic), you need to get the PortNum for VM and pNic used for that particular VM. 

In this example I am taking Test-VM which has IP address 10.10.10.10

You can get Port-ID after connecting to esxi host using SSH on which that VM is running, using command net-stats -l 

example:

net-stats -l

PortNum             Type SubType SwitchName       MACAddress         ClientName

33577595            5       9    DvsPortset-0        00:50:56:82:70:48     Test-VM.eth0

You can also get the VM PORT-ID and pNic used for impacted VM for which you need to do packet capture.

esxtop

then press n for network detrails of host and VMs

PORT-ID   USED-BY                            TEAM-PNIC    DNAME              PKTTX/s   MbTX/s   PSZTX      PKTRX/s  MbRX/s   PSZRX  %DRPTX %DRPRX

33577595  60324549:Test-VM.eth0       vmnic1       DvsPortset-0          1.79    0.00    82.00       2.26    0.00     76.00   0.00   0.00

Once you note the details for Port-ID and pNIC used for VM. You can go ahead and run the packet capture using pktcap-uw

Test-VM using port-id 33577595 and pNIC vmnic1

There are two way to get the capute one you can see the output direct on esxi console or you can capture the same in pcap file for later analysis.

To capture at VM Port-ID level:

Below command will show all the traffic on that VM port id which has 10.10.10.10 either source or destination IP.

pktcap-uw --capture VnicTx,VnicRx --switchport 33577595 --ip 10.10.10.10 -o - | tcpdump-uw -enr -

To capture the output of capture to a file you can use below command, you need to provide the path to capture file:

pktcap-uw --capture VnicTx,VnicRx --switchport 33577595 --ip 10.10.10.10 -o /vmfs/volumes/DatastoreName/FileName.pcap

To capture packet for particular source and destination for any port:

pktcap-uw --capture VnicTx,VnicRx --switchport 33577595 --srcip 10.10.10.10 --dstip 10.10.10.20  -o - | tcpdump-uw -enr -

To capture packet for particular source and destination for specific port:

pktcap-uw --capture VnicTx,VnicRx --switchport 33577595 --srcip 10.10.10.10 --dstip 10.10.10.20 --tcpport 22 -o - | tcpdump-uw -enr -

To capture output to a file:

pktcap-uw --capture VnicTx,VnicRx --switchport 33577595 --srcip 10.10.10.10 --dstip 10.10.10.20 --tcpport 22 -o /vmfs/volumes/DatastoreName/FileName.pcap

To capture at pNic level used for that VM:

Below command will show all the traffic on pNIC level,  where you can add filter using source IP, Destination IP and Port Number along with direction.

In below example I want to see the traffic sent or received on IP 10.10.10.10 on port number 22 on vmnic1

pktcap-uw --uplink vmnic1 --capture UplinkSndKernel,UplinkRcvKernel --ip 10.10.10.10  --tcpport 22 -o - | tcpdump-uw -enr -

To capture above command output to a pcap file for later analysis.

pktcap-uw --uplink vmnic1 --capture UplinkSndKernel,UplinkRcvKernel --ip 10.10.10.10  --tcpport 22 -o /vmfs/volumes/DatastoreName/FileName.pcap

To capture traffic between 2 particular end points on given pNIC

pktcap-uw --uplink vmnic1 --capture UplinkSndKernel,UplinkRcvKernel --srcip 10.10.10.10 --dstip 10.10.10.20  -o - | tcpdump-uw -enr -

To Save output to a file:

pktcap-uw --uplink vmnic1 --capture UplinkSndKernel,UplinkRcvKernel --srcip 10.10.10.10 --dstip 10.10.10.20  -o /vmfs/volumes/DatastoreName/FileName.pcap 

To capture traffic between 2 particular end points along with port on given pNIC

pktcap-uw --uplink vmnic1 --capture UplinkSndKernel,UplinkRcvKernel --srcip 10.10.10.10 --dstip 10.10.10.20 --tcpport 22 -o - | tcpdump-uw -enr -

To Save output to a file:

pktcap-uw --uplink vmnic1 --capture UplinkSndKernel,UplinkRcvKernel --srcip 10.10.10.10 --dstip 10.10.10.20 --tcpport 22 -o /vmfs/volumes/DatastoreName/FileName.pcap 

You can change the port-ID, pNIC, IP and Port Number according your use case.

vRA 7.6 IaaS Management Agent Certificate replace with New Self Sign Certificate

The system administrator can replace the Management Agent certificate when it expires or

about to expire with new self-signed certificate.

Each IaaS host runs its own Management Agent. Repeat this procedure on each IaaS node whose

Management Agent you want to update.

Prerequisites:

Obtain vRA Management Agent hostname (hn):

Copy the vRA Management Agent hostname from the vRA VAMI interface under cluster tab, where you see the warning or error for certificate for agent.

Obtain vRA Management Agent ID (nd):

Copy the details from the existing Agent Config file from the vRA Management Agent installation directory.

On each of the IaaS nodes where you will be replacing the certificate, obtain the Management Agent identifier by opening the below file.

<vra-installation-dir>\Management Agent\VMware.IaaS.Management.Agent.exe.config file. 

agentConfiguration id

Obtain vRA appliance SSL Thumbprint (tp):

You can find this in the same file <vra-installation-dir>\Management Agent\VMware.IaaS.Management.Agent.exe.config file

Find below screenshot for sample file and data to copy

Once all the required data copied perform below 2 steps, before generating and registering new self-sign certificate with vRA appliance management site.

1. Stop the Management Agent service from your Windows Services snap-in.

 VMware vCloud Automation Center Management Agent service

Stop to stop the service.

2 Remove the current certificate from the machine. 

Local Machine certificate snap-in in MMC console, remove the current Management Agent certificate.

• Register the Management Agent certificate with the vRA appliance management site.

Open a command prompt as an administrator and navigate to the Cafe directory on the

machine on which the Management Agent is installed at <vra-installation-dir>

\Management Agent\Tools\Cafe, typically C:\Program Files (x86)\VMware\vCAC

\Management Agent\Tools\Cafe.

Enter the Vcac-Config.exe RegisterNode command with options to register the

Management Agent identifier and certificate in one step. Include the Management Agent

identifier you recorded earlier as the value for the -nd option.

Vcac-Config.exe RegisterNode -v -vamih "vra-va.eng.mycompany:5480" -cu "root" -cp

"secret" -hn "iaas.eng.mycompany" -nd "C816CFBX-4830-4FD2-8951-C17429CEA291" -tp

"70928851D5B72B206E4B1CF9F6ED953EE1103DED"

Restart the Management Agent.

Now you can verify from vRA VAMI UI under cluster Agent certificate warning/error should disappeared.

To get more details use VMware documentation: https://kb.vmware.com/s/article/86046

Unregister vCenter with External PSC or Embedded PSC - Remove vCenter from ELM

Prerequisite:

1. Make sure you have root and administrator@vsphere.local credential

2. Take offline snapshot of vCenter servers in ELM.

vCenter Server Appliance:

1. Power off the vCenter Server Appliance you are decommissioning.
2. SSH to one of the other working External PSC or Embedded PSC/vCenter appliance within the same SSO domain via root
3. To enable the Bash shell, run:

shell.set --enabled true

4. Run the shell command to start the Bash shell and log in.
5. Run the cmsso-util unregister command to unregister the vCenter Server Appliance:

   cmsso-util unregister --node-pnid vCenterServer_System_Name --username administrator@your_domain_name --passwd 'vCenter_Single_Sign_On_password'

   Where vCenter_Server_Appliance_System_Name is the FQDN or IP address of the vCenter Server Appliance that you want to decommission. Please enter the vCenter_Single_Sign_On_password in quotes.
   
   Note: there is downtime involved in executing the cmsso-util unregister command. It will restart the services on the node where you are executing the command.

6. Delete the vCenter Server Appliance that you no longer need from the vSphere inventory.

Form more details: https://kb.vmware.com/s/article/2106736

Updating the vCenter Server Appliance via CLI and Offline Patch ISO.

***Before starting the patching activity, in case of vCenter in ELM need to shutdown and take offline snapshot for all the vCenters in ELM . So that if rollback require there is working state to revert back, during vCenter patch activity recommended to pause VM backups.

Download patch ISO from VMware Patch download page, by logging with your customer connect credential.

https://customerconnect.vmware.com/patch (for example see screenshot below):

1. Find venter VM and note ESXi host it is running on. Put DRS Automation in manual mode if host part of DRS cluster.

2. Connect ESXi host UI on which vCenter VM is running.

3. Shutdown vCenter VM.

4. Take snapshot of VM.

5. PowerON vCenter VM

6. Mount vCenter latest patch iso 

7.  SSH to vCenter 

Check the shell

If not on Vcenter appliance shell prompt.

Run this appliancesh

If already on Vcenter shell then run below commands.

Be on below prompt of venter appliance.

Command>

Run the below commands one by one.

Command> software-packages stage --iso --acceptEulas

Command> software-packages list --staged

Command> software-packages install --staged

After successful upgrade, this will reboot venter VM, wait for all the service come up.

To check version of vCenter after patch you can run the command (from bash shell) 

Type shell and press Enter

vpxd -vl

To check service status on vcenter server run below command:

service-control --status

This completes the vCenter patch activity. You can remove snapshot as per your org policy. Keeping snapshot for longtime not recommended.

Enable SSH on NSX-T manger using CLI

 Login to NSX-T manager using admin account from the console

To start just for current session:

nsxmanager> start service ssh

To keep service started even after reboot:

nsxmanager> set service ssh start-on-boot

To check service status:

nsxmanager> get service ssh

To see the list of all services:

nsxmanager> get services