Monday, January 22, 2024

Packet capture for VM on esxi host for vNic and pNic using pktcap-uw

To capture packet on ESXi host for particular VM and related uplink (pNic), you need to get the PortNum for VM and pNic used for that particular VM. 

In this example I am taking Test-VM which has IP address 10.10.10.10

You can get Port-ID after connecting to esxi host using SSH on which that VM is running, using command net-stats -l 

example:

net-stats -l

PortNum             Type SubType SwitchName       MACAddress         ClientName

33577595            5       9    DvsPortset-0        00:50:56:82:70:48     Test-VM.eth0

You can also get the VM PORT-ID and pNic used for impacted VM for which you need to do packet capture.

esxtop

then press n for network detrails of host and VMs

PORT-ID   USED-BY                            TEAM-PNIC    DNAME              PKTTX/s   MbTX/s   PSZTX      PKTRX/s  MbRX/s   PSZRX  %DRPTX %DRPRX

33577595  60324549:Test-VM.eth0       vmnic1       DvsPortset-0          1.79    0.00    82.00       2.26    0.00     76.00   0.00   0.00

Once you note the details for Port-ID and pNIC used for VM. You can go ahead and run the packet capture using pktcap-uw

Test-VM using port-id 33577595 and pNIC vmnic1

There are two way to get the capute one you can see the output direct on esxi console or you can capture the same in pcap file for later analysis.

To capture at VM Port-ID level:

Below command will show all the traffic on that VM port id which has 10.10.10.10 either source or destination IP.

pktcap-uw --capture VnicTx,VnicRx --switchport 33577595 --ip 10.10.10.10 -o - | tcpdump-uw -enr -

To capture the output of capture to a file you can use below command, you need to provide the path to capture file:

pktcap-uw --capture VnicTx,VnicRx --switchport 33577595 --ip 10.10.10.10 -o /vmfs/volumes/DatastoreName/FileName.pcap

To capture packet for particular source and destination for any port:

pktcap-uw --capture VnicTx,VnicRx --switchport 33577595 --srcip 10.10.10.10 --dstip 10.10.10.20  -o - | tcpdump-uw -enr -

To capture packet for particular source and destination for specific port:

pktcap-uw --capture VnicTx,VnicRx --switchport 33577595 --srcip 10.10.10.10 --dstip 10.10.10.20 --tcpport 22 -o - | tcpdump-uw -enr -

To capture output to a file:

pktcap-uw --capture VnicTx,VnicRx --switchport 33577595 --srcip 10.10.10.10 --dstip 10.10.10.20 --tcpport 22 -o /vmfs/volumes/DatastoreName/FileName.pcap

To capture at pNic level used for that VM:

Below command will show all the traffic on pNIC level,  where you can add filter using source IP, Destination IP and Port Number along with direction.

In below example I want to see the traffic sent or received on IP 10.10.10.10 on port number 22 on vmnic1

pktcap-uw --uplink vmnic1 --capture UplinkSndKernel,UplinkRcvKernel --ip 10.10.10.10  --tcpport 22 -o - | tcpdump-uw -enr -


To capture above command output to a pcap file for later analysis.


pktcap-uw --uplink vmnic1 --capture UplinkSndKernel,UplinkRcvKernel --ip 10.10.10.10  --tcpport 22 -o /vmfs/volumes/DatastoreName/FileName.pcap


To capture traffic between 2 particular end points on given pNIC


pktcap-uw --uplink vmnic1 --capture UplinkSndKernel,UplinkRcvKernel --srcip 10.10.10.10 --dstip 10.10.10.20  -o - | tcpdump-uw -enr -


To Save output to a file:


pktcap-uw --uplink vmnic1 --capture UplinkSndKernel,UplinkRcvKernel --srcip 10.10.10.10 --dstip 10.10.10.20  -o /vmfs/volumes/DatastoreName/FileName.pcap 


To capture traffic between 2 particular end points along with port on given pNIC


pktcap-uw --uplink vmnic1 --capture UplinkSndKernel,UplinkRcvKernel --srcip 10.10.10.10 --dstip 10.10.10.20 --tcpport 22 -o - | tcpdump-uw -enr -


To Save output to a file:


pktcap-uw --uplink vmnic1 --capture UplinkSndKernel,UplinkRcvKernel --srcip 10.10.10.10 --dstip 10.10.10.20 --tcpport 22 -o /vmfs/volumes/DatastoreName/FileName.pcap 


You can change the port-ID, pNIC, IP and Port Number according your use case.



Friday, January 19, 2024

vRA 7.6 IaaS Management Agent Certificate replace with New Self Sign Certificate

The system administrator can replace the Management Agent certificate when it expires or

about to expire with new self-signed certificate.

Each IaaS host runs its own Management Agent. Repeat this procedure on each IaaS node whose

Management Agent you want to update.

Prerequisites:



















Obtain vRA Management Agent hostname (hn):

Copy the vRA Management Agent hostname from the vRA VAMI interface under cluster tab, where you see the warning or error for certificate for agent.

Obtain vRA Management Agent ID (nd):

Copy the details from the existing Agent Config file from the vRA Management Agent installation directory.

On each of the IaaS nodes where you will be replacing the certificate, obtain the Management Agent identifier by opening the below file.

<vra-installation-dir>\Management Agent\VMware.IaaS.Management.Agent.exe.config file. 

agentConfiguration id

Obtain vRA appliance SSL Thumbprint (tp):

You can find this in the same file <vra-installation-dir>\Management Agent\VMware.IaaS.Management.Agent.exe.config file

Find below screenshot for sample file and data to copy


Once all the required data copied perform below 2 steps, before generating and registering new self-sign certificate with vRA appliance management site.

1. Stop the Management Agent service from your Windows Services snap-in.

 VMware vCloud Automation Center Management Agent service

Stop to stop the service.

2 Remove the current certificate from the machine. 

Local Machine certificate snap-in in MMC console, remove the current Management Agent certificate.

  • Register the Management Agent certificate with the vRA appliance management site.

Open a command prompt as an administrator and navigate to the Cafe directory on the

machine on which the Management Agent is installed at <vra-installation-dir>

\Management Agent\Tools\Cafe, typically C:\Program Files (x86)\VMware\vCAC

\Management Agent\Tools\Cafe.

Enter the Vcac-Config.exe RegisterNode command with options to register the

Management Agent identifier and certificate in one step. Include the Management Agent

identifier you recorded earlier as the value for the -nd option.

Vcac-Config.exe RegisterNode -v -vamih "vra-va.eng.mycompany:5480" -cu "root" -cp

"secret" -hn "iaas.eng.mycompany" -nd "C816CFBX-4830-4FD2-8951-C17429CEA291" -tp

"70928851D5B72B206E4B1CF9F6ED953EE1103DED"

Restart the Management Agent.

Now you can verify from vRA VAMI UI under cluster Agent certificate warning/error should disappeared.

To get more details use VMware documentation: https://kb.vmware.com/s/article/86046




Tuesday, September 5, 2023

Unregister vCenter with External PSC or Embedded PSC - Remove vCenter from ELM

Prerequisite:

1. Make sure you have root and administrator@vsphere.local credential

2. Take offline snapshot of vCenter servers in ELM.

vCenter Server Appliance:

  1. Power off the vCenter Server Appliance you are decommissioning.
  2. SSH to one of the other working External PSC or Embedded PSC/vCenter appliance within the same SSO domain via root
  3. To enable the Bash shell, run:
shell.set --enabled true
  1. Run the shell command to start the Bash shell and log in.
  2. Run the cmsso-util unregister command to unregister the vCenter Server Appliance:

    cmsso-util unregister --node-pnid vCenterServer_System_Name --username administrator@your_domain_name --passwd 'vCenter_Single_Sign_On_password'

    Where vCenter_Server_Appliance_System_Name is the FQDN or IP address of the vCenter Server Appliance that you want to decommission. Please enter the vCenter_Single_Sign_On_password in quotes.

    Note: there is downtime involved in executing the cmsso-util unregister command. It will restart the services on the node where you are executing the command.

  3. Delete the vCenter Server Appliance that you no longer need from the vSphere inventory.

Form more details: https://kb.vmware.com/s/article/2106736

Wednesday, July 5, 2023

Updating the vCenter Server Appliance via CLI and Offline Patch ISO.

***Before starting the patching activity, in case of vCenter in ELM need to shutdown and take offline snapshot for all the vCenters in ELM . So that if rollback require there is working state to revert back, during vCenter patch activity recommended to pause VM backups.


Download patch ISO from VMware Patch download page, by logging with your customer connect credential.

https://customerconnect.vmware.com/patch (for example see screenshot below):



1. Find venter VM and note ESXi host it is running on. Put DRS Automation in manual mode if host part of DRS cluster.

2. Connect ESXi host UI on which vCenter VM is running.

3. Shutdown vCenter VM.


4. Take snapshot of VM.


5. PowerON vCenter VM


6. Mount vCenter latest patch iso 


7.  SSH to vCenter 


Check the shell

If not on Vcenter appliance shell prompt.

Run this appliancesh

If already on Vcenter shell then run below commands.

Be on below prompt of venter appliance.


Command>


Run the below commands one by one.


Command> software-packages stage --iso --acceptEulas


Command> software-packages list --staged


Command> software-packages install --staged


After successful upgrade, this will reboot venter VM, wait for all the service come up.


To check version of vCenter after patch you can run the command (from bash shell) 


Type shell and press Enter


vpxd -vl


To check service status on vcenter server run below command:


service-control --status


This completes the vCenter patch activity. You can remove snapshot as per your org policy. Keeping snapshot for longtime not recommended.






Monday, May 22, 2023

Enable SSH on NSX-T manger using CLI

 Login to NSX-T manager using admin account from the console

To start just for current session:

nsxmanager> start service ssh

To keep service started even after reboot:

nsxmanager> set service ssh start-on-boot

To check service status:

nsxmanager> get service ssh

To see the list of all services:

nsxmanager> get services


Tuesday, April 4, 2023

Check TCP/UDP port connectivity from ESXi host

Run below command from esxi shell:

For TCP Port:

nc -vz x.x.x.x/name Port

example: if you want to check connectivity from esxi to 10.10.10.10 and TCP port (8000).

nc -vz  10.10.10.10 8000

Connection to 10.10.10.10 8000 port [tcp/*] succeeded!

For UDP Port:

nc -vz -u x.x.x.x/name Port

example: if you want to check connectivity from esxi to Syslog server (10.10.10.10) and UDP port (514).

nc -vz -u 10.10.10.10 514

Connection to 10.10.10.10 514 port [udp/*] succeeded!




VMware Appliances Reset Forget|Expired passwords

To check current password policy:

chage -l root


To change password expiry policy to never expire you can use below command:


chage -M -1 root


vROPS

————


vROPS Admin Account:


https://kb.vmware.com/s/article/2078313


vROPS root Account:


https://kb.vmware.com/s/article/2001476


Unlock Admin Account 


https://kb.vmware.com/s/article/2131633



vRLI

————

vRLI root Account:


https://kb.vmware.com/s/article/53649


vRLI admin Account:


https://docs.vmware.com/en/vRealize-Log-Insight/8.2/com.vmware.log-insight.administration.doc/GUID-48C871F8-6289-406C-9C9A-59E4EA1AF2E5.html


vCenter:

—————


vCenter Root Account:


https://kb.vmware.com/s/article/2147144



vCenter Administrator@vsphere.local Account:


https://kb.vmware.com/s/article/2146224



vRA

—————


How to reset administrator@vsphere.local account password in vRA 7.x. (2150055)


https://kb.vmware.com/s/article/2150055#:~:text=Navigate%20to%20vRA%20Settings%20%3E%20SSO,Save%20settings


vRA root password


https://docs.vmware.com/en/vRealize-Automation/8.8/Administering/GUID-1BC4BD89-73C5-4C2A-875D-B8787418EE85.html#:~:text=At%20the%20root%20%5B%2F%5D%23,and%20press%20the%20Enter%20key



SRM 


SRM Appliance root pass


https://kb.vmware.com/s/article/304686