Technical Note:
First Published 21 Jul 2016
Updated 5 Oct 2016
On upgrading from ESXi 5.5 to version 6.0 (using HP’s custom ISO on HP hardware) we found that we could no longer make a connection from the Windows client to any vSphere hosts. In fact, the connection could be made, but was only successful around 1 in 10 times.
This problem appeared on multiple client machines and to multiple hosts running V6.0
Initially the fix below was employed and appeared to resolve the issue. However, the problem re-occured and logging in was not possible. The original fix therefore appeared to be a separate issue. It was later found that it is the account lockout feature that is implemented in VMWare V6.0 onwards which caused the issue.
- On the problem machine, click Start > Run, type regedit, and click OK. The Registry Editor window is displayed.
- Navigate to the HKCU\Software\VMware\VMware Infrastructure Client\Preferences\CLIENT_CMD_TIMEOUT registry key.
- Increase the timeout value.Note: If CLIENT_CMD_TIMEOUT does not exist, manually create a new key with the value ’90’. Ensure that the key is created with the type REG_SZ (String).
The answer to this problem can be found by looking in the logs. However – if you can’t get remote access due to account lockout – you will not be able to obtain this information.
Checking the event log, you will see…
Remote access for ESXi local user account ‘root’ has been locked for 120 seconds after xxx failed login attempts.
This behavious changed in V6.0, and is described in the documentation.
In the section ESXi Passwords, ESXi Pass Phrases, and Account Lockout of the ESXi and vCenter Server 6.0 Documentation you can find the following information:
ESXi Account Lockout Behavior
Starting with vSphere 6.0, account locking is supported for access through SSH and through the vSphere Web Services SDK. The Direct Console Interface (DCUI) and the ESXi Shell do not support account lockout. By default, a maximum of ten failed attempts is allowed before the account is locked. The account is unlocked after two minutes by default.
You can configure the login behavior with the following advanced options:
Security.AccountLockFailures. Maximum number of failed login attempts before a user’s account is locked. Zero disables account locking.
Security.AccountUnlockTime. Number of seconds that a user is locked out.
And indeed this is new in version 6.0.
In this specific case the host was directly connected to the Internet (in a remote data centre) and had shell access via ssh permanently enabled. The IP, being publicly available, was being scanned by ‘hackers’ who then repeatedly tried to break into the ESXi host using brute force. The new account lockout feature helps to mitigate this kind of attack, but it will also prevents legitimate access to the system like in this case. It is important to understand that not only ssh access, but also access to the host via the vSphere client or API calls is prevented when the root account is locked out.
To prevent this problem, modify firewall entries to only allow access from your authorised ‘management’ IP addresses.
This can be done using the ESXi builtin firewall to limit access for at least ssh (port 22) and the vSphere client (port 443) to known trusted IP addresses.
In our case the whole data centre rack is front ended by a Cisco hardware firewall, so rules were entered that only allowed access to the VMWare hosts via a controlled set of addresses. Take care that external backup addresses are also included if necessary for on-line backups to take place.