NTLM authentication, as one of alternatives of Integrated Windows authentication, is integrated into the Windows operating system since it has built-in network support. NTLM authenticates clients through a challenge/response mechanism that is based on a threeway handshake between the client and the server. This authentication works only if the client and the server are running Windows:
1. The client starts the communication by sending a message to the server, which indicates that the client wants to talk to the server.
2. The server generates a 64-bit random value called the nonce and responds to the client’s request by returning this nonce. The response is called the challenge.
3. The client operating system asks the user for a user name and password. After the user has entered this information, the system hashes the password. This password hash is called the master key and used for encrypting the nonce.
4. The client transmits the encrypted nonce together with the user name in his response to the server (completing the challenge/response mechanism).
5. The server now needs to validate the returned nonce. This validation takes place locally when the user is local or remotely on the domain controller when the user is domain one. In both cases, the user’s master key, which is the hashed version of the password, is retrieved from the security account database.
6. The master key, retrieved from the security account database, encrypts encrypts the clear-text nonce again on the server (of course, the server has cached the clear-text nonce before it transmits the data to the client).
7. The user is authenticated successfully, and a logon session is created on the server for the user if the re-created encrypted version of the nonce matches the encrypted version returned from the client.
The next picture presents the process of NTLM authentication:
The NTLM protocol at a glance