A handshake state machine is the protocol logic that tracks each step of connection setup and message exchange. It records what has already happened, what message is expected next, and how to recover when a packet is lost, delayed, or repeated. In TLS and DTLS, this state governs client and server negotiation, key exchange, and session setup.
It matters because security often depends on the state being correct. If the machine resumes from the wrong step, reuses a stale offset, or accepts messages out of order, the code may read or write the wrong memory, skip validation, or leak data. In attacks, flaws in handshake state handling can expose heap contents, break authentication, or create denial-of-service conditions. In defenses, careful state tracking, retransmission limits, and strict message checks help keep protocol recovery logic safe on unreliable networks.