Transactions are typically discussed in a database context. Most Windows developers know them from classic databases or some ORM layer that handles the plumbing. But NTFS supports them too. So does the Windows registry. And they have been there since Vista.
That is worth knowing for a few reasons. The obvious one is correctness: file and registry operations under a transaction either all succeed or all fail. The less obvious one is the attack surface. A well-known malware technique called Process Doppelganging taks advantage of NTFS transactions. Understanding how they work is the prerequisite for understanding that.
What Is a Transaction, Exactly?
In the classical sense, a transaction is a set of operations that adheres to the ACID properties. These come from databases, but they apply here just as well.
Atomicity: the entire set of operations succeeds together, or nothing goes through. There is no in-between. Either the transaction commits, or everything rolls back.
Consistency: anyone outside the transaction always sees a consistent state. While a transaction is in progress, other processes see the data as it was before the transaction started. They do not see partial changes.
Isolation: multiple concurrent transactions do not interfere with each other. Each one operates as if it is the only one running.
Durability: once committed, the changes survive crashes. The data ends up in a consistent state even after an unexpected failure. That last property is largely why NTFS already has a journal; durability requires some kind of recovery mechanism. The transaction layer builds on top of that.
The Kernel Transaction Manager
Windows implements file and registry transactions through the Kernel Transaction Manager (KTM), which has been part of the OS since Windows Vista. It provides a kernel object type called TmTx (transaction manager transaction) that represents an active transaction.
The relevant header is KtmW32.h and the import library is KtmW32.lib. These are not pulled in automatically. Most Visual Studio projects will produce a linker error until you add the lib explicitly. That is a common first stumbling block.
You can verify an active transaction handle in Process Explorer: enable “View > Show Unnamed Handles and Mappings” and look for a handle of type TmTx in your process. Without admin privileges, Process Explorer will not show the object address, but you will see the type.
Creating a Transaction
The entry point is CreateTransaction:
#include <KtmW32.h>
HANDLE htx = CreateTransaction(
nullptr, // security attributes
nullptr, // UOW (reserved)
0, // create options
0, // isolation level (reserved)
0, // isolation flags (reserved)
0, // timeout in ms (0 = no limit)
nullptr // description
);Most parameters are reserved and not actually supported by the current transaction manager. Real databases implement multiple isolation levels (read committed, serializable, and so on) because they are worth the complexity there. KTM does not go there. You get one level, and it does what you would expect.
The timeout parameter is worth knowing: specify a non-zero value and the transaction auto-rolls-back after that many milliseconds if you have not committed. Zero means no limit.
File Operations Under a Transaction
With a transaction handle in hand, use CreateFileTransacted instead of CreateFile:
HANDLE hfile = CreateFileTransacted(
L"E:\\Temp\\sample.txt",
GENERIC_WRITE,
0, // exclusive access
nullptr, // security attributes
CREATE_ALWAYS,
FILE_ATTRIBUTE_NORMAL,
nullptr, // template file
htx, // the transaction
nullptr, // unimportant here
nullptr // reserved
);The key extension is the transaction handle at the end. Everything else is identical to CreateFile. The handle you get back is a normal file handle, and here is the important part: there is no WriteFileTransacted or ReadFileTransacted. The handle carries the transaction context. Any read or write through that handle is automatically part of the transaction. You do not need to change your I/O code at all.
The same pattern applies to other file operations: CopyFileTransacted, CreateDirectoryTransacted, and similar. Each takes a transaction handle; the rest of the signature matches the non-transacted version.
$1,938
$1,356 or $140 X 10 payments
Windows Internals Master
Broadens and deepens your understanding of the inner workings of Windows.
Registry Transactions
The registry side is symmetrical. RegCreateKeyTransacted and RegOpenKeyTransacted are the entry points. Once you have a registry key handle opened under a transaction, normal calls like RegSetValueEx and RegQueryValueEx operate within that transaction context automatically.
This means you can bundle file system changes and registry changes under a single transaction. Either all of it commits, or none of it does. That is the practical value for installation logic, update routines, or any operation where partial success leaves the system in a broken state.
Committing and Rolling Back
When you are done:
CommitTransaction(htx); // changes become permanent and visibleor
RollbackTransaction(htx); // changes are discardedIf you close the transaction handle without calling either, that is an implicit rollback. The changes disappear. This is easy to forget. You can write to a file, read back data that looks correct, close the file handle, and then let the transaction handle close when your process exits. The file is never changed on disk. The APIs reported success the whole time.
Process Doppelganging
Now for the reason this matters in a security context.
Process Doppelganging is a technique that abuses NTFS transactions to load a malicious executable while making it look, from any external view, like a legitimate one.
The core idea: create a transaction and use CreateFileTransacted to open a legitimate Windows executable, say notepad.exe. Write your malicious payload into it as part of the transaction. At this point the on-disk file is unchanged; the modifications exist only inside the transaction, invisible to any other process.
Before committing or rolling back, call CreateFileMapping on that file handle. This creates a section object backed by the transacted view of the file (your malicious content). Then roll back the transaction. The file on disk reverts to the original. notepad.exe is untouched.
But you still have the section object.
// Inside the transaction, after writing malicious content:
HANDLE hmap = CreateFileMapping(
hfile,
nullptr,
PAGE_READONLY,
0, 0, // entire file
nullptr
);
CloseHandle(hfile);
RollbackTransaction(htx);
CloseHandle(htx);
// hmap still reflects the transacted content that was never committedThe mapping captured the transacted view at creation time. After the rollback, the section still holds the data that existed during the transaction. That data was never written to disk. But it is available in memory, and you can use it to back a process.
Creating a process from a section directly (via NtCreateProcessEx) is a lower-level step worth its own coverage. But the NTFS transaction piece — creating the mapping while the transaction is open, then rolling back, is the foundation. Process Explorer and most security tools check the file on disk. The file on disk is clean. The process running in memory is not.
Key Takeaways
- NTFS and the registry support ACID transactions via the Kernel Transaction Manager, available since Windows Vista.
CreateTransactionis the entry point;CreateFileTransactedandRegCreateKeyTransactedhook file and registry operations into the transaction.- Normal I/O functions (WriteFile, ReadFile, RegSetValueEx) do not need transacted variants; the handle carries the context.
- Closing a transaction handle without calling
CommitTransactionis an implicit rollback. The changes vanish silently. CreateFileMappingon a transacted file handle captures the transacted view of the file. After rollback, the section retains that view even though the file on disk is unchanged.- Process Doppelganging uses exactly that property to run a malicious payload under the guise of a legitimate executable.
Keep Learning
For a full grounding in how the Windows kernel manages objects, handles, file systems, and security internals, the Windows Internals course series is the right starting point. The complete track runs five days and covers the object manager, memory, I/O, process internals, and more.
For hands-on use of the Windows API in C++ (processes, handles, file I/O, synchronization), Windows System Programming 1 covers the foundations you need before going deeper into internals.
