Prerequisites: Requirements for the users that are planning to learn Malware Development are first of all basics of programming, since mostly deals with the C programming lanaguage, it is a requirement that you are familiar with fundamentals of C.

So what is actually Windows Architecture? It is what happens under the hood of Windows processes and applications, the internal structure of the Windows operating system, envolves layers like hardware, kernel, executive services, and subsystem DLLs.

Windows Architecture Link to heading

First things firts, let’s clear out a few terms:

What is a processor?

The processor is the CPU, the hardware that performs instructions. It fetches, decodes, and executes machine instructions, perform arithmetic and logical operations, also manages the flow of control between various programs (or processes) running on the machine.

What is an application?

An application is a program that runs on the operating system, usi its resources to function. Is stored on disk as an executable file(or set of files), when run, Windows loads the application into memory as one or more processes. The application’s code along with its resources (data, libraries, etc), then runs under the direction of the operating system and is executed by the processor. There is a separtion that you must get it, between the program file (the code) and its process (the running instance).

Subsystem DLLs:

Subsystem DLLs are dynamic link libraries that implement the user-mode subsystems of Windows (explained down below). In Windows, much of the functionality provided to applications isn’t built into the kernel—it’s offered via these modular DLLs. Examples include libraries like kernel32.dll, user32.dll, gdi32.dll, and others. Each of these is designed to provide a specific set of services like kernel32.dll that offers ore functions for memory management, process creation, and file I/O, user32.dll that implements user interface components (windows, messages, input handling) or gdi32.dll that deals with graphics device interface functions for drawing and font management.

Bridge the gap between high-level application code and the Windows kernel. When an application makes an API call to perform a task (like opening a window or reading from a file), it typically calls a function in one of these subsystem DLLs. The DLL then translates that request into lower-level system calls that the kernel can process. This layered, modular design keeps the system efficient and easier to update.

Windows API:

The Windows API (Application Programming Interface) is the complete set of functions, data structures, and conventions that Microsoft exposes to allow applications to interact with Windows. Informally known as Win32 (for the 32-bit version) or simply Windows API, it is the standard way for developers to request services from the operating system.

The API is vast, covering functions for:

• Process management, file I/O, memory allocation (e.g., CreateProcess, ReadFile).
• Creating and managing windows, processing messages, drawing graphics (like CreateWindow, MessageBox).
• GDI/GDI+ for drawing text and images.

How It Works:
When you write a Windows application, you include header files (windows.h) that declare these functions and types. At runtime, your application’s calls are routed through the Windows loader to the appropriate subsystem DLL, which then communicates with the kernel. This call flow from the application, through the DLL (using functions like LoadLibrary and GetProcAddress if needed), and finally to the kernel is the backbone of how Windows maintains control while allowing third-party programs to perform complex tasks.

So basically the Subsystem DLLs are the modular libraries that provides Windows core services to applications, isolating them from directly hardware interactions and the Windows API is the full set of functions and conventions that let applications request and use Windows services, enabling tasks from file operations to GUI management.

So after this has been cleared out, let’s actually understand the Windows Architecture. A processor inside a machine running the Windows OS can operate under two different modes:

User Mode and Kernel Mode, Applications run in user mode, and operating system components run in kernel mode. When an application wants to accomplish a task, such as creating a file, it cannot do so on its own. The only entity that can complete the task is the Kernel, so instead applications must follow a specific call flow.

What all of that means?

User mode is where regular apps run with limited privileges, while kernel mode is where the system’s core functions, like device drivers, operate with full access to hardware

User Mode vs Kernel Mode

User Mode:

Regular applications and most noncritical processes run, the code is executed with limited privileges, why this isolation occurs?

This isolation prevents a faulty or malicious application from directly tampering with system hardware or core services, but as we will see in the future, it has its work arounds. Each application running in user mode is isolated in its own virtual address space, so for example a crash or error in one process won’t necessarily bring down the entire system. So the applications utilize high-level APIs provided by the subsystem DLLs (like, kernel32.dll, user32.dll) to perform tasks, these handle the necessary preliminary processing and then request services from the operating system.

Kernel Mode:

The kernel mode is the highly privilege layer of the Windows OS, is where the Windows Kernel, device drivers, and core system services run, the code that runs in kernel mode can directly access hardware and system memory.

Kernel mode has unrestricted access, critical for performing essential operations like hardware management, memory allocation, and I/O operations(input/output), a failure here can bring the whole system down.

System Calls:

When a user mode program needs to access hardware or perform privileged operations, it makes a system call. This essentially triggers a request that gets processed by kernel mode.

For better understanding, let’s breakdown:

So the User Processes (program/application executed by the user) calls Subsystem DLLs (DLLs that contain API functions that are called by User Processes, like kernel32.dll exporting CreateFile Windows API) then calls Ntdll.dll (A system-wide DLL which is the lowest layer available in user mode. This is a special DLL that creates the transition from user mode to kernel mode. This is often referred to as the Native API or NTAPI.) after that Executive Kernel (This is what is known as the Windows Kernel and it calls other drivers and modules available within kernel mode to complete tasks. The Windows kernel is partially stored in a file called ntoskrnl.exe under "C:\Windows\System32".)

How this transition works from user mode to kernel mode?

This transition is part of what is know as Function Call Flow:

Essentially, when a program in user mode makes a system call (like reading a file), the CPU needs to switch to kernel mode to access protected resources. The transition typically happens through an interrupt or system call mechanism. This allows the operating system to safely manage privileged actions, ensuring user programs don’t directly interfere with the system’s low-level operations.

Step by step:

User Process: A user mode process (an application) starts in user mode and calls a high-level Windows API function. For example, when your application calls CreateFile (from kernel32.dll), it’s invoking a function provided by a subsystem DLL.

Subsystem DLLs:
The subsystem DLLs (such as kernel32.dll, user32.dll, etc.) provide the API functions that the application uses. These DLLs contain wrappers that set up the call and then invoke lower-level routines as needed. In the case of functions like CreateFile, the DLL wraps the call and then delegates to a lower layer.

Ntdll.dll (Native API):
The next step involves ntdll.dll. This special system-wide DLL sits at the bottom of the user mode layer and contains the system call stubs(part of) these are the routines (often referred to as the Native API or NTAPI) that prepare the transition from user mode to kernel mode. Essentially, when a subsystem DLL calls into ntdll.dll, it’s setting up a system call that the CPU can execute to switch to kernel mode.

Executive Kernel (ntoskrnl.exe and drivers):
Once the system call is issued (using instructions like syscall or sysenter), the CPU switches to kernel mode. The request is then handled by the Windows kernel (primarily in ntoskrnl.exe, which is part of the Executive Kernel). The kernel executes the requested operation by calling the appropriate drivers and kernel modules to complete the task.

Return Path:
After the kernel completes the operation, the result is passed back through ntdll.dll to the subsystem DLLs, which in turn return the result to the original user mode process.

So again, for consolidating, it begins with the user application calling the CreateFile WinAPI function which is available in kernel32.dll . Kernel32.dll is a critical DLL that exposes applications to the WinAPI and is therefore can be seen loaded by most applications. Next, CreateFile calls its equivalent NTAPI function, NtCreateFile , which is provided through ntdll.dll . Ntdll.dll then executes an assembly sysenter (x86) or syscall (x64) instruction, which transfers execution to kernel mode. The kernel NtCreateFile function is then used which calls kernel drivers and modules to perform the requested task.

Directly Invoking The Native API (NTAPI)
It’s important to note that applications can invoke syscalls (NTDLL functions) directly without having to go through the Windows API. The Windows API simply acts as a wrapper for the Native API. With that being said, the Native API is more difficult to use because it is not officially documented by Microsoft.

Normally, applications use the Windows API (like CreateFile, WriteFile), which is provided by DLLs like kernel32.dll.

Directly calling native functions from ntdll.dll (NTAPI functions) is possible.

The Windows API is a stable, documented interface, used by most apps it’s built on top of the Native API (NTAPI, in ntdll.dll). NTAPI functions are the system call stubs that handle system-level tasks, transitioning to kernel mode.

How the wrapping works:
When you call a function like CreateFile from kernel32.dll, you’re not directly invoking the lowest-level system routines. Instead, CreateFile prepares the parameters and calls a corresponding function in ntdll.dll (such as NtCreateFile).

The transitions flow works as follow:

1. User Process: Your application calls CreateFile in kernel32.dll.
2. Kernel32.dll: This function performs any parameter checking, error translation, or compatibility work and then calls the native function (NtCreateFile) in ntdll.dll.
3. Ntdll.dll (NTAPI): Here, the function issues the actual system call (using CPU instructions like syscall or sysenter) that transitions execution from user mode to kernel mode.
4. Kernel Mode: The Windows kernel (ntoskrnl.exe) processes the request, interacts with drivers, and completes the operation.
5. Return Path: The result flows back the same way, eventually returning to your application.

May sound kinda repetitive, but it’s really important for you to get this.

Memory Management in Windows Link to heading

Memory management is a critical component of any operating system, and Windows is no exception. The Windows memory manager is responsible for handling memory allocation, virtual memory, paging, and other low level operations that ensure efficient and memory usage by both user mode applications and the kernel.

Virtual Memory Link to heading

Windows employs a virtual memory system that provides each process with its own virtual address space. This abstraction allows applications to operate as if they have a contiguous block of memory, even if the actual physical memory is fragmented or even partially stored on disk.

• Each process in user mode has a virtual address space of 4GB (for 32-bit systems) or significantly larger on 64-bit systems.
• The lower half of the address space (0x00000000 to 0x7FFFFFFF in 32-bit) is reserved for the application, while the upper half (0x80000000 to 0xFFFFFFFF) is reserved for the kernel.
• The Memory Manager is responsible for translating virtual addresses to physical addresses using page tables.

Paging and Page Faults Link to heading

The Windows memory manager uses a technique called paging to manage memory efficiently:

• The system divides memory into fixed-size pages (usually 4KB).
• When an application requests memory, it is assigned pages from physical memory (RAM).
• If a requested page is not in RAM, a page fault occurs, and Windows loads the page from disk (paging file or another storage source).
• The paging file (pagefile.sys) is used when RAM is insufficient, allowing memory pages to be temporarily stored on disk.

Page fault handling:

1. The CPU attempts to access a memory page.
2. If the page is not in physical memory, a page fault occurs.
3. The memory manager retrieves the page from disk and maps it to physical memory.
4. The process resumes execution once the page is available.

Address Space Layout Randomization (ASLR) Link to heading

ASLR is a security feature that randomizes the memory addresses used by system and application components to make it harder for attackers to predict addresses in memory.

• Executable images (e.g., ntdll.dll, kernel32.dll) are loaded at randomized locations.
• Stack and heap allocations are randomized.
• This prevents predictable memory layouts, making it difficult for exploits like buffer overflows and return-oriented programming (ROP) attacks to succeed.

Heap and Stack Management Link to heading

Windows manages memory using two primary structures:

The Stack Link to heading

• Used for function calls and local variables.
• Managed automatically by the CPU and OS.
• Has a fixed size, typically 1MB in Windows.
• Stack Overflow occurs when too much memory is used, often due to recursive function calls.

The Heap Link to heading

• Used for dynamic memory allocation (malloc, HeapAlloc, VirtualAlloc).
• Managed by the Windows heap manager.
• Can grow as needed.
• Applications are responsible for freeing heap memory (free, HeapFree).

Windows Memory APIs Link to heading

Windows provides several APIs for memory management:

• VirtualAlloc/VirtualFree – Allocates and frees memory pages directly.
• HeapAlloc/HeapFree – Manages dynamic memory in the process heap.
• GlobalAlloc/LocalAlloc – Legacy functions for allocating memory.
• MapViewOfFile – Used for memory-mapped files.

Kernel Mode Memory Management Link to heading

Kernel mode memory management is more privileged and operates differently:

• The kernel uses paged pool and nonpaged pool memory.
• Paged Pool – Can be swapped to disk.
• Nonpaged Pool – Stays in RAM, used for critical components like drivers.
• The kernel uses Memory Descriptor Lists (MDLs) to track memory regions.
• Windows uses IRQLs (Interrupt Request Levels) to manage priority access to memory.