Understanding Shellcode: Key Concepts for Cybersecurity Experts

Introduction

Shellcode is a set of machine code instructions typically written in assembly language, designed to be executed directly by a computer's processor. Because assembly's instructions are architecture-specific that restricts the portability of shellcodes among different processors.

In general, shellcode focuses on direct manipulation of processor registers, configuring them for various system calls using opcodes. Once the assembly code is crafted to execute the desired operation, then it must be converted into machine code. However, this will not be enough, removing all null bytes is crucial. The reason is that many string operations, such as strcpy(), stop when encountering null bytes.

To better understand shellcodes, we'll first examine two fundamental concepts: Assembler and System Calls (syscall).

Netwide Assembler (NASM)

Assembly language is a low-level programming language that is specific to a particular computer architecture. It needs an assembler that translates assembly language code into machine code. There are different assemblers for different architecture. In our case, we will use the Netwide Assembler (NASM) which is an x86 and x86-64 assembler that supports several object file formats, including ELF, Win32, and others. You can specify the object file format with the -f switch.

System Calls (syscall)

To execute any functionalities like opening system ports or modifying permissions, the operating system provides system calls. System calls serve as a means to manage communication with hardware and access kernel functionality that might not be present in the application's address space. On UNIX-based operating systems, each function is assigned a unique system call number.

For instance, when a user-level program needs to access a function beyond its address space, such as write(), it must first determine the system call number associated with write() function. Then, it triggers an interrupt which signals to the operating system that a request needs attention.

The actual numbers assigned to each system call can vary between operating systems. In Linux, syscall numbers are defined in the kernel headers. One easy way to get the system call numbers is to use ausyscall tool

As in the case of most system calls, one or more arguments are necessary. The system call number is loaded into the EAX register, while the arguments intended for the desired function are typically loaded into EBX, ECX, and EDX, following this order. In 64-bit architecture, the arguments for system call are placed in RDI, RSI, RDX, RCX, R8 and R9 in that order (calling convention).

Example: Writing a message to the standard output

In this example, we will print the message “System Calls” to our screen. Within our current environment, we identify the syscalls for two functions:

• The write() function and,
• The exit()

The write() function, size_t write(int files, const void *buf, size_t nbyte), has the system call number 0x1 stored in the rax register, and it takes three arguments:

• The file descriptor for standard output is passed through the rdi
• The buffer pointing to the message "System Calls" is passed via the rsi
• The length of the message (nbyte) is passed through the rdx

Once all the required registers are configured, a call to the write() function is executed.

As for the exit() function, void exit(int status), it possesses the system call number 0x3C and requires only one argument. In this case, zero is passed via the rdi register.

Now, it is time to assemble, compile, and run our simple write program which prints "System Calls" on screen

Note
In 32-bit x86 systems, the int 0x80 instruction is commonly employed to initiate an interrupt. In 64-bit systems, this is substituted with the syscall instruction.

Creating Shellcode

As a general security best-practices, applications typically drop their privileges whenever possible. To enable our shellcode to call a root shell, we must invoke a function that restores the application's privileges. This can be achieved using the setreuid() system call. Therefor, our objective is to create a position-independent assembly code that calls a root shell.

The following code first escalates its privileges by setting the effective user ID to zero and then spawns a shell with the newly acquired privileges using the execve system call. After compiling it, we grant ownership to the root user and enable the SetUID (SUID) permission. This configuration enables our shellcode to showcase the process of reverting ownership to the root user.

The sequence “0x68732f6e69622f” represents the reverse order of the hexadecimal value corresponds to the ASCII representation of the string "/bin//sh"

Again, let's assemble, compile, and run

Looking at the xxd output of the object file syscall_shell.o, it's clear that our shellcode contains a lot of null bytes, which is a problem (For demonstration purposes, I am using a complete assembly code).

In many exploitation scenarios, we often rely on string manipulation functions like strcpy() or gets() to copy data into a buffer. However, when these functions encounter a null byte, they interpret it as the end of the string, leading to the failure of our shellcode execution.

At this point, we're facing two issues. First, we need to eliminate the null bytes (0x00). Second, the previous code is a complete assembly code with its sections, which was meant to illustrate our concept. However, in practical terms, we only need the instructions to spawn a shell. Let's tackle these two challenges in the next steps.

Removing Null Bytes

We're facing a challenge with null bytes in our assembly code. Take, for instance, the instruction to move 10 (0x0a) into the 64-bit register rax. This operation results in '0x000000000000000a', introducing 7 null bytes. These null bytes can cause trouble, especially in string operations and can break our shellcode.

To overcome this issue, we can use tricks like accessing smaller portions of a 64-bit register directly. For instance, the lower 8 bits of rax can be accessed using the register name 'al'.

In addition, you'll often come across the instruction “xor eax, eax” in assembly to reset a register because it doesn't mess with the EFLAGS register. XORing something with itself always gives you 0. Another way to zero out a register is by subtracting it from itself, like “sub eax, eax”. You can also move the value of an existing register that's already contains 0 into another register using mov instruction “mov eax, ecx”.

However, keep in mind that these instructions can make your shellcode larger than it needs to be (that's for another article). Let's apply these concepts to our code.

As you can see in the following code snippet, the .text section has been excluded. XOR operations are employed to reset registers, and the least significant byte of rax (al) is used to store the system call numbers.

Let's assemble, compile, and run

Now we see that the null bytes have been successfully eliminated. Additionally, the size of the shellcode has significantly reduced to just 40 bytes, cool, right.

Test Shellcode

Now that we have our machine code (opcode), which is essentially our set of instructions to spawn a shell, the next step is to test our shellcode. Fortunately, there are several C programs designed specifically for this purpose.

In the code snippet below, the shellcode is casted to a function pointer. This function pointer is then invoked, executing the instructions contained within the shellcode array.

We then assign ownership to root and turn on SUID. This allows our shellcode to demonstrate the restoring of root privileges prior to spawning a shell. After executing shellcode if you run the id command you should get uid=0

Happy Learning...

Appendix: Test Environment
All code and tools in this article were executed in a testing environment consisting of:

• Ubuntu 16.04.7 LTS (Xenial Xerus), x86_64 GNU/Linux)
  • gcc
  • nasm
  • xxd
  • ausyscall