InCTF 2018 also consisted of an individual CTF apart from the main Attack and Defense CTF. There were 5 total challenges in this CTF. So in this post i am going to discuss the “vm” challenge that I solved during the CTF. There were multiple approaches that could have been used for this challenge since the source code of the vm was given. During the CTF i didn’t use this approach and i directly went after the flag by ignoring the VM since it was a very simple VM as source code was given but I decided to solve the challenge again and this time without using the source code because I usually have some trouble solving complex VMs (that 10th flare-on level this year with a kvm was a pain) and this looks like a good challenge to learn something new.

What is a VM?

In the context of this challenge, VM(Virtual Machine) is an application/machine that reads and executes the custom bytecode. We need to write a parser/disassembler so as to convert this bytecode to readable format/pseudo assembly.


We are given 3 files in this challenge – net.bsd, vm, vm.c. We are not gonna look at the source in this method and will try to reverse engineer the VM. Also, “net.bsd” contains the custom bytecode which is executed by the VM. Taking a look at the .bss section it is pretty clear that there are 4 virtual registers. Analyzing the Switch Jump Table at 0x2024 and the different functions (handle_….) it is clear that there are 7 different virtual opcodes in the VM and there is a default function which is called when VM encounters an invalid opcode. Now onto the main function, the vm opens net.bsd files and reads it line by line. Each line is converted to integer and then different parts of the instruction are extracted from this integer.

Code Snippet of VM converting the integer to the different parts of the virtual instruction:

Some important points we can derive from our knowledge till now:

  • Each line in net.bsd contains an integer which is actually an virtual instruction.
  • Different parts of the integer are different parts of the instruction.

With the further analysis of the above code snippet we can see that the integer is divided into various parts:

The last byte that is 24th to 32nd bit of the integer are saved separately and we will refer this as 1st part further in the post. The bits from 20th to 24th bit form the 2nd part. The bits from 16th to 20th bit form the 3rd part and 12th to 16th bit form the 4th part of the instruction. 0 to 16 bit are also saved and lets say this is the 5th part of the instruction. Now lets see what these parts are and where they are used in the VM.

So now we have almost reached to the core of the VM. The 1st part that was found earlier is now compared with 6 and if its greater then handle_default is called and VM exits. So it is the 1st part which decides which function will be called and rest all the parts are actually the custom instructions or the parameters to the functions. Now since we know how the functions are called from the integer lets take a look at all the 7 functions that is the 7 opcodes supported by the VM. We will analyze all the 7 functions and will try to parse the code at last.


Analyzing all the functions will help us write a parser for this vm :

handle_add – On analysis we can see that this function expects three parameters, edi and esi contain first 2 parameters which are actually the 2nd part (20th to 24th bit) and 3rd part (16th to 20th bit) of the integer which we got in the earlier part. The third parameter is actually the 16 bit integer (5th part). If 2nd or 3rd parameter are greater than 3 then again the VM exits. And this made me remember that there are just 4 virtual registers which are stored in continuous memory as array. First two parameters are actually the indexes of 2 registers which can be further confirmed by looking at the code.

So what happens in above code is the value in the virtual register at the index (2nd param) is added to the 3rd param and is stored in vreg (1st param). So its kind of like reg[1st param] = reg[2nd param] + 3rd param.


handle_xor – This function also has 3 parameters but this time the 2nd, 3rd and 4th parts of the integer are passed as parameters. what happens in this function is the register at first param is set equal to the xor of the data at 2nd and 3rd register.


handle_out – This function has a single parameter. It expects the index of the virtual register whose data has to be printed. Only a single character is printed at a time.

handle_in This function reads a value into a virtual register. Only a single character is read at a time.

handle_mov – This function moves a 16 bit value to a register. The index of the register and the value are passed as parameter.

handle_chk – This function checks if 2 virtual registers are equal or not.

handle_hlt – Exits the VM.

This was just some brief info about the VM functions, Now as to parse the custom bytecode set we would need to convert the custom bytecode to the corresponding opcodes. So here is the format of the different opcodes which we will get after parsing the code.

  • ADD R1, R2, val – Sets R1 equal to sum of R2 and val.
  • XOR R1, R2, R3 – Sets R1 equal to xor of R2 and R3.
  • IN R1 Reads a character from the stdin to a virtual register R1.
  • OUT R1 Prints data from virtual register R1 to stdout.
  • MOV R1, val – Moves val to R1.
  • CHK R1, R2 – Checks if R1 is equal to R2 or not, Sets flag = 1 if they are unequal.
  • HLT – VM exit

So from the knowledge of all the above opcodes and their working it is pretty easy to write a parser, so here is the parser i wrote for this VM :

Code executed by the VM

By using the above script we get the code executed by the VM:

Since the VM only supports 7 virtual opcodes and 4 virtual registers and in the above instructions the ADD opcode and 4th register are not even used, So the analysis of the code is really easy. The comments in the parsed code can help us to understand what the VM is doing.

The flag that we get from code is – inctf{1nC7F_B3St}

This is pretty good challenge to play with if you want to get started with VMs.

Categories: CTFsWriteups