We are given 3 files in this challenge:

  • reverse – This is the main binary and it implements a VM as the challenge name suggests.
  • encrypt – This file contains the code that is executed by the VM.
  • data.enc – This is the encrypted data which we need to decrypt to get the flag

From the initial analysis we can see the main function which is a bit weird, since some functions are not recognised and some paths look unreachable at first glance.

With help the vm jump table at 0x13D0, I fixed the function manually and was able to recover the functions. We can see that VM only supports 15 instructions.

The vm reads 2 bytes from the code that was read from “encrypt” and then calculates which function to execute. For each of these instruction the vm jumps to the corresponding addresses.

0x0 - 0xfc0 
0x1 - 0xf20
0x2 - 0xed0
0x3 - 0xe08
0x4 - 0x10c5
0x5 - 0xd98
0x6 - 0xd20
0x7 - 0xcb0
0x8 - 0xc10
0x9 - 0x10c5
0xa - 0xbc0
0xb - 0x10c5
0xc - 0x10c5
0xd - 0x10c5
0xe - 0x10c5
0xf - 0xb20

The functions that i focused on are:

  • 0xa (0xbc0) – This function reads 64 bytes from “encrypt” after the 2 bytye instruction and stores them to address contained in rax.
  • 0x7 (0xcb0) – This function accepts 2 arguments both of which contain 64 bytes and then it xor’s both of them and stores the result.
  • 0x8 (0xc10) – This performs some kind of rolling OR on the 64 byte string which is passed to it and shifted the places of all the bytes.
  • 0x2 (0xed0) – This instruction reduces the VM instruction pointer thus helping to implement loops.

0x4, 0x9, 0xb, 0xc, 0xd, 0xe are exit and 0xf creates the encrypted file.

encrypt

After some static/dynamic analysis I was able to figure out what the encrypt vm code did. So there are mainly 2 layers of encoding/encryption that protect the encrypted data. First of all 64 byte key is read, then a 64 byte block is read from file to be encrypted. Both of these are xor’ed with each other using 0x7 instruction. After that 0x8 instruction is used to encode the above result and a new key is derived for the next 64 byte block using the 0x8 instruction. Then it loops with help of 0x2 instruction untill the whole file is encrypted.

Here is the implementation of the first function that encrypts the data:

def firstalgo(a, k):
	res = [0] * 64
	for i in range(0, 64):
		res[i] = ord(a[i]) ^ k[i]
	return res

The second function is used as second layer of encoding (rcx = 5, shift = 9) and is also used to derive the key(rcx = 7, shift 0xd) for next block

def secondalgo(arr, rcx, shift):
	res = [0] * 64
	last = arr[63]
	for i in range(0, 64):
		first = arr[i] << rcx
		if i == 0:
			second = (last << rcx) >> 8
		else:
			second = (arr[i - 1] << rcx) >> 8
		res[(i + shift) % 64] = (first | second) % 0x100
	return res

I was able to implement the whole vm in a python script (not posted since it is almost same as decryption script). One of my teamates (@7feilee) reversed the second algo so as to decode the data encoded by it. After it was easy to write a decryption script because we have initial key which was used to encrypt the first block, so from there we can get the first block and then derive the key for 2nd block and repeat the process similarly.

from functools import partial

def rev_second(arr, rcx, shift):
    res = [0] * 64
    last = ord(arr[0])
    for i in range(0, 64):
        first = ord(arr[i]) << rcx
        if i == 64:
            second = (last << rcx) >> 8
        else:
            second = (ord(arr[i - 1]) << rcx) >> 8
        res[(i + shift) % 64] = (first | second) % 0x100
    return res

def firstalgo(a, k):
    res = [0] * 64
    for i in range(0, 64):
        res[i] = a[i] ^ k[i]
    return res

def secondalgo(arr, rcx, shift):
    res = [0] * 64
    last = arr[63]
    for i in range(0, 64):
        first = arr[i] << rcx
        if i == 0:
            second = (last << rcx) >> 8
        else:
            second = (arr[i - 1] << rcx) >> 8
        res[(i + shift) % 64] = (first | second) % 0x100
    return res

key =[0xe1, 0xa9, 0xe1, 0x2e, 0x0b, 0x15, 0x44, 0x9c, 0x08, 0xdc, 0xdc, 0xf3, 0x1a, 0x91, 0x9c, 0x6e, 0x34, 0x5c, 0xe4, 0x5e, 0xf9, 0xe2, 0x5f, 0xf1, 0xf0, 0x86, 0x05, 0xa8, 0x70, 0x6e, 0x04, 0x53, 0x9d, 0x31, 0xec, 0x10, 0xab, 0xea, 0xf6, 0x74, 0x44, 0x79, 0x0f, 0x28, 0x53, 0x40, 0x37, 0x2c, 0x17, 0x9a, 0xc3, 0x67, 0x95, 0x2f, 0x4b, 0x27, 0xd9, 0x3f, 0xf9, 0x1d, 0x2a, 0x70, 0x77, 0x5d]

blocks = []
i = 0
inp = ""
with open('data.enc', 'rb') as openfileobject:
    for chunk in iter(partial(openfileobject.read, 64), b''):
            blocks.append(chunk)

decrypted = []
for block in blocks:
    rcx = 5
    shift = 9
    dat = rev_second(block, 8 - rcx, 63 - shift)
    block = firstalgo(dat, key)
    decrypted.append([chr(x) for x in block])
    firstalgo_res = firstalgo(block, key)
    rcx = 7
    shift = 0xd
    key = secondalgo(key, rcx, shift)
    key = firstalgo(block, key)

f = open("data", "wb")
for i in decrypted:
    for j in i:
        f.write(j)

The script decrypts the encrypted data which turns out to be a PNG.

The flag is VolgaCTF{y0u_ju5t_rever5ed_a_512_b1t_Virtu4l_Mach1nE}

Categories: CTFs