Hi, in this blog post, I will present my solution to the CTF challenge Chatouille. This challenge is a medium-rated reverse-engineering challenge published at FCSC 2025 and available on hackropole.

main

The main function is quite simple. First, it reserves 128 bytes on the stack (add rsp, 0FFFFFFFFFFFFFF80h) and fill it with 0 using rep stosq. Then, the program calls read(stdin, rsp, 128) to let the user enter some text into this buffer. Finally, some checks are done :

  • if the user input length is different from 118 then, the input is incorrect (the \n is removed from the buffer and therefore from the size of the input).
  • modify 3 bytes (see the code snippet below) of the buffer and then call sub_1220(rsp). If the return value is true then the input is incorrect :
mov byte ptr [rsp+118], 80h
mov word ptr [rsp+126], 0B003h

rsp+118 is the byte just after our input (since our input must be 118 characters so it goes from *rsp to *(rsp+117) and rsp+126 is also after but it’s still in the buffer area allocated at the start.

  • calls sub_1790(rsp), if the return value is true then the input is correct.

It’s not possible to pass the first check (sub_1220) and therefore it’s a waste of time to reverse this function. To keep the write-up short, I am going to talk directly about the second check (sub_1790) but you can get more information about sub_1220 at the end of this post.

sub_1790

This function only contains AVX instructions and is very big (4000 lines) but it’s mostly composed of these two instructions :

vpshufd xmm1, xmm6, 39h
sha256msg1 xmm0, xmm6
  • vpshufd a, b, c : shuffle the dwords in b based on the 8-bit immediate c and store the result in a :
    • First, split the immediate into 2-bit values. For example, in the function, the immedate is always the same: 0x39 = 0011 1001 = 00 11 10 01.
    • Then, use each 2-bit value as an index into the xmm register (or memory location) b to get a dword. In this case, we have :
    a[0] = b[0b01] = b[1] // a[0] means the first dword of xmm register a
    a[1] = b[0b10] = b[2]
    a[2] = b[0b11] = b[3]
    a[3] = b[0b00] = b[0]
    We can see that vpshufd with an immediate of 0x39 corresponds to a rotation of 1 to the right of the source register (or memory location).
  • sha256msg1 a, b : this instruction does computation related to the message schedule of SHA-256. I will go into more details about the calculations later. For now, we just need to note that the first register is both a source and a destination register (it gets overwritten with the result of the instruction).

Note : we don’t have to know what the message schedule is to solve this challenge but I wrote some notes so i included them at the end of this post.

Going back to the function, we can see a pattern with sha256msg1 and vpshufd (IDA highlighting helps):

vpshufd xmm1, xmm6, 39h ; '9'
sha256msg1 xmm0, xmm6
vpshufd xmm2, xmm1, 39h ; '9'
sha256msg1 xmm0, xmm1
vpshufd xmm3, xmm2, 39h ; '9'
sha256msg1 xmm0, xmm2
vmovdqu xmm2, xmmword ptr [rdi+30h]
vpshufd xmm4, xmm3, 39h ; '9'
sha256msg1 xmm0, xmm3
vpshufd xmm5, xmm4, 39h ; '9'
sha256msg1 xmm0, xmm4
vpshufd xmm7, xmm5, 39h ; '9'
sha256msg1 xmm0, xmm5
vpshufd xmm8, xmm7, 39h ; '9'
sha256msg1 xmm0, xmm7
sha256msg1 xmm0, xmm8
vpshufd xmm9, xmm8, 39h ; '9'
  1. We can see that the vpushfd are just rotating the same 16 bytes and initialising registers for sha256msg1.
  2. For 8 consecutives sha256msg1, the first source register is always the same.

Basically, we can rewrote the previous code as:

vpshufd xmm1, xmm6, 39h ; '9'
vpshufd xmm2, xmm1, 39h ; '9'
vpshufd xmm3, xmm2, 39h ; '9'
// vmovdqu xmm2, xmmword ptr [rdi+30h]
vpshufd xmm4, xmm3, 39h ; '9'
vpshufd xmm5, xmm4, 39h ; '9'
vpshufd xmm7, xmm5, 39h ; '9'
vpshufd xmm8, xmm7, 39h ; '9'
vpshufd xmm9, xmm8, 39h ; '9'
sha256msg1 xmm0, xmm6
sha256msg1 xmm0, xmm1
sha256msg1 xmm0, xmm2
sha256msg1 xmm0, xmm3
sha256msg1 xmm0, xmm4
sha256msg1 xmm0, xmm5
sha256msg1 xmm0, xmm7
sha256msg1 xmm0, xmm8

(there is just one sha256msg1 that must be done after a vpushfd as xmm2 gets a new value at some point but you get the idea).

When this block has finished executing, the next block uses the next 16 bytes of the input as a source for the first sha256msg1 of the block and the result of the previous last sha256msg1 for the rotation.

  1. What happens when all the input is “consumed” ?
    The last read of the input is done in xmm15 (vmovdqu xmm15, xmmword ptr [rdi+112]).
    After the block that processes xmm15 finishes, the next block uses xmm9 as the source for the first sha256msg1:
vpshufd xmm13, xmm15, 39h // as usual, the result of a block is used as the rotate stuff
sha256msg1 xmm9, xmm15

What is xmm9 at that point ? if we trace back, we end up to the last vpshufd of the first block of the function:

vpshufd xmm9, xmm8, 39h ; '9'

Since 1 rotation moves 4 bytes (in a 16 bytes register) then, after 4 rotations, we are back to the original value. xmm9 is the register that receive the rotation so it’s the same idea, xmm9 contains the original content before any rotation. What is this content ? it’s just the first 16 bytes of the input ! (yes, I forgot to mention it, but the first block actually process input[16:31] so the first 16 bytes never got used).

This time, all the buffer has been consumed, let’s look at the next block to see what is used as the source of the first sha256msg1. The block just after uses xmm8 for sha256msg1. At that point, xmm8 is the result of the first block The block just after uses the result of the second block for sha256msg1 and so on…

To summarize what we learned :

  • a block is a sequence of 8 sha256msg1 on the same register (the result of the first sha is used as the source for the next…). The second source of sha256msg1 is some bytes of the last sha256msg1 of the previous block.
  • after each 16-byte part of the buffer has been processed (so after 8 blocks since , the size of the buffer), the source of the first sha256msg1 of a block is the result of the block (with being the first block).
  1. For how long does it go? We can get this information by counting the number of times sha256msg1 is used in the function :
cat sub_1790.txt | grep sha256msg1 | wc -l
2040

Since sha256msg1 is used 8 times in a block, there is 255 blocks.

  1. How this function finishes ? Let’s take a look at the end of the function :
vpcmpeqd xmm0, xmm13, xmm12
vpmovmskb eax, xmm0
cmp eax, 0FFFFh
setz dl
movzx eax, dl
retn

If you remember, we want this function to return true. The only way to get true is to have eax = 0xFFFF, which is only possible if xmm13 == xmm12. Let’s substitute :

  • xmm12 is easy as it is xored with itself just before (vpxor xmm12, xmm12, xmm12) so xmm12 is 0
  • xmm13 is xmm14 | xmm4 = (xmm11 | xmm8) | (xmm5 | xmm7) = … =
    xmm0 | xmm9 | xmm1 | xmm8 | xmm15 | xmm10 | xmm2 | xmm7
    • xmm7 is the result of the last block (block 254) xored with a constant (0x27E2993E7DDB9BCE388260CED6DF027E)
    • xmm8 is xmm12 xored with a constant and xmm12 is the result of the 253th block)
    • xmm2 is a xor between the result of the block 252 and a constant
    • xmm1 is a xor between the result of the block 251 and a constant
    • xmm10 : result of block 250 xored with a constant
    • xmm15 : result of block 249 xored with a constant
    • xmm0 : result of block 248 xored with a constant
    • xmm9 : result of block 247 xored with a constant

xmm0 | xmm9 | xmm1 | xmm8 | xmm15 | xmm10 | xmm2 | xmm7 = 0
The only solution to this equation is that all those registers must be 0.
We know that means that and since each register is a xor between the result of a block and a constant then we know what values the last 8 blocks must have to pass the check.

From these known values, can we get the values of previous blocks?
Let’s take a look at sha256msg1 more closely. For
and , sha256msg1 computes:

What do we know?

  • : we know of the sha256msg1 of the last 8 blocks
  • : it’s either the result of the block or the result of the previous sha256msg1. We don’t know that and this is what we try to retrieve !
  • is the last dword of which is rotated after each sha256msg1. We can think that we don’t know this value but we know it! is the last dword of , where does come from ? is the result of the previous block, do we know the result of the previous block ? yes, we know the result of the last 8 blocks.

With , we can get easily :

Basically, we retrieve the main source of a sha256msg1 (). We can use the same method to invert the 8 sha256msg1 of a block. At the end, the last inversion gives the original source, which is the result of the block .

In summary, by knowing the result of 2 consecutives blocks, one can invert the last sha256msg1 within the second block which makes it possible to invert all the sha256msg1 within one block. The inversion of a block gives the source entry of the block, the result of the block . Thus, by knowing the result of 8 consecutives blocks, it’s possible to retrieve the source of these 8 blocks, which after 255 inversions give us the first source (the input or the flag).

Additional information

sub_1220

This function only contains AVX instructions which my CPU doesn’t support (Program received signal SIGILL, Illegal instruction.). The function starts by moving some constants from .rodata to xmm registers using vmovdqa. We can also see that our input is moved into xmm registers (vmovdqu xmm5, xmmword ptr [rdi]). By skimming over the instructions, we can see that this function has no loop, only one condition at the end and the only interesting instructions are instructions related to sha256. The first sha256 instruction is this one:

sha256rnds2 xmm1, xmm14, xmm0

The instruction performs two round of sha256. At some point, someone reminded me that emulator existed so I could just run the program under qemu-x86-64 with a gdbserver and attach to it with gdb to get the value of xmm1, xmm14 and xmm0… I didn’t think of it at the start and I already wrote how to compute xmm14 so might as well put it in this writeup. xmm14 is the result of this instruction :

vpalignr xmm14, xmm0, xmm13, 8
  • vpalignr a, b, c, d : concatenate b and c, shift the result of d bytes then save the right-aligned bytes in a :
    • Example :
      • 0x3C6EF372A54FF53A6A09E667BB67AE85 : 0x 510E527F9B05688C1F83D9AB5BE0CD19 (concatenate xmm0 and xmm13, they are initialised with .rodata constant at the start)
      • shift by 8 bytes (or 64 bits) : 0x3C6EF372A54FF53A6A09E667BB67AE85 510E527F9B05688C
      • save 16 byte from the right most bit :
        xmm14 = 0x6A09E667BB67AE85510E527F9B05688C

We confirm our result with gdb :

(gdb) x /i $pc
=> 0x555555557276: sha256rnds2 xmm1,xmm14,xmm0
(gdb) p /x $xmm14.uint128
$4 = 0x6a09e667bb67ae85510e527f9b05688c

Anyway, going back to our sha256 instruction, here is what the doc says :

The SHA256RNDS2 instruction performs 2 rounds of SHA256 operation using an initial SHA256 state (C,D,G,H) from the first operand, an initial SHA256 state (A,B,E,F) from the second operand, and a pre-computed sum of the next 2 round message dwords and the corresponding round constants from the implicit operand xmm0

When hashing a message with SHA, one preprocessing step is to set the “initial hash value” (see section 5.3). For SHA256 :
h_0 sha256 values

Dumping the value of xmm1 and xmm14 :

{0x5be0cd19, 0x1f83d9ab, 0xa54ff53a, 0x3c6ef372}
{0x9b05688c, 0x510e527f, 0xbb67ae85, 0x6a09e667}

or


which is correct according to the documentaton : “using an initial SHA256 state (C,D,G,H) from the first operand, an initial SHA256 state (A,B,E,F) from the second operand”.

At that point, I made the assumption that this function correctly hash the input and since it doesn’t seem to touch the original buffer, I decided to leave this function and just go to the next one.
The only problem is that we don’t know how to pass this check so we can’t debug the second function.

sha256msg1: SHA256 message schedule

to understand this instruction, we have to understand a little bit how sha256 works. After initialising , the message , is split into multiple blocks of 512 bits. (a padding is added to the message if necessary). Then, for each block, a message schedule is computed. The message schedule is just a set of 64, 32-bit values:
sha256 message schedule
means the word (starting from the left) of the block of the message

We can see that … is just the original block. The sha256msg1 instruction is used to compute for 4 values of :
sha256msg1
The first register is viewed a set of 4 dwords (32-bit values)
and only the last dword of the second register is used . Then the computation is done and stored in the first register (see image above).

A second instruction, sha256msg2 must be used to finish the computation (with ) but it’s never used in the function.