sha256msg1 reverse-engineering challenge (Chatouille - Hackropole)
Hi, in this blog post, I will present my solution to the CTF challenge Chatouille. This challenge is a medium-rated reverse-engineering challenge published at FCSC 2025 and available on hackropole.
main
The main function is quite simple. First, it reserves 128 bytes on the stack (add rsp, 0FFFFFFFFFFFFFF80h) and fill it with 0 using rep stosq. Then, the program calls read(stdin, rsp, 128) to let the user enter some text into this buffer. Finally, some checks are done :
- if the user input length is different from 118 then, the input is incorrect (the
\nis removed from the buffer and therefore from the size of the input). - modify 3 bytes (see the code snippet below) of the buffer and then call
sub_1220(rsp). If the return value is true then the input is incorrect :
mov byte ptr [rsp+118], 80hmov word ptr [rsp+126], 0B003hrsp+118 is the byte just after our input (since our input must be 118 characters so it goes from *rsp to *(rsp+117) and rsp+126 is also after but it’s still in the buffer area allocated at the start.
- calls
sub_1790(rsp), if the return value is true then the input is correct.
It’s not possible to pass the first check (sub_1220) and therefore it’s a waste of time to reverse this function. To keep the write-up short, I am going to talk directly about the second check (sub_1790) but you can get more information about sub_1220 at the end of this post.
sub_1790
This function only contains AVX instructions and is very big (4000 lines) but it’s mostly composed of these two instructions :
vpshufd xmm1, xmm6, 39hsha256msg1 xmm0, xmm6vpshufd a, b, c: shuffle the dwords inbbased on the 8-bit immediatecand store the result ina:- First, split the immediate into 2-bit values. For example, in the function, the immedate is always the same:
0x39 = 0011 1001 = 00 11 10 01. - Then, use each 2-bit value as an index into the xmm register (or memory location)
bto get a dword. In this case, we have :
We can see thata[0] = b[0b01] = b[1] // a[0] means the first dword of xmm register aa[1] = b[0b10] = b[2]a[2] = b[0b11] = b[3]a[3] = b[0b00] = b[0]vpshufdwith an immediate of0x39corresponds to a rotation of 1 to the right of the source register (or memory location).- First, split the immediate into 2-bit values. For example, in the function, the immedate is always the same:
sha256msg1 a, b: this instruction does computation related to the message schedule of SHA-256. I will go into more details about the calculations later. For now, we just need to note that the first register is both a source and a destination register (it gets overwritten with the result of the instruction).
Note : we don’t have to know what the message schedule is to solve this challenge but I wrote some notes so i included them at the end of this post.
Going back to the function, we can see a pattern with sha256msg1 and vpshufd (IDA highlighting helps):
vpshufd xmm1, xmm6, 39h ; '9'sha256msg1 xmm0, xmm6vpshufd xmm2, xmm1, 39h ; '9'sha256msg1 xmm0, xmm1vpshufd xmm3, xmm2, 39h ; '9'sha256msg1 xmm0, xmm2vmovdqu xmm2, xmmword ptr [rdi+30h]vpshufd xmm4, xmm3, 39h ; '9'sha256msg1 xmm0, xmm3vpshufd xmm5, xmm4, 39h ; '9'sha256msg1 xmm0, xmm4vpshufd xmm7, xmm5, 39h ; '9'sha256msg1 xmm0, xmm5vpshufd xmm8, xmm7, 39h ; '9'sha256msg1 xmm0, xmm7sha256msg1 xmm0, xmm8vpshufd xmm9, xmm8, 39h ; '9'- We can see that the
vpushfdare just rotating the same 16 bytes and initialising registers forsha256msg1. - For 8 consecutives
sha256msg1, the first source register is always the same.
Basically, we can rewrote the previous code as:
vpshufd xmm1, xmm6, 39h ; '9'vpshufd xmm2, xmm1, 39h ; '9'vpshufd xmm3, xmm2, 39h ; '9'// vmovdqu xmm2, xmmword ptr [rdi+30h]vpshufd xmm4, xmm3, 39h ; '9'vpshufd xmm5, xmm4, 39h ; '9'vpshufd xmm7, xmm5, 39h ; '9'vpshufd xmm8, xmm7, 39h ; '9'vpshufd xmm9, xmm8, 39h ; '9'
sha256msg1 xmm0, xmm6sha256msg1 xmm0, xmm1sha256msg1 xmm0, xmm2sha256msg1 xmm0, xmm3sha256msg1 xmm0, xmm4sha256msg1 xmm0, xmm5sha256msg1 xmm0, xmm7sha256msg1 xmm0, xmm8(there is just one sha256msg1 that must be done after a vpushfd as xmm2 gets a new value at some point but you get the idea).
When this block has finished executing, the next block uses the next 16 bytes of the input as a source for the first sha256msg1 of the block and the result of the previous last sha256msg1 for the rotation.
- What happens when all the input is “consumed” ?
The last read of the input is done inxmm15(vmovdqu xmm15, xmmword ptr [rdi+112]).
After the block that processesxmm15finishes, the next block usesxmm9as the source for the firstsha256msg1:
vpshufd xmm13, xmm15, 39h // as usual, the result of a block is used as the rotate stuffsha256msg1 xmm9, xmm15What is xmm9 at that point ? if we trace back, we end up to the last vpshufd of the first block of the function:
vpshufd xmm9, xmm8, 39h ; '9'Since 1 rotation moves 4 bytes (in a 16 bytes register) then, after 4 rotations, we are back to the original value. xmm9 is the register that receive the xmm9 contains the original content before any rotation. What is this content ? it’s just the first 16 bytes of the input ! (yes, I forgot to mention it, but the first block actually process input[16:31] so the first 16 bytes never got used).
This time, all the buffer has been consumed, let’s look at the next block to see what is used as the source of the first sha256msg1. The block just after uses xmm8 for sha256msg1. At that point, xmm8 is the result of the first block The block just after uses the result of the second block for sha256msg1 and so on…
To summarize what we learned :
- a block is a sequence of 8
sha256msg1on the same register (the result of the first sha is used as the source for the next…). The second source ofsha256msg1is some bytes of the lastsha256msg1of the previous block. - after each 16-byte part of the buffer has been processed (so after 8 blocks since
, the size of the buffer), the source of the first sha256msg1of a blockis the result of the block (with being the first block).
- For how long does it go? We can get this information by counting the number of times
sha256msg1is used in the function :
cat sub_1790.txt | grep sha256msg1 | wc -l2040Since sha256msg1 is used 8 times in a block, there is 255 blocks.
- How this function finishes ? Let’s take a look at the end of the function :
vpcmpeqd xmm0, xmm13, xmm12vpmovmskb eax, xmm0cmp eax, 0FFFFhsetz dlmovzx eax, dlretnIf you remember, we want this function to return true. The only way to get true is to have eax = 0xFFFF, which is only possible if xmm13 == xmm12. Let’s substitute :
xmm12is easy as it is xored with itself just before (vpxor xmm12, xmm12, xmm12) so xmm12 is 0xmm13isxmm14 | xmm4=(xmm11 | xmm8) | (xmm5 | xmm7)= … =
xmm0 | xmm9 | xmm1 | xmm8 | xmm15 | xmm10 | xmm2 | xmm7- xmm7 is the result of the last block (block 254) xored with a constant (
0x27E2993E7DDB9BCE388260CED6DF027E) - xmm8 is xmm12 xored with a constant and xmm12 is the result of the 253th block)
- xmm2 is a xor between the result of the block 252 and a constant
- xmm1 is a xor between the result of the block 251 and a constant
- xmm10 : result of block 250 xored with a constant
- xmm15 : result of block 249 xored with a constant
- xmm0 : result of block 248 xored with a constant
- xmm9 : result of block 247 xored with a constant
- xmm7 is the result of the last block (block 254) xored with a constant (
xmm0 | xmm9 | xmm1 | xmm8 | xmm15 | xmm10 | xmm2 | xmm7 = 0
The only solution to this equation is that all those registers must be 0.
We know that
From these known values, can we get the values of previous blocks?
Let’s take a look at sha256msg1 more closely. For
sha256msg1 computes:
What do we know?
: we know of the sha256msg1of the last 8 blocks: it’s either the result of the block or the result of the previous sha256msg1. We don’t know that and this is what we try to retrieve !is the last dword of which is rotated after each sha256msg1. We can think that we don’t know this value but we know it!is the last dword of , where does come from ? is the result of the previous block, do we know the result of the previous block ? yes, we know the result of the last 8 blocks.
With
Basically, we retrieve the main source of a sha256msg1 (sha256msg1 of a block. At the end, the last inversion gives the original source, which is the result of the block
In summary, by knowing the result of 2 consecutives blocks, one can invert the last sha256msg1 within the second block which makes it possible to invert all the sha256msg1 within one block. The inversion of a block gives the source entry of the block, the result of the block
Additional information
sub_1220
This function only contains AVX instructions which my CPU doesn’t support (Program received signal SIGILL, Illegal instruction.). The function starts by moving some constants from .rodata to xmm registers using vmovdqa. We can also see that our input is moved into xmm registers (vmovdqu xmm5, xmmword ptr [rdi]). By skimming over the instructions, we can see that this function has no loop, only one condition at the end and the only interesting instructions are instructions related to sha256. The first sha256 instruction is this one:
sha256rnds2 xmm1, xmm14, xmm0The instruction performs two round of sha256. At some point, someone reminded me that emulator existed so I could just run the program under qemu-x86-64 with a gdbserver and attach to it with gdb to get the value of xmm1, xmm14 and xmm0… I didn’t think of it at the start and I already wrote how to compute xmm14 so might as well put it in this writeup. xmm14 is the result of this instruction :
vpalignr xmm14, xmm0, xmm13, 8vpalignr a, b, c, d: concatenatebandc, shift the result ofdbytes then save the right-aligned bytes ina:- Example :
- 0x3C6EF372A54FF53A6A09E667BB67AE85 : 0x 510E527F9B05688C1F83D9AB5BE0CD19 (concatenate xmm0 and xmm13, they are initialised with
.rodataconstant at the start) - shift by 8 bytes (or 64 bits) : 0x3C6EF372A54FF53A6A09E667BB67AE85 510E527F9B05688C
- save 16 byte from the right most bit :
xmm14 = 0x6A09E667BB67AE85510E527F9B05688C
- 0x3C6EF372A54FF53A6A09E667BB67AE85 : 0x 510E527F9B05688C1F83D9AB5BE0CD19 (concatenate xmm0 and xmm13, they are initialised with
- Example :
We confirm our result with gdb :
(gdb) x /i $pc=> 0x555555557276: sha256rnds2 xmm1,xmm14,xmm0(gdb) p /x $xmm14.uint128$4 = 0x6a09e667bb67ae85510e527f9b05688cAnyway, going back to our sha256 instruction, here is what the doc says :
The SHA256RNDS2 instruction performs 2 rounds of SHA256 operation using an initial SHA256 state (C,D,G,H) from the first operand, an initial SHA256 state (A,B,E,F) from the second operand, and a pre-computed sum of the next 2 round message dwords and the corresponding round constants from the implicit operand xmm0When hashing a message with SHA, one preprocessing step is to set the “initial hash value”

Dumping the value of xmm1 and xmm14 :
{0x5be0cd19, 0x1f83d9ab, 0xa54ff53a, 0x3c6ef372}{0x9b05688c, 0x510e527f, 0xbb67ae85, 0x6a09e667}or
which is correct according to the documentaton : “using an initial SHA256 state (C,D,G,H) from the first operand, an initial SHA256 state (A,B,E,F) from the second operand”.
At that point, I made the assumption that this function correctly hash the input and since it doesn’t seem to touch the original buffer, I decided to leave this function and just go to the next one.
The only problem is that we don’t know how to pass this check so we can’t debug the second function.
sha256msg1: SHA256 message schedule
to understand this instruction, we have to understand a little bit how sha256 works. After initialising

We can see that sha256msg1 instruction is used to compute

The first register is viewed a set of 4 dwords (32-bit values)
A second instruction, sha256msg2 must be used to finish the computation (with