Application is occasionally aborted when runs as a Windows service during boot

[ancwrd1]

Problem:

I have a problem with the aws-lc-rs/aws-lc which causes the application that runs as a Windows service to abort during early boot (error code 0x40000015, the abort() call).

This does not happen always, but is reproducible, for example on Windows 11 virtual machine, 25H2, build 26200.8037.

From the crash dump the following stack trace is produced:

 	myapp.exe!abort() Line 77	C++
>	myapp.exe!tree_jitter_initialize_once() Line 274	C
 	myapp.exe!call_once_init(_RTL_RUN_ONCE * once, void * arg, void * * out) Line 39	C
 	[External Code]	
 	myapp.exe!aws_lc_0_38_0_CRYPTO_once(_RTL_RUN_ONCE * once, void(*)() init) Line 43	C
 	myapp.exe!aws_lc_0_38_0_tree_jitter_initialize(entropy_source_t * entropy_source) Line 305	C
 	myapp.exe!aws_lc_0_38_0_get_entropy_source() Line 139	C
 	myapp.exe!rand_state_initialize(rand_thread_local_state * state) Line 323	C
 	myapp.exe!rand_bytes_impl(<unnamed-tag> tls_key, unsigned char * out, unsigned __int64 out_len, const unsigned char * user_pred_resistance, int use_user_pred_resistance) Line 502	C
 	myapp.exe!aws_lc_0_38_0_RAND_bytes(unsigned char * out, unsigned __int64 out_len) Line 529	C
 	myapp.exe!aws_lc_rs::rand::fill(ref_mut$<slice2$<u8>>) Line 179	Unknown
 	myapp.exe!aws_lc_rs::rand::impl$5::fill_impl(aws_lc_rs::rand::SystemRandom * self, ref_mut$<slice2$<u8>>) Line 168	Unknown
 	myapp.exe!aws_lc_rs::rand::impl$0::fill<aws_lc_rs::rand::SystemRandom>(aws_lc_rs::rand::SystemRandom * self, ref_mut$<slice2$<u8>>) Line 63	Unknown

[justsmth]

Hello! Thanks for the report and stack trace. That was really helpful.

Based on the stack trace, the crash appears to be in tree_jitter_get_root_seed (possibly inlined into tree_jitter_initialize_once?), where the Jitter Entropy source fails to produce entropy via jent_read_entropy even after retries, ultimately hitting abort(). My suspicion is that during early boot the system's timing characteristics aren't stable enough for the Jitter Entropy library to generate entropy reliably. We have not seen this ourselves in testing, so we can't be certain exactly.

Is it possible you somehow got the jitter entropy code to compile with optimizations?

As a workaround, you can disable CPU Jitter Entropy at build time by setting this environment variable when you build:

AWS_LC_SYS_NO_JITTER_ENTROPY=1

This will fall back to using the OS CSPRNG (BCryptGenRandom) for seeding instead. That should be reliable during early boot and is still sound, though you do lose the second independent entropy source.

We'll also look into whether we can handle this more gracefully on the library side.

[ancwrd1]

Thanks for the reply, the stacktrace is from the build without optimizations, for x86_64-pc-windows-msvc target. I have tried it also for x86_64-pc-windows-gnu with the same result, both with and without optimizations.
The workaround can certainly work, yes, thanks for the suggestion. I think it should be handled in the library though, to prevent unexpected app termination. May be fallback to BCryptGenRandom automatically if jent_read_entropy fails.