(i'm in the middle of writing a blog post about all the ways that standard c is incompatible with c as it exists in the real world, and this section ended up getting pretty big (and arguably off-topic), so i decided to split it off into its own post. enjoy :3)
c99 specifies the macro math_errhandling, which, depending on which methods of error reporting are supported by the math.h functions, expands to either MATH_ERRNO (1), MATH_ERREXCEPT (2), or the bitwise OR of both. in a freestanding implementation it's also allowed to be 0.
math_error(7) has this to say about support for math_errhandling in glibc:
Themath_errhandlingidentifier specified by C99 and POSIX.1 is not supported by glibc. This identifier is supposed to indicate which of the two error-notification mechanisms (errno, exceptions retrievable viafetestexcept(3)) is in use. The standards require that at least one be in use, but permit both to be available. The current (glibc 2.8) situation under glibc is messy. Most (but not all) functions raise exceptions on errors. Some also seterrno. A few functions seterrno, but don't raise an exception. A very few functions do neither.
this sounds pretty bad right? but here's the thing: glibc 2.8 was released in april 2008. the commit which added that paragraph to the man page is from july 2008. this paragraph was written when i was 4 years old and hasn't been updated since, so it's now just straight up misinformation.
from what i can tell, modern glibc supports both errno and floating-point exceptions in all applicable math.h functions (though admittedly i haven't checked through every single function to confirm this). from combing through the git log (and the man pages of individual functions), it seems like work on this mostly started with the release of glibc 2.10 (may 2009), and a definition for math_errhandling was added in 2.11 (october 2009).
but just because modern glibc gets math_errhandling right doesn't mean you can always reliably use it in portable programs: musl always defines math_errhandling to MATH_ERREXCEPT, regardless of the presence of compiler flags like -ffast-math:
#define MATH_ERRNO 1
#define MATH_ERREXCEPT 2
#define math_errhandling 2(musl never sets errno in any of its math functions, interestingly)
-ffast-math is non-conforming anyway, so there's an argument to be made that this is conforming behavior, because setting math_errhandling to 0 (as glibc does when -ffast-math is used) is technically disallowed by the standard. but i think glibc's behavior is much more useful than just straight up lying.
one more note: the complex.h functions don't report errors at all. this is kinda funny because of the implications it has on tgmath.h: whether or not the macros report errors is dependent on the type of their argument. for example, pow will only report an error on overflow if its argument is non-complex. this doesn't really matter that much but i thought it was kinda funny.
IMO the only portable and "Correct" way to check for errors in libc math functions is to check the inputs beforehand for domain errors (test for NaNs and infinities), and to check for outputs like HUGE_VAL if overflow is a possibility (checking for underflow in this way is more complicated and dependent on the specific function, but i'm pretty sure it's always possible by checking for specific input/output combinations, e.g. base != 0.0 && result == 0.0 for pow). although you can rely on errno or floating-point exceptions in modern non-buggy implementations, neither approach is guaranteed to be supported, so you'll need to support both, and that's just cumbersome (and a pain in the ass to test), so it's really just not worth it.
(floating-point exceptions in c are their own can of worms, which i'll write much more about in the future.)