Rendered at 16:03:40 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
tonnydourado 56 minutes ago [-]
Am I missing something or is this written up to seem a lot worse than it actually is?
It starts with a bold claim: "ssh accounts that should not accept a password will accept any password between 256 and 1024 characters". But then it keeps adding qualifiers:
- sshd compiled with musl
- not using PAM
- password logins enabled
- at least one account with '*' as the password "hash"
- at least one OTHER account with a valid password
- server is aligned with 2 ley lines
- hacker is wearing a blue t-shirt and green socks
Ok, the last two are an exaggeration, but still. The conditions are all ANDed, miss a single one and you're fine.
On top of that, as someone else in these comments reminded, the target account needs to have a valid login shell or other exploitable program assigned to it, and these passwordless accounts tend to have nologin or something similar.
To me this seems to fall on the "technically possible, practically very unlikely" bucket, kinda like the PHP vulnerability from the other day. Maybe this has gone unnoticed for so long because it just doesn't happen.
pizlonator 45 minutes ago [-]
There are a lot of systems that meet those requirements.
> - sshd compiled with musl
So alpine as well as a bunch of other embedded Linuxes. Huge install base.
> - not using PAM
What you get if you try to have a minimal setup. Default sshd config if you install from source.
> - password logins enabled
The default. Super common.
> - at least one account with '*' as the password "hash"
Almost every Linux system.
> - at least one OTHER account with a valid password
Almost every Linux system.
So sure, there are caveats, but they cover a massive install base.
This is a super bad bug
tonnydourado 34 minutes ago [-]
How many alpine installs are running sshd? Most containers using it as a base image aren't. Embedded Linux is unlikely to be running it, too. And if one IS running sshd, how often is it with unchanged defaults? I don't buy the inference here.
I'll concede that embedded linux might be my blindspot here. Maybe cheap web hosting, too, but that seems much less likely to be using musl.
svnt 32 minutes ago [-]
Embedded linux is your blindspot here.
tchebb 4 minutes ago [-]
Most embedded systems I've seen use glibc or uClibc. I agree it would make sense for them to use musl, but the embedded space moves very slowly.
pizlonator 29 minutes ago [-]
Yes
andix 32 minutes ago [-]
A lot of embedded systems use musl and no PAM. Some of them might use Alpine, but the distribution doesn't matter that much.
3m4r 9 minutes ago [-]
Most of those would be using dropbear rather than openssh.
tonnydourado 29 minutes ago [-]
And ssh? I'd assume most would use proprietary or industry-specific standard for data IO, or just https, for domestic IoT stuff
andix 27 minutes ago [-]
And also ssh in many cases.
pera 10 minutes ago [-]
I thought Dropbear was pretty much the defacto SSH server for embedded applications, is OpenSSH+musl actually a common combination for these scenarios?
vadansky 32 minutes ago [-]
> > - password logins enabled
> The default. Super common.
This always seems bonkers to me. I think SSHD should give you ONE password free connection of setup to copy your key over and then it just shuts the door behind it.
hnfong 27 minutes ago [-]
I think it has to be a time limited FREE offer. (like, seriously, what if the user doesn't take the offer?)
It might actually a good idea, but I would likely avoid it since my built-in Bayesian filter would just mark it as spam.
hnfong 40 minutes ago [-]
Alpine Linux (which uses musl) images are popularly deployed by those who use docker, and apparently the issue is there on the default install.
Not sure whether the PHP vulnerability actually had been exploited, but I'm pretty sure there are literally thousands of machines out there affected by this ssh issue right now as we speak.
This isn't going to end well.
---
Addendum (which may or may not be related to the above) --
Everyone has heard of the tale of the boy who cried wolf. The main moral of the story is not to troll your fellow townsfolks so they don't distrust you. But everyone with a sane mind gets that.
The (IMHO) under-appreciated lesson is actually: don't be those villagers. Believe stories you are given, even if you might look like a fool for believing in them. Don't assume what happens is the same as what happened in the past. Look at the facts yourself. Trust and verify.
VorpalWay 30 minutes ago [-]
You don't typically run sshd in containers, instead you only run whatever single service you set up your container for.
Sure there are native Alpine installs too, but that is far less common of a use case.
tonnydourado 32 minutes ago [-]
Who's out there running sshd inside a docker container? It's almost MORE work than exec-ing into the container.
andix 33 minutes ago [-]
Alpine in containers is usually not affected. Most images/containers haven't ssh/sshd installed at all. Remote access is usually done from the host system.
tcdent 4 minutes ago [-]
Really, "password auth enabled" should be enough to be a non starter these days.
itemize123 37 minutes ago [-]
only 1) is a narrowing filter. the rest is basically the majority.
VorpalWay 29 minutes ago [-]
Most linux distros use PAM, so 2 is also a narrowing filter. Debian, Ubuntu, Fedora, Arch, etc all use PAM.
mpyne 2 hours ago [-]
OK, I think I see what’s going on with this, in this configuration.
The goal from an attacker’s perspective is to get this to be true: `$ret = crypt($pw, $hash); $ret == $hash` when called by OpenSSH’s sshd. This is how sshd validates that the user logging in knew the account password.
The `$hash` is the “settings” to the crypt call and will normally look like “$6$--gibberish” for users with passwords set, but could look like “*” or “!” for users not intended to be able to login.
The bug is from the intersection of all of these things:
1. musl can return “*” for errors, rather than leave the return value NULL. Other libc have adopted something like this too, but seem to use a different result “*0”.
2. musl can also return “+” for errors, but only does so if $hash starts with “*”. This is intended to eliminate the possibility of an error causing `$ret == $hash` to be true, however it assumes that applications calling `crypt()` will set $pw and $hash to values corresponding to the `$ret == $hash` comparison to be made immediately afterwards.
3. OpenSSH can be convinced to pass the hash of a different user if the hash is “*” or “!” to avoid timing attacks that can enumerate the user list, so it will set $pw and $hash to correspond to different users. $hash will be a real “$6$--gibberish” style hash in this case, and OpenSSH expects `crypt()` to either return NULL or return a hash string. But now there’s two hashes OpenSSH uses. It does: `$ret = crypt($pw, $hash2); $ret == $hash1`;
4. This should normally not be a problem because OpenSSH will pass in valid $pw and $hash2 that should not lead to errors in hashing, so the $ret that musl generates should be a real hash that cannot match the “*” or “!” $hash1 for the login-disabled user. However as a DDoS mitigation, musl will return error if the password is too long when using SHA-2-based hashes (the default, it seems).
5. This error will result in a non-NULL result of “*” or “+”. Since OpenSSH passed in a real hash ($hash2), musl will return “*”, which compares equal to the actual user hash $hash1 for the disabled-login user.
The OpenSSL adjustment ensures a minimum hash length for the user before logging them in, even if crypt() succeeds somehow.
chuckadams 44 minutes ago [-]
I think the conclusion that can be drawn from this is that the behavior of crypt() is howling-at-the-moon insane, and it should no longer be considered for any system built in this century, at least not without extensive wrappers that correct and normalize all these edge cases.
byronsharman 15 minutes ago [-]
Thank you for explaining it with proper capitalization. I understood your explanation much better than the author's.
idlewords 2 hours ago [-]
This is a vastly clearer explanation than the linked page. Thank you!
alexgoodhart 1 hours ago [-]
Because Claude wrote the linked page. Claude is impressively bad at communicating these days. with gpt-5.6+ approaching Claude level like 30% of time and Gemini much clearer but also the most annoying 7 year old ever, it’s a pretty lame environment especially for people who r new to everything
endymi0n 44 minutes ago [-]
Until Opus 5.0 I'd have agreed with you, but Opus 5.5 is honestly one of the best I've ever sparred with, vastly better than the rest, including Astra imho. Especially so when prompting for clear top-down, first-principles communication.
ericbarrett 7 minutes ago [-]
Disagree, I review a lot of Opus 5.5 code and its comments and docs are still pretty inane and Claudish. Is it better than the 4.x and 5.0 era, sure. But it's still nouning the verbs, using colons everywhere, a-not-b, "a foos the b" etc. A lot of lingual knots that need a human hand to untie.
yearolinuxdsktp 25 minutes ago [-]
Opus 5.5 is much better, but IME I always have to force it to review its writing (Opus 5.5) to get it to put the most important facts first (and not bury the lede). It loves to park the main point far away after intermediate clauses. It always writes “premise, so consequence.” It takes a prompt asking to review against rules in context for it to fix it.
The ‘concise’ output style setting is bad with Opus 5.5, as it will often spit out a cryptic (is this really English?!), allergic-to-commas concise response, which takes more time to parse.
dboreham 10 minutes ago [-]
> musl can return “*” for errors
wtaf ?
BoppreH 3 hours ago [-]
In-band signaling[1] strikes again. * should not have been used as special value, neither for "no password" in /etc/passwd nor "error while hashing the password" in crypt. I'm afraid we'll find many more of these abuses, specially in string manipulations. The null was ok, an exception (or equivalent) better, and Result<T> (or equivalent) best.
PSA: you can typically declare different types of string so that your type system objects to mixing data of different types even though deep down we know they're both strings.
BoppreH 35 minutes ago [-]
If you mean type aliases, that's a good idea within a single codebase, but the distinction between different types of strings doesn't survive serialization so it wouldn't have prevented this error. What they need is a way to signal errors out-of-band, like Result<T> or exceptions.
29 minutes ago [-]
mkl 1 hours ago [-]
Even using "no_password" and "error_while_hashing" would have prevented it. The problem was picking a generic, meaningless signal string.
mort96 48 minutes ago [-]
That would've happened to fix it, because "error_while_hashing" and "no_password" happen to not be valid password hashes. The right way would've been to have different syntax for hashes and non-hashes; you could've made it so that a hash starts with a '#' and if it doesn't start with a '#', the rest of the string should be ignored.
Or, for backwards compatibility, start non-hash strings with a character which can't appear in a hash, such as '@'. Your examples would then become '@no_password' and '@error_while_hashing'.
throw10920 2 hours ago [-]
This is a side-effect of Unix-y "it's all plain text" design that leans extremely heavily upon in-band signalling (among other design decisions).
If /etc/shadow was a JSON (or XML, even, although that would make me feel gross), this bug would have been much easier to see, because a field that reads '"hash": ""' sticks out a lot* more than a custom microformat that you don't understand.
With data structures, there's never "plain text" - it's all structured. The question is whether you want to use existing formats and tools, or create a new one for every tool and literally make footguns like this.
__MatrixMan__ 1 hours ago [-]
A few years back I un-bashed my life as much as possible in favor of nushell (though it's not the only shell to encourage structure). I feel like it has really helped me notice a lot of this-needs-more-structure type problems outside of my shell that I otherwise wouldn't have.
The elegance of the unix-y approach is actually that it's all bits. You can just join things together and shove the bits through, that's a wild simplification compared with having vendor-specific kinds of plugs. But we could have that kind of compatibility while also benefiting from structure.
So I encourage everybody to give a shell with structured output a try, let it rewire your brain, and then ask yourself: how is the OS holding this back from being everything it could be? Because if we build that OS, it won't encourage these problems.
When I imagine it, it would probe either side of a pipe to see if they support negotiation. If not, send the bits through in the same old unixy way. If so, use the newest schema for structured data that is supported by both sides.
jonhohle 40 minutes ago [-]
At Amazon we had RecordStream which implemented most of the standard Unix tools on structured streams. It was pretty easy to write filters, aggregators, and formatted output in something not much more complicated than sed/awk/cut/join/sort pipelines. There was a convenient structured log format for services that was often used as input.
The issue isn't the config, but the fact that crypt() doesn't have any way to signal failure. In better designed languages you would have exceptions, result types, or something. Even in C you have "return integer indicating success or error" which would've prevented the confusion.
Now the public API is hard to change but you could still avoid the issue by returning a longer token like "==DANGEROUS_CRYPT_ERROR_length==" or "==DANGEROUS_CRYPT_ERROR_unsupported_alg==" which would never match the password field unless someone was really trying to induce the issue.
knome 53 minutes ago [-]
>but the fact that crypt() doesn't have any way to signal failure
>Upon successful completion, crypt() shall return a pointer to the encoded string. The first two characters of the returned value shall be those of the salt argument. Otherwise, it shall return a null pointer and set errno to indicate the error.
returning a sentinel value of NULL and setting errno is pretty standard C, and shouldn't cause an error.
OP's sshhole link specifically notes that it _should_ return NULL, but some MUSL devs decided they wanted it to return trash instead, and the trash they returned instead of NULL caused this.
All because some devs were too lazy to check the result and another dev decided to be clever and "fix" NULL errors from crypt by sending back 'unmatchable' trash banking on devs not checking NULL.
edit: apparently crypt implementations have been trash ignoring the NULL return for a long time. this isn't only on MUSL, but it's still because people ignored and tried to work around the existing out-of-band error mechanism.
BoppreH 2 hours ago [-]
No, that's just more in-band signaling!
Some badly designed password hashing algorithms (cough bcrypt cough) have maximum password lengths, ignoring everything after X characters. So it could happen that I set my password to a 100-character string, it gets saved as "==DANGEROUS_CRYPT_ERROR_length==", and then anybody can log in to my account by entering an unrelated long string that returns the same "error".
If a function promises a string result, it should only provide a string if the function succeeded. The null was much better than magic strings.
dns_snek 1 hours ago [-]
I know, it's bad. I agree that it should've stayed NULL. But if you really really really think you have an acceptable reason to do something like this (and you probably don't), the least you can do is use a non-trivial token.
BoppreH 1 hours ago [-]
If I was really desperate and could only return a valid string, I would probably use a message and add a random component to avoid ever matching two errors. Like SQL, where null != null.
UqWBcuFx6NV4r 2 hours ago [-]
At this point I’m resigning myself to think that we’ve just gotta wait for some people to die. It feels very dramatic and glib (no pun intended lol) and whatever to say, but at this point it feels like there’s a certain hostility in some parts to changing anything from the platonic ideals of however computing happened to work in 2004. “Oh we don’t need to mess with it!”
gspr 1 hours ago [-]
I think this is a bit unfair. The way I see it, a lot of people are saying that changing breaks lots of old stuff, and risks complete fragmentation. Those are real costs to change. It might still be worth it, but let's not pretend that everyone is resisting change just because it's change.
abirch 1 hours ago [-]
The costs to change are dropping significantly with AI.
I'd say another UNIX-y thing - "C is good enough" - is part of the problem. C makes programming painful which results in all sorts of hacks.
E.g. in Pascal you can can define a record and then defined file of record to have typed I/O. I.e. it comes with built-in serialization format supported by compiler. Not a good one - but perhaps a step in a good direction.
OTOH C people starving for serialization often believe that `struct` byte layout made by a specific version of a compiler is a serialization format - a view not endorsed by the standard.
yearolinuxdsktp 20 minutes ago [-]
Returning “*” instead of NULL to signal failure is the real footgun. Don’t blame the format.
For historical reasons, we could’ve ended up JSON as “hash”: “*” just the same. The /etc/shadow format lets you skip values completely, like “::”, so your argument doesn’t hold.
Terminating an app because of a NULL dereference is the other footgun. Sadly, we still have “panic” in modern languages like Go and Rust.
raverbashing 45 minutes ago [-]
> "it's all plain text" design that leans extremely heavily upon in-band signalling
Honestly I've been shaking my fist at these since 0 becomes True/Success in Bash and the opposite everywhere else
cwillu 13 minutes ago [-]
That's not specifically bash though, that's process exit codes
BoppreH 2 hours ago [-]
Yes, absolutely. Take for example this cURL command:
Random parts of it will be interpreted by the shell, others by cURL, or the filesystem, or the remote server. They each have their own encoding rules, nested. I count ten distinct separators and nine ways of escaping/tagging. It's one command. No, just no.
We have seriously overused strings and we're now paying the price.
dolmen 1 hours ago [-]
Also: free text is the input format of LLMs...
2 hours ago [-]
amluto 16 minutes ago [-]
The whole story is a bit worse than this. Musl already worked around this type of failure. See the second-to-last line here:
The vulnerable path in musl is only a little bit vulnerable, because "" is not actually a sha256* encrypted configuration and should not reach this code: musl's crypt(..., "*") will not call that code! Turns out that sshd detects that the password hash is too short and does not forward the salt!
Look, no else statement! If that if doesn't get taken, then salt remains NULL. And the sshd code is really quite messy and doesn't keep track of the fact that this code path is inappropriate for validating an existing password hash.
A workaround in the sha256 path in musl wouldn't even have prevented the bug!
I'm going to give some credit to musl for unwisely using "*" and some credit to OpenSSH for having nonsensical control and data flow.
NewJazz 9 minutes ago [-]
The formatting in your comment got messed btw
nxobject 3 hours ago [-]
Above all, I’m a little sad that the musl developers decided to make a lot of “well it’s convenient” changes to core crypto functions — at least without the level of auditing that would have caught this! It’s tarnished their reputation in my eyes a little.
rustyhancock 3 hours ago [-]
I think everyone is over fixated on AI finding complex bugs beyond a typical person's understanding either to locate or exploit.
I think for any one of those bugs there might simply be half a dozen footguns or bugs that once it's spelled out you wonder how anyone let it slide.
I think that's the case where LLMs might indentify most.
It's challenging though as the current AI audits produce so many false positives it's hard to know when it's being pedantic or when it's genuinely caught a human being a bone head
svnt 29 minutes ago [-]
This one might be hard to find using the author’s approach of throwing everything at various builds of ssh.
It is not really that hard to find in the wild if you had a device built this way, when overflowing buffers on embedded devices is a standard attack vector.
It might be hard to root cause, but it would not be that hard to find.
jamal-kumar 2 hours ago [-]
Finding more and more that these complex bugs people gloss over like you are now are being used in exploit chains and the end result is still my clients getting hacked
trebligdivad 3 hours ago [-]
In many ways each individual decision makes some sense;
wanting to stop loads of programs being affected by a (remotely triggerable?) crash
wanting to avoid time based attacks
wanting to be able to set accounts to not have a password
Each one relies on a perfectly valid argument - that * can't match because we all know it'll be rejected by the other thing.
yangm97 2 hours ago [-]
I would be more concerned about the lack of explicit return types, one of the arguments for strongly typed languages is that you should not have ambiguous data and yet…
Granted I mostly work with high level programming languages so I may be missing something.
Retr0id 3 hours ago [-]
> Each one relies on a perfectly valid argument - that * can't match because we all know it'll be rejected by the other thing.
But that's not a perfectly valid argument, and it never was.
zarzavat 3 hours ago [-]
The real problem here is the C language. Our industry should have dumped it a long time ago and we’re now paying the price of prevarication.
oneeyedpigeon 2 hours ago [-]
Which of the several flaws in this bug chain would have been materially different with a language other than C?
nh2 2 hours ago [-]
Any use of sum types would have prevented this immediately.
The whole thing exists because APIs that return a NULL value that "nobody checks", plus a string that you shouldn't use if you actually did check the null value.
In languages with sum types, this cannot happen.
Either you get your hash, or no result at all.
You couldn't get eg NULL and *.
jppittma 2 hours ago [-]
C has unions. C has sum types. People have been doing tagged unions in C for longer than rust has been a language
K0nserv 1 hours ago [-]
I don't think I've ever seen a C function that uses unions for this purpose though?
I can think of two common patterns that I've seen frequently:
* Return NULL for errors (sometimes with a global errorno() function).
* Return an integer error with any return values as out parameters.
So if it is possible to improve, why doesn't anyone do it?
aliceryhl 1 hours ago [-]
Yet the authors of crypt did not use them. The C language is not designed in a way to encourage people to write code that correctly uses tagged unions (or nullable pointers for that matter).
jppittma 1 hours ago [-]
Where are you going with those goal posts?
rini17 5 minutes ago [-]
Goalposts are fine. If crypt did return union type, C still allows cowboy programmers to straight extract the char* without any check whether it's valid. So we'd end up in exactly the same situation.
Philpax 15 minutes ago [-]
Nowhere. It's the same argument as upthread. A language that forces downstream consumers of an API to correctly deal with all possible outcomes would have prevented this.
dolmen 57 minutes ago [-]
Nope.
Because in fact all the changes were about changing the function signatures. The function signature is enforced by the linker. The semantic of strings that go through isn't.
Returning 2 values is possible in C (via pointers input parameters). The main issue here is that libc is a strong API contract based on C function signatures, that nobody can change, only augment with new symbols. Such strong API compatibility constraints happen in any programming language, and workaround to fit new behaviours in old APIs happens everywhere.
K0nserv 23 minutes ago [-]
I don't think nh2 is saying that this is how the "issue" should've been solved here, but that it's a fatal flaw of C itself (yes I know, unions, but no one uses them).
But also, its not like there were many alternative languages when the POSIX standards were designed so it's fair that it works like this.
2 hours ago [-]
muvlon 2 hours ago [-]
C's lack of all of
- sum/union types
- multiple return values/tuple types
- exceptions
means that most ways of returning an error from a function are not available in C. Because of this, many C APIs, including `crypt` in this case, resort to in-band error signalling by using a sentinel value (like NULL, a negative integer, "*" etc.).
Most other languages lack one or two of the above, but very few lack all three, and so most have developed a different culture around error reporting (Result in Rust, multiple return values in Go, exceptions in Java, Python etc.).
tialaramex 1 hours ago [-]
One of the small but important significances of Rust is that it very often gets to have the same machine code as a correct "in-band error signalling" strategy from C but with source code that the maintainers can actually understand.
And really often when your "in-band error signalling" in Rust is more expensive you realise (perhaps to your horror) that your C wouldn't be correct, so you just dodged a bullet. For example Option<OwnedFd> is, like OwnedFd, a 32-bit value, Unix has always reserved -1 to mean invalid, so Rust can treat that as a niche. But the Windows equivalent Option<OwnedHandle> can't do this because, well, cue Raymond Chen: https://devblogs.microsoft.com/oldnewthing/20040302-00/?p=40...
sscaryterry 1 hours ago [-]
The first 2 it has, and the 3rd, you really don't want.
jppittma 2 hours ago [-]
C has sum types/unions? What are you talking about? You don't need multiple return. You can use a struct and it's the same thing.
muvlon 5 minutes ago [-]
C unions are sum types if you just "draw the rest of the fucking owl". C has sum types the way x86 assembly has function calls. You see, there are "call" and "ret" instructions. Just figure out calling convention and the stack yourself, it's easy. And yes I can stick the tag next to the union myself in C and write a bunch of functions that hopefully deal with it correctly. I can even write some macros that look like pattern matching and maybe convince myself that it's a language feature, but it just isn't.
And it's very evident why the difference matters. People gravitate to what's readily available and ergonomic. That's why every C API under the sun (most definitely including libc itself) does janky stuff like returning -1 or "*" on error instead of building their own bespoke Result-style abstraction. A programming language is not just the things that are in some way possible within it, but more importantly the things that are easy in it.
adgjlsfhk1 12 minutes ago [-]
not in a way that forces the user to actually handle them properly. C is a weakly typed language and as such is pretty badly suited to systems programming.
K0nserv 2 hours ago [-]
For the other part of the issue (the truncation in SHA512), in Rust you could've had:
Now the truncation is a terrible idea in the first place, but at least with this it would have been entirely obvious to the caller that it was there.
dmantis 2 hours ago [-]
Wouldn't Rust's Result or Go's error checking prevent that?
While not mandatory, a person would probably use it instead of the * hack.
Simply because the calling side wouldn't have an option to forget the check like with the NULL case
ninkendo 2 hours ago [-]
Rust and go still make it pretty easy to not check results if you don’t care about the success value (lots of code just does `let _ = foo()`), but if you do want the value (like with a hashing function), Rust makes it really hard to not check the error too. There’s no syntax to say “assume this worked and give me the success value, including whatever garbage is there if it’s not set”:
- result.unwrap() will panic if it failed
- result.ok() throws away the error but gives you a None value instead (still can’t use it as if it’s a success)
- matching on the Ok variant and doing nothing on the Err variant, you only get to use the success value inside the Ok arm
Etc etc.
The problem is when the compiler says “NULL is a valid instance of T”, which go lets you do, and rust doesn’t (without using the unsafe keyword and intentionally casting a null pointer, which you don’t do by accident.) Go lets you write code like `var hash string; hash, _ = hashfn()` and it compiles and now you have null instead of a valid hash.
tialaramex 2 hours ago [-]
> NULL is a valid instance of T
This by the way is Tony Hoare's famous "Billion Dollar Mistake"
randomNumber7 1 hours ago [-]
Do you work in industry or academia?
01HNNWZ0MV43FF 2 hours ago [-]
There is no industry, it's just people
WesolyKubeczek 2 hours ago [-]
You must be new here. While I applauded the idea of musl when it first came out (coincidentally, glibc wasn't having the best time in its development then), the developer's arguments why <some behavior drastically different from every libc under the sun> was akchyually correct, which was why you had to fix a program that hadn't been broken for 30 years and now suddenly was, made me put him under the "clowns" category and never look at musl seriously again.
When later it turned out that musl's procrustian stack size made Python not really runnable under it, it cemented my resolve to not go near musl.
And now we have this.
acdha 2 hours ago [-]
I’m also reminded of the clowns pushing “it’s smaller!” container images based on social media vibes rather than analysis where, yes, not using Debian+glibc saved tens of megabytes on the base OS but did so at the expense of measurable performance reductions & compatibility issues like the one you mentioned, and literally every single one of those dudes I worked with was shipping containers pushing a gigabyte so the size reduction had no measurable impact on deployments.
NewJazz 16 minutes ago [-]
Uh, isn't this the case of the opposite? The musl devs trying to accomodate broken programs?
semiquaver 3 hours ago [-]
A little?!? I’m sorry, but musl is a joke, this is yet another proof of that fact, and I have no idea why anyone uses it in production, they have only themselves to blame after the many warning signs.
wolrah 2 hours ago [-]
Also how many times do we have to learn that letting badly programmed software die when unexpected things happen is the better option because it means people actually notice that there's a problem. Silent compatibility hacks are the worst, they guarantee problems never get fixed and the stack of hacks never gets smaller.
Microsoft has historically been one of the leaders on the wrong side of this one, but they have also demonstrated how effective it is to call out badly behaved software and make it annoy the users with UAC. When Vista came out, so many applications needed elevated privileges to run because they were badly designed to write user data in system locations, but now their application was more annoying to use and came with a scary dialog about it making changes to your computer that other better-designed software didn't get.
More often than not if a program didn't actually intend to change system settings it was just a matter of badly chosen default paths which are pretty trivial to fix, but so many vendors just would not do it despite years of complaints from NT admins wanting to not have to spend hours with Process Explorer figuring out specific permissions to grant limited users enough access. Once their failures were put front and center, calling their application out as the one getting things wrong every time the user ran it, they suddenly had motivation to fix it.
simias 2 hours ago [-]
For security it's better, but in many cases if you're not writing security-oriented software it can be worse from a user perspective. I remember the first job I had over 20 years ago was working on VoD streaming and one of the "tricks" to improve stability was mmap'ing the 0 page so that NULL dereferences wouldn't immediately segfault. Ugly as hell but when you were working with all sorts of more or less well coded media codecs and security wasn't a concern at all (worst case scenario the user would stream movies without paying) it made some amount of sense.
I suspect that Microsoft has the same history: the entire Windows platform was built with single-user PCs in mind, mostly only dealing with trusted input. The users were also expected to be tech-illiterate for the most part and unable to troubleshoot deep technical issues.
More recently I'm still working on video transmission, but on the production side this time. I also originally adopted the mentality that I should crash early if some unexpected situation was encountered, but I ended up walking that back a lot, because having a critical piece of live media production reboot mid FIFA worldcup final is generally Not Great. Degraded functionality is usually preferable. I still crash early in dev builds, but the production code just logs an error and continues if at all possible.
That said, I definitely agree with you for something like password handling. In this case, I do think it's warranted to crash the program without survivors if something is amiss.
pmontra 1 hours ago [-]
> if you're not writing security-oriented software
It's hard to define what's security oriented and what is not. For example, is this HN web app security oriented? Not in the way of sshd but it must keep our accounts data separated and it must not let people exfiltrate data from the db, where for example it is storing the emails of some of us.
funcDropShadow 26 minutes ago [-]
It depends on the time. Twenty or thirty years most software was not security-oriented. Nowadays, everything is online and therefore everything needs to be vigilant about security.
pixl97 1 hours ago [-]
> because having a critical piece of live media production reboot mid
It's not great, but these days a trickster with AI will find some way to start broadcasting porn to a billion people because of the flaw, so may end up being the better option.
whoanoway 1 hours ago [-]
[dead]
stackghost 1 hours ago [-]
> because having a critical piece of live media production reboot mid FIFA worldcup final is generally Not Great.
Story time?
zahlman 17 minutes ago [-]
Agreed. I feel like this attitude has been holding Python back, too. It's one thing to criticize "move fast and break things", but it's also not good to ignore already existing breakage just because the pieces still fit together neatly, and it's not good to refuse to move at all.
drdexebtjl 26 minutes ago [-]
Now everything installs to a user local folder, runs as the user, there are no UAC prompts, and malware can just freely tamper with other applications, act as the user, and harvest all data the user has access to.
Hey, but at least it can’t touch the System32 folder.
Great job guys.
pizlonator 52 minutes ago [-]
> Also how many times do we have to learn that letting badly programmed software die when unexpected things happen is the better option because it means people actually notice that there's a problem.
100%
hnfong 1 hours ago [-]
> Microsoft has historically been one of the leaders on the wrong side of this one
Objective-C ahem nil messaging...
sscaryterry 1 hours ago [-]
Objective-C is an Apple thingie :)
usefulcat 1 hours ago [-]
By way of NeXTSTEP
quotemstr 1 hours ago [-]
> Also how many times do we have to learn that letting badly programmed software die when unexpected things happen is the better option
Sadly, we will never learn this because failing fast runs counter to intuition, and a class of person will never override it.
bulatb 3 minutes ago [-]
Moralizing leaves them trapped in epistemic closure.
They will look at problems caused by failing late and they'll see proof that failing fast is bad.
zicixisi 2 hours ago [-]
[flagged]
evgpbfhnr 2 hours ago [-]
So to recap, with a non-upgraded sshd you need:
- an account with /etc/shadow hash set to '*'
- (note root won't work by default because of PermitRootLogin prohibit-password default)
- attempt twice, first attempt fails second long password is accepted (that took me a while to understand)
In practice passwd -l seems to lock with a ! prefix, and new account without passwords are created with a single ! which is also not affected, I'm not sure how big the impact actually will be in practice and don't understand the recommendation to upgrade urgently right now... Just don't use `*`?
EDIT: ah, some default container images apparently have a default alpine user with that hash, so I guess that's bad for them. This makes a bit more sense now.
cowlevel 37 seconds ago [-]
Why do you need to attempt twice?
babagan0ush 2 hours ago [-]
> Just don't use `*`
Well but, alas, a non-zero number of user probably do use it, though and "upgrade X" is a clearer and better instruction to communicate than "change this setting where maybe you don't even know what it means and it might break your workflow, but saves you an upgrade".
evgpbfhnr 1 hours ago [-]
In some contexts (e.g. embedded) you'd be surprised at how reluctant some people are to upgrade when they can run through a dozen of hoops for workarounds such as changing hash instead...
scandinavian 3 hours ago [-]
Did anyone get it to work? I tried on the official alpine docker image, enabled password login for root. The shadow entry has ! as password by default, this didn't work with a > 256 character password (i also did passwd -l which also does !). I manually changed it to * and it still didn't work.
Am I doing anything wrong here?
OpenSSH_10.3p1, OpenSSL 3.5.9 29 Sep 2026
PRETTY_NAME="Alpine Linux v3.24"
markuman123 2 hours ago [-]
It's because alpine has updated all openssh-server packages already. Even from EOL alpine versions.
bonyt 2 hours ago [-]
Ahh! Wasn't working on 3.16, which does indeed have an updated package.
But, got it to work on 3.15 - which requires changing the ! to a * in the shadow file in addition to other steps (making another user with a hash, setting PermitRootLogin to yes, etc.).
Didn't notice that the openssh-server-10.3_p1-r2 was released today.
atmosx 3 hours ago [-]
Nope, wasn’t able to reproduced it in my firecracker VM, which isn’t publicly accessible anyway.
f311a 3 hours ago [-]
As far as I understand, it wont work on default setups.
You need musl-c, no pam and openssh.
zamadatix 2 hours ago [-]
In Alpine, these would all be the case for an out of the box install with openssh added unless you specifically install openssh-server-pam and configure sshd_config to use PAM (Alpine is already musl-c based and doesn't default to PAM).
Aachen 2 hours ago [-]
Doesn't it also need a user to be added with a password entry?
I only ever see Alpine Linux used for containers which don't need accounts with interactive user logins, which is not to say it doesn't exist, just that it doesn't seem like Alpine comes with all preconditions met out of the box
zamadatix 44 minutes ago [-]
Isn't it the opposite, accounts which have no password and hence had * instead of a hash?
> accounts that should never take a password at all (root, your key-only deploy user, ldap users, the default user on alpine's aws and azure images) take any password longer than 256 characters.
But yeah, Alpine for persistant installs isn't unheard of - just not as popular as the container use case. They offer official VM, Xen, Pi, and bare metal images users deploy as well. I actually had a fleet of VMs to update!
Edit: The Alpine notice made it clearer to me that at least one other account has to have a password, it's just the target account that doesn't. So if you truly have no users with passwords in your container deployment then those passwordless users aren't vulnerable to this bug, but any of the image types which have a user will likely have root exposed.
> The attacker needs the name of an account whose password field is * (an account with no password set, which is normal for key-only accounts), and needs the machine to have at least one other account with a $5$ or $6$ password hash.
SSHing into test@ with one 'wrong' password then one 'long' password doesn't work. I have UsePAM no and PasswordAuthentication yes in my sshd_config.
justinsaccount 2 hours ago [-]
I can't reproduce it either. I used alpine:3.24.1 and openssh is openssh-server-10.3_p1-r2
I tried adding PermitRootLogin yes and PasswordAuthentication yes. Tried adding another user with a valid password.
Verified shadow entry has a *. Can't login as root or a test user. Tried another incorrect password first as well.
edit: can reproduce after `git revert 384e25e6bd1f2e7b30095900e48b570d46953510` in aports and rebuilding openssh without the fix. It also accepts the password on the 2nd+ attempt, NOT the first.
hagbard_c 2 hours ago [-]
No, didn't work for me open OpenWRT x64 running in a container but that is probably 'cause I'm using dropbear instead of openssh-server. That's what OpenWRT uses by default so the impact there will be minimal assuming that most installations don't change to openssh-server - it isn't installed by default.
2 hours ago [-]
ryanw02 2 hours ago [-]
[dead]
so_it_be 2 hours ago [-]
[dead]
sidneythekidney 2 hours ago [-]
Fun!
I was able to reproduce this on Alpine 3.23.7 with OpenSSH_10.2p1.
Luckily PermitRootLogin [0] hasn't allowed logins with passwords for root since 2015 [1]. I'm guessing that's at least part of why this went undiscovered for so long.
Offtopic: this is a very cool way to present commits and code excerpts. Love classic Windows styling.
Were those pictures with links generated using a publicly available tool that we could use, or were it custom (vibe)coded specifically to create this page?
dpacmittal 1 hours ago [-]
This looks entirely vibe-coded to me. No one is going to put so much effort for a vuln.
mmoll 12 minutes ago [-]
The musl vulnerability looks a lot like an intentional backdoor, doesn’t it?
Cakez0r 7 minutes ago [-]
Choosing * instead of NULL is indeed very fishy coincidence. Of all the return values they could've picked... What are the odds?
0x0 3 hours ago [-]
For OpenWRT, AsusWRT etc, I believe they use dropbear ssh by default instead of OpenSSH, which apparently isn't affected?
dhx 2 hours ago [-]
Yes, correct for OpenWrt. OpenWrt defaults to dropbear for both server and client. However, more advanced users may optionally choose to replace it with OpenSSH, or install OpenSSH side-by-side, to gain additional features of OpenSSH.
Dropbear added the 100 character password limit in 2019.77, so earlier versions might be susceptible.
neuroticnews25 3 hours ago [-]
That's a well-made site, yet I feel nothing.
tasuki 3 hours ago [-]
Maybe that's on you though. I enjoyed it, though the discrepancy between the `*sshole` name and the `sshhole.com` domain made me twitch. How is anyone[0] obsessive enough to find this and write it up and yet not obsessive enough to make the amount of `h` consistent?
[0]: Be they human or LLM. Or an entity with any proportion of the two. I do not judge nor discriminate.
glibg10b 3 hours ago [-]
Look closely. The second h is behind the word and almost as dark as the background
tasuki 3 hours ago [-]
Oh. Thank you!
oneeyedpigeon 2 hours ago [-]
I thought it was a great write up. It did feel a bit AI-coded, but whether it actually was or not, it's still a good bit of work imo.
xnorswap 2 hours ago [-]
It would be weird to spend this much effort doing the site by hand, for a bug that the author says was found by a swarm of agents.
phyzome 2 hours ago [-]
I dunno, there were some strange discrepancies in the explanation (and captions). Didn't really hang together.
user43928 2 hours ago [-]
Looks good overall, but I was wondering if eg. "the timer learns nothing" and "no X, no Y, no Z" was slop rather than a deliberate choice.
cpach 3 hours ago [-]
LLM output fatigue?
amelius 3 hours ago [-]
Apple sideways traffic light fatigue.
hagbard_c 2 hours ago [-]
Good to know I'm not the only one annoyed by that superfluous piece of ${whatever} signalling.
riffraff 3 hours ago [-]
I didn't hate the writing even if it's AI, but I deeply disliked the animations. I don't know if that's LLM fatigue or not.
bonplan23 1 hours ago [-]
I do actually. My guess is that this was not just one-prompted, but someone took the time to refine it based on their tastes and expectations until it matched something they would actually like.
However, if someone already developed a strong AI-slop-tingle it will still be triggered by some telltale signs and let their subconsciousness know that they can't possible enjoy this.
jonstewart 3 hours ago [-]
What is dead can never die. All blueteamers are ironborn these days.
p4bl0 36 minutes ago [-]
Nice web site. However, I've got a server with 14 alpine-based docker containers in production. Among them 13 are widely distributed images (like web server, databases, etc.) and one is a custom build image. Only one of them have an SSH server publicly exposed, Forgejo, but it's its own internal SSH server which only accept public keys and Git commands. On the internal network, only 3 of respond to SSH. I haven't tweaked their default SSH config. None were vulnerable to this attack.
RVuRnvbM2e 3 hours ago [-]
Wait so musl, famously a stickler for standards, just changed the behaviour of a POSIX function?
tomaskafka 2 hours ago [-]
At this point it feels to me we aren’t discovering security bugs but a decade of planted backdoors.
jephs 2 hours ago [-]
Never attribute to malice that which yada yada yada
And like who even needed to plant back doors? If you're the NSA you already had an agent swarm in 1993, but it was made out of dudes from Berkeley and New Haven instead of racks in Prince William County and 印西 (and they were literal secret agents I guess, pretty metal). You can get back doors just by foraging, no need for agriculture.
tialaramex 2 hours ago [-]
You have considerably underestimated both how fallible humans are, and how high their tolerance is for risks which aren't immediately visible.
This isn't even just the software industry, humans are like this about everything. Imagine explaining to something rational that (a) humans didn't realise that they won't be able to competently operate a motor vehicle while watching TokTok dances, and (b) when we finally made that illegal because it's such an obviously bad idea they would keep doing it anyway.
This is why those "What if it builds SkyNet?" conspiracies were so stupid. Why on Earth would you build SkyNet when you can just invent Weeaboo and the idiot humans will die playing ?
ImPostingOnHN 2 hours ago [-]
True, but also, that is exactly the pretext that would allow someone to maliciously, intentionally commit a backdoor.
There was even a contest for hiding/obfuscating malicious code, e.g. as an honest mistake: https://www.underhanded-c.org/ . And what do you know, the most recent, common approach shown on the page, is: improper comparison inputs that give unexpected comparison results, which is... well... this submission.
tomaskafka 1 hours ago [-]
Yes, "this looks like an underhanded c challenge" was my first thought when reading this. See one mistake by one project, inject the matching intentional mistake into another project (that also conveniently fixes some known bug), and boom - plausibly deniable backdoor.
If I had a free afternoon, I'd ask a bunch of agents to track the people responsible and their other contributions (but usually the scheme is "contribute valuable patches to build reputation, and only inject an exploit when safe to do so").
zicixisi 2 hours ago [-]
[flagged]
TZubiri 2 hours ago [-]
actually this looks very underhanded indeed, what are the odds they pick the same sentinel character?!
jephs 2 hours ago [-]
Like 1 in 5? The set of go-to characters for that kind of thing is pretty small.
robinpie 1 hours ago [-]
Pretty big? For this purpose, my instinct would be to select an ASCII character that's not alphanumeric, not directional, and not a common separator, and that's like, what, a set of 11 characters?
Wow. I wonder which of the common fuzzers would have found this a decade ago. If literally any password over 256 chars works, that means nobody has ever fuzzed openssh on musl linux
illusive4080 3 hours ago [-]
From the other thread’s source [1], it affects musl without PAM. You can set PasswordAuthentication=no or update to remediate.
What is out there that uses musl without PAM and hosts an OpenSSH SSHd with PasswordAuthentication?
As in: "Actually out there"
The hypothetical Alpine exists, yes, but what runs that like that?
Bender 3 hours ago [-]
What is out there that uses musl without PAM and hosts an OpenSSH SSHd with PasswordAuthentication?
Possibly public SFTP servers. I have allowed null passwords in the past for anonymous public SFTP and I run Alpine Linux exclusively for my servers both VM's and physical. The shadow package (PAM) did not get installed by default. I installed it but I had to specify it in the installation so there are likely people without it. There is probably a way to find them on shodan.io.
I will activate one for people to play around with for today:
sftp hn@ping.ohftp.org
There's no firewall and no password. Hack away HN.
Edit: Sure, a small minority presumably run sshd under these conditions. But a small minority times the number of Alpine containers running in the world gets to a large absolute number.
Edit 2: You really don't need to look very hard to find public container examples at million-download scale.
ekidd 3 hours ago [-]
Many, many Docker containers rely on "docker exec" or the kubectl equivalent to gain CLI access when needed. To have sshd in an Alpine Docker container, you're generally need to be running init and a bunch of demons, not just a single standalone service.
I'm guessing in practice relatively few people are actually exposed. But if they are, it's catastrophic.
hypfer 1 hours ago [-]
> Edit 2: You really don't need to look very hard to find public container examples at million-download scale.
It would be super helpful if you would answer a "what, specifically" question with a "this, specifically" itemized list or not answer at all.
This is not helpful.
hypfer 3 hours ago [-]
Docker containers that run sshds?
stingraycharles 3 hours ago [-]
Docker containers run sshd?
figassis 3 hours ago [-]
Agent sandboxes today probably
mkj 36 minutes ago [-]
It would be nice if the passwd crypt functions didn't have timing dependence on password length. Then there'd never need to be hacky workarounds like what musl did, and various tricks to avoid exposing whether users exist on systems.
No idea if it's likely to happen though.
0812037123158 2 hours ago [-]
It's amazing how similar this exploit payload is to the 20+ year-old Quake 3 vulnerabilities found by Luigi Auriemma (aluigi), which also used the exact same 256-to-1024 character range and nothing more than a long string of AAAA.... Here you're feeding >256 A's up to a 1024-byte limit to trip musl's defensive length cap, whereas back in the mid-2000s, Luigi was famous for publishing one-line .cfg scripts where the entire exploit was literally "say aaaaa..." repeated a few hundred times overflowing server's memory.¹ ²
A great example is Jedi Academy, a 2003 Quake 3 engine game whose multiplayer is still actively played today (its open-source engine port, openjk, is even in the Debian repos btw). Luigi had two classic scripts for it on his site: Jedi Academy <= 1.011 in-game server buffer-overflow (jamsgbof)¹, which crashed the server just by pushing a string of as past the 1024-byte console buffer, and Quake 3 engine in-game players kicking (q3msgboom)², where a wall of as hit the engine's 1022-char truncation limit and the leftover bytes confused the parser into disconnecting everyone on the server.
Kind of wild that 20 years later, the exact same payload goes from crashing Quake 3 matches from the tilde console to getting root on some SSH servers.
The last few weeks have made me feel very good about my policy of zero internet-accessible ssh ports, ever. On AWS, use SSM. For a VPS or home server, access everything over WireGuard.
Aachen 1 hours ago [-]
Such solutions always felt like swapping one risk for another. With good separation (running the VPN on a separate host from the SSH login, otherwise a bug in the VPN solution still might result in a shell on the box), at best you'd need two vulnerabilities before someone gets in, but it's not fundamentally different unless you find a way simpler protocol where you can be more certain that it's secure than ssh is (for all its scrutiny and track record of no pre-auth bugs for decades)
Putting services behind ssh forwarding was the way to secure something you're worried about rather than the other way around, but Wireguard actually makes a good argument nowadays. Iirc they did proofs for the cryptography and it has good design in terms of attack surface, and it's old/mature enough now that I'll say it's probably better than exposing SSH. On the other hand, when this thing running in your kernel gets compromised, you're slightly worse off than with someone logging into a passwordless service account as in the current bug
bspammer 20 minutes ago [-]
One thing that makes me feel a lot more confident about exposing a WireGuard port is it’s completely invisible unless you have the host’s public key. Nothing will show up on a port scan.
Its security model doesn’t rely on that fact of course, but it’s the cherry on top.
Kotlopou 1 hours ago [-]
Tangent: Pangram says the text is 60-ish% AI-generated. Not that surprising given the look of the page and the work being done with AI in the first place, but it is interesting to me because this doesn't look like your standard wall-of-text AI purple prose. So it's notable that a) you can generate convincing text in this style and b) Pangram detects it anyway.
yuck39 1 hours ago [-]
Panagram is just guessing.
traceroute66 3 hours ago [-]
To be fair, I have less than zero sympathy for anyone who runs SSH on any platform without `PasswordAuthentication=no` in 2026.
Its not asking for trouble, it is there will be trouble, not if but when.
As for the people who say "well, I run SSH on port 942329053052". Well guess what, the bots make short work on that. We tried it at $work as an experiment, multiple bots found it in less than a couple of hours. That's why your SSH config must be hardened, preferably running on a hardened OS too.
dwedge 2 hours ago [-]
> Its not asking for trouble, it is there will be trouble, not if but when.
Did any of those bots brute force a password? Because a 20 character random password is comparable in brute force complexity to an SSH key. The experiment you ran was just as valid for SSH keys on that port, and had no bearing on password vs key.
traceroute66 2 hours ago [-]
> Did any of those bots brute force a password?
The point is they found the security by obscurity so-called "hidden" port in less than a couple of hours.
At that point, what they did is irrelevant. The point is that security by obscurity does not work in 2026.
> Because a 20 character random password is comparable in brute force complexity to an SSH key.
You can't be serious with your comment.
No its not.
And even then, SSH key is asymmetric, you're not sending the private key to the server. But with a password you are, which leaves you vulnerable to many attack vectors that simply do not exist with keys.
Aachen 2 hours ago [-]
> > Because a 20 character random password is comparable in brute force complexity to an SSH key.
> You can't be serious with your comment. \n\n No its not.
If you differ in opinion, maybe provide evidence? Especially for a claim like long randomly generated value 1 being absurdly ("can't be serious"-level) weak as compared to long randomly generated value 2
If you mean it's log(62^20,2)≈119 bits of entropy instead of 128+ and so it's theoretically weaker when we build a Dyson sphere around the sun to brute force that key (after somehow obtaining the shadow file, since an online attack would be way too slow), then sure
minitech 2 hours ago [-]
Other points are fine, but why do you say a 20-character random password isn’t comparable in brute force complexity to an SSH key?
TZubiri 2 hours ago [-]
> The point is that security by obscurity does not work in 2026.
I feel the opposite, note how most all AI vulns discovered use source code level analysis, empirically keeping source code hidden is more secure.
kerchoff’s principle is misconstrued by a voyeuristic internet that wants to see source code
Ardon 1 hours ago [-]
This is a sampling error - we're only hearing about the public disclosures.
(almost) No company actually has secret source code, only source code not publicly viewable. And the LLMs crawling those don't belong to people who make public disclosures.
Ecco 3 hours ago [-]
How is password auth weaker than key auth? Bugs in auth code can happen in both pathways. And apart from stupidly short passwords both should be equally impossible to brute-force.
In other words, conceptually, at the end if the day, isn’t a key just another name for a password, really?
minitech 2 hours ago [-]
Password gets sent to the server. Private key does not get sent to the server, and authentication protocol can be forwarded safely, enabling things like ssh-agent, FIDO SSH keys. This is solvable with passwords (see: augmented PAKE), but `PasswordAuthentication` isn’t that. But there are also settings where none of this really matters, yes, to be fair if that’s what you’re referring to.
SSH certificates are cool too.
TZubiri 2 hours ago [-]
password doesn’t get sent to the server, hash does.
but I guess that the hash is always the same unlike with pub priv key mechanism.
traceroute66 2 hours ago [-]
> password doesn’t get sent to the server, hash does.
Kind of irrelevant what gets sent to the server and what happens on the server.
Password is already easily defeatable long before it even leaves your computer, thanks to keyloggers and/or password manager breaches and/or old-school shoulder surfing (assisted in the 2026 era by covert filming and AI).
All of which are not vulnerabilities with hardware-backed keys which you don't even need a Yubikey for these days since most modern computers and smartphones have secure enclaves and/or TPMs. (Yubikeys are still great though).
mkj 1 hours ago [-]
No, you are wrong. The password gets sent to the server. That said, I still enable passwords on servers, but don't ever share passwords between trust domains.
Borealid 2 hours ago [-]
What people think of as a very strong password often has <100 "security bits" of entropy.
Keys are usually at least 128 bits of entropy, and it's easy to take them up to 256.
Yes, both are a shared secret, but password auth - even with challenge-response - is weaker than key auth exactly because key auth provides more guess resistance.
Elucalidavah 2 hours ago [-]
But the SSH keys are based on asymmetric crypto, not shared secrets.
traceroute66 2 hours ago [-]
> What people think of as a very strong password often has <100 "security bits" of entropy.
The best password in the world will still be defeated by a simple keylogger or breach of your password manager.
scotty79 2 hours ago [-]
2^100 is a lot of things to try though
jandrese 2 hours ago [-]
Especially if you have Fail2Ban set up.
traceroute66 2 hours ago [-]
> if you have Fail2Ban set up.
Worth noting that if you are running a remotely recent version of SSH it has everything you need built-in. No need for Fail2Ban on SSH these days if your config is fully modernised.
belorn 2 hours ago [-]
Fail2Ban is kind of required unless one like running out of sockets and are unburden by high cpu and disk load.
Aachen 1 hours ago [-]
Huh? I run ssh (OpenSSH) on port 22 with root login enabled and no fail2ban on a single-core laptop cpu from fifteen years ago. The load average is 0.00 even with the small mail and web server that runs on it
What cpu and especially what disk load are you talking about? Also not running out of sockets, not sure if that's RAM-dependent (the box has 2GB)
8organicbits 2 hours ago [-]
Password reuse is the problem. The password hash is stolen from some other site and brute forced offline.
dwedge 2 hours ago [-]
Same issue with key reuse. Unless you have a policy to rotate them and verify which host they login from, someone's unencrypted hard drive from 5 years ago is more likely to have a key than password.
traceroute66 1 hours ago [-]
> someone's unencrypted hard drive from 5 years ago is more likely to have a key than password.
Even 5+ years ago, hardware-backed SSH keys were a thing (Yubikey). The only difference today is that its 100% easier to do keys properly because of the widespread commoditisation of secure enclaves in desktops and smartphones to support FIDO, its basically free of charge now because you're not obliged to buy a Yubikey.
ant0ni 1 hours ago [-]
you claimed my favorite game's character name!
2 hours ago [-]
znnajdla 3 hours ago [-]
This particular bug works even with password authentication turned off.
sjamaan 3 hours ago [-]
Where did you get that idea? The entire bug is about password hashing. With password auth off, that's not even attempted.
kijin 2 hours ago [-]
The exploit works for a user account that has no password. This is often confused with turning off password auth, especially in containers that have only one usable account.
It's not just the parent commenter. I'm sure somebody out there is going to remove their account password instead of editing their sshd_config in an attempt to "patch" this vulnerability, actually exacerbating the problem.
tialaramex 3 hours ago [-]
Bug > are password logins still on? they are by default
You > This particular bug works even with password authentication turned off.
No. The bug is related to password authentication. If you do not have password authentication then you are not affected.
What freaks some people out is they chose parameters where they do have password authentication but they expect to always fail. That's a bad choice, like OK you did fit a cheap manual keyhole, but you threw the keys away thinking now nobody can open that - you realise somebody with a cheap jiggler can open it right ?
znnajdla 2 hours ago [-]
read the article again
traceroute66 3 hours ago [-]
> This particular bug works even with password authentication turned off.
Does it ?
Somebody here has already said "You can set PasswordAuthentication=no or update to remediate"
We don't use musl at $work anyway, so I've no means to test.
kccqzy 2 hours ago [-]
I run SSH with password authentication on but the port binds to a ULA address. It’s fine. No need to be so absolutist.
traceroute66 2 hours ago [-]
> No need to be so absolutist.
Why not ?
Even if you don't want to go all the way to SSH certificates, SSH public-key authentication is laughably easy to do.
I also don't buy the argument "but its on a ULA address". Attacker's old favourite these days is finding a way in (North–South) and then using that as a stepping stone (East–West). Also the general concept of "lower security is OK because its on the LAN" is so 1997.
kccqzy 2 hours ago [-]
Yeah because if attackers are already finding a way in, they can easily read my ~/.ssh/id_ed25519. A password is strictly better, although not by much because lateral movement is very easy anyways. The only exception is if the private key is actually on a HSM, which is the case for all my SSH private keys on iOS.
traceroute66 2 hours ago [-]
> read my ~/.ssh/id_ed25519.
No excuse for doing that these days.
You can use a Yubikey or you can use your computer's secure enclave or TPM with FIDO at no extra cost.
A password is never, ever "strictly better". Seriously, you're just making shit up now.
ghm2180 3 hours ago [-]
100% and with local tool calls many AI systems will do ssh "for you" and end run around any security theatre that you may have going on with LLMs.
unglaublich 1 hours ago [-]
I mean there are 65k possible ports so it's the equivalent of guessing an 16 bit password, i.e. two (extended) ascii characters. It's not very surprising that 'multiple bots' found it: a single `nmap` command will give you the open ports in a few seconds.
Edit: My bad as pointed below this is not in OpenSSH but in musl and it appears in the relase notes https://www.openssh.org/releasenotes.html#10.6p2. I had scrolled straight to the bottom of the website because the animations annoyed me. Still would like a CVE for proper tracking though!
doodlesdev 1 hours ago [-]
OpenSSH considers this to be a vulnerability in musl and not OpenSSH, but shipped a fix anyways since that helps mitigate impact... Whether you agree with that or not, it's up to you :)
mkj 54 minutes ago [-]
Probably because mitre take days to weeks to give out CVE numbers (if ever). Just get the fix out there.
qwel-1asj 3 minutes ago [-]
All that site is lacking is marquee and furry creatures. AI slop sites are so horrible with their blinking animations, l33t dark background and chaotic text.
Did we declare "break the world Thursday" and nobody told me? I feel like it's three weeks in a row that we've had a once-in-a-decade security issue pop up.
vova_hn2 2 hours ago [-]
I think we should get used to eternal "break the world Thursday", just like our predecessors had to come to terms with Eternal September [0].
I'm starting to realize that trusting some random people who wrote the gruddiest, boringest, most torturous low-level stuff might have been a bad plan.
At this point I feel embarrassed in front of the AIs.
throooooo 3 hours ago [-]
The bugs and commits that "fix" them reveal somewhat of a shitshow for how musl is managed. All projects have bugs, but I can't help but question why the wheel was reinvented here. Many toes have been stubbed along the way.
chekibreki 2 hours ago [-]
PSA for OpenWrt users: If you are getting the post quantum message on login which points to openssh.com, you are likely using dropbear nonetheless. Check in System->Software. I was confused and worried why I seemingly had the vulnerable OpenSSH installed, but its just dropbear using an OpenSSH link.
So AFAIU, you are fine.
nasso_dev 55 minutes ago [-]
something interesting to note here is that while this isn't a memory safety vulnerability, a modern systems programming language with better error handling could have easily avoided this whole situation. error handling in C is a PITA, and we're paying the cost of decades of pretending that it's not a problem
draxen 3 hours ago [-]
I avoided using alpine-based containers ever since finding out a lot of Linux server software is actually Linux + SystemD software. The union set API of both Linux and SystemD seems to be the default "Linux API" today.
puzzlingcaptcha 2 hours ago [-]
but alpine famously does not use systemd but openrc?
vel0city 2 hours ago [-]
That's their point. The software they want to use often assumes its being run on a platform with systemd, so running that software on Alpine which makes a point to not use systemd causes challenges for them compared to using a different distro.
sam_lowry_ 3 hours ago [-]
I use alpine-based containers whenever I can just to keep the world safer from SystemD rotting Linux.
Never enabled sshd there... except on my PostmarketOS PineBook Pro... oh shit!
andunie 3 hours ago [-]
What is the argument against systemd?
Is it mainly that it is too big?
I feel like everybody still loves the unix philosophy, but the previous tools were too bad, systemd just filled user demand for easier tools.
The moment someone makes a pid 1 software that is good and still modular, open and pluggable I think that will be an instant success.
timcobb 9 minutes ago [-]
I just want to point out that, aside from LinkedIn content, I am decreasingly hearing about "AI slop" or "human-in-the loop".
ghm2180 2 hours ago [-]
Thanks you to the team that found this. Exceptionally well explained and funny at the same time. What is most important is how people just checking stuff( remember the .xz hack that was tried?) has save the internets ass so many times. We should give good security researchers UBI for life for just doing good work.
This is a super duper annoying site for people who just want to know if their systems are vulnerable, after patching like a dozen critical vulnerabilities in other software over the last week and are almost at the breaking point of resigning.
Can we have a “don’t treat me like five/toddler” / “idgaf about the explanation” quick link to a section for adults who need to patch systems?
(And if your reaction is “just scroll down, stupid”, I’m happy to report that with the scroll jacking, I scroll not enough / past the critical sections nine out of ten times.)
Thanks for mentioning that before I tried to scroll down. Forewarned, I was able to click on the "2026" link once the left-handed "scrollbar" (not the right word but I can't figure out what the right word should be) appeared, which let me jump past most of the verbiage and get to the "are you affected?" section just a couple pages later.
jamal-kumar 2 hours ago [-]
It's people like you I fire with glee
dcchambers 18 minutes ago [-]
This one feels negligent. How does this go uncaught for 10 years? How did it make it into production period?
We need to re-think EVERY assumption about security.
drob518 2 hours ago [-]
So, nobody fuzz tested sshd before?!?!
jsiepkes 1 hours ago [-]
> So, nobody fuzz tested sshd before?!?!
OpenBSD probably did (which doesn't use musl), RedHat probably did (or some other distro using glibc), but they all don't use musl.
bensyverson 2 hours ago [-]
Yeah, I am mystified as to how this could have happened.
2 hours ago [-]
paaloeye 3 hours ago [-]
Not the worst site, but it's clearly a publicity machine from wakefield.co, whom I haven't heard of.
Also, exploitation conditions are super narrow, albeit, any password is wild.
cube00 3 hours ago [-]
> except a lot of programs never check for NULL. they just compare whatever crypt() gives back with the stored field, hit the NULL, and crash
> so musl (the libc under alpine) goes ok fine, crypt() just won't return NULL anymore. if it fails, it returns a single character instead.
Considering it's been accepted that crashing if malloc returns a NULL is your fault, did musl really feel the need to accommodate bad implementations like this or is this just AI speculation?
red75prime 3 hours ago [-]
"unfortunately, a large portion of programs which call crypt are not
prepared for its failure and do not check that the return value is
non-null before using it. thus, always "succeeding" but giving an
unmatchable hash is reportedly a better behavior than failing on
error." [1]
Such a questionable decision! "Rather than let a bug in a caller (not handling the valid null output and thus crashing) surface, let's just silence it and pretend that the caller is correct! A potential clash in some situations (resulting in this vulnerability) is better than a crash!"
Maybe I should update my plea to fail loudly (https://alejo.ch/3gk) to list this as an example.
01HNNWZ0MV43FF 2 hours ago [-]
It's always a people problem. Doing a bad patch on my side is easier than bothering anyone else to fix their code
Good4boothee 3 hours ago [-]
> did musl really feel the need to accommodate bad implementations like this
I don't know about this specific case but in general? Yes, totally. No one is going to fix all existing software that depends on libs behaving exactly as glibc, you have to accept some trade-offs. Either your new implementation stops being perfectly simple and clean, or no one uses it.
Palomides 2 hours ago [-]
it's not "glibc behavior", returning null on error is in the spec
vova_hn2 2 hours ago [-]
See also: Hyrum's Law
gabucino 49 minutes ago [-]
rich felker was such a f* (funny) guy back in the mplayer days too
never changed i guess
philipwhiuk 3 hours ago [-]
Magic characters are awesome.
muddi900 2 hours ago [-]
I would like to read this but the scrolling is completely broken on iPad Safari
ctrlaltdileep 3 hours ago [-]
That's kind of shocking. I guess the swarm of AI related vulnerability findings continues.
vdfs 2 hours ago [-]
It's shocking that it took so long with the amount of investment and hype you'd expect this kind of bugs to be already fixed, but it still need a human to tell the agents what to do, which could also work with any fuzzing framework
ChrisRR 3 hours ago [-]
Can we just have a site with the text and images without the awkward scrolling crap? Hijacking basic features is an instant click away from sites for me
Edit: It seems to work fine on mobile. I'm not sure why the scrolling behaviour was added on the desktop version of the site
littlecranky67 2 hours ago [-]
> how did I ran into it? [...] i had agent swarms. a lot of them. let's not get into it
Can we please now normalize AI usage in development? Just some other day people asked for "NO AI used" badges for software as measure for quality, and praised i.e. NetBSD for being AI-free. Yes, we come from a phase of AI slop, but it has become obvious that there are plenty of legit usecases where AI beats human-only effort nowadays.
neuronexmachina 2 hours ago [-]
There's a decent-size "any AI usage is theft" contingent that reacts dramatically to any usage of AI whatsoever.
__MatrixMan__ 2 hours ago [-]
You've got a great point. But let's also acknowledge that there's a pretty stark contrast between these:
- I found a way to get AI to help me while I made this
- I asked AI to make this
The people you're objecting to are complaining about the latter, this is an example of the former.
rglover 2 hours ago [-]
Not sure why but a lot of people seem to be missing that you don't have to look at AI as a binary "you're either with us or you're with the terrorists."
Just do HITL (human in the loop). Use an LLM to help you author, but steer the AI in whatever direction you want. It's like having a slightly delusional copy of yourself to pair program with. If you take the time to provide an LLM with some additional skills files and the like, with you in the pilot seat, you can get solid results (it doesn't look as cool as a hands free agent setup but the quality far surpasses anything they can do solo).
vova_hn2 2 hours ago [-]
> Can we please now normalize AI usage in development?
1. It is already normalized outside of a tiny, but very vocal bubble that lives mostly on Bluesky (sometimes also on Reddit/HN/Mastodon).
2. I don't think that even the most staunch AI hater would object to using AI for security research as long as all the artifacts (code, docs) that were produced based on the research are human-written. "Yes, there is a bug in my program, but I refuse to fix it, because I know that you found it using AI" would be a pretty weird position. Not sure, how would you defend it.
littlecranky67 40 minutes ago [-]
I feel like there is a lot of pushback from delusional engineers, that won't accept their coding skills have been rendered useless - so they come up with tons of reasons to avoid AI, because it just has to be bad. Anything to hold up the ilusion you are an well-paid white-collar worker that can pick employers anytime. And I say this myself coding since the late 90s and making a living off it - it is a painful pill to swallow, but the genie does not go back in the box so just accept it and move on.
stogot 3 hours ago [-]
Of all the named vulnerabilities, I appreciate the history lesson and screenshots of how it got there
bombcar 3 hours ago [-]
The real name of this is sshole - the is the "star" of the show, after all.
yreg 3 hours ago [-]
*sshole, the * is the star of the show
You can use \ to escape formatting.
bombcar 38 minutes ago [-]
Ugh! HN got me, too
3 hours ago [-]
shaolinspirit 2 hours ago [-]
came for the name, stayed for the story
andreagrandi 1 hours ago [-]
This is fine :)
DonHopkins 2 hours ago [-]
This is bottom-up branding I can get behind. ()*()
cowlevel 59 minutes ago [-]
This is a pretty well done AI generated website, congrats you have proven it's not solely for slop.
gosen 3 hours ago [-]
I really like creativity in the name, the way website built and the content
"tl;dr" was a bit of a downgrade though, but that's fine
it's a nice way to promote the company behind it
globular-toast 14 minutes ago [-]
Tbh the most disappointing part of this is musl (and later glibc) changing the function to stop callers crashing because they didn't RTFM.
HackerThemAll 2 hours ago [-]
I just created an Alpine container, did "apk add openssh" and this doesn't work for me. I assume it's been updated already.
lclc 3 hours ago [-]
One more reason to build your (own) Docker images on 'FROM scratch' (basically nothing).
blueflow 3 hours ago [-]
No, the Alpine containers don't come with sshd preinstalled. You'd have to manually install via dockerfile and leave it unconfigured.
Good4boothee 3 hours ago [-]
No one runs sshd in containers, at least I hope so. What would even be the point when you have docker/kubectl commands available?
2 hours ago [-]
MarkBlah 3 hours ago [-]
Haha insane, great find and write up
jdw64 3 hours ago [-]
I liked that this site explained the vulnerability in an easy-to-understand way.
ordu 3 hours ago [-]
I can't understand it. All this scrolling and pictures are distracting me, can't concentrate really.
vavkamil 2 hours ago [-]
Yep, scrolling through my phone and reading slowly to understand. I'm seeing distracting animations below what I'm reading. And when I go to them, there is no way to restart them. So it completely misses the point.
jdw64 3 hours ago [-]
I think it varies from person to person. Since I'm kind of from the TikTok generation, I don't have much trouble reading this kind of animation, but some people prefer static sites. That's just how it is.
I also found scroll-triggered animations hard to read at first, but these days a lot of sites use them, so I've gotten used to them.
alternative implementations in general slightly down
karel-3d 2 hours ago [-]
does this have CVE or something?
henry700 3 hours ago [-]
at this point my only reaction is "are you fucking kidding me"
sneela 3 hours ago [-]
Yet another clankered finding creating a clankered website with clankered explanations. To be fair, this one's a lot better than the other clankered equivalents, but I wonder whether these people even know enough about their bug to have their own write-up?
Also, what's with this damn trend of typing in lowercase only? Why is this a thing?
> oh did i say "i"? sorry. i had agent swarms. a lot of them. let's not get into it
Still a cool finding... it's just the communication is becoming a lot worse these days - I want to understand the findings please, not be impressed by your webpage! Biggest gripe in recent memory is https://github.com/xoreaxeaxeax/skitter-creek-bath-salts
BugsJustFindMe 3 hours ago [-]
> Also, what's with this damn trend of typing in lowercase only?
Anticapitalism.
dsunds 2 hours ago [-]
You made my morning. Got a new dad joke.
DonHopkins 2 hours ago [-]
you could make a strong case for that!!!
account42 2 hours ago [-]
Anticapitalism that uses an army of clankers?
DonHopkins 2 hours ago [-]
sigh, it's not a new trend. it's just yet another pop-cultural appropriation from the gay community, now mainstream thanks to ai. as trump's gay dementia progresses, pretty soon even he will be truthing about other men's asses in lowercase, just to get a rise out of them!!! however, the *sshole project's shibboleths of lowercasing and anal asterisk punctuation strongly suggest its authors may be bona fide members of the lgbtq+ community.
> Let’s sit and twink about the data. 24.6% of my 19,000+ respondents abandoned capitalization entirely. If you control for age, queer people were 1.5x more likely than straight people to type in lowercase. Lesbians out-lowercased every other sexuality at 28.3%, followed by bisexuals at 24.9%. Honorable mention to the lowercase enbies, at 29.2%.
New and Changing Social Evaluations of All-lowercase and Exclamation Points
It starts with a bold claim: "ssh accounts that should not accept a password will accept any password between 256 and 1024 characters". But then it keeps adding qualifiers:
- sshd compiled with musl
- not using PAM
- password logins enabled
- at least one account with '*' as the password "hash"
- at least one OTHER account with a valid password
- server is aligned with 2 ley lines
- hacker is wearing a blue t-shirt and green socks
Ok, the last two are an exaggeration, but still. The conditions are all ANDed, miss a single one and you're fine.
On top of that, as someone else in these comments reminded, the target account needs to have a valid login shell or other exploitable program assigned to it, and these passwordless accounts tend to have nologin or something similar.
To me this seems to fall on the "technically possible, practically very unlikely" bucket, kinda like the PHP vulnerability from the other day. Maybe this has gone unnoticed for so long because it just doesn't happen.
> - sshd compiled with musl
So alpine as well as a bunch of other embedded Linuxes. Huge install base.
> - not using PAM
What you get if you try to have a minimal setup. Default sshd config if you install from source.
> - password logins enabled
The default. Super common.
> - at least one account with '*' as the password "hash"
Almost every Linux system.
> - at least one OTHER account with a valid password
Almost every Linux system.
So sure, there are caveats, but they cover a massive install base.
This is a super bad bug
I'll concede that embedded linux might be my blindspot here. Maybe cheap web hosting, too, but that seems much less likely to be using musl.
> The default. Super common.
This always seems bonkers to me. I think SSHD should give you ONE password free connection of setup to copy your key over and then it just shuts the door behind it.
It might actually a good idea, but I would likely avoid it since my built-in Bayesian filter would just mark it as spam.
Not sure whether the PHP vulnerability actually had been exploited, but I'm pretty sure there are literally thousands of machines out there affected by this ssh issue right now as we speak.
This isn't going to end well.
---
Addendum (which may or may not be related to the above) --
Everyone has heard of the tale of the boy who cried wolf. The main moral of the story is not to troll your fellow townsfolks so they don't distrust you. But everyone with a sane mind gets that.
The (IMHO) under-appreciated lesson is actually: don't be those villagers. Believe stories you are given, even if you might look like a fool for believing in them. Don't assume what happens is the same as what happened in the past. Look at the facts yourself. Trust and verify.
Sure there are native Alpine installs too, but that is far less common of a use case.
The goal from an attacker’s perspective is to get this to be true: `$ret = crypt($pw, $hash); $ret == $hash` when called by OpenSSH’s sshd. This is how sshd validates that the user logging in knew the account password.
The `$hash` is the “settings” to the crypt call and will normally look like “$6$--gibberish” for users with passwords set, but could look like “*” or “!” for users not intended to be able to login.
The bug is from the intersection of all of these things:
1. musl can return “*” for errors, rather than leave the return value NULL. Other libc have adopted something like this too, but seem to use a different result “*0”.
2. musl can also return “+” for errors, but only does so if $hash starts with “*”. This is intended to eliminate the possibility of an error causing `$ret == $hash` to be true, however it assumes that applications calling `crypt()` will set $pw and $hash to values corresponding to the `$ret == $hash` comparison to be made immediately afterwards.
3. OpenSSH can be convinced to pass the hash of a different user if the hash is “*” or “!” to avoid timing attacks that can enumerate the user list, so it will set $pw and $hash to correspond to different users. $hash will be a real “$6$--gibberish” style hash in this case, and OpenSSH expects `crypt()` to either return NULL or return a hash string. But now there’s two hashes OpenSSH uses. It does: `$ret = crypt($pw, $hash2); $ret == $hash1`;
4. This should normally not be a problem because OpenSSH will pass in valid $pw and $hash2 that should not lead to errors in hashing, so the $ret that musl generates should be a real hash that cannot match the “*” or “!” $hash1 for the login-disabled user. However as a DDoS mitigation, musl will return error if the password is too long when using SHA-2-based hashes (the default, it seems).
5. This error will result in a non-NULL result of “*” or “+”. Since OpenSSH passed in a real hash ($hash2), musl will return “*”, which compares equal to the actual user hash $hash1 for the disabled-login user.
The OpenSSL adjustment ensures a minimum hash length for the user before logging them in, even if crypt() succeeds somehow.
The ‘concise’ output style setting is bad with Opus 5.5, as it will often spit out a cryptic (is this really English?!), allergic-to-commas concise response, which takes more time to parse.
wtaf ?
[1] https://en.wikipedia.org/wiki/In-band_signaling#Computer_dat...
Or, for backwards compatibility, start non-hash strings with a character which can't appear in a hash, such as '@'. Your examples would then become '@no_password' and '@error_while_hashing'.
If /etc/shadow was a JSON (or XML, even, although that would make me feel gross), this bug would have been much easier to see, because a field that reads '"hash": ""' sticks out a lot* more than a custom microformat that you don't understand.
With data structures, there's never "plain text" - it's all structured. The question is whether you want to use existing formats and tools, or create a new one for every tool and literally make footguns like this.
The elegance of the unix-y approach is actually that it's all bits. You can just join things together and shove the bits through, that's a wild simplification compared with having vendor-specific kinds of plugs. But we could have that kind of compatibility while also benefiting from structure.
So I encourage everybody to give a shell with structured output a try, let it rewire your brain, and then ask yourself: how is the OS holding this back from being everything it could be? Because if we build that OS, it won't encourage these problems.
When I imagine it, it would probe either side of a pipe to see if they support negotiation. If not, send the bits through in the same old unixy way. If so, use the newest schema for structured data that is supported by both sides.
https://github.com/benbernard/RecordStream
Now the public API is hard to change but you could still avoid the issue by returning a longer token like "==DANGEROUS_CRYPT_ERROR_length==" or "==DANGEROUS_CRYPT_ERROR_unsupported_alg==" which would never match the password field unless someone was really trying to induce the issue.
https://pubs.opengroup.org/onlinepubs/009604599/functions/cr...
>Upon successful completion, crypt() shall return a pointer to the encoded string. The first two characters of the returned value shall be those of the salt argument. Otherwise, it shall return a null pointer and set errno to indicate the error.
returning a sentinel value of NULL and setting errno is pretty standard C, and shouldn't cause an error.
OP's sshhole link specifically notes that it _should_ return NULL, but some MUSL devs decided they wanted it to return trash instead, and the trash they returned instead of NULL caused this.
All because some devs were too lazy to check the result and another dev decided to be clever and "fix" NULL errors from crypt by sending back 'unmatchable' trash banking on devs not checking NULL.
edit: apparently crypt implementations have been trash ignoring the NULL return for a long time. this isn't only on MUSL, but it's still because people ignored and tried to work around the existing out-of-band error mechanism.
Some badly designed password hashing algorithms (cough bcrypt cough) have maximum password lengths, ignoring everything after X characters. So it could happen that I set my password to a 100-character string, it gets saved as "==DANGEROUS_CRYPT_ERROR_length==", and then anybody can log in to my account by entering an unrelated long string that returns the same "error".
If a function promises a string result, it should only provide a string if the function succeeded. The null was much better than magic strings.
E.g. in Pascal you can can define a record and then defined file of record to have typed I/O. I.e. it comes with built-in serialization format supported by compiler. Not a good one - but perhaps a step in a good direction.
OTOH C people starving for serialization often believe that `struct` byte layout made by a specific version of a compiler is a serialization format - a view not endorsed by the standard.
For historical reasons, we could’ve ended up JSON as “hash”: “*” just the same. The /etc/shadow format lets you skip values completely, like “::”, so your argument doesn’t hold.
Terminating an app because of a NULL dereference is the other footgun. Sadly, we still have “panic” in modern languages like Go and Rust.
Honestly I've been shaking my fist at these since 0 becomes True/Success in Bash and the opposite everywhere else
We have seriously overused strings and we're now paying the price.
https://git.musl-libc.org/cgit/musl/tree/src/crypt/crypt_des...
(hey, this workaround is wrong if the salt is "x"!)
but they forgot to work around it here:
https://git.musl-libc.org/cgit/musl/tree/src/crypt/crypt_sha...
The vulnerable path in musl is only a little bit vulnerable, because "" is not actually a sha256* encrypted configuration and should not reach this code: musl's crypt(..., "*") will not call that code! Turns out that sshd detects that the password hash is too short and does not forward the salt!
https://github.com/openssh/openssh-portable/blob/master/auth...
Look, no else statement! If that if doesn't get taken, then salt remains NULL. And the sshd code is really quite messy and doesn't keep track of the fact that this code path is inappropriate for validating an existing password hash.
A workaround in the sha256 path in musl wouldn't even have prevented the bug!
I'm going to give some credit to musl for unwisely using "*" and some credit to OpenSSH for having nonsensical control and data flow.
I think for any one of those bugs there might simply be half a dozen footguns or bugs that once it's spelled out you wonder how anyone let it slide.
I think that's the case where LLMs might indentify most.
It's challenging though as the current AI audits produce so many false positives it's hard to know when it's being pedantic or when it's genuinely caught a human being a bone head
It is not really that hard to find in the wild if you had a device built this way, when overflowing buffers on embedded devices is a standard attack vector.
It might be hard to root cause, but it would not be that hard to find.
Each one relies on a perfectly valid argument - that * can't match because we all know it'll be rejected by the other thing.
Granted I mostly work with high level programming languages so I may be missing something.
But that's not a perfectly valid argument, and it never was.
The whole thing exists because APIs that return a NULL value that "nobody checks", plus a string that you shouldn't use if you actually did check the null value.
In languages with sum types, this cannot happen.
Either you get your hash, or no result at all. You couldn't get eg NULL and *.
I can think of two common patterns that I've seen frequently:
* Return NULL for errors (sometimes with a global errorno() function).
* Return an integer error with any return values as out parameters.
So if it is possible to improve, why doesn't anyone do it?
Because in fact all the changes were about changing the function signatures. The function signature is enforced by the linker. The semantic of strings that go through isn't.
Returning 2 values is possible in C (via pointers input parameters). The main issue here is that libc is a strong API contract based on C function signatures, that nobody can change, only augment with new symbols. Such strong API compatibility constraints happen in any programming language, and workaround to fit new behaviours in old APIs happens everywhere.
But also, its not like there were many alternative languages when the POSIX standards were designed so it's fair that it works like this.
- sum/union types
- multiple return values/tuple types
- exceptions
means that most ways of returning an error from a function are not available in C. Because of this, many C APIs, including `crypt` in this case, resort to in-band error signalling by using a sentinel value (like NULL, a negative integer, "*" etc.).
Most other languages lack one or two of the above, but very few lack all three, and so most have developed a different culture around error reporting (Result in Rust, multiple return values in Go, exceptions in Java, Python etc.).
And really often when your "in-band error signalling" in Rust is more expensive you realise (perhaps to your horror) that your C wouldn't be correct, so you just dodged a bullet. For example Option<OwnedFd> is, like OwnedFd, a 32-bit value, Unix has always reserved -1 to mean invalid, so Rust can treat that as a niche. But the Windows equivalent Option<OwnedHandle> can't do this because, well, cue Raymond Chen: https://devblogs.microsoft.com/oldnewthing/20040302-00/?p=40...
And it's very evident why the difference matters. People gravitate to what's readily available and ergonomic. That's why every C API under the sun (most definitely including libc itself) does janky stuff like returning -1 or "*" on error instead of building their own bespoke Result-style abstraction. A programming language is not just the things that are in some way possible within it, but more importantly the things that are easy in it.
While not mandatory, a person would probably use it instead of the * hack.
Simply because the calling side wouldn't have an option to forget the check like with the NULL case
- result.unwrap() will panic if it failed
- result.ok() throws away the error but gives you a None value instead (still can’t use it as if it’s a success)
- matching on the Ok variant and doing nothing on the Err variant, you only get to use the success value inside the Ok arm
Etc etc.
The problem is when the compiler says “NULL is a valid instance of T”, which go lets you do, and rust doesn’t (without using the unsafe keyword and intentionally casting a null pointer, which you don’t do by accident.) Go lets you write code like `var hash string; hash, _ = hashfn()` and it compiles and now you have null instead of a valid hash.
This by the way is Tony Hoare's famous "Billion Dollar Mistake"
When later it turned out that musl's procrustian stack size made Python not really runnable under it, it cemented my resolve to not go near musl.
And now we have this.
Microsoft has historically been one of the leaders on the wrong side of this one, but they have also demonstrated how effective it is to call out badly behaved software and make it annoy the users with UAC. When Vista came out, so many applications needed elevated privileges to run because they were badly designed to write user data in system locations, but now their application was more annoying to use and came with a scary dialog about it making changes to your computer that other better-designed software didn't get.
More often than not if a program didn't actually intend to change system settings it was just a matter of badly chosen default paths which are pretty trivial to fix, but so many vendors just would not do it despite years of complaints from NT admins wanting to not have to spend hours with Process Explorer figuring out specific permissions to grant limited users enough access. Once their failures were put front and center, calling their application out as the one getting things wrong every time the user ran it, they suddenly had motivation to fix it.
I suspect that Microsoft has the same history: the entire Windows platform was built with single-user PCs in mind, mostly only dealing with trusted input. The users were also expected to be tech-illiterate for the most part and unable to troubleshoot deep technical issues.
More recently I'm still working on video transmission, but on the production side this time. I also originally adopted the mentality that I should crash early if some unexpected situation was encountered, but I ended up walking that back a lot, because having a critical piece of live media production reboot mid FIFA worldcup final is generally Not Great. Degraded functionality is usually preferable. I still crash early in dev builds, but the production code just logs an error and continues if at all possible.
That said, I definitely agree with you for something like password handling. In this case, I do think it's warranted to crash the program without survivors if something is amiss.
It's hard to define what's security oriented and what is not. For example, is this HN web app security oriented? Not in the way of sshd but it must keep our accounts data separated and it must not let people exfiltrate data from the db, where for example it is storing the emails of some of us.
It's not great, but these days a trickster with AI will find some way to start broadcasting porn to a billion people because of the flaw, so may end up being the better option.
Story time?
Hey, but at least it can’t touch the System32 folder.
Great job guys.
100%
Objective-C ahem nil messaging...
Sadly, we will never learn this because failing fast runs counter to intuition, and a class of person will never override it.
They will look at problems caused by failing late and they'll see proof that failing fast is bad.
- an account with /etc/shadow hash set to '*'
- (note root won't work by default because of PermitRootLogin prohibit-password default)
- attempt twice, first attempt fails second long password is accepted (that took me a while to understand)
In practice passwd -l seems to lock with a ! prefix, and new account without passwords are created with a single ! which is also not affected, I'm not sure how big the impact actually will be in practice and don't understand the recommendation to upgrade urgently right now... Just don't use `*`?
EDIT: ah, some default container images apparently have a default alpine user with that hash, so I guess that's bad for them. This makes a bit more sense now.
Well but, alas, a non-zero number of user probably do use it, though and "upgrade X" is a clearer and better instruction to communicate than "change this setting where maybe you don't even know what it means and it might break your workflow, but saves you an upgrade".
Am I doing anything wrong here?
OpenSSH_10.3p1, OpenSSL 3.5.9 29 Sep 2026
PRETTY_NAME="Alpine Linux v3.24"
But, got it to work on 3.15 - which requires changing the ! to a * in the shadow file in addition to other steps (making another user with a hash, setting PermitRootLogin to yes, etc.).
https://pkgs.alpinelinux.org/package/v3.16/main/x86_64/opens... Build time 2026-11-05 06:53:46
on 3.21 with the previous version ...
Didn't notice that the openssh-server-10.3_p1-r2 was released today.
I only ever see Alpine Linux used for containers which don't need accounts with interactive user logins, which is not to say it doesn't exist, just that it doesn't seem like Alpine comes with all preconditions met out of the box
> accounts that should never take a password at all (root, your key-only deploy user, ldap users, the default user on alpine's aws and azure images) take any password longer than 256 characters.
But yeah, Alpine for persistant installs isn't unheard of - just not as popular as the container use case. They offer official VM, Xen, Pi, and bare metal images users deploy as well. I actually had a fleet of VMs to update!
Edit: The Alpine notice made it clearer to me that at least one other account has to have a password, it's just the target account that doesn't. So if you truly have no users with passwords in your container deployment then those passwordless users aren't vulnerable to this bug, but any of the image types which have a user will likely have root exposed.
> The attacker needs the name of an account whose password field is * (an account with no password set, which is normal for key-only accounts), and needs the machine to have at least one other account with a $5$ or $6$ password hash.
https://alpinelinux.org/posts/2026-11-05-openssh-password-au...
Also didn't have much luck on Alpine 3.20.10 with OpenSSH_9.7p1 which I've freshly installed from an official ISO.
Last two entries in /etc/shadow:
SSHing into test@ with one 'wrong' password then one 'long' password doesn't work. I have UsePAM no and PasswordAuthentication yes in my sshd_config.I tried adding PermitRootLogin yes and PasswordAuthentication yes. Tried adding another user with a valid password.
Verified shadow entry has a *. Can't login as root or a test user. Tried another incorrect password first as well.
edit: can reproduce after `git revert 384e25e6bd1f2e7b30095900e48b570d46953510` in aports and rebuilding openssh without the fix. It also accepts the password on the 2nd+ attempt, NOT the first.
I was able to reproduce this on Alpine 3.23.7 with OpenSSH_10.2p1.
Luckily PermitRootLogin [0] hasn't allowed logins with passwords for root since 2015 [1]. I'm guessing that's at least part of why this went undiscovered for so long.
0: https://man.openbsd.org/sshd_config#PermitRootLogin
1: https://cvsweb.openbsd.org/diff/src/usr.bin/ssh/sshd_config?...
Were those pictures with links generated using a publicly available tool that we could use, or were it custom (vibe)coded specifically to create this page?
[0]: Be they human or LLM. Or an entity with any proportion of the two. I do not judge nor discriminate.
However, if someone already developed a strong AI-slop-tingle it will still be triggered by some telltale signs and let their subconsciousness know that they can't possible enjoy this.
And like who even needed to plant back doors? If you're the NSA you already had an agent swarm in 1993, but it was made out of dudes from Berkeley and New Haven instead of racks in Prince William County and 印西 (and they were literal secret agents I guess, pretty metal). You can get back doors just by foraging, no need for agriculture.
This isn't even just the software industry, humans are like this about everything. Imagine explaining to something rational that (a) humans didn't realise that they won't be able to competently operate a motor vehicle while watching TokTok dances, and (b) when we finally made that illegal because it's such an obviously bad idea they would keep doing it anyway.
This is why those "What if it builds SkyNet?" conspiracies were so stupid. Why on Earth would you build SkyNet when you can just invent Weeaboo and the idiot humans will die playing ?
There was even a contest for hiding/obfuscating malicious code, e.g. as an honest mistake: https://www.underhanded-c.org/ . And what do you know, the most recent, common approach shown on the page, is: improper comparison inputs that give unexpected comparison results, which is... well... this submission.
If I had a free afternoon, I'd ask a bunch of agents to track the people responsible and their other contributions (but usually the scheme is "contribute valuable patches to build reputation, and only inject an exploit when safe to do so").
[1] https://hachyderm.io/@damienmiller/117563026759098135
As in: "Actually out there"
The hypothetical Alpine exists, yes, but what runs that like that?
Possibly public SFTP servers. I have allowed null passwords in the past for anonymous public SFTP and I run Alpine Linux exclusively for my servers both VM's and physical. The shadow package (PAM) did not get installed by default. I installed it but I had to specify it in the installation so there are likely people without it. There is probably a way to find them on shodan.io.
I will activate one for people to play around with for today:
There's no firewall and no password. Hack away HN. cd /pubget so*
Edit: Sure, a small minority presumably run sshd under these conditions. But a small minority times the number of Alpine containers running in the world gets to a large absolute number.
Edit 2: You really don't need to look very hard to find public container examples at million-download scale.
I'm guessing in practice relatively few people are actually exposed. But if they are, it's catastrophic.
It would be super helpful if you would answer a "what, specifically" question with a "this, specifically" itemized list or not answer at all.
This is not helpful.
No idea if it's likely to happen though.
A great example is Jedi Academy, a 2003 Quake 3 engine game whose multiplayer is still actively played today (its open-source engine port, openjk, is even in the Debian repos btw). Luigi had two classic scripts for it on his site: Jedi Academy <= 1.011 in-game server buffer-overflow (jamsgbof)¹, which crashed the server just by pushing a string of as past the 1024-byte console buffer, and Quake 3 engine in-game players kicking (q3msgboom)², where a wall of as hit the engine's 1022-char truncation limit and the leftover bytes confused the parser into disconnecting everyone on the server.
Kind of wild that 20 years later, the exact same payload goes from crashing Quake 3 matches from the tilde console to getting root on some SSH servers.
[1] https://web.archive.org/web/20101227181948/http://aluigi.alt... [2] https://web.archive.org/web/20060522175643/http://aluigi.alt...
Putting services behind ssh forwarding was the way to secure something you're worried about rather than the other way around, but Wireguard actually makes a good argument nowadays. Iirc they did proofs for the cryptography and it has good design in terms of attack surface, and it's old/mature enough now that I'll say it's probably better than exposing SSH. On the other hand, when this thing running in your kernel gets compromised, you're slightly worse off than with someone logging into a passwordless service account as in the current bug
Its security model doesn’t rely on that fact of course, but it’s the cherry on top.
Its not asking for trouble, it is there will be trouble, not if but when.
As for the people who say "well, I run SSH on port 942329053052". Well guess what, the bots make short work on that. We tried it at $work as an experiment, multiple bots found it in less than a couple of hours. That's why your SSH config must be hardened, preferably running on a hardened OS too.
Did any of those bots brute force a password? Because a 20 character random password is comparable in brute force complexity to an SSH key. The experiment you ran was just as valid for SSH keys on that port, and had no bearing on password vs key.
The point is they found the security by obscurity so-called "hidden" port in less than a couple of hours.
At that point, what they did is irrelevant. The point is that security by obscurity does not work in 2026.
> Because a 20 character random password is comparable in brute force complexity to an SSH key.
You can't be serious with your comment.
No its not.
And even then, SSH key is asymmetric, you're not sending the private key to the server. But with a password you are, which leaves you vulnerable to many attack vectors that simply do not exist with keys.
> You can't be serious with your comment. \n\n No its not.
If you differ in opinion, maybe provide evidence? Especially for a claim like long randomly generated value 1 being absurdly ("can't be serious"-level) weak as compared to long randomly generated value 2
If you mean it's log(62^20,2)≈119 bits of entropy instead of 128+ and so it's theoretically weaker when we build a Dyson sphere around the sun to brute force that key (after somehow obtaining the shadow file, since an online attack would be way too slow), then sure
I feel the opposite, note how most all AI vulns discovered use source code level analysis, empirically keeping source code hidden is more secure.
kerchoff’s principle is misconstrued by a voyeuristic internet that wants to see source code
(almost) No company actually has secret source code, only source code not publicly viewable. And the LLMs crawling those don't belong to people who make public disclosures.
In other words, conceptually, at the end if the day, isn’t a key just another name for a password, really?
SSH certificates are cool too.
but I guess that the hash is always the same unlike with pub priv key mechanism.
Kind of irrelevant what gets sent to the server and what happens on the server.
Password is already easily defeatable long before it even leaves your computer, thanks to keyloggers and/or password manager breaches and/or old-school shoulder surfing (assisted in the 2026 era by covert filming and AI).
All of which are not vulnerabilities with hardware-backed keys which you don't even need a Yubikey for these days since most modern computers and smartphones have secure enclaves and/or TPMs. (Yubikeys are still great though).
Keys are usually at least 128 bits of entropy, and it's easy to take them up to 256.
Yes, both are a shared secret, but password auth - even with challenge-response - is weaker than key auth exactly because key auth provides more guess resistance.
The best password in the world will still be defeated by a simple keylogger or breach of your password manager.
Worth noting that if you are running a remotely recent version of SSH it has everything you need built-in. No need for Fail2Ban on SSH these days if your config is fully modernised.
What cpu and especially what disk load are you talking about? Also not running out of sockets, not sure if that's RAM-dependent (the box has 2GB)
Even 5+ years ago, hardware-backed SSH keys were a thing (Yubikey). The only difference today is that its 100% easier to do keys properly because of the widespread commoditisation of secure enclaves in desktops and smartphones to support FIDO, its basically free of charge now because you're not obliged to buy a Yubikey.
It's not just the parent commenter. I'm sure somebody out there is going to remove their account password instead of editing their sshd_config in an attempt to "patch" this vulnerability, actually exacerbating the problem.
You > This particular bug works even with password authentication turned off.
No. The bug is related to password authentication. If you do not have password authentication then you are not affected.
What freaks some people out is they chose parameters where they do have password authentication but they expect to always fail. That's a bad choice, like OK you did fit a cheap manual keyhole, but you threw the keys away thinking now nobody can open that - you realise somebody with a cheap jiggler can open it right ?
Does it ?
Somebody here has already said "You can set PasswordAuthentication=no or update to remediate"
We don't use musl at $work anyway, so I've no means to test.
Why not ?
Even if you don't want to go all the way to SSH certificates, SSH public-key authentication is laughably easy to do.
I also don't buy the argument "but its on a ULA address". Attacker's old favourite these days is finding a way in (North–South) and then using that as a stepping stone (East–West). Also the general concept of "lower security is OK because its on the LAN" is so 1997.
No excuse for doing that these days.
You can use a Yubikey or you can use your computer's secure enclave or TPM with FIDO at no extra cost.
A password is never, ever "strictly better". Seriously, you're just making shit up now.
Edit: My bad as pointed below this is not in OpenSSH but in musl and it appears in the relase notes https://www.openssh.org/releasenotes.html#10.6p2. I had scrolled straight to the bottom of the website because the animations annoyed me. Still would like a CVE for proper tracking though!
[0] https://github.com/leonjza/sshhole-lab
[0] https://en.wikipedia.org/wiki/Eternal_September
At this point I feel embarrassed in front of the AIs.
So AFAIU, you are fine.
Never enabled sshd there... except on my PostmarketOS PineBook Pro... oh shit!
Is it mainly that it is too big?
I feel like everybody still loves the unix philosophy, but the previous tools were too bad, systemd just filled user demand for easier tools.
The moment someone makes a pid 1 software that is good and still modular, open and pluggable I think that will be an instant success.
Can we have a “don’t treat me like five/toddler” / “idgaf about the explanation” quick link to a section for adults who need to patch systems?
(And if your reaction is “just scroll down, stupid”, I’m happy to report that with the scroll jacking, I scroll not enough / past the critical sections nine out of ten times.)
Edit: You may want to read https://sshhole.com/#exposure.
We need to re-think EVERY assumption about security.
OpenBSD probably did (which doesn't use musl), RedHat probably did (or some other distro using glibc), but they all don't use musl.
> so musl (the libc under alpine) goes ok fine, crypt() just won't return NULL anymore. if it fails, it returns a single character instead.
Considering it's been accepted that crashing if malloc returns a NULL is your fault, did musl really feel the need to accommodate bad implementations like this or is this just AI speculation?
[1] https://git.musl-libc.org/cgit/musl/commit/?id=b3c4cc121f70f...
Maybe I should update my plea to fail loudly (https://alejo.ch/3gk) to list this as an example.
I don't know about this specific case but in general? Yes, totally. No one is going to fix all existing software that depends on libs behaving exactly as glibc, you have to accept some trade-offs. Either your new implementation stops being perfectly simple and clean, or no one uses it.
never changed i guess
Edit: It seems to work fine on mobile. I'm not sure why the scrolling behaviour was added on the desktop version of the site
Can we please now normalize AI usage in development? Just some other day people asked for "NO AI used" badges for software as measure for quality, and praised i.e. NetBSD for being AI-free. Yes, we come from a phase of AI slop, but it has become obvious that there are plenty of legit usecases where AI beats human-only effort nowadays.
- I found a way to get AI to help me while I made this
- I asked AI to make this
The people you're objecting to are complaining about the latter, this is an example of the former.
Just do HITL (human in the loop). Use an LLM to help you author, but steer the AI in whatever direction you want. It's like having a slightly delusional copy of yourself to pair program with. If you take the time to provide an LLM with some additional skills files and the like, with you in the pilot seat, you can get solid results (it doesn't look as cool as a hands free agent setup but the quality far surpasses anything they can do solo).
1. It is already normalized outside of a tiny, but very vocal bubble that lives mostly on Bluesky (sometimes also on Reddit/HN/Mastodon).
2. I don't think that even the most staunch AI hater would object to using AI for security research as long as all the artifacts (code, docs) that were produced based on the research are human-written. "Yes, there is a bug in my program, but I refuse to fix it, because I know that you found it using AI" would be a pretty weird position. Not sure, how would you defend it.
You can use \ to escape formatting.
"tl;dr" was a bit of a downgrade though, but that's fine
it's a nice way to promote the company behind it
I also found scroll-triggered animations hard to read at first, but these days a lot of sites use them, so I've gotten used to them.
[0] https://news.ycombinator.com/item?id=50377221
alpine stocks are crashing?
musl stocks are crashing?
glibc stonks are up
alternative implementations in general slightly down
Also, what's with this damn trend of typing in lowercase only? Why is this a thing?
> oh did i say "i"? sorry. i had agent swarms. a lot of them. let's not get into it
Still a cool finding... it's just the communication is becoming a lot worse these days - I want to understand the findings please, not be impressed by your webpage! Biggest gripe in recent memory is https://github.com/xoreaxeaxeax/skitter-creek-bath-salts
Anticapitalism.
why do gay people love typing in lowercase?
https://thetastefultwink.substack.com/p/the-queer-lowercase
> Let’s sit and twink about the data. 24.6% of my 19,000+ respondents abandoned capitalization entirely. If you control for age, queer people were 1.5x more likely than straight people to type in lowercase. Lesbians out-lowercased every other sexuality at 28.3%, followed by bisexuals at 24.9%. Honorable mention to the lowercase enbies, at 29.2%.
New and Changing Social Evaluations of All-lowercase and Exclamation Points
https://repository.upenn.edu/server/api/core/bitstreams/edbc...
> both exclamation points and lowercase usage are associated with queer identity.