Internet2 provides InCommon Certificate Service, issuing certificates from "InCommon Intermediate CA" which chains up to emSign Root CA - G1. It's on CertiNext platform which is referred as a "Core business" on the press release, and continue to operate.
Somehow this is making it even worse. Like, I'm having issues explaining the whole thing to my wife, because I can't come up with a metaphor or parable that's even remotely as ridiculous as what they've done. Anyone's got a good one?
Maybe, like, setting up a factory that builds both LHD and RHD cars and building cars with two passenger seats in the front for two years straight without anyone noticing?
In the old days a king would "sign" his proclamations by pressing his "unforgeable" signet into a wax seal. This proved the authenticity of the proclamation.
Normally, the king would declare and sign a proc stating that anything signed by his secretary's "unforgeable" signet was to be respected as well. The secretary could delegate authority to others in turn.
Well turns out that one of the signets in the chain of trust was just a picture of a "four" that anybody could have forged, but nobody noticed because in all of the ceremony and stacks of parchment with wax seals, nobody actually checked if the signet was secure.
They were "unforgeable" because commissioning a jeweler to make an intricate carving on a ring by hand was very expensive in the first place. Finding one who could precisely replicate an existing carving, down to the mistakes of its original author and any wear and tear that would be visible on the wax seal, was almost impossible and even more expensive.
I just explained what it actually is. I said "you know how you go to a website HTTPS and it's secure? Well Apple chose a couple of trusted parties that your phone inherently trusts, and those people approve a few thousand intermediate companies who do the day-to-day certificate work for end websites. One of those intermediates, rather than choosing a 200-digit random number, chose 4, no not four-digits, the number 4". And then we could laugh at some of the memes.
Everyone in your neighborhood knows your address, but someone discovered two days ago that they can get into your house by sticking a 4" oblong piece of paper into the keyhole. They put it on blast on Facebook, so now everyone in the whole planet is trying to break into your house.
As a 90s kid, the Dodge Omni was a common shitbox student car. I think they had like 8 key patterns lol.
My brother’s ignition key unlocked my friends’s trunk. There were a few people who could start other peoples cars. In one case, a soccer player “stole” a football players unlocked car as they looked identical and were parked near each other. In another as a prank we unlocked cars to roll them out of parking spots and physically lift them into unusual places.
How about, someone is selling a secure box of some sort, and their factory starts putting glitter nail polish over the screws that hold it together, as is sometimes done for security reasons.
Then the intern decides, “I can just glue these identical little white circles with blue sparkles onto the screw, and it will be faster and I can take more breaks!”
>I can't come up with a metaphor or parable that's even remotely as ridiculous as what they've done
The best example I can think is (some) international borders: yeah, they're heavily regulated entities with lots of written protections (and thousands of miles of impenitrable barricades), but occassionally a hole opens up (whether physically or engineered).
Perhaps a section of border fence is occassionally destroyed (and people sneak in), but this won't happen if the border guards are approving [false] passports from a compromised government actor ("they have a valid passport, according to our inspection"), the border fence doesn't even matter!
Presume anybody can now go to that bad-passport-issuing government and request a passport to then enter any other country (i.e. the other country will see the passport as valid in the official global database): BAM, borders no longer mean anything; welcome to Privileged Accessia.
I'm also wondering who the hell would use a registrar called "eMudhra" with Arial font in the subtext in their logo. It's so incredibly 90s.
I'm actually being serious, I do judge companies by their logo and website design taste, and more modern-looking companies also tend to pay more attention to doing things properly.
we could stick with the metaphor of keys. The modern web requires fancy ones capable of hundreds of notches. The company in question basically skipped the step of notches and just used the blank key
> However, the level of social media amplification around this incident has brought scrutiny that is out of proportion to SSL/TLS’s place in our business
Scrutiny is inversely proportional to bit complexity of the private keys they used. Without a failure to apologize or expose how it happened: a human mistake, process mistake, etc. this looks pretty bad on them.
> eMudhra’s global SSL/TLS infrastructure runs separately from its other operations. The company’s Global digital signature certificate (DSC), eSign, and other signing services, including those in India, its enterprise PKI Solutions (emCA, CertiNext platforms), emSigner eSignature Suite, and its identity and authentication solutions (SecurePass, emAS) are not affected and continue to operate as norma
Anyone relying on them with anything related to security or trust is playing Russian roulette at this point.
Various Indian regulations (particularly around commerce and banking) require you to use India companies for solutions like the above. It gets worse when companies are "steered" by a regulator/compliance person who has a bias for a specific solutions provider.
> About eMudhra / With over 17 years of expertise, eMudhra is a global provider of digital identity, trust, and security solutions, enabling secure digital transformation for banks and financial institutions (BFSI), enterprises, and governments across more than 35 countries worldwide. Its product suite spans CertiNext
I was hoping the statement at the bottom would be written in past tense: eMudhra was a global provider of digital identity ... had temporarily enabled secure digital transformation ... Its product suite spanned CertiNext...
Like, that had to be malicious right? ssh-keygen won't generate a key like this, and neither will openssl or whatever Microsoft is doing behind their CA server.
Maybe it was just a mistake and they intended to generate a `--bitsize 4` thinking it was in kilobyte units.
shrug
I think they're just absolutely inept and should never be trusted with anything security related as an organisation if this is a mistake they could make.
You can't do that because an RSA private key contains two numbers - the modulus and the exponent. Even better, an RSA private key usually has the modulus stored as its two factors rather than their product.
Also note that using small numbers like 65537 for RSA public exponents is common practice.
Today "racism" is used as a synonym for "bigotry". Would you agree it was bigotry? Or do you only have a problem with people using words wrong, and not bigotry itself?
Whynot use the correct word for it, which is prejudice? Racism is prejudice against a race. You can be prejudiced to any group of people, doesnt mean that you are biggoted.
> Whynot use the correct word for it, which is prejudice?
Again you seem more concerned with grammar than, y'know, condemning racism, prejudice, or bigotry. They're all equally bad and closely related concepts. "A rose by another name" and so on.
Your own comment has 3 typos and several grammatical errors ("prejudiced to" is wrong, for example). Irony.
Finally, modern usages of "racism" encapsulate all of these meanings. Languages change over time.
> You can be prejudiced to any group of people, doesnt mean that you are biggoted
They are not being pedantic here, they are establishing a meaningful difference in words to help distinguish a shared phenomena. I think the original comment was unnecessary and certainly bigoted, but I would need more to say racism. The "grammar" guy clarified twice the semantic difference they personally observed (note they did not labor on syntax like you wish to suggest).
All that said, I make this comment as I believe the other commenter's worry is on display im this comment here...
> Again you seem more concerned with grammar than, y'know, condemning racism, prejudice, or bigotry. They're all equally bad and closely related concepts
These are _not_ all equally bad. Prejudice is foundational to understanding, there is no escape from this. In this understanding, we should treat particular postures of prejudice more serious than collapsing any "bad prejudice" under the same label.
> I think the original comment was unnecessary and certainly bigoted, but I would need more to say racism
Do you think racism is worse than bigotry? Why? I understand that the first word has a more emotional charge in our times. But as far as "bad things to do" go they're equal in gravity. Racism is just a subset of bigotry.
> These are _not_ all equally bad
Why not?
> Prejudice is foundational to understanding
That's one definition of the word - "judging without knowing the facts".
But there's another definition - "an irrational hostile attitude, fear, or hatred towards a particular group, race, or religion" - which was the one we are operating with in this thread.
My mistake. English is not my first language. In my native language, the word racism is used for discrimination or prejudice against anything foreign (nation, race, language, ...).
But either way, I find the reduction of the argument to the semantics of the word racism to be deflection. Wouldn't you agree the original comment was made in poor taste? Why did it need to include a reference to Indian and imply that all things Indian made are cheap or low quality?
We applied this kind of mindset to the Japanese in the 60s and 70s, and look where that got us. I witness it happening again in the 00s and 10s applied to Korean electronics (LG and Samsung mostly) while we were content with minor improvements over the status quo and innovating far too slowly.
Yes, this incident has an almost farcical amount of stupid attached to it, but the fact that it happened to an Indian company is in my eyes only coincidental. There are plenty of Silicon Valley startups and tech companies that are equally cavalier about aspects of security and apply the mentality of "we'll fix it in post" to anything but shipping features.
Dutch. The word is "racisme", and the entry for it in our dictionary[0] states:
> Racism is a form of discrimination in which a person is judged on the basis of his or her ethnic origin, skin color, nationality or religion. It is a form of prejudice and inequality that prevents people from gaining equal opportunities.
While I'm happy to admit that perhaps my use of the word in English is incorrect and its meaning there is more narrow, in Dutch it covers nationality or religion as well. Islamophobia is considered racist ("racistisch") in Dutch, for example.
"There are plenty of Silicon Valley startups and tech companies that are equally cavalier about aspects of security and apply the mentality of "we'll fix it in post" to anything but shipping features."
Almost all browsers require certificates be recorded in Certificate Transparency logs <https://en.wikipedia.org/wiki/Certificate_Transparency#Manda...>. Since those logs are empty in this instance, you can state that no one used a forged certificate with a mainstream browser, which is the most obvious (and desirable) attack type.
You can also monitor if they show up in certificate transparency logs. Which won't catch all fraudulently issued certs, but might show up some, especially if it's being widely abused.
Yes, but in the end that's not their fault, and in a sense it's good that this happened. It was well understood from the invention of security certificates that private keys might leak and certificates will need to be revoked. Systems that were not designed with this in mind should never have been deployed, and should now be decommissioned and replaced.
It might even be sensible to intentionally revoke a small number of certificates every month, to make sure nobody is complacent.
Mmmh so you will be trying to supplant the destination server with your own, with this emudhra leaked cert, to see what's being sent? Or how are you planning to do that? Just curious :)
This reminds me - I need to get around to setting up a PiHole and convince my partner that her freemium mobile games aren't necessarily worth the telemetry scrapes.
a high quality set of FOSS, community inspected "freemium clones" may do more for national security and individual privacy compared to yet another crypto library with zero novel mathematics just another reimplementation of specs...
> Yes, but in the end that's not their fault, and in a sense it's good that this happened.
I am reminded of the US DST changes back in the 2000s: there was all sorts of drama because a lot of software is written in the US, and there was much 'computerization' in the 1990s, but US coders had never personally experienced a DST rule change during that time period, so any change needed a reboot to pick up the new rules.
That drama led to (e.g.) periodic dynamic checking of /etc/localtime on various Unix-y system.
Sometimes you don't realize that things can be improved until you experience situations in which they currently suck.
The leadership page is comedy gold ("Leadership That Inspires Trust"):
> He has wide experience spanning over 45 years in Banking, Financial Services and Technology Industry. His profound understanding of the digital security landscape has been instrumental in eMudhra's global expansion and innovation. ...
> A thought leader and author, Mr. Srinivasan penned "New Age Management Philosophy from Ancient Indian Wisdom," ...
> Under his visionary leadership, eMudhra has become a cornerstone in digital security, known for its pioneering solutions and steadfast adherence to global standards.
Tell Congress to change the rules again soon, lest a new generation of programmers become complacent and hardcode transitions again. Many current junior programmers don’t remember the 2000s rule change.
I can even hear this conversation happening: what? I need to open this America/Los_Angeles file just to convert a timestamp? What pretentious overengineering! I want a single-file executable with zero dependencies. Let me hard code the rules.
There's a bunch of changes recently / coming up in Canada, so that might be good enough. I've started occasionally reading the tzdb mailing list archives for fun because of that.
Google has decided to stop trusting certificates with dual purpose (server and client) and managed to negotiate a decrease in the maximum (SSL/TLS) certificate lifetime because certificate revocation doesn't work.
We've had to rework part of our application and certificate use because of this.
The problem isn't revocation of end-user identity certificates. There's hundreds of millions of those floating around at any given point, so revocation happens very regularly. The issue is revocation of intermediates and roots, which involves invalidating tens of millions of certificates and often find themselves in embedded/non-web applications (financial, legal signatures, medical data exchange).
An analogy would be shutting down neighborhood streets versus shutting down major highways in a state for a hurricane evacuation. You can't test the latter at scale.
It really is. It's basically chaos monkey for pki.
Code paths and procedures that aren't exercised regularly are broken.
Even better is having a way to reoke keys and big players occasionally include revoked keys in trust chains and blast a warning if the revoked key is considered valid.
It rages me as well. Some random attacker could have MITM attack any https website without being noticed by anyone(including eMudhra), surely it won't contribute any to their revenue!
I mean, yeah I get your point but also if these pieces of the stack weren't prepared to deal with a compromised CA, then they were also at least a little bit negligent. There's always a chance this is going to happen. There's a chance it could happen and there not be any negligence or incompetence on the part of the CA
But at the end of the day if you're not prepared to revoke the trust of any CA you trust, it's a you problem
As I understand it, the reason for so much work is that eMudhra was not a root CA. It wasn't added to any official root CA list, instead it was signed by a different root CA with the assumption that eMudhra will complete the work required to become a root CA and this hack will become unnecessary. This is a common, temporary process. So while all the mechanisms for root CA revocation were in place, no one was quite prepared for intermediate certificate revocation.
Take out the word root from my original reply. If you aren't prepared to immediately stop trusting a CA--root, intermediate, internal, then your process is broken
"no one was quite prepared for intermediate certificate revocation" is actually worse because as an intermediate CA, they obviously didn't meet the requirements to be a full root CA, so why wouldn't they be more likely to need revocation than a root? Honestly more negligent than being unable to revoke a root.
This is just wrong though, they were a root CA. There was cross-signing which is typical, but their `emSign ECC Root CA - G3` root certificate was in most CA bundles.
Good point. I get why some stacks use their own trust stores and not the system trust store. But there should be some mechanism to revoke trust to some of the CAs, without applying patches or rebuilding. Even if its just a convention based /etc/certs/revoked-certs file.
It looks to me like the world was decently prepared to deal with it. The bad certificates are being blocked and there don't appear to be widespread compromises.
That doesn't mean it's cheap or easy, or that it's reasonable for the responsible party to brush off all the havoc they've caused.
"eMudhra’s global SSL/TLS infrastructure runs separately from its other operations. The company’s Global digital signature certificate (DSC), eSign, and other signing services, including those in India, its enterprise PKI Solutions (emCA, CertiNext platforms), emSigner eSignature Suite, and its identity and authentication solutions (SecurePass, emAS) are not affected and continue to operate as normal."
So they didn't even use their own certificates? :)
> Its infrastructure was not compromised and no data was breached.
The private key of your globally trusted CA is infrastructure data and it was breached.
> All physical and logical security controls remained intact.
No, because you promised the CA Program that those keys would be HSM generated, FIPS 140-2 level 3 compliant. Ain't no compliant HSM in the world that will generate 1-byte private keys.
> There were no end entity certificates issued out of the affected CA's.
I charitably read this as "revoking these weak CA didn't affect customers". But now your whole root is being revoked next week (which you forgot to mention). All your TLS customers get screwed.
Overall, good choice to give up, but lying on your way out is not a good look for a company with the tagline "Trust Delivered".
If this company was so incompetent that their private keys were a single byte, I seriously wonder what other areas they're incompetent in but that they've ventured into anyway. When you fail so spectacularly and so publicly at something like being a certificate authority, in the manner that eMudra did, it's rather hard not to wonder.
> However, the level of social media amplification around this incident has brought scrutiny that is out of proportion to SSL/TLS’s place in our business.
What a weak minded statement. Just admit you messed up and call it a day.
Immediately and totally exiting the domain is about as close to admit you messed up and call it a day as you'll ever get from a modern company.
The statement can use whatever weasel words it likes, but the action pretty clearly says "we aren't mature enough as an organisation to handle this responsibility." Which would be quite the concern for me, if I were a customer for one of eMudhra's other product lines.
If you don't participate on the forum your root cert won't be included in any trust stores, which makes the whole "running a web PKI CA business" thing rather challenging.
As someone who's well-versed with the "face-saving" aspects of Indian culture, I can state with clarity that the statement they put out is completely in character.
Folks in India — especially the boomer class — are trepidated by how they're perceived over actual substance and quality of their work (exceptions exist, of course).
Edit: Sudhir Kakar's (an Indian psychoanalyst) work
on Indian identity (The Inner World, Intimate Relations) helped me decode my own origin culture's internal logic. Highly recommend it for people of South Asian diaspora.
I am curious, not that I agree it exists or understand the reality of izzat, how does it differ in saving face provided by gentlemen who allow the loser to have a graceful defeat?
In martial arts the winner will often say "I got lucky" or "you almost had me" to allow the loser to save face, at least publicly, when in reality the skill gap is evident.
if it only affected the company itself rather than ~the entire world, there may be face to be saved. trusted authorities have to be held to a MUCH higher standard, for good reason.
no grace if you use 4 as the private key for a trusted authority.
> Izzat (Bengali: ইজ্জত; Hindi: इज़्ज़त; Urdu: عزت, romanized: 'izzat, lit. 'honour') literally translated: "respect" is a concept of prestige, dignity, and social respect.
If it is a component, then is OP wrong? Fwiw, a lot of nations have interesting variants of that kind of honor though they don't always express themselves the same way.
It's more than I'm not encouraging to engage with OP's point because he just wants to be a bigot. But that is separate from India having a shame based culture.
“One of eMudhra’s certificate authority private keys is the number 4. Not a number that starts with 4, or the number 4 followed by a few hundred random digits, just, 4… Another one is 56, and another is 115. I know, because I recovered all of them from their public certificates on my laptop just now, and it took less than a second,” wrote Helme.
Security researcher Scott Helme was among those who recovered the keys. This is a hall-of-fame level fail.
> Bruh, you just literally went ——— BEGIN PUBLIC KEY ——— 4 ——— END PUBLIC KEY ———
No, their private key is 4. Moreover, you wouldn't see the literal number 4 because the PEM key file would have the cryptographic sections encoded in base64.
> As someone who's well-versed with the "face-saving" aspects of Indian work culture, I can state with clarity that the statement they put out is completely in character.
As someone who's blood is tainted with teflon and microplastics same as you, I can state with clarity that attributing universal corporate tendencies to national work culture is completely in character for a Sagittarius.
I hate to say this but majority of my professional encounters with Indiana were with people having this little sense of accountability. But then again, with many awesome exceptions! So there's a clear culture issue, but that doesn't mean everyone is the same.
Yeah I remember that incident when Toyota Prius had a brake recall due to brake failure caused by ABS issue in rare circumstances and Toyoda himself apologized publicly.
This one is basically Japan being social media IRL since before it. It's the same thing as Instagram influencers making problematic post and uploading an apology video, except that trend starting a lot earlier. Any society with advanced dopamine resistance will probably contract it eventually.
War crimes stuffs is a separate and a bit more nuanced issue. It's also not just the simple matter of admitting or denying it.
Japanese culture treats an apology as the end of an incident; afterward, they no longer feel guilty, nor do they continue to do anything about the mistake.
To admit having messed up they would need to include the full private key in the post, which they seem to have decided is still more embarrassing than not including it.
What I hear is that he is saying that the SSL/TLS business is only a small part of their overall business and that they can close that part down as a result of the loss of reputation for that this has rendered.
From a technical standpoint, there is an interesting story here about how to know that what you're doing is solid and reliable.
I have to assume the fact this didn't get discovered suggests they weren't investing in trying to break their own crypto. A good takeaway for anyone else who wants to be in this space.
It's not quite as egregious as someone purposely choosing the key. According to the write up on the Mozilla bugtracker there was a bug in the script used for there key generation ceremony which caused the bad certificates to be generated. Obviously completely unacceptable for a CA, but makes more sense than someone plucking 4 out of the air while generating a root cert.
Correct. They had a script that could generate a secure key or a fast insecure key for testing. The script generated both at the same time.
Human error was selecting the wrong output file and promoting it forward.
It's not at all the kind of thing you want to see in a reliable, secure, mature process... There shouldn't be a step where a human even makes a choice there.
And it's exactly the kind of thing most companies are doing, most of the time, more often than we would like.
An awful lot of the world runs on spit and bailing wire and things too easy to automate yet, and the parts that don't cost more.
Thousand line shell scripts that use expect to drive 30+ answers to a multi-choice interactive tui (that only works on one version of redhat from 2008 or something) is not so unusual, can totally understand how it happened. No excuses in that though.
There were no end entity certificates issued out of the affected CA’s - that you know of
The company’s Global digital signature certificate (DSC), eSign, and other signing services, including those in India, its enterprise PKI Solutions (emCA, CertiNext platforms), emSigner eSignature Suite, and its identity and authentication solutions (SecurePass, emAS) are not affected and continue to operate as normal - maybe to a full review of those? Are you rolling your own crypto there by chance?
I just wonder how many times private keys were lost by or stolen from by slightly less incompetent companies?
I think based on this incident a discussion will open into how much we trust that system at all. With the level of espionage tactics some countries are keen to show the rest of the world, I personally assume absolutely anything can be sniffed.
At the bare minimum certificate pinning should become mandatory in some way.
X.509 needs to be replaced with a simple key you get for free during domain registration. Then the blast radius of key compromise is a single domain name and not the whole internet.
How does a client know what public key to expect for a given domain, then?
If they key is distributed by DNS, then you need to secure DNS, e.g. with DNSSEC. Well, with DNSSEC the TLDs and root basically have the same status as CAs (although it does cut down the huge number of CAs that can issue certs).
Beware even the countries internet regulatory authority (NIC) issued rogue certificates in past, about time NIC's root certificates should be ejected once and for all.
> There were no end entity certificates issued out of the affected CA’s.
What does this mean from the press release? Were they only issuing other chained intermediates? Or did they never issue any ECC certificates from these at all, and everyone is paying the price?
I wonder if big roots with chained intermediates will now need to have serious pressure on their downstream intermediates to re-prove their need to exist. I know RHEL and Debian's strategy (if not others) was to pull the Comodo AAA Services root as well ...
The certs were intended to be used by new US and EU subsidiaries. Those were still in the process of being set up, so the certs were never used in production for customer signing.
IIRC the Comodo AAA Services root was already in the process of being retired, with its "trust bit" having been removed by Mozilla and Chrome last year. Nuking it completely should have been a no-op for end users.
The whole chaining & cross-signing song-and-dance will never go away completely. There's just no way around if if you want enrollment of a new CA to take less than a decade, or if you want CAs to be able to do any form of non-world-breaking key rotation. Intermediates signed by third-party roots are already treated with the same scrutiny as root certs, so they should have been fine...
But yes, this will undoubtedly result in even stricter verification. For example, some participants are already calling for mandating attestation that a key has been generated inside a HSM running the right config.
# of issued leaf certs isn't a great indicator of propagation, user-impact, revenue etc.
1. 59 million of those 60 million could be very short validity, duplicate enrollments from same private key/csr, etc.
2. I'm still not certain that that figure isn't including for example private roots/intermediates they've issued from.
That said they're a low volume (in terms of root/int) provider so I don't think the impact of them exiting will be great.
> All physical and logical security controls remained intact.
Did the HSM have a fault, or was the key generation script at fault?
I suspect the later given the choice to exit the business. If they had strong evidence that a FIPS-compliant HSM misbehaved, I suspect they'd be more willing to fight this.
Wow this press release is a real PR disaster. It's really just about their business value.
Starting the statement with "a global digital trust company" is so ironic on that context.
honestly, props to the c-suite to have enough awareness that this buisness was already dead for them and there was absolutely no pathway where they are getting their certificates trusted in PKI ever again.
this lets them exit this without public humiliation of getting made fun (and example of) in the mailing lists.
Nevermind the legal liabilities, how is their company viable with this sort of brand damage? Their other business lines aren't making toasters. It's the same line of work. They sell trust.
True, but the founder's two sons are at the top of the company, so mission was accomplished regardless.
Really though, this is no different than people relying on accountancy firms in the rest of the world for "trust" and "security" as technology concepts. Accountancy firms have no business consulting on technology topics, and yet that is a major part of their business all over the world, because we've allowed "security" and "compliance" to become muddled concepts.
Typical bullshit where India regulations force people in India to buy local - and because the landscape is filled with incompetent and inept companies you get shit like this.
Yes they didn't close down the business, it was already gone. There's not a business called "selling certs that aren't accepted by any browser or OS". That's not a product.
Absolutely no way any OS or browser manufacturer forgives them and let's their certs validate anything in the future.
They also have to convince the root programs to let them stay trusted, which can't be done by issuing vague press releases until the news cycle moves on.
What about CAA records? And of course, browsers handle this with the Certificate Transparency log and the OneCRL, made less painful by baking a short-lived proof that the cert is not revoked into the handshake. The problem is all the other parts of the ecosystem that basically ignore all this new complex security stuff and just check the cert against a static list of vendored roots.
For example, node, java, and python all have vendored roots and don't check any CRLs.
Well, what about them? If you've got access to the private key, nothing is going to stop you from generating a cert violating CAA records - and it won't even be easy to detect! They are intended to protect against, say, someone hacking into your server, running ACME to get a cert, cleaning up their traces, and then intercepting your traffic for 90 days. CAA records assume the CA is trustworthy and following the rules.
now the question for the media for the next week :
ban LLM discovery of horrendous security practices because as it turns out security-thru-obscurity is being practiced as 'good enough' across the world , or actually enforce realistic and livable security practice ?
given the insistence on picking overblown and totally partisan poor choices lately, I presume the media will sit on the side of 'we'll just ban LLMs from checking the rest of our security apparatus for single digit keys' and charge 'Wayne' with terrorism statutes.' -- but hopefully i'm just being a jaded pessimist.
I think we live in an age where programming mistakes should be attributed to llm by default. In the old days, it was just the coder and stack overflow to blame, now its layer on layer of ai assisted work, from ceo level on down. Something we haven’t perfected yet: how to prove 0 ai use in a product. Until we get there, we have to assume llm are everywhere and that their use carries liability risk.
https://uit.stanford.edu/service/ssl/emSign
Maybe, like, setting up a factory that builds both LHD and RHD cars and building cars with two passenger seats in the front for two years straight without anyone noticing?
Normally, the king would declare and sign a proc stating that anything signed by his secretary's "unforgeable" signet was to be respected as well. The secretary could delegate authority to others in turn.
Well turns out that one of the signets in the chain of trust was just a picture of a "four" that anybody could have forged, but nobody noticed because in all of the ceremony and stacks of parchment with wax seals, nobody actually checked if the signet was secure.
- emudhra lawyer maybe (:
Except it's still not good enough, because you could use that key to eavesdrop nearly anything and anyone.
As a 90s kid, the Dodge Omni was a common shitbox student car. I think they had like 8 key patterns lol.
My brother’s ignition key unlocked my friends’s trunk. There were a few people who could start other peoples cars. In one case, a soccer player “stole” a football players unlocked car as they looked identical and were parked near each other. In another as a prank we unlocked cars to roll them out of parking spots and physically lift them into unusual places.
Mine was the lock company selling locks but also making freely available keys that would open any of their locks.
Then the intern decides, “I can just glue these identical little white circles with blue sparkles onto the screw, and it will be faster and I can take more breaks!”
The best example I can think is (some) international borders: yeah, they're heavily regulated entities with lots of written protections (and thousands of miles of impenitrable barricades), but occassionally a hole opens up (whether physically or engineered).
Perhaps a section of border fence is occassionally destroyed (and people sneak in), but this won't happen if the border guards are approving [false] passports from a compromised government actor ("they have a valid passport, according to our inspection"), the border fence doesn't even matter!
Presume anybody can now go to that bad-passport-issuing government and request a passport to then enter any other country (i.e. the other country will see the passport as valid in the official global database): BAM, borders no longer mean anything; welcome to Privileged Accessia.
I'm actually being serious, I do judge companies by their logo and website design taste, and more modern-looking companies also tend to pay more attention to doing things properly.
And when that clicks, extend the metaphor, literally, to a key with ~384 grooves.
Then they locked it with the key you find on a chew toy for a baby.
house collapses and people get injured.
concrete manufacture says we no longer build houses cause it’s too small of a business.
Scrutiny is inversely proportional to bit complexity of the private keys they used. Without a failure to apologize or expose how it happened: a human mistake, process mistake, etc. this looks pretty bad on them.
> eMudhra’s global SSL/TLS infrastructure runs separately from its other operations. The company’s Global digital signature certificate (DSC), eSign, and other signing services, including those in India, its enterprise PKI Solutions (emCA, CertiNext platforms), emSigner eSignature Suite, and its identity and authentication solutions (SecurePass, emAS) are not affected and continue to operate as norma
Anyone relying on them with anything related to security or trust is playing Russian roulette at this point.
I was hoping the statement at the bottom would be written in past tense: eMudhra was a global provider of digital identity ... had temporarily enabled secure digital transformation ... Its product suite spanned CertiNext...
It would need to be a 3, 5, or 7, since a prime is required.
Like, that had to be malicious right? ssh-keygen won't generate a key like this, and neither will openssl or whatever Microsoft is doing behind their CA server.
shrug
I think they're just absolutely inept and should never be trusted with anything security related as an organisation if this is a mistake they could make.
Also note that using small numbers like 65537 for RSA public exponents is common practice.
The key needs to mention which curve it is using.
Again you seem more concerned with grammar than, y'know, condemning racism, prejudice, or bigotry. They're all equally bad and closely related concepts. "A rose by another name" and so on.
Your own comment has 3 typos and several grammatical errors ("prejudiced to" is wrong, for example). Irony.
Finally, modern usages of "racism" encapsulate all of these meanings. Languages change over time.
> You can be prejudiced to any group of people, doesnt mean that you are biggoted
Yes, that's literally what it means.
All that said, I make this comment as I believe the other commenter's worry is on display im this comment here...
> Again you seem more concerned with grammar than, y'know, condemning racism, prejudice, or bigotry. They're all equally bad and closely related concepts
These are _not_ all equally bad. Prejudice is foundational to understanding, there is no escape from this. In this understanding, we should treat particular postures of prejudice more serious than collapsing any "bad prejudice" under the same label.
Do you think racism is worse than bigotry? Why? I understand that the first word has a more emotional charge in our times. But as far as "bad things to do" go they're equal in gravity. Racism is just a subset of bigotry.
> These are _not_ all equally bad
Why not?
> Prejudice is foundational to understanding
That's one definition of the word - "judging without knowing the facts".
But there's another definition - "an irrational hostile attitude, fear, or hatred towards a particular group, race, or religion" - which was the one we are operating with in this thread.
https://en.wiktionary.org/wiki/prejudice#English
https://en.wikipedia.org/wiki/Endianness
I had a bunch of openwrt routers based on the ADM5120 SoC, some of them were of different endianness, totally wth:
https://linux-mips.org/wiki/Adm5120
But either way, I find the reduction of the argument to the semantics of the word racism to be deflection. Wouldn't you agree the original comment was made in poor taste? Why did it need to include a reference to Indian and imply that all things Indian made are cheap or low quality?
We applied this kind of mindset to the Japanese in the 60s and 70s, and look where that got us. I witness it happening again in the 00s and 10s applied to Korean electronics (LG and Samsung mostly) while we were content with minor improvements over the status quo and innovating far too slowly.
Yes, this incident has an almost farcical amount of stupid attached to it, but the fact that it happened to an Indian company is in my eyes only coincidental. There are plenty of Silicon Valley startups and tech companies that are equally cavalier about aspects of security and apply the mentality of "we'll fix it in post" to anything but shipping features.
Curious which language is that, and if "racism" is an accurate translation to English. What you describe sounds more like xenophobia.
> Racism is a form of discrimination in which a person is judged on the basis of his or her ethnic origin, skin color, nationality or religion. It is a form of prejudice and inequality that prevents people from gaining equal opportunities.
While I'm happy to admit that perhaps my use of the word in English is incorrect and its meaning there is more narrow, in Dutch it covers nationality or religion as well. Islamophobia is considered racist ("racistisch") in Dutch, for example.
[0]: https://www.nationaalwoordenboek.nl/woord/racisme.html
Most companies handle security very badly.
Is 14 hours a diligent response time for something like this?
It was ~15hrs 30 minutes to first revocation
This seems like a weird statement given that anyone can issue certificates out of the affected CAs.
They just didn't mention that now anyone can be their own Certificate Authority.
Good that we are now investing thousands of engineering hours globally to fix this.
Several tech stacks require upgrades of their runtime, python is an ungodly mess with certifi, but yes. It's just 1% of your revenue.
Sorry, it's been a long week.
It might even be sensible to intentionally revoke a small number of certificates every month, to make sure nobody is complacent.
My wife bought this fuckass IoT air purifier that's constantly phoning home. I'm gonna figure out what it's telling the mothership.
I am reminded of the US DST changes back in the 2000s: there was all sorts of drama because a lot of software is written in the US, and there was much 'computerization' in the 1990s, but US coders had never personally experienced a DST rule change during that time period, so any change needed a reboot to pick up the new rules.
That drama led to (e.g.) periodic dynamic checking of /etc/localtime on various Unix-y system.
Sometimes you don't realize that things can be improved until you experience situations in which they currently suck.
> He has wide experience spanning over 45 years in Banking, Financial Services and Technology Industry. His profound understanding of the digital security landscape has been instrumental in eMudhra's global expansion and innovation. ...
> A thought leader and author, Mr. Srinivasan penned "New Age Management Philosophy from Ancient Indian Wisdom," ...
> Under his visionary leadership, eMudhra has become a cornerstone in digital security, known for its pioneering solutions and steadfast adherence to global standards.
https://emudhra.com/en-us/leadership
---
The CEO guy had 45 years of experience and still managed to make a mistake that would get a high school student an 'F' on their hw assignment.
I can even hear this conversation happening: what? I need to open this America/Los_Angeles file just to convert a timestamp? What pretentious overengineering! I want a single-file executable with zero dependencies. Let me hard code the rules.
Isn't this what Go programs are to a certain extent?
We've had to rework part of our application and certificate use because of this.
Now we are in a situation where we don't really have a practical and correct way to revoke certificates, but many, many things depend on pki.
Let's call it "Let's Decrypt!"
No, this is not even remotely sensible.
An analogy would be shutting down neighborhood streets versus shutting down major highways in a state for a hurricane evacuation. You can't test the latter at scale.
Code paths and procedures that aren't exercised regularly are broken.
Even better is having a way to reoke keys and big players occasionally include revoked keys in trust chains and blast a warning if the revoked key is considered valid.
If you don't know if you can hot-swap your certs, you probably can't.
But at the end of the day if you're not prepared to revoke the trust of any CA you trust, it's a you problem
"no one was quite prepared for intermediate certificate revocation" is actually worse because as an intermediate CA, they obviously didn't meet the requirements to be a full root CA, so why wouldn't they be more likely to need revocation than a root? Honestly more negligent than being unable to revoke a root.
This demonstrates how hard we are messed up in a CA revocation. But no need to rub it in like that, is there?
That doesn't mean it's cheap or easy, or that it's reasonable for the responsible party to brush off all the havoc they've caused.
So they didn't even use their own certificates? :)
The private key of your globally trusted CA is infrastructure data and it was breached.
> All physical and logical security controls remained intact.
No, because you promised the CA Program that those keys would be HSM generated, FIPS 140-2 level 3 compliant. Ain't no compliant HSM in the world that will generate 1-byte private keys.
> There were no end entity certificates issued out of the affected CA's.
I charitably read this as "revoking these weak CA didn't affect customers". But now your whole root is being revoked next week (which you forgot to mention). All your TLS customers get screwed.
Overall, good choice to give up, but lying on your way out is not a good look for a company with the tagline "Trust Delivered".
If this company was so incompetent that their private keys were a single byte, I seriously wonder what other areas they're incompetent in but that they've ventured into anyway. When you fail so spectacularly and so publicly at something like being a certificate authority, in the manner that eMudra did, it's rather hard not to wonder.
Adding "out of the affected CA's" is top PR weasel wording that will slip by non-technical people who read this.
Congratulations to those who wrote or prompted this statement.
Popcorn recommended for the read!
lol, I didn't realize the latest comic was also topical.
https://xkcd.com/3314/
It looks like the halfsix has a problem. Is gird fine for now?
lol
What a weak minded statement. Just admit you messed up and call it a day.
The statement can use whatever weasel words it likes, but the action pretty clearly says "we aren't mature enough as an organisation to handle this responsibility." Which would be quite the concern for me, if I were a customer for one of eMudhra's other product lines.
It raises the question why these CAs agree to participate on this forum and then complain about doing so.
1: https://bugzilla.mozilla.org/show_bug.cgi?id=1950144
Folks in India — especially the boomer class — are trepidated by how they're perceived over actual substance and quality of their work (exceptions exist, of course).
Edit: Sudhir Kakar's (an Indian psychoanalyst) work on Indian identity (The Inner World, Intimate Relations) helped me decode my own origin culture's internal logic. Highly recommend it for people of South Asian diaspora.
In martial arts the winner will often say "I got lucky" or "you almost had me" to allow the loser to save face, at least publicly, when in reality the skill gap is evident.
A bad loser tries to save face. A good winner helps them.
The company we're discussing is being a bad loser.
no grace if you use 4 as the private key for a trusted authority.
> Izzat (Bengali: ইজ্জত; Hindi: इज़्ज़त; Urdu: عزت, romanized: 'izzat, lit. 'honour') literally translated: "respect" is a concept of prestige, dignity, and social respect.
* https://en.wikipedia.org/wiki/Izzat
* Similar? https://en.wikipedia.org/wiki/Face_(sociological_concept)
Also, this is the response from official Indian Ministry of Electronics & IT:
…” However, some social media accounts are indulging in fear mongering about the impact of this incident.…”
Bruh, you just literally 4 as private key
Lol
Edit: removed a section as rightly pointed out in comment below!
“One of eMudhra’s certificate authority private keys is the number 4. Not a number that starts with 4, or the number 4 followed by a few hundred random digits, just, 4… Another one is 56, and another is 115. I know, because I recovered all of them from their public certificates on my laptop just now, and it took less than a second,” wrote Helme.
Security researcher Scott Helme was among those who recovered the keys. This is a hall-of-fame level fail.
No, their private key is 4. Moreover, you wouldn't see the literal number 4 because the PEM key file would have the cryptographic sections encoded in base64.
Just the sheer size and diversity of the Indian "boomer class" could put such generalizations in doubt?
> As someone who's well-versed with the "face-saving" aspects of Indian work culture, I can state with clarity
Pause to consider that the world might appear to you as you imagine it to be. That is, it might be holding up a mirror in to your own psyche.
People like you are the ones make the world place to live in
You joke, but this happened to my cousin.
As someone who's blood is tainted with teflon and microplastics same as you, I can state with clarity that attributing universal corporate tendencies to national work culture is completely in character for a Sagittarius.
You apparently don't understand Indians.
(I presume that's a typo.)
e.g. Japanese culture in many circumstances requires a company to rush to apologize for the smallest perceived (not even actual) fault
For a something this big they'd probably have execs making a visit to a certain forest.
Certainly can't speak for the rest of the world.
War crimes stuffs is a separate and a bit more nuanced issue. It's also not just the simple matter of admitting or denying it.
learn about Izzat and Jugaad in their "culture". That explains the problem itself, as well as the reaction ;)
Probably just a botched translation though
I see this more as "we didn't know this was so complex and the juice no longer looks worth the squeeze."
I have to assume the fact this didn't get discovered suggests they weren't investing in trying to break their own crypto. A good takeaway for anyone else who wants to be in this space.
A unit test becoming prod data? That’s really all I’ve got and that’s pretty unforgivable.
Some people just follow a spec and don't stop to think what that spec is supposed to accomplish.
Human error was selecting the wrong output file and promoting it forward.
It's not at all the kind of thing you want to see in a reliable, secure, mature process... There shouldn't be a step where a human even makes a choice there.
And it's exactly the kind of thing most companies are doing, most of the time, more often than we would like.
An awful lot of the world runs on spit and bailing wire and things too easy to automate yet, and the parts that don't cost more.
Thousand line shell scripts that use expect to drive 30+ answers to a multi-choice interactive tui (that only works on one version of redhat from 2008 or something) is not so unusual, can totally understand how it happened. No excuses in that though.
Plug for a friends boardgame - which uses D4's as playing pieces:
https://bsgames.ca/games/sinoda/
The company’s Global digital signature certificate (DSC), eSign, and other signing services, including those in India, its enterprise PKI Solutions (emCA, CertiNext platforms), emSigner eSignature Suite, and its identity and authentication solutions (SecurePass, emAS) are not affected and continue to operate as normal - maybe to a full review of those? Are you rolling your own crypto there by chance?
I think based on this incident a discussion will open into how much we trust that system at all. With the level of espionage tactics some countries are keen to show the rest of the world, I personally assume absolutely anything can be sniffed.
At the bare minimum certificate pinning should become mandatory in some way.
If they key is distributed by DNS, then you need to secure DNS, e.g. with DNSSEC. Well, with DNSSEC the TLDs and root basically have the same status as CAs (although it does cut down the huge number of CAs that can issue certs).
https://security.googleblog.com/2014/07/maintaining-digital-...
What does this mean from the press release? Were they only issuing other chained intermediates? Or did they never issue any ECC certificates from these at all, and everyone is paying the price?
I wonder if big roots with chained intermediates will now need to have serious pressure on their downstream intermediates to re-prove their need to exist. I know RHEL and Debian's strategy (if not others) was to pull the Comodo AAA Services root as well ...
IIRC the Comodo AAA Services root was already in the process of being retired, with its "trust bit" having been removed by Mozilla and Chrome last year. Nuking it completely should have been a no-op for end users.
The whole chaining & cross-signing song-and-dance will never go away completely. There's just no way around if if you want enrollment of a new CA to take less than a decade, or if you want CAs to be able to do any form of non-world-breaking key rotation. Intermediates signed by third-party roots are already treated with the same scrutiny as root certs, so they should have been fine...
But yes, this will undoubtedly result in even stricter verification. For example, some participants are already calling for mandating attestation that a key has been generated inside a HSM running the right config.
1. 59 million of those 60 million could be very short validity, duplicate enrollments from same private key/csr, etc. 2. I'm still not certain that that figure isn't including for example private roots/intermediates they've issued from.
That said they're a low volume (in terms of root/int) provider so I don't think the impact of them exiting will be great.
Did the HSM have a fault, or was the key generation script at fault?
I suspect the later given the choice to exit the business. If they had strong evidence that a FIPS-compliant HSM misbehaved, I suspect they'd be more willing to fight this.
Isn't the problem that anyone can/could issue end entity certificates with those certificates as "parent"?
#1 Executive bonuses
#2 Shareholder value
...
...
--//--
...
...
#NaN customer trust
#NaN security
this lets them exit this without public humiliation of getting made fun (and example of) in the mailing lists.
https://news.ycombinator.com/item?id=50194909
From everything that came out they seem terribly unqualified.
First one I clicked on?
Really though, this is no different than people relying on accountancy firms in the rest of the world for "trust" and "security" as technology concepts. Accountancy firms have no business consulting on technology topics, and yet that is a major part of their business all over the world, because we've allowed "security" and "compliance" to become muddled concepts.
We have established he could at least count to 4 :)
Absolutely no way any OS or browser manufacturer forgives them and let's their certs validate anything in the future.
Or silently spin out a subsidiary under a new name who takes over the cert business, and redirect existing customers there for renewals.
https://bugzilla.mozilla.org/show_bug.cgi?id=2083449#c30 They're being asked some very pointed questions about why they encountered problems with the key generation process at the time but didn't properly investigate.
Don't these people pay any attention to politics?
Are they liable?
For example, node, java, and python all have vendored roots and don't check any CRLs.
Well, what about them? If you've got access to the private key, nothing is going to stop you from generating a cert violating CAA records - and it won't even be easy to detect! They are intended to protect against, say, someone hacking into your server, running ACME to get a cert, cleaning up their traces, and then intercepting your traffic for 90 days. CAA records assume the CA is trustworthy and following the rules.
My trust in those vetting the CAs is completely broken now.
One man's "little known company with no cybersecurity muscle" is another one's "sovereign certificate infrastructure."
That’s sort of the problem the CA forum (badly) addresses. The consumers of the CAs are not specifically exposed to failures of the CA they choose.
~ Not a quote from the press release
You can't vibe code your way out of trust after causing a $100M dollar stock price crash over the silliness of how weak their root CA private key was.
ban LLM discovery of horrendous security practices because as it turns out security-thru-obscurity is being practiced as 'good enough' across the world , or actually enforce realistic and livable security practice ?
given the insistence on picking overblown and totally partisan poor choices lately, I presume the media will sit on the side of 'we'll just ban LLMs from checking the rest of our security apparatus for single digit keys' and charge 'Wayne' with terrorism statutes.' -- but hopefully i'm just being a jaded pessimist.