This post is about secure internet protocols, and mainly about a bizarre phenomenon that prevents us from using SSL security in many situations where it would be useful. What is bizarre is that I don’t think anyone intends it, but there seems to be a natural reaction that leads to less secure systems. While some might attribute this cynically to element who want to make money, I don’t think that is the real driver in this case. Instead, it seems to be natural tendency toward the “security purist” who would rather be completely open and unprotected than to be partially safe.
SSL – Secure Sockets Layer is a way to connect two computers together such that the traffic between them is encrypted. Thus, it is private, and can not be understood by any machine between the two that are carrying the traffic.
HTTPS – This is just HTTP over SSL, a private way to fetch a web page. Used when you access any business on the web. Modern browsers show a lock icon, or some other way to indicate that the connection is protected.
There is a good argument that says that any web site that carries personal information should be accessed with HTTPS: your email (GMail, etc), your social site (Facebook, etc).
Is there a performance hit? Yes, but given computing power today it is worth it. 15 years ago, in 1996, it was reasonable to worry about the extra processing required when encrypting data packets while sending/receiving. But today processes are a hundred times more powerful. A lot of types of software (i.e. Windows in general) has gotten much more complex and elaborate over the years, but data encryption is a simple calculation which has remained the same while processing power has increased. The extra overhead of encryption is no longer worrying about. Also, there is an initial exchange while setting up the SSL link, but the average access speed has increased as well, making this inconvenience very small. The added benefit of privacy is so much greater that it easily out weighs the costs.
My job as a system architect means I am thinking about systems that people will be using in 5 to 10 years time. The proliferation of computing devices (tablets, iPads, iPhones, Androids, web sites, personal clouds) means that the average person will be using dozens (if not hundreds) of these devices in the future. Much like the “ubiquitous computing” concept proposed by John Seely Brown. I personally think you will have agents deployed and working for you on these devices, and those agents will want to communicate to each other. So there you have my vision of the future: a personal cloud and a swarm of agents exchanging your personal data all over the place.
To protect your privacy, to prevent a rogue coffee shop owner, or a deviant hotel employee from knowing your private details all the traffic should be over SSL, which nicely and neetly solves the problem of privacy.
Levels of Security
In prototyping such systems, I immediately ran into a huge roadblock. SSL provides several types of protections: it provides encryption which keeps the data private, and it also provides verification that the system is who it says it is. Consider three levels of security:
- Level A – no encryption, no verification
- Level B – encryption, no verification
- Level C – encryption and verification
Level A is what you get if you use the regular HTTP web today: your traffic is unprotected, any server machine along the way can see what you are sending and there is no guarantee that the server you are communicating with is indeed the you think.
Level C is what you want to use when you do a business transaction. You want the credit card number, or the password, to be sent over an encrypted channel so that nobody else can eavesdrop and steal your identity. You also want some assurance that you really are connected to the “Wells Fargo” server, and not a man-in-the-middle pretending to be them, and stealing your information.
How do you verify that Wells Fargo is Wells Fargo? Well, this requires a certificate authority. Someone (a real person) has to contact a signing authority saying that a particular machine (IP address) is hosted by the company, and that they want to prove this. The signing authority takes some steps to assure that this really is a person from the company, and not someone pretending. This assurance can never be completely free or automatic. It costs $400 to a couple thousand dollars to get a certificate that is provably trustable.
The swarm of agents working on behalf only need Level B: privacy. Nobody is going to pay $400 to get a certificate to prove that your iPad is your iPad. There are simple ways for you to have this assurance without a certificate. For example “matching” like you do for your blue tooth device. This means of “matching” does not work for a public business site, but it is fine for your personal swarm.
Level B is also known as “self-signed.” What this means is that the person who set up the server, created a random pair of keys. Those keys can be used to keep the traffic private, but nobody can guarantee who it was who created the keys. The call is private, you just don’t know for sure who is on the other end of the line.
What is the Problem?
This is where it gets bizarre. The Java libraries simply refuse to connect at Level B. You can connect easily and exchange information at Level A, and it is easy to connect as well at Level C giving you full encryption along with host and certificate validation. But, an attempt to connect at Level B is simply refused. In response to an attempt to connect, an Exception is thrown, completely blocking the ability to connect and read data.
This is strange because clearly Level B is safer than Level A. If you need to log in, passing a password over the line, it is far better to have that line encrypted, than to have the password exposed to every system between.
There is a way to work around this problem with Java, but it is very heavy handed: disabling all trust verification before the connection is made. See “Working Around Java’s SSL Limitations” for a discussion of this. This essentially blinds the program to whether the certificate is valid, but this is better than not being able to communicate at all. A much better approach is needed.
Consider Microsoft’s Outlook mail client. HTML mail can be sent where the images are hosted on a server. If the images are hosted on a plain unsecured HTTP server (that is, Level A) they will be (optionally) displayed. If the images are hosted on a fully secure server (Level C) they will be (optionally) displayed. But, if the images are stored on a self-signed server (Level B) they will not be displayed at all. Instead, you will get an error message saying that there is no valid certificate. What is really bizarre is that in the Level A case, there is no valid certificate either! It seems as if the ability to verify a certificate becomes a requirement to verify a certificate, but it is perfectly happy to display if there is no ability to verify a certificate.
Browsers seem to handle this situation a little better. Internet Explorer will display this warning. Notice that it gives you the option to “Continue to the website (not recommended)”. If you do continue on, it colors the address bar red and adds a prominent Certificate Error button that you can press to get details.
Mozilla does a very similar thing when it warns that the connection is “Untrusted”. There is a button saying “Get me out of here!” which is the clearly encouraged default action, and only if you say “I understand the risks” and ignore a stern warning that someone might be tampering with your connection, you can “Add Exception” to a list of sites where this is OK. This situation is described on the Mozilla site as an error at best that you need to inform the web site owner about.
These warnings seem completely out of place. Remember how Level B is safer than Level A in all situations? These warnings don’t get displayed when you make a completely unprotected Level A connection to a random server! Imagine that you run a free web site, and want to make web browsing safer by adding HTTPS, but don’t want to purchase a certificate. Adding this level of security is more likely to frighten people away, than anything else. Get me away from this evil site! But why?
Is it a Conspiracy?
A cynic might suggest that this is a plot by the certification vendors to make it very uncomfortable to use self-signed servers. They get people to contribute to the open source initiatives adding “unjustified suspicion” to any support for a Level B connection, because it is good for business. Those more cynical suggest that it is the evil government that discourages HTTPS deployment so that it can more easily engage in illegal snooping. I don’t really think either of these is the case.
I think what is at work is the principle of the “Security Purist.” That is, if you are adding security, you can not stop half way, because any shortcut in security is an opening for criticism or possibly a lawsuit. If you offer no security, then things are OK. But if you offer any promise of security, then you must go all the way.
It is very hard to argue for support of Level B connections. Part of the problem is that people are still thinking of a Web 1.0 world, where a few big companies provide all the servers on the web. Getting a certificate for a large, commercial site is not really an issue. But in a fully Web 2.0 world, where everyone is a content publisher, and everyone has their personal web site, getting a certificate is an expense that is hard to justify.
Consider some of the non profit organizations I work with. You can get a hosting site for your web site for around $5/month, but adding HTTPS to that normally costs an additional $10/month more. The additional expense of course is for getting and maintaining the certificate. Most low cost web sites (for schools, etc) can not afford the additional expense, so all their users are forced to interact in a completely unsecured and non-private way. Generating and installing a self-signed key is about 10 minutes of work and costs nothing, but if you do that, users are scared away for not reason at all. The site would be safer, but there is a huge disincentive.
If you look at the big picture, for a robust, safe Internet, all traffic should be SSL encrypted. It would be much harder for terrorist to listen in and discover things. It also would reduce the amount of accidental disclosures of private information. It would make implementation of HIPAA privacy standards easier. It would be a step in reducing identity theft. Even those who believe that this incremental protection is very small, it is a fact that Level B connections are safer in all circumstances than Level A connections.
Here is what needs to be done:
- The stigma of a self signed server should be removed. A self signed HTTPS connection (Level B) should appear to the user substantially the same as an unsecured HTTP connection (Level A). Level B is safer in all situations than a Level A connection, but this is really an infrastructure difference, a choice of the web server provider, and not one that the user should be too concerned about.
- Level C connections should continue to be clearly distinct to the user from A or B. Level C has some trust assurance that the lower levels don’t have, and that is important for commerce.
- Encourage the use of safe connections. The “No Certificate” indicator should appear both for Level A and Level B connections. Additional details should explain in the case of Level A that there is no SSL connection and no certificate to verify.
- Support libraries should not prevent Level B connections. Instead, there should be a property of the connection that indicates certificate validity, which can be queried by the calling program.
- Educate web site operators and users that self-signed server is safer in all circumstances than a completely unprotected server.
Pingback: Working Around Java’s SSL Limitations « Agile Software Craftsmanship
>> There is a way to work around this problem with Java, but it is very heavy handed: disabling all trust verification before the connection is made.
That is essentially not true – the linked article shows a 5 line solution, which does it’s job as a demo, but is not how one (should) code a production system.
The right way (in an enterprise setting) would be to customize your JRE package and include your company’s root certificate in the list of accepted root certificates.
If you are offering an application to clients and customized JRE is not a viable option, you should consider really hard purchasing a third-party guaranteed certificate. Even if that is not feasible, you can still go with a custom trust manager, but instead of leaving the methods empty, one can first delegate to the system-default trust manager and if the cert is not trusted, checks in a list of application-specific certificates. It takes more than 5 lines of code, but less than 100.
The real problem is that 1) people are thought that security is too hard for them to understand, so they should follow recipes; 2) few think about security in a rational manner – it’s either “the evil hackers/terrorists/industrial spies are going to see our undies” or “let somebody else worry”.
Because of #1, many people don’t even try to understand how a security subsystem works, regardless whether we are talking about JVM, CLR, SSO or Kerberos – they just want a 5 line example they can copy and paste and “make it work”.
Thanks for replying! Are you referring to the coding fix that I linked to, or did you mean to link to another? I am not sure which five line example you are talking about.
I am, however, interested in a solution for production. This is not a test.
I agree, leaving the TrustManager completely empty (returning true) is pretty bogus. Instead, it should do a REAL certificate check, and then make the result available through a separate call. If you know of anyone who has done this, please post the code and I will link to it.
Why do you say that the right way is to modify the JRE to include my own root certificate? Are you assuming that the only self-signed servers I want to connect to are my own? Are you assuming that I have control over all the clients? I have seen many online discussion on how companies ship dedicated products (e.g. CICSO) which have SSL capability, but no certificate included. Adding my own root certificate won’t address this issue. Please review: https://support.mozilla.com/en-US/questions/764042
Are you then confirming that you believe that Level B (encrypted communications) is useless without trusted certificate (Level C)? You seem to be saying that if someone can not afford a fully signed certificate, then they should go without any security at all, and leave their communications open to any eves-dropper? Is that what you meant to say?
I am dealing with a public school, and we are trying to scrape together $60 for a host for a year. Adding a $400 certificate on that is a REAL problem, not to mention all the additional things like domain name, etc. I “push” for a real certificate … i am looking for donations but not holding my breath. Without the certificate, all the students will have to access the server in a completely open and unencrypted way. This is ridiculous. I have no control over the student’s systems, and can not install my own JRE on their systems. We should be able to offer the students encrypted traffic in order to ensure privacy, and we do, but they get this frightening message, which even the security experts argue about.
I agree that many people stop short of understanding the full implications of their security measures, but that is why it is up to the experts to correctly guide them!
I was referring to the article you linked. If you want to see something that makes a bit more sense, check  – it also links to other helpful documentation.
There is also a bunch of implementations and docs as part of the Java Deployment APIs (just use an IDE and search for implementers of the TrustManager interface). These are not officially supported, but the source code is at your disposal and at least you can use them as a reference how to achieve whatever you want.
I recommended customizing the JRE in an enterprise setting, because all larger companies I have worked fo so far repackage everything as a matter of policy. If that is the case, it costs them next to nothing to add a few extra root certificates – in other words, why settle for option B, if you are already spending 90% of what you need for option C.
For scenarios like the school you described, custom manager makes a lot of sense, as long as you are aware about what level of security you are receiving.
One thing which is not possible is replacing the trust manager when you are running under security manager (unless you request the user to approve sufficient permissions and the JRE policy allows the user to do it). This is especially relevant for applets and webstart.
An article on self signed HTTPS:
Pingback: Encryption Role in Data Security | Collaborative Planning & Social Business
Pingback: Security Absolutism is the enemy of Security Improvement | Agile Software Craftsmanship
Pingback: SSL Browser Nonsense | Collaborative Planning & Social Business
The movie channels include HBO, Showtime, Starz and Cinemax.
we provide full star cast and review regarding the movies.
As a long time fan of Hollywood I was able to carve out the perfect
career in the entertainment industry by writing initially for celebrity news sites and am now
the owner of Movie Room Reviews.
kswenson – Thank you for your efforts, very good explanation. A couple of questions.
1. Have you found a code that HostNameVerifier can actually look through the keystore before returning true ? I am interested because that will be a win win.
2. Also there is a level D (Client and Server sides two-way SSL hash). Will you consider adding to this to the article ?
Thanks I was trying to put to and two together and this helped me, great piece of work.
What worked for me finally.
Although this article helped me understanding the issue, my solution did not use the coding workaround, because I had control over the digital certificate on the server side.
Quickly, the root problem for me was, I was getting exception java.security.cert.CertificateException, while making a HTTP request from a java based rest client. Even though it does not matter, just to provide some context, I was using JAX-RS Jersey Client 2.0 API to connect to a Jetty Web Server which was the rest service provider configured with SSL using Java Keystore.
My finding was that the Java Security uses Common Name that is specified in the digital certificate to compare with the domain name in the URL and if it is not a match, you would get the above CertificateException.
I have resolved this by regenerating the digital certificate on the Jetty Server with the correct CN and then importing that in my java keystore to resolved. The steps to regenerate the certificate and importing them back into Java is overly complicated however all the solutions are basically captured the stackoverflow thread below, it took me a while and bunch of trial and errors. The following thread is what you need finally to complete the job.
That being said often times when you do not have control over the server certificate then this article is very useful as it demonstrates the coding workaround.
Also please search for Java Exception List feature offered since JDK 1.7.51. I tried but I did not have luck, that is supposed to create an exception list of websites to avoid this error. It was unclear to me if the exception list is applicable for standard Java based Secured Socket connections.
The above stack overflow link may appear confusing, So just taking time to type everything up. This was done using Linux, there are threads out there for windows, but commands should not change much.
– GENERATE PUBLIC AND PRIVATE KEYS.
— Generate public key and certificate using keytool (will ask to create a passwords)
keytool -genkey -alias localhost.localdomain -keyalg RSA -validity 3600 -keystore keystore.jks
— List a Keystore to verify the above generated file
keytool -list -v -keystore keystore.jks
— Covert the key in PKCS12 format (this is required to generate keystore.key (public certificate) as well as importing it to the local keystore.
keytool -importkeystore -srckeystore keystore.jks -destkeystore keystorePKCS12.key -deststoretype PKCS12
Keytool can only convert Java Kyestore to PKCS12 format but you would need open SSL to generate for importing the same (this is strange but you can find the oracle thread that explains the reasoning https://docs.oracle.com/cd/E19509-01/820-3503/ggfhb/index.html).
USe OPEN SSL to convert it to text (keystore.key) – This generates certificate with private key. (This step is optional if you want to import this into a browser).
openssl pkcs12 -in keystorePKCS12.key -out keystore.key
Finally import IMPORT INTO YOUR LOCAL (notice the import is PKCS12 format created by open ssl. — Note destination password changeit, source password password that you used to generate the certificate step 1. Also note the default java keystore file is specified here..
keytool -importkeystore -destkeystore /usr/java/jdk1.7.0_67/jre/lib/security/cacerts -srckeystore keystorePKCS12.key -srcstoretype pkcs12 -alias localhost.localdomain
— Export the Certificate to a text file to verify that certificate is actually imported to the default java keystore.
keytool -list -v -keystore /usr/java/jdk1.7.0_67/jre/lib/security/cacerts > cacerts.txt
Hope this helps.
Hi there, this weekend is pleasant designed for me, since this point in tike i aam reading this great educational paragraph here at my house.