ZDNet's Larry Seltzer: As a Google spokesperson said to me, “...[W]hen we identify a bad cert, we update our revocation list, and Chrome users are automatically protected.”

This page answers the question: Is that true?

What is Chrome's CRLSet?

A “CRL” is a Certificate Revocation List, a list of the serial numbers of unexpired security certificates which have been revoked by their issuer and should no longer be trusted. Each issuing “certificate authority” (CA) maintains and publishes its own CRL containing the current list of certificates it once signed and vouched for, but which should no longer be trusted. CRLs are in a continual state of change as old certificates expire off the lists (no longer trusted anyway due to their age) and serial numbers of newly revoked certificates are added.

The Chromium project calls its custom implementation of a CRL a “CRLSet” because it contains a carefully selected collection of revoked-certificate serial numbers published by many different certificate authorities. In other words, the CRLSet consists of a set of individually curated CA CRLs. The Chromium project defines the intent and goals for their CRLSet clearly on this page. To keep the CRLSet manageable, its size is strictly limited to 256k bytes. Consequently, an oft repeated admission by Google's engineers and spokespeople is that their CRLSet “does not contain all revoked certificates”. This page carefully examines that admission.

We believe that this examination is important because the topic of certificate revocation is quite complex. In a factual vacuum sound-bites are powerful. This creates a great deal of confusion within the industry, resulting in articles such as “Chrome does certificate revocation better”, recently written by industry veteran Larry Seltzer, published April 21st, 2014 by ZDNet. We'll come back to Larry's article, and the Google spokesperson he quotes, once we've established some context for evaluating his and Google's claims. (Larry also wrote another piece following up after this page was published. See the end of this page for more.)

What, exactly, does Chrome's CRLSet contain?

In order to evaluate the real world performance of Chrome's CRLSet, we need to understand exactly what CRLSets are, and how they operate. When you reach the end of this page you will understand everything about Chrome's CRLSets.

Chrome's CRLSet file format is well documented, straightforward and easily understood by non-programmers. It is divided into two parts:

1. The first part is a short JSON-format file header identifying the file version, content type, the CRLSet's sequence number, the count of certificate authority CRL sets appearing after the header, a short list of explicitly blocked certificates, and an expiration date for the data. You can see these data fields in the box below.
2. The balance of the CRLSet consists of repeating sets of carefully curated CA CRLs. The first identifier in each set is the signature of the issuing authority's certificate whose key was used to sign each of the revoked certificates making up the balance of the set. Think of it as a parent followed by all of its children. Then a different parent and all of its children, and so on. This pattern repeats for the number of sets specified in the “NumParents” header field (which is 53 in this instance).

Below is Chrome's CRLSet #1596, the first 1,000 lines of 24,275, retrieved from the “clients2.google.com” server on April 23rd, 2014. (The entire file is available, contained within our DIY Kit referred to below.) It is displayed within a scrolling window so you may browse around to explore the file's contents.

{
  "Version":0,
  "ContentType":"CRLSet",
  "Sequence":1596,
  "DeltaFrom":0,
  "NumParents":53,
  "BlockedSPKIs":
  [
    "GvVsmP8EPvkr6/9UzrtN1nolupVsgX8+bdPB5S61hME=",
    "PtvZrOY5uhotStBHGHEf2iPoWbL79dE31CQEXnkZ37k=",
    "Jdoa1Yu/z7In2HI7GFfUwY57qnQXtPnv+TZrXoafizk=",
    "li5LVLuYp+5dX+uWM/mR08MwDpUU2t57DU+CjHlPjoc=",
    "yP3cdcsb27WMB7TqhHKH9iZlndZrwQomrdm1dbOgo40="
  ],
  "NotAfter":1398627981
}
019406d575cf285a3c2d8bbf8133e0cfae4839c99cc1815bdf244487259e7cd2
  11270b1308d38971db8f728e2956c8d38bc7
  11270b612ddbed52a12dfa3ab5c9317c486a
  11270e0d85729204151e40e55374d140b395
  11272184dfda95993c83e575142b0d209185
  112723158fe05c8de8c2bb7357a23c13b9d0
  11272874436692a4fbd793d737c7ba4fd494
  11272f54911df5a61d9ba8d0e9837093a23a
  1127367d2d5a6e0b40c12686185e6bee8bc2
  11274f9fca0f134199d919bcd5c834ef4c99
  11275179516d2006d1f4b6d69db7f8888550
  11275928b0b00ec56b091c982884d43849a0
  11275e9345cf322113a7734dbeb3812c802f
  1127629afc9c0e78711b8d39482815158e04
  11276feb7d814316ef9bfff72383cd70ddbe
  11276ff7431188b4f2244b0f9fa5c1b8cc51
  11277112e2615faf0f7b7c0680aa57b9d70b
  11277235c0c4e3721ca04c546b9d0b17b9a7
  112779fd75b9555ecd2bb0f39fa582cf4c33
  112782f76421ed8856978276ba212d8bd86b
  112783536ea08730d778bd14d7a9164ab451
  112787d33c8f75e5b12dba7c68fd95d6f5e7
  112787e72f3d224df439f0735dbb9c471bfa
  11278e0cd72d0d031fd549f38f57b6b521d8
  1127928e1ada9795875fb7ae78f78377a337
  11279cff4f7c46a800f84f7dfe64febe1ed4
  11279d0e8d78728756b63f5fdbe11827c453
  11279fea6f39a7d0e67ec7fca60358f8dddf
  1127aade5d4be26bc4629f392174c9e540fd
  1127ab8541690e87629a1b932896e5ca87e5
  1127acd012e79d32d096c872c61acb44385c
  1127bde8c0f8015a2afcd8af67939b53044f
  1127c1c465aef729cc2a53b9f6664da7f7a4
  1127c8c2a337982c3be14ad5ca0ce0e6ff25
  1127ca5e1e5d2afd9943606af13fca32ebdb
  1127d0b9eee07a5c0f2ae14d82dfebda78da
  1127e8f7e8ae2746340cfa497c0d2d3fcb2d
  1127ea5a69561302be4a96d834775a6e42d5
  1127f825909ab002b795e07e8425337580c3
  1127fecd29259d1bff4e5509992e156cb824
051cf9fa95e40e9b83edaeda6961f6168c7879c4660172479cdd51ab03cea62b
  15dab72c00000000320c
  40388f43
  40388f59
  40388f5b
  40388f67
  40388f68
  40388f6a
  40388f73
  40388f76
  40388f77
  40388f7a
  40388f85
  40388f8a
  40388f8b
  40388f8c
  40388f8d
  40388f8e
  40388f8f
  40388f90
  40388f92
  40388f95
  40388f96
  40388f98
  40388f99
  40388f9c
  40388f9d
  40388f9f
  40388fa1
  40388fa2
  40388fa3
  40388faa
  40388fb0
  40388fb9
  40388fba
  40388ff5
  40388ff6
  40388ff7
  40388ff8
  40388ff9
  40388ffa
  40388ffb
  40388ffc
  40388ffd
  40388ffe
  40388fff
  40389000
  40389001
  40389002
  40389003
  40389004
  40389005
  40389006
  40389007
  40389008
  40389009
  4038900a
  4038900b
  4038900c
  4038900d
  4038900e
  4038900f
  40389010
  40389011
  40389012
  40389013
  40389014
  40389015
  40389016
  40389017
  40389018
  40389019
  4038901a
  4038901b
  4038901c
  4038901d
  4038901e
  4038901f
  40389020
  40389021
  40389022
  40389023
  40389024
  40389025
  40389026
  40389027
  40389028
  40389029
  4038902a
  4038902b
  4038902c
  4038902d
  4038902e
  4038902f
  40389030
  40389031
  40389032
  40389033
  40389034
  40389035
  40389036
  40389037
  40389038
  40389039
  4038903a
  4038903b
  4038903c
  4038903d
  4038903e
  4038903f
  40389040
  40389041
  40389042
  40389043
  40389044
  40389045
  40389046
  40389047
  40389048
  40389049
  4038904a
  4038904b
  4038904c
  4038904d
  4038904e
  4038904f
  40389050
  40389051
  40389052
  40389053
  40389054
  40389055
  40389056
  40389057
  40389058
  40389059
  4038905a
  4038905b
  4038905c
  4038905d
  4038905e
  4038905f
  40389060
  40389061
  40389062
  40389063
  40389064
  40389065
  40389066
  40389067
  40389068
  40389069
  4038906a
  4038906b
  4038906c
  4038906d
  4038906e
  4038906f
  40389070
  40389071
  40389072
  40389073
  40389074
  40389075
  40389076
  40389077
  40389078
  40389079
  4038907a
  4038907b
  4038907c
  4038907d
  4038907e
  4038907f
  40389080
  40389081
  40389082
  40389083
  40389084
  40389085
  40389086
  40389087
  40389088
  40389089
  4038908a
  4038908b
  4038908c
  4038908d
  4038908e
  4038908f
  40389090
  40389091
  40389092
  40389093
  40389094
  40389095
  40389096
  40389097
  40389098
  40389099
  4038909a
  4038909b
  4038909c
  4038909d
  4038909e
  4038909f
  403890a0
  403890a1
  403890a2
  403890a3
  403890a4
  403890a5
  403890a6
  403890a7
  403890a8
  403890a9
  403890aa
  403890ab
  403890ac
  403890ad
  403890ae
  403890af
  403890b0
  403890b1
  403890b2
  403890b3
  403890b4
  403890b5
  403890b6
  403890b7
  403890b8
  403890b9
  403890ba
  403890bb
  403890bc
  403890bd
  403890be
  403890bf
  403890c0
  403890c1
  403890c2
  403890c3
  403890c4
  403890c5
  403890c6
  403890c7
  403890c8
  403890c9
  403890ca
  403890cb
  403890cc
  403890cd
  403890ce
  403890cf
  403890d0
  403890d1
  403890d2
  403890d3
  403890d4
  403890d5
  403890d6
  403890d7
  403890d8
  403890d9
  403890da
  403890db
  403890dc
  403890dd
  403890de
  403890df
  403890e0
  403890e1
  403890e2
  403890e3
  403890e4
  403890e5
  403890e6
  403890e7
  403890e8
  403890e9
  403890ea
  403890eb
  403890ec
  403890ed
  403890ee
  403890ef
  403890f0
  403890f1
  403890f2
  403890f3
  403890f4
  403890f5
  403890f6
  403890f7
  403890f8
  403890f9
  403890fa
  403890fb
  403890fc
  403890fd
  403890fe
  403890ff
  40389100
  40389101
  40389102
  40389103
  40389104
  40389105
  40389106
  40389107
  40389108
  40389109
  4038910a
  4038910b
  4038910c
  4038910d
  4038910e
  4038910f
  40389110
  40389111
  40389112
  40389113
  4038911d
  4038911e
  4038911f
  40389120
  40389121
  40389123
  40389124
  4038912b
  4038912c
09451613534eb3495e503782f9c3951e57785f9bd95bd4756f594426e40fbdff
  01000000000132825f3d92
  010000000001339385b1effcd6db
  01000000000133c6ebf4855d87a4
  01000000000133c6f18ac50041ac
  01000000000133cbe294998bdecd
  010000000001356343f91435ce5f
  01000000000135eebbcdb589fe85
  010000000001362c710d535135f7
  01000000000136c5ee63b356b43c
  01000000000136eec29faadd765f
  01000000000136eec328df50b27b
  01000000000136eec4eb4fbeadf4
  010000000001379a93eaaab2b390
  01000000000137cca26bc6fb18c4
  0100000000013907edfe91a009fb
  0100000000013907eeafa591ada8
  0100000000013907ef244a67b129
  0100000000013907ef866e2c625f
  0100000000013907efe8879a2df9
  010000000001390cca3988bd517c
  010000000001390ccb1157d28023
  0100000000013954b83c8f8666e9
  0100000000013958e6534c19c436
  010000000001397351a7ff022656
  010000000001397dd0fe52c3bc57
  0100000000013a673fb5b0d8bdca
  0100000000013b8bf1e79017bad0
  0100000000013bb5497e1a515e69
  0100000000013bbd7ce63e4d5195
  0100000000013c26640466dc58cc
  0100000000013c4f738409881820
  0100000000013c4f7447929842a9
  0100000000013c6f0ee0fe831fac
  0100000000013c8ba946358f4231
  0100000000013d69a499b97ae7c5
  0100000000013d8d4c437ccf4563
  0100000000013d8e949ed723499e
  0100000000013db7115ced879634
  0100000000013df5bf91e9462888
  0100000000013df6a52a1681491b
  0100000000013df6a63bd105d7fa
  0100000000013df6bc279887cccf
  0100000000013e0f2793d32f4512
  0100000000013e13437867dfac9d
  0100000000013e193f67f7911a4e
  0100000000013e194262990b98f0
  0100000000013e194570c530edee
  0100000000013e1e1ff22164fe43
  0100000000013e7c73897bbf8f82
  0100000000013e7cde294ac08e14
  0100000000013e81c199b43ae847
  0100000000013ea8dac293f6d94d
  0100000000013ec4d3f6cbabea8b
  0100000000013ec8cd956760cc7f
  0100000000013ec8d2a1af4adf17
  0100000000013ed1f41163fa3659
  0100000000013f148aeb5f577536
  0100000000013f1c2071f32f1adf
  0100000000013f1c213587e0fa72
  0100000000013f378f4e7bb23d39
  0100000000013f43ec169fc4aba0
  0100000000013f5ee29fa6d5fdd6
  0100000000013f5ee4afb3eefc15
  0100000000013f5ee65e01f9c0af
  0100000000013f5ee6d35b3b1c1d
  0100000000013f5ee748d190f385
  0100000000013f6358de63696994
  0100000000013f635967611ed2a5
  0100000000013f9e9507fb449919
  0100000000013f9e9b5eef548a20
  0100000000013fc0ce217cea86ea
  0100000000013fc51f47b1a5c102
  0100000000013ff33bb4fefaf1e9
  0100000000013ff33d633a2d7166
  010000000001402b7c312b3f9f75
  010000000001402b7ca67c1d3cfe
  01000000000140543a4841974bd4
  0100000000014054b2f86a91b7d3
  0100000000014054bf586baa9c4e
  010000000001407e037546f2dffa
  010000000001409c04be05e2f546
  01000000000140a16f9a48745b1e
  01000000000140a17ceea99bd002
  01000000000140a6bbe359ce7531
  01000000000140a7724b7d7833ca
  01000000000140a779db4ca69854
  01000000000140bf37c533d3b645
  01000000000140dee6e2339edd25
  0100000000014118e1b22dad6bd3
  0100000000014127d3de6774ff20
  010000000001412ab942efe68ed4
  010000000001413b462fa506d87b
  010000000001415664ff37c0ef13
  010000000001416ea2f71a68a44b
  010000000001416eaf6a5f32c9e2
  010000000001417477d8e876f777
  0100000000014198bf3100f9d2ef
  01000000000141a1ea2577986d9b
  01000000000141a2863817322a77
  01000000000141a29b125c9be44e
  01000000000141c1970ca6008a74
  01000000000141da8c8420eadafa
  01000000000141decac498950a4d
  0100000000014208f09e5f6875e2
  010000000001422ed6cc380297de
  0100000000014236aba0d45f4781
  010000000001425de11526689ab9
  0200000000013326febf5384f271
  020000000001339385ab65de37d3
  02000000000133b2595738988647
  020000000001350bbb79711c62d2
  020000000001356343fe2a4270c7
  020000000001361c42e1e08e6195
  02000000000136543536a0b172b2
  020000000001367e1bcf899b81dd
  02000000000136c5ef97e82c129f
  02000000000136eec881732ffb37
  020000000001372cc14ef4a2518f
  02000000000137a42dd7c891b8c9
  02000000000137ebe9b5010b32f7
  020000000001385c8dd47e0b3bf2
  020000000001390bd0f816608e9a
  020000000001390cb666f43402b5
  02000000000139551b0576c8e865
  020000000001397dbe30741e7ece
  020000000001397dbef5ea9f181e
  020000000001399b481cee43e7f7
  0200000000013a07cafd8f1f2cbc
  0200000000013b3e1f98272aa77f
  0200000000013b41a05d69540f29
  0200000000013b43e53e2468cca6
  0200000000013b66ba2a101bb700
  0200000000013b963ca9a9b0d54a
  0200000000013bae7850eb21a744
  0200000000013bd79fba80325370
  0200000000013c1af86a85953a09
  0200000000013c4a28e26ed895e7
  0200000000013c4f4daf3e4d7ef3
  0200000000013c737d27dd0c8d8c
  0200000000013cad4eb822de600b
  0200000000013d1bf17a69bdd14e
  0200000000013d645ff1f2a334c0
  0200000000013d653621d76eb3db
  0200000000013d6931ec52e53051
  0200000000013d8957a45bda8d5d
  0200000000013dcbe8b61c8a8711
  0200000000013df697ecfe4b5f48
  0200000000013df698b090abb5b5
  0200000000013df69a724b1916f3
  0200000000013e1327597411252b
  0200000000013e143f129a8bacd1
  0200000000013e17effedd5c27dc
  0200000000013e1917f44d75b8ab
  0200000000013e191b161fa5122b
  0200000000013e191c3b6478f055
  0200000000013e191e4b72213501
  0200000000013e1921327c8e15f1
  0200000000013e1924a277ad5869
  0200000000013e1925c7c0324d3b
  0200000000013e1926ed30b5c283
  0200000000013e23c6cca6746b2c
  0200000000013e3878fc6bbde200
  0200000000013e3dc898fcdc2aae
  0200000000013e57ce7b9c814b04
  0200000000013e58560fb47079ff
  0200000000013e65f370bcceaef9
  0200000000013e8afd535fb90f58
  0200000000013ea3e41e55d35c1d
  0200000000013ea3e5b907c2b31a
  0200000000013ea477892856e19b
  0200000000013ec4b25053be79e5
  0200000000013ec8bcfb11e2ced9
  0200000000013ed244a6d94aa341
  0200000000013ef1ed99a3bd0137
  0200000000013efc9cda7b565040
  0200000000013f1ad8ea715da345
  0200000000013f1c1fc8d0ae5edb
  0200000000013f1c20c6fda6dc8a
  0200000000013f1c2338ea6e70cc
  0200000000013f3539130004082b
  0200000000013f375f0ad4d6f8bb
  0200000000013f5ee4d9b2383e57
  0200000000013f5ee5629f106f43
  0200000000013f5ee6d625cb14b1
  0200000000013f5ee737f75e692a
  0200000000013f5ee7ad5759bd2f
  0200000000013f5ee95c874f3a04
  0200000000013f5ee9d1f723e430
  0200000000013f5eea33da7ddcbc
  0200000000013f67674ea36d3d67
  0200000000013f9a172b50ec4da8
  0200000000013fd24c51f7d9b187
  0200000000013fd4238f3ec741ad
  0200000000013fe69a526b62cfd8
  0200000000014006c5cfb8716869
  0200000000014006c7dff63d65ee
  0200000000014008ca6fca575e24
  0200000000014008cb5a6ffd08f5
  020000000001401bc17eba24c718
  020000000001401c822a35cd5e43
  020000000001402b7ef2d74328e7
  0200000000014053117819860b7a
  02000000000140534b5dc7343990
  0200000000014053c9db6409306c
  0200000000014053d66250ed450a
  0200000000014053eec028645071
  02000000000140541f2ea55bf601
  02000000000140543010380fca2a
  020000000001405435cafa1a98ab
  0200000000014054449cf391b3a0
  02000000000140544a4428fb2a91
  0200000000014054bbda92ebadb8
  0200000000014054ca98c8aeaff0
  0200000000014083708d4c75207e
  02000000000140a770e45901d659
  02000000000140a774dd49b61119
  02000000000140c07f96e5f96720
  02000000000140c080cfcb8bb53f
  020000000001410165472f40e7c0
  0200000000014126dd006670fec8
  02000000000141279b10db897488
  020000000001416eb82f2aebf038
  020000000001419cc32cf851df7b
  02000000000141a18087bb594db0
  02000000000141a263b0647749f0
  02000000000141c3cf9883ffb9d7
  02000000000141cba36aee89c309
  02000000000141e70ddb1c4c2278
  020000000001420cd2bab39b5852
  02000000000142229e960175bfe6
  020000000001422879a84ac9fefa
  0200000000014229648decc12fbb
  0200000000014229676176cc3826
  02000000000144085d02a9d1587e
0c97420a5c8c51800720f65508f3c95adcbca45922811992aae6bc452f142699
  7e240a4986f84ba7
0ee0d04926b8b9b693cda6f93018e1d8dbeb1240f3b75d243c47670d050f9d9d
  011444d7fa7044453c6bd7a165fce70b
  01eb4d84b2f2cfd2817249d4ee7fef61
  0243352a12001fef31a99b0b16c30f7a
  0246ed711cb180f92b0b4c03ba865fae
  032f058e63abce632ee3271278361d48
  033c1520fbd0c1c630cca3b55a27d3f6
  03671f8ba5276f57b972aa9467eda14c
  03f917eae05b2a952c45b2b728578e9e
  044346af53f06e3192a7c976d140b28c
  045c5c8349844af9d1f923c24f3d2c13
  0474462702771ad4a63f3e999660c41c
  048de9ddc53ef7ffd253f8e296046957
  04bbebaf3d94526096fd5a644cb4d6a6
  05f68f60c30e66e1b5de9d84b3135e93
  06449fc6f92b397e24cb66933a804581
  06c6f4503d93bf70d861a37af8559079
  06da57a57bd63ef5be35ac43162ca1cf
  071590952587be7a2b37e44e5ee361e8
  07ac9be3d25475efd1a6ee5aa2cd17e9
  07f7095350df13121ef00158772d73cb
  098fac5de86ea51b8342bde5a5cb12d8
  09fbefffbe3034cd00fa03daf5ab2990
  0a419f4f9196e8804c1adceb6e29f8d6
  0a9ef4166ae8a26f5c52b0014996eb9f
  0af8ba45194b2b765a75edabe813c7f8
  0b0b6e055947268e41c0caad185f79ea
  0b15e143be1788af83916ed3967662a7
  0bc4c59ae662999defd7342f8dfae0c7
  0bcd8d5c6e6930335e19e16d531b48fe
  0bf447c3a803284eb2758309f1ad3195
  0bfd5e13fca843e73456aaf758718380
  0c4930845f1765e652c5a2c4f90fc11d
  0c8270de4f8731d7bdf79be97746e46d
  0cc8ebea550e5223a493033cd785b5c0
  0e7a1b052623137135d676497d44be87
  0e7f498919a466274b7e878bdf10e464
  0e9bac5262bb0ef3bc3e56a5ae112fa4
  0f3ec633d8b7b3f1b3ba1438f5eedb98
  0fb349cd8d26d0d803b26d8c95ca2f61
  0ff2bb486a62c7db824f1bd6a940316a
  1055dd788182e46f84ce68e340ceb3b9
  1075c9f76bc155d6fe9f5f82abec1f89
  109e7d126aeca06e4f0416ac8b3ea65f
  10bb1c8947a45412da0ba22601205094
  110f2ab26d4f7e69f83dd9386916e658
  11469cc16f133fccf4c4ec1579d92c0f
  11d34dbc3ac6f5612ec8d262393f3e35
  122c16da3996271019a387ea02abcfe5
  125d31f099a9119143d4cff29d19623e
  12c7d823c688dba5d446628a411195ae
  12c7ef9cd052b491ab9b5958ccc161c8
  12e2cecea1dcc35f2ffa5c5107a6aeb2
  13b90280a57432f4cfda87e124653f9a
  13e164ae7a9a164d1562993db38a3556
  148f1dac7a1a56aa48acd4f2739480e6
  14a8fbad33fe3f7bf165ea718c04c836
  14c98efb21ab557ea7037bc84bdb5d90
  14d5019881267235f8971d64e8d5daa3
  15b5aa9c6fef2c614152d06306a00786
  15f07b07ba023d8ee8b2844b646a8421
  1628ab8474af95d4ce24d39b770a7808
  165b8fa08d9de04bcd931bff924d70bf
  170c3af70195145fd6ed4c166d7d1584
  171d0b83baed760aceddf831bbe26fa5
  177f7bfdd9c1baf40ef116d1561e2fad
  17bcc83bcc699ae576823e187c3ba541
  17c81503006e153a868555325360e743
  17d7170f6d60405333624bf669a11a0d
  180a520e4b4e0b08270b3034c67c0241
  180ed0ebd9838a85add8469cd85e6364
  182c62111a835ebc311467b49b21c8e8
  183f9c1be17539fab36aa3f0eb9ab32f
  188aea9b4807ed1c0cf53ba526f358ba
  18bfa63a45bac3de78708ae6f16940b8
  1908986446940a2afb495c211d1a335f
  199fabf08197c33b9b5825717b09f5c2
  1a40283e63e05d351f2e12d8353a6b8b
  1a4d4ea963f7825e850377e369e10123
  1a91cc553510fe2400269702a47a4717
  1a94769fe93ea351c6aeb68eebb04d6c
  1aa27e98a10546489054ba35d208d1fb
  1aa64d4815194d38fb4d463eabf86628
  1aa8cf10f0b9a355d2e0c908d1b23c3c
  1b192c9e0a318346dbdf514c3a684531
  1b3d44f60a3065bc3e3990bdd06ffce2
  1c119e4f12ee1ceef28c1ee5cc197735
  1c61b2b17a9206ba272eb0a281107f8c
  1e4d5f25f9a4427f6951f85e1ad5d462
  1e62e9942fdbc7448f385541b3be93e7
  1f13cb784ca161daa02772f2a9332dc2
  1f3a52a9f16b937ee6be3c82bf00cb0b
  2008207f05d26f5feb18e874feea89c8
  2017227e137f723291d07ceea69ececd
  20448b1a678ecaec722d4e0efaad0bc5
  20779e46efbc82c4f3a1d5a4d8547960
  21878f3deb48aa30e677a0bc5b8610e8
  219435fdd5174af03587f00b84daf5a4
  2203088686117bf249401138b4e2631d
  2210e965e15d4e8e960efe3e3f482657
  227e60247b88c9516e6d5ba89c7b758b
  2356da3a5c0251f81152c87b4b18b956
  235b709a64d073bc9bb01094293d214f
  23e8c2ef8f7344afbcdeba72a956e4ee
  2442ee9443c759b63ae97a9f07826595
  250841e1712d727099064010fd562251
  2513a6291085784bb2abd13103b51a19
  26b434ecb6df042244604b5a3be5fded
  273de69b25acc9ae9950a09bcd49747b
  2785977725c666504505bb713d1b4967
  27c8f76674a802b41f0c56422fee276f
  282c88d34d8fbdc4b315fa18efc170d0
  2857c004aa0e0ca33f20b65b03381437
  28f58f4c34983056e6fd323bcdd2edba
  2918a8b67ff6cf5474cbc0af03e54d3d
  29a7c2d77ef0053cfa8042e5fd21a06b
  29dbe551650ed26e309cd2650c8bc736
  29e0811b9c9a17b96b3f29c45306a70c
  2ad301eabb3048774bee7d1af8c121e9
  2b857c7d258e3836ea3d83d2255200ca
  2b96dc6990a46a250c958da93d8a608c
  2bd63501d165548982cb047069bdd0f2
  2c0ec7f77c660e3808ef5cb61c4de85c
  2caff2473aa441051394a0f1d857f4d5
  2cee285860ba8899dd0f648bd3967543
  2cfa67777a60b42075a72075d88e8e3f
  2d31d98aaa7a15db3b9015a2cd64cf44
  2d4ad01dc2675ce44a80d8dd85f7e748
  2e34d255e364997fa439ebe2e638721f
  2f71b417c857026ccb5abae10424a21f
  2f99328217a025289cc65b3703b85a8b
  30664f1e6df68f387376106ecf7475be
  30fba900b263d7328c4f73a22ce56749
  31980603dea9dfe743b8961669e5ee2f
  31ba7b863598f6501ab9b7a0488c1c35
  32ebbadb708b3239693af8c9c6bf017c
  33e724bde4c2f5fb1ec83ba64f4e06b2
  34268631936fc0e9a5a9c7f790e9d97a
  344cf23913c2c490d545c48f70928ea0
  34549f318ab80416fe26a5abdf353b85
  3486101d9131f0cc7bb40b75df16be65
  349a9818abe3cf1dec8a3c3e02f0fbff
  349d6c8fb1f01ddd160ff4f02f1b2e19
  34d48b0b3e27f38f9b5a634b48dc4867
  34e8dd18af32cc7b96711e94c9efe4ad
  350c17924ca62dc1bc2e10299e4e2880
  35223d2129d3de29dcbcc1de0e175d1b
  35c8a0312cc175c142fbe17ddd4b715a
  360a0df121739273e4cd2aab969d67dc
  36bb17555bf3def0a5694ee085bf011b
  388fbb4981188c8122d46d98018d5500
  38924baca5f21b60955ba507e1c9628b
  3945f8594da353b262b5c914506c5981
  3947025d202eb30327f34bea804588e8
  3a78099d9280945ce553366d5593f707
  3ab37c7eac7aec0114359b694c92288a
  3af0645a5362657c18272bf6f2fc025f
  3b2333ea3936edab7ebd26395b34551a
  3b7caac5aaf695041e402fd046374f69
  3c1ae78bff841dc9fbf7c4a91459e125
  3c8e4249ef142786e9b56601191ad424
  3d09c14c080c2664e4993dac274e5411
  3d2622bf1a04079f86e350a08d6b5f36
  3e0ca0282ca32194290cf18238f1d949
  3f56cbaafa810d003dbb585f9f6dd1bd
  3f5a567c2c1635994d4bd884d930f355
  40053fe7934d1e28fbfd4bc932c96d45
  405d2b2b47b9c0abfad3c7e430eeac56
  40c81277d901a592fbc0b771d7417db8
  410fda288acf35eef957778022f6733a
  426effdcdba456988e9f98667de68a34
  42d6835df03b65557ab48a5fad4fa6e1
  43c38ca7b6d89d43b040293c5fe53084
  4488005d506b2a5c59b02f15179f73ab
  44bed28abe75286a2ea0e74654ac6513
  44c405edfa5d02a9742dd8532dee4676
  454460c09b689b5aa765efa050884ae6
  466224db2e5cec56ae1a9fb57db83969
  47757909a6008c2cce0706723208e3c3
  493667ea6218ceb59f3ba54240b0a49d
  4994e8fcc0a121f732d21478d026af27
  49dddd641ce8941ad9ab258e18ba1038
  4a2ebc5e51290a3f76579f5a0bb5781c
  4a38d3280a5c0c75e39f18a5ae943c33
  4aab4948d7455e065802199e41b94163
  4acb98faa730715965201c5de234a75f
  4b4db6ee2d2ffda92ddbd1ac0cd03548
  4b9c95aeabacb75e5726cf8bdb368db6
  4bafe6bbaae04b640a44b56910aa9d74
  4be19fedb6c8ece012bf7743c33d5f9c
  4c264b11237cbfa937c9fc1af8128321
  4c87795576af05327d258d061bebecf7
  4da6a7a70766b0afa47bc2e08f0396c5
  4da7193f58f61f769525d8eb6d249870
  4daf282c20b6d47e2ac4694345eabbed
  4de33e5d12a2e2550aba87986f152667
  4e517525511b42dec0442eb821b09aed
  4e639a56bc10a4664b2876c26df03df1
  4ea50cdcd9046d04e9716945e34148c5
  4f1955b52ad021965a659e90bddb80de
  4f30d79a1e899e81e30bf084c9027c53
  4f32804c0ed749daa1a21ae7d25aca12
  4f3d13a64184151c5f97ab942510c268
  4f5e826493633d0072f9e6da81cec632
  4f8e55122df71469bc2fe7cccdcd1606
  51434414ed379477cd91ba648bc011d7
  519c628c5b8df952ba4601026464334b
  51cce734ba5d2d012e1911c96b50face
  52111405542fbcf12c82f12cf0c00582
  526d03fa0cac900541bfbf1ffad23b16
  526e1df56122a829d0ef2c4e0136474b
  537beb024f9d46175bf2e1cf7575f86b
  53fc64c3d81e30a461a5abd7c5504de6
  5413b950e0851a56c34afd32e9db1448
  546b76c3e98cc943dfdd6a8aa48b858b
  54d8c910519fe20eb1064cd1e6dbc0d8
  552e9df12a0a3e26d92145499f2fcdee
  5536c833e99539190b632a6ccc712055
  558596f0708c61acbb377e82b806012c
  55e2a5605d5a91838229ff8b6729b531
  55f73d3cc76679ab69fa4336e3e93a97
  55ffe37a94f80ec87d0933bac3fd6363
  56048fd56ae5ba2ab60e441981d7c8b9
  56238baeb37cb9143d914c36bdba1f8f
  5703fa0e43719d2c9eb176b5800018f1
  57dce7bebd7cca21021b0cf634707ed9
  57f20dde3851d0449d497afdf5e42ec3
  58223457a2c1f86f88571b75309548db
  584ec09d01289577a687e7b1bac1c9cc
  5895462d8bb46e6f1832ab937e7ccec7
  594bc76107ccdf3fe014e8b18159955a
  597f365dbd78b46c5a8a613ea726b69a
  59d8b2570f94d04138ddda1c1be6dd02
  5a08e81c7c168e08935eb0564c0e4c22
  5aa95f2d5b359df58e8574f1e1bc0b12
  5bf74017b1e14f9c126dda9c70701da6
  5d487986e597f44054c7841a23945c76
  5ddbef0aaa802f33e7ef94fdcb9e1850
  5e2b73f9d36eb346be9c77327c17477a
  5e3416f143b277b9bd27d07a40323eec
  5e394787dc4437cd36eb0cea892e5e23
  5ed3b72bd3abadd8fdc1c63f576310d5
  5ee137c48fa8befe771cd13bdb936b03
  5eee69793bc721810aab1fee2e83e2a5
  5f808536172ca2b40ad916eb8ae6ebbf
  60195ec309123c1d703fe7c6a5e7ebf3
  606f0feace4ddfae08b18037a6ed3493
  609091fcf9ac0bb720b22be37851c785
  60a07b5a69c7978dbe8c1d7c505a38a9
  60a1a47ab7773540b0a3f1be3dfb21d8
  6137fbfe99405a4c6fc03ec04428d04c
  61dbe19741460b40cd8a02dbf5417e97
  627f4b0b3c71ce6a19d9499b8c4567f7
  62bf12a978c15673b74dcaaf3a5b7b2e
  62c6f656ceb5b9f8f50c2f58c74d6196
  6349fc6003aea4d2224049fd016bef4f
  63f7a840e91201c81cc6e386dfe0871f
  64510c9b8580f4f1ec8e335847bd742d
  649e2b5c681f25c105a8f41d05e66708
  65852e60b24a2fa63b8b075717987a2f
  65b85cbcd9559c6a806c622447df089e
  6660a0b8ff4f4a1b1285355c5474c760
  66662878f61a505331f04ba5dbfed5d8
  6673cfa5b13e2e048b3c6d4e2c7c9f0b
  678f231bd5996a7e650705725c5e2785
  67c8c0204354c409ece0b56a6f80b39a
  67c97da56bbd0cd68a7d42eb1514bf7a
  681f617d1babb4e70542ff2ab8b51cc1
  68af86a478eed00c4284b221836dda0f
  68da4b0e45510bb9aa0a5f7af760317a
  68effa320fa74779c0e7270c9ccab69b
  69690f90799aeb398bc63ea651874787
  6a63e20e73bb4115e5d3490b82e44b9f
  6b07732933eb37aeadd3c43f58b266c7
  6c4c007e203d356971e2954614cb13df
  6ce4b1c2477a4342f8e81768e7c1f972
  6cffed95647a23b6bd802ef3f2b53692
  6d9fea49db5300c1f035898b76a4a974
  6db039df27841220fb7a65874f74d86d
  6de5b77740d5a200b3aee50198e6ad09
  6e1b5baa8ad5bc0ccf4bdd9efc0dea2d
  6e268f19e88612d1a0e030cc26b60034
  6e7979d57209752a982950a23dc4b481
  6e7dcb4914f3a331fc60df4fd3f34438
  6e9ae7a008054423dbb70f55673cf02f
  6f59a96e9f1b91dc989992c4f6253a30
  6f65d2bc9022a8047455c18eedd3390a
  6fda53dc72b88b83e9c9e999428fd3ef
  6feba3f747593431e629743308159567
  701a8884729a1aa4306d27b2965128fc
  70ace8db41ed7171c8422d883507a006
  70e428b3987aea50eb52676ec85c1b7a
  70f03cffa6f6efeae0ea8ba95455d9ae
  7126f793d6b14255425cb0046b0a480a
  7127d3b417582faaf3054033b91d7f55
  7131c25e5c3649785bc557fce7a82c2a
  724feff7b4a1e52be4b1cb20e0660e39
  728e928290cb5bb8242f1d97604ec84f
  72d1ceaf172c8b15c4d2022fa8444557
  73203e6a488a006b9216819b40d9a1e6
  732dcf309bdbec724a887c1c66d856c7
  73b3cdc5e9ccfec6438dae9999c20a8f
  73dee78b55e813221b835b152734bb99
  74a13d08dce8b01cbd493eedc99d01a5
  74aee724c459324ae2912fa1bc11944f
  74bfa1f51843ceaae2b8fb585a1b76a1
  74f5bc244233966c50fca313b25b35a1
  7615859c410f68bdb20dad9901f0abec
  765c0993515112e16684c6ec2d898f7b
  76a83ce2a444ab45177d8889804895df
  783e69223432b6aeca46703d3c63fb8a
  78683fab09d92efef92a9952c3e5ec40
  789e91750a6371c15dcaf6a1a9209f50
  79706345766c0b7a0b64ea5ef365d46d
  79a547780408ea77d77cccf2abf16131
  79ae8fa83aee118af998bdd5bd5da1a9
  7a2ead9ea031ae61d8979f512cb2ce21
  7a3d3e7d78d79db4be410cd789beed2b
  7a907628e707ff0893ce34e34cb7b213
  7b21d32904d3dd51d1f51adbd266adda
  7b3fa1123366c497895d0ff943a2af1c
  7b63b271963d909b0eb1c36ddeeeb3e7
  7c0a6fe56598e4d6ac639a7bc952488e
  7c13ff4aadc9a53b278797d41aea5355
  7c41becb222fb014fc88e13fce440fba
  7c66dc1361753f734779bd6a3c89699b
  7cf1af48c60c5fcec5042584888bb4a4
  7d2fb015dc0572b2a2650491a2465633
  7e1f2e938b13ccb77e828f395000352e
  7e7105e37bdaa09b64d03f58da4608fd
  7f458440c7d31dce12c233f4dd3aa34b
  7f9f2084bc39a1e34d38494bc1a2778e
  7fbd6c8b4ce60f42baa33597cb578cc1
  7ff67c56a3055b1624a954da635d02c4
  808bb3995968192c29e0ddc69cd44166
  81a0b84c73057e97a5199b529d41d4df
  81b26f1313208fafba04552cb9d9f3f7
  81bf4e36c5ec152237635e72cf8ef0b7
  821b7dd5cf27ea24049aecd9ac2acb02
  826332738db15d82229610c1b895b24c
  826b23c18ece07eb90b90c4ae453434d
  82ac6548dbbfaf6fd3dc359e13b17845
  82d6030b11fff3affa94c1e83c5af9c7
  843bcdce77bf62f85027d5006df5f897
  849c20fbb522f729c075122ce387ceb8
  84cac522c7a7048585e60db6694f3b0b
  84e7a677e5e34687c4368266d1d3a680
  861535ed93decea3d9ea8a0ff59c551a
  861f653c69faa71c4b7ee5969fb0b536
  8647b8be37b0abbd7b9776827ecd325f
  866edf67adff94f9e0a7aea1eae3e241
  867487d0a5074f09d665df4130e3e306
  8683f5d5c43299c550c1d3ff997083b8
  872624e12a7d67d475f383c9515ad72c
  87f3d84aac9aba1aedb9441de7956d26
  8830f6a6ace9e7a97ba924b0a40b9a84
  8853bcd7110749441f22833efb52db3a
  887450f33fd07e3d679337d9946f33ab
  88b0eada7375ded151e4018461cb4947
  8910757c5bce2781e994649355579ffd
  8921e8ade41d23ede958fb7dfad4148d
  89b16d369feb37c240329f2ca8dd9827
  8a1e552dabbcb3526576b054557b9861
  8a2fa3a8eae7fff276ad943dbda27747
  8a62699540a680ddb398904820843049
  8af46d329d02743f05b49f9d08fb5575
  8b3ffcbb510a715e4587ea7fa1255240
  8b544f463d587bfda148c34df1e36bbd
  8b8b9687ee7e8f9a51e36f5fd3ba60b1
  8d14919a573ad9f35880dc8dab175e46
  8dffbe074314ab4fa55e748859c2299c
  8e7d0dd9914d6a504cc2c18677d10086
  8e9cd992b604bb239c9bf7e7ff8d0120
  8e9f582d11c88c6b3647a0183869ca4e
  8ec7f560347625e2038f373786c7862a
  8f40f8d6c12870c9593334ecfe9cf7c3

What do these characteristics mean for the CRLSet's effectiveness?

The header indicates that the body of the file contains 53 sets of revoked certificates (NumParents:53). In other words, the entire current CRLSet contains sets of revoked certificates signed by 53 certificate authorities.

Let's look at some numbers:

Apple's site provides their list of 211 root certificate authorities which are built into and trusted by Apple products, and Microsoft's site provides their listings (on four pages) of 353 root certificate authorities built into and trusted by Microsoft products.

If you are using Chrome on an Apple product, which trusts the certificates signed by 211 certificate authorities, the limitations inherent in Chrome's CRLSet blinds it to ALL of the certificates which have been revoked by 158 certificate authorities trusted by Apple's products. To rephrase that: Chrome will blindly trust every revoked certificate that was originally signed by three quarters of the certificate authorities Apple trusts.

If you are using Chrome on a Microsoft system, which trusts the certificates signed by 353 certificate authorities, the limitations inherent in Chrome's CRLSet blinds it to ALL of the certificates which have been revoked by 300 certificate authorities trusted by Microsoft's systems. To rephrase that: Chrome will blindly trust every revoked certificate that was originally signed by more than four fifths of the certificate authorities Microsoft trusts.

How does it perform in sheer number of revoked certificates?

Examining CRLSet #1596, we find 24,259 lines of certificate identifiers, one per line. Since we know that 53 of those lines are the certificate signatures of each section's “parent,” we subtract 53 from the total body line count to obtain 24,206 revoked serial numbers contained within CRLSet #1596.

But . . . compared to what?  That's the question.

In July, 2013, Websense harvested data from their “ThreatSeeker Intelligence Cloud” as described in detail in this Websense Security Labs Blog article titled “Digging Into Certificate Revocation Lists.” During approximately one hour of sampling the Internet for certificate traffic, Websense gathered 10,000 records. From those 10,000 records, 9,066 contained the URL of a certificate authority's online certificate revocation list (CRL). Within that set they found 819 unique URLs (there were many repeats).

Here is Websense's graph of the Top 30 requested CRL URLs:

Credit: Websense blog, July 10th, 2013

Websense notes that the Globalsign CRL (the lower line on the chart above) is the most popular, having been referenced almost twice as often as most of the other top 30, not to mention the other 481 URLs that didn't make it into the Top 30 list. We'll come back to Globalsign a bit later.

Websense then compiled the certificate revocation list content from 511 of the 819 URLs that responded to their queries. Given the nature of the ad hoc URL collection, and their unknown provenance, Websense was not surprised by non-responses. Other facets of Websense's research is further discussed on our OCSP Must-Staple page.

What's significant here, is that by pulling from the Internet's readily available certificate revocation lists, Websense gathered records of a total of 2,232,845 revoked certificates. This means that Chrome's CRLSet, which currently lists the serial numbers of 24,206 revoked certificates, reflects only 1.08% of the revoked certificates collected by Websense in one hour.

The 2,232,845 number should probably be regarded as a worst-case high-end count. Certificates are used and revoked for purposes other than protecting web traffic. So for the sake of argument and scrupulous fairness, lets assume that only half of the revoked certificates collected by Websense are relevant to web security. That's probably very conservative, as additional data will shortly demonstrate. But it allows us to very conservatively conclude . . .

So that we're not basing our entire case upon a single research study, here are two additional data points from Internet monitoring companies to further characterize the limited reach of Chrome's list of 24,206 revoked serial numbers:

• Websense:
  Starting in the year 2000, Websense has tracked the rate of certificate revocations. This chart clearly illustrates the trend:

  
  Credit: Websense blog, July 10th, 2013

  The apparent decline in 2013 is due to the fact that data for only the first five months of 2013 was available and shown. Websense notes that during those first five months, 766,451 certificate revocations were seen. A calculation of the average monthly revocation rate based upon those first five months will significantly underestimate the year's total, because the historical trend has clearly been increasing. But it can give us a working minimum estimate.

  766,451 revocations in five months means an average of 153,290 revocations per month through those first five months. Extrapolating that for the year, which, again, won't account for any rate increase, gives us a low-end estimate of 1,839,482 newly published certificate revocations occurring only during 2013.

  Standard web certificates have a lifetime of either two or three years (24 or 36 months) though most are issued for three years to obtain a discounted price. If we assume that revocation events are uniformly (randomly) spread across the lifetime of certificates, this means that the average revoked certificate will remain on its CRL for half its lifetime, or 18 months. Then it will be expired off the list due to age. For the sake of obtaining a scrupulously fair low-end estimate, we will ignore the effects of an accelerating revocation rate which would cause more certificates to be added every month than expire. At any given time, assuming a low-end revocation rate of 153,290 revocations per month, the Internet's collected revocation lists would contain an average of 18 months of certificate revocations, or 2,759,220 listed revoked certificates. Remember that this figure assumes a constant revocation rate based upon the observed revocation rate during the first five months of 2013. We know the real number is significantly higher.

  From this data, even after making every conservative assumption, Chrome's list of 24,206 revoked certificates, accounts for just 0.877% of the Internet's currently published non-expired and possibly fraudulent certificates.

  Update: In a communication with Netcraft engineers attempting to verify our back-of-the-envelope estimates, they wrote: “In our data, from 1800 GMT yesterday, there were 2.85M revocations in CRLs we’ve fetched after seeing them in publicly-trusted certificates. This is very close to your estimated 2.76M.” — This additional data directly confirms an aggregate global CRL size of well over two million.
• Netcraft:
  Early in the Heartbleed adventure, Netcraft, a well known and respected Internet monitoring company, wondered aloud in a posting when the expected Heartbleed “tsunami” of certificate revocations was going to show up. Most sites took longer to respond than many industry observers expected, with a great many occurring the following week. Quoting form Netcraft's post:

  Activity on certificate revocation lists peaked at a rate of 3,900 revocations per hour on the day the Heartbleed bug was announced (Monday April 7, 2014). On a typical Monday, we would expect to see a total of around 22,000-30,000 SSL certificates being revoked over the course of the day. On the Monday that the Heartbleed bug was announced to the public, there were 29,000 revocations. On the next day (Tuesday), 33,000 certificates were revoked, followed by 32,000 on Wednesday. These were both above average, suggesting that around 5,000 certificates were revoked in direct response to the Heartbleed bug each day. Note that fewer revocations usually take place over weekends.

  The crucial takeaway from Netcraft's rather matter-of-fact baseline, is that they state that on a typical Monday before the Heartbleed incident, they would expect to see between 22,000 and 30,000 certificates revoked. With most certificates having either a three year (36 month) life, we'd expect the average certificate to remain on revocation lists for half that time, 18 months or 548 days. Some will expire sooner, and some later. But Chrome's CRLSet #1596 contains a grand total of 24,206 revoked certificate serial numbers. Given Netcraft's observations, that implies that Chrome's CRLSet might be able to handle a single day's Internet certificate revocations. But given the average revoked certificate CRL presence of 548 days, Chrome can only retain approximately 1/548^th of the Internet's revoked CRLs.

  Update: In a communication with Netcraft engineers, they explained that the numbers obtained during this particular work were fetched from all of the CRLs they have access to, including some published by certificate authorities not trusted by mainstream web browsers. So this somewhat horrifying number is reduced somewhat.

Since signatures are made from hash functions which are deliberately “one-way,” it's not possible to take a signature hash and “decode it” to recover the value that was originally hashed. The only way to determine which certificate authorities are represented in Chrome's CRLSet is to take candidate CA signing certificates, generate their signature, and see whether that value matches any of the values in Chrome's CRLSet.

Let's first take a look at Globalsign:

As we saw above, during the single hour of Internet traffic capture which Websense analyzed last summer (long before anyone knew about the critical vulnerability in OpenSSL and had imagined or named “Heartbleed”), references to Globalsign's CRL were made nearly twice as often as most of the other CRL's among the Top 30, to say nothing of the remaining 481 responding CRLs in the data set. In other words, as reflected by this sample . . .

In other words, the integrity of Globalsign's certificates matters to Internet users. This makes the importance of intercepting and blocking any attempted abuse of these certificates clear.

As a consequence of the Heartbleed incident, hundreds of thousands of possibly-compromised certificates were revoked and replaced across the Internet. This chart, compiled by Netcraft, shows the regular daily cycle of Internet certificate revocations per hour in the days following the Heartbleed revelation:

Credit: Netcraft Security Blog, April 18th, 2014

It also shows, separately in green, the April 16th to 17th period during which the large web performance and security company, CloudFlare, which was vulnerable to Heartbleed as were so many sites, revoked all of its web certificates issued by Globalsign.

To obtain an estimate of the total number of CloudFlare's certificates revoked by Globalsign, we next turn to the SANS Technology Institute's certificate revocation tracker:

Credit: SANS Technology Institute, Internet Storm Center, SSL CRL Activity monitor

At the time of this writing, the SANS Internet Storm Center is monitoring the CRLs of 81 Internet certificate authorities including Globalsign's. Although this is only a subset of all trusted certificate authorities, it is perfect for our purposes, because their monitoring of fewer means that skewing of the Globalsign data will be reduced.

The two data points of interest occur on April 16th and 17th. Examining the chart, we note that the midpoint of the line connecting the data for the two days falls almost exactly across the 70,000 certs-per-day scale line. This means that the value for April 16th is as far above 70,000 as the value for the 17th is below 70,000. In other words, the average of the two days is slightly below 70,000, so the total for the two days is slightly less than twice that, or 140,000.

Therefore, we can confidently conclude that just short of 140,000 possibly-compromised and now revoked certificates were originally issued by Globalsign, the most heavily used certificate authority (based upon Websense's traffic analysis).

What does Chrome's CRLSet have
to say about that 140,000?

The Globalsign CRL in question, which was monitored by Netcraft and SANS for their reports, is http://crl.globalsign.com/gs/gsorganizationvalg2.crl.  This handy page provides the intermediate CA certificate whose private key was used to sign the Organization Validated (OV) certificates whose subsequent revocations were listed in the Globalsign OV CRL. Since this certificate is the “parent” certificate of all those revoked, its signature will introduce a set of revoked serial numbers within Chrome's CRLSet.

We generate any certificate's “signature” with the following concatenation of OpenSSL commands:

This extracts the Subject's Public Key Info (SPKI) record, converts it from base64 into binary, obtains its SHA256 hash and outputs the final result in hexadecimal, which is the format Chrome's CRLSet uses for each of its certificate set's headers. When this command is run against the Globalsign OV Intermediate certificate we obtain this signature:

Now we perform a simple text search for that string of characters in the ASCII-converted CRLSet. And it is not found. In other words . . .

But . . .

First, a brief bit of history to set the stage . . .

Certificate revocation has not been receiving sufficient attention from the Internet community. And it seemed clear that the news of the “Heartbleed” vulnerability would result in a “tsunami” of revocations of possibly-compromised certificates. So the day after Heartbleed went public, work on GRC's revocation testing “https://revoked.grc.com” website began. It was secured by a certificate that had been revoked before the site was ever made public. Its intent was clearly stated: To allow users to test the “revocation awareness” of their various browsers, devices, and computing platforms.

As expected, the revoked.grc.com site immediately and vividly demonstrated that “the revocation problem” has been a well-kept secret among industry insiders. The Internet's many millions of web surfers have been encouraged to believe that revocation is working and that their web browsers are protecting them from fraudulent web sites. (They're not.)

Because few people understand the reality of today's broken revocation system, Chrome's users assumed that their web browser's willing display of the revoked.grc.com demonstration page must be a bug . . . and reported it as such to the Chromium developers: Issue 362696: Missing warning on revoked certificate. (For additional background, read the thread there. It's enlightening.)

And sure enough, though it took a few days, we started receiving reports that Chrome was now “properly” blocking the special revoked.grc.com test site . . . as expected.

Wanting to take a closer look, we grabbed a copy of Adam's Langley's CRLSet fetch & dump utility from his github repository. (This is how we originally obtained and created the complete CRLSet listing above.) Knowing how the CRLSet worked, we simply searched it for the serial number of the certificate of our now-blocked revoked.grc.com site . . .

 . . . assuming, of course, that it would be there somewhere.  But it wasn't!

The serial number of GRC's “revoked.grc.com”
certificate wasn't in the CRLSet file anywhere.

How could that be? Chrome had begun blocking access to revoked.grc.com once people obtained the updated CRLSet. Adam Langley himself, answering in the dialog following the Issue 362696 bug report, wrote: “CRLSets push 1576 (I think) contains the revocation for revoked.grc.com. (See chrome://components.)” (Adam's reference to “chrome://components” is a means to have Chrome display the versions of its various components, among them the last retrieved and loaded CRLSet.)

Was our CA, Digicert, missing the same way Globalsign was?

We decided to see whether our parent's (Digicert) intermediate certificate was present as one of the headers for a set of its revoked certificates. So just as we did with the Globalsign certificate, we ran the OpenSSL incantation on the certificate for the key that signed “revoked.grc.com”:

It dutifully dumped out the signature of the certificate for the key that signed “revoked.grc.com”:

And, once again, we searched the CRLSet for that string . . . and it was not present in the file. That meant that no certificates signed by Digicert's High Assurance CA-3 intermediate certificate were present in the file, including ours.

But it had to be since Chrome had started blocking the site.

A bunch of us in the grc.securitynow newsgroup, who had been collaborating on this research, puzzled over this intractable mystery for a day or two. Finally, one of the brighter bulbs in the box decided to pursue a wild hunch . . . which shocked us all, but instantly solved the mystery:

By adding one more term to the growing OpenSSL invocation, we convert its hexadecimal signature output into base64 encoding, which is the format used within the “BlockedSPKI” header block:

Rather than using just our certificate's serial number, our entire certificate is fed into this invocation. It extracts the Subject's Public Key Info (SPKI) record and returns a base64 encoding of it's SHA256 hash:

 . . . which is exactly what's listed as the last entry in the list of certificates to be blocked by the certificate's public key (not its serial number). You'll note that we also discovered that one other high profile recently revoked domain certificate had also recently been added into the CRLSet header: cloudflarechallenge.com

We have no idea what certs those first three “BlockedSPKI” entries are blocking, and no one (except the Chrome Devs) would ever know what those last two were, if this mystery hadn't been solved. Now we all know.

• We know that Chrome's CRLSet currently includes certificates from only 53 different certificate authorities. ALL of the other many hundreds of CAs are publishing their own CRLs of revoked certificates which Chrome will blindly trust.
• We know that Chrome's CRLSet includes at most 2% of the revoked certificates currently published by the Internet's certificate authorities. Chrome will blindly trust the remaining 98% of the Internet's revoked and not-yet-expired certificates. And, of course, new revocations are being published every day, 98% of which Chrome will never be aware of.
• We know that Chrome is blind to every one of the approximately 140,000 possibly-compromised certificates recently revoked by Globalsign on behalf of CloudFlare. Chrome trusts them all.
• We know that the appearance of the entirely benign revoked.grc.com test, and news that the certificate for the cloudflarechallenge.com website had been revoked, threatened to reveal that Chrome would not automatically block those revoked certificates. (It's not a bug . . . it's a feature.)
• We know that the Chromium developers (secretly) know all of the above, and that they don't rely upon their own CRLSet system to block sites that users are aware of and expect to be blocked.
• We know that when they need embarrassing certificates to be blocked, they make special cases of them, adding them manually, as static entries, into the header of Chrome's CRLSet, because that's the only thing that works.

Thus, by any reasonable definition of “broken” . . .

• We know that Chrome's CRLSet certificate revocation system is completely broken. It cannot, and should not, be relied upon to protect the security interests of its users.

You'll recall that near the top of this page we referred to Larry Seltzer's recent ZDNet column with that title. We now have sufficient context, truth and fact to evaluate Larry's conclusion. In his column, Larry quotes an unnamed Google spokesperson saying:

Somebody, somewhere, has been drinking some serious Google Kool-Aid.

As another point of interest, it's worth noting that in practice, updates to Chrome's CRLSet appear to often take several days and may be unreliable. In the online dialog following the developers' addition of the cloudflarechallenge.com and revoked.grc.com certificates, participants can be seen continuing to report that the sites were not being blocked. They were jumping through hoops, rebooting and restarting their browsers, forcing update refreshes, and doing everything they could think of to get the blocking to work. The actual process seemed to go a bit less smoothly in practice than in theory.

Larry then goes on to quote Adam Langley:

Let's address Adam's last sentence first: If anything has ever been deserving of the title “security theater” it's a certificate revocation detection system that is completely blind to all of the certificates revoked by 4/5ths of the Internet's certificate authorities, at least 98% of revoked certificates overall, doesn't even attempt to deal with the consequences of major Internet events such as Heartbleed, and must have known-revoked certificates, which would be embarrassing to not block, manually added by its maintainers.  As we all now know, that describes Chrome's CRLSet to a tee. “Security theater” . . . by design.

What about Adam's first assertion?:

“That's why I claim that online revocation checking is useless — because it doesn't stop attacks. Turning it on does nothing but slow things down.”

Unfortunately, in our view, Adam is allowing the perfect to be the enemy of the good. He is absolutely correct in asserting that using most of today's technology there are situations in which online revocation checking can fail and/or be blocked by a determined and properly positioned attacker. But we believe he's wrong to completely ignore the many situations in which it cannot be blocked and is guaranteed to succeed in protecting its user from online fraud.

One example is Firefox's available option to refuse any secure connection whose certificate cannot be positively verified as valid. Although this can result in Firefox blocking when it should not, it provides extremely good protection to users who have enabled it. (We have always had it enabled and have never received a single false-positive block.) This page is not the place for a detailed pro and con discussion of this issue, but we do have such a page: For a thorough discussion, including the examination of worst-case attack scenarios, please see our OCSP Must-Staple page.

Given Adam's continual assertions (here and here and here) that “revocation checking is useless” one quick example of certificate revocation working valuably and perfectly is in order. For brevity and clarity, and because we haven't discussed the newer OCSP protocol here, we'll look at the case of classic CRL lists which have been supported for decades:

• A typical Internet user visits secured sites such as Amazon, eBay, PayPal, their bank, and so on. With revocation checking enabled, every time they connect their web browser will query the CRL pointed to by each institution's security certificate to verify that the certificate is NOT currently listed as revoked. CRL's are typically updated by their issuing authorities daily, with a week-long expiration, so the most recently obtained CRL covering a given certificate will be retained (cached) by the browser and/or operating system and updated as needed.
• Then, at some point, something happens to cause one of the secured institutions to worry that their certificate might have been compromised. So they ask their CA to revoke and replace their no-longer-trusted certificate. The replacement will be from the same CA, of the same class and have the same intermediate signing certificate, so it will almost certainly be covered by the same CRL.
• Now imagine that the institution was correct, and that bad guys got control of the revoked certificate. At some point before the revoked certificate expires they create a fraudulent clone of the target institution's website and arrange, perhaps through internet router, DNS poisoning, or traffic hijack, to divert the institution's traffic to their fake site.
• If revocation checking is enabled, before accepting the connection the user's operating system and/or web browser will examine the pending connection's certificate to find its CRL. Since this will likely be the same CRL listed for the institution's replaced and non-revoked certificate, which they've been using, they probably already have a locally cached copy which lists the previously revoked certificate as such . . . thus absolutely preventing them from trusting the fraudulent website. They are protected.

In the entirely feasible real world scenario described above, you'll note that there was absolutely nothing any attacker could do at the time of the attack to block the browser's awareness of the revoked certificate. Thanks to background CRL fetching and caching, the user's device already knew of the certificate's revoked status. And note that since no additional Internet queries and retrievals were required, in this instance nothing was slowed down . . . except for the attacker, who was stopped cold.

It is unfortunate that anyone would advise everyone to disable this long-standing feature of Internet security using the argument that there are some scenarios in which it can be defeated. No one argues that. But that certainly doesn't render it useless, and it appears to be far more generally useful than Google's own CRLSet which presumes to replace it.

With OCSP Must-Staple, the same web server that serves the security certificate also “staples” to that certificate a freshly signed assertion from the issuing certificate authority attesting to the certificate's current status. The web browser doesn't need to make any additional fetches, so no delays or performance penalty, ever. The web server periodically obtains a fresh OCSP assertion from its certificate's authority and provides that to every browser.

“Stapling” already exists and it's supported by all major web servers (Apache, nginx, LiteSpeed, IIS and others):

• Microsoft IIS: enabled in all web servers by default for years.
• Apache: https://httpd.apache.org/docs/current/mod/mod_ssl.html#sslusestapling
• nginx: http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_stapling

Firefox has supported stapling since version 26 and Microsoft with Internet Explorer has been well ahead of the curve.

The final piece of the puzzle is the enforcement of stapling, known is must-staple, and we're close to having that too.

Let's get this fixed!

1. If you haven't already, please read and absorb our The Case for OCSP Must-Staple page. You'll come away with a complete and comprehensive understanding and appreciation of the whole problem.
2. You'll learn and understand why “OCSP Must Staple” is where we need to be.
3. Then use Digicert's excellent help page to check whether the various websites you need to trust currently have OCSP Stapling enabled. (GRC does, Google and most others do not). All major web servers have supported OCSP stapling for some time, but with the exception of Microsoft's servers which enabled it by default, it is disabled by default and most have not seen any need to turn it on. Help them to see the need. Let them know you need them to care more about their (and your!) security.
4. It appears that Firefox will be the first browser to offer support for server response-header “Must Staple”—they're very close and the Mozilla guys “get it.” Internet Explorer has all of the plumbing and foundation required to quickly follow. So until Google catches up, either of those two browsers will likely offer the best protection against Internet certificate abuse. There are guys at Google who also “get it” and who know what needs to be done. They're just not as close to being there as they would like to be today.

Note: Our next and final page in this series will describe and characterize the current “revocation awareness” state of the industry's various OS and web browser platforms . . . stay tuned!

• Google's Adam Langley's blog response to this page: Revocation still doesn't work.
• The official Certificate Authority (CA) Security Council industry group has also weighed-in on the issue of Google's claims and the proven CRLSet ineffectiveness.
• The Google Chromium team have responded by disabling all external certificate revocation checking (CRL and OCSP) and then removing the checkbox to optionally allow the user to turn it on, using the justification that it confuses users. (Even though it was already well hidden under the Advanced Settings. See near the bottom of this “bug” page, and the comments in this code commit page: “Remove the confusing certificate revocation checkbox.”
• ArsTechnica's coverage and interpretation: Google Chrome protection for Heartbleed-hacked sites called “completely broken”
• SlashGear's coverage offers a sane and coherent conclusion: What we can gauge from this is that a security standard — a real standard — must be arrived at soon. The issue with CRLSet revolves around transport layer security (TLS), and if one of the major browsers is only identifying 2% of compromised sites, that’s just plain inadequate. When it comes to security, there simply shouldn’t be disparate parties solving things their own way. There needs to be a call to arms, and a dissolving of egos.
• ZDNet's Larry Seltzer followed up his earlier piece, quoted above, with: “Certificate revocation controversy heats up” Within his column, this part came as welcome news:
  After my last article praising CRLSets I was approached by the CA Security Council, a CA industry group. They made some of the same arguments that Gibson makes about CRLSets, but they didn't try very hard to defend the current regime.
  
  I did ask them about what could work in the long term, and we agreed that a combination of a new standard called OCSP Must Staple and hard failing would be effective, particularly in combination with short OCSP validity periods.
  Yep. That's exactly where we need to go. See: The case for OCSP Must-Staple.
• Softpedia covers this issue with: GRC: Google Chrome Only Blocks 3% of Sites Compromised by Heartbleed