1Password 4 Cloud Keychain design

Introduction

This is a technical document detailing the 1Password Cloud Keychain format, introduced with 1Password 4 for iOS in December 2012.

The 1Password Cloud Keychain format is the new external data format used for syncing with iCloud. Dropbox syncing still leverages the Agile Keychain format to enable cross platform support with 1Password 3 on other platforms; see Rolling out the 1Password 4 keychain for details.

While this document specifically covers the Cloud keychain format, 1Password also includes a local copy of your data in an SQLite file for better performance. The data stored in SQLite is secured using the same design decisions described below.

A bit of history

The Agile Keychain format was introduced in 2008 as a successor to using OS X Keychain integration. It proved to be much more reliable for syncing, and gave us flexibility in design, efficient and reliable syncing, and portability across a variety of platforms. It was definitely a good thing. We designed it not only to withstand threats from 2008, but from future threats as well.

Changes in available technology allow us to improve upon that design. And we can again work to design against threats that may not exist today, but which may develop in the coming years.

There are many design features that we keep in the 1Password 4 Cloud Keychain format, including

  1. We continue to use JSON files for most everything.
  2. We continue to rely on existing cryptographic libraries, and do everything we can to avoid rolling our own. In particular, we use the CommonCrypto libraries for OS X and iOS.
  3. We continue to use PBKDF2 for key derivation. (And we’ve dramatically simplified other portions of our key derivation)
  4. We continue to have as little information as necessary decrypted at any one time when the data is “unlocked”
  5. We continue to be as open about our data format, implementations, and design as is practical for a non-open-source product.

Contents

Major changes

Fewer separate files

The Agile Keychain format used a separate file for each item in the user’s keychain. Experience has taught us that not only that having each item in its own file creates difficulties for some filesystems, but that it also brings a penalty with data syncing as well. There is an overhead for synching each individual file irrespective of its size. Having hundreds or thousands of small files led to inefficient syncing in some cases. On the other hand, we don’t want a single monolithic file, which would introduce its own problems for synchronizing 1Password data across systems.

We combine items into 16 bands depending on the the item’s unique identifier (UUID). For example, all items that have a UUID beginning with “C” will be listed in the file band_C.js. Band files range from band_0.js to band_F.j. If there are no items in a particular band, there will be no band file for it.

We also design a data format that is appropriate for each platform and have a process that synchronizes the Cloud Keychain with a format that is most appropriate for the application. We had already been doing this in some situations with earlier versions of 1Password, but in 1Password 4 it is a fundamental design decision. The file formats used for synchronization through cloud services is never directly used by the local application.

We will refer to the particular data files and formats used by used directly by apps on particular platforms as the “local data”. The local formats will use the same encryption design as discussed here, but will (typically) be SQLite files. But this document just focuses on the 1Password 4 Cloud Keychain, which is the format of data that will be used for synchronization and storage across networks, initially with iCloud.

Encrypting everything

The Agile Keychain kept some information (most notably Location and Title) unencrypted so that these could be used to search for or identify a particular item, while the more sensitive content could remain encrypted. With the Agile Keychain format, the browser extensions could identify and list potential matches for a website without having to be “unlocked”. With the JavaScript 1Password extensions, we have moved away from that. The user must unlock the data with their Master Password before they can see a list of Logins.

Overview data and full content

However even when 1Password is unlocked, we wish to keep as little sensitive data decrypted at any one time. So in the new format we make a distinction between “overview” data and the full encrypted content. Overview data contains the information that is used to sort, index, and search for items in the keychain. We can decrypt the overview data for all items after the Master Password is entered, but keep the details for each item encrypted until the particular item’s content is needed.

Unencrypted metadata

Some metadata remains unencrypted: Which folder an item is in; what category (Login, Credit Card, …) an item belongs to; creation time; modify time; and last sync time.

The item UUIDs are fully available, which can be used to determine how many attachments, if any, an item has associated with it. The UUID of any folder an item belongs to is unencrypted, and thus an attacker can determine which items are in the same folder.

Ciphersuits

We use Encrypt-then-MAC authenticated encryption everywhere we use encryption. The MAC is HMAC-SHA256 and encryption is AES-CBC using 256-bit keys. Key derivation is uses PBKDF2-HMAC-SHA512. More detail about these choices will be presented in the relevant sections on key derivation and item encryption.

In this document we will refer to “blocks of data”. Unless otherwise stated, blocks are the length of AES blocks, 128-bits (16 bytes).

Authenticated encryption and data integrity

When the Agile Keychain format was developed, chosen ciphertext attacks (CCA) were seen as theoretical. Furthermore the primary threat to 1Password users was thought to be from an attacker stealing the data once and pursuing an off-line attack. It did not anticipate an attacker who could tamper with user data that would be subsequently processed by the legitimate owner.

CCAs are no longer just theoretical, and we also see (and encourage) widespread storage of 1Password data in “the cloud” for syncing. Thus data integrity needs to be addressed in our new design.

Instead of trying to design against particular CCAs or particular mischief that can be done through data manipulation, we simply authenticate everything we can. Authenticated encryption is used whenever we encrypt, and HMACs are calculated over the elements in each item. The item is rejected if the MAC does not verify. The Encrypt-then-MAC construction is better thought of as “Verify-and-only-then-Decrypt.”

Building blocks

Keys

We handle a variety of keys, the purpose and derivation of which will be described in later sections.

  1. Master Password (OK, this one isn’t a “key”)
  2. Derived encryption key.
  3. Derived MAC key.
  4. Master encryption key.
  5. Master MAC key.
  6. Overview encryption key.
  7. Overview MAC key.
  8. Item encryption key (item specific).
  9. Item MAC key (item specific).

As an overview of the role of these various keys, note that the secret content of each item is encrypted a unique pair of keys (encryption and MAC) for that item. The relationship between an item key and the users Master Password can be summarized in song:

Each item key’s encrypted with the master key
And the master key’s encrypted with the derived key
And the derived key comes from the MP
Oh hear the word of the XOR
Them keys, them keys, them random keys (3x)
Oh hear the word of the XOR

All of the encryption keys are 256-bit keys for use with AES. This has been a highly requested change. Although 128-bit keys remain more than sufficiently resistant to any brute force attack, moving to 256-bit keys has the side effect of leading us to use a SHA512 in our key derivation, which may add meaningful strength defenses agains password crackers. Additionally, as the systems that 1Password normally runs on are much more powerful than they were years ago, the additional overhead of using 256-bit keys is no longer a reason not to.

We are aware that the key schedule concerns surrounding 256-bit AES keys, and we will continue to keep an eye on attacks against it. We can modify our format and cipher suites to return to 128-bit keys if necessary. The MAC keys are also all 256-bit keys.

All keys (other than the key encryption keys derived from the Master Password) are created randomly using cryptographically appropriate random number generation.

UUIDs

As in the Agile Keychain format, each item is associated with a unique ID, the UUID. These are 128-bit numbers that are chosen at random when an item is first created. In what follows, we will often use the term “UUID” to refer to the uppercase hexadecimal representations of a UUID. It should be clear from context when we mean the hexadecimal representation and when we mean the number itself.

Because each UUID is chosen at random, it contains no information about the content of an item. When a user modified information in an item the UUID remains the same, although the time stamp associated with it will change.

opdata01

All encrypted data with the exception of some encrypted keys is a specific format we are calling “opdata01”.

The first 8 bytes of the data are the string “opdata01”. The next 8 bytes contain the length in bytes of the plaintext as a little endian unsigned 64 bit integer. The next 16 bytes are the randomly chosen initialization vector.

The plaintext is padded using the following scheme.

If the size of the plaintext is an even multiple of the block size then 1 block of random data is _pre_pended to the plaintext, otherwise between 1 and 15 (inclusive) bytes of random data are _pre_pended to the plaintext to achieve an even multiple of blocks.

The data, including the IV and prepended padding, is encrypted using AES in CBC mode with a 256-bit encryption key.

The HMAC-SHA256 is computed over the entirety of the opdata including header, length, IV and ciphertext using a 256-bit MAC key. The 256-bit MAC is not truncated. It is appended to the ciphertext.

Depending on where opdata is stored within data files it may be base64 encoded.

Implementation

On the Mac and on iOS we use the CommonCrypto libraries, using kCCHmacAlgSHA256 and kCCAlgorithmAES128. We set up the cryptor with

CCCryptorCreate(kCCEncrypt, kCCAlgorithmAES128, 0, encryptionKey, 32, IV, &cryptor)

and the MAC with

CCHmacInit(&context, kCCHmacAlgSHA256, MACKey, kCCKeySizeAES256)

Rationale

We perform authenticated encryption for the reasons described in the authenticated encryption section. CommonCrypto does not, at this time, offer any direct authenticated encryption modes, so we use Encrypt-then-MAC composition.

We use CBC mode because at this time support for CTR mode is not fully supported in CommonCrypto. With OS X 10.8 and iOS 6 CCCryptorCreateWithMode() appears to be available, but is undocumented outside of the source itself.

We do not use kCCOptionPKCS7Padding as this can lead to padding oracle CCAs (although our use of Encrypt-then-MAC should prevent all CCAs). Instead we do our own random padding as described above. By putting the padding up front, this has the effect of acting like a smaller, additional, IV.

Note that IETF draft for authenticated encryption with associated data was not available in time for our first version of this, but we will certainly consider the simpler padding proposed there in the future.

Random number generation

1Password on iOS and OS X uses SecRandomCopyBytes() throughout for its cryptographically secure random numbers. These are used to create the master keys (encryption and MAC), overview keys (encryption and MAC), and PBKDF2 salt when a new Keychain is created. Subsequently, random numbers are used for item keys (encryption and MAC), initialization vectors for CBC encryption, random CBC padding, and UUIDs.

Directory layout

The top level directory is called onepassword_data and will contain a directory for each profile, currently default is the only profile. The default folder will contain 16 band files, a folders.js file, a profile.js file, and any number of attachments.

Here is the layout for a sample Cloud Keychain:

onepassword_data
`-- default
    |-- 026AA7B7333B4F925F16DE9E21B912B7_5754B83288A34CD39DE64B45C2F05A9D.attachment
    |-- 6F8CDF100CC99FD55053B77492D97487_072A1462CBDE4E2488FB2DA16D96B84B.attachment
    |-- 7BAFBD42CA6F487C94DEC95C187A88A7_C10EA572E11246019A76C4657E302C8B.attachment
    |-- BD1B7C38FDC647E188E172DE2C06C391_A766BB1A9D484174B0D3453E8142118C.attachment
    |-- EE2B42ABEB4D4276AEA51006B3A9C47C_CC20BEA583014AD38E2899853CC48DB8.attachment
    |-- F6A28725D3F0F4DCB857C62A10E612F6_E390C5572D1848E3B128C416162CA60B.attachment
    |-- band_0.js
    |-- band_1.js
    |-- band_2.js
    |-- band_3.js
    |-- band_4.js
    |-- band_5.js
    |-- band_6.js
    |-- band_7.js
    |-- band_8.js
    |-- band_9.js
    |-- band_A.js
    |-- band_B.js
    |-- band_C.js
    |-- band_D.js
    |-- band_E.js
    |-- band_F.js
    |-- folders.js
    `-- profile.js

Band files

An individual band file, say, band_4.j, will contain all of the items with UUIDs that begin with ‘4’.

The contents of a band file containing two Login items may look like

ld({
  "468B1E24F93B413DAD57ABE6F1C01DF6": {
    "category": "001",
    "created": 1325483950,
    "d": "b3BkYXRhMDFcAQ ...",
    "folder": "C8CE328220DF4157961787FBA30DAB96",
    "hmac": "30d0rq4tFQvU++mIOElzzI1zilZS0M4Ya10TXmsFz2c=",
    "k": "KGRbxc/qWol4sbs7bwwctO ...",
    "o": "b3BkYXRhMDGCAAAAAAAAAB0ZLRV9xZidd ...",
    "tx": 1347560906,
    "updated": 1325483950,
    "uuid": "468B1E24F93B413DAD57ABE6F1C01DF6"
  },
  "4E36C011EE8348B1B24418218B04018C": {
    "category": "001",
    "created": 1325483950,
    "d": "b3BkYXRhMDFcAQAAAAAAAIyTActXjlxJEgx ...",
    "hmac": "T/8VMqYTr8WJDJqPrJyJl+bFcmS28dn2ZBICwS6d+Tw=",
    "k": "87VRNWy3Pj6 ...",
    "o": "b3BkYXRhMDFyAAAAAAAAADkKPiRdEPFXmLQG9T ...",
    "tx": 1347560906,
    "updated": 1325483951,
    "uuid": "4E36C011EE8348B1B24418218B04018C"
  }
});

Items

And individual item within a band file may look like

"468B1E24F93B413DAD57ABE6F1C01DF6": {
    "category": "001",
    "created": 1325483950,
    "d": "b3BkYXRhMDFcAQ ...",
    "folder": "C8CE328220DF4157961787FBA30DAB96",
    "hmac": "30d0rq4tFQvU++mIOElzzI1zilZS0M4Ya10TXmsFz2c=",
    "k": "KGRbxc/qWol4sbs7bwwctO ...",
    "o": "b3BkYXRhMDGCAAAAAAAAAB0ZLRV9xZidd ...",
    "tx": 1347560906,
    "updated": 1325483950,
    "uuid": "468B1E24F93B413DAD57ABE6F1C01DF6"
  }

The item is labeled by its UUID which is also one of the elements within the data. The MAC is calculated over all of the items (in C lexicographic order) except for hmac itself and folder. The folder is excluded because there may be legitimate instances of an item changing its folder when the MAC key is unavailable.

category

Item category indicates whether the item is a Login (“100”), Secure Note (“003”), Credit Card (“002”), etc. These are three digit decimal numbers.

Current categories are

Login 001
Credit Card 002
Secure Note 003
Identity 004
Password 005
Tombstone 099
Software License 100
Bank Account 101
Database 102
Driver License 103
Outdoor License 104
Membership 105
Passport 106
Rewards 107
SSN 108
Router 109
Server 110
Email 111

Rationale

category is left unencrypted so that a list of, say, Email items can be presented without having to decrypt the overviews of every item in within the keychain. If category was specified within the encrypted overviews then it would be necessary to decrypt every overview in order to identify the particular items to be listed.

created

Unix time of item creation. All time stamps are Unix time and are written as an ASCII decimal number. Internal representation of Unix time stamps involves data types that are based on 64 bit integers and so will handle dates beyond January 2038.

d

This is the encrypted details for the item. It is base64 encoded opdata. It is encrypted using the item keys in “k”.

fave

The presence of fave indicates that the item is to be listed among “favorites”. The value, an ASCII representation of an unsigned long integer is used as a sort index, with lower numbers sorted first.

It is useful to have large gaps in the sort indices, so that items added later to favorites can be inserted in the sort order without having to renumber existing items. For example, the item to be sorted first may have a sort index of 1000 and the item to be sorted second may have a sort index of 2000. The allows someone to add a new favorite item to be sorted be sorted between those two, sort index 1500, without having to modify the existing items.

Rationale

fave, like category, is left unencrypted so that a list of favorites can be displayed without having to decrypt the overviews of every item in the keychain. If fave were to be among the overview data, then the overviews of all items would need to be decrypted to identify which items are favorites.

folder

This is the UUID of any folder an item may belong to.

hmac

This MAC (HMAC-SHA256) is computed over all of the elements of the item with the exception of the hmac itself. It’s computed over the item elements and their values. Very pseudo code:

elements = Sort_with:@selector(compare:);
for (element in elements) {
	if (element == "hmac") continue;

	CCHmacUpdate(&context, [element UTF8String], [element length]);

	*data = [dict[element] description];
	CCHmacUpdate(&context, [data UTF8String], [data length]);
}

The HMAC is computed using SHA256 using the overview MAC key, and it is base64 encoded.

k

This is the encrypted item and MAC keys. It is encrypted with the master encryption key and authenticated with the master MAC key.

The last 32 bytes comprise the HMAC-SHA256 of the IV and the encrypted data. The MAC is computed with the master MAC key. The data before the MAC is the AES-CBC encrypted item keys using unique random 16-byte IV.

The decrypted object is defined by

typedef struct {
  uint8_t crypto_key[32];
  uint8_t mac_key[32];
};

This is base64 encoded.

Rationale

One reason to provide a separate key for each item is to ensure that only a safe amount of data is encrypted under one key. Although we don’t expect item details to grow particularly large, any attachment to an item is also encrypted under the item key.

o

This is the encrypted overview for the item. It is base64 encoded opdata, encrypted with the overview encryption and MAC keys.

There are still cases where we need to find items without having to decrypt all of the data. So we separate the data in each item into “overview data” and “encrypted data”. In the clients, the overview data may remain unencrypted (in memory) for the entire time that 1Password is unlocked, while the restricted data for any item is only briefly decrypted.

The decrypted overview data includes the URLs (yes, that is plural) associated with an item, the Title associated with the item, and some additional information depending on the category of an item. For a Login, the username for the item is included. For a Password, the creation time is included. For a secure note, the first 80 bytes of the note are included. For an Identity, the full name is included. For a Credit Card, the masked credit card number is included.

trashed

If "trashed": true appears in the item, it indicates that the item has been moved to the trash. The default for trashed is false, so the absence of it indicates that the item is not in the Trash.

tx

This is the “transaction timestamp” (Unix time) indicating when this particular item was last modified or added to the current database through synchronization. When an item is modified or added to the database through synchronization, this item will be updated to reflect that. This along with the updated timestamp are used to manage synchronization.

updated

This is a Unix timestamp indicating when the contents of the item were last modified.

uuid

This is the UUID of the item. Its inclusion within the item ensures that it is included when the hmac is calculated.

folders.js

The file folders.js has a similar structure to a band file, and each folder listed has a structure very similar to an item in the band file. For security purposes, it is worth noting that the folder name and the search query for defining a smart folder are encrypted with the overview key. In this version, other than the actual encrypted data, there is no MAC for each folder nor for the list of folders themselves.

Here is a sample entry

"379A3A7E5D5A47A6AA3A69C4D1E57D1B": {
    "created": 0,
    "overview": "b3BkYXRhMDES ...",
    "tx": 1373753421,
    "updated": 0,
    "uuid": "379A3A7E5D5A47A6AA3A69C4D1E57D1B"
}

The overview is encrypted with the overview keys, and contains the name of the folder for ordinary folders.

In this sample, it is clear that this folder was created and last modified at the very beginning of time itself, though possibly it was merely created while there were still some bugs in the pre-beta version of 1Password used to create it this data sample.

Some folders may also contain "smart": true,, indicating that it is a smart folder. The overview will in those cases will contain, in addition to its name, a definition of the search predicate used to define the smart folder. The specification of those is still in flux and is not, at this point, expected to be portable.

profile.js

This file contains the encrypted master and overview keys. Here is a sample

var profile={
	"lastUpdatedBy":"Dropbox",
	"updatedAt":1325483930,
	"profileName":"default",
	"salt":"LlfotX7aTZnfL+AjJJYARA==",
	"passwordHint":"fred",
	"masterKey":"b3BkY ...",
	"iterations":227272,
	"uuid":"FFF0089247244BDEAAE714B9AC6FF129",
	"overviewKey":"b3BkYXRhMD ...",
	"createdAt":1347560904};

The masterKey and the overviewKey are in base64 encoded opdata objects. iterations is the number of PBKDF2 iterations. salt is base64 encoded 16 byte random salt. These will be described in more detail in the section on key derivation.

lastUpdatedBy is used to help make the syncing process more efficient. The password hint is stored unobfuscated. We did not feel that we could devise an obfuscation system that could sufficiently resist reverse engineering for the very limited gain of obfuscating the hint.

Key derivation

We have been able to simplify key derivation because we no longer have multiple security levels of the Agile Keychain format, where each security level required its own master key. The key derived from the user’s Master Password is used to authenticate and decrypt the masterKey and the overviewKey in profile.js.

From Master Password to derived encryption and MAC keys

The Master Password is converted to a UTF8 null terminated string. This is then used in the Common Crypto key derivation function CCKeyDerivationPBKDF, using PBKDF2, the salt, and HMAC-SHA512 as the pseudorandom function to derive the encryption key and the MAC key. The first 256-bits are the derived encryption key and the remaining 256-bits are the derived MAC key.

Using SHA2-512 serves two purposes:

  1. It gives us enough raw material to for both an encryption key and a MAC key (each 256-bits).
  2. By using a wide hash (512 bits) in PBKDF2, we make it more difficult for GPUs which are constrained by their memory access limitations.

At the moment our choice of PBKDF2 over scrypt is largely motivated by the fact PBKDF2 is available in the system libraries for the platforms that we use. We may migrate to scrypt, or some other successor to PBKDF2, in future.

From derived keys to master keys and beyond

The derived keys are used to decrypt the contents of masterKey and overviewKey. The masterKey and the overviewKey are base64 encoded opdata objects.

The plaintext of the masterKey is 256 bytes of data selected randomly when the keychain was first created. The plaintext of the overviewKey is 64 bytes selected randomly when the keychain was first created.

The 256 byte (2048 bit) plaintext content of the masterKey is then hashed with SHA-512. The first 32 bytes (256-bits) of the resulting hash are the master encryption key, and the second 32 bytes are the master hmac key.

The master keys (encryption and MAC) are used to encrypt the item specific item keys within each item. These in turn are used to encrypt the data within the item and also any attachment associated with the item.

The overview keys (encryption and MAC) are derived in the same way as the master keys. The contents of overviewKey is verified and decrypted with the derived keys; the result is then hashed with SHA-512; the first 32 bytes of the hash output will be the overview encryption key; the last 32 bytes of the hash output will be the overview hmac key.

The overview key is used for encrypting the overviews in each item as well as the overviews in for each folder.

Attachments

Attachment files are named with the UUID of the item that they are attached to followed by an underscore and then followed by the UUID of the attachment itself. The file is then given the extension “.attachment”.

The contents can be divided into four sections.

  1. The header
  2. The meta-data (up to 216 bytes)
  3. The encrypted icon (up to 232 bytes)
  4. The encrypted data (up to 232 bytes)

Counting from zero

Bytes 0-6 The literal OPCLDAT
Byte 7 Version (0x01 for this first version)
Bytes 8-9 Metadata size in bytes [little endian]
Bytes 10-11 Junk [structure padding]
Bytes 12-15 Icon size in bytes [little endian]

metadata

This is then followed by a JSON object containing the meta data (displayed here with whitespace, but there is no whitespace in the actual attachment file.)

{
	"itemUUID":"EE2B4 ...",
	"contentsSize":277153,
	"external":true,
	"updatedAt":1328222369,
	"txTimestamp":1347487884,
	"overview":"b3BkYXRhMDEiAAAAAAAAACMP4CJz/gLE4lwhnyahSY0jBL ...",
	"createdAt":1328222369,
	"uuid":"CC20BE..."
}

encrypted icon

This is opdata. In attachment files these are not base64 encoded.

encrypted contents

This is opdata, encrypted with the associated item and MAC keys. In attachment files these are not base64 encoded.

Sample data and further details

For those who wish to explore and analyze this further, we posted a sample keychain.

Although we don’t anticipate publishing source code for manipulating 1Password keychains, others, unaffiliated with AgileBits, have begun to.

Copyright © 2014 AgileBits Inc.