JemDrive - Technical overview

JemDrive was designed to tick the following boxes:

  • a holistic view of information: notes AND files, side by side. It’s all information. Files don’t exist in a vacuum, they come with context. Context is what we write down in a note. I want the files to live in the notes, next to their context.
  • low tech[1]: text-files and folders. The data should be accessible using a text editor and/or file explorer, don’t tie me to a single app or service. It should be accessible in 20 years, from any platform.
  • sync: it must be accessible across devices (phone, laptop, etc.)
  • write it down when it happens. 95% of the time, if I didn’t write it down when it happened I never wrote it down at all
  • WHEN is a key word: organizing things FIRST by subject or theme, and THEN chronologically is a powerful storage format.
  • A month is a great length of time for grouping files together: if I take any one subject and put all the files related to it from the span of a single month into one folder, I can usually find what I want in there.
  • encryption. Information in our digital age increasingly includes many secrets: passwords, PINs, recovery questions, etc. These need a safe place I can store them.

Below is a (technical) description of how JemDrive is designed in accordance with these principles.

Storage of files and notes, tags

JemDrive uses text files to store notes and timelines, using the markdown format. Notes are regular markdown files, while timelines use a specific timeline format. JemDrive treats any file whose extension is .timeline.md as a timeline, and any other file whose extension is .md as a note. For example,some-note.md is a note, while some-project.timeline.md is a timeline.

Files are stored in folders next to the timelines/notes, grouped into folders according to the month during which they were added. So for example, if we have a timeline called project-a, there will be a files directory next to it to store attachments, and in it folders for each month in which those attachments were added:

├── project-a.timeline.md
└── files/
     ├── 2019-01
     ├── 2019-02
     ├── ...
     └── 2022-10

By using text files, JemDrive ensures that notes and timelines stay readable across a wide variety of platforms, and far into the future, even when JemDrive itself may be unavailable. By storing files next to the timelines/notes, the files are accessible using any platform’s file explorer. Files are grouped into folders covering month-long periods, which experience has shown to be useful.

Timelines and notes are stored in folders, which act as tags. So for example, if we have multiple timelines for home, say home-improvement and insurance, then we might create a folder called home to act as a tag, and have two timelines inside that folder:

home
 ├── insurance.timeline.md
 ├── home-improvement.timeline.md
 └── files/
      ├── 2019-01
      ├── 2019-02
      ├── ...
      └── 2022-10

The two timelines then share the files directory for their attachments. A future enhancement is planned that would allow tagging timelines with more than one tag, by use of symlinks, where the parent folder would be a first-among-equals “primary” tag and the symlinks pointing to it would be other tags.

JemDrive stores its files in a folder called JemDrives, which it places in the “home” directory on desktop platforms. This renders the files accessible from other programs - for example, if we need to add an attachment to an email related to our home insurance from January 2022, we can use the platform’s file explorer to browse to

JemDrives/My Drive/home/files/2022-01/

and find the file we need.

This format also makes JemDrives backup friendly: there is no need to export the data, just copy the JemDrive right out of ~/JemDrives/.

On mobile platforms, due to data access restrictions, this is less flexible, and the JemDrive folder cannot be accessed directly by other applications - files must be accessed from within the app.

Sync

JemDrive uses git for syncing data between devices. When the app first runs it generates a private/public key pair specific to that device, and the private key is stored in the keychain. When you create/add a device to a JemDrive account, the public key is transmitted to the server, where it is used to authenticate that device.

Each drive corresponds to a git repository. Repositories created in the app do not have an associated repository - they must be either added to a JemDrive account, or alternatively you may specify your own private git repository to use for that drive. This last option is the basis of the “subscription-less offer” for developers, for which one would have to add the public key to their custom repository and then configure the git server in the drive’s sync options.

Sync (when enabled) happens automatically in the background, subject to restrictions in some platforms (especially iOS, where currently there is no background sync). It can also be triggered manually, via the sync pane from the sidebar. JemDrive delegates conflict handling to the user, allowing them to compare local vs remote versions and asking them to choose.

Text-editing

Text editing is done via a block-based editor, which treats each paragraph as a “block”, where blocks are specialized to the type of information contained in the block. So for example there are regular paragraphs, lists, indented blocks, encrypted blocks, attachment blocks, image blocks, etc. Each block has separate operations that may be applied to it, and that are available in the editing action bar at the bottom of the screen when that block is being edited.

Encryption

JemDrive supports 128-bit AES encryption in CBC mode. Each drive has its own encryption key. The encryption key may be generated using a secure random generator, or entered manually. Each encryption key is protected by a password, which is used to encrypt and decrypt the encryption-key, which is itself stored in the keychain.

Neither the encryption key nor the password are ever transmitted off of the device by JemDrive, and once entered an encryption key is never again shown to the user in the clear.

In day-to-day encrypting and decrypting, only the password is asked of the user. This is used to decrypt the encryption key and then encrypt or decrypt the data. The encryption key itself is only required when configuring a new device.

This design has several consequences:

  • JemDrive’s servers can never know the encryption key chosen by a user (even for devices which use JemDrive accounts for sync). This significantly reduces the attack surface for a malicious actor to obtain the encryption key.
  • The responsibility of keeping and protecting the encryption key falls to the user. If it is lost, encrypted data will (most likely) become inaccessible.

Currently encryption is supported for individual blocks of text. Encrypted data is stored using the following format:

--ENCRYPTED--
base64_encoded([magic-byte][128-bit-IV][cipher-text])

The magic byte is used to version the encryption format and as a sanity check to detect encrypted content.

A future enhancement is planned that would allow encrypting and decrypting entire files - both entire notes as well as attachments.

Timeline format

Timelines are markdown files that use the extension .timeline.md and which contain notes ordered in increasing chronological order according to date, with the date specified as a level-2 header in YYYY-mm-dd format:

## 2022-10-01

<markdown note....>

## 2022-10-03

<another markdown note....>

Date headers SHOULD have a blank line before and after them, and SHOULD appear in increasing chronological order in the file.

Timeline files can “contain” attachments, which are specified as markdown-links into a files folder that is a sibling of the timeline file, and more specifically into a subfolder specific to the month corresponding to the date header under which that file appears. So for example an attachment file form.pdf that is labeled my form and is under the date header 2022-10-03 would look like:

## 2022-10-03

....

  [my form](files/2022-10/form.pdf)

And on disk:

├── project-a.timeline.md
└── files/
     └── 2022-10
          └── document.pdf

Any relative link in the timeline that starts with files/(YYYY)-(mm) SHOULD be an attachment that is present in the files folder.

Due to this design, sibling timelines share a files folder. If a timeline is moved, any attachments in the folder SHOULD be moved along with it, unless other sibling timelines also reference it, in which case they SHOULD be copied instead.

A future enhancement is intended to provide support for partitioning a single timeline into multiple files, probably by appending the year to the timeline’s name, e.g.:

project-a.2020.timeline.md
project-a.2021.timeline.md
project-a.2022.timeline.md

Would all be considered one timeline that is spread across three files, with dates belonging to the each year contained in the corresponding file. This functionality is not yet implemented in JemDrive.