Service database

The service has a built-in database for storing devices, admins, remote users, labels, and so on. By default, RADKit uses an encrypted file on disk, but there are multiple storage options, each with their own advantages/disadvantages. Besides the encrypted file, we support a Postgres database, or an in memory database.

Regardless of the storage backend, the Database constraints remain the same.

Storage backends

Encrypted file

By default, all data is stored in an encrypted file. The superadmin password is used to open the vault from where the encryption key is taken to decrypt the database file. Only one process at a time can interact with this encrypted file.

The default file location on Linux and Mac OS for the service database is here:

~/.radkit/service/service-db.json.encrypted

A different location can be configured using the service_database_name configuration option:

[service.database]
# These are the default options:
storage_backend = "encrypted-file"
service_database_name = "service-db.json.encrypted"

Warning

It is not recommended to store this database file on a network filesystem. The database uses advisory locks in order to prevent concurrent writes, but this is often not reliable over filesystems like NFS.

There are two layers of encryption: the database file as a whole is an encrypted JSON blob, with a plain text header on top. Secondly, passwords within that JSON blob are encrypted once more with a separate encryption key. Both encryption keys are stored in the vault. Having these two layers of encryption makes it possible to decrypt the outer layer for debugging, without revealing all the passwords.

In the vault, these encryption keys are identified by the following names:

db-secrets-encryption-key/{db_uuid}
db-encryption-key/{db_uuid}

This storage backend can be used by multiple processes at the same time, but it’s rather slow when both processes are expected to do writes.

In memory storage

The in-memory storage is initialized as an empty database every time the service starts. This storage doesn’t persist any data. It can be useful for testing or for situations where the database is populated each time through radkit-control right after it starts.

In order to use the in-memory storage backend, use the following config:

[service.database]
storage_backend = "in-memory"

Postgres database

The Postgres storage backend is useful for situations where scalability is important; for when there is a need to store the data on a different server; or for when different back-up strategies or replication is important.

Unlike the encrypted file, when data is stored in Postgres, it’s not fully encrypted. Only the passwords within the database are encrypted. The Postgres database is assumed to be protected by a sufficiently strong authentication method.

[service.database]
storage_backend = "postgres"

[service.database.postgres]
database = "database-name"
username = "postgres-user"
password = "postgres-password"
host = "localhost"
port = 5432

Once radkit-service is started, the password is automatically moved into the vault, and the settings file is updated so that it refers to the vault like this:

[service.database]
...
password = [
    "from_vault",
    "encrypted-setting/service.database.postgres.password/719f989d-d99d-4961-b348-db99809ee26d"
]
...

The Postgres database storage allows for multiple radkit-service processes to connect to the same database, but only if:

they all have the required encryption key in their vault;
they are running the same RADKit version (at least with the same database schema).

If one radkit-service process tries to migrate the database, other radkit-service processes that assume an older database schema will terminate and can no longer use the database until they are upgraded.

One note: the Postgres database backend isn’t used like a traditional relational database. It’s rather a storage backend for an in-memory database. Every radkit-service process keeps an in-memory read replica of the data of the Postgres database for querying. This read-replica is also used as a starting point of any write transaction. When a write transaction terminates, it will be merged into the Postgres database if our data models validate after applying the change on top of the most recent data from the DB.

Connecting directly to this Postgres database from application other than radkit-service is NOT recommended and can break the data model. However, it is possible to run a separate radkit-service instance as a REST API on top of the Postgres database.

Import and export as JSON

In order to export the database as JSON, run the following command. This will export from the currently configured storage backend (if not in-memory):

radkit-service export-db --output dump.json

Use the --decrypt-secrets option to also decrypt the passwords in the JSON structure.

To import the database from a JSON file, do:

radkit-service import-db --input dump.json

Note that for importing a JSON dump, if the secrets within the JSON dump were not decrypted, a corresponding encryption key in the vault is required.

Note

Both the import-db and export-db commands can be performed while the service is running. This works for both the encrypted file storage as well as the Postgres storage.

For the in-memory storage these commands don’t work, because there is no way for an external process to get access to the data. Things like device names and their attributes can be exported through radkit-control, but there is no way to get access to any credentials in this case.

Snapshots

In order to create a complete snapshot of all radkit-service files, including the database (if an encrypted file is used), use the radkit-service snapshot capture command.

radkit-service snapshot capture

Similarly, the radkit-service snapshot restore command will restore the snapshot by importing it in the current Postgres database.

A snapshot contains only copies of local files, so if Postgres is used as a storage backend, no back-up of the data in the database will be taken. In that case, use the export-db command from above to export the database content as JSON, or use Postgres specific back-up tooling.

Note

A snapshot can be captured while the service is running, because it copies files out of the radkit directory. However, in order to restore a snapshot, the service must not be running.

Database constraints

The RADKit service database imposes some constraints to the amount of data that can be stored in order to guarantee flawless operation. The service will refuse to store any data that will surpass any of these limits. There are no configuration options for changing this behavior.

Maximum number of entries for each table

radkit_service.database.constraints.MAX_DEVICES = 50000: Maximum number of devices.

radkit_service.database.constraints.MAX_LABELS = 1000: Maximum number of labels.

radkit_service.database.constraints.MAX_EXTERNAL_SOURCES = 1000: Maximum number of external sources.

radkit_service.database.constraints.MAX_DEVICE_TEMPLATES = 1000: Maximum number of device templates.

radkit_service.database.constraints.MAX_REMOTE_USERS = 1000: Maximum number of remote users.

radkit_service.database.constraints.MAX_ADMINS = 1000: Maximum number of admins.

Metadata specific constraints

There are additional constraints specifically for the device metadata:

radkit_service.database.constraints.MAX_METADATA_ENTRIES_PER_DEVICE = 200: Maximum number of metadata entries for a single device.

radkit_service.database.constraints.MAX_METADATA_KEY_LENGTH = 50: Maximum length for a metadata key (which is a string).

radkit_service.database.constraints.MAX_METADATA_VALUE_LENGTH = 1000: Maximum length for a metadata value (which is a string).

radkit_service.database.constraints.MAX_METADATA_SIZE = 5000: Maximum metadata size (in characters) for all keys+values combined for a single device.

Other constraints

radkit_service.database.constraints.MAX_STRING_LENGTH = 10000: Max length any string in the database can have.