Database Encryption and Configuration

Session 171 completed two features that bridge the gap between "works on my machine" and "ready for production": database encryption and configuration management. Encryption ensures that data at rest is unreadable without the correct password -- critical for applications handling sensitive information. Configuration management ensures that the same FLIN application behaves correctly in development, testing, and production environments.

Thirty-nine tests. Six hundred lines of configuration code. Three hundred lines of encryption code. Three new Rust dependencies. By the end of Session 171, FlinDB was 97% complete.

Part 1: Backup Encryption

The Security Requirements

An unencrypted backup file is a liability. If someone gains access to the file -- through a misconfigured S3 bucket, a stolen laptop, or a compromised server -- they can read every record in the database. For applications handling personal data (healthcare, financial, educational), this is a compliance violation.

FlinDB's encryption uses industry-standard algorithms with OWASP-recommended parameters:

AES-256-GCM

AES-256-GCM (Advanced Encryption Standard with 256-bit key in Galois/Counter Mode) provides both confidentiality and integrity. The "GCM" part is critical -- it produces an authentication tag alongside the ciphertext. If even one bit of the encrypted data is modified (accidentally or maliciously), decryption fails. This prevents both corruption and tampering.

Argon2id Key Derivation

A password is not a cryptographic key. "my-secure-password" is 19 bytes of low-entropy text. AES-256 needs exactly 32 bytes of high-entropy key material. Key derivation functions (KDFs) bridge this gap.

We chose Argon2id because it is the winner of the Password Hashing Competition and the OWASP-recommended KDF for 2024+. Unlike PBKDF2 (which can be parallelized on GPUs) or bcrypt (which has a fixed memory requirement), Argon2id requires significant memory (64 MiB per derivation), making brute-force attacks expensive even on specialized hardware.

The Encryption API

Creating an encrypted backup:

rustlet options = BackupOptions::encrypted("my-secure-password");
Backup::full(&db, "backup.flindb.bak", options)?;

Restoring an encrypted backup:

rustlet options = RestoreOptions::with_password("my-secure-password");
let db = Backup::restore("backup.flindb.bak", options)?;

Checking if a backup is encrypted:

rustlet is_encrypted = Backup::is_encrypted("backup.flindb.bak")?;

The Encryption Flow

The encryption process has five steps:

1. Serialize: Convert the database state to JSON
2. Compress: Apply Zstd compression (optional, enabled by default)
3. Derive key: Use Argon2id to derive a 256-bit key from the password
4. Generate nonce: Create a random 12-byte nonce (unique per backup)
5. Encrypt: Apply AES-256-GCM with the derived key and nonce

Decryption reverses the process:

1. Read metadata: Extract the salt and nonce from the backup header
2. Derive key: Use Argon2id with the same salt to derive the same key
3. Decrypt: Apply AES-256-GCM decryption
4. Decompress: Apply Zstd decompression (if compressed)
5. Deserialize: Parse the JSON back into database state

The salt is stored alongside the encrypted data in the backup file. The nonce is also stored. Neither is secret -- they exist to ensure that the same password produces different keys for different backups (salt) and that the same key produces different ciphertext for different encryptions (nonce).

Wrong Password Detection

If the wrong password is provided, key derivation produces a different key, and AES-GCM decryption fails immediately (the authentication tag does not match). The error is clear: "Decryption failed: invalid password or corrupted backup." There is no partial decryption. No garbled output. The authentication tag guarantees all-or-nothing.

The Dependencies

Three new crates were added to Cargo.toml:

tomlaes-gcm = "0.10"    # AES-256-GCM encryption
argon2 = "0.5"       # Argon2id key derivation
rand = "0.8"         # Random nonce generation

These are pure Rust implementations with no C bindings. They compile on every platform Rust supports, including WebAssembly (for future browser-side encryption).

Part 2: Configuration System

The Problem

FlinDB's zero-configuration promise works for development. But production applications need to tune behavior: WAL thresholds, logging levels, database paths, backup schedules. And test environments need different behavior entirely -- in-memory databases, verbose logging, reset between tests.

The FLIN Config Format

FlinDB's configuration uses FLIN's own syntax -- not JSON, not YAML, not TOML:

flin// flin.config
app {
    name: "MyApp"
    mode: "prod"
}

database {
    path: "./.flindb"
    wal: true
    compaction_interval: "1h"
}

backup {
    enabled: true
    interval: "1h"
    retention: 24
}

ai {
    embeddings: "local"
}

This was a deliberate choice. FLIN developers already know FLIN syntax. Requiring them to learn TOML or YAML for configuration adds unnecessary cognitive load. The config parser is a simplified version of FLIN's own parser, handling string values, numbers, booleans, and nested blocks.

Environment Modes

Three modes with different defaults:

rustpub enum AppMode {
    Dev,
    Test,
    Prod,
}

Test mode is the most interesting. The database is entirely in-memory -- no disk I/O, no file cleanup between tests, no test isolation issues. When the test process exits, the database vanishes. This makes FLIN's test suite fast and deterministic.

Duration Parsing

Configuration values that represent time durations support human-readable formats:

rustfn parse_duration(s: &str) -> Option<Duration> {
    if s.ends_with("ms") {
        s[..s.len()-2].parse::<u64>().ok().map(Duration::from_millis)
    } else if s.ends_with("s") {
        s[..s.len()-1].parse::<u64>().ok().map(Duration::from_secs)
    } else if s.ends_with("m") {
        s[..s.len()-1].parse::<u64>().ok().map(|m| Duration::from_secs(m * 60))
    } else if s.ends_with("h") {
        s[..s.len()-1].parse::<u64>().ok().map(|h| Duration::from_secs(h * 3600))
    } else if s.ends_with("d") {
        s[..s.len()-1].parse::<u64>().ok().map(|d| Duration::from_secs(d * 86400))
    } else if s.ends_with("w") {
        s[..s.len()-1].parse::<u64>().ok().map(|w| Duration::from_secs(w * 604800))
    } else {
        None
    }
}

So "1h" means one hour, "30m" means thirty minutes, "5000ms" means five seconds, and "1w" means one week. No more counting seconds ("is 3600 one hour or one day?").

Environment Variable Overrides

Every configuration value can be overridden by an environment variable:

Environment variables take precedence over the config file. This follows the twelve-factor app principle: configuration that varies between deploys (dev vs staging vs production) should come from the environment.

Config File Discovery

The configuration system searches for config files in three locations, in order:

1. flin.config in the project root
2. ~/.flin/config in the user's home directory
3. /etc/flin/config for system-wide defaults

The first file found is used. This supports both project-specific configuration (most common) and user-wide or system-wide defaults (useful for CI/CD environments where every project should use the same database path).

The Config Struct

rustpub struct FlinConfig {
    pub app_name: String,
    pub mode: AppMode,
    pub log_level: LogLevel,
    pub database: DatabaseConfig,
    pub backup: BackupConfig,
    pub ai: AiConfig,
}

pub struct DatabaseConfig {
    pub path: String,
    pub wal: bool,
    pub compaction_interval: Duration,
}

pub struct BackupConfig {
    pub enabled: bool,
    pub interval: Duration,
    pub retention: usize,
    pub compression: bool,
}

The struct is fully typed. No string-keyed hashmaps. No dynamic access. The compiler verifies that every configuration value is used correctly.

The Thirty-Nine Tests

Session 171's test suite was comprehensive:

Configuration tests (29): - Default values for all settings - Parsing each config section (app, database, backup, ai) - Dev/Test/Prod mode defaults - AppMode string conversion - LogLevel ordering and comparison - Duration parsing for all units (ms, s, m, h, d, w) - Environment variable overrides (mode, path, WAL, log level) - Config value type validation - Config discovery with no file present

Encryption tests (10): - Encrypt/decrypt roundtrip - Wrong password rejection - Empty data encryption - Large data encryption (stress test) - Full backup with encryption roundtrip - Wrong password on encrypted backup - No password on encrypted backup (error) - Encrypted + compressed backup - Unencrypted backup detection - Encryption metadata serialization

The encryption roundtrip test is the most critical -- it verifies that data encrypted with a password can be decrypted with the same password, producing identical output. The wrong-password test is equally important -- it verifies that decryption with the wrong password fails cleanly rather than producing garbage.

FlinDB at 97%

After Session 171, FlinDB's implementation status was:

The remaining 3% was primarily real embedding models (replacing mock embeddings with actual multilingual-e5-small), advanced index types, and the remaining configuration options. The core database engine -- CRUD, constraints, queries, transactions, backup, encryption, event sourcing, and configuration -- was complete.

Total test count: 2,404 (1,787 library + 617 integration).

The gap between "interesting project" and "production database" is measured in encryption algorithms, configuration management, and thousands of tests. Session 171 closed that gap.

This is Part 11 of the "How We Built FlinDB" series, documenting how we built a complete embedded database engine for the FLIN programming language.

Series Navigation: - [064] Graph Queries and Semantic Search - [065] The EAVT Storage Model - [066] Database Encryption and Configuration (you are here) - [067] Tree Traversal and Integration Testing - [068] FlinDB Hardening for Production