Database Persistence Audit

A database that does not persist data is not a database. It is a cache with pretensions. FLIN's embedded database, ZEROCORE, was designed to be invisible -- developers write save entity and the data survives server restarts. No configuration, no connection strings, no schema migrations. Just save and it persists.

Except when it does not.

Session 203 of FLIN's development was the day we discovered that entity saves were silently failing. Users would add todos through the UI, see them appear on screen, and then lose everything on page refresh. The Write-Ahead Log file was being created, but it contained zero bytes. The database was going through the motions of persistence without actually persisting anything.

The audit of this session, documented as AUDIT-SESSION-203-DATABASE-PERSISTENCE, became one of the most instructive debugging exercises in FLIN's history. Not because the bugs were complex -- they were not -- but because three separate, independent bugs conspired to produce a single symptom, and fixing any two of the three would still leave the system broken.

The Symptom

The report was simple and devastating:

Running: flin dev from embedded/todo-app/
1. User adds a todo via the UI
2. Action handler executes (returns 302 redirect)
3. Todo appears on screen
4. WAL file created: 0 bytes
5. Page refresh: all todos gone

The WAL (Write-Ahead Log) is ZEROCORE's persistence mechanism. Every save, delete, and destroy operation writes a log entry to the WAL file before modifying in-memory state. On recovery, the WAL is replayed to reconstruct the database. A WAL with zero bytes means no operations were ever logged -- yet the UI showed that the save had apparently succeeded.

The Audit Methodology

The investigation followed FLIN's standard audit methodology -- trace the code path from HTTP request to disk write, verifying each step:

1. Trace the action request flow from HTTP POST to database save
2. Verify entity schema registration
3. Trace OpCode::Save execution through the VM
4. Verify database.save() parameters
5. Trace the WAL write path
6. Compare test flow vs. server flow

rust// The investigation started here: where does the save happen?
// server/http.rs -- action handler
async fn handle_action(req: Request) -> Response {
    let vm = create_vm_for_action(&req)?;

    // Inject form data as global variables
    for (key, value) in req.form_data() {
        vm.set_global(key, Value::Text(value));
    }

    // Execute the FLIN source (which contains the save logic)
    vm.run(&compiled_bytecode)?;

    // Return redirect
    Response::redirect(302, &req.referer())
}

The flow looked correct. Form data was injected as global variables, the bytecode executed (including the addTodo() function that performs the save), and the response was sent. Every step produced no errors. And yet, the WAL was empty.

Root Cause 1: Bytecode Overwrites Injected State

The first bug was in the interaction between state injection and bytecode execution. When the action handler injected form data as global variables, the subsequent bytecode execution re-ran the top-level variable initialization -- which overwrote the injected values.

rust// The sequence of operations:

// Step 1: Action handler injects form data
vm.set_global("newTodo", Value::Text("Buy groceries"));

// Step 2: VM executes bytecode, which includes:
//   newTodo = ""  (the initialization from FLIN source)
//
// OpCode::StoreGlobal("newTodo", Value::Text(""))
// This OVERWRITES the injected "Buy groceries"!

// Step 3: addTodo() function checks:
//   if newTodo.trim() != ""
//   But newTodo is now "" -- condition is false!

// Step 4: save todo -- NEVER EXECUTED

The fix introduced protected globals -- variables that cannot be overwritten by bytecode execution:

rust// Added to VM struct
protected_globals: HashSet<String>,

// New method for action handler
pub fn set_global_protected(&mut self, name: String, value: Value) {
    self.globals.insert(name.clone(), value);
    self.protected_globals.insert(name);
}

// Modified OpCode::StoreGlobal handler
OpCode::StoreGlobal => {
    let name = self.read_constant_string(code)?;
    let value = self.pop()?;
    // Only store if not protected
    if !self.protected_globals.contains(&name) {
        self.globals.insert(name, value);
    }
}

Root Cause 2: Value::Text Not Handled by String Operations

Even after fixing Root Cause 1, the save still failed. The protected global mechanism ensured that newTodo retained its injected value. But when the FLIN code executed newTodo.trim(), the result was an empty string.

The problem was that state injection creates Value::Text variants (inline strings), but OpCode::Trim only handled Value::Object variants (heap-allocated strings). For any Value::Text input, the trim operation returned an empty string:

rust// OpCode::Trim -- BEFORE fix
let s = match &string {
    Value::Object(id) => self.get_string(*id)?.trim().to_string(),
    _ => String::new(),  // BUG: Value::Text returns empty!
};

// OpCode::Trim -- AFTER fix
let s = match &string {
    Value::Object(id) => self.get_string(*id)?.trim().to_string(),
    Value::Text(t) => t.trim().to_string(),  // Handle inline strings
    _ => String::new(),
};

This was the same category of bug as the duplicate CreateMap issue -- a failure to handle both string representations consistently. FLIN has two string representations (Value::Text for short inline strings and Value::Object pointing to heap-allocated ObjectData::String), and every string operation must handle both. The trim operation was not the only offender -- the same gap existed in extract_string() and potentially other string operations.

Root Cause 3: Validators Causing Silent Failures

With Root Causes 1 and 2 fixed, the save finally reached the database layer. But entity validation intercepted it before it could be persisted. The todo entity had validators:

flinentity Todo {
    title: text @required @min(1)
    done: bool = false
}

The @required and @min(1) validators rejected the save -- but the rejection was silent. No error was returned to the calling code. No message appeared in the console. The validator simply prevented the save from executing and returned control to the caller as if nothing had happened.

rust// The validation path -- before fix
fn validate_before_save(
    &self,
    entity: &Entity,
    schema: &EntitySchema,
) -> bool {
    for (field, validators) in &schema.validators {
        for validator in validators {
            if !validator.check(entity.get(field)) {
                return false;  // Silent rejection!
            }
        }
    }
    true
}

The temporary fix for Session 203 was removing the validators from the entity definition. The proper fix, tracked for a later session, was to make validation failures return descriptive errors that the developer could handle.

The Three-Bug Conspiracy

What made this persistence failure so difficult to diagnose was the conspiracy of three independent bugs. Fix any two of the three, and the system still appears broken:

• Fix Root Cause 1 (protected globals) but not Root Cause 2 (trim): newTodo retains its value, but .trim() returns empty, so the condition fails.
• Fix Root Cause 1 and 2 but not Root Cause 3 (validators): newTodo retains its value, .trim() works, but the validator silently rejects the save.
• Fix Root Cause 2 and 3 but not Root Cause 1: .trim() works and validators are handled, but newTodo gets overwritten to empty before any of it matters.

All three bugs had to be fixed for the persistence to work. This is the kind of bug composition that makes debugging a language runtime fundamentally different from debugging an application. In an application, you can usually isolate the problem to one cause. In a runtime, the interaction between subsystems -- state injection, opcode execution, type handling, validation -- creates emergent failures that only manifest when specific code patterns exercise all the broken paths simultaneously.

Verification

After fixing all three root causes, the audit created five new tests that verified the exact server flow:

rust#[test]
fn test_dev_server_flow_save_entity() {
    // Basic save without conditions
}

#[test]
fn test_dev_server_flow_with_state_injection() {
    // State injection with conditional save
}

#[test]
fn test_state_injection_without_condition() {
    // Isolate state injection
}

#[test]
fn test_recovery_between_vms() {
    // Save in VM1, verify in VM2 (simulates restart)
}

#[test]
fn test_entity_queries_after_recovery() {
    // Todo.all and Todo.count after recovery
}

The verification was conclusive:

Before fix:
$ ls -la embedded/todo-app/.flindb/wal.log
-rw-r--r--  1 juste  staff  0 Jan 16 12:00 wal.log

After fix:
$ ls -la embedded/todo-app/.flindb/wal.log
-rw-r--r--  1 juste  staff  177 Jan 16 12:40 wal.log

One hundred seventy-seven bytes. A single WAL entry for a single todo item. The database was finally persisting data.

Lessons from the Persistence Audit

The database persistence audit produced four principles that guided subsequent FLIN development:

State injection must be protected. When a runtime injects values into a VM for a specific purpose (like processing form data), those values must be shielded from re-initialization by the program's own code.

Value types must be handled consistently. Every operation that works on strings must handle both Value::Text and Value::Object(String). There are no exceptions to this rule.

Silent failures are unacceptable. No operation -- especially not validation -- should fail without producing a visible signal. A rejected save with no error message is worse than a crash, because the developer cannot diagnose it.

Test the exact production flow. Unit tests that call vm.save_entity() directly may pass while the actual server flow fails, because the server flow involves state injection, bytecode execution, and validation steps that the unit test skips. Integration tests must reproduce the complete request lifecycle.

The test suite grew from 2,870 to 2,875 tests after this session. More importantly, it gained coverage of the precise code path that production users would exercise.

This is Part 151 of the "How We Built FLIN" series, documenting how a CEO in Abidjan and an AI CTO designed and built a programming language from scratch.

Series Navigation: - [150] Function Audit Day 7 Complete - [151] Database Persistence Audit (you are here) - [152] 3,452 Tests, Zero Failures