Data mapping and validation

This guide looks at the role schemas play when validating and casting data through changesets. As we will see, sometimes the best answer is not to bend a single schema to fit every job, but to define several schemas — one per concern. A schema that maps your database table, another that maps a form, another for reads. Because a lode Schema(row) is an ordinary value rather than compile-time magic, spinning up an extra schema costs one function declaration — and nothing forces a schema to ever touch the database.

Schemas are mappers

The lode/schema module documentation describes a schema as mapping any data source into a typed Gleam record. The word any is the important one: schemas are most often used to map database tables, but they work just as well for mapping API payloads, form parameters, or any other external data — the schema is just metadata plus a load/dump codec pair.

Let’s make this concrete with a Sign Up feature. The form asks for a first name, a last name, and an email. On the database side, suppose we persist into two tables: accounts (holding the email) and profiles (holding a single name column).

A first instinct is to graft the form’s shape onto the persisted schema with virtual fields — fields that are cast and validated like any other but never written to the database:

import lode/schema
import lode/types/primitive
import lode/value.{VInt, VString}
import gleam/dict
import gleam/result

pub type Profile {
  Profile(id: Int, name: String, first_name: String, last_name: String)
}

fn profile_schema() -> schema.Schema(Profile) {
  schema.new(
    source: "profiles",
    fields: [
      schema.primary_key(name: "id", of: primitive.id()),
      schema.field(name: "name", of: primitive.string()),
      // Virtual: cast in changesets, never stored.
      schema.virtual_field(name: "first_name", of: primitive.string()),
      schema.virtual_field(name: "last_name", of: primitive.string()),
    ],
    load: fn(row) {
      use id <- result.try(schema.load_field(row, "id", primitive.integer()))
      use name <- result.try(schema.load_field(row, "name", primitive.string()))
      let first_name =
        schema.load_virtual(row, "first_name", primitive.string(), "")
      let last_name =
        schema.load_virtual(row, "last_name", primitive.string(), "")
      Ok(Profile(id:, name:, first_name:, last_name:))
    },
    dump: fn(p: Profile) {
      dict.from_list([#("id", VInt(p.id)), #("name", VString(p.name))])
    },
  )
}

This works — first_name and last_name cast and validate like stored fields, and the repo silently drops them from every write payload. But it muddies the schema: Profile now carries fields that exist only for one form, its record has fields that are blank everywhere except during sign-up, and every future form with a slightly different shape tempts you to add more virtual fields. The schema stops describing the data and starts describing every UI that ever touched it.

The better split is two mappings: one schema shaped like the form, and plain functions that convert the validated form data into the persisted shapes.

In Elixir’s Ecto the form-shaped schema would be declared with embedded_schema, a special form for schemas without a data source. lode needs no separate construct: a Schema value is inert metadata — it only touches the database when you hand it to a repo function — so a schema for a record you never persist is built exactly like any other. (The lode/embed module is a different thing: schemas persisted inside a jsonb column — see Embedded Schemas. The design rationale for explicit schema values is in DESIGN.md in the repository.)

import lode/schema
import lode/types/primitive
import lode/value.{VString}
import gleam/dict
import gleam/result

pub type Registration {
  Registration(first_name: String, last_name: String, email: String)
}

fn registration_schema() -> schema.Schema(Registration) {
  schema.new(
    // Never queried or written — the source is just a label here.
    source: "registrations",
    fields: [
      schema.field(name: "first_name", of: primitive.string()),
      schema.field(name: "last_name", of: primitive.string()),
      schema.field(name: "email", of: primitive.string()),
    ],
    load: fn(row) {
      use first_name <- result.try(schema.load_field(
        row,
        "first_name",
        primitive.string(),
      ))
      use last_name <- result.try(schema.load_field(
        row,
        "last_name",
        primitive.string(),
      ))
      use email <- result.try(schema.load_field(row, "email", primitive.string()))
      Ok(Registration(first_name:, last_name:, email:))
    },
    dump: fn(r: Registration) {
      dict.from_list([
        #("first_name", VString(r.first_name)),
        #("last_name", VString(r.last_name)),
        #("email", VString(r.email)),
      ])
    },
  )
}

Now cast form parameters against the registration schema and validate them, exactly as you would for a persisted one. Parameters arrive as a Dict(String, Value) — in a web app, decoded from the request body:

import lode/changeset
import lode/value.{type Value, VString}
import gleam/dict.{type Dict}
import gleam/option.{Some}

// e.g. decoded from the request body:
// dict.from_list([
//   #("first_name", VString("Ada")),
//   #("last_name", VString("Lovelace")),
//   #("email", VString("ada@example.com")),
// ])

pub fn registration_changeset(
  params: Dict(String, Value),
) -> changeset.Changeset(Registration) {
  let permitted = field.fields(["first_name", "last_name", "email"])
  changeset.cast(
    data: Registration(first_name: "", last_name: "", email: ""),
    schema: registration_schema(),
    params: params,
    permitted: permitted,
  )
  |> changeset.validate_required(permitted)
  |> changeset.validate_length(field.field("email"), min: Some(3), max: Some(160))
}

Columns are typed Field(row) values, not bare strings. field.field("email") names one column and field.fields(["a", "b"]) a list; the row is inferred from the schema:/changeset alongside, so a column of another schema is a compile error. To also catch typos, use the codegen-emitted column record — registration_fields.email — or reuse a query accessor with expr.to_field(registration_email()). (Where Ecto’s validate_length takes keyword options, here min and max are explicit Option(Int) arguments — pass None for an unbounded side.)

If the changeset is valid, we turn it into a Registration record, convert that into the persisted shapes, and write both rows in one transaction. In Ecto this is an if changeset.valid? branch; lode folds the branch into changeset.apply_action, which returns Ok(record) for a valid changeset and Error(ChangesetInvalid(errors)) otherwise — so the whole flow composes with result.try:

import lode/changeset
import lode/error.{type LodeError}
import lode/repo
import lode/value.{type Value}
import gleam/dict.{type Dict}
import gleam/result

pub fn sign_up(
  r: repo.Repo,
  params: Dict(String, Value),
) -> Result(Registration, LodeError) {
  let cs = registration_changeset(params)

  // Valid -> the typed Registration; invalid -> ChangesetInvalid(errors).
  use registration <- result.try(changeset.apply_action(cs, changeset.Insert))

  use _ <- result.try(
    repo.transaction(r, fn(tx) {
      use _ <- result.try(repo.insert_all(
        repo: tx,
        schema: account_schema(),
        rows: [to_account(registration)],
      ))
      repo.insert_all(
        repo: tx,
        schema: profile_schema(),
        rows: [to_profile(registration)],
      )
    }),
  )

  Ok(registration)
}

Two notes on the translation. First, Ecto tags the failed changeset with a custom action (like :registration) before handing it back to the form layer; lode’s Action is a closed set (Insert/Update/Delete/Replace/ Ignore), and apply_action returns the errors, not the changeset — if your form layer wants the changeset itself for re-rendering, check cs.valid and keep cs before calling apply_action. Second, Ecto’s insert_all can write bare maps into a table name with no schema; lode’s repo.insert_all(repo:, schema:, rows:) always goes through a schema and typed rows — which is exactly what we want here. (See Schemaless queries for the raw-SQL escape hatch.)

The conversion functions are ordinary record-to-record code. With the registration schema carrying the form’s shape, Account and Profile shrink back to mapping just their tables:

pub type Account {
  Account(id: Int, email: String)
}

pub type Profile {
  Profile(id: Int, name: String)
}

pub fn to_account(r: Registration) -> Account {
  // id: 0 is a placeholder — autogenerated columns are omitted from
  // the insert payload, so the database assigns the real id.
  Account(id: 0, email: r.email)
}

pub fn to_profile(r: Registration) -> Profile {
  Profile(id: 0, name: r.first_name <> " " <> r.last_name)
}

account_schema() and profile_schema() are declared the usual way (see Getting Started); each now maps exactly one table and nothing else.

The Registration schema cleanly separates what the user sees from what the database stores: validations that belong to the form (the name fields, length limits, validate_confirmation/validate_acceptance over virtual fields) live on the registration changeset, while constraints that belong to the data (a unique email) live on the account changeset. Each schema has one reason to change.

In Ecto there is a second option for one-off validations — changesets without any schema at all, driven by a bare data-and-types tuple:

data  = %{}
types = %{name: :string, email: :string}

changeset =
  {data, types}
  |> Ecto.Changeset.cast(params["sign_up"], Map.keys(types))
  |> validate_required(...)
  |> validate_length(...)

Schemaless changesets. Ecto builds a changeset from a {data, types} map pair, skipping schema definition — for search forms or ad-hoc API params. schema.schemaless([schema.field("q", primitive.string()), ...]) is the equivalent: it builds a Schema(Dict(String, Value)) whose load/dump are the identity, so you cast(dict.new(), types, params, keys), run the validators, and apply_changes returns a Dict(String, Value) — no record type and no codec to write. TASK:schemaless-changesets

So which to reach for? In Ecto the trade-off is schemas versus schemaless changesets; in lode every changeset has a schema, and the question becomes how many schemas to define and what each one maps. The underlying lesson is the same either way: don’t solve the big problem — “validate this form and persist two tables” — in one place. Break it apart. Map the form with one schema, validate it there, then convert the result into the persisted shapes with plain functions. Each piece stays small, typed, and testable on its own, and none of them needs a database to run.