lode - v0.1.0

Schemaless queries

Most queries in lode are written using schemas. For example, to retrieve all posts in a database, one may write:

repo.all(repo: r, schema: post_schema(), query: query.from("posts", "p"))

Where Ecto’s macro rewrites Repo.all(Post) into a query that selects every schema field into a %Post{} struct, lode does the same job with the schema’s load codec: the adapter returns each row as a name-keyed Dict(String, Value), and schema.load turns that dict into your record (see DESIGN.md in the repository). In fact, lode queries are already schemaless by construction — query.from("posts", "p") names a table, not a schema, and the schema only enters at the edges, when rows are loaded or values dumped.

Although you might use schemas for most of your queries, lode also lets you query without one. The schemaless read path is repo.query_raw, which plays the role of both Ecto’s Repo.query and its schemaless from "posts", select: [:title] queries: you write parameterized SQL with positional $1, $2, … placeholders, and rows come back as name-keyed Dict(String, Value) maps:

import lode/repo
import lode/value.{VInt, VString}
import gleam/dict

let assert Ok(rows) =
  repo.query_raw(
    repo: r,
    sql: "SELECT title, body FROM posts WHERE visits >= $1",
    params: [VInt(100)],
  )

let assert [first, ..] = rows
let assert Ok(VString(title)) = dict.get(first, "title")

Because rows are keyed by column name rather than position, reads are order-independent and projection-tolerant: select columns in any order, or only a subset, and load each field by name. The typed way to do that is schema.load_field, which runs the column through an LodeType — the same machinery schema fields use — and treats an absent column as VNull, so a projection that omits a nullable column still loads:

import lode/schema
import lode/types/primitive.{Absent, Present}

// `age` was not selected, so for a nullable read it loads as Absent
// rather than erroring.
let assert Ok(title) =
  schema.load_field(row: first, name: "title", of: primitive.string())
let assert Ok(Absent) =
  schema.load_field(
    row: first,
    name: "age",
    of: primitive.nullable(primitive.integer()),
  )

For required columns there are also the blunter schema.require_int and schema.require_string, and schema.get(row, name) fetches the raw Value if you want to pattern match yourself. This idiom — query_raw, then load_field/require_* per column — is the library’s own; see lode_sql/test/query_raw_test.gleam for it running against a live Postgres.

When Ecto sees from "posts", select: [:title, :body], it converts the field list into a map or struct for you. lode’s equivalent is repo.query_raw_as, which runs raw SQL and loads each row into a typed struct via the schema’s load codec, in one call — as long as your SELECT covers every column the codec requires:

let assert Ok(posts) =
  repo.query_raw_as(
    repo: r,
    schema: post_schema(),
    sql: "SELECT id, title, body FROM posts",
    params: [],
  )

(The load codec is a plain function carried on the Schema value, so you can also apply it per row yourself — list.try_map(rows, post_schema().load) — when working from a query_raw result directly.)

Two behavioral notes versus Ecto’s Repo.query: query_raw returns only the rows (there is no num_rows/columns result struct — a statement with no result set returns Ok([])), and it needs a SQL-speaking adapter: the in-memory adapter returns an error (see Testing with Lode).

Schemaless writes go through the query builder. Ecto changes a post’s title without a schema using update: [set: [title: ^new_title]]; the lode equivalent is repo.update_all with a set list of column/Value pairs. The where condition needs a column reference — either build an ad-hoc typed FieldRef (binding 0 is the from source) and use the typed combinators, or assemble expr.Col/expr.Lit directly:

import lode/query
import lode/query/expr.{FieldRef}
import lode/types/primitive
import lode/value.{VString}

pub fn update_title(
  r: Repo,
  post_id: Int,
  new_title: String,
) -> Result(Int, LodeError) {
  let id_column = FieldRef(binding: 0, name: "id", lode_type: primitive.id())
  let q =
    query.from("posts", "p")
    |> query.where(expr.eq(field: id_column, to: post_id))

  repo.update_all(repo: r, query: q, set: [#("title", VString(new_title))])
}

The ad-hoc FieldRef carries an LodeType, so expr.eq(field: id_column, to: post_id) is still type-checked — you get the casting guarantees a schema field would give, without declaring a schema.

Ecto’s update construct supports four commands: set, inc (atomic increment), push and pull (array append/remove). repo.update_all_set covers all of them via query.Assignments — and returns an affected count:

pub fn increment_page_views(r: Repo, post_id: Int) -> Result(Int, LodeError) {
  repo.update_all_set(
    repo: r,
    query: query.from("posts", "p")
      |> query.where(expr.BinOp(expr.Eq, expr.Col(0, "id"), expr.Lit(VInt(post_id)))),
    set: [query.inc("page_views", by: VInt(1))],
  )
}

Set a column from an expression with query.set_expr. One caveat: Postgres rejects two assignments to the same column in one UPDATE, so push and pull the same column in separate calls. TASK:update-all-inc-push-pull

Let’s look at another example. Imagine you are writing a reporting view. It may be counter-productive to think about how your existing application schemas relate to the report being generated; it is often simpler to write a query that returns only the data you need. In Ecto this is a schemaless from u in "users", join: ..., select: %{...} query; the lode query model does have joins, group-by and aggregates, but repo.all always pairs a query with a schema for loading — so a report that fits no schema goes through query_raw:

import lode/error.{type LodeError}
import lode/repo.{type Repo}
import lode/schema
import lode/value.{VTimestamp}
import gleam/list
import gleam/result
import gleam/time/timestamp.{type Timestamp}

pub type Report {
  Report(user_id: Int, count: Int)
}

pub fn running_activities(
  r: Repo,
  start_at: Timestamp,
  end_at: Timestamp,
) -> Result(List(Report), LodeError) {
  use rows <- result.try(repo.query_raw(
    repo: r,
    sql: "
      SELECT a.user_id, count(u.id) AS count
        FROM users AS u
        JOIN activities AS a ON a.user_id = u.id
       WHERE a.start_at > $1 AND a.end_at < $2
       GROUP BY a.user_id",
    params: [VTimestamp(start_at), VTimestamp(end_at)],
  ))
  list.try_map(rows, fn(row) {
    use user_id <- result.try(schema.require_int(row, "user_id"))
    use count <- result.try(schema.require_int(row, "count"))
    Ok(Report(user_id:, count:))
  })
}

The function above does not rely on schemas and returns only the data that matters for the report. Notice the role Ecto’s type/2 plays here: in Ecto you write type(^start_at, :naive_datetime) to tell the planner how to cast an interpolated value. In lode the parameters are already runtime-tagged Values, so the constructor is the type annotation — VTimestamp encodes as timestamptz, VNaiveDatetime(date:, time:) as a zone-less timestamp, and so on for every variant in lode/value.

By keeping the query model schemaless at its core and pushing schemas to the edges, lode makes queries with and without schemas equally accessible — and because queries and expressions are plain values rather than macros, it also makes dynamic queries (where fields, filters and ordering are not known upfront) ordinary code. See Dynamic queries.

insert_all, update_all and delete_all

Ecto allows all database operations to be expressed without a schema, starting with Repo.insert_all, which accepts a bare table name and a list of field/value lists:

MyApp.Repo.insert_all(
  "posts",
  [
    [title: "hello", body: "world"],
    [title: "another", body: "post"]
  ]
)

Schemaless insert_all. repo.insert_all_raw bulk-inserts value lists into a bare table — no schema:

let assert Ok(2) =
  repo.insert_all_raw(
    repo: r,
    source: "posts",
    columns: ["title", "body"],
    rows: [
      [VString("hello"), VString("world")],
      [VString("another"), VString("post")],
    ],
  )

It returns the inserted count; insert_all_raw_returning returns the inserted rows. The schema-based repo.insert_all stays for typed rows. TASK:schemaless-insert-all

Updates and deletes, by contrast, are fully schemaless: repo.update_all and repo.delete_all take a query (no schema anywhere) and return the affected count — no rows need loading, so no codec is needed:

// Use the ID to select which posts to act on
let id_column = FieldRef(binding: 0, name: "id", lode_type: primitive.id())
let post =
  query.from("posts", "p")
  |> query.where(expr.eq(field: id_column, to: id))

// Update the title of all matching posts
let assert Ok(1) =
  repo.update_all(repo: r, query: post, set: [#("title", VString("new title"))])

// Delete all matching posts
let assert Ok(1) = repo.delete_all(repo: r, query: post)

The same is true of the aggregate reads: repo.count(repo:, query:) and repo.exists(repo:, query:) take only a query, since a count or a boolean needs no row loading either.

It is not hard to see how these operations map directly to their SQL variants, keeping the database at your fingertips without the need to intermediate every operation through schemas — and when even the query model is in the way, query_raw is the SQL itself. For the schema-ful side of this spectrum, see Data mapping and validation.

✨ Search Document