chitai/backend/AGENTS.md

# Chitai backend

Litestar REST API for the eBook library. See the repo-root `AGENTS.md` for the overall picture and
dev-environment setup.

**Stack:** Python 3.13 · Litestar 2 · advanced-alchemy over async SQLAlchemy 2 · asyncpg ·
PostgreSQL 17 · Alembic (through advanced-alchemy's `alchemy` CLI) · pydantic-settings · uv.

## Layering

```
controllers/      HTTP surface only — parse, delegate, serialise. Keep thin.
services/         Business logic. One SQLAlchemyAsyncRepositoryService subclass per aggregate.
database/models/  SQLAlchemy models.
schemas/          Pydantic DTOs for request bodies and responses.
```

Controllers call **service** methods, not the repository. `app.py` assembles the app: route
handlers, JWT auth, exception handlers, the SQLAlchemy plugin, and two lifespan context managers.

## advanced-alchemy idioms

These are the conventions that are easy to get wrong if you write plain SQLAlchemy here:

- Models extend `BigIntAuditBase` (adds id/created_at/updated_at); pure link and child rows extend
  `BigIntBase` (e.g. `Identifier`, `FileMetadata`, `BookAuthorLink`).
- A service declares an inner repository and points at it:

  ```python
  class BookService(SQLAlchemyAsyncRepositoryService[Book]):
      class Repo(SQLAlchemyAsyncRepository[Book]):
          model_type = Book
      repository_type = Repo
  ```

- Transform incoming data with the **`to_model_on_create` / `to_model_on_update` hooks**, not by
  overriding `create`/`update` wholesale — see `services/book.py:407` onward.
- Serialise responses through `service.to_schema(obj, schema_type=s.SomeRead)`; for lists,
  `to_schema(items, total, filters, schema_type=…)` produces the `OffsetPagination` envelope.
- `Author`, `Tag`, `Publisher` and `BookSeries` are deduplicated with **`as_unique_async`**. Never
  construct them directly when attaching to a book — use
  `await Author.as_unique_async(session, name=name)` as
  `BookService._populate_with_unique_relationships` does, or you will create duplicate rows.
- Domain methods on services are named for the domain (`create_book`, `update_book`, `add_files`),
  deliberately distinct from the inherited CRUD names.

## Dependency injection

`services/dependencies.py` is the hub; controllers wire providers in their `dependencies` dict.

- Most providers come from `create_service_provider(SomeService, …)` — one line each.
- `provide_book_service` is hand-written because it must inject eager loads *and* scope
  user-specific rows: `selectinload` for authors/tags/files/etc., plus `with_loader_criteria` so
  `BookProgress` and `BookListLink` only load rows belonging to `current_user`. If you add a
  relationship that the API returns, add it to that `load` list.
- `create_book_filter_dependencies` intentionally **overrides** advanced-alchemy's stock providers:
  the search filter becomes a trigram search, and order-by gains a `random` sort order. Do not
  replace it with the stock `create_filter_dependencies`.
- `get_library_by_id` resolves the target library from either a `library_id` query param or the
  book's own `library_id`, and raises 404 for either miss.

## Filters

`services/filters/` holds `StatementFilter` dataclasses that compose into any `list` /
`list_and_count` call: `TagFilter`, `AuthorFilter`, `BookshelfFilter`, `ProgressFilter`,
`TrigramSearchFilter`, `CustomOrderBy`, `FileHashFilter`, plus the `*LibraryFilter` variants used by
OPDS.

To add list-filtering behaviour: write the dataclass here, add a `provide_*_filter` function in
`dependencies.py`, register it in the controller's `dependencies`, and fold it into
`provide_book_filters` — the controller handler itself does not change.

## Authentication — three schemes

| Scheme | Where | Used by |
| --- | --- | --- |
| JWT bearer (`OAuth2PasswordBearerAuth`) | `app.py` | The web frontend and the main API. Public paths are listed in its `exclude=[…]`. |
| HTTP Basic (`middleware/basic_auth.py`) | `OpdsController` | E-reader / OPDS clients, which only speak Basic. |
| `X-AUTH-USER` API key (`middleware/kosync_auth.py`) | `KosyncController` | KOReader devices; the key maps to a `KosyncDevice` row, which maps to a user. |

Each middleware resolves a `User` onto the connection; the matching
`provide_user_via_basic_auth` / `provide_user_via_kosync_auth` dependencies expose it to handlers.

## Filesystem behaviour

The backend owns files on disk, not just rows:

- **Layout** — `services/filesystem_library.py` (`BookPathGenerator`) renders a Jinja2 template
  against book metadata to decide where a book lives under the library's `root_path`
  (default: `author/series/position - title/`).
- **One directory per book, never shared.** The generated path is a pure function of the metadata,
  so two books with the same author and title produce the same one — two editions, or an
  `allow_duplicates` copy. `BookService._reserve_book_path` moves the later one to `title (2)`
  before anything is written, and `update_book` reserves the same way so a rename cannot move a
  book in on top of another. This matters because `book.path` is what deletes, moves and file
  lookups act on: books sharing a directory means one overwrites the other's files, and deleting
  either takes both. `_unused_path` does the same job for filenames within a directory.
  A book that already has a `path` keeps it — `add_files` must follow the book, not the template.
- **Metadata extraction** — `services/metadata_extractor.py` reads EPUB (ebooklib) and PDF
  (pypdfium2) files; extracted values fill only *empty* fields on the incoming payload.
- **Covers** — converted to WebP with a UUID filename under `settings.book_cover_path`, served by a
  static-files router mounted at `/covers`.
- **Consume directory** — `services/consume.py` (`ConsumeDirectoryWatcher`) watches
  `settings.consume_path` with `watchfiles`, creates one subdirectory per library slug, batches
  additions (3 s debounce) and imports them via `BookService.create_many_from_existing_files`.
  Started as an asyncio task from the `setup_directory_watcher` lifespan hook.
- **Updates move files.** `BookService.update_book` regenerates the path from the new metadata and,
  if it differs, moves the directory contents and prunes empty parents. Keep that in mind before
  changing metadata handling.

### Content types come from the extension, and null means null

Every ingest path names a file's format with `guess_content_type` (`services/utils.py`), never with
`mimetypes.guess_type` directly and never with what the client said. Python's built-in map answers
`None` for `.mobi`, `.azw`, `.prc`, `.fb2`, `.fbz`, `.lit`, `.lrf` and `.cb7` — most of what a
library imported from elsewhere carries — so `EBOOK_CONTENT_TYPES` fills those in. A browser's
`application/octet-stream` is discarded rather than used as a fallback: it is the client saying it
does not know, and storing it is indistinguishable from having determined a format.

When nothing can name the extension the column **stays null**. That is the honest answer, and only
one consumer cannot take it: OPDS `Link.type` is a required string, so
`services/opds/opds.py` substitutes `application/octet-stream` at that boundary. Litestar's
`ASGIFileResponse` already does its own fallback, so `get_file` can pass a null straight through.
`FileMetadataRead.content_type` is nullable for the same reason — it was once required, which
turned a stored null into a 500 on a book that was otherwise fine.

## Duplicate detection

Every ingest path screens incoming files against what is already stored, keyed on
**`(hash, size)`** — never the hash alone, because it samples 12 KiB (see below) and
EPUBs from one toolchain often share their first window. `FileMetadata.hash` carries a
plain, deliberately **non-unique** index: a collision must not be able to fail an import,
and older databases may already hold duplicates.

Scope comes from `CHITAI_DUPLICATE_SCOPE` (`library`, the default | `global` | `off`).

The policy differs by how deliberate the import is:

| Path | Behaviour |
| --- | --- |
| `create_many_from_files` (browser bulk) | Skip per file, skip a whole group whose files are all known, report everything skipped in `ImportResult.duplicates`. Re-dropping a folder to pick up what is new is the case this serves. |
| `create_book` (single, with metadata) | All-or-nothing: raises `DuplicateFilesError`, which `controllers/book.py` renders as a **409** carrying the refused files in `extra`. |
| `add_files` | A file the book already carries is a no-op; one stored under another book raises `DuplicateFilesError`. |
| `create_many_from_existing_files` (consume watcher) | Skips, and **moves the file to `CHITAI_DUPLICATE_PATH/<library slug>/`** — nothing is deleted, and it cannot stay put because `watchfiles` only reports additions. That path must stay outside `consume_path` or the watcher re-imports it and tries to read the directory name as a library slug. |
| `create_many_from_calibre` (Calibre import) | Skips per file, and skips a whole book whose files are all known. Nothing is moved — the source is somebody else's library. This is what makes a re-run a no-op and an interrupted import resumable by running it again. |

`allow_duplicates=true` overrides all of it, on every endpoint. Keep that working — the
hash is not proof of identity, so a wrong verdict has to be recoverable, and a scripted
import needs a way through. The **web UI deliberately does not offer it**: storing the
same bytes twice splits reading progress and shelf membership across two records that
can never converge, which is nothing anyone wants on purpose.

Three things to preserve when touching this code:

- **A match only counts while the file is on disk.** `find_duplicate_files` stats each
  candidate, and `add_files` writes a missing file back into the row that already
  describes it (`_restore_file`) instead of adding a second row beside it. The hash
  lives in the database and the file does not, so without this a file deleted behind
  the app's back would go on refusing its own replacement.

- **Screening runs before anything is written.** `fingerprint_upload` reads the spooled
  upload and rewinds it; the resulting fingerprints are handed to `_save_book_files`,
  which skips its own `StreamingHasher` when it already has the answer. Passing them
  through is what keeps the file from being read twice.
- **`_screen_for_duplicates` extends the `known` dict as it goes**, so the same bytes
  submitted twice in one request are caught. Those duplicates report `book_id: None` —
  there is no row to point at yet.

`POST /books/duplicate-files` answers the same question from fingerprints alone, for
clients that want to ask before uploading anything.

### Book-level detection — a different question

The file check answers "are these the same bytes?". `find_duplicate_books` answers "is
this the same book?", which a re-scan, a re-zipped EPUB or another edition cannot be
asked with a hash. Two signals, either sufficient: a **shared identifier**, or a
**matching normalized title with at least one shared author**.

It **never blocks**. A metadata match is a guess — a work shares title and author with
its own translation, its own second edition and its own audiobook — so the book is
created and the candidates are reported alongside it in
`ImportResult.possible_duplicates`. File-level dedupe keeps its refuse/skip behaviour;
that one is near-certain and this one is not. Do not "improve" this into a refusal.

Two rules narrow it, both applied in Python over the small candidate set:

- **A shared author is required for a title match.** Without it every book the
  extractors gave up on and titled `Unknown` is a duplicate of every other one. A book
  with no authors can therefore only match on an identifier.
- **The same series at a different `series_position` disqualifies a match.** A trilogy
  shares an author and often most of its title; the position is the library saying
  outright that these are two books.

A book is compared under **several title keys, not one** (`_title_keys`). Ebook files
are overwhelmingly named `Title - Author.epub`, and wherever nothing inside the file
overrode that name the author ended up in the title column — so one copy is stored as
`Building Microservices` and another as `Building Microservices Sam Newman`. Both
directions are generated, the author stripped off and the author added on, which is why
the query is `normalized_title.in_(keys)` rather than `==`. This is still exact matching
on an indexed column: no similarity score, nothing to tune. It does not weaken the
shared-author requirement, which is a separate condition.

`find_duplicate_book_groups` is the library-wide pass behind
`GET /books/duplicate-books`, since the import-time check says nothing about a
collection someone already has. It buckets books by every key they carry and merges the
buckets with union-find, so A~B by ISBN and B~C by title land in one group. Pairs in
`duplicate_dismissals` are never merged — a reader disagreeing with one pairing must not
silently break a group that stands on other evidence.

### Author names have one stored form

`Author.name` is always the canonical form, produced by `format_author_name`. Extractors
hand over whatever the file said — `Newman, Sam;` from a `DC:creator` list, `Sam Newman`
from a PDF, `Sam Newman.epub` from a filename — and storing those verbatim is how one
person becomes four rows in the sidebar, four entries in the author filter, and four
books that never look like each other.

This is **display** canonicalization, distinct from `normalize_author`, which throws
away case, accents and spacing to build a comparison key nobody sees. Tidying only
removes what an extractor added: a trailing separator, a file extension, and the
`Surname, Given` ordering. It never touches case or accents — `Michał Płachta` and
`Steve McConnell` are the author's own spelling, not something to correct.

Three places have to agree, and `Author` keeps them together: the `@validates("name")`
hook, `unique_hash`, and `unique_filter`. `as_unique_async` looks a row up with the
filter and then constructs with the validator, so if the lookup used the raw name and
the insert used the tidy one, every variant spelling would miss the existing row and
then collide with it on the unique index.

A form the rule does not recognise is **left exactly as it was found** — `Dave Thomas,
Andy Hunt` is two people in one string, and flipping it would invent a third. Leaving a
mess visible beats rewriting it wrongly.

### The normalized columns are written by validators

`Book.normalized_title`, `Author.normalized_name` and `Identifier.normalized_value` are
derived from `services/matching.py` and kept current by **SQLAlchemy `@validates` hooks
on the models**, not by any service. Assigning them directly is always wrong.

This is deliberate and it is invisible at the call sites: `BookService` writes titles
through at least three paths (`to_model_on_create`, `to_model_on_update`, and the
`setattr` loop in `_populate_with_unique_relationships`), and a validator is the only
thing a fourth cannot bypass. The columns carry plain, **non-unique** btree indexes —
two spellings collapsing onto one value is the entire point.

`services/matching.py` is imported *inside* those validators rather than at module
scope: reaching it initialises the `chitai.services` package, which imports the
services, which import the models. Keep the local import.

The validators only fire on write, so a migration that adds one of these columns must
backfill existing rows through the same helpers — see the `data_upgrades()` hook in
`2026-08-15_add_book_matching_keys_and_duplicate__4358e7d4743a.py`.

**Changing anything in `services/matching.py` needs a revision that recomputes them.**
The keys are derived and already written, so a normalization change silently invalidates
every stored row: a book written under the old rules just stops matching one written
under the new rules, with nothing to show that anything is wrong. Copy
`2026-08-15_recompute_book_matching_keys_ed41acf21270.py`, which exists because
`normalize_title` learned to strip compact edition markers (`2E`, `5e`). It is
idempotent and safe to re-run.

## Importing from Calibre

Two pieces, deliberately separated:

- **`services/calibre.py`** reads `metadata.db` and the tree beside it. It knows nothing about
  `Book`, `BookService` or a session, so it is testable without Postgres, and it reports what
  Calibre wrote rather than what Chitai wants — identifiers come back keyed by `identifiers.type`
  verbatim. It also unpacks a zipped library (`extract_calibre_archive`).
- **`BookService.create_many_from_calibre`** does the ingest, next to the two other ingest paths
  because it needs the same privates they do (`_reserve_book_path`, `_save_cover_image`,
  `_screen_for_duplicates`). The CLI in `cli.py` is a thin wrapper over it.
- **`services/calibre_import.py`** is lifecycle only — the job registry behind the endpoints:
  state, progress, cancellation, and a session of its own.

`docs/calibre-import.md` is the full brief, including the phases not built yet. What matters here:

- **Files are copied, never moved.** `metadata.db` would go on pointing at files that are gone,
  which quietly ruins a library somebody still uses. `copy_file` streams rather than using
  `shutil.copy`, which would block the loop for a 40 MB read.
- **The extractors are not run.** This is the one ingest path that trusts its input: Calibre's
  catalogue is curated and its filenames are truncated to ~42 characters, so `data.name` locates a
  file and the database carries the metadata. The title is stored verbatim for the same reason —
  no edition split out, no subtitle guessed.
- **The catalogue is copied before it is read**, and the copy is opened read-write. Calibre may be
  running; opening the live file either sees a torn state or needs to recover a write-ahead log,
  which read-only access cannot do. `close()` removes the copy in a `finally`, or a failure leaves
  a catalogue-sized file in the temp directory.
- **`check_same_thread=False` plus an `asyncio.Lock`.** Every query runs through
  `asyncio.to_thread`, which hands out whichever worker is free, so the connection outlives the
  thread that opened it. The lock is what makes that safe. Removing either one reintroduces
  `SQLite objects created in a thread can only be used in that same thread`, intermittently —
  the pool often reuses one thread, so it passes until it does not.
- **One book never costs the run.** A failure is recorded in `CalibreImportResult.failed`, the
  session is rolled back so the next book can use it, and the files that book had already copied
  are deleted — an orphaned directory would make the next attempt reserve `title (2)` and look as
  though it had worked. A cover that PIL cannot open costs the cover, not the book.
- **`calibre-uuid`, not `uuid`.** `books.uuid` is stable for the life of the row, so it is the
  durable link back to the source and worth matching on. `uuid` is the name `normalize_identifier`
  refuses, because an EPUB regenerates one per build.

### The API takes an uploaded archive; the CLI takes a path

**`POST /libraries/{id}/imports/calibre/upload`** is the only way in over HTTP. It takes a zipped
Calibre library, answers **202** with a job handle, and unpacks into a temp directory the job owns.
`GET /libraries/imports/{job_id}` is polled; `DELETE` on the same path stops it.

There is deliberately **no endpoint that imports from a server path**. A desktop Calibre install is
not on the server, and importing from a path the server can already see is a server-side operation
— which is what `litestar --app-dir src/chitai/ calibre-import <path> --library <slug>` is for,
including its `--dry-run`. Do not add the path endpoint back without being asked: it was built,
then removed on purpose.

- **The registry is in memory, so it assumes one worker process.** That holds today (`litestar run`
  is single-process, and the consume watcher is already an in-process singleton), but the day
  `TODO.md`'s "production image runs the development server" item is fixed with a worker count, a
  poll can land on a worker that never heard of the job. `services/calibre_import.py` says so at
  the top; an `import_jobs` table is the answer when that happens.
- **The job opens its own session.** The request that started it is long gone and its session
  closed with it.
- **Cancelling is not aborting.** A flag is read between books, never during one, so a cancelled
  import leaves whole books behind and never half of one. `task.cancel()` would abandon a book
  mid-copy and leave files with no row describing them.
- **The job deletes its workspace** — the unpacked archive is a second copy of the whole library,
  and the books worth keeping have been copied into the library proper by the time it ends. Removed
  even when the run failed, since nothing will come back for it.
- **Extraction refuses** an entry pointing outside the archive (zip slip), an archive that will not
  fit on disk, and one with no `metadata.db` within three levels. All three answer 400 before a job
  exists, rather than as a job that reports FAILED a moment later.
- **The upload is streamed both sides.** The archive reaches disk in chunks rather than being read
  whole, and the SvelteKit proxy passes `request.body` through instead of buffering it — see
  `frontend/AGENTS.md`.

Things Calibre does that will produce wrong data if you forget them are documented at the top of
`services/calibre.py` — the `0101-01-01` date sentinel, `|` for a comma in an author name, the
REAL `series_index` that defaults to 1.0 for every book, HTML in `comments`, the views that need
SQLite functions Calibre registers from Python, and `books_pages_link` being both recent and
usually empty. `tests/calibre_fixtures.py` builds a library exercising all of them.

## KOReader hashing

`services/utils.py` reimplements KOReader's partial-MD5 document identifier: 1 KiB samples at
offsets produced by LuaJIT's 32-bit `bit.lshift`, including its shift-masking wrap-around
(`shift & 0x1F`, so `i=-1` yields offset 0). `_lshift32` looks wrong and is not — the overflow is
what makes hashes match real devices. Don't "simplify" it; `tests/integration/test_file_hash.py`
guards the behaviour.

## Database

- **`pg_trgm` is required.** `Book.__table_args__` declares a GIN trigram index on `title`, which
  `TrigramSearchFilter` uses for fuzzy title search. The extension is enabled by a migration and, in
  tests, by `conftest.py`.
- Migrations live in `migrations/versions/`. Generate with
  `alchemy --config chitai.database.config.config make-migrations` (add `--no-autogenerate` for a
  blank revision), apply with `… upgrade`. `database/config.py` sets `create_all=False`, so nothing
  is auto-created at runtime; production applies migrations from `entrypoint.sh`.
- Sessions use `expire_on_commit=False` and Litestar's `before_send_handler="autocommit"`, so a
  handler that returns 2xx commits automatically.

## Testing

`pytest tests/` — `asyncio_mode = "auto"`, so async tests need no marker.

- `tests/unit/` exercises services directly against a real session; `tests/integration/` drives the
  whole app through `AsyncTestClient(app=create_app())`.
- The database is a throwaway container from `pytest-databases`, so **Docker must be running**.
- `tests/conftest.py` provides the shared fixtures: `client`, `authenticated_client`,
  `other_authenticated_client` (a second user, for access-control tests), one fixture per service,
  `test_user` / `test_library`, and an autouse fixture that redirects cover storage into `tmp_path`.
- `tests/integration/conftest.py` monkeypatches the module-level alchemy `config` onto the test
  engine/sessionmaker and drops+recreates+reseeds the schema for every test.
- Real EPUB and PDF fixtures live in `tests/data_files/`.

## Known rough edges

Observed in the current tree — don't mistake these for intentional patterns to copy:

- `controllers/book.py` — file-level TODO: `book_id` is a path parameter on some endpoints and a
  query parameter on others. `set_book_progress_batch` does a documented N+1 (one select + one
  upsert per book).
- `services/filesystem_library.py` — TODO to replace Jinja2 templating with simple placeholders;
  `generate_filename` accepts a `filename_template` but currently ignores it and returns the
  original filename.
- `app.py` — `watcher_task` is declared as a module-level global but assigned locally inside
  `setup_directory_watcher`, so the global is never populated (cancellation still works via the
  closure).
- `BookService.get_files` (the multi-book ZIP download) opens `Path(file.path)`, but `file.path` is
  stored relative to `book.path` — worth verifying before relying on that endpoint.