chitai/backend/AGENTS.md

Chitai backend

Litestar REST API for the eBook library. See the repo-root AGENTS.md for the overall picture and dev-environment setup.

Stack: Python 3.13 · Litestar 2 · advanced-alchemy over async SQLAlchemy 2 · asyncpg · PostgreSQL 17 · Alembic (through advanced-alchemy's alchemy CLI) · pydantic-settings · uv.

Layering

controllers/      HTTP surface only — parse, delegate, serialise. Keep thin.
services/         Business logic. One SQLAlchemyAsyncRepositoryService subclass per aggregate.
database/models/  SQLAlchemy models.
schemas/          Pydantic DTOs for request bodies and responses.

Controllers call service methods, not the repository. app.py assembles the app: route handlers, JWT auth, exception handlers, the SQLAlchemy plugin, and two lifespan context managers.

advanced-alchemy idioms

These are the conventions that are easy to get wrong if you write plain SQLAlchemy here:

• Models extend BigIntAuditBase (adds id/created_at/updated_at); pure link and child rows extend BigIntBase (e.g. Identifier, FileMetadata, BookAuthorLink).

• A service declares an inner repository and points at it:

  class BookService(SQLAlchemyAsyncRepositoryService[Book]):
      class Repo(SQLAlchemyAsyncRepository[Book]):
          model_type = Book
      repository_type = Repo
  

• Transform incoming data with the to_model_on_create / to_model_on_update hooks, not by overriding create/update wholesale — see services/book.py:407 onward.

• Serialise responses through service.to_schema(obj, schema_type=s.SomeRead); for lists, to_schema(items, total, filters, schema_type=…) produces the OffsetPagination envelope.

• Author, Tag, Publisher and BookSeries are deduplicated with as_unique_async. Never construct them directly when attaching to a book — use await Author.as_unique_async(session, name=name) as BookService._populate_with_unique_relationships does, or you will create duplicate rows.

• Domain methods on services are named for the domain (create_book, update_book, add_files), deliberately distinct from the inherited CRUD names.

Dependency injection

services/dependencies.py is the hub; controllers wire providers in their dependencies dict.

• Most providers come from create_service_provider(SomeService, …) — one line each.
• provide_book_service is hand-written because it must inject eager loads and scope user-specific rows: selectinload for authors/tags/files/etc., plus with_loader_criteria so BookProgress and BookListLink only load rows belonging to current_user. If you add a relationship that the API returns, add it to that load list.
• create_book_filter_dependencies intentionally overrides advanced-alchemy's stock providers: the search filter becomes a trigram search, and order-by gains a random sort order. Do not replace it with the stock create_filter_dependencies.
• get_library_by_id resolves the target library from either a library_id query param or the book's own library_id, and raises 404 for either miss.

Filters

services/filters/ holds StatementFilter dataclasses that compose into any list / list_and_count call: TagFilter, AuthorFilter, BookshelfFilter, ProgressFilter, TrigramSearchFilter, CustomOrderBy, FileHashFilter, plus the *LibraryFilter variants used by OPDS.

To add list-filtering behaviour: write the dataclass here, add a provide_*_filter function in dependencies.py, register it in the controller's dependencies, and fold it into provide_book_filters — the controller handler itself does not change.

Authentication — three schemes

Scheme	Where	Used by
JWT bearer (OAuth2PasswordBearerAuth)	app.py	The web frontend and the main API. Public paths are listed in its exclude=[…].
HTTP Basic (middleware/basic_auth.py)	OpdsController	E-reader / OPDS clients, which only speak Basic.
X-AUTH-USER API key (middleware/kosync_auth.py)	KosyncController	KOReader devices; the key maps to a KosyncDevice row, which maps to a user.

Each middleware resolves a User onto the connection; the matching provide_user_via_basic_auth / provide_user_via_kosync_auth dependencies expose it to handlers.

Filesystem behaviour

The backend owns files on disk, not just rows:

• Layout — services/filesystem_library.py (BookPathGenerator) renders a Jinja2 template against book metadata to decide where a book lives under the library's root_path (default: author/series/position - title/).
• One directory per book, never shared. The generated path is a pure function of the metadata, so two books with the same author and title produce the same one — two editions, or an allow_duplicates copy. BookService._reserve_book_path moves the later one to title (2) before anything is written, and update_book reserves the same way so a rename cannot move a book in on top of another. This matters because book.path is what deletes, moves and file lookups act on: books sharing a directory means one overwrites the other's files, and deleting either takes both. _unused_path does the same job for filenames within a directory. A book that already has a path keeps it — add_files must follow the book, not the template.
• Metadata extraction — services/metadata_extractor.py reads EPUB (ebooklib) and PDF (pypdfium2) files; extracted values fill only empty fields on the incoming payload.
• Covers — converted to WebP with a UUID filename under settings.book_cover_path, served by a static-files router mounted at /covers.
• Consume directory — services/consume.py (ConsumeDirectoryWatcher) watches settings.consume_path with watchfiles, creates one subdirectory per library slug, batches additions (3 s debounce) and imports them via BookService.create_many_from_existing_files. Started as an asyncio task from the setup_directory_watcher lifespan hook.
• Updates move files. BookService.update_book regenerates the path from the new metadata and, if it differs, moves the directory contents and prunes empty parents. Keep that in mind before changing metadata handling.

Content types come from the extension, and null means null

Every ingest path names a file's format with guess_content_type (services/utils.py), never with mimetypes.guess_type directly and never with what the client said. Python's built-in map answers None for .mobi, .azw, .prc, .fb2, .fbz, .lit, .lrf and .cb7 — most of what a library imported from elsewhere carries — so EBOOK_CONTENT_TYPES fills those in. A browser's application/octet-stream is discarded rather than used as a fallback: it is the client saying it does not know, and storing it is indistinguishable from having determined a format.

When nothing can name the extension the column stays null. That is the honest answer, and only one consumer cannot take it: OPDS Link.type is a required string, so services/opds/opds.py substitutes application/octet-stream at that boundary. Litestar's ASGIFileResponse already does its own fallback, so get_file can pass a null straight through. FileMetadataRead.content_type is nullable for the same reason — it was once required, which turned a stored null into a 500 on a book that was otherwise fine.

Duplicate detection

Every ingest path screens incoming files against what is already stored, keyed on (hash, size) — never the hash alone, because it samples 12 KiB (see below) and EPUBs from one toolchain often share their first window. FileMetadata.hash carries a plain, deliberately non-unique index: a collision must not be able to fail an import, and older databases may already hold duplicates.

Scope comes from CHITAI_DUPLICATE_SCOPE (library, the default | global | off).

The policy differs by how deliberate the import is:

Path	Behaviour
create_many_from_files (browser bulk)	Skip per file, skip a whole group whose files are all known, report everything skipped in ImportResult.duplicates. Re-dropping a folder to pick up what is new is the case this serves.
create_book (single, with metadata)	All-or-nothing: raises DuplicateFilesError, which controllers/book.py renders as a 409 carrying the refused files in extra.
add_files	A file the book already carries is a no-op; one stored under another book raises DuplicateFilesError.
create_many_from_existing_files (consume watcher)	Skips, and moves the file to CHITAI_DUPLICATE_PATH/<library slug>/ — nothing is deleted, and it cannot stay put because watchfiles only reports additions. That path must stay outside consume_path or the watcher re-imports it and tries to read the directory name as a library slug.
create_many_from_calibre (Calibre import)	Skips per file, and skips a whole book whose files are all known. Nothing is moved — the source is somebody else's library. This is what makes a re-run a no-op and an interrupted import resumable by running it again.

allow_duplicates=true overrides all of it, on every endpoint. Keep that working — the hash is not proof of identity, so a wrong verdict has to be recoverable, and a scripted import needs a way through. The web UI deliberately does not offer it: storing the same bytes twice splits reading progress and shelf membership across two records that can never converge, which is nothing anyone wants on purpose.

Three things to preserve when touching this code:

• A match only counts while the file is on disk. find_duplicate_files stats each candidate, and add_files writes a missing file back into the row that already describes it (_restore_file) instead of adding a second row beside it. The hash lives in the database and the file does not, so without this a file deleted behind the app's back would go on refusing its own replacement.

• Screening runs before anything is written. fingerprint_upload reads the spooled upload and rewinds it; the resulting fingerprints are handed to _save_book_files, which skips its own StreamingHasher when it already has the answer. Passing them through is what keeps the file from being read twice.

• _screen_for_duplicates extends the known dict as it goes, so the same bytes submitted twice in one request are caught. Those duplicates report book_id: None — there is no row to point at yet.

POST /books/duplicate-files answers the same question from fingerprints alone, for clients that want to ask before uploading anything.

Book-level detection — a different question

The file check answers "are these the same bytes?". find_duplicate_books answers "is this the same book?", which a re-scan, a re-zipped EPUB or another edition cannot be asked with a hash. Two signals, either sufficient: a shared identifier, or a matching normalized title with at least one shared author.

It never blocks. A metadata match is a guess — a work shares title and author with its own translation, its own second edition and its own audiobook — so the book is created and the candidates are reported alongside it in ImportResult.possible_duplicates. File-level dedupe keeps its refuse/skip behaviour; that one is near-certain and this one is not. Do not "improve" this into a refusal.

Two rules narrow it, both applied in Python over the small candidate set:

• A shared author is required for a title match. Without it every book the extractors gave up on and titled Unknown is a duplicate of every other one. A book with no authors can therefore only match on an identifier.
• The same series at a different series_position disqualifies a match. A trilogy shares an author and often most of its title; the position is the library saying outright that these are two books.

A book is compared under several title keys, not one (_title_keys). Ebook files are overwhelmingly named Title - Author.epub, and wherever nothing inside the file overrode that name the author ended up in the title column — so one copy is stored as Building Microservices and another as Building Microservices Sam Newman. Both directions are generated, the author stripped off and the author added on, which is why the query is normalized_title.in_(keys) rather than ==. This is still exact matching on an indexed column: no similarity score, nothing to tune. It does not weaken the shared-author requirement, which is a separate condition.

find_duplicate_book_groups is the library-wide pass behind GET /books/duplicate-books, since the import-time check says nothing about a collection someone already has. It buckets books by every key they carry and merges the buckets with union-find, so AB by ISBN and BC by title land in one group. Pairs in duplicate_dismissals are never merged — a reader disagreeing with one pairing must not silently break a group that stands on other evidence.

Author names have one stored form

Author.name is always the canonical form, produced by format_author_name. Extractors hand over whatever the file said — Newman, Sam; from a DC:creator list, Sam Newman from a PDF, Sam Newman.epub from a filename — and storing those verbatim is how one person becomes four rows in the sidebar, four entries in the author filter, and four books that never look like each other.

This is display canonicalization, distinct from normalize_author, which throws away case, accents and spacing to build a comparison key nobody sees. Tidying only removes what an extractor added: a trailing separator, a file extension, and the Surname, Given ordering. It never touches case or accents — Michał Płachta and Steve McConnell are the author's own spelling, not something to correct.

Three places have to agree, and Author keeps them together: the @validates("name") hook, unique_hash, and unique_filter. as_unique_async looks a row up with the filter and then constructs with the validator, so if the lookup used the raw name and the insert used the tidy one, every variant spelling would miss the existing row and then collide with it on the unique index.

A form the rule does not recognise is left exactly as it was found — Dave Thomas, Andy Hunt is two people in one string, and flipping it would invent a third. Leaving a mess visible beats rewriting it wrongly.

The normalized columns are written by validators

Book.normalized_title, Author.normalized_name and Identifier.normalized_value are derived from services/matching.py and kept current by SQLAlchemy @validates hooks on the models, not by any service. Assigning them directly is always wrong.

This is deliberate and it is invisible at the call sites: BookService writes titles through at least three paths (to_model_on_create, to_model_on_update, and the setattr loop in _populate_with_unique_relationships), and a validator is the only thing a fourth cannot bypass. The columns carry plain, non-unique btree indexes — two spellings collapsing onto one value is the entire point.

services/matching.py is imported inside those validators rather than at module scope: reaching it initialises the chitai.services package, which imports the services, which import the models. Keep the local import.

The validators only fire on write, so a migration that adds one of these columns must backfill existing rows through the same helpers — see the data_upgrades() hook in 2026-08-15_add_book_matching_keys_and_duplicate__4358e7d4743a.py.

Changing anything in services/matching.py needs a revision that recomputes them. The keys are derived and already written, so a normalization change silently invalidates every stored row: a book written under the old rules just stops matching one written under the new rules, with nothing to show that anything is wrong. Copy 2026-08-15_recompute_book_matching_keys_ed41acf21270.py, which exists because normalize_title learned to strip compact edition markers (2E, 5e). It is idempotent and safe to re-run.

Importing from Calibre

Two pieces, deliberately separated:

• services/calibre.py reads metadata.db and the tree beside it. It knows nothing about Book, BookService or a session, so it is testable without Postgres, and it reports what Calibre wrote rather than what Chitai wants — identifiers come back keyed by identifiers.type verbatim. It also unpacks a zipped library (extract_calibre_archive).
• BookService.create_many_from_calibre does the ingest, next to the two other ingest paths because it needs the same privates they do (_reserve_book_path, _save_cover_image, _screen_for_duplicates). The CLI in cli.py is a thin wrapper over it.
• services/calibre_import.py is lifecycle only — the job registry behind the endpoints: state, progress, cancellation, and a session of its own.

docs/calibre-import.md is the full brief, including the phases not built yet. What matters here:

• Files are copied, never moved. metadata.db would go on pointing at files that are gone, which quietly ruins a library somebody still uses. copy_file streams rather than using shutil.copy, which would block the loop for a 40 MB read.
• The extractors are not run. This is the one ingest path that trusts its input: Calibre's catalogue is curated and its filenames are truncated to ~42 characters, so data.name locates a file and the database carries the metadata. The title is stored verbatim for the same reason — no edition split out, no subtitle guessed.
• The catalogue is copied before it is read, and the copy is opened read-write. Calibre may be running; opening the live file either sees a torn state or needs to recover a write-ahead log, which read-only access cannot do. close() removes the copy in a finally, or a failure leaves a catalogue-sized file in the temp directory.
• check_same_thread=False plus an asyncio.Lock. Every query runs through asyncio.to_thread, which hands out whichever worker is free, so the connection outlives the thread that opened it. The lock is what makes that safe. Removing either one reintroduces SQLite objects created in a thread can only be used in that same thread, intermittently — the pool often reuses one thread, so it passes until it does not.
• One book never costs the run. A failure is recorded in CalibreImportResult.failed, the session is rolled back so the next book can use it, and the files that book had already copied are deleted — an orphaned directory would make the next attempt reserve title (2) and look as though it had worked. A cover that PIL cannot open costs the cover, not the book.
• calibre-uuid, not uuid. books.uuid is stable for the life of the row, so it is the durable link back to the source and worth matching on. uuid is the name normalize_identifier refuses, because an EPUB regenerates one per build.

The API takes an uploaded archive; the CLI takes a path

POST /libraries/{id}/imports/calibre/upload is the only way in over HTTP. It takes a zipped Calibre library, answers 202 with a job handle, and unpacks into a temp directory the job owns. GET /libraries/imports/{job_id} is polled; DELETE on the same path stops it.

There is deliberately no endpoint that imports from a server path. A desktop Calibre install is not on the server, and importing from a path the server can already see is a server-side operation — which is what litestar --app-dir src/chitai/ calibre-import <path> --library <slug> is for, including its --dry-run. Do not add the path endpoint back without being asked: it was built, then removed on purpose.

• The registry is in memory, so it assumes one worker process. That holds today (litestar run is single-process, and the consume watcher is already an in-process singleton), but the day TODO.md's "production image runs the development server" item is fixed with a worker count, a poll can land on a worker that never heard of the job. services/calibre_import.py says so at the top; an import_jobs table is the answer when that happens.
• The job opens its own session. The request that started it is long gone and its session closed with it.
• Cancelling is not aborting. A flag is read between books, never during one, so a cancelled import leaves whole books behind and never half of one. task.cancel() would abandon a book mid-copy and leave files with no row describing them.
• The job deletes its workspace — the unpacked archive is a second copy of the whole library, and the books worth keeping have been copied into the library proper by the time it ends. Removed even when the run failed, since nothing will come back for it.
• Extraction refuses an entry pointing outside the archive (zip slip), an archive that will not fit on disk, and one with no metadata.db within three levels. All three answer 400 before a job exists, rather than as a job that reports FAILED a moment later.
• The upload is streamed both sides. The archive reaches disk in chunks rather than being read whole, and the SvelteKit proxy passes request.body through instead of buffering it — see frontend/AGENTS.md.

Things Calibre does that will produce wrong data if you forget them are documented at the top of services/calibre.py — the 0101-01-01 date sentinel, | for a comma in an author name, the REAL series_index that defaults to 1.0 for every book, HTML in comments, the views that need SQLite functions Calibre registers from Python, and books_pages_link being both recent and usually empty. tests/calibre_fixtures.py builds a library exercising all of them.

KOReader hashing

services/utils.py reimplements KOReader's partial-MD5 document identifier: 1 KiB samples at offsets produced by LuaJIT's 32-bit bit.lshift, including its shift-masking wrap-around (shift & 0x1F, so i=-1 yields offset 0). _lshift32 looks wrong and is not — the overflow is what makes hashes match real devices. Don't "simplify" it; tests/integration/test_file_hash.py guards the behaviour.

Database

• pg_trgm is required. Book.__table_args__ declares a GIN trigram index on title, which TrigramSearchFilter uses for fuzzy title search. The extension is enabled by a migration and, in tests, by conftest.py.
• Migrations live in migrations/versions/. Generate with alchemy --config chitai.database.config.config make-migrations (add --no-autogenerate for a blank revision), apply with … upgrade. database/config.py sets create_all=False, so nothing is auto-created at runtime; production applies migrations from entrypoint.sh.
• Sessions use expire_on_commit=False and Litestar's before_send_handler="autocommit", so a handler that returns 2xx commits automatically.

Testing

pytest tests/ — asyncio_mode = "auto", so async tests need no marker.

• tests/unit/ exercises services directly against a real session; tests/integration/ drives the whole app through AsyncTestClient(app=create_app()).
• The database is a throwaway container from pytest-databases, so Docker must be running.
• tests/conftest.py provides the shared fixtures: client, authenticated_client, other_authenticated_client (a second user, for access-control tests), one fixture per service, test_user / test_library, and an autouse fixture that redirects cover storage into tmp_path.
• tests/integration/conftest.py monkeypatches the module-level alchemy config onto the test engine/sessionmaker and drops+recreates+reseeds the schema for every test.
• Real EPUB and PDF fixtures live in tests/data_files/.

Known rough edges

Observed in the current tree — don't mistake these for intentional patterns to copy:

• controllers/book.py — file-level TODO: book_id is a path parameter on some endpoints and a query parameter on others. set_book_progress_batch does a documented N+1 (one select + one upsert per book).
• services/filesystem_library.py — TODO to replace Jinja2 templating with simple placeholders; generate_filename accepts a filename_template but currently ignores it and returns the original filename.
• app.py — watcher_task is declared as a module-level global but assigned locally inside setup_directory_watcher, so the global is never populated (cancellation still works via the closure).
• BookService.get_files (the multi-book ZIP download) opens Path(file.path), but file.path is stored relative to book.path — worth verifying before relying on that endpoint.