chitai/backend/AGENTS.md

Chitai backend

Litestar REST API for the eBook library. See the repo-root AGENTS.md for the overall picture and dev-environment setup.

Stack: Python 3.13 · Litestar 2 · advanced-alchemy over async SQLAlchemy 2 · asyncpg · PostgreSQL 17 · Alembic (through advanced-alchemy's alchemy CLI) · pydantic-settings · uv.

Layering

controllers/      HTTP surface only — parse, delegate, serialise. Keep thin.
services/         Business logic. One SQLAlchemyAsyncRepositoryService subclass per aggregate.
database/models/  SQLAlchemy models.
schemas/          Pydantic DTOs for request bodies and responses.

Controllers call service methods, not the repository. app.py assembles the app: route handlers, JWT auth, exception handlers, the SQLAlchemy plugin, and two lifespan context managers.

advanced-alchemy idioms

These are the conventions that are easy to get wrong if you write plain SQLAlchemy here:

• Models extend BigIntAuditBase (adds id/created_at/updated_at); pure link and child rows extend BigIntBase (e.g. Identifier, FileMetadata, BookAuthorLink).

• A service declares an inner repository and points at it:

  class BookService(SQLAlchemyAsyncRepositoryService[Book]):
      class Repo(SQLAlchemyAsyncRepository[Book]):
          model_type = Book
      repository_type = Repo
  

• Transform incoming data with the to_model_on_create / to_model_on_update hooks, not by overriding create/update wholesale — see services/book.py:407 onward.

• Serialise responses through service.to_schema(obj, schema_type=s.SomeRead); for lists, to_schema(items, total, filters, schema_type=…) produces the OffsetPagination envelope.

• Author, Tag, Publisher and BookSeries are deduplicated with as_unique_async. Never construct them directly when attaching to a book — use await Author.as_unique_async(session, name=name) as BookService._populate_with_unique_relationships does, or you will create duplicate rows.

• Domain methods on services are named for the domain (create_book, update_book, add_files), deliberately distinct from the inherited CRUD names.

Dependency injection

services/dependencies.py is the hub; controllers wire providers in their dependencies dict.

• Most providers come from create_service_provider(SomeService, …) — one line each.
• provide_book_service is hand-written because it must inject eager loads and scope user-specific rows: selectinload for authors/tags/files/etc., plus with_loader_criteria so BookProgress and BookListLink only load rows belonging to current_user. If you add a relationship that the API returns, add it to that load list.
• create_book_filter_dependencies intentionally overrides advanced-alchemy's stock providers: the search filter becomes a trigram search, and order-by gains a random sort order. Do not replace it with the stock create_filter_dependencies.
• get_library_by_id resolves the target library from either a library_id query param or the book's own library_id, and raises 404 for either miss.

Filters

services/filters/ holds StatementFilter dataclasses that compose into any list / list_and_count call: TagFilter, AuthorFilter, BookshelfFilter, ProgressFilter, TrigramSearchFilter, CustomOrderBy, FileHashFilter, plus the *LibraryFilter variants used by OPDS.

To add list-filtering behaviour: write the dataclass here, add a provide_*_filter function in dependencies.py, register it in the controller's dependencies, and fold it into provide_book_filters — the controller handler itself does not change.

Authentication — three schemes

Scheme	Where	Used by
JWT bearer (OAuth2PasswordBearerAuth)	app.py	The web frontend and the main API. Public paths are listed in its exclude=[…].
HTTP Basic (middleware/basic_auth.py)	OpdsController	E-reader / OPDS clients, which only speak Basic.
X-AUTH-USER API key (middleware/kosync_auth.py)	KosyncController	KOReader devices; the key maps to a KosyncDevice row, which maps to a user.

Each middleware resolves a User onto the connection; the matching provide_user_via_basic_auth / provide_user_via_kosync_auth dependencies expose it to handlers.

Filesystem behaviour

The backend owns files on disk, not just rows:

• Layout — services/filesystem_library.py (BookPathGenerator) renders a Jinja2 template against book metadata to decide where a book lives under the library's root_path (default: author/series/position - title/).
• One directory per book, never shared. The generated path is a pure function of the metadata, so two books with the same author and title produce the same one — two editions, or an allow_duplicates copy. BookService._reserve_book_path moves the later one to title (2) before anything is written, and update_book reserves the same way so a rename cannot move a book in on top of another. This matters because book.path is what deletes, moves and file lookups act on: books sharing a directory means one overwrites the other's files, and deleting either takes both. _unused_path does the same job for filenames within a directory. A book that already has a path keeps it — add_files must follow the book, not the template.
• Metadata extraction — services/metadata_extractor.py reads EPUB (ebooklib) and PDF (pypdfium2) files; extracted values fill only empty fields on the incoming payload.
• Covers — converted to WebP with a UUID filename under settings.book_cover_path, served by a static-files router mounted at /covers.
• Consume directory — services/consume.py (ConsumeDirectoryWatcher) watches settings.consume_path with watchfiles, creates one subdirectory per library slug, batches additions (3 s debounce) and imports them via BookService.create_many_from_existing_files. Started as an asyncio task from the setup_directory_watcher lifespan hook.
• Updates move files. BookService.update_book regenerates the path from the new metadata and, if it differs, moves the directory contents and prunes empty parents. Keep that in mind before changing metadata handling.

Duplicate detection

Every ingest path screens incoming files against what is already stored, keyed on (hash, size) — never the hash alone, because it samples 12 KiB (see below) and EPUBs from one toolchain often share their first window. FileMetadata.hash carries a plain, deliberately non-unique index: a collision must not be able to fail an import, and older databases may already hold duplicates.

Scope comes from CHITAI_DUPLICATE_SCOPE (library, the default | global | off).

The policy differs by how deliberate the import is:

Path	Behaviour
create_many_from_files (browser bulk)	Skip per file, skip a whole group whose files are all known, report everything skipped in ImportResult.duplicates. Re-dropping a folder to pick up what is new is the case this serves.
create_book (single, with metadata)	All-or-nothing: raises DuplicateFilesError, which controllers/book.py renders as a 409 carrying the refused files in extra.
add_files	A file the book already carries is a no-op; one stored under another book raises DuplicateFilesError.
create_many_from_existing_files (consume watcher)	Skips, and moves the file to CHITAI_DUPLICATE_PATH/<library slug>/ — nothing is deleted, and it cannot stay put because watchfiles only reports additions. That path must stay outside consume_path or the watcher re-imports it and tries to read the directory name as a library slug.

allow_duplicates=true overrides all of it, on every endpoint. Keep that working — the hash is not proof of identity, so a wrong verdict has to be recoverable, and a scripted import needs a way through. The web UI deliberately does not offer it: storing the same bytes twice splits reading progress and shelf membership across two records that can never converge, which is nothing anyone wants on purpose.

Three things to preserve when touching this code:

• A match only counts while the file is on disk. find_duplicate_files stats each candidate, and add_files writes a missing file back into the row that already describes it (_restore_file) instead of adding a second row beside it. The hash lives in the database and the file does not, so without this a file deleted behind the app's back would go on refusing its own replacement.

• Screening runs before anything is written. fingerprint_upload reads the spooled upload and rewinds it; the resulting fingerprints are handed to _save_book_files, which skips its own StreamingHasher when it already has the answer. Passing them through is what keeps the file from being read twice.

• _screen_for_duplicates extends the known dict as it goes, so the same bytes submitted twice in one request are caught. Those duplicates report book_id: None — there is no row to point at yet.

POST /books/duplicate-files answers the same question from fingerprints alone, for clients that want to ask before uploading anything.

Book-level detection — a different question

The file check answers "are these the same bytes?". find_duplicate_books answers "is this the same book?", which a re-scan, a re-zipped EPUB or another edition cannot be asked with a hash. Two signals, either sufficient: a shared identifier, or a matching normalized title with at least one shared author.

It never blocks. A metadata match is a guess — a work shares title and author with its own translation, its own second edition and its own audiobook — so the book is created and the candidates are reported alongside it in ImportResult.possible_duplicates. File-level dedupe keeps its refuse/skip behaviour; that one is near-certain and this one is not. Do not "improve" this into a refusal.

Two rules narrow it, both applied in Python over the small candidate set:

• A shared author is required for a title match. Without it every book the extractors gave up on and titled Unknown is a duplicate of every other one. A book with no authors can therefore only match on an identifier.
• The same series at a different series_position disqualifies a match. A trilogy shares an author and often most of its title; the position is the library saying outright that these are two books.

A book is compared under several title keys, not one (_title_keys). Ebook files are overwhelmingly named Title - Author.epub, and wherever nothing inside the file overrode that name the author ended up in the title column — so one copy is stored as Building Microservices and another as Building Microservices Sam Newman. Both directions are generated, the author stripped off and the author added on, which is why the query is normalized_title.in_(keys) rather than ==. This is still exact matching on an indexed column: no similarity score, nothing to tune. It does not weaken the shared-author requirement, which is a separate condition.

find_duplicate_book_groups is the library-wide pass behind GET /books/duplicate-books, since the import-time check says nothing about a collection someone already has. It buckets books by every key they carry and merges the buckets with union-find, so AB by ISBN and BC by title land in one group. Pairs in duplicate_dismissals are never merged — a reader disagreeing with one pairing must not silently break a group that stands on other evidence.

Author names have one stored form

Author.name is always the canonical form, produced by format_author_name. Extractors hand over whatever the file said — Newman, Sam; from a DC:creator list, Sam Newman from a PDF, Sam Newman.epub from a filename — and storing those verbatim is how one person becomes four rows in the sidebar, four entries in the author filter, and four books that never look like each other.

This is display canonicalization, distinct from normalize_author, which throws away case, accents and spacing to build a comparison key nobody sees. Tidying only removes what an extractor added: a trailing separator, a file extension, and the Surname, Given ordering. It never touches case or accents — Michał Płachta and Steve McConnell are the author's own spelling, not something to correct.

Three places have to agree, and Author keeps them together: the @validates("name") hook, unique_hash, and unique_filter. as_unique_async looks a row up with the filter and then constructs with the validator, so if the lookup used the raw name and the insert used the tidy one, every variant spelling would miss the existing row and then collide with it on the unique index.

A form the rule does not recognise is left exactly as it was found — Dave Thomas, Andy Hunt is two people in one string, and flipping it would invent a third. Leaving a mess visible beats rewriting it wrongly.

The normalized columns are written by validators

Book.normalized_title, Author.normalized_name and Identifier.normalized_value are derived from services/matching.py and kept current by SQLAlchemy @validates hooks on the models, not by any service. Assigning them directly is always wrong.

This is deliberate and it is invisible at the call sites: BookService writes titles through at least three paths (to_model_on_create, to_model_on_update, and the setattr loop in _populate_with_unique_relationships), and a validator is the only thing a fourth cannot bypass. The columns carry plain, non-unique btree indexes — two spellings collapsing onto one value is the entire point.

services/matching.py is imported inside those validators rather than at module scope: reaching it initialises the chitai.services package, which imports the services, which import the models. Keep the local import.

The validators only fire on write, so a migration that adds one of these columns must backfill existing rows through the same helpers — see the data_upgrades() hook in 2026-08-15_add_book_matching_keys_and_duplicate__4358e7d4743a.py.

Changing anything in services/matching.py needs a revision that recomputes them. The keys are derived and already written, so a normalization change silently invalidates every stored row: a book written under the old rules just stops matching one written under the new rules, with nothing to show that anything is wrong. Copy 2026-08-15_recompute_book_matching_keys_ed41acf21270.py, which exists because normalize_title learned to strip compact edition markers (2E, 5e). It is idempotent and safe to re-run.

KOReader hashing

services/utils.py reimplements KOReader's partial-MD5 document identifier: 1 KiB samples at offsets produced by LuaJIT's 32-bit bit.lshift, including its shift-masking wrap-around (shift & 0x1F, so i=-1 yields offset 0). _lshift32 looks wrong and is not — the overflow is what makes hashes match real devices. Don't "simplify" it; tests/integration/test_file_hash.py guards the behaviour.

Database

• pg_trgm is required. Book.__table_args__ declares a GIN trigram index on title, which TrigramSearchFilter uses for fuzzy title search. The extension is enabled by a migration and, in tests, by conftest.py.
• Migrations live in migrations/versions/. Generate with alchemy --config chitai.database.config.config make-migrations (add --no-autogenerate for a blank revision), apply with … upgrade. database/config.py sets create_all=False, so nothing is auto-created at runtime; production applies migrations from entrypoint.sh.
• Sessions use expire_on_commit=False and Litestar's before_send_handler="autocommit", so a handler that returns 2xx commits automatically.

Testing

pytest tests/ — asyncio_mode = "auto", so async tests need no marker.

• tests/unit/ exercises services directly against a real session; tests/integration/ drives the whole app through AsyncTestClient(app=create_app()).
• The database is a throwaway container from pytest-databases, so Docker must be running.
• tests/conftest.py provides the shared fixtures: client, authenticated_client, other_authenticated_client (a second user, for access-control tests), one fixture per service, test_user / test_library, and an autouse fixture that redirects cover storage into tmp_path.
• tests/integration/conftest.py monkeypatches the module-level alchemy config onto the test engine/sessionmaker and drops+recreates+reseeds the schema for every test.
• Real EPUB and PDF fixtures live in tests/data_files/.

Known rough edges

Observed in the current tree — don't mistake these for intentional patterns to copy:

• controllers/book.py — file-level TODO: book_id is a path parameter on some endpoints and a query parameter on others. set_book_progress_batch does a documented N+1 (one select + one upsert per book).
• services/filesystem_library.py — TODO to replace Jinja2 templating with simple placeholders; generate_filename accepts a filename_template but currently ignores it and returns the original filename.
• app.py — watcher_task is declared as a module-level global but assigned locally inside setup_directory_watcher, so the global is never populated (cancellation still works via the closure).
• BookService.get_files (the multi-book ZIP download) opens Path(file.path), but file.path is stored relative to book.path — worth verifying before relying on that endpoint.