chitai/backend/AGENTS.md

Chitai backend

Litestar REST API for the eBook library. See the repo-root AGENTS.md for the overall picture and dev-environment setup.

Stack: Python 3.13 · Litestar 2 · advanced-alchemy over async SQLAlchemy 2 · asyncpg · PostgreSQL 17 · Alembic (through advanced-alchemy's alchemy CLI) · pydantic-settings · uv.

Layering

controllers/      HTTP surface only — parse, delegate, serialise. Keep thin.
services/         Business logic. One SQLAlchemyAsyncRepositoryService subclass per aggregate.
database/models/  SQLAlchemy models.
schemas/          Pydantic DTOs for request bodies and responses.

Controllers call service methods, not the repository. app.py assembles the app: route handlers, JWT auth, exception handlers, the SQLAlchemy plugin, and two lifespan context managers.

advanced-alchemy idioms

These are the conventions that are easy to get wrong if you write plain SQLAlchemy here:

• Models extend BigIntAuditBase (adds id/created_at/updated_at); pure link and child rows extend BigIntBase (e.g. Identifier, FileMetadata, BookAuthorLink).

• A service declares an inner repository and points at it:

  class BookService(SQLAlchemyAsyncRepositoryService[Book]):
      class Repo(SQLAlchemyAsyncRepository[Book]):
          model_type = Book
      repository_type = Repo
  

• Transform incoming data with the to_model_on_create / to_model_on_update hooks, not by overriding create/update wholesale — see services/book.py:407 onward.

• Serialise responses through service.to_schema(obj, schema_type=s.SomeRead); for lists, to_schema(items, total, filters, schema_type=…) produces the OffsetPagination envelope.

• Author, Tag, Publisher and BookSeries are deduplicated with as_unique_async. Never construct them directly when attaching to a book — use await Author.as_unique_async(session, name=name) as BookService._populate_with_unique_relationships does, or you will create duplicate rows.

• Domain methods on services are named for the domain (create_book, update_book, add_files), deliberately distinct from the inherited CRUD names.

Dependency injection

services/dependencies.py is the hub; controllers wire providers in their dependencies dict.

• Most providers come from create_service_provider(SomeService, …) — one line each.
• provide_book_service is hand-written because it must inject eager loads and scope user-specific rows: selectinload for authors/tags/files/etc., plus with_loader_criteria so BookProgress and BookListLink only load rows belonging to current_user. If you add a relationship that the API returns, add it to that load list.
• create_book_filter_dependencies intentionally overrides advanced-alchemy's stock providers: the search filter becomes a trigram search, and order-by gains a random sort order. Do not replace it with the stock create_filter_dependencies.
• get_library_by_id resolves the target library from either a library_id query param or the book's own library_id, and raises 404 for either miss.

Filters

services/filters/ holds StatementFilter dataclasses that compose into any list / list_and_count call: TagFilter, AuthorFilter, BookshelfFilter, ProgressFilter, TrigramSearchFilter, CustomOrderBy, FileHashFilter, plus the *LibraryFilter variants used by OPDS.

To add list-filtering behaviour: write the dataclass here, add a provide_*_filter function in dependencies.py, register it in the controller's dependencies, and fold it into provide_book_filters — the controller handler itself does not change.

Authentication — three schemes

Scheme	Where	Used by
JWT bearer (OAuth2PasswordBearerAuth)	app.py	The web frontend and the main API. Public paths are listed in its exclude=[…].
HTTP Basic (middleware/basic_auth.py)	OpdsController	E-reader / OPDS clients, which only speak Basic.
X-AUTH-USER API key (middleware/kosync_auth.py)	KosyncController	KOReader devices; the key maps to a KosyncDevice row, which maps to a user.

Each middleware resolves a User onto the connection; the matching provide_user_via_basic_auth / provide_user_via_kosync_auth dependencies expose it to handlers.

Filesystem behaviour

The backend owns files on disk, not just rows:

• Layout — services/filesystem_library.py (BookPathGenerator) renders a Jinja2 template against book metadata to decide where a book lives under the library's root_path (default: author/series/position - title/).
• One directory per book, never shared. The generated path is a pure function of the metadata, so two books with the same author and title produce the same one — two editions, or an allow_duplicates copy. BookService._reserve_book_path moves the later one to title (2) before anything is written, and update_book reserves the same way so a rename cannot move a book in on top of another. This matters because book.path is what deletes, moves and file lookups act on: books sharing a directory means one overwrites the other's files, and deleting either takes both. _unused_path does the same job for filenames within a directory. A book that already has a path keeps it — add_files must follow the book, not the template.
• Metadata extraction — services/metadata_extractor.py reads EPUB (ebooklib) and PDF (pypdfium2) files; extracted values fill only empty fields on the incoming payload.
• Covers — converted to WebP with a UUID filename under settings.book_cover_path, served by a static-files router mounted at /covers.
• Consume directory — services/consume.py (ConsumeDirectoryWatcher) watches settings.consume_path with watchfiles, creates one subdirectory per library slug, batches additions (3 s debounce) and imports them via BookService.create_many_from_existing_files. Started as an asyncio task from the setup_directory_watcher lifespan hook.
• Updates move files. BookService.update_book regenerates the path from the new metadata and, if it differs, moves the directory contents and prunes empty parents. Keep that in mind before changing metadata handling.

Duplicate detection

Every ingest path screens incoming files against what is already stored, keyed on (hash, size) — never the hash alone, because it samples 12 KiB (see below) and EPUBs from one toolchain often share their first window. FileMetadata.hash carries a plain, deliberately non-unique index: a collision must not be able to fail an import, and older databases may already hold duplicates.

Scope comes from CHITAI_DUPLICATE_SCOPE (library, the default | global | off).

The policy differs by how deliberate the import is:

Path	Behaviour
create_many_from_files (browser bulk)	Skip per file, skip a whole group whose files are all known, report everything skipped in ImportResult.duplicates. Re-dropping a folder to pick up what is new is the case this serves.
create_book (single, with metadata)	All-or-nothing: raises DuplicateFilesError, which controllers/book.py renders as a 409 carrying the refused files in extra.
add_files	A file the book already carries is a no-op; one stored under another book raises DuplicateFilesError.
create_many_from_existing_files (consume watcher)	Skips, and moves the file to CHITAI_DUPLICATE_PATH/<library slug>/ — nothing is deleted, and it cannot stay put because watchfiles only reports additions. That path must stay outside consume_path or the watcher re-imports it and tries to read the directory name as a library slug.

allow_duplicates=true overrides all of it, on every endpoint. Keep that working — the hash is not proof of identity, so a false positive has to be recoverable.

Two things to preserve when touching this code:

• Screening runs before anything is written. fingerprint_upload reads the spooled upload and rewinds it; the resulting fingerprints are handed to _save_book_files, which skips its own StreamingHasher when it already has the answer. Passing them through is what keeps the file from being read twice.
• _screen_for_duplicates extends the known dict as it goes, so the same bytes submitted twice in one request are caught. Those duplicates report book_id: None — there is no row to point at yet.

POST /books/duplicates answers the same question from fingerprints alone, for clients that want to ask before uploading anything.

KOReader hashing

services/utils.py reimplements KOReader's partial-MD5 document identifier: 1 KiB samples at offsets produced by LuaJIT's 32-bit bit.lshift, including its shift-masking wrap-around (shift & 0x1F, so i=-1 yields offset 0). _lshift32 looks wrong and is not — the overflow is what makes hashes match real devices. Don't "simplify" it; tests/integration/test_file_hash.py guards the behaviour.

Database

• pg_trgm is required. Book.__table_args__ declares a GIN trigram index on title, which TrigramSearchFilter uses for fuzzy title search. The extension is enabled by a migration and, in tests, by conftest.py.
• Migrations live in migrations/versions/. Generate with alchemy --config chitai.database.config.config make-migrations (add --no-autogenerate for a blank revision), apply with … upgrade. database/config.py sets create_all=False, so nothing is auto-created at runtime; production applies migrations from entrypoint.sh.
• Sessions use expire_on_commit=False and Litestar's before_send_handler="autocommit", so a handler that returns 2xx commits automatically.

Testing

pytest tests/ — asyncio_mode = "auto", so async tests need no marker.

• tests/unit/ exercises services directly against a real session; tests/integration/ drives the whole app through AsyncTestClient(app=create_app()).
• The database is a throwaway container from pytest-databases, so Docker must be running.
• tests/conftest.py provides the shared fixtures: client, authenticated_client, other_authenticated_client (a second user, for access-control tests), one fixture per service, test_user / test_library, and an autouse fixture that redirects cover storage into tmp_path.
• tests/integration/conftest.py monkeypatches the module-level alchemy config onto the test engine/sessionmaker and drops+recreates+reseeds the schema for every test.
• Real EPUB and PDF fixtures live in tests/data_files/.

Known rough edges

Observed in the current tree — don't mistake these for intentional patterns to copy:

• controllers/book.py — file-level TODO: book_id is a path parameter on some endpoints and a query parameter on others. set_book_progress_batch does a documented N+1 (one select + one upsert per book).
• services/filesystem_library.py — TODO to replace Jinja2 templating with simple placeholders; generate_filename accepts a filename_template but currently ignores it and returns the original filename.
• app.py — watcher_task is declared as a module-level global but assigned locally inside setup_directory_watcher, so the global is never populated (cancellation still works via the closure).
• BookService.get_files (the multi-book ZIP download) opens Path(file.path), but file.path is stored relative to book.path — worth verifying before relying on that endpoint.