# Implementation brief: importing a Calibre library Written for an agent picking this up cold. Read the repo-root `AGENTS.md` and `backend/AGENTS.md` first — this brief assumes both, particularly the **Filesystem behaviour** and **Duplicate detection** sections. Every claim about Calibre's schema and on-disk layout below was checked against a real library at `~/Documents/Calibre Library` (6 books, current Calibre). Where a fact came from Calibre's source rather than that library, it says so. ## Feasibility: high, and most of the machinery already exists `metadata.db` is plain SQLite with a schema that has been stable for a decade, and the files sit beside it in a predictable tree. Chitai already has every piece an import needs: | Needed | Already in the tree | | --- | --- | | Ingest files that are already on disk | `BookService.create_many_from_existing_files` (`services/book.py:1280`) | | Deduplicate authors/tags/publishers/series | `_populate_with_unique_relationships` (`services/book.py:1752`), via `as_unique_async` | | Generic identifiers with a scheme map | `Identifier`, and `parse_identifier` (`services/metadata_extractor.py:58`) — whose map already covers `isbn`, `amazon`, `mobi-asin`, `google`, `goodreads`, `doi`, `calibre` | | Decide where a book lives on disk | `BookPathGenerator`, `_reserve_book_path` (`services/book.py:1066`) | | Not import the same book twice | `find_duplicate_files` (`:461`) and `find_duplicate_books` (`:535`) | | Store a cover | `_save_cover_image` (`:1994`) | So this is **a reader, not new ingest machinery**: turn Calibre rows into the metadata dict `BookService` already accepts, and hand it to a slightly generalised version of the consume-directory path. The parsing is the easy half. The hard parts are elsewhere, and all three are addressed below: 1. Calibre libraries hold formats Chitai cannot describe, let alone read — and one of them **currently 500s the book detail endpoint** (see prerequisites). 2. A 5,000-book import is a long-running job, and the app has no job/progress concept. 3. Whether files are **copied** into the library or **referenced in place** — which is a product decision with a large blast radius, because in-place means Chitai's write paths point at somebody's Calibre library. ## What a Calibre library actually is ``` Calibre Library/ ├── metadata.db the whole catalogue ├── metadata_db_prefs_backup.json ignore ├── .caltrash/ .calnotes/ ignore — deleted books still live in .caltrash └── / └── (<book id>)/ == books.path ├── cover.jpg iff books.has_cover ├── metadata.opf ignore; the db is authoritative └── <data.name>.<format> one per row in `data` ``` The tables that matter, and nothing else: `books`, `authors` + `books_authors_link`, `publishers` + `books_publishers_link`, `tags` + `books_tags_link`, `series` + `books_series_link`, `languages` + `books_languages_link`, `comments`, `identifiers`, `data`, `books_pages_link`, `last_read_positions`. Ten things that will produce wrong data if you do not know them: - **Never query the views.** `meta`, `tag_browser_*` and friends call SQLite functions Calibre registers from Python at connection time. Verified: `SELECT * FROM meta` fails with `no such function: sortconcat`. Query base tables only. - **`pubdate` has a sentinel, not a null.** An unknown publication date is stored as `0101-01-01 00:00:00+00:00` (Calibre's `UNDEFINED_DATE`, year 101). It parses fine as a `date`, so nothing will complain — two of the six books in the reference library carry it. Drop any `pubdate` with year < 1000. The same sentinel appears in `timestamp`. - **`data.name` is lossy and is not the title.** It is the on-disk stem, truncated to Calibre's filename limit and sanitised. Verified in the reference library: the book titled `The Project Gutenberg eBook #33283: Calculus Made Easy, 2nd Edition` is stored as `The Project Gutenberg eBook #33283_ Calcul - Silvanus Phillips Thompson.pdf`. So the file extractors must not be consulted for metadata (see decision 2), and `books.path`/`data.name` are for *locating* files only. - **`books.title` may contain characters Calibre strips from its own paths** — `:` became `_` above, and titles legitimately contain `/` (`AC/DC`). `BookPathGenerator` interpolates the title straight into a path and only collapses repeated slashes (`services/filesystem_library.py`), so an unsanitised Calibre title can silently add a directory level. Sanitise `/` and control characters out of `title` before path generation. - **`authors.name` escapes commas as `|`.** Calibre's `AuthorsTable` unserialises with `name.replace('|', ',')` (from Calibre's `db/tables.py`; the reference library has no such name, so this one is unverified locally). Do the same replacement, and pass `authors.name` — **not** `authors.sort`, which is `Melville, Herman`. `format_author_name` would flip the sort form correctly anyway, but there is no reason to hand it the worse input. - **`series_index` is a REAL.** `7.0` must become `"7"`, not `"7.0"` — `Book.series_position` is a string, and `find_duplicate_books`'s series-position disqualifier compares it as one. - **`languages.lang_code` is ISO 639-2/B** (`eng`), while `EpubExtractor` stores raw `DC:language` (`en`). Both will coexist in the column. `Book.language` is free text and the edit form is a plain `<input>`, so nothing breaks; normalising to two letters is optional polish, not part of this work. - **`comments.text` is HTML.** `Book.description` is rendered as plain text by `CollapsibleText`, so `<p>` tags will show literally. Strip to text on import. - **`books_pages_link` is usually empty of real data.** It carries `needs_scan` and, in the reference library, `pages = 0` for all six books. Only use it when `pages > 0`. - **`identifiers.type` is free text.** The reference library holds `isbn`, `amazon` and `mobi-asin`, all of which `parse_identifier` already maps. Feed every identifier through it and keep whatever survives; do not filter to a known list. ## Decisions to settle before writing code 1. **Copy files into the library. Do not move, do not reference in place** — for the first version. Moving leaves `metadata.db` pointing at files that are gone, which quietly destroys a library the user still uses. Referencing in place is genuinely desirable (nobody wants two copies of 80 GB) but it points `book.path` at the Calibre tree, and `update_book` **moves directories** while `delete_books` **deletes files** — so a metadata edit in Chitai would rearrange somebody's Calibre library. `Library.read_only` exists but is enforced in exactly one place (`services/library.py:54`, at creation). See phase 3. 2. **Trust Calibre's metadata; do not run the extractors.** Calibre's catalogue is curated, its filenames are truncated garbage, and running `Extractor.extract_metadata` over thousands of files means opening every EPUB and rendering a cover page from every PDF. Take the cover from `cover.jpg` directly. The one exception worth allowing: fill `pages` from the file when Calibre has no useful value, behind a flag, off by default. 3. ~~**The source is a server-side path, not an upload.**~~ **Reversed in review, and the reason this brief was wrong is worth keeping.** The premise — "the library lives on the same host as the backend in every realistic deployment" — is false for the common case: Calibre is a desktop application, and its library is on the desktop. So the split is by *surface*, not by preference: - **Over HTTP: an uploaded zip only.** `…/imports/calibre/upload`. There is no endpoint taking a server path; one was built and then removed deliberately. - **On the server: a path only.** `litestar calibre-import <path>`, which is where a very large library or a headless migration belongs — an upload has to carry the whole archive across first. A CLI taking a path needs no justification. An *endpoint* taking one would have: it would let any authenticated caller read any directory the backend can, and `TODO.md` records there is no authorization tier at all. Not adding it is one less thing to gate later. 4. **Import into an existing Chitai library**, chosen by the caller. Creating a library is already one action, and the Calibre tree is rewritten by `BookPathGenerator` regardless. 5. **Re-running an import must be safe, and file-level dedupe already makes it so.** The same bytes are recognised by `(hash, size)` whatever their path, so a second run over the same library skips everything. No import bookkeeping is needed for idempotency. ## Prerequisite: a `.mobi` file breaks the book endpoint — **done** > Landed ahead of the import itself. `guess_content_type` in `services/utils.py` now names every > format from its extension, the column keeps a null when nothing can name one, and the OPDS feed > substitutes `application/octet-stream` at the one place a string is required. The Read control is > driven by `isReadable` rather than by the file count. See the section below for why it mattered, > and `backend/AGENTS.md` for the rule as it now stands. `FileMetadataRead.content_type` is a required `str` (`schemas/book.py:33`), but every ingest path fills it from `mimetypes.guess_type`, which returns `None` for `.mobi`, `.azw`, `.fb2`, `.lit` and `.htmlz` (verified). `FileMetadata.content_type` is nullable in the model, so the row stores fine and then fails response validation on the way out — a book whose only file is a MOBI would be unreadable through the API. Nothing in the tree hits this today because the browser upload path is used with EPUBs and PDFs. A Calibre library is full of MOBI and AZW3. Fix it first, either way round: - make the schema field `str | None`, and/or - add a small extension→MIME table for the ebook formats `mimetypes` does not know (`application/x-mobipocket-ebook`, `application/vnd.amazon.ebook`, `application/x-fictionbook+xml`). Do both, in fact: the table is the right answer for OPDS clients, which choose an acquisition link by MIME type, and the nullable field is the safety net. **Related, but not a blocker:** Chitai reads EPUB and PDF only. `openBookInReader` (`book/[bookId]/+page.svelte:66`) branches on `getFileType(...) === 'EPUB' | 'PDF'` and does nothing for anything else, so an AZW3-only book gets a Read button that silently fails. Importing those files is still right — they are downloadable and they are the user's — but the button should be disabled for a book with no readable file. One `$derived` on the page, worth doing in the same branch. ## Design ### 1. `services/calibre.py` — a pure reader, no Chitai types ```python @dataclass(frozen=True) class CalibreFile: path: Path # absolute, resolved against the library root format: str # "EPUB", as stored size: int # data.uncompressed_size, for a cheap sanity check @dataclass(frozen=True) class CalibreBook: calibre_id: int uuid: str title: str authors: list[str] ... # one field per row of the mapping table below cover: Path | None files: list[CalibreFile] class CalibreLibrary: def __init__(self, root: Path) -> None: ... async def open(self) -> None: ... # copy + connect, see below async def books(self) -> AsyncIterator[CalibreBook]: ... async def close(self) -> None: ... ``` Deliberately knows nothing about `Book`, `BookService` or the session — it is a file-format reader, unit-testable against a fixture database with no Postgres and no app. Two implementation notes: - **Copy `metadata.db` to a temp file and read the copy.** Calibre may be running and writing; opening the live file read-only either sees a torn state or needs the `-wal` sidecar. The database is small (438 KB for six books, single-digit MB for thousands), so a copy costs nothing and removes the whole problem. - **`sqlite3` inside `asyncio.to_thread`, not a new dependency.** The connection is used for a handful of queries. Do not add `aiosqlite` for this. Read the whole catalogue in **one query per table** and join in Python — six or so `SELECT`s and a few dicts, versus a per-book N+1 across ten tables. At self-hosted scale the entire catalogue minus descriptions fits in memory comfortably; if `comments.text` for 20k books is a concern, fetch that one table per batch. ### 2. The mapping | Calibre | Chitai | Notes | | --- | --- | --- | | `books.title` | `title` | Sanitise `/` and control chars for path generation. `Extractor.format_book_title` may still be worth applying to split a subtitle at the second colon — but **do not** run `split_edition`, Calibre's title is the curated one. | | `authors.name` via `books_authors_link` | `authors` | `\|` → `,`. Order by `books_authors_link.id`; `Book.author_links` is an `ordering_list`, so insertion order is the displayed order. | | `comments.text` | `description` | Strip HTML to text. | | `books.pubdate` | `published_date` | Drop the year-101 sentinel. | | `series.name`, `books.series_index` | `series`, `series_position` | `7.0` → `"7"`. | | `tags.name` | `tags` | | | `publishers.name` | `publisher` | `books_publishers_link` is unique per book. | | `languages.lang_code` (lowest `item_order`) | `language` | Chitai holds one. | | `identifiers.type` / `.val` | `identifiers` | Through `parse_identifier`; keep what survives. | | `books.uuid` | `identifiers["calibre-uuid"]` | The one durable link back to the source row. `normalize_identifier` returns a key for it (it is not in `_PER_BUILD_NAMES`), which is *desirable*: a book re-imported from the same Calibre library matches on it exactly. | | `books_pages_link.pages` | `pages` | Only when `> 0`. | | `cover.jpg` when `has_cover` | `cover_image` | | | `data` rows | `files` | | | `books.timestamp` | — | `Book.created_at` is audit-managed; do not fight it. | | `ratings`, `annotations`, `custom_columns` | — | No home in the model. Out of scope. | | `last_read_positions` | `BookProgress` | Phase 2. | ### 3. The ingest > **As built, this went on `BookService` as `create_many_from_calibre`, not into a separate > `services/calibre_import.py`.** The orchestration needs `_reserve_book_path`, > `_save_cover_image`, `_screen_for_duplicates` and `_record_possible_duplicates`, and reaching > into four privates from another module is worse than one more method in the file where the other > two ingest paths already live. `_record_possible_duplicates` was changed to take the list it > appends to rather than an `ImportResult`, so every ingest path can share it whatever its own > result type is. Two other deviations: `CalibreLibrary.books()` returns a list rather than an > async iterator, because the caller needs the total up front anyway; and the reader reports > identifiers exactly as Calibre keyed them, with the fold onto Chitai's schemes done by the > importer, which keeps the reader free of Chitai imports. Per book, in this order — it mirrors `create_many_from_existing_files`, which is the closest existing shape: 1. `fingerprint_file` each source file (`services/utils.py:164`). 2. `find_duplicate_files` against the target library. All files known → skip the book entirely, recording it. Some known → import the rest. 3. Build the metadata dict from the `CalibreBook`. 4. `_reserve_book_path(path_gen.generate_path(data))`. 5. **Copy** each file to `parent / _unused_path(...)`, building `FileMetadata` from the fingerprint already computed. `services/utils.py` has `move_file` but no copy — add `copy_file` beside it, streaming through `aiofiles` in `CHUNK_SIZE` blocks like `_save_book_files` does, not `shutil.copy` (a 40 MB blocking read inside the event loop). 6. Cover: open `cover.jpg` with PIL and hand the `Image` to `_save_cover_image`, which already accepts one and converts to WebP. 7. `super().create(data)` through `BookService`, then `find_duplicate_books` and record candidates — same as `_record_possible_duplicates` (`services/book.py:1253`). 8. **Commit per book.** A 5,000-book import inside one transaction is one failure away from nothing, and per-book commits are what lets the library page show books arriving — which is the behaviour commit `85367da` deliberately built. Report an `ImportResult`-shaped outcome; reuse `ImportResult` itself if it fits, extending it with a `failures: list[tuple[int, str]]` keyed by Calibre id. **One book must never fail the run** — a missing file, an unreadable cover or a `NOT NULL` violation gets recorded and skipped. ### 4. Progress, and where the import runs > **As built, the HTTP surface is an uploaded archive and nothing else** — see decision 3. > > `POST …/imports/calibre/upload` takes a zipped library. The job owns the temp directory it is > unpacked into and deletes it when it ends. Extraction refuses zip slip, an archive too big for the > disk, and one with no `metadata.db` within three levels — all answered 400 before a job exists. > > This forced a fix to the SvelteKit proxy, which buffered request bodies with `arrayBuffer()`: > survivable for one book, not for a multi-gigabyte archive. POST and PATCH now stream > `request.body` through with `duplex: 'half'`. > > **A preview endpoint was built and then removed with the path route.** It read a server-side > catalogue and reported its size before writing anything, which is only useful when the caller > named a directory. An upload has already been carried across by the time anything can be read, so > unpacking it *is* the validation step — an archive that is not a Calibre library is refused there. The import outlives its request, so the handler starts it and returns a handle: - `POST /libraries/{library_id:int}/imports/calibre` — body `{path, copy_files: true}`, returns `{job_id, total}`. 202. - `GET /libraries/imports/{job_id}` — `{state, total, processed, created, skipped, failed, current_title, errors}`. - `DELETE /libraries/imports/{job_id}` — cancel; the task checks a flag between books. Keep the registry **in memory**, a `dict[str, ImportJob]` on a module-level singleton, with the task created by `asyncio.create_task`. This matches what the app already does — the consume watcher is an in-process singleton started from a lifespan hook — and it is roughly thirty lines against a model, a migration and a service for the alternative. State that limitation explicitly in the docstring: **it assumes one worker process.** The production `CMD` is `litestar run`, which is single-process, so this holds today; `TODO.md` already records that the production image should move to uvicorn with a worker count, and doing that would mean a poll landing on a worker that has never heard of the job. The consume watcher has the same problem, so this is not a new constraint — but the next person to add workers needs to find it written down. If import *history* is ever wanted, that is when an `import_jobs` table earns its migration. **Also add a CLI entry point.** A 200 GB library imported through a browser tab that must stay open is a bad experience, and `pyproject.toml` already declares a `chitai` script. A Litestar CLI command (`litestar --app-dir src/chitai/ calibre-import <path> --library <slug>`) is ~20 lines over the same service and is the right tool for the initial migration, which is the case this whole feature exists for. The endpoint is for people who would rather click. ### 5. Frontend Model it on the duplicates screen, which is the closest precedent in shape and placement: - Route `(root)/settings/libraries/[libraryId]/import` — beside `settings/libraries/[libraryId]/duplicates`, reached from the library settings page. - `getCalibreImport` (a `query`) and `cancelCalibreImport` (a `command`) in `src/lib/api/calibre-import.remote.ts`, re-exported from `src/lib/api/index.ts`. **Starting an import is not a remote function**: the archive goes to the backend through the proxy so the browser streams straight through, where a remote function would put the whole thing through the SvelteKit process first. The screen uses `XMLHttpRequest` for it, which is the only way to get upload progress. - A file input, an upload progress bar, then a progress bar polling `getCalibreImport` every second or two, a running count, and the failures listed at the end with their Calibre ids. - Finish with a link to the library's duplicates screen. An import into a non-empty library is the single most likely way to produce duplicate books, and that screen already handles them. Do **not** route this through the upload tray. The tray reports on a client-driven queue it owns (`upload-queue.svelte.ts`); this is server-side work whose state survives a page reload, and conflating the two would mean teaching the tray to poll. Regenerate `src/lib/schema/openapi/schema.d.ts` against a backend running **your** branch — a stale server silently writes a stale file. ## Testing The fixture is the interesting part. Build a Calibre library in a `tmp_path` fixture rather than committing a binary `metadata.db`: a helper that executes the subset of Calibre's `CREATE TABLE` statements (they are in this document's shape, and in any real library's `sqlite_master`), inserts a handful of books, and lays out `<Author>/<Title> (id)/` directories containing the existing EPUB and PDF fixtures from `backend/tests/data_files/` plus a copy of `cover.jpg`. Generated beats committed here because the tests need to assert on *odd* rows — the pubdate sentinel, a `|` in an author name, a title with a colon — and those are clearer written in Python than hidden in a blob. - **Unit** (`tests/unit/test_calibre.py`) — the reader alone: field mapping; the year-101 pubdate dropped; `series_index` 7.0 → `"7"`; `|` unescaped in an author name; HTML stripped from `comments`; `pages = 0` ignored; identifiers passed through `parse_identifier`; a `data` row whose file is missing from disk reported rather than raised; `.caltrash` never walked. - **Service** (`tests/unit/test_services/test_calibre_import.py`) — a book with two formats lands as one record with two files; the source files still exist afterwards; a second run over the same library creates nothing; a library with one broken book imports the rest; authors and tags shared between two books produce one `Author` / `Tag` row each; `possible_duplicates` reported when the target library already holds the same book. - **Integration** (`tests/integration/test_calibre_import.py`) — `POST` returns 202 with a job id, polling reaches a terminal state, and the books are then listable through `GET /books`. A `.mobi`-only book must come back from `GET /books/{id}` without a 500 — that is the prerequisite's regression test. `pytest` needs Docker (`pytest-databases`). ## Verification ```bash nix-shell # postgres + migrations applied cd backend pytest tests/ # take your own baseline first uv run litestar --app-dir src/chitai/ run --port 8001 # for the OpenAPI regeneration cd ../frontend && pnpm check # baseline: 30 errors, 1 warning, 8 files ``` Neither `pnpm check` nor `pnpm lint` is clean on this repo — baseline before assuming an error is yours. End to end, against a real library (`~/Documents/Calibre Library` will do): import into an empty Chitai library and confirm the six books arrive with their covers, authors, tags, series positions and identifiers intact; that the AZW3 book is listed and downloadable; that the source library is byte-for-byte untouched (`diff -r` a copy taken beforehand); and that re-running the import creates nothing and reports six skipped books. Then import the same library into a library that already holds one of those books by upload, and confirm it lands on the duplicates screen rather than as a second copy. ## Phasing | Phase | Scope | | --- | --- | | **0** ✅ | The `content_type` prerequisite, plus the Read button. **Done** — see below. | | **1** ✅ | `services/calibre.py`, the ingest, the CLI command, copy-only. **Done** — a headless one-time migration works today. | | **2** ◐ | The upload endpoint, the job registry and the settings screen — **done**. The HTTP surface is an uploaded zip only; see decision 3, which this reversed. `last_read_positions` → `BookProgress` is **not** done: it needs a Calibre-user → Chitai-user mapping, and the answer differs between the endpoint (which has a `current_user`) and the CLI (which has none). That decision is the next thing to make. | | **3** | Reference-in-place import. Its real content is **enforcing `Library.read_only`** across `update_book`, `delete_books`, `add_files` and `remove_files` — which is a feature of its own and should not be smuggled in under an import. | ## Out of scope Annotations and highlights (no model to put them in), custom columns, ratings, virtual libraries and saved searches → bookshelves, format conversion, writing anything back to Calibre, and any form of continuing two-way sync. This is a one-way migration.