patrick/chitai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: import a Calibre library

Browse Source 
Reads metadata.db and copies the books into a library — from a zip uploaded on
the library settings page, or from a path with `litestar calibre-import`. The
source is never touched, and re-running only picks up what is new.

Also names the formats mimetypes does not know: a Calibre library is full of
MOBI and AZW3, and a null content type used to fail the book endpoint.
This commit is contained in:
patrick
2026-08-17 13:38:44 -04:00
parent 85367daf7e
commit 55e00ba960
34 changed files with 4723 additions and 50 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/calibre-import.md
+407
View File
@@ -0,0 +1,407 @@
# Implementation brief: importing a Calibre library
Written for an agent picking this up cold. Read the repo-root `AGENTS.md` and
`backend/AGENTS.md` first — this brief assumes both, particularly the **Filesystem behaviour**
and **Duplicate detection** sections.
Every claim about Calibre's schema and on-disk layout below was checked against a real library at
`~/Documents/Calibre Library` (6 books, current Calibre). Where a fact came from Calibre's source
rather than that library, it says so.
## Feasibility: high, and most of the machinery already exists
`metadata.db` is plain SQLite with a schema that has been stable for a decade, and the files sit
beside it in a predictable tree. Chitai already has every piece an import needs:
| Needed | Already in the tree |
| --- | --- |
| Ingest files that are already on disk | `BookService.create_many_from_existing_files` (`services/book.py:1280`) |
| Deduplicate authors/tags/publishers/series | `_populate_with_unique_relationships` (`services/book.py:1752`), via `as_unique_async` |
| Generic identifiers with a scheme map | `Identifier`, and `parse_identifier` (`services/metadata_extractor.py:58`) — whose map already covers `isbn`, `amazon`, `mobi-asin`, `google`, `goodreads`, `doi`, `calibre` |
| Decide where a book lives on disk | `BookPathGenerator`, `_reserve_book_path` (`services/book.py:1066`) |
| Not import the same book twice | `find_duplicate_files` (`:461`) and `find_duplicate_books` (`:535`) |
| Store a cover | `_save_cover_image` (`:1994`) |
So this is **a reader, not new ingest machinery**: turn Calibre rows into the metadata dict
`BookService` already accepts, and hand it to a slightly generalised version of the consume-directory
path. The parsing is the easy half.
The hard parts are elsewhere, and all three are addressed below:
1. Calibre libraries hold formats Chitai cannot describe, let alone read — and one of them
**currently 500s the book detail endpoint** (see prerequisites).
2. A 5,000-book import is a long-running job, and the app has no job/progress concept.
3. Whether files are **copied** into the library or **referenced in place** — which is a
product decision with a large blast radius, because in-place means Chitai's write paths point
at somebody's Calibre library.
## What a Calibre library actually is
```
Calibre Library/
├── metadata.db the whole catalogue
├── metadata_db_prefs_backup.json ignore
├── .caltrash/ .calnotes/ ignore — deleted books still live in .caltrash
└── <Author Name>/
└── <Title> (<book id>)/ == books.path
├── cover.jpg iff books.has_cover
├── metadata.opf ignore; the db is authoritative
└── <data.name>.<format> one per row in `data`
```
The tables that matter, and nothing else: `books`, `authors` + `books_authors_link`,
`publishers` + `books_publishers_link`, `tags` + `books_tags_link`, `series` +
`books_series_link`, `languages` + `books_languages_link`, `comments`, `identifiers`, `data`,
`books_pages_link`, `last_read_positions`.
Ten things that will produce wrong data if you do not know them:
- **Never query the views.** `meta`, `tag_browser_*` and friends call SQLite functions Calibre
registers from Python at connection time. Verified: `SELECT * FROM meta` fails with
`no such function: sortconcat`. Query base tables only.
- **`pubdate` has a sentinel, not a null.** An unknown publication date is stored as
`0101-01-01 00:00:00+00:00` (Calibre's `UNDEFINED_DATE`, year 101). It parses fine as a
`date`, so nothing will complain — two of the six books in the reference library carry it. Drop
any `pubdate` with year < 1000. The same sentinel appears in `timestamp`.
- **`data.name` is lossy and is not the title.** It is the on-disk stem, truncated to Calibre's
filename limit and sanitised. Verified in the reference library: the book titled
`The Project Gutenberg eBook #33283: Calculus Made Easy, 2nd Edition` is stored as
`The Project Gutenberg eBook #33283_ Calcul - Silvanus Phillips Thompson.pdf`. So the file
extractors must not be consulted for metadata (see decision 2), and `books.path`/`data.name`
are for *locating* files only.
- **`books.title` may contain characters Calibre strips from its own paths** — `:` became `_`
above, and titles legitimately contain `/` (`AC/DC`). `BookPathGenerator` interpolates the
title straight into a path and only collapses repeated slashes
(`services/filesystem_library.py`), so an unsanitised Calibre title can silently add a
directory level. Sanitise `/` and control characters out of `title` before path generation.
- **`authors.name` escapes commas as `|`.** Calibre's `AuthorsTable` unserialises with
`name.replace('|', ',')` (from Calibre's `db/tables.py`; the reference library has no such
name, so this one is unverified locally). Do the same replacement, and pass `authors.name` —
**not** `authors.sort`, which is `Melville, Herman`. `format_author_name` would flip the sort
form correctly anyway, but there is no reason to hand it the worse input.
- **`series_index` is a REAL.** `7.0` must become `"7"`, not `"7.0"` — `Book.series_position` is
a string, and `find_duplicate_books`'s series-position disqualifier compares it as one.
- **`languages.lang_code` is ISO 639-2/B** (`eng`), while `EpubExtractor` stores raw
`DC:language` (`en`). Both will coexist in the column. `Book.language` is free text and the
edit form is a plain `<input>`, so nothing breaks; normalising to two letters is optional
polish, not part of this work.
- **`comments.text` is HTML.** `Book.description` is rendered as plain text by
`CollapsibleText`, so `<p>` tags will show literally. Strip to text on import.
- **`books_pages_link` is usually empty of real data.** It carries `needs_scan` and, in the
reference library, `pages = 0` for all six books. Only use it when `pages > 0`.
- **`identifiers.type` is free text.** The reference library holds `isbn`, `amazon` and
`mobi-asin`, all of which `parse_identifier` already maps. Feed every identifier through it and
keep whatever survives; do not filter to a known list.
## Decisions to settle before writing code
1. **Copy files into the library. Do not move, do not reference in place** — for the first
version. Moving leaves `metadata.db` pointing at files that are gone, which quietly destroys a
library the user still uses. Referencing in place is genuinely desirable (nobody wants two
copies of 80 GB) but it points `book.path` at the Calibre tree, and `update_book` **moves
directories** while `delete_books` **deletes files** — so a metadata edit in Chitai would
rearrange somebody's Calibre library. `Library.read_only` exists but is enforced in exactly one
place (`services/library.py:54`, at creation). See phase 3.
2. **Trust Calibre's metadata; do not run the extractors.** Calibre's catalogue is curated, its
filenames are truncated garbage, and running `Extractor.extract_metadata` over thousands of
files means opening every EPUB and rendering a cover page from every PDF. Take the cover from
`cover.jpg` directly. The one exception worth allowing: fill `pages` from the file when
Calibre has no useful value, behind a flag, off by default.
3. ~~**The source is a server-side path, not an upload.**~~ **Reversed in review, and the reason
this brief was wrong is worth keeping.** The premise — "the library lives on the same host as
the backend in every realistic deployment" — is false for the common case: Calibre is a desktop
application, and its library is on the desktop. So the split is by *surface*, not by preference:
- **Over HTTP: an uploaded zip only.** `…/imports/calibre/upload`. There is no endpoint taking a
server path; one was built and then removed deliberately.
- **On the server: a path only.** `litestar calibre-import <path>`, which is where a very large
library or a headless migration belongs — an upload has to carry the whole archive across
first.
A CLI taking a path needs no justification. An *endpoint* taking one would have: it would let any
authenticated caller read any directory the backend can, and `TODO.md` records there is no
authorization tier at all. Not adding it is one less thing to gate later.
4. **Import into an existing Chitai library**, chosen by the caller. Creating a library is
already one action, and the Calibre tree is rewritten by `BookPathGenerator` regardless.
5. **Re-running an import must be safe, and file-level dedupe already makes it so.** The same
bytes are recognised by `(hash, size)` whatever their path, so a second run over the same
library skips everything. No import bookkeeping is needed for idempotency.
## Prerequisite: a `.mobi` file breaks the book endpoint — **done**
> Landed ahead of the import itself. `guess_content_type` in `services/utils.py` now names every
> format from its extension, the column keeps a null when nothing can name one, and the OPDS feed
> substitutes `application/octet-stream` at the one place a string is required. The Read control is
> driven by `isReadable` rather than by the file count. See the section below for why it mattered,
> and `backend/AGENTS.md` for the rule as it now stands.
`FileMetadataRead.content_type` is a required `str` (`schemas/book.py:33`), but every ingest path
fills it from `mimetypes.guess_type`, which returns `None` for `.mobi`, `.azw`, `.fb2`, `.lit`
and `.htmlz` (verified). `FileMetadata.content_type` is nullable in the model, so the row stores
fine and then fails response validation on the way out — a book whose only file is a MOBI would
be unreadable through the API.
Nothing in the tree hits this today because the browser upload path is used with EPUBs and PDFs.
A Calibre library is full of MOBI and AZW3. Fix it first, either way round:
- make the schema field `str | None`, and/or
- add a small extension→MIME table for the ebook formats `mimetypes` does not know
(`application/x-mobipocket-ebook`, `application/vnd.amazon.ebook`, `application/x-fictionbook+xml`).
Do both, in fact: the table is the right answer for OPDS clients, which choose an acquisition link
by MIME type, and the nullable field is the safety net.
**Related, but not a blocker:** Chitai reads EPUB and PDF only. `openBookInReader`
(`book/[bookId]/+page.svelte:66`) branches on `getFileType(...) === 'EPUB' | 'PDF'` and does
nothing for anything else, so an AZW3-only book gets a Read button that silently fails. Importing
those files is still right — they are downloadable and they are the user's — but the button
should be disabled for a book with no readable file. One `$derived` on the page, worth doing in
the same branch.
## Design
### 1. `services/calibre.py` — a pure reader, no Chitai types
```python
@dataclass(frozen=True)
class CalibreFile:
path: Path # absolute, resolved against the library root
format: str # "EPUB", as stored
size: int # data.uncompressed_size, for a cheap sanity check
@dataclass(frozen=True)
class CalibreBook:
calibre_id: int
uuid: str
title: str
authors: list[str]
... # one field per row of the mapping table below
cover: Path | None
files: list[CalibreFile]
class CalibreLibrary:
def __init__(self, root: Path) -> None: ...
async def open(self) -> None: ... # copy + connect, see below
async def books(self) -> AsyncIterator[CalibreBook]: ...
async def close(self) -> None: ...
```
Deliberately knows nothing about `Book`, `BookService` or the session — it is a file-format
reader, unit-testable against a fixture database with no Postgres and no app.
Two implementation notes:
- **Copy `metadata.db` to a temp file and read the copy.** Calibre may be running and writing;
opening the live file read-only either sees a torn state or needs the `-wal` sidecar. The
database is small (438 KB for six books, single-digit MB for thousands), so a copy costs
nothing and removes the whole problem.
- **`sqlite3` inside `asyncio.to_thread`, not a new dependency.** The connection is used for a
handful of queries. Do not add `aiosqlite` for this.
Read the whole catalogue in **one query per table** and join in Python — six or so `SELECT`s and
a few dicts, versus a per-book N+1 across ten tables. At self-hosted scale the entire catalogue
minus descriptions fits in memory comfortably; if `comments.text` for 20k books is a concern,
fetch that one table per batch.
### 2. The mapping
| Calibre | Chitai | Notes |
| --- | --- | --- |
| `books.title` | `title` | Sanitise `/` and control chars for path generation. `Extractor.format_book_title` may still be worth applying to split a subtitle at the second colon — but **do not** run `split_edition`, Calibre's title is the curated one. |
| `authors.name` via `books_authors_link` | `authors` | `\|` → `,`. Order by `books_authors_link.id`; `Book.author_links` is an `ordering_list`, so insertion order is the displayed order. |
| `comments.text` | `description` | Strip HTML to text. |
| `books.pubdate` | `published_date` | Drop the year-101 sentinel. |
| `series.name`, `books.series_index` | `series`, `series_position` | `7.0` → `"7"`. |
| `tags.name` | `tags` | |
| `publishers.name` | `publisher` | `books_publishers_link` is unique per book. |
| `languages.lang_code` (lowest `item_order`) | `language` | Chitai holds one. |
| `identifiers.type` / `.val` | `identifiers` | Through `parse_identifier`; keep what survives. |
| `books.uuid` | `identifiers["calibre-uuid"]` | The one durable link back to the source row. `normalize_identifier` returns a key for it (it is not in `_PER_BUILD_NAMES`), which is *desirable*: a book re-imported from the same Calibre library matches on it exactly. |
| `books_pages_link.pages` | `pages` | Only when `> 0`. |
| `cover.jpg` when `has_cover` | `cover_image` | |
| `data` rows | `files` | |
| `books.timestamp` | — | `Book.created_at` is audit-managed; do not fight it. |
| `ratings`, `annotations`, `custom_columns` | — | No home in the model. Out of scope. |
| `last_read_positions` | `BookProgress` | Phase 2. |
### 3. The ingest
> **As built, this went on `BookService` as `create_many_from_calibre`, not into a separate
> `services/calibre_import.py`.** The orchestration needs `_reserve_book_path`,
> `_save_cover_image`, `_screen_for_duplicates` and `_record_possible_duplicates`, and reaching
> into four privates from another module is worse than one more method in the file where the other
> two ingest paths already live. `_record_possible_duplicates` was changed to take the list it
> appends to rather than an `ImportResult`, so every ingest path can share it whatever its own
> result type is. Two other deviations: `CalibreLibrary.books()` returns a list rather than an
> async iterator, because the caller needs the total up front anyway; and the reader reports
> identifiers exactly as Calibre keyed them, with the fold onto Chitai's schemes done by the
> importer, which keeps the reader free of Chitai imports.
Per book, in this order — it mirrors `create_many_from_existing_files`, which is the closest
existing shape:
1. `fingerprint_file` each source file (`services/utils.py:164`).
2. `find_duplicate_files` against the target library. All files known → skip the book entirely,
recording it. Some known → import the rest.
3. Build the metadata dict from the `CalibreBook`.
4. `_reserve_book_path(path_gen.generate_path(data))`.
5. **Copy** each file to `parent / _unused_path(...)`, building `FileMetadata` from the
fingerprint already computed. `services/utils.py` has `move_file` but no copy — add
`copy_file` beside it, streaming through `aiofiles` in `CHUNK_SIZE` blocks like
`_save_book_files` does, not `shutil.copy` (a 40 MB blocking read inside the event loop).
6. Cover: open `cover.jpg` with PIL and hand the `Image` to `_save_cover_image`, which already
accepts one and converts to WebP.
7. `super().create(data)` through `BookService`, then `find_duplicate_books` and record
candidates — same as `_record_possible_duplicates` (`services/book.py:1253`).
8. **Commit per book.** A 5,000-book import inside one transaction is one failure away from
nothing, and per-book commits are what lets the library page show books arriving — which is
the behaviour commit `85367da` deliberately built.
Report an `ImportResult`-shaped outcome; reuse `ImportResult` itself if it fits, extending it with
a `failures: list[tuple[int, str]]` keyed by Calibre id. **One book must never fail the run** —
a missing file, an unreadable cover or a `NOT NULL` violation gets recorded and skipped.
### 4. Progress, and where the import runs
> **As built, the HTTP surface is an uploaded archive and nothing else** — see decision 3.
>
> `POST …/imports/calibre/upload` takes a zipped library. The job owns the temp directory it is
> unpacked into and deletes it when it ends. Extraction refuses zip slip, an archive too big for the
> disk, and one with no `metadata.db` within three levels — all answered 400 before a job exists.
>
> This forced a fix to the SvelteKit proxy, which buffered request bodies with `arrayBuffer()`:
> survivable for one book, not for a multi-gigabyte archive. POST and PATCH now stream
> `request.body` through with `duplex: 'half'`.
>
> **A preview endpoint was built and then removed with the path route.** It read a server-side
> catalogue and reported its size before writing anything, which is only useful when the caller
> named a directory. An upload has already been carried across by the time anything can be read, so
> unpacking it *is* the validation step — an archive that is not a Calibre library is refused there.
The import outlives its request, so the handler starts it and returns a handle:
- `POST /libraries/{library_id:int}/imports/calibre` — body `{path, copy_files: true}`, returns
`{job_id, total}`. 202.
- `GET /libraries/imports/{job_id}` — `{state, total, processed, created, skipped, failed,
current_title, errors}`.
- `DELETE /libraries/imports/{job_id}` — cancel; the task checks a flag between books.
Keep the registry **in memory**, a `dict[str, ImportJob]` on a module-level singleton, with the
task created by `asyncio.create_task`. This matches what the app already does — the consume
watcher is an in-process singleton started from a lifespan hook — and it is roughly thirty lines
against a model, a migration and a service for the alternative.
State that limitation explicitly in the docstring: **it assumes one worker process.** The
production `CMD` is `litestar run`, which is single-process, so this holds today; `TODO.md`
already records that the production image should move to uvicorn with a worker count, and doing
that would mean a poll landing on a worker that has never heard of the job. The consume watcher
has the same problem, so this is not a new constraint — but the next person to add workers needs
to find it written down. If import *history* is ever wanted, that is when an `import_jobs` table
earns its migration.
**Also add a CLI entry point.** A 200 GB library imported through a browser tab that must stay
open is a bad experience, and `pyproject.toml` already declares a `chitai` script. A Litestar CLI
command (`litestar --app-dir src/chitai/ calibre-import <path> --library <slug>`) is ~20 lines
over the same service and is the right tool for the initial migration, which is the case this
whole feature exists for. The endpoint is for people who would rather click.
### 5. Frontend
Model it on the duplicates screen, which is the closest precedent in shape and placement:
- Route `(root)/settings/libraries/[libraryId]/import` — beside
`settings/libraries/[libraryId]/duplicates`, reached from the library settings page.
- `getCalibreImport` (a `query`) and `cancelCalibreImport` (a `command`) in
`src/lib/api/calibre-import.remote.ts`, re-exported from `src/lib/api/index.ts`. **Starting an
import is not a remote function**: the archive goes to the backend through the proxy so the
browser streams straight through, where a remote function would put the whole thing through the
SvelteKit process first. The screen uses `XMLHttpRequest` for it, which is the only way to get
upload progress.
- A file input, an upload progress bar, then a progress bar polling `getCalibreImport` every second
or two, a running count, and the failures listed at the end with their Calibre ids.
- Finish with a link to the library's duplicates screen. An import into a non-empty library is
the single most likely way to produce duplicate books, and that screen already handles them.
Do **not** route this through the upload tray. The tray reports on a client-driven queue it owns
(`upload-queue.svelte.ts`); this is server-side work whose state survives a page reload, and
conflating the two would mean teaching the tray to poll.
Regenerate `src/lib/schema/openapi/schema.d.ts` against a backend running **your** branch —
a stale server silently writes a stale file.
## Testing
The fixture is the interesting part. Build a Calibre library in a `tmp_path` fixture rather than
committing a binary `metadata.db`: a helper that executes the subset of Calibre's `CREATE TABLE`
statements (they are in this document's shape, and in any real library's `sqlite_master`), inserts
a handful of books, and lays out `<Author>/<Title> (id)/` directories containing the existing
EPUB and PDF fixtures from `backend/tests/data_files/` plus a copy of `cover.jpg`. Generated
beats committed here because the tests need to assert on *odd* rows — the pubdate sentinel, a
`|` in an author name, a title with a colon — and those are clearer written in Python than hidden
in a blob.
- **Unit** (`tests/unit/test_calibre.py`) — the reader alone: field mapping; the year-101 pubdate
dropped; `series_index` 7.0 → `"7"`; `|` unescaped in an author name; HTML stripped from
`comments`; `pages = 0` ignored; identifiers passed through `parse_identifier`; a `data` row
whose file is missing from disk reported rather than raised; `.caltrash` never walked.
- **Service** (`tests/unit/test_services/test_calibre_import.py`) — a book with two formats lands
as one record with two files; the source files still exist afterwards; a second run over the
same library creates nothing; a library with one broken book imports the rest; authors and tags
shared between two books produce one `Author` / `Tag` row each; `possible_duplicates` reported
when the target library already holds the same book.
- **Integration** (`tests/integration/test_calibre_import.py`) — `POST` returns 202 with a job id,
polling reaches a terminal state, and the books are then listable through `GET /books`. A
`.mobi`-only book must come back from `GET /books/{id}` without a 500 — that is the
prerequisite's regression test.
`pytest` needs Docker (`pytest-databases`).
## Verification
```bash
nix-shell # postgres + migrations applied
cd backend
pytest tests/ # take your own baseline first
uv run litestar --app-dir src/chitai/ run --port 8001 # for the OpenAPI regeneration
cd ../frontend && pnpm check # baseline: 30 errors, 1 warning, 8 files
```
Neither `pnpm check` nor `pnpm lint` is clean on this repo — baseline before assuming an error is
yours.
End to end, against a real library (`~/Documents/Calibre Library` will do): import into an empty
Chitai library and confirm the six books arrive with their covers, authors, tags, series
positions and identifiers intact; that the AZW3 book is listed and downloadable; that the source
library is byte-for-byte untouched (`diff -r` a copy taken beforehand); and that re-running the
import creates nothing and reports six skipped books. Then import the same library into a
library that already holds one of those books by upload, and confirm it lands on the duplicates
screen rather than as a second copy.
## Phasing
| Phase | Scope |
| --- | --- |
| **0** ✅ | The `content_type` prerequisite, plus the Read button. **Done** — see below. |
| **1** ✅ | `services/calibre.py`, the ingest, the CLI command, copy-only. **Done** — a headless one-time migration works today. |
| **2** ◐ | The upload endpoint, the job registry and the settings screen — **done**. The HTTP surface is an uploaded zip only; see decision 3, which this reversed. `last_read_positions` → `BookProgress` is **not** done: it needs a Calibre-user → Chitai-user mapping, and the answer differs between the endpoint (which has a `current_user`) and the CLI (which has none). That decision is the next thing to make. |
| **3** | Reference-in-place import. Its real content is **enforcing `Library.read_only`** across `update_book`, `delete_books`, `add_files` and `remove_files` — which is a feature of its own and should not be smuggled in under an import. |
## Out of scope
Annotations and highlights (no model to put them in), custom columns, ratings, virtual libraries
and saved searches → bookshelves, format conversion, writing anything back to Calibre, and any
form of continuing two-way sync. This is a one-way migration.