chitai/docs/calibre-import.md

Implementation brief: importing a Calibre library

Written for an agent picking this up cold. Read the repo-root AGENTS.md and backend/AGENTS.md first — this brief assumes both, particularly the Filesystem behaviour and Duplicate detection sections.

Every claim about Calibre's schema and on-disk layout below was checked against a real library at ~/Documents/Calibre Library (6 books, current Calibre). Where a fact came from Calibre's source rather than that library, it says so.

Feasibility: high, and most of the machinery already exists

metadata.db is plain SQLite with a schema that has been stable for a decade, and the files sit beside it in a predictable tree. Chitai already has every piece an import needs:

Needed	Already in the tree
Ingest files that are already on disk	BookService.create_many_from_existing_files (services/book.py:1280)
Deduplicate authors/tags/publishers/series	_populate_with_unique_relationships (services/book.py:1752), via as_unique_async
Generic identifiers with a scheme map	Identifier, and parse_identifier (services/metadata_extractor.py:58) — whose map already covers isbn, amazon, mobi-asin, google, goodreads, doi, calibre
Decide where a book lives on disk	BookPathGenerator, _reserve_book_path (services/book.py:1066)
Not import the same book twice	find_duplicate_files (:461) and find_duplicate_books (:535)
Store a cover	_save_cover_image (:1994)

So this is a reader, not new ingest machinery: turn Calibre rows into the metadata dict BookService already accepts, and hand it to a slightly generalised version of the consume-directory path. The parsing is the easy half.

The hard parts are elsewhere, and all three are addressed below:

1. Calibre libraries hold formats Chitai cannot describe, let alone read — and one of them currently 500s the book detail endpoint (see prerequisites).
2. A 5,000-book import is a long-running job, and the app has no job/progress concept.
3. Whether files are copied into the library or referenced in place — which is a product decision with a large blast radius, because in-place means Chitai's write paths point at somebody's Calibre library.

What a Calibre library actually is

Calibre Library/
├── metadata.db                       the whole catalogue
├── metadata_db_prefs_backup.json     ignore
├── .caltrash/  .calnotes/            ignore — deleted books still live in .caltrash
└── <Author Name>/
    └── <Title> (<book id>)/          == books.path
        ├── cover.jpg                 iff books.has_cover
        ├── metadata.opf              ignore; the db is authoritative
        └── <data.name>.<format>      one per row in `data`

The tables that matter, and nothing else: books, authors + books_authors_link, publishers + books_publishers_link, tags + books_tags_link, series + books_series_link, languages + books_languages_link, comments, identifiers, data, books_pages_link, last_read_positions.

Ten things that will produce wrong data if you do not know them:

• Never query the views. meta, tag_browser_* and friends call SQLite functions Calibre registers from Python at connection time. Verified: SELECT * FROM meta fails with no such function: sortconcat. Query base tables only.

• pubdate has a sentinel, not a null. An unknown publication date is stored as 0101-01-01 00:00:00+00:00 (Calibre's UNDEFINED_DATE, year 101). It parses fine as a date, so nothing will complain — two of the six books in the reference library carry it. Drop any pubdate with year < 1000. The same sentinel appears in timestamp.

• data.name is lossy and is not the title. It is the on-disk stem, truncated to Calibre's filename limit and sanitised. Verified in the reference library: the book titled The Project Gutenberg eBook #33283: Calculus Made Easy, 2nd Edition is stored as The Project Gutenberg eBook #33283_ Calcul - Silvanus Phillips Thompson.pdf. So the file extractors must not be consulted for metadata (see decision 2), and books.path/data.name are for locating files only.

• books.title may contain characters Calibre strips from its own paths — : became _ above, and titles legitimately contain / (AC/DC). BookPathGenerator interpolates the title straight into a path and only collapses repeated slashes (services/filesystem_library.py), so an unsanitised Calibre title can silently add a directory level. Sanitise / and control characters out of title before path generation.

• authors.name escapes commas as |. Calibre's AuthorsTable unserialises with name.replace('|', ',') (from Calibre's db/tables.py; the reference library has no such name, so this one is unverified locally). Do the same replacement, and pass authors.name — not authors.sort, which is Melville, Herman. format_author_name would flip the sort form correctly anyway, but there is no reason to hand it the worse input.

• series_index is a REAL. 7.0 must become "7", not "7.0" — Book.series_position is a string, and find_duplicate_books's series-position disqualifier compares it as one.

• languages.lang_code is ISO 639-2/B (eng), while EpubExtractor stores raw DC:language (en). Both will coexist in the column. Book.language is free text and the edit form is a plain <input>, so nothing breaks; normalising to two letters is optional polish, not part of this work.

• comments.text is HTML. Book.description is rendered as plain text by CollapsibleText, so <p> tags will show literally. Strip to text on import.

• books_pages_link is usually empty of real data. It carries needs_scan and, in the reference library, pages = 0 for all six books. Only use it when pages > 0.

• identifiers.type is free text. The reference library holds isbn, amazon and mobi-asin, all of which parse_identifier already maps. Feed every identifier through it and keep whatever survives; do not filter to a known list.

Decisions to settle before writing code

1. Copy files into the library. Do not move, do not reference in place — for the first version. Moving leaves metadata.db pointing at files that are gone, which quietly destroys a library the user still uses. Referencing in place is genuinely desirable (nobody wants two copies of 80 GB) but it points book.path at the Calibre tree, and update_book moves directories while delete_books deletes files — so a metadata edit in Chitai would rearrange somebody's Calibre library. Library.read_only exists but is enforced in exactly one place (services/library.py:54, at creation). See phase 3.

2. Trust Calibre's metadata; do not run the extractors. Calibre's catalogue is curated, its filenames are truncated garbage, and running Extractor.extract_metadata over thousands of files means opening every EPUB and rendering a cover page from every PDF. Take the cover from cover.jpg directly. The one exception worth allowing: fill pages from the file when Calibre has no useful value, behind a flag, off by default.

3. The source is a server-side path, not an upload. Reversed in review, and the reason this brief was wrong is worth keeping. The premise — "the library lives on the same host as the backend in every realistic deployment" — is false for the common case: Calibre is a desktop application, and its library is on the desktop. So the split is by surface, not by preference:

   • Over HTTP: an uploaded zip only. …/imports/calibre/upload. There is no endpoint taking a server path; one was built and then removed deliberately.
   • On the server: a path only. litestar calibre-import <path>, which is where a very large library or a headless migration belongs — an upload has to carry the whole archive across first.

   A CLI taking a path needs no justification. An endpoint taking one would have: it would let any authenticated caller read any directory the backend can, and TODO.md records there is no authorization tier at all. Not adding it is one less thing to gate later.

4. Import into an existing Chitai library, chosen by the caller. Creating a library is already one action, and the Calibre tree is rewritten by BookPathGenerator regardless.

5. Re-running an import must be safe, and file-level dedupe already makes it so. The same bytes are recognised by (hash, size) whatever their path, so a second run over the same library skips everything. No import bookkeeping is needed for idempotency.

Prerequisite: a .mobi file breaks the book endpoint — done

Landed ahead of the import itself. guess_content_type in services/utils.py now names every format from its extension, the column keeps a null when nothing can name one, and the OPDS feed substitutes application/octet-stream at the one place a string is required. The Read control is driven by isReadable rather than by the file count. See the section below for why it mattered, and backend/AGENTS.md for the rule as it now stands.

FileMetadataRead.content_type is a required str (schemas/book.py:33), but every ingest path fills it from mimetypes.guess_type, which returns None for .mobi, .azw, .fb2, .lit and .htmlz (verified). FileMetadata.content_type is nullable in the model, so the row stores fine and then fails response validation on the way out — a book whose only file is a MOBI would be unreadable through the API.

Nothing in the tree hits this today because the browser upload path is used with EPUBs and PDFs. A Calibre library is full of MOBI and AZW3. Fix it first, either way round:

• make the schema field str | None, and/or
• add a small extension→MIME table for the ebook formats mimetypes does not know (application/x-mobipocket-ebook, application/vnd.amazon.ebook, application/x-fictionbook+xml).

Do both, in fact: the table is the right answer for OPDS clients, which choose an acquisition link by MIME type, and the nullable field is the safety net.

Related, but not a blocker: Chitai reads EPUB and PDF only. openBookInReader (book/[bookId]/+page.svelte:66) branches on getFileType(...) === 'EPUB' | 'PDF' and does nothing for anything else, so an AZW3-only book gets a Read button that silently fails. Importing those files is still right — they are downloadable and they are the user's — but the button should be disabled for a book with no readable file. One $derived on the page, worth doing in the same branch.

Design

1. services/calibre.py — a pure reader, no Chitai types

@dataclass(frozen=True)
class CalibreFile:
    path: Path          # absolute, resolved against the library root
    format: str         # "EPUB", as stored
    size: int           # data.uncompressed_size, for a cheap sanity check

@dataclass(frozen=True)
class CalibreBook:
    calibre_id: int
    uuid: str
    title: str
    authors: list[str]
    ...                 # one field per row of the mapping table below
    cover: Path | None
    files: list[CalibreFile]

class CalibreLibrary:
    def __init__(self, root: Path) -> None: ...
    async def open(self) -> None: ...          # copy + connect, see below
    async def books(self) -> AsyncIterator[CalibreBook]: ...
    async def close(self) -> None: ...

Deliberately knows nothing about Book, BookService or the session — it is a file-format reader, unit-testable against a fixture database with no Postgres and no app.

Two implementation notes:

• Copy metadata.db to a temp file and read the copy. Calibre may be running and writing; opening the live file read-only either sees a torn state or needs the -wal sidecar. The database is small (438 KB for six books, single-digit MB for thousands), so a copy costs nothing and removes the whole problem.
• sqlite3 inside asyncio.to_thread, not a new dependency. The connection is used for a handful of queries. Do not add aiosqlite for this.

Read the whole catalogue in one query per table and join in Python — six or so SELECTs and a few dicts, versus a per-book N+1 across ten tables. At self-hosted scale the entire catalogue minus descriptions fits in memory comfortably; if comments.text for 20k books is a concern, fetch that one table per batch.

2. The mapping

Calibre	Chitai	Notes
books.title	title	Sanitise / and control chars for path generation. Extractor.format_book_title may still be worth applying to split a subtitle at the second colon — but do not run split_edition, Calibre's title is the curated one.
authors.name via books_authors_link	authors	| → ,. Order by books_authors_link.id; Book.author_links is an ordering_list, so insertion order is the displayed order.
comments.text	description	Strip HTML to text.
books.pubdate	published_date	Drop the year-101 sentinel.
series.name, books.series_index	series, series_position	7.0 → "7".
tags.name	tags
publishers.name	publisher	books_publishers_link is unique per book.
languages.lang_code (lowest item_order)	language	Chitai holds one.
identifiers.type / .val	identifiers	Through parse_identifier; keep what survives.
books.uuid	identifiers["calibre-uuid"]	The one durable link back to the source row. normalize_identifier returns a key for it (it is not in _PER_BUILD_NAMES), which is desirable: a book re-imported from the same Calibre library matches on it exactly.
books_pages_link.pages	pages	Only when > 0.
cover.jpg when has_cover	cover_image
data rows	files
books.timestamp	—	Book.created_at is audit-managed; do not fight it.
ratings, annotations, custom_columns	—	No home in the model. Out of scope.
last_read_positions	BookProgress	Phase 2.

3. The ingest

As built, this went on BookService as create_many_from_calibre, not into a separate services/calibre_import.py. The orchestration needs _reserve_book_path, _save_cover_image, _screen_for_duplicates and _record_possible_duplicates, and reaching into four privates from another module is worse than one more method in the file where the other two ingest paths already live. _record_possible_duplicates was changed to take the list it appends to rather than an ImportResult, so every ingest path can share it whatever its own result type is. Two other deviations: CalibreLibrary.books() returns a list rather than an async iterator, because the caller needs the total up front anyway; and the reader reports identifiers exactly as Calibre keyed them, with the fold onto Chitai's schemes done by the importer, which keeps the reader free of Chitai imports.

Per book, in this order — it mirrors create_many_from_existing_files, which is the closest existing shape:

1. fingerprint_file each source file (services/utils.py:164).
2. find_duplicate_files against the target library. All files known → skip the book entirely, recording it. Some known → import the rest.
3. Build the metadata dict from the CalibreBook.
4. _reserve_book_path(path_gen.generate_path(data)).
5. Copy each file to parent / _unused_path(...), building FileMetadata from the fingerprint already computed. services/utils.py has move_file but no copy — add copy_file beside it, streaming through aiofiles in CHUNK_SIZE blocks like _save_book_files does, not shutil.copy (a 40 MB blocking read inside the event loop).
6. Cover: open cover.jpg with PIL and hand the Image to _save_cover_image, which already accepts one and converts to WebP.
7. super().create(data) through BookService, then find_duplicate_books and record candidates — same as _record_possible_duplicates (services/book.py:1253).
8. Commit per book. A 5,000-book import inside one transaction is one failure away from nothing, and per-book commits are what lets the library page show books arriving — which is the behaviour commit 85367da deliberately built.

Report an ImportResult-shaped outcome; reuse ImportResult itself if it fits, extending it with a failures: list[tuple[int, str]] keyed by Calibre id. One book must never fail the run — a missing file, an unreadable cover or a NOT NULL violation gets recorded and skipped.

4. Progress, and where the import runs

As built, the HTTP surface is an uploaded archive and nothing else — see decision 3.

POST …/imports/calibre/upload takes a zipped library. The job owns the temp directory it is unpacked into and deletes it when it ends. Extraction refuses zip slip, an archive too big for the disk, and one with no metadata.db within three levels — all answered 400 before a job exists.

This forced a fix to the SvelteKit proxy, which buffered request bodies with arrayBuffer(): survivable for one book, not for a multi-gigabyte archive. POST and PATCH now stream request.body through with duplex: 'half'.

A preview endpoint was built and then removed with the path route. It read a server-side catalogue and reported its size before writing anything, which is only useful when the caller named a directory. An upload has already been carried across by the time anything can be read, so unpacking it is the validation step — an archive that is not a Calibre library is refused there.

The import outlives its request, so the handler starts it and returns a handle:

• POST /libraries/{library_id:int}/imports/calibre — body {path, copy_files: true}, returns {job_id, total}. 202.
• GET /libraries/imports/{job_id} — {state, total, processed, created, skipped, failed, current_title, errors}.
• DELETE /libraries/imports/{job_id} — cancel; the task checks a flag between books.

Keep the registry in memory, a dict[str, ImportJob] on a module-level singleton, with the task created by asyncio.create_task. This matches what the app already does — the consume watcher is an in-process singleton started from a lifespan hook — and it is roughly thirty lines against a model, a migration and a service for the alternative.

State that limitation explicitly in the docstring: it assumes one worker process. The production CMD is litestar run, which is single-process, so this holds today; TODO.md already records that the production image should move to uvicorn with a worker count, and doing that would mean a poll landing on a worker that has never heard of the job. The consume watcher has the same problem, so this is not a new constraint — but the next person to add workers needs to find it written down. If import history is ever wanted, that is when an import_jobs table earns its migration.

Also add a CLI entry point. A 200 GB library imported through a browser tab that must stay open is a bad experience, and pyproject.toml already declares a chitai script. A Litestar CLI command (litestar --app-dir src/chitai/ calibre-import <path> --library <slug>) is ~20 lines over the same service and is the right tool for the initial migration, which is the case this whole feature exists for. The endpoint is for people who would rather click.

5. Frontend

Model it on the duplicates screen, which is the closest precedent in shape and placement:

• Route (root)/settings/libraries/[libraryId]/import — beside settings/libraries/[libraryId]/duplicates, reached from the library settings page.
• getCalibreImport (a query) and cancelCalibreImport (a command) in src/lib/api/calibre-import.remote.ts, re-exported from src/lib/api/index.ts. Starting an import is not a remote function: the archive goes to the backend through the proxy so the browser streams straight through, where a remote function would put the whole thing through the SvelteKit process first. The screen uses XMLHttpRequest for it, which is the only way to get upload progress.
• A file input, an upload progress bar, then a progress bar polling getCalibreImport every second or two, a running count, and the failures listed at the end with their Calibre ids.
• Finish with a link to the library's duplicates screen. An import into a non-empty library is the single most likely way to produce duplicate books, and that screen already handles them.

Do not route this through the upload tray. The tray reports on a client-driven queue it owns (upload-queue.svelte.ts); this is server-side work whose state survives a page reload, and conflating the two would mean teaching the tray to poll.

Regenerate src/lib/schema/openapi/schema.d.ts against a backend running your branch — a stale server silently writes a stale file.

Testing

The fixture is the interesting part. Build a Calibre library in a tmp_path fixture rather than committing a binary metadata.db: a helper that executes the subset of Calibre's CREATE TABLE statements (they are in this document's shape, and in any real library's sqlite_master), inserts a handful of books, and lays out <Author>/<Title> (id)/ directories containing the existing EPUB and PDF fixtures from backend/tests/data_files/ plus a copy of cover.jpg. Generated beats committed here because the tests need to assert on odd rows — the pubdate sentinel, a | in an author name, a title with a colon — and those are clearer written in Python than hidden in a blob.

• Unit (tests/unit/test_calibre.py) — the reader alone: field mapping; the year-101 pubdate dropped; series_index 7.0 → "7"; | unescaped in an author name; HTML stripped from comments; pages = 0 ignored; identifiers passed through parse_identifier; a data row whose file is missing from disk reported rather than raised; .caltrash never walked.
• Service (tests/unit/test_services/test_calibre_import.py) — a book with two formats lands as one record with two files; the source files still exist afterwards; a second run over the same library creates nothing; a library with one broken book imports the rest; authors and tags shared between two books produce one Author / Tag row each; possible_duplicates reported when the target library already holds the same book.
• Integration (tests/integration/test_calibre_import.py) — POST returns 202 with a job id, polling reaches a terminal state, and the books are then listable through GET /books. A .mobi-only book must come back from GET /books/{id} without a 500 — that is the prerequisite's regression test.

pytest needs Docker (pytest-databases).

Verification

nix-shell                                                    # postgres + migrations applied
cd backend
pytest tests/                                                # take your own baseline first
uv run litestar --app-dir src/chitai/ run --port 8001        # for the OpenAPI regeneration
cd ../frontend && pnpm check                                 # baseline: 30 errors, 1 warning, 8 files

Neither pnpm check nor pnpm lint is clean on this repo — baseline before assuming an error is yours.

End to end, against a real library (~/Documents/Calibre Library will do): import into an empty Chitai library and confirm the six books arrive with their covers, authors, tags, series positions and identifiers intact; that the AZW3 book is listed and downloadable; that the source library is byte-for-byte untouched (diff -r a copy taken beforehand); and that re-running the import creates nothing and reports six skipped books. Then import the same library into a library that already holds one of those books by upload, and confirm it lands on the duplicates screen rather than as a second copy.

Phasing

Phase	Scope
0 ✅	The content_type prerequisite, plus the Read button. Done — see below.
1 ✅	services/calibre.py, the ingest, the CLI command, copy-only. Done — a headless one-time migration works today.
2 ◐	The upload endpoint, the job registry and the settings screen — done. The HTTP surface is an uploaded zip only; see decision 3, which this reversed. last_read_positions → BookProgress is not done: it needs a Calibre-user → Chitai-user mapping, and the answer differs between the endpoint (which has a current_user) and the CLI (which has none). That decision is the next thing to make.
3	Reference-in-place import. Its real content is enforcing Library.read_only across update_book, delete_books, add_files and remove_files — which is a feature of its own and should not be smuggled in under an import.

Out of scope

Annotations and highlights (no model to put them in), custom columns, ratings, virtual libraries and saved searches → bookshelves, format conversion, writing anything back to Calibre, and any form of continuing two-way sync. This is a one-way migration.