Compare commits

..
25 Commits
Author SHA1 Message Date
patrick 7e33a8fe05 Merge branch 'duplicate-detection' 2026-08-17 11:06:23 -04:00
patrick 202cbed30e refactor: name the merge options and drop their subtext 2026-08-17 11:06:19 -04:00
patrick a3443d36f1 feat: offer a merge that keeps the first book, without the workbench 2026-08-17 10:35:35 -04:00
patrick 75fe41e266 fix: reopen the merge dialog after dismissing it 2026-08-17 10:35:24 -04:00
patrick d16a90f08f feat: draw generated covers where a book has none 2026-08-17 10:35:01 -04:00
patrick 904d8fc76f refactor: move duplicates into library settings 2026-08-17 10:32:57 -04:00
patrick 315149e8a2 docs: brief for moving duplicates into library settings
Option B — libraries expand in the settings nav, duplicates becomes a section
within a library's pane.
2026-08-16 20:21:25 -04:00
patrick 699a1a7fa2 feat: merge books from the duplicates screen and the toolbar
A workbench with the folded records in a rail, the survivor's fields live and
editable beside them, and per-field actions chosen by what kind of field it is.
Fields the records agree on stay out of the way. Reachable from a duplicate
group and from the selection toolbar, which is the only way to merge a pair the
detector never proposed.
2026-08-16 20:21:19 -04:00
patrick 22539644a9 feat: merge books into one record
Files move on disk before any row moves, then everything repoints in one
transaction and the folded records are deleted. Progress keeps whichever is
furthest, shelves and tags union, and identifiers move only under a name the
survivor lacks. Metadata is left alone unless the caller resolves it, since
choosing between two titles is a judgement this endpoint cannot make. Nothing
is removed from disk.
2026-08-16 20:21:10 -04:00
patrick a48be517e4 fix: type the ids filter from its configured id type
advanced_alchemy annotates the `ids` query parameter as list[str] whatever the
config says, so a bigint primary key was compared against strings and Postgres
refused. Nothing had called ?ids= until now.
2026-08-16 20:20:44 -04:00
patrick 2e0d556c33 fix: return the publisher an EPUB declares
The lookup discarded its own result and fell off the end of the function, so
no EPUB ever contributed a publisher.
2026-08-16 00:01:02 -04:00
patrick ff75f2c758 chore: clear the lint in the files this branch touches
Unused imports, duplicated import lines, and bare excepts that swallowed
KeyboardInterrupt along with everything else.
2026-08-15 22:01:04 -04:00
patrick e967019964 feat: split the edition out of a book's title
"Fluent Python, 2nd Edition" is one book with a field for the edition. Left in
the title it also splits the library, since the second edition never looks like
the first. Only a numbered statement is moved, so "Catch 22" keeps its number
and "Global Edition" — which has nowhere to go in an integer column — stays put.
2026-08-15 21:57:17 -04:00
patrick f6bb06ac6e fix: merge identifiers across a book's formats
Identifiers are a collection, but the whole dict was replaced per format, so
the last file to report won outright — an EPUB declaring an ASIN, a Google id
and a Calibre id kept none of them once a PDF contributed one ISBN. Accumulate
instead, letting a declared identifier outrank one scraped off a page.
2026-08-15 21:57:01 -04:00
patrick 8ee406533c feat: canonicalise author names
Extractors wrote whatever the file said, so one person held several rows:
"Sam Newman" beside "Newman, Sam;" beside "Sam Newman.epub", the last because
the upload path never stripped the file extension. Tidy on write, in the
validator and in the uniqueness lookup alike, and merge the rows that collide.
2026-08-15 21:56:35 -04:00
patrick 373c96d6d6 fix: match the compact edition markers a cover carries
"Building Microservices, 2E" never matched "Building Microservices". Key the
strip on the trailing "e" so 2E, 5e and 3 Ed are caught, while a bare number
leaves "Catch 22" and "Blade Runner 2049" alone. Stored keys are recomputed.
2026-08-15 21:55:45 -04:00
patrick fc6b97bf38 feat: review and dismiss duplicate books
Adds the per-library review screen, a muted note in the upload tray for a book
that may already be held, and the remote functions behind them. The tray note
is deliberately weaker than the file-level one, which means something stronger.
2026-08-15 21:55:20 -04:00
patrick 5047277845 feat: detect books that may already be in the library
Match on a shared identifier, or on a normalised title credited to a shared
author, and report candidates rather than refusing anything — a metadata match
is a guess, and a second edition is not a mistake. Adds a library-wide review
pass, dismissals, and renames the fingerprint pre-flight to duplicate-files.
2026-08-15 21:55:13 -04:00
patrick 6d1890ce04 feat: store matching keys and duplicate dismissals
Derive normalized_title, normalized_name and normalized_value with @validates
so no write can bypass them, and add the table recording pairs a reader has
said are not the same book.
2026-08-15 21:54:32 -04:00
patrick 523117ec28 fix: keep the identifiers an EPUB declares
DC:identifier was validated verbatim, so hyphenated and urn:isbn: forms never
reached the checksum and every non-ISBN identifier was discarded. Normalise
first, and name whatever survives.
2026-08-15 21:54:12 -04:00
patrick 86e1d096ef feat: add normalisation helpers for book matching
Reduce a title, an author and an identifier to a single comparison key, so
two copies of one book can be recognised by equality rather than by a
similarity score.
2026-08-15 21:44:50 -04:00
patrick 3f39f1f8ae fix: drop the add-anyway override from the upload tray
Storing the same bytes twice splits reading progress and shelves across two
records that can never converge. The skipped row still names the book that
holds the file; allow_duplicates stays on the API for a wrong verdict.
2026-08-14 14:18:08 -04:00
patrick d321315acf fix: ignore duplicate matches whose file is gone
The hash lives in the database and the file does not, so a file deleted behind
the app's back went on refusing its own replacement. Matches are now checked
against disk, and re-adding a book's own missing file writes it back into the
row that already describes it.
2026-08-14 14:18:08 -04:00
patrick b124a65d6e feat: report skipped duplicates in the upload tray
A book whose files were all already stored settles as skipped rather than done,
naming the book that holds them and offering to add it anyway.
2026-08-13 17:23:57 -04:00
patrick d78b21c27f feat: check for duplicate files when importing books
Incoming files are matched against what is already stored, keyed on the
KOReader hash and the file size. Bulk uploads skip and report them, deliberate
creates are refused with a 409, the consume directory parks them aside, and
allow_duplicates overrides all three.

Also: books whose metadata generates a path another book already owns are moved
aside, so a forced copy cannot overwrite the original's files.
2026-08-13 17:23:48 -04:00
44 changed files with 6742 additions and 180 deletions
+9
View File
@@ -5,6 +5,15 @@ CHITAI_TOKEN_SECRET=secret
CHITAI_DEFAULT_LIBRARY_NAME=Books CHITAI_DEFAULT_LIBRARY_NAME=Books
CHITAI_DEFAULT_LIBRARY_PATH="libraries/books" CHITAI_DEFAULT_LIBRARY_PATH="libraries/books"
# Duplicate detection when importing files (optional).
# Scope: "library" compares against the library being uploaded to, "global" against
# every library, "off" disables the check.
CHITAI_DUPLICATE_SCOPE=library
# Where the consume watcher parks files it refused as duplicates. Keep it outside
# CHITAI_CONSUME_PATH, or the watcher picks them straight back up.
CHITAI_DUPLICATE_PATH="duplicates"
# You probably should not change these # You probably should not change these
CHITAI_API_URL="http://backend:8000" CHITAI_API_URL="http://backend:8000"
CHITAI_API_DEBUG=false CHITAI_API_DEBUG=false
+142
View File
@@ -88,6 +88,14 @@ The backend owns files on disk, not just rows:
- **Layout** — `services/filesystem_library.py` (`BookPathGenerator`) renders a Jinja2 template - **Layout** — `services/filesystem_library.py` (`BookPathGenerator`) renders a Jinja2 template
against book metadata to decide where a book lives under the library's `root_path` against book metadata to decide where a book lives under the library's `root_path`
(default: `author/series/position - title/`). (default: `author/series/position - title/`).
- **One directory per book, never shared.** The generated path is a pure function of the metadata,
so two books with the same author and title produce the same one — two editions, or an
`allow_duplicates` copy. `BookService._reserve_book_path` moves the later one to `title (2)`
before anything is written, and `update_book` reserves the same way so a rename cannot move a
book in on top of another. This matters because `book.path` is what deletes, moves and file
lookups act on: books sharing a directory means one overwrites the other's files, and deleting
either takes both. `_unused_path` does the same job for filenames within a directory.
A book that already has a `path` keeps it — `add_files` must follow the book, not the template.
- **Metadata extraction** — `services/metadata_extractor.py` reads EPUB (ebooklib) and PDF - **Metadata extraction** — `services/metadata_extractor.py` reads EPUB (ebooklib) and PDF
(pypdfium2) files; extracted values fill only *empty* fields on the incoming payload. (pypdfium2) files; extracted values fill only *empty* fields on the incoming payload.
- **Covers** — converted to WebP with a UUID filename under `settings.book_cover_path`, served by a - **Covers** — converted to WebP with a UUID filename under `settings.book_cover_path`, served by a
@@ -100,6 +108,140 @@ The backend owns files on disk, not just rows:
if it differs, moves the directory contents and prunes empty parents. Keep that in mind before if it differs, moves the directory contents and prunes empty parents. Keep that in mind before
changing metadata handling. changing metadata handling.
## Duplicate detection
Every ingest path screens incoming files against what is already stored, keyed on
**`(hash, size)`** — never the hash alone, because it samples 12 KiB (see below) and
EPUBs from one toolchain often share their first window. `FileMetadata.hash` carries a
plain, deliberately **non-unique** index: a collision must not be able to fail an import,
and older databases may already hold duplicates.
Scope comes from `CHITAI_DUPLICATE_SCOPE` (`library`, the default | `global` | `off`).
The policy differs by how deliberate the import is:
| Path | Behaviour |
| --- | --- |
| `create_many_from_files` (browser bulk) | Skip per file, skip a whole group whose files are all known, report everything skipped in `ImportResult.duplicates`. Re-dropping a folder to pick up what is new is the case this serves. |
| `create_book` (single, with metadata) | All-or-nothing: raises `DuplicateFilesError`, which `controllers/book.py` renders as a **409** carrying the refused files in `extra`. |
| `add_files` | A file the book already carries is a no-op; one stored under another book raises `DuplicateFilesError`. |
| `create_many_from_existing_files` (consume watcher) | Skips, and **moves the file to `CHITAI_DUPLICATE_PATH/<library slug>/`** — nothing is deleted, and it cannot stay put because `watchfiles` only reports additions. That path must stay outside `consume_path` or the watcher re-imports it and tries to read the directory name as a library slug. |
`allow_duplicates=true` overrides all of it, on every endpoint. Keep that working — the
hash is not proof of identity, so a wrong verdict has to be recoverable, and a scripted
import needs a way through. The **web UI deliberately does not offer it**: storing the
same bytes twice splits reading progress and shelf membership across two records that
can never converge, which is nothing anyone wants on purpose.
Three things to preserve when touching this code:
- **A match only counts while the file is on disk.** `find_duplicate_files` stats each
candidate, and `add_files` writes a missing file back into the row that already
describes it (`_restore_file`) instead of adding a second row beside it. The hash
lives in the database and the file does not, so without this a file deleted behind
the app's back would go on refusing its own replacement.
- **Screening runs before anything is written.** `fingerprint_upload` reads the spooled
upload and rewinds it; the resulting fingerprints are handed to `_save_book_files`,
which skips its own `StreamingHasher` when it already has the answer. Passing them
through is what keeps the file from being read twice.
- **`_screen_for_duplicates` extends the `known` dict as it goes**, so the same bytes
submitted twice in one request are caught. Those duplicates report `book_id: None` —
there is no row to point at yet.
`POST /books/duplicate-files` answers the same question from fingerprints alone, for
clients that want to ask before uploading anything.
### Book-level detection — a different question
The file check answers "are these the same bytes?". `find_duplicate_books` answers "is
this the same book?", which a re-scan, a re-zipped EPUB or another edition cannot be
asked with a hash. Two signals, either sufficient: a **shared identifier**, or a
**matching normalized title with at least one shared author**.
It **never blocks**. A metadata match is a guess — a work shares title and author with
its own translation, its own second edition and its own audiobook — so the book is
created and the candidates are reported alongside it in
`ImportResult.possible_duplicates`. File-level dedupe keeps its refuse/skip behaviour;
that one is near-certain and this one is not. Do not "improve" this into a refusal.
Two rules narrow it, both applied in Python over the small candidate set:
- **A shared author is required for a title match.** Without it every book the
extractors gave up on and titled `Unknown` is a duplicate of every other one. A book
with no authors can therefore only match on an identifier.
- **The same series at a different `series_position` disqualifies a match.** A trilogy
shares an author and often most of its title; the position is the library saying
outright that these are two books.
A book is compared under **several title keys, not one** (`_title_keys`). Ebook files
are overwhelmingly named `Title - Author.epub`, and wherever nothing inside the file
overrode that name the author ended up in the title column — so one copy is stored as
`Building Microservices` and another as `Building Microservices Sam Newman`. Both
directions are generated, the author stripped off and the author added on, which is why
the query is `normalized_title.in_(keys)` rather than `==`. This is still exact matching
on an indexed column: no similarity score, nothing to tune. It does not weaken the
shared-author requirement, which is a separate condition.
`find_duplicate_book_groups` is the library-wide pass behind
`GET /books/duplicate-books`, since the import-time check says nothing about a
collection someone already has. It buckets books by every key they carry and merges the
buckets with union-find, so A~B by ISBN and B~C by title land in one group. Pairs in
`duplicate_dismissals` are never merged — a reader disagreeing with one pairing must not
silently break a group that stands on other evidence.
### Author names have one stored form
`Author.name` is always the canonical form, produced by `format_author_name`. Extractors
hand over whatever the file said — `Newman, Sam;` from a `DC:creator` list, `Sam Newman`
from a PDF, `Sam Newman.epub` from a filename — and storing those verbatim is how one
person becomes four rows in the sidebar, four entries in the author filter, and four
books that never look like each other.
This is **display** canonicalization, distinct from `normalize_author`, which throws
away case, accents and spacing to build a comparison key nobody sees. Tidying only
removes what an extractor added: a trailing separator, a file extension, and the
`Surname, Given` ordering. It never touches case or accents — `Michał Płachta` and
`Steve McConnell` are the author's own spelling, not something to correct.
Three places have to agree, and `Author` keeps them together: the `@validates("name")`
hook, `unique_hash`, and `unique_filter`. `as_unique_async` looks a row up with the
filter and then constructs with the validator, so if the lookup used the raw name and
the insert used the tidy one, every variant spelling would miss the existing row and
then collide with it on the unique index.
A form the rule does not recognise is **left exactly as it was found** — `Dave Thomas,
Andy Hunt` is two people in one string, and flipping it would invent a third. Leaving a
mess visible beats rewriting it wrongly.
### The normalized columns are written by validators
`Book.normalized_title`, `Author.normalized_name` and `Identifier.normalized_value` are
derived from `services/matching.py` and kept current by **SQLAlchemy `@validates` hooks
on the models**, not by any service. Assigning them directly is always wrong.
This is deliberate and it is invisible at the call sites: `BookService` writes titles
through at least three paths (`to_model_on_create`, `to_model_on_update`, and the
`setattr` loop in `_populate_with_unique_relationships`), and a validator is the only
thing a fourth cannot bypass. The columns carry plain, **non-unique** btree indexes —
two spellings collapsing onto one value is the entire point.
`services/matching.py` is imported *inside* those validators rather than at module
scope: reaching it initialises the `chitai.services` package, which imports the
services, which import the models. Keep the local import.
The validators only fire on write, so a migration that adds one of these columns must
backfill existing rows through the same helpers — see the `data_upgrades()` hook in
`2026-08-15_add_book_matching_keys_and_duplicate__4358e7d4743a.py`.
**Changing anything in `services/matching.py` needs a revision that recomputes them.**
The keys are derived and already written, so a normalization change silently invalidates
every stored row: a book written under the old rules just stops matching one written
under the new rules, with nothing to show that anything is wrong. Copy
`2026-08-15_recompute_book_matching_keys_ed41acf21270.py`, which exists because
`normalize_title` learned to strip compact edition markers (`2E`, `5e`). It is
idempotent and safe to re-run.
## KOReader hashing ## KOReader hashing
`services/utils.py` reimplements KOReader's partial-MD5 document identifier: 1 KiB samples at `services/utils.py` reimplements KOReader's partial-MD5 document identifier: 1 KiB samples at
@@ -0,0 +1,80 @@
"""add file hash index
Revision ID: e9c2c7e875ae
Revises: 6d72d1bbc0ee
Create Date: 2026-08-13 14:52:09.341906
"""
import warnings
from typing import TYPE_CHECKING
import sqlalchemy as sa
from alembic import op
from advanced_alchemy.types import EncryptedString, EncryptedText, GUID, ORA_JSONB, DateTimeUTC, StoredObject, PasswordHash, FernetBackend
from advanced_alchemy.types.encrypted_string import PGCryptoBackend
from advanced_alchemy.types.password_hash.argon2 import Argon2Hasher
from advanced_alchemy.types.password_hash.passlib import PasslibHasher
from advanced_alchemy.types.password_hash.pwdlib import PwdlibHasher
from sqlalchemy import Text # noqa: F401
if TYPE_CHECKING:
from collections.abc import Sequence
__all__ = ["downgrade", "upgrade", "schema_upgrades", "schema_downgrades", "data_upgrades", "data_downgrades"]
sa.GUID = GUID
sa.DateTimeUTC = DateTimeUTC
sa.ORA_JSONB = ORA_JSONB
sa.EncryptedString = EncryptedString
sa.EncryptedText = EncryptedText
sa.StoredObject = StoredObject
sa.PasswordHash = PasswordHash
sa.Argon2Hasher = Argon2Hasher
sa.PasslibHasher = PasslibHasher
sa.PwdlibHasher = PwdlibHasher
sa.FernetBackend = FernetBackend
sa.PGCryptoBackend = PGCryptoBackend
# revision identifiers, used by Alembic.
revision = 'e9c2c7e875ae'
down_revision = '6d72d1bbc0ee'
branch_labels = None
depends_on = None
def upgrade() -> None:
with warnings.catch_warnings():
warnings.filterwarnings("ignore", category=UserWarning)
with op.get_context().autocommit_block():
schema_upgrades()
data_upgrades()
def downgrade() -> None:
with warnings.catch_warnings():
warnings.filterwarnings("ignore", category=UserWarning)
with op.get_context().autocommit_block():
data_downgrades()
schema_downgrades()
def schema_upgrades() -> None:
"""schema upgrade migrations go here."""
# ### commands auto generated by Alembic - please adjust! ###
with op.batch_alter_table('file_metadata', schema=None) as batch_op:
batch_op.create_index('ix_file_metadata_hash', ['hash'], unique=False)
# ### end Alembic commands ###
def schema_downgrades() -> None:
"""schema downgrade migrations go here."""
# ### commands auto generated by Alembic - please adjust! ###
with op.batch_alter_table('file_metadata', schema=None) as batch_op:
batch_op.drop_index('ix_file_metadata_hash')
# ### end Alembic commands ###
def data_upgrades() -> None:
"""Add any optional data upgrade migrations here!"""
def data_downgrades() -> None:
"""Add any optional data downgrade migrations here!"""
@@ -0,0 +1,163 @@
"""add book matching keys and duplicate dismissals
Revision ID: 4358e7d4743a
Revises: e9c2c7e875ae
Create Date: 2026-08-15 15:09:02.708914
"""
import warnings
import sqlalchemy as sa
from alembic import op
from advanced_alchemy.types import EncryptedString, EncryptedText, GUID, ORA_JSONB, DateTimeUTC, StoredObject, PasswordHash, FernetBackend
from advanced_alchemy.types.encrypted_string import PGCryptoBackend
from advanced_alchemy.types.password_hash.argon2 import Argon2Hasher
from advanced_alchemy.types.password_hash.passlib import PasslibHasher
from advanced_alchemy.types.password_hash.pwdlib import PwdlibHasher
from sqlalchemy import Text # noqa: F401
__all__ = ["downgrade", "upgrade", "schema_upgrades", "schema_downgrades", "data_upgrades", "data_downgrades"]
sa.GUID = GUID
sa.DateTimeUTC = DateTimeUTC
sa.ORA_JSONB = ORA_JSONB
sa.EncryptedString = EncryptedString
sa.EncryptedText = EncryptedText
sa.StoredObject = StoredObject
sa.PasswordHash = PasswordHash
sa.Argon2Hasher = Argon2Hasher
sa.PasslibHasher = PasslibHasher
sa.PwdlibHasher = PwdlibHasher
sa.FernetBackend = FernetBackend
sa.PGCryptoBackend = PGCryptoBackend
# revision identifiers, used by Alembic.
revision = '4358e7d4743a'
down_revision = 'e9c2c7e875ae'
branch_labels = None
depends_on = None
def upgrade() -> None:
with warnings.catch_warnings():
warnings.filterwarnings("ignore", category=UserWarning)
with op.get_context().autocommit_block():
schema_upgrades()
data_upgrades()
def downgrade() -> None:
with warnings.catch_warnings():
warnings.filterwarnings("ignore", category=UserWarning)
with op.get_context().autocommit_block():
data_downgrades()
schema_downgrades()
def schema_upgrades() -> None:
"""schema upgrade migrations go here."""
# ### commands auto generated by Alembic - please adjust! ###
op.create_table('duplicate_dismissals',
sa.Column('id', sa.BigInteger().with_variant(sa.Integer(), 'sqlite'), nullable=False),
sa.Column('book_a_id', sa.BigInteger().with_variant(sa.Integer(), 'sqlite'), nullable=False),
sa.Column('book_b_id', sa.BigInteger().with_variant(sa.Integer(), 'sqlite'), nullable=False),
sa.ForeignKeyConstraint(['book_a_id'], ['books.id'], name=op.f('fk_duplicate_dismissals_book_a_id_books'), ondelete='cascade'),
sa.ForeignKeyConstraint(['book_b_id'], ['books.id'], name=op.f('fk_duplicate_dismissals_book_b_id_books'), ondelete='cascade'),
sa.PrimaryKeyConstraint('id', name=op.f('pk_duplicate_dismissals')),
sa.UniqueConstraint('book_a_id', 'book_b_id', name=op.f('uq_duplicate_dismissals_book_a_id'))
)
with op.batch_alter_table('duplicate_dismissals', schema=None) as batch_op:
batch_op.create_index(batch_op.f('ix_duplicate_dismissals_book_a_id'), ['book_a_id'], unique=False)
batch_op.create_index(batch_op.f('ix_duplicate_dismissals_book_b_id'), ['book_b_id'], unique=False)
# `server_default` so the column can be added to a table that already has rows;
# `data_upgrades` fills in the real keys immediately afterwards.
with op.batch_alter_table('authors', schema=None) as batch_op:
batch_op.add_column(sa.Column('normalized_name', sa.String(), nullable=False, server_default=''))
batch_op.create_index(batch_op.f('ix_authors_normalized_name'), ['normalized_name'], unique=False)
with op.batch_alter_table('books', schema=None) as batch_op:
batch_op.add_column(sa.Column('normalized_title', sa.String(), nullable=False, server_default=''))
batch_op.create_index(batch_op.f('ix_books_normalized_title'), ['normalized_title'], unique=False)
with op.batch_alter_table('identifiers', schema=None) as batch_op:
batch_op.add_column(sa.Column('normalized_value', sa.String(), nullable=True))
batch_op.create_index(batch_op.f('ix_identifiers_normalized_value'), ['normalized_value'], unique=False)
# ### end Alembic commands ###
def schema_downgrades() -> None:
"""schema downgrade migrations go here."""
# ### commands auto generated by Alembic - please adjust! ###
with op.batch_alter_table('identifiers', schema=None) as batch_op:
batch_op.drop_index(batch_op.f('ix_identifiers_normalized_value'))
batch_op.drop_column('normalized_value')
with op.batch_alter_table('books', schema=None) as batch_op:
batch_op.drop_index(batch_op.f('ix_books_normalized_title'))
batch_op.drop_column('normalized_title')
with op.batch_alter_table('authors', schema=None) as batch_op:
batch_op.drop_index(batch_op.f('ix_authors_normalized_name'))
batch_op.drop_column('normalized_name')
with op.batch_alter_table('duplicate_dismissals', schema=None) as batch_op:
batch_op.drop_index(batch_op.f('ix_duplicate_dismissals_book_b_id'))
batch_op.drop_index(batch_op.f('ix_duplicate_dismissals_book_a_id'))
op.drop_table('duplicate_dismissals')
# ### end Alembic commands ###
def data_upgrades() -> None:
"""
Fill the matching keys in for rows that already exist.
The validators on the models only fire when something is written, so without this
every book imported before today is invisible to duplicate detection. Run through
the same helpers the validators use, so a backfilled row and a freshly written one
are guaranteed to agree.
"""
from chitai.services.matching import (
normalize_author,
normalize_identifier,
normalize_title,
)
connection = op.get_bind()
books = connection.execute(sa.text("SELECT id, title FROM books")).fetchall()
_apply(
connection,
"UPDATE books SET normalized_title = :key WHERE id = :id",
[{"id": id, "key": normalize_title(title)} for id, title in books],
)
authors = connection.execute(sa.text("SELECT id, name FROM authors")).fetchall()
_apply(
connection,
"UPDATE authors SET normalized_name = :key WHERE id = :id",
[{"id": id, "key": normalize_author(name)} for id, name in authors],
)
identifiers = connection.execute(
sa.text("SELECT id, name, value FROM identifiers")
).fetchall()
_apply(
connection,
"UPDATE identifiers SET normalized_value = :key WHERE id = :id",
[
{"id": id, "key": normalize_identifier(name, value)}
for id, name, value in identifiers
],
)
def _apply(connection, statement: str, parameters: list[dict]) -> None:
"""Run one update per row, in batches, skipping the work when there are none."""
batch_size = 1000
for start in range(0, len(parameters), batch_size):
connection.execute(sa.text(statement), parameters[start : start + batch_size])
def data_downgrades() -> None:
"""Add any optional data downgrade migrations here!"""
@@ -0,0 +1,130 @@
"""canonicalize author names
Revision ID: 49a9e85a0ffc
Revises: ed41acf21270
Create Date: 2026-08-15 15:59:47.331545
"""
import warnings
import sqlalchemy as sa
from alembic import op
from advanced_alchemy.types import EncryptedString, EncryptedText, GUID, ORA_JSONB, DateTimeUTC, StoredObject, PasswordHash, FernetBackend
from advanced_alchemy.types.encrypted_string import PGCryptoBackend
from advanced_alchemy.types.password_hash.argon2 import Argon2Hasher
from advanced_alchemy.types.password_hash.passlib import PasslibHasher
from advanced_alchemy.types.password_hash.pwdlib import PwdlibHasher
from sqlalchemy import Text # noqa: F401
__all__ = ["downgrade", "upgrade", "schema_upgrades", "schema_downgrades", "data_upgrades", "data_downgrades"]
sa.GUID = GUID
sa.DateTimeUTC = DateTimeUTC
sa.ORA_JSONB = ORA_JSONB
sa.EncryptedString = EncryptedString
sa.EncryptedText = EncryptedText
sa.StoredObject = StoredObject
sa.PasswordHash = PasswordHash
sa.Argon2Hasher = Argon2Hasher
sa.PasslibHasher = PasslibHasher
sa.PwdlibHasher = PwdlibHasher
sa.FernetBackend = FernetBackend
sa.PGCryptoBackend = PGCryptoBackend
# revision identifiers, used by Alembic.
revision = '49a9e85a0ffc'
down_revision = 'ed41acf21270'
branch_labels = None
depends_on = None
def upgrade() -> None:
with warnings.catch_warnings():
warnings.filterwarnings("ignore", category=UserWarning)
with op.get_context().autocommit_block():
schema_upgrades()
data_upgrades()
def downgrade() -> None:
with warnings.catch_warnings():
warnings.filterwarnings("ignore", category=UserWarning)
with op.get_context().autocommit_block():
data_downgrades()
schema_downgrades()
def schema_upgrades() -> None:
"""schema upgrade migrations go here."""
pass
def schema_downgrades() -> None:
"""schema downgrade migrations go here."""
pass
def data_upgrades() -> None:
"""
Rewrite every author into the canonical form, merging the rows that collide.
`Author.name` is only now guaranteed tidy — until this revision extractors wrote
whatever the file said, so one person could hold several rows: "Sam Newman" beside
"Newman, Sam;" beside "Sam Newman.epub" (the last from a filename whose extension
was never stripped). Each showed up as its own author in the sidebar and its own
filter, and no amount of fixing the extractors repairs a row already written.
Rows that canonicalize onto one name are merged into the lowest id, which keeps
whichever row the library has been referring to longest. `book.path` is stored, not
derived, so renaming an author moves nothing on disk.
"""
from chitai.services.matching import format_author_name, normalize_author
connection = op.get_bind()
authors = connection.execute(sa.text("SELECT id, name FROM authors")).fetchall()
groups: dict[str, list[int]] = {}
for id, name in sorted(authors):
# A name with nothing left of it after tidying is left exactly as it was:
# merging those together would invent one author out of several unrelated
# broken rows, which is worse than leaving the mess visible.
if canonical := format_author_name(name):
groups.setdefault(canonical, []).append(id)
for canonical, ids in groups.items():
winner, losers = ids[0], ids[1:]
for loser in losers:
# A book credited to both rows would otherwise breach the
# (book_id, author_id) unique constraint the moment the link is repointed.
connection.execute(
sa.text(
"DELETE FROM book_author_links WHERE author_id = :loser AND book_id IN"
" (SELECT book_id FROM book_author_links WHERE author_id = :winner)"
),
{"loser": loser, "winner": winner},
)
connection.execute(
sa.text(
"UPDATE book_author_links SET author_id = :winner"
" WHERE author_id = :loser"
),
{"loser": loser, "winner": winner},
)
connection.execute(
sa.text("DELETE FROM authors WHERE id = :loser"), {"loser": loser}
)
# Only after the losers are gone, or this collides with the unique index.
connection.execute(
sa.text(
"UPDATE authors SET name = :name, normalized_name = :key WHERE id = :id"
),
{"id": winner, "name": canonical, "key": normalize_author(canonical)},
)
def data_downgrades() -> None:
"""
Nothing to undo.
The rows a merge removed are gone, and the spellings it replaced were never
recorded anywhere else — there is nothing to restore them from.
"""
@@ -0,0 +1,122 @@
"""recompute book matching keys
Revision ID: ed41acf21270
Revises: 4358e7d4743a
Create Date: 2026-08-15 15:44:28.341020
"""
import warnings
import sqlalchemy as sa
from alembic import op
from advanced_alchemy.types import EncryptedString, EncryptedText, GUID, ORA_JSONB, DateTimeUTC, StoredObject, PasswordHash, FernetBackend
from advanced_alchemy.types.encrypted_string import PGCryptoBackend
from advanced_alchemy.types.password_hash.argon2 import Argon2Hasher
from advanced_alchemy.types.password_hash.passlib import PasslibHasher
from advanced_alchemy.types.password_hash.pwdlib import PwdlibHasher
from sqlalchemy import Text # noqa: F401
__all__ = ["downgrade", "upgrade", "schema_upgrades", "schema_downgrades", "data_upgrades", "data_downgrades"]
sa.GUID = GUID
sa.DateTimeUTC = DateTimeUTC
sa.ORA_JSONB = ORA_JSONB
sa.EncryptedString = EncryptedString
sa.EncryptedText = EncryptedText
sa.StoredObject = StoredObject
sa.PasswordHash = PasswordHash
sa.Argon2Hasher = Argon2Hasher
sa.PasslibHasher = PasslibHasher
sa.PwdlibHasher = PwdlibHasher
sa.FernetBackend = FernetBackend
sa.PGCryptoBackend = PGCryptoBackend
# revision identifiers, used by Alembic.
revision = 'ed41acf21270'
down_revision = '4358e7d4743a'
branch_labels = None
depends_on = None
def upgrade() -> None:
with warnings.catch_warnings():
warnings.filterwarnings("ignore", category=UserWarning)
with op.get_context().autocommit_block():
schema_upgrades()
data_upgrades()
def downgrade() -> None:
with warnings.catch_warnings():
warnings.filterwarnings("ignore", category=UserWarning)
with op.get_context().autocommit_block():
data_downgrades()
schema_downgrades()
def schema_upgrades() -> None:
"""schema upgrade migrations go here."""
pass
def schema_downgrades() -> None:
"""schema downgrade migrations go here."""
pass
def data_upgrades() -> None:
"""
Recompute every matching key against the current normalization.
The keys are derived, so changing a helper in `services/matching.py` silently
invalidates every row already written — a book stored under the old rules simply
stops matching one stored under the new ones, with nothing to show that anything
is wrong. `normalize_title` learned to strip the compact edition markers a cover
actually carries ("2E", "5e"), which moved `Building Microservices, 2E` onto the
same key as `Building Microservices`.
Any later change to those helpers wants a revision that looks exactly like this
one. It is idempotent and safe to re-run.
"""
from chitai.services.matching import (
normalize_author,
normalize_identifier,
normalize_title,
)
connection = op.get_bind()
books = connection.execute(sa.text("SELECT id, title FROM books")).fetchall()
_apply(
connection,
"UPDATE books SET normalized_title = :key WHERE id = :id",
[{"id": id, "key": normalize_title(title)} for id, title in books],
)
authors = connection.execute(sa.text("SELECT id, name FROM authors")).fetchall()
_apply(
connection,
"UPDATE authors SET normalized_name = :key WHERE id = :id",
[{"id": id, "key": normalize_author(name)} for id, name in authors],
)
identifiers = connection.execute(
sa.text("SELECT id, name, value FROM identifiers")
).fetchall()
_apply(
connection,
"UPDATE identifiers SET normalized_value = :key WHERE id = :id",
[
{"id": id, "key": normalize_identifier(name, value)}
for id, name, value in identifiers
],
)
def _apply(connection, statement: str, parameters: list[dict]) -> None:
"""Run one update per row, in batches, skipping the work when there are none."""
batch_size = 1000
for start in range(0, len(parameters), batch_size):
connection.execute(sa.text(statement), parameters[start : start + batch_size])
def data_downgrades() -> None:
"""Add any optional data downgrade migrations here!"""
+24
View File
@@ -1,9 +1,25 @@
from enum import StrEnum
from pydantic import Field, PostgresDsn, computed_field from pydantic import Field, PostgresDsn, computed_field
from pydantic_settings import BaseSettings, SettingsConfigDict from pydantic_settings import BaseSettings, SettingsConfigDict
from advanced_alchemy.extensions.litestar import ( from advanced_alchemy.extensions.litestar import (
SQLAlchemyAsyncConfig, SQLAlchemyAsyncConfig,
) )
class DuplicateScope(StrEnum):
"""How widely an incoming file is compared against what is already stored."""
LIBRARY = "library"
"""Only files in the library being uploaded to count as duplicates."""
GLOBAL = "global"
"""A file already held by any library counts as a duplicate."""
OFF = "off"
"""No duplicate detection at all."""
class Settings(BaseSettings): class Settings(BaseSettings):
version: str = Field("0.0.1") version: str = Field("0.0.1")
project_name: str = Field("chitai") project_name: str = Field("chitai")
@@ -33,6 +49,14 @@ class Settings(BaseSettings):
# Path to consume directory # Path to consume directory
consume_path: str = Field("./consume") consume_path: str = Field("./consume")
# Duplicate detection
duplicate_scope: DuplicateScope = Field(DuplicateScope.LIBRARY)
# Where the consume watcher parks files it refused as duplicates. Must sit
# outside `consume_path`, or the watcher picks them straight back up and
# tries to resolve the directory name as a library slug.
duplicate_path: str = Field("./duplicates")
@computed_field @computed_field
@property @property
def postgres_uri(self) -> PostgresDsn: def postgres_uri(self) -> PostgresDsn:
+248 -9
View File
@@ -12,7 +12,12 @@ from litestar.params import Dependency, Body
from litestar.enums import RequestEncodingType from litestar.enums import RequestEncodingType
from litestar.response import File, Stream from litestar.response import File, Stream
from litestar.exceptions import HTTPException from litestar.exceptions import HTTPException
from litestar.status_codes import HTTP_400_BAD_REQUEST from litestar.status_codes import (
HTTP_200_OK,
HTTP_204_NO_CONTENT,
HTTP_400_BAD_REQUEST,
HTTP_409_CONFLICT,
)
from litestar.datastructures import UploadFile from litestar.datastructures import UploadFile
from advanced_alchemy.service.pagination import OffsetPagination from advanced_alchemy.service.pagination import OffsetPagination
from advanced_alchemy.filters import CollectionFilter from advanced_alchemy.filters import CollectionFilter
@@ -23,6 +28,24 @@ from chitai.services import dependencies as deps
from chitai import schemas as s from chitai import schemas as s
from chitai.database import models as m from chitai.database import models as m
from chitai.services import BookService, BookProgressService from chitai.services import BookService, BookProgressService
from chitai.services.book import DuplicateFilesError
def _duplicate_conflict(exc: DuplicateFilesError) -> HTTPException:
"""
Turn refused files into a 409 the caller can act on.
The files ride along in `extra` so the client can name them and offer to send them
again with `allow_duplicates`, rather than being told only that something clashed.
"""
return HTTPException(
status_code=HTTP_409_CONFLICT,
detail="These files are already in the library",
extra=[
s.DuplicateFileRead.model_validate(duplicate).model_dump(mode="json")
for duplicate in exc.duplicates
],
)
class BookController(Controller): class BookController(Controller):
@@ -63,6 +86,7 @@ class BookController(Controller):
books_service: BookService, books_service: BookService,
library: m.Library, library: m.Library,
data: Annotated[s.BookCreate, Body(media_type=RequestEncodingType.MULTI_PART)], data: Annotated[s.BookCreate, Body(media_type=RequestEncodingType.MULTI_PART)],
allow_duplicates: bool = False,
) -> s.BookRead: ) -> s.BookRead:
""" """
Create a new book with metadata and files. Create a new book with metadata and files.
@@ -73,6 +97,10 @@ class BookController(Controller):
Path Parameters: Path Parameters:
library_id: The ID of the library the book belongs to. library_id: The ID of the library the book belongs to.
Query Parameters:
allow_duplicates: If True, store the files even if the library already
holds them.
Request Body: Request Body:
data: Book creation data including metadata and files. data: Book creation data including metadata and files.
@@ -83,9 +111,17 @@ class BookController(Controller):
Returns: Returns:
The created book as a BookRead schema. The created book as a BookRead schema.
Raises:
HTTPException: 409 if any of the files is already in the library.
""" """
result = await books_service.create_book(data, library) try:
result = await books_service.create_book(
data, library, screen_duplicates=not allow_duplicates
)
except DuplicateFilesError as exc:
raise _duplicate_conflict(exc)
book = await books_service.get(result.id) book = await books_service.get(result.id)
return books_service.to_schema(book, schema_type=s.BookRead) return books_service.to_schema(book, schema_type=s.BookRead)
@@ -97,13 +133,20 @@ class BookController(Controller):
data: Annotated[ data: Annotated[
s.BooksCreateFromFiles, Body(media_type=RequestEncodingType.MULTI_PART) s.BooksCreateFromFiles, Body(media_type=RequestEncodingType.MULTI_PART)
], ],
) -> OffsetPagination[s.BookRead]: allow_duplicates: bool = False,
) -> s.BooksUploadResult:
""" """
Create multiple books from uploaded files. Create multiple books from uploaded files.
Groups files by directory and creates separate books for each group. Groups files by directory and creates separate books for each group.
Metadata is automatically extracted from the files. Metadata is automatically extracted from the files.
Files the library already holds are skipped rather than refused, and reported
back so the caller can say which ones did not make it in and why.
Query Parameters:
allow_duplicates: If True, store every file, even one already held.
Request Body: Request Body:
data: Container with list of uploaded files. data: Container with list of uploaded files.
@@ -112,19 +155,200 @@ class BookController(Controller):
library: The library the books belong to. library: The library the books belong to.
Returns: Returns:
Paginated list of created books. The books created, and the files skipped as duplicates.
""" """
try: try:
results = await books_service.create_many_from_files(data, library) result = await books_service.create_many_from_files(
data, library, allow_duplicates=allow_duplicates
)
except ValueError: except ValueError:
raise HTTPException( raise HTTPException(
status_code=HTTP_400_BAD_REQUEST, detail="Must upload at least one file" status_code=HTTP_400_BAD_REQUEST, detail="Must upload at least one file"
) )
books = await books_service.list( books = (
CollectionFilter("id", [result.id for result in results]) await books_service.list(
CollectionFilter("id", [book.id for book in result.books])
)
if result.books
else []
) )
return books_service.to_schema(books, schema_type=s.BookRead)
return s.BooksUploadResult(
created=[
books_service.to_schema(book, schema_type=s.BookRead) for book in books
],
skipped=[
s.DuplicateFileRead.model_validate(duplicate)
for duplicate in result.duplicates
],
possible_duplicates=[
s.PossibleDuplicateRead.model_validate(possible)
for possible in result.possible_duplicates
],
)
@get(path="duplicate-books")
async def list_duplicate_books(
self, books_service: BookService, library: m.Library
) -> list[s.DuplicateBookGroupRead]:
"""
Report books already in the library that look like copies of one another.
The import-time check only ever sees what is arriving, so this is what covers
a collection someone already has. Matching is on metadata and therefore a
guess: a group is a question for the reader, not a verdict.
Query Parameters:
library_id: The library to review.
Injected Dependencies:
books_service: The book service for database operations.
library: The library to review.
Returns:
One entry per group of two or more books. Empty when there is nothing to
review, or when duplicate detection is switched off.
"""
groups = await books_service.find_duplicate_book_groups(library)
return [
s.DuplicateBookGroupRead(
books=[s.DuplicateBookRead.model_validate(book) for book in group]
)
for group in groups
]
@post(path="merge")
async def merge_books(
self, books_service: BookService, library: m.Library, data: s.BookMerge
) -> s.BookRead:
"""
Fold several books into one and delete the records folded in.
The survivor keeps its id, so links and bookmarks still resolve. Files, reading
progress, shelves, tags and unheld identifiers move onto it; metadata is only
changed by what `metadata` names, because choosing between two titles is the
reader's judgement rather than this endpoint's.
Nothing is removed from disk — a wrong merge should cost metadata that can be
retyped, not a book.
Query Parameters:
library_id: The library the books belong to.
Request Body:
data: The survivor, the books to fold in, and the resolved metadata.
Injected Dependencies:
books_service: The book service for database operations.
library: The library the books belong to.
Returns:
The surviving book.
Raises:
HTTPException: 400 if fewer than two distinct books were named, one is
unknown, or they do not all belong to one library.
"""
try:
book = await books_service.merge_books(
data.survivor_id,
data.merged_ids,
library,
metadata=data.metadata.model_dump(exclude_unset=True)
if data.metadata
else None,
)
except ValueError as exc:
raise HTTPException(status_code=HTTP_400_BAD_REQUEST, detail=str(exc))
return books_service.to_schema(book, schema_type=s.BookRead)
@post(path="duplicate-books/dismissals", status_code=HTTP_204_NO_CONTENT)
async def dismiss_duplicate_books(
self, books_service: BookService, data: s.DuplicateDismissal
) -> None:
"""
Record that two books are not the same book.
Without this the review screen proposes the same wrong pair forever, which is
how a reader learns to stop looking at it.
Request Body:
data: The two book IDs. Order does not matter.
Injected Dependencies:
books_service: The book service for database operations.
Raises:
HTTPException: 400 if the two IDs are the same or either book is unknown.
"""
try:
await books_service.dismiss_duplicates(data.book_a_id, data.book_b_id)
except ValueError as exc:
raise HTTPException(status_code=HTTP_400_BAD_REQUEST, detail=str(exc))
@delete(path="duplicate-books/dismissals")
async def restore_duplicate_books(
self, books_service: BookService, book_a_id: int, book_b_id: int
) -> None:
"""
Undo a dismissal, so the pair is proposed again.
Query Parameters:
book_a_id: One of the two books.
book_b_id: The other. Order does not matter.
Injected Dependencies:
books_service: The book service for database operations.
"""
await books_service.restore_duplicates(book_a_id, book_b_id)
# A question, not a change: 200 rather than the 201 a POST would default to.
@post(path="duplicate-files", status_code=HTTP_200_OK)
async def check_duplicate_files(
self,
books_service: BookService,
library: m.Library,
data: list[s.FileFingerprint],
) -> list[s.DuplicateFileRead]:
"""
Report which of the given files the library already holds.
Lets a client ask before it uploads anything, which is the difference between
re-sending a folder of books and re-sending twelve kilobytes of hashes.
Query Parameters:
library_id: The library to check against.
Request Body:
data: Hash and size for each file, optionally with the name to echo back.
Injected Dependencies:
books_service: The book service for database operations.
library: The library to check against.
Returns:
One entry per submitted file that is already stored. Files that are not
are absent.
"""
matches = await books_service.find_duplicate_files(
((item.hash, item.size) for item in data), library
)
return [
s.DuplicateFileRead(
filename=item.filename or match.filename,
hash=match.hash,
size=match.size,
library_id=match.library_id,
book_id=match.book_id,
book_title=match.book_title,
)
for item in data
if (match := matches.get((item.hash, item.size))) is not None
]
@get(path="/{book_id:int}") @get(path="/{book_id:int}")
async def get_book_by_id( async def get_book_by_id(
@@ -303,13 +527,20 @@ class BookController(Controller):
], ],
library: m.Library, library: m.Library,
books_service: BookService, books_service: BookService,
allow_duplicates: bool = False,
) -> s.BookRead: ) -> s.BookRead:
""" """
Add files to an existing book. Add files to an existing book.
A file the book already carries is ignored, so re-sending one is harmless.
Path Parameters: Path Parameters:
book_id: The ID of the book to modify book_id: The ID of the book to modify
Query Parameters:
allow_duplicates: If True, store the files even if the library already
holds them.
Request Body: Request Body:
files: The files to add to the book files: The files to add to the book
@@ -320,9 +551,17 @@ class BookController(Controller):
Returns: Returns:
The modified book The modified book
Raises:
HTTPException: 409 if a file is already stored under a different book.
""" """
await books_service.add_files(book_id, data, library) try:
await books_service.add_files(
book_id, data, library, allow_duplicates=allow_duplicates
)
except DuplicateFilesError as exc:
raise _duplicate_conflict(exc)
book = await books_service.get(book_id) book = await books_service.get(book_id)
return books_service.to_schema(book, schema_type=s.BookRead) return books_service.to_schema(book, schema_type=s.BookRead)
@@ -3,6 +3,7 @@ from .book import Book, Identifier, FileMetadata
from .book_list import BookList, BookListLink from .book_list import BookList, BookListLink
from .book_progress import BookProgress from .book_progress import BookProgress
from .book_series import BookSeries from .book_series import BookSeries
from .duplicate_dismissal import DuplicateDismissal
from .kosync_device import KosyncDevice from .kosync_device import KosyncDevice
from .kosync_progress import KosyncProgress from .kosync_progress import KosyncProgress
from .library import Library from .library import Library
+40 -2
View File
@@ -5,6 +5,7 @@ from sqlalchemy import ColumnElement, ForeignKey, UniqueConstraint
from sqlalchemy.orm import Mapped from sqlalchemy.orm import Mapped
from sqlalchemy.orm import mapped_column from sqlalchemy.orm import mapped_column
from sqlalchemy.orm import relationship from sqlalchemy.orm import relationship
from sqlalchemy.orm import validates
from advanced_alchemy.base import BigIntAuditBase, BigIntBase from advanced_alchemy.base import BigIntAuditBase, BigIntBase
from advanced_alchemy.mixins import UniqueMixin from advanced_alchemy.mixins import UniqueMixin
@@ -16,18 +17,55 @@ if TYPE_CHECKING:
class Author(BigIntAuditBase, UniqueMixin): class Author(BigIntAuditBase, UniqueMixin):
__tablename__ = "authors" __tablename__ = "authors"
# Always the canonical form — see `_canonicalize`. Extractors hand over whatever
# the file happened to say: "Newman, Sam;" from a `DC:creator` list, or
# "Sam Newman.epub" from a filename. Storing those verbatim is how one person ends
# up as several rows in the sidebar.
name: Mapped[str] = mapped_column(unique=True, index=True) name: Mapped[str] = mapped_column(unique=True, index=True)
# Kept current by `_canonicalize` too — never assign it directly. Not unique: two
# spellings that survive canonicalization, "Steve Mcconnell" and "Steve McConnell",
# are still one person to a reader, which is what this column exists to express.
normalized_name: Mapped[str] = mapped_column(default="", index=True)
description: Mapped[Optional[str]] description: Mapped[Optional[str]]
@validates("name")
def _canonicalize(self, _key: str, name: str) -> str:
"""
Store the tidied name, and derive the matching key from it.
A validator so no write can get around it, and `unique_hash` / `unique_filter`
below tidy the same way so `as_unique_async` looks the row up under the name it
would actually be stored as. All three have to agree: if the lookup used the
raw name and the insert used the tidy one, every variant spelling would miss
the existing row and then collide with it on the unique index.
"""
# Imported here rather than at module scope: `chitai.services.matching` cannot
# be reached without initialising the `chitai.services` package, which imports
# the services, which import this module.
from chitai.services.matching import format_author_name, normalize_author
name = format_author_name(name)
self.normalized_name = normalize_author(name)
return name
@classmethod
def _tidy(cls, name: str) -> str:
"""The name as `_canonicalize` would store it."""
from chitai.services.matching import format_author_name
return format_author_name(name)
@classmethod @classmethod
def unique_hash(cls, name: str) -> Hashable: def unique_hash(cls, name: str) -> Hashable:
"""Generate a unique hash for deduplication.""" """Generate a unique hash for deduplication."""
return name return cls._tidy(name)
@classmethod @classmethod
def unique_filter(cls, name: str) -> ColumnElement[bool]: def unique_filter(cls, name: str) -> ColumnElement[bool]:
"""SQL filter for finding existing records.""" """SQL filter for finding existing records."""
return cls.name == name return cls.name == cls._tidy(name)
def __repr__(self) -> str: def __repr__(self) -> str:
return f"Author({self.name!r})" return f"Author({self.name!r})"
+53 -4
View File
@@ -2,13 +2,10 @@ from datetime import date
from typing import TYPE_CHECKING, Any, Optional from typing import TYPE_CHECKING, Any, Optional
from sqlalchemy import Index, ForeignKey, UniqueConstraint from sqlalchemy import Index, ForeignKey, UniqueConstraint
from sqlalchemy.orm import Mapped, mapped_column, relationship from sqlalchemy.orm import Mapped, mapped_column, relationship, validates
from sqlalchemy.orm import mapped_column
from sqlalchemy.orm import relationship
from sqlalchemy.ext.orderinglist import ordering_list from sqlalchemy.ext.orderinglist import ordering_list
from sqlalchemy.ext.associationproxy import association_proxy from sqlalchemy.ext.associationproxy import association_proxy
from sqlalchemy.ext.associationproxy import AssociationProxy from sqlalchemy.ext.associationproxy import AssociationProxy
from sqlalchemy.orm.collections import attribute_keyed_dict
from advanced_alchemy.base import BigIntAuditBase, BigIntBase from advanced_alchemy.base import BigIntAuditBase, BigIntBase
@@ -44,6 +41,13 @@ class Book(BigIntAuditBase):
library: Mapped["Library"] = relationship(back_populates="books") library: Mapped["Library"] = relationship(back_populates="books")
title: Mapped[str] title: Mapped[str]
# Kept current by `_normalize_title` below — never assign it directly.
#
# Deliberately not unique: two spellings collapsing onto one value is the whole
# point of the column, and a second edition is allowed to exist.
normalized_title: Mapped[str] = mapped_column(default="", index=True)
subtitle: Mapped[Optional[str]] subtitle: Mapped[Optional[str]]
description: Mapped[Optional[str]] description: Mapped[Optional[str]]
published_date: Mapped[Optional[date]] published_date: Mapped[Optional[date]]
@@ -111,6 +115,24 @@ class Book(BigIntAuditBase):
def progress(self) -> Optional["BookProgress"]: def progress(self) -> Optional["BookProgress"]:
return self.progress_records[0] if self.progress_records else None return self.progress_records[0] if self.progress_records else None
@validates("title")
def _normalize_title(self, _key: str, title: str) -> str:
"""
Derive `normalized_title` from whatever writes the title.
A validator rather than a service call because `BookService` sets titles from
at least three places — `to_model_on_create`, `to_model_on_update` and the
`setattr` loop in `_populate_with_unique_relationships` — and a fourth would
otherwise leave the key silently stale.
"""
# Imported here rather than at module scope: `chitai.services.matching` cannot
# be reached without initialising the `chitai.services` package, which imports
# the services, which import this module.
from chitai.services.matching import normalize_title
self.normalized_title = normalize_title(title)
return title
def __repr__(self) -> str: def __repr__(self) -> str:
return f"Book({self.title=!r})" return f"Book({self.title=!r})"
@@ -132,6 +154,24 @@ class Identifier(BigIntBase):
book_id: Mapped[int] = mapped_column(ForeignKey("books.id", ondelete="cascade")) book_id: Mapped[int] = mapped_column(ForeignKey("books.id", ondelete="cascade"))
value: Mapped[str] value: Mapped[str]
# Kept current by `_normalize` below — never assign it directly. Null for an
# identifier that cannot carry a match: a per-build UUID, or an ISBN that fails
# its own checksum.
normalized_value: Mapped[Optional[str]] = mapped_column(index=True)
@validates("name", "value")
def _normalize(self, key: str, value: str) -> str:
"""Recompute `normalized_value` whenever either half of the pair changes."""
from chitai.services.matching import normalize_identifier
name = value if key == "name" else self.name
raw = value if key == "value" else self.value
self.normalized_value = (
normalize_identifier(name, raw) if name and raw else None
)
return value
def __repr__(self): def __repr__(self):
return f"Identifier({self.name!r} : {self.value!r})" return f"Identifier({self.name!r} : {self.value!r})"
@@ -139,6 +179,15 @@ class Identifier(BigIntBase):
class FileMetadata(BigIntBase): class FileMetadata(BigIntBase):
__tablename__ = "file_metadata" __tablename__ = "file_metadata"
__table_args__ = (
# Deliberately not unique. The hash is KOReader's partial MD5, which samples
# 12 KiB of the file, so two genuinely different files can collide — and an
# existing database may already hold duplicates, which a unique index would
# refuse to build over. Duplicate detection pairs it with `size` and treats a
# match as advisory, so this only has to make the lookup cheap.
Index("ix_file_metadata_hash", "hash"),
)
book_id: Mapped[int] = mapped_column(ForeignKey("books.id", ondelete="cascade")) book_id: Mapped[int] = mapped_column(ForeignKey("books.id", ondelete="cascade"))
book: Mapped[Book] = relationship(back_populates="files") book: Mapped[Book] = relationship(back_populates="files")
hash: Mapped[str] hash: Mapped[str]
@@ -0,0 +1,37 @@
from sqlalchemy import ForeignKey, UniqueConstraint
from sqlalchemy.orm import Mapped, mapped_column
from advanced_alchemy.base import BigIntBase
class DuplicateDismissal(BigIntBase):
"""
Two books a reader has said are not the same book.
Title and author matching is probabilistic, so it will keep proposing a second
edition, a translation and a sequel that shares its predecessor's name. A review
screen with no way to disagree with it nags forever, which is how people learn to
ignore a screen.
The pair is stored ordered — `book_a_id` is always the lower id — so "A and B" and
"B and A" are one row and the unique constraint can do its job. Use `pair()` rather
than assigning the columns directly.
"""
__tablename__ = "duplicate_dismissals"
__table_args__ = (UniqueConstraint("book_a_id", "book_b_id"),)
book_a_id: Mapped[int] = mapped_column(
ForeignKey("books.id", ondelete="cascade"), index=True
)
book_b_id: Mapped[int] = mapped_column(
ForeignKey("books.id", ondelete="cascade"), index=True
)
@staticmethod
def pair(first: int, second: int) -> tuple[int, int]:
"""The two book ids in the order this table stores them."""
return (first, second) if first <= second else (second, first)
def __repr__(self) -> str:
return f"DuplicateDismissal({self.book_a_id!r}, {self.book_b_id!r})"
+8
View File
@@ -4,7 +4,15 @@ from .book import (
BookProgressCreate, BookProgressCreate,
BookProgressRead, BookProgressRead,
BooksCreateFromFiles, BooksCreateFromFiles,
BooksUploadResult,
BookMerge,
BookMetadataUpdate, BookMetadataUpdate,
DuplicateBookGroupRead,
DuplicateBookRead,
DuplicateDismissal,
DuplicateFileRead,
FileFingerprint,
PossibleDuplicateRead,
FileMetadataRead, FileMetadataRead,
BookSeriesRead, BookSeriesRead,
) )
+91
View File
@@ -129,6 +129,83 @@ class BooksCreateFromFiles(BaseModel):
model_config = ConfigDict(arbitrary_types_allowed=True) model_config = ConfigDict(arbitrary_types_allowed=True)
class FileFingerprint(BaseModel):
"""What a client can say about a file it has not uploaded yet."""
hash: str
size: int
filename: str = ""
class DuplicateFileRead(BaseModel):
"""A file that was not stored because the library already holds its bytes."""
model_config = ConfigDict(from_attributes=True)
filename: str
hash: str
size: int
library_id: int
# Null when the match was another file in the same upload, which has no row yet.
book_id: int | None = None
book_title: str | None = None
class DuplicateBookRead(BaseModel):
"""
A stored book that may be the same book as another one.
Unlike `DuplicateFileRead` this is a guess: the evidence is metadata two editions
of one work legitimately share. Nothing was refused on the strength of it.
"""
model_config = ConfigDict(from_attributes=True)
book_id: int
title: str
authors: list[str]
library_id: int
cover_image: Path | None = None
# Why it matched: "identifier" and/or "title-author".
matched_on: list[str]
class PossibleDuplicateRead(BaseModel):
"""A book that was imported, together with what it might be a second copy of."""
model_config = ConfigDict(from_attributes=True)
book_id: int
title: str
candidates: list[DuplicateBookRead]
class DuplicateBookGroupRead(BaseModel):
"""Books the library holds that all look like copies of one book."""
books: list[DuplicateBookRead]
class DuplicateDismissal(BaseModel):
"""Two books a reader is saying are not the same book."""
book_a_id: int
book_b_id: int
class BooksUploadResult(BaseModel):
"""The outcome of a multi-file upload: what was created, and what was skipped."""
created: list["BookRead"]
skipped: list[DuplicateFileRead]
# Created, not skipped — these are books that went in and look like something the
# library already had. The reader decides what to do about it.
possible_duplicates: list[PossibleDuplicateRead] = Field(default_factory=list)
class BookMetadataUpdate(BaseModel): class BookMetadataUpdate(BaseModel):
title: str | None = None title: str | None = None
subtitle: str | None = None subtitle: str | None = None
@@ -170,6 +247,20 @@ class BookMetadataUpdate(BaseModel):
return v return v
class BookMerge(BaseModel):
"""
Fold several books into one.
`metadata` is the reader's resolution of the fields the records disagreed on.
Anything it does not name keeps the survivor's value — merging metadata is a
judgement, so nothing is guessed on the caller's behalf.
"""
survivor_id: int
merged_ids: list[int]
metadata: Optional["BookMetadataUpdate"] = None
class BookProgressCreate(BaseModel): class BookProgressCreate(BaseModel):
percentage: float percentage: float
epub_cfi: str | None = None epub_cfi: str | None = None
File diff suppressed because it is too large Load Diff
+23 -3
View File
@@ -1,6 +1,7 @@
import asyncio import asyncio
from pathlib import Path from pathlib import Path
from collections import defaultdict from collections import defaultdict
from chitai.config import settings
from chitai.database.models.library import Library from chitai.database.models.library import Library
from chitai.services import BookService, LibraryService from chitai.services import BookService, LibraryService
from chitai.services.metadata_extractor import Extractor from chitai.services.metadata_extractor import Extractor
@@ -111,14 +112,33 @@ class ConsumeDirectoryWatcher:
"""Process a batch of files.""" """Process a batch of files."""
try: try:
books = await self.book_service.create_many_from_existing_files( result = await self.book_service.create_many_from_existing_files(
list(file_paths), list(file_paths),
self.watch_path / Path(library_slug), self.watch_path / Path(library_slug),
library=await self._get_library(library_slug), library=await self._get_library(library_slug),
) )
print(f"Created {len(books)} books!") print(f"Created {len(result.books)} books!")
if result.duplicates:
print(
f"Moved {len(result.duplicates)} already-stored file(s) "
f"to {settings.duplicate_path}"
)
# Imported all the same — a metadata match is a guess, and there is nobody
# here to ask. The library's duplicates screen is where these get decided.
for possible in result.possible_duplicates:
names = ", ".join(
f"{candidate.title} (#{candidate.book_id})"
for candidate in possible.candidates
)
print(
f"Imported {possible.title!r} (#{possible.book_id}), which may "
f"already be in the library as: {names}"
)
except Exception as e: except Exception as e:
print(f"Error processing batch: {e}") print(f"Error processing batch: {e}")
raise e raise e
@@ -127,6 +127,31 @@ def create_book_filter_dependencies(
# Get base filters first # Get base filters first
filters = create_filter_dependencies(config, dep_defaults) filters = create_filter_dependencies(config, dep_defaults)
# OVERRIDE: id filter typed by the configured id type, not always `str`
#
# advanced_alchemy's `provide_id_filter` annotates `ids` as `list[str]` and
# ignores `config["id_filter"]` entirely, so `?ids=12` reaches the database as
# the string "12" and Postgres refuses `bigint = character varying`. Nothing
# called `?ids=` until the duplicates screen needed to fetch a handful of books
# by id, which is why it went unnoticed.
if id_type := config.get("id_filter"):
id_field = config.get("id_field", "id")
def provide_typed_id_filter(
ids=Parameter(query="ids", default=None, required=False),
) -> CollectionFilter:
return CollectionFilter(field_name=id_field, values=ids)
# Attached as a type object rather than written as an annotation: this module
# has `from __future__ import annotations`, so a written one is stored as the
# string "Optional[list[id_type]]" and resolved against module globals, where
# a local named `id_type` does not exist.
provide_typed_id_filter.__annotations__["ids"] = Optional[list[id_type]]
filters[dep_defaults.ID_FILTER_DEPENDENCY_KEY] = Provide(
provide_typed_id_filter, sync_to_thread=False
)
# OVERRIDE: Custom search filter with trigram search # OVERRIDE: Custom search filter with trigram search
if config.get("search"): if config.get("search"):
search_fields = config.get("search") search_fields = config.get("search")
+207
View File
@@ -0,0 +1,207 @@
# src/chitai/services/matching.py
"""
Normalisation for book-level duplicate detection.
Two copies of one book rarely agree on how it is written down. One says
`The Metamorphosis`, the other `Metamorphosis`; one credits `Kafka, Franz`, the other
`Franz Kafka`; one carries the ISBN-10 and the other the ISBN-13 of the same edition.
These functions reduce each of those to a single key, so the comparison is an equality
test the database can index rather than a similarity score nobody can explain.
Everything here is pure: the keys are computed once and stored on the row (see
`Book.normalized_title`, `Author.normalized_name`, `Identifier.normalized_value`), so
no Postgres extension is needed at query time.
"""
from __future__ import annotations
import re
import unicodedata
from chitai.services.utils import is_valid_isbn, isbn10_to_isbn13
# Asides a title carries that say nothing about which book it is:
# "Frankenstein (Illustrated)", "Dune [Deluxe]".
_BRACKETED = re.compile(r"[(\[{][^)\]}]*[)\]}]")
# Edition and format qualifiers, matched only as a *trailing* run of words. Anchoring
# to the end is what keeps "The Illustrated Man" a book and "Moby Dick Illustrated" a
# format note — a qualifier trails the title, it is never the thing the title is about.
_EDITION_NOISE = re.compile(
r"\s+(?:"
# "2nd edition", but also the compact forms publishers actually print on a
# cover: "2E", "3 Ed", "5e". The number alone is never enough — "Catch 22" is
# a title and must survive.
r"\d+(?:st|nd|rd|th)?\s*(?:edition|edn|ed|e)"
r"|(?:first|second|third|fourth|fifth|sixth|new|revised|expanded|updated|"
r"annotated|illustrated|unabridged|abridged|complete|definitive|deluxe|"
r"anniversary|collectors|international|kindle|paperback|hardcover|hardback|"
r"ebook|audiobook)"
r"(?:\s+(?:and|&)\s+\w+)*"
r"(?:\s+ed(?:ition|n)?)?"
r")$"
)
_LEADING_ARTICLE = re.compile(r"^(?:the|a|an)\s+")
# Anything that is not a letter, a digit or a space, once accents are gone.
_PUNCTUATION = re.compile(r"[^0-9a-z ]+")
_WHITESPACE = re.compile(r"\s+")
# Junk an extractor leaves on the end of a name: the `;` from a `DC:creator` list, the
# `.epub` from a filename the author's name was read out of.
_TRAILING_SEPARATORS = " ;,&/"
_FILE_EXTENSION = re.compile(
r"\.(?:epub|pdf|mobi|azw3?|djvu|fb2|txt|cbz|cbr)$", re.IGNORECASE
)
# `J. R. R.` survives punctuation stripping as three one-letter words; `J.R.R.` as one.
# Joining any run of them makes both `jrr`.
_INITIAL_RUN = re.compile(r"\b(?:[a-z] )+[a-z]\b")
# ISBNs are the same number under several names; everything else keeps its own.
_ISBN_NAMES = {"isbn", "isbn-10", "isbn10", "isbn-13", "isbn13"}
# Generated fresh for every build of a file, so two copies of one book never share one.
# Matching on them would only re-find files the hash check already catches.
_PER_BUILD_NAMES = {"uuid", "urn:uuid"}
# Below this an identifier is not specific enough to be evidence: a Calibre `id` of
# "42" would otherwise pair two unrelated books.
_MIN_IDENTIFIER_LENGTH = 4
def _fold(text: str) -> str:
"""Casefolded, accent-free, punctuation-free, single-spaced."""
decomposed = unicodedata.normalize("NFKD", text.casefold())
unaccented = "".join(c for c in decomposed if not unicodedata.combining(c))
return _WHITESPACE.sub(" ", _PUNCTUATION.sub(" ", unaccented)).strip()
def normalize_title(title: str | None) -> str:
"""
Reduce a title to the key two copies of one book should share.
`Book.subtitle` is already split off by `Extractor.format_book_title`, so only what
is left in the title column is considered here.
Args:
title: The title as it was stored.
Returns:
The comparison key, or an empty string if nothing survives normalisation —
which is the signal not to match on the title at all.
"""
if not title:
return ""
folded = _fold(_BRACKETED.sub(" ", title).replace("&", " and "))
# Repeated because qualifiers stack: "Dune Deluxe Edition Illustrated".
while (trimmed := _EDITION_NOISE.sub("", folded)) != folded:
folded = trimmed
# An article says nothing, but a title that is only an article is not improved by
# having none, and neither is one that noise removal emptied out.
return _LEADING_ARTICLE.sub("", folded, count=1) or folded
def format_author_name(name: str | None) -> str:
"""
Tidy an author's name into the one form the library writes them in.
Distinct from `normalize_author`, which throws away case, accents and spacing to
build a comparison key. This one is what a reader sees, so it keeps everything
that belongs to the name and only removes what an extractor added: a trailing
separator left over from a creator list, a file extension carried in from a
filename, and the `Surname, Given` ordering that EPUBs file names under.
Args:
name: The name as the file or filename gave it.
Returns:
The name to store, or an empty string if there is nothing left of it.
"""
if not name:
return ""
tidied = _FILE_EXTENSION.sub("", name.strip().strip(_TRAILING_SEPARATORS).strip())
if tidied.count(",") == 1:
surname, given = (part.strip() for part in tidied.split(","))
# Only when the part before the comma is a single word. "Dave Thomas, Andy
# Hunt" is two people in one string, and flipping it would invent a third
# person who does not exist. Leaving an unrecognised form alone is the safe
# failure; rewriting it wrongly is not.
if surname and given and " " not in surname:
tidied = f"{given} {surname}"
return _WHITESPACE.sub(" ", tidied).strip()
def normalize_author(name: str | None) -> str:
"""
Reduce an author's name to the key their other books should share.
Deliberately not reduced to surname plus initial: that collides unrelated people,
and a wrong match here is a book pointed at a stranger's shelf.
Args:
name: The name as it was stored, in either `Franz Kafka` or `Kafka, Franz` form.
Returns:
The comparison key, or an empty string if nothing survives normalisation.
"""
if not name:
return ""
# `Kafka, Franz` is one name written backwards. More than one comma is a list, or a
# suffix, and guessing at either does more harm than leaving it alone.
if name.count(",") == 1:
surname, forename = name.split(",")
name = f"{forename.strip()} {surname.strip()}"
folded = _fold(name)
return _INITIAL_RUN.sub(lambda run: run.group().replace(" ", ""), folded)
def normalize_identifier(name: str, value: str) -> str | None:
"""
Reduce one identifier to a `scheme:value` key, if it can carry a match at all.
Args:
name: What kind of identifier it is, as stored.
value: The identifier itself.
Returns:
The key, or None when the identifier is no use for matching: a per-build UUID,
something too short to be evidence, or an ISBN that fails its own checksum.
"""
name = (name or "").strip().casefold()
value = (value or "").strip()
if not name or not value or name in _PER_BUILD_NAMES:
return None
if name in _ISBN_NAMES:
digits = re.sub(r"[^0-9Xx]", "", value).upper()
if not is_valid_isbn(digits):
return None
# One scheme for both forms: a publisher prints whichever it likes, and the
# ISBN-10 and ISBN-13 of an edition are the same number written twice.
isbn = digits if len(digits) == 13 else isbn10_to_isbn13(digits)
return f"isbn:{isbn}" if isbn else None
folded = _fold(value) or value.casefold()
if len(folded) < _MIN_IDENTIFIER_LENGTH:
return None
return f"{name}:{folded}"
+197 -17
View File
@@ -3,7 +3,6 @@
# TODO: Code is a mess. Clean it up and add docstrings # TODO: Code is a mess. Clean it up and add docstrings
# Standard library # Standard library
from abc import ABC, abstractmethod
import datetime import datetime
from pathlib import Path from pathlib import Path
from io import BytesIO from io import BytesIO
@@ -31,6 +30,147 @@ from chitai.services.utils import (
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
# Identifier schemes an EPUB can declare, mapped onto the names the rest of the app
# uses. A scheme arrives either as an `opf:scheme` attribute or as a prefix on the
# value itself (`urn:isbn:…`, `calibre:…`), and the two say the same thing.
_IDENTIFIER_SCHEMES = {
"isbn": "isbn",
"isbn10": "isbn-10",
"isbn-10": "isbn-10",
"isbn13": "isbn-13",
"isbn-13": "isbn-13",
"uuid": "uuid",
"calibre": "calibre",
"doi": "doi",
"asin": "asin",
"amazon": "asin",
"mobi-asin": "asin",
"google": "google",
"goodreads": "goodreads",
}
# `scheme:rest`, with an optional `urn:` in front of it.
_SCHEME_PREFIX = re.compile(r"^(?:urn:)?([A-Za-z][A-Za-z0-9.-]*):(.+)$")
_UUID = re.compile(r"^[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}$", re.IGNORECASE)
def parse_identifier(value: str, scheme: str | None = None) -> tuple[str, str] | None:
"""
Work out what one raw identifier is, and what it is worth storing as.
EPUBs write the same ISBN as `9780486282114`, `978-0-486-28211-4` and
`urn:isbn:978-0-486-28211-4`, and carry plenty of identifiers that are not ISBNs
at all. Validating the string verbatim keeps only the first form and throws the
rest away, so normalise first and name whatever survives.
Args:
value: The identifier as the file wrote it.
scheme: What the file said it is, if it said anything — an `opf:scheme`
attribute. A prefix on the value takes precedence over this.
Returns:
The `(name, value)` to store, or `None` when there is nothing usable: an
empty value, or one declared to be an ISBN that fails its own checksum.
"""
value = (value or "").strip()
if not value:
return None
name = _IDENTIFIER_SCHEMES.get((scheme or "").strip().casefold())
# An unrecognised prefix is part of the value rather than a scheme —
# "http://example.com/book" is not an identifier called "http".
if (match := _SCHEME_PREFIX.match(value)) and (
prefixed := _IDENTIFIER_SCHEMES.get(match.group(1).casefold())
):
name = prefixed
value = match.group(2).strip()
if name is None or name.startswith("isbn"):
digits = re.sub(r"[^0-9Xx]", "", value).upper()
if is_valid_isbn(digits):
return f"isbn-{len(digits)}", digits
# Something that announced itself as an ISBN and is not one carries no
# information: storing it would link the reader to a page that does not exist.
if name is not None:
return None
return name or ("uuid" if _UUID.match(value) else "id"), value
# Numbered editions, in the forms covers and catalogue records actually use:
# "3rd Edition", "2E", "4e", "8_e", "/6e", "(2nd edition)", "Third International Edition".
_ORDINAL_WORDS = {
"first": 1, "second": 2, "third": 3, "fourth": 4, "fifth": 5, "sixth": 6,
"seventh": 7, "eighth": 8, "ninth": 9, "tenth": 10, "eleventh": 11, "twelfth": 12,
}
# Words that sit between the number and "Edition" and belong to the same statement.
_EDITION_QUALIFIER = (
r"(?:international|global|revised|updated|expanded|anniversary|deluxe|student|"
r"instructors?|annotated|illustrated|reprint)"
)
_EDITION = re.compile(
rf"""
[\s,;:/\-–—(\[]+ # the separator the statement hangs off
(?:
(?P<num>\d{{1,2}})\s*(?:st|nd|rd|th)?[\s_]*
(?:{_EDITION_QUALIFIER}\s+)*(?:edition\b|edn\b|ed\b\.?|e\b)
| (?P<word>{"|".join(_ORDINAL_WORDS)})\s+
(?:{_EDITION_QUALIFIER}\s+)*(?:edition\b|edn\b|ed\b\.?)
)
[\s)\]]* # and its closing bracket, if it had one
""",
re.IGNORECASE | re.VERBOSE,
)
def split_edition(title: str | None) -> tuple[str | None, int | None]:
"""
Separate a numbered edition statement from the title it is written into.
"Fluent Python, 2nd Edition" is one book with a field for the edition, not a
title. Left in place it also splits the library: the second edition never looks
like the first, and neither matches the copy whose file simply did not mention it.
The number is what makes this safe. Nothing is stripped without one, so
"Catch 22" and "Blade Runner 2049" keep their numbers and "Global Edition"
which is a variant, not a numbered edition, and has nowhere to go in an
integer column — is left in the title where it can still be read.
Args:
title: The title as the file or filename gave it.
Returns:
The title without the edition statement, and the edition number. The title
unchanged and None when there is no numbered edition in it, or when removing
it would leave nothing behind.
"""
if not title:
return title, None
if (match := _EDITION.search(title)) is None:
return title, None
edition = (
int(match["num"]) if match["num"] else _ORDINAL_WORDS[match["word"].casefold()]
)
stripped = _EDITION.sub(" ", title)
stripped = re.sub(r"\s{2,}", " ", stripped)
stripped = re.sub(r"\s+([,;:.!?])", r"\1", stripped) # "Works : What" → "Works: What"
stripped = stripped.strip(" ,;:-–—/")
# A title that is only an edition statement is not improved by having none.
if not stripped:
return title, None
return stripped, edition
class FileExtractor(Protocol): class FileExtractor(Protocol):
@classmethod @classmethod
async def extract_metadata( async def extract_metadata(
@@ -54,15 +194,36 @@ class Extractor:
# EPUB tends to give better metadata results over pdf # EPUB tends to give better metadata results over pdf
sorted_files = sorted(files, key=lambda f: Extractor._get_file_priority(f)) sorted_files = sorted(files, key=lambda f: Extractor._get_file_priority(f))
# Identifiers accumulate across formats instead of replacing each other. Every
# other field is a single value where the later, better-trusted format simply
# wins, but identifiers are a *collection*: a book holding an EPUB and a PDF
# genuinely carries what both of them declare, and merging the dict wholesale
# threw away everything the earlier format found. An EPUB that declares an
# ASIN, a Google volume id and a Calibre id kept none of them once a PDF
# contributed a single ISBN.
identifiers: dict[str, str] = {}
for file in sorted_files: for file in sorted_files:
match get_file_extension(file): match get_file_extension(file):
case "epub": case "epub":
metadata = metadata | await EpubExtractor.extract_metadata(file) extracted = await EpubExtractor.extract_metadata(file)
case "pdf": case "pdf":
metadata = metadata | await PdfExtractor.extract_metadata(file) extracted = await PdfExtractor.extract_metadata(file)
case _: case _:
break break
# First writer wins per name, and the files are already ordered by how
# far their metadata can be trusted. A `dc:identifier` the publisher
# declared outranks an ISBN scraped out of a PDF's copyright page, which
# routinely prints the ISBNs of other formats and older editions too.
for name, value in (extracted.pop("identifiers", None) or {}).items():
identifiers.setdefault(name, value)
metadata = metadata | extracted
if identifiers:
metadata["identifiers"] = identifiers
# Get metadata from file names # Get metadata from file names
for file in files: for file in files:
metadata = FilenameExtractor.extract_metadata(file) | metadata metadata = FilenameExtractor.extract_metadata(file) | metadata
@@ -75,7 +236,14 @@ class Extractor:
# format the title # format the title
if metadata.get('title', None): if metadata.get('title', None):
title, subtitle = Extractor.format_book_title(metadata["title"]) # Before the subtitle split, so the edition cannot be mistaken for one:
# "How Linux Works, 3rd Edition: What Every Superuser Should Know" has to
# lose the edition first for the colon count to mean anything.
title, edition = split_edition(metadata["title"])
if edition is not None:
metadata.setdefault("edition", edition)
title, subtitle = Extractor.format_book_title(title)
metadata["title"] = title metadata["title"] = title
metadata["subtitle"] = subtitle metadata["subtitle"] = subtitle
@@ -236,7 +404,7 @@ class PdfExtractor(FileExtractor):
try: try:
return datetime.datetime.strptime(date_portion, "%Y%m%d").date() return datetime.datetime.strptime(date_portion, "%Y%m%d").date()
except Exception as e: except Exception:
return None return None
@classmethod @classmethod
@@ -363,15 +531,23 @@ class EpubExtractor(FileExtractor):
@classmethod @classmethod
def _extract_identifiers(cls, epub: epub.EpubBook) -> dict[str, str]: def _extract_identifiers(cls, epub: epub.EpubBook) -> dict[str, str]:
"""
Every `DC:identifier` the file carries, keyed by what kind of thing it is.
Non-ISBN identifiers are kept: `Identifier` is a free-form name/value pair, so
a Calibre id or an ASIN costs nothing to store and is one more thing two copies
of a book can be recognised by.
"""
identifiers = {} identifiers = {}
for id in epub.get_metadata("DC", "identifier"): for value, attributes in epub.get_metadata("DC", "identifier"):
if is_valid_isbn(id[0]): scheme = None
if len(id[0]) == 13: if isinstance(attributes, dict):
identifiers.update({"isbn-13": id[0]}) scheme = attributes.get("opf:scheme") or attributes.get("scheme")
elif len(id[0]) == 10: if (parsed := parse_identifier(value, scheme)) is not None:
identifiers.update({"isbn-10": id[0]}) name, parsed_value = parsed
identifiers[name] = parsed_value
return identifiers return identifiers
@@ -380,7 +556,7 @@ class EpubExtractor(FileExtractor):
try: try:
return epub.get_metadata("DC", "description")[0][0] return epub.get_metadata("DC", "description")[0][0]
except: except Exception:
return None return None
@classmethod @classmethod
@@ -389,15 +565,15 @@ class EpubExtractor(FileExtractor):
date_str = epub.get_metadata("DC", "date")[0][0].split("T")[0] date_str = epub.get_metadata("DC", "date")[0][0].split("T")[0]
return datetime.date.fromisoformat(date_str) return datetime.date.fromisoformat(date_str)
except: except Exception:
return None return None
@classmethod @classmethod
def _extract_publisher(cls, epub: epub.EpubBook) -> str | None: def _extract_publisher(cls, epub: epub.EpubBook) -> str | None:
try: try:
epub.get_metadata("DC", "publisher")[0][0] return epub.get_metadata("DC", "publisher")[0][0]
except: except Exception:
return None return None
@classmethod @classmethod
@@ -421,7 +597,7 @@ class EpubExtractor(FileExtractor):
cover_item = epub.get_item_with_id(cover_id) cover_item = epub.get_item_with_id(cover_id)
if cover_item: if cover_item:
return PIL.Image.open(BytesIO(cover_item.content)) return PIL.Image.open(BytesIO(cover_item.content))
except Exception as e: except Exception:
pass # Fallback to next strategy pass # Fallback to next strategy
# Strategy 2: Search image filenames for "cover" keyword # Strategy 2: Search image filenames for "cover" keyword
@@ -489,7 +665,11 @@ class FilenameExtractor(FileExtractor):
elif isinstance(input, Path): elif isinstance(input, Path):
filename = get_filename(input, ext=False) filename = get_filename(input, ext=False)
elif isinstance(input, UploadFile): elif isinstance(input, UploadFile):
filename = Path(input.filename).name # `.stem`, not `.name`: the extension is not part of the metadata, and
# this is the browser upload path, so keeping it is how a library fills
# up with authors called "Sam Newman.epub". The other two branches have
# always stripped it.
filename = Path(input.filename).stem
else: else:
raise ValueError("Input type not supported") raise ValueError("Input type not supported")
+84 -9
View File
@@ -3,13 +3,11 @@
# Standard library # Standard library
from __future__ import annotations from __future__ import annotations
import errno
import hashlib import hashlib
from pathlib import Path from pathlib import Path
import shutil import shutil
from typing import TYPE_CHECKING, BinaryIO from typing import BinaryIO
if TYPE_CHECKING:
from hashlib import _Hash
# Third-party libraries # Third-party libraries
import PIL import PIL
@@ -32,6 +30,9 @@ KO_STEP = 1024
KO_SAMPLE_SIZE = 1024 KO_SAMPLE_SIZE = 1024
KO_INDICES = range(-1, 11) # -1 to 10 inclusive KO_INDICES = range(-1, 11) # -1 to 10 inclusive
# How much is read at a time while hashing.
HASH_CHUNK_SIZE = 262144 # 256 KiB
def _lshift32(val: int, shift: int) -> int: def _lshift32(val: int, shift: int) -> int:
""" """
@@ -100,10 +101,9 @@ async def calculate_koreader_hash(file_path: Path) -> str:
offsets = _get_koreader_offsets() offsets = _get_koreader_offsets()
file_pos = 0 file_pos = 0
chunk_size = 262144 # 256 KiB
async with aiofiles.open(file_path, "rb") as f: async with aiofiles.open(file_path, "rb") as f:
while chunk := await f.read(chunk_size): while chunk := await f.read(HASH_CHUNK_SIZE):
_partial_md5_from_chunk(chunk, hasher, offsets, file_pos) _partial_md5_from_chunk(chunk, hasher, offsets, file_pos)
file_pos += len(chunk) file_pos += len(chunk)
@@ -132,6 +132,49 @@ class StreamingHasher:
"""Return the final hash.""" """Return the final hash."""
return self.hasher.hexdigest() return self.hasher.hexdigest()
@property
def size(self) -> int:
"""Total number of bytes fed in so far."""
return self.position
async def fingerprint_upload(file: UploadFile) -> tuple[str, int]:
"""
Calculate the hash and byte size of an uploaded file without storing it.
Duplicate detection has to answer before anything is written to the library, so
the file is read here and rewound for whoever writes it afterwards.
Args:
file: The uploaded file to read.
Returns:
The file's `(hash, size)` pair.
"""
hasher = StreamingHasher()
await file.seek(0)
while chunk := await file.read(HASH_CHUNK_SIZE):
hasher.update(chunk)
await file.seek(0)
return hasher.hexdigest(), hasher.size
async def fingerprint_file(file_path: Path) -> tuple[str, int]:
"""
Calculate the hash and byte size of a file already on disk.
Args:
file_path: Path to the file to read.
Returns:
The file's `(hash, size)` pair.
"""
stats = await aios.stat(file_path)
return await calculate_koreader_hash(file_path), stats.st_size
################################## ##################################
# Filesystem related utilities # # Filesystem related utilities #
################################## ##################################
@@ -172,7 +215,7 @@ async def create_directory(dir_path: Path | str) -> None:
async def move_file(src_path: Path, dest_path: Path, create_dirs=True) -> None: async def move_file(src_path: Path, dest_path: Path, create_dirs=True) -> None:
""" """
Move a file from source to destination asynchronously. Move a file from source to destination asynchronously.
Args: Args:
source_path: Path to the source file source_path: Path to the source file
destination_path: Path to the destination file destination_path: Path to the destination file
@@ -184,7 +227,16 @@ async def move_file(src_path: Path, dest_path: Path, create_dirs=True) -> None:
if dest_dir: # Only create if there's a directory path if dest_dir: # Only create if there's a directory path
await aios.makedirs(dest_dir, exist_ok=True) await aios.makedirs(dest_dir, exist_ok=True)
await aios.rename(src_path, dest_path) try:
await aios.rename(src_path, dest_path)
except OSError as exc:
if exc.errno != errno.EXDEV:
raise
# Source and destination are on different filesystems, which rename cannot
# cross. Libraries, the consume directory and the duplicates directory are
# all configured separately, so they can easily be separate mounts.
shutil.move(str(src_path), str(dest_path))
async def move_dir_contents(source_dir: Path | str, target_dir: Path | str) -> None: async def move_dir_contents(source_dir: Path | str, target_dir: Path | str) -> None:
""" """
@@ -432,7 +484,7 @@ def is_valid_isbn(isbn: str) -> bool:
return is_valid_isbn13(isbn) return is_valid_isbn13(isbn)
else: else:
return False return False
except: except Exception:
return False return False
@@ -457,6 +509,29 @@ def is_valid_isbn10(isbn: str) -> bool:
return str(check_digit) == isbn[-1] or (check_digit == 10 and isbn[-1] in "Xx") return str(check_digit) == isbn[-1] or (check_digit == 10 and isbn[-1] in "Xx")
def isbn10_to_isbn13(isbn: str) -> str | None:
"""
Convert an ISBN-10 to the ISBN-13 naming the same edition.
The two are the same number written twice: prefix `978`, drop the ISBN-10 check
digit, recompute the check digit under the ISBN-13 rule. Matching only works if
both forms collapse onto one, since a publisher may print either.
Args:
isbn: A 10-character ISBN, digits and an optional trailing `X` only.
Returns:
The equivalent ISBN-13, or None if the input is not a valid ISBN-10.
"""
if not is_valid_isbn(isbn) or len(isbn) != 10:
return None
digits = f"978{isbn[:9]}"
total = sum(int(digit) * (1 if i % 2 == 0 else 3) for i, digit in enumerate(digits))
return f"{digits}{(10 - total % 10) % 10}"
def is_valid_isbn13(isbn: str) -> bool: def is_valid_isbn13(isbn: str) -> bool:
""" """
Validate an ISBN-13 number using its check digit. Validate an ISBN-13 number using its check digit.
+7 -1
View File
@@ -40,7 +40,13 @@ pytest_plugins = [
@pytest.fixture(autouse=True) @pytest.fixture(autouse=True)
def _patch_settings(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: def _patch_settings(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setattr(settings, "book_cover_path", f"{tmp_path}/covers") # PIL will not create the directory it is asked to save into, so anything that
# imports a file carrying a cover needs it to exist first.
covers = tmp_path / "covers"
covers.mkdir()
monkeypatch.setattr(settings, "book_cover_path", str(covers))
monkeypatch.setattr(settings, "duplicate_path", str(tmp_path / "duplicates"))
@pytest.fixture(name="engine") @pytest.fixture(name="engine")
+346 -4
View File
@@ -37,7 +37,8 @@ from pathlib import Path
( (
Path("tests/data_files/Calculus Made Easy - Silvanus Thompson.pdf"), Path("tests/data_files/Calculus Made Easy - Silvanus Thompson.pdf"),
2, 2,
"The Project Gutenberg eBook #33283: Calculus Made Easy, 2nd Edition", # The ", 2nd Edition" is split off into `edition`, not kept in the title.
"The Project Gutenberg eBook #33283: Calculus Made Easy",
["Silvanus Phillips Thompson"], ["Silvanus Phillips Thompson"],
), ),
], ],
@@ -208,6 +209,21 @@ async def test_get_book_file(
assert downloaded_content == file_content assert downloaded_content == file_content
async def test_list_books_by_id(populated_authenticated_client: AsyncClient) -> None:
"""
`?ids=` has to reach the database as integers.
advanced_alchemy's stock id filter annotates the parameter as `list[str]` whatever
the configured id type, so the ids arrived as strings and Postgres refused to
compare a bigint primary key against them. Nothing called it until a screen needed
to fetch a handful of books by id.
"""
response = await populated_authenticated_client.get("/books?ids=1&ids=2&pageSize=10")
assert response.status_code == 200
assert sorted(book["id"] for book in response.json()["items"]) == [1, 2]
async def test_get_book_by_id(populated_authenticated_client: AsyncClient) -> None: async def test_get_book_by_id(populated_authenticated_client: AsyncClient) -> None:
"""Test retrieving a specific book by ID.""" """Test retrieving a specific book by ID."""
@@ -366,7 +382,8 @@ async def test_create_multiple_books_from_directory(
assert response.status_code == 201 assert response.status_code == 201
data = response.json() data = response.json()
assert len(data.get("items") or data.get("data")) >= 1 assert len(data["created"]) == 2
assert data["skipped"] == []
async def test_create_books_from_parent_directory_keeps_embedded_title( async def test_create_books_from_parent_directory_keeps_embedded_title(
@@ -396,7 +413,7 @@ async def test_create_books_from_parent_directory_keeps_embedded_title(
assert response.status_code == 201 assert response.status_code == 201
books = response.json()["items"] books = response.json()["created"]
assert len(books) == 1 assert len(books) == 1
assert books[0]["title"] == "Metamorphosis" assert books[0]["title"] == "Metamorphosis"
@@ -419,11 +436,336 @@ async def test_create_books_groups_formats_within_one_folder(
assert response.status_code == 201 assert response.status_code == 201
books = response.json()["items"] books = response.json()["created"]
assert len(books) == 1 assert len(books) == 1
assert len(books[0]["files"]) == 2 assert len(books[0]["files"]) == 2
class TestDuplicateHandling:
"""A file the library already holds must not be stored a second time."""
epub_path = Path("tests/data_files/Metamorphosis - Franz Kafka.epub")
def upload(self, name: str | None = None) -> list[tuple[str, tuple]]:
return [
(
"files",
(
name or self.epub_path.name,
self.epub_path.read_bytes(),
"application/epub+zip",
),
)
]
async def test_bulk_upload_reports_skipped_files(
self, authenticated_client: AsyncClient
) -> None:
"""Re-dropping a folder must import what is new and name what was not."""
first = await authenticated_client.post(
"/books/fromFiles?library_id=1", files=self.upload()
)
assert first.status_code == 201
created = first.json()["created"][0]
second = await authenticated_client.post(
"/books/fromFiles?library_id=1", files=self.upload()
)
assert second.status_code == 201
result = second.json()
assert result["created"] == []
assert len(result["skipped"]) == 1
skipped = result["skipped"][0]
assert skipped["filename"] == self.epub_path.name
assert skipped["book_id"] == created["id"]
assert skipped["book_title"] == created["title"]
async def test_bulk_upload_can_be_forced(
self, authenticated_client: AsyncClient
) -> None:
await authenticated_client.post(
"/books/fromFiles?library_id=1", files=self.upload()
)
response = await authenticated_client.post(
"/books/fromFiles?library_id=1&allow_duplicates=true", files=self.upload()
)
assert response.status_code == 201
assert len(response.json()["created"]) == 1
assert response.json()["skipped"] == []
async def test_single_book_create_conflicts(
self, authenticated_client: AsyncClient
) -> None:
"""Naming files deliberately earns a refusal rather than a silent drop."""
await authenticated_client.post(
"/books/fromFiles?library_id=1", files=self.upload()
)
response = await authenticated_client.post(
"/books?library_id=1", files=self.upload(), data={"library_id": 1}
)
assert response.status_code == 409
assert response.json()["extra"][0]["filename"] == self.epub_path.name
forced = await authenticated_client.post(
"/books?library_id=1&allow_duplicates=true",
files=self.upload(),
data={"library_id": 1},
)
assert forced.status_code == 201
async def test_adding_another_books_file_conflicts(
self, authenticated_client: AsyncClient
) -> None:
created = await authenticated_client.post(
"/books/fromFiles?library_id=1", files=self.upload()
)
book_id = created.json()["created"][0]["id"]
other = await authenticated_client.post(
"/books?library_id=1",
files=[
(
"files",
(
"war.epub",
Path("tests/data_files/The Art of War - Sun Tzu.epub").read_bytes(),
"application/epub+zip",
),
)
],
data={"library_id": 1},
)
other_id = other.json()["id"]
response = await authenticated_client.post(
f"/books/{book_id}/files",
files=[
(
"files",
(
"war.epub",
Path("tests/data_files/The Art of War - Sun Tzu.epub").read_bytes(),
"application/epub+zip",
),
)
],
)
assert response.status_code == 409
assert response.json()["extra"][0]["book_id"] == other_id
async def test_resending_a_books_own_file_changes_nothing(
self, authenticated_client: AsyncClient
) -> None:
created = await authenticated_client.post(
"/books/fromFiles?library_id=1", files=self.upload()
)
book_id = created.json()["created"][0]["id"]
response = await authenticated_client.post(
f"/books/{book_id}/files", files=self.upload()
)
assert response.status_code == 201
assert len(response.json()["files"]) == 1
async def test_duplicates_can_be_checked_before_uploading(
self, authenticated_client: AsyncClient
) -> None:
"""The pre-flight check answers from hashes alone, with no file sent."""
created = await authenticated_client.post(
"/books/fromFiles?library_id=1", files=self.upload()
)
book = created.json()["created"][0]
stored = book["files"][0]
response = await authenticated_client.post(
"/books/duplicate-files?library_id=1",
json=[
{
"hash": stored["hash"],
"size": stored["size"],
"filename": "local-copy.epub",
},
{"hash": stored["hash"], "size": stored["size"] + 1},
{"hash": "0" * 32, "size": 1234},
],
)
assert response.status_code == 200
matches = response.json()
assert len(matches) == 1
assert matches[0]["filename"] == "local-copy.epub"
assert matches[0]["book_id"] == book["id"]
@pytest.mark.asyncio
class TestDuplicateBooks:
"""A second copy of a book is imported and reported, never refused."""
epub_path = Path("tests/data_files/Metamorphosis - Franz Kafka.epub")
def upload(self, name: str, pad: bool = False) -> list[tuple[str, tuple]]:
"""
The fixture, optionally padded so it is a different file and the same book.
Padding the archive changes its size and its sampled hash without disturbing
the metadata, which is exactly the case the file-level check cannot see.
"""
data = self.epub_path.read_bytes() + (b"\0" * 64 if pad else b"")
return [("files", (name, data, "application/epub+zip"))]
async def test_a_second_edition_is_created_and_reported(
self, authenticated_client: AsyncClient
) -> None:
first = await authenticated_client.post(
"/books/fromFiles?library_id=1", files=self.upload("first.epub")
)
original = first.json()["created"][0]
second = await authenticated_client.post(
"/books/fromFiles?library_id=1", files=self.upload("second.epub", pad=True)
)
assert second.status_code == 201
body = second.json()
# Created, not skipped: a metadata match is a guess, and refusing a legitimate
# second edition costs more than a note does.
assert len(body["created"]) == 1
assert body["skipped"] == []
assert len(body["possible_duplicates"]) == 1
possible = body["possible_duplicates"][0]
assert possible["book_id"] == body["created"][0]["id"]
candidate = possible["candidates"][0]
assert candidate["book_id"] == original["id"]
assert candidate["title"] == original["title"]
assert candidate["authors"] == ["Franz Kafka"]
assert "title-author" in candidate["matched_on"]
async def test_the_review_screen_groups_them(
self, authenticated_client: AsyncClient
) -> None:
first = await authenticated_client.post(
"/books/fromFiles?library_id=1", files=self.upload("first.epub")
)
second = await authenticated_client.post(
"/books/fromFiles?library_id=1", files=self.upload("second.epub", pad=True)
)
book_ids = sorted(
[first.json()["created"][0]["id"], second.json()["created"][0]["id"]]
)
response = await authenticated_client.get("/books/duplicate-books?library_id=1")
assert response.status_code == 200
groups = response.json()
assert len(groups) == 1
assert [book["book_id"] for book in groups[0]["books"]] == book_ids
async def test_a_dismissed_group_stays_dismissed(
self, authenticated_client: AsyncClient
) -> None:
first = await authenticated_client.post(
"/books/fromFiles?library_id=1", files=self.upload("first.epub")
)
second = await authenticated_client.post(
"/books/fromFiles?library_id=1", files=self.upload("second.epub", pad=True)
)
pair = {
"book_a_id": first.json()["created"][0]["id"],
"book_b_id": second.json()["created"][0]["id"],
}
dismissed = await authenticated_client.post(
"/books/duplicate-books/dismissals", json=pair
)
assert dismissed.status_code == 204
response = await authenticated_client.get("/books/duplicate-books?library_id=1")
assert response.json() == []
restored = await authenticated_client.delete(
"/books/duplicate-books/dismissals"
f"?book_a_id={pair['book_b_id']}&book_b_id={pair['book_a_id']}"
)
assert restored.status_code == 204
response = await authenticated_client.get("/books/duplicate-books?library_id=1")
assert len(response.json()) == 1
async def test_two_books_merge_into_one(
self, authenticated_client: AsyncClient
) -> None:
"""The survivor keeps its id and gains the other's file; the other is gone."""
first = await authenticated_client.post(
"/books/fromFiles?library_id=1", files=self.upload("first.epub")
)
second = await authenticated_client.post(
"/books/fromFiles?library_id=1", files=self.upload("second.epub", pad=True)
)
keep = first.json()["created"][0]
fold = second.json()["created"][0]
response = await authenticated_client.post(
"/books/merge?library_id=1",
json={
"survivor_id": keep["id"],
"merged_ids": [fold["id"]],
"metadata": {"title": "Metamorphosis", "edition": 2},
},
)
assert response.status_code == 201
merged = response.json()
assert merged["id"] == keep["id"]
assert merged["edition"] == 2
assert len(merged["files"]) == 2
# The folded record is gone, and the group it formed with it.
assert (await authenticated_client.get(f"/books/{fold['id']}")).status_code == 404
groups = await authenticated_client.get("/books/duplicate-books?library_id=1")
assert groups.json() == []
async def test_merging_an_unknown_book_is_refused(
self, authenticated_client: AsyncClient
) -> None:
created = await authenticated_client.post(
"/books/fromFiles?library_id=1", files=self.upload("first.epub")
)
keep = created.json()["created"][0]["id"]
response = await authenticated_client.post(
"/books/merge?library_id=1",
json={"survivor_id": keep, "merged_ids": [9999]},
)
assert response.status_code == 400
async def test_dismissing_an_unknown_book_is_refused(
self, authenticated_client: AsyncClient
) -> None:
response = await authenticated_client.post(
"/books/duplicate-books/dismissals",
json={"book_a_id": 1, "book_b_id": 9999},
)
assert response.status_code == 400
# NOTE: the multi-book ZIP download is covered at the service level, in # NOTE: the multi-book ZIP download is covered at the service level, in
# tests/unit/test_services/test_book_service.py. Driving `/books/download` through # tests/unit/test_services/test_book_service.py. Driving `/books/download` through
# AsyncTestClient hangs in fixture teardown: it is the only `Stream` endpoint in the # AsyncTestClient hangs in fixture teardown: it is the only `Stream` endpoint in the
+233
View File
@@ -0,0 +1,233 @@
"""Tests for the normalization behind book-level duplicate detection."""
import pytest
from chitai.services.matching import (
format_author_name,
normalize_author,
normalize_identifier,
normalize_title,
)
from chitai.services.metadata_extractor import parse_identifier
from chitai.services.utils import isbn10_to_isbn13
class TestNormalizeTitle:
"""Two copies of one book rarely agree on how the title is written."""
@pytest.mark.parametrize(
("title", "expected"),
[
("The Metamorphosis", "metamorphosis"),
("Metamorphosis", "metamorphosis"),
("METAMORPHOSIS", "metamorphosis"),
("A Tale of Two Cities", "tale of two cities"),
("An Enquiry", "enquiry"),
# Accents, punctuation and ampersands are spelling, not identity.
("Les Misérables", "les miserables"),
("Moby Dick; Or, The Whale", "moby dick or the whale"),
("Sense & Sensibility", "sense and sensibility"),
# Bracketed asides and trailing edition noise say nothing about the book.
("Frankenstein (Illustrated)", "frankenstein"),
("Frankenstein [Kindle Edition]", "frankenstein"),
("Frankenstein, 2nd Edition", "frankenstein"),
# The compact forms a cover actually carries.
("Building Microservices, 2E", "building microservices"),
("Building Microservices 2e", "building microservices"),
("Frankenstein 3 Ed", "frankenstein"),
("Dungeons & Dragons 5e", "dungeons and dragons"),
("Frankenstein Revised Edition", "frankenstein"),
("Dune Deluxe Edition Illustrated", "dune"),
("", ""),
(None, ""),
],
)
def test_titles_that_should_agree(self, title: str | None, expected: str) -> None:
assert normalize_title(title) == expected
def test_a_qualifier_that_is_the_title_survives(self) -> None:
"""A trailing qualifier is noise; the same word at the front is the book."""
assert normalize_title("The Illustrated Man") == "illustrated man"
def test_normalization_never_empties_a_title(self) -> None:
"""An article-only title is not improved by having no article left."""
assert normalize_title("The") == "the"
@pytest.mark.parametrize(
"title", ["Catch 22", "Fahrenheit 451", "Blade Runner 2049", "1984", "Apollo 13"]
)
def test_a_number_is_not_an_edition(self, title: str) -> None:
"""Edition stripping keys on the `e`; a bare number is part of the title."""
assert normalize_title(title) == title.casefold()
class TestNormalizeAuthor:
"""One person, written down several ways."""
@pytest.mark.parametrize(
("name", "expected"),
[
("Franz Kafka", "franz kafka"),
("Kafka, Franz", "franz kafka"),
("KAFKA, FRANZ", "franz kafka"),
("Émile Zola", "emile zola"),
("Doyle, Arthur Conan", "arthur conan doyle"),
# Runs of initials are joined, so spacing them out changes nothing.
("J.R.R. Tolkien", "jrr tolkien"),
("J. R. R. Tolkien", "jrr tolkien"),
("JRR Tolkien", "jrr tolkien"),
("", ""),
(None, ""),
],
)
def test_names_that_should_agree(self, name: str | None, expected: str) -> None:
assert normalize_author(name) == expected
def test_two_people_are_not_reduced_together(self) -> None:
"""Surname plus initial would collide unrelated writers; it is not used."""
assert normalize_author("Charles Dickens") != normalize_author("Colin Dexter")
def test_a_list_is_left_alone(self) -> None:
"""More than one comma is a list or a suffix, and guessing does more harm."""
assert normalize_author("Smith, John, Jr.") == "smith john jr"
class TestFormatAuthorName:
"""What gets stored and shown, as opposed to what gets compared."""
@pytest.mark.parametrize(
("written", "expected"),
[
# A leftover separator from a `DC:creator` list.
("Newman, Sam;", "Sam Newman"),
("Sam Newman ", "Sam Newman"),
(" Dan Vanderkam ", "Dan Vanderkam"),
# `Surname, Given` is how EPUBs file a name, not how anyone reads it.
("Kleppmann, Martin", "Martin Kleppmann"),
("Huxley, Aldous", "Aldous Huxley"),
("Liu, Cixin", "Cixin Liu"),
# An extension carried in from the filename the name was read out of.
("Sam Newman.epub", "Sam Newman"),
("Franz Kafka.mobi", "Franz Kafka"),
("Brian W. Kernighan.epub", "Brian W. Kernighan"),
("", ""),
(None, ""),
],
)
def test_names_are_tidied(self, written: str | None, expected: str) -> None:
assert format_author_name(written) == expected
@pytest.mark.parametrize(
"written",
[
# Two people in one string. Flipping it would invent a third.
"Dave Thomas, Andy Hunt",
"Mark Richards, Neal Ford",
# A compound surname is not recognised, and is left alone rather than
# rearranged wrongly.
"García Márquez, Gabriel",
],
)
def test_an_unrecognised_form_is_left_alone(self, written: str) -> None:
assert format_author_name(written) == written
def test_case_and_accents_belong_to_the_author(self) -> None:
"""Tidying removes what an extractor added; it does not correct spelling."""
assert format_author_name("Michał Płachta.epub") == "Michał Płachta"
assert format_author_name("Steve McConnell") == "Steve McConnell"
@pytest.mark.parametrize(
"written", ["Newman, Sam;", "Sam Newman.epub", "Kleppmann, Martin"]
)
def test_tidying_is_idempotent(self, written: str) -> None:
"""`unique_filter` tidies a name that may already be tidy; it must not drift."""
once = format_author_name(written)
assert format_author_name(once) == once
class TestNormalizeIdentifier:
"""Identifiers only help if the same edition produces the same key."""
def test_isbn_10_and_isbn_13_are_one_key(self) -> None:
assert normalize_identifier("isbn-10", "0486282112") == "isbn:9780486282114"
assert normalize_identifier("isbn-13", "9780486282114") == "isbn:9780486282114"
@pytest.mark.parametrize(
"written", ["978-0-486-28211-4", "978 0 486 28211 4", "9780486282114"]
)
def test_formatting_is_not_part_of_an_isbn(self, written: str) -> None:
assert normalize_identifier("isbn", written) == "isbn:9780486282114"
def test_an_isbn_that_fails_its_checksum_is_no_evidence(self) -> None:
assert normalize_identifier("isbn-13", "9780486282115") is None
def test_uuids_are_refused(self) -> None:
"""Generated per build, so they only re-find what the hash check catches."""
assert normalize_identifier("uuid", "3f2b1c4e-1111-2222-3333-444455556666") is None
assert normalize_identifier("urn:uuid", "3f2b1c4e-1111-2222-3333-444455556666") is None
def test_other_schemes_keep_their_own_key(self) -> None:
assert normalize_identifier("asin", "B000FC0PDA") == "asin:b000fc0pda"
assert normalize_identifier("ASIN", "b000fc0pda") == "asin:b000fc0pda"
def test_something_too_short_is_not_evidence(self) -> None:
"""A Calibre id of "42" would otherwise pair two unrelated books."""
assert normalize_identifier("calibre", "42") is None
@pytest.mark.parametrize(("name", "value"), [("", "1234567"), ("asin", "")])
def test_half_an_identifier_is_no_identifier(self, name: str, value: str) -> None:
assert normalize_identifier(name, value) is None
class TestIsbnConversion:
def test_isbn_10_converts_to_its_isbn_13(self) -> None:
assert isbn10_to_isbn13("0486282112") == "9780486282114"
def test_a_trailing_x_is_a_digit(self) -> None:
assert isbn10_to_isbn13("043942089X") == "9780439420891"
@pytest.mark.parametrize("isbn", ["0486282113", "9780486282114", "nonsense"])
def test_anything_that_is_not_an_isbn_10_converts_to_nothing(self, isbn: str) -> None:
assert isbn10_to_isbn13(isbn) is None
class TestParseIdentifier:
"""What an EPUB writes, and what is worth storing for it."""
@pytest.mark.parametrize(
"written",
[
"9780486282114",
"978-0-486-28211-4",
"urn:isbn:9780486282114",
"urn:isbn:978-0-486-28211-4",
"ISBN:978-0-486-28211-4",
],
)
def test_isbns_survive_however_they_are_written(self, written: str) -> None:
assert parse_identifier(written) == ("isbn-13", "9780486282114")
def test_the_scheme_attribute_is_read_too(self) -> None:
assert parse_identifier("0-486-28211-2", "ISBN") == ("isbn-10", "0486282112")
def test_non_isbn_identifiers_are_kept(self) -> None:
assert parse_identifier("urn:uuid:3f2b1c4e-1111-2222-3333-444455556666") == (
"uuid",
"3f2b1c4e-1111-2222-3333-444455556666",
)
assert parse_identifier("calibre:1234") == ("calibre", "1234")
assert parse_identifier("B000FC0PDA", "mobi-asin") == ("asin", "B000FC0PDA")
def test_an_unrecognised_prefix_is_part_of_the_value(self) -> None:
""""http://example.com/book" is not an identifier called "http"."""
assert parse_identifier("http://www.gutenberg.org/5200") == (
"id",
"http://www.gutenberg.org/5200",
)
def test_a_declared_isbn_that_is_not_one_is_dropped(self) -> None:
assert parse_identifier("urn:isbn:not-an-isbn") is None
@pytest.mark.parametrize("written", ["", " ", None])
def test_nothing_yields_nothing(self, written: str | None) -> None:
assert parse_identifier(written) is None
+176 -1
View File
@@ -1,7 +1,13 @@
import pytest import pytest
from ebooklib import epub
from pathlib import Path from pathlib import Path
from datetime import date from datetime import date
from chitai.services.metadata_extractor import EpubExtractor from chitai.services.metadata_extractor import (
EpubExtractor,
Extractor,
PdfExtractor,
split_edition,
)
@pytest.mark.asyncio() @pytest.mark.asyncio()
@@ -15,3 +21,172 @@ class TestEpubExtractor:
assert metadata["authors"] == ["Herman Melville"] assert metadata["authors"] == ["Herman Melville"]
assert metadata["published_date"] == date(year=2001, month=7, day=1) assert metadata["published_date"] == date(year=2001, month=7, day=1)
EPUB = Path("tests/data_files/Metamorphosis - Franz Kafka.epub")
PDF = Path("tests/data_files/Calculus Made Easy - Silvanus Thompson.pdf")
@pytest.mark.asyncio()
class TestIdentifierMerging:
"""A book's formats each contribute identifiers; none of them replaces the rest."""
async def test_every_format_contributes(self, monkeypatch: pytest.MonkeyPatch) -> None:
"""
Identifiers are a collection, not a single value.
Merging the whole dict meant the last format to report won outright: an EPUB
declaring an ASIN, a Google volume id and a Calibre id kept none of them once
a PDF contributed one ISBN.
"""
async def epub(_file):
return {
"title": "How Linux Works",
"identifiers": {"isbn-13": "9781718500419", "asin": "1718500408"},
}
async def pdf(_file):
return {"identifiers": {"isbn-13": "9781718500402", "isbn-10": "1593270356"}}
monkeypatch.setattr(EpubExtractor, "extract_metadata", epub)
monkeypatch.setattr(PdfExtractor, "extract_metadata", pdf)
metadata = await Extractor.extract_metadata(
[Path("How Linux Works.epub"), Path("How Linux Works.pdf")]
)
assert metadata["identifiers"] == {
# Declared by the publisher's toolchain, so it outranks the PDF's, which
# was scraped off a copyright page that also prints the print edition's.
"isbn-13": "9781718500419",
"asin": "1718500408",
"isbn-10": "1593270356",
}
async def test_a_second_format_that_finds_nothing_erases_nothing(self) -> None:
"""The PDF fixture carries no ISBN, so it must leave the EPUB's alone."""
metadata = await Extractor.extract_metadata([EPUB, PDF])
assert metadata["identifiers"] == {"id": "http://www.gutenberg.org/5200"}
async def test_one_format_on_its_own_is_unaffected(self) -> None:
metadata = await Extractor.extract_metadata([EPUB])
assert metadata["identifiers"] == {"id": "http://www.gutenberg.org/5200"}
async def test_no_identifiers_anywhere_leaves_the_field_absent(self) -> None:
"""An empty dict would count as extracted metadata and overwrite nothing."""
metadata = await Extractor.extract_metadata([PDF])
assert "identifiers" not in metadata
class TestSplitEdition:
"""An edition is a field on the book, not part of what the book is called."""
@pytest.mark.parametrize(
("title", "stripped", "edition"),
[
# Every form below is one that turned up in a real library.
("Fluent Python, 2nd Edition", "Fluent Python", 2),
("Building Microservices, 2E", "Building Microservices", 2),
("Digital Image Processing, 4e", "Digital Image Processing", 4),
(
"Network Security Essentials: Applications and Standards/6e",
"Network Security Essentials: Applications and Standards",
6,
),
(
"Refactoring: Improving the Design of Existing Code (2nd edition)",
"Refactoring: Improving the Design of Existing Code",
2,
),
# Ordinal words, including a qualifier sitting inside the statement.
(
"The Art of Computer Programming: Volume 1 / Fundamental Algorithms, Third Edition",
"The Art of Computer Programming: Volume 1 / Fundamental Algorithms",
3,
),
(
"Introduction to the Theory of Computation, Third International Edition",
"Introduction to the Theory of Computation",
3,
),
# Mid-title, before a subtitle and before a trailing author.
(
"How Linux Works, 3rd Edition: What Every Superuser Should Know",
"How Linux Works: What Every Superuser Should Know",
3,
),
(
"Code Complete, 2nd Edition - Steve McConnell",
"Code Complete - Steve McConnell",
2,
),
# An underscore between the number and the "e", beside an unnumbered
# qualifier that has nowhere to go in an integer column and so stays put.
(
"Cryptography and Network Security, Global Edition, 8_e - Stallings",
"Cryptography and Network Security, Global Edition - Stallings",
8,
),
],
)
def test_editions_are_split_out(self, title: str, stripped: str, edition: int) -> None:
assert split_edition(title) == (stripped, edition)
@pytest.mark.parametrize(
"title",
[
# A number alone is never an edition — these are titles.
"Catch 22",
"Fahrenheit 451",
"Blade Runner 2049",
"1984",
"Apollo 13",
"Slaughterhouse 5",
"The Art of Computer Programming: Volume 1",
# "Edition" with no number cannot be stored, so it stays where it can
# still be read.
"Cryptography and Network Security: Principles and Practice, Global Edition",
"Building Microservices",
],
)
def test_titles_are_left_alone(self, title: str) -> None:
assert split_edition(title) == (title, None)
def test_a_title_that_is_only_an_edition_is_kept(self) -> None:
"""Stripping must never leave a book with no title at all."""
assert split_edition("2nd Edition") == ("2nd Edition", None)
@pytest.mark.parametrize("title", ["", None])
def test_nothing_yields_nothing(self, title: str | None) -> None:
assert split_edition(title) == (title, None)
@pytest.mark.asyncio()
class TestEditionFromFiles:
async def test_extraction_moves_the_edition_off_the_title(self) -> None:
"""The PDF fixture calls itself a 2nd edition in its own metadata title."""
metadata = await Extractor.extract_metadata([PDF])
assert metadata["title"] == "The Project Gutenberg eBook #33283: Calculus Made Easy"
assert metadata["edition"] == 2
class TestEpubPublisher:
"""The publisher was looked up and then dropped on the floor."""
def test_a_declared_publisher_is_returned(self) -> None:
"""
The lookup discarded its own result and fell off the end of the function, so
every EPUB reported no publisher no matter what it said.
"""
book = epub.EpubBook()
book.add_metadata("DC", "publisher", "No Starch Press")
assert EpubExtractor._extract_publisher(book) == "No Starch Press"
def test_no_publisher_is_none(self) -> None:
assert EpubExtractor._extract_publisher(epub.EpubBook()) is None
File diff suppressed because it is too large Load Diff
+146
View File
@@ -0,0 +1,146 @@
# Implementation brief: move Duplicates into library settings
Written for an agent picking this up cold. Read the repo-root `AGENTS.md` and
`frontend/AGENTS.md` first — this brief assumes both.
**This is a frontend-only change.** The backend already scopes everything by library
(`GET /books/duplicate-books?library_id=`), so no endpoint, schema or migration is
involved.
## Where this starts from
The duplicates review screen exists and works. It currently lives at
```
frontend/src/routes/(root)/(library)/library/[libraryId]/duplicates/
+page.server.ts loads the groups, plus the full Book records merge needs
+page.svelte group cards, "Not duplicates", "Merge…"
```
and is reached from a **Duplicates entry in the main sidebar**
(`frontend/src/lib/components/layout/nav-main.svelte`), which is what this change
removes.
Settings today is a flat, entirely global four-item nav
(`frontend/src/routes/(root)/settings/+layout.svelte`): Account, Appearance, Libraries,
Devices. `settings/libraries/+page.svelte` is a single table of every library whose rows
link *out* to the library itself. **There is nowhere that means "settings for this
library"** — that is the gap this change fills.
## What to build — option B
Libraries expands in the settings nav. Every library is a sub-item; selecting one swaps
the pane; Duplicates is a section inside that pane. All libraries and all their sections
end up one click apart.
```
/settings/libraries the existing table (leave it as the index)
/settings/libraries/[libraryId] redirects to the first section
/settings/libraries/[libraryId]/duplicates the review screen, moved
```
Suggested files:
| Path | What |
| --- | --- |
| `settings/libraries/[libraryId]/+layout.svelte` | Library name, and the section tabs |
| `settings/libraries/[libraryId]/+page.ts` | `redirect(303, …/duplicates)` |
| `settings/libraries/[libraryId]/duplicates/+page.server.ts` | Moved verbatim |
| `settings/libraries/[libraryId]/duplicates/+page.svelte` | Moved verbatim |
Duplicates is the **only** real section today. Build the tab strip so General and Danger
zone have somewhere obvious to land, but do not invent them now — an empty tab is worse
than no tab.
## The nav
In `settings/+layout.svelte`, `items` is a flat `as const` array matched on
`page.route.id`. Libraries needs to render its children beneath it:
```svelte
{#each libraryState.libraries as library (library.id)}
<a href={resolve('/(root)/settings/libraries/[libraryId]/duplicates', {
libraryId: String(library.id) })}> </a>
{/each}
```
`getLibraryState()` **is** available under `/settings` — it is set in
`(root)/+layout.svelte`, above the settings group, and `settings/libraries/+page.svelte`
already uses it. No new load function is needed to list the libraries.
**Active state is matched on route id, not pathname.** There is a comment in
`settings/+layout.svelte` explaining why: `resolve()` returns an absolute path on the
client and a relative one during SSR, so a pathname comparison is false on the server and
true after hydration, and the highlight flashes in. A nested library item is active when
the route id matches **and** `page.params.libraryId === String(library.id)` — both, or
every library lights up at once.
## Things that will bite
1. **Remove the sidebar entry in the same change.** `nav-main.svelte` gained a
`Duplicates` item and a `CopyCheck` import when the screen was built. Delete both, and
delete the old route directory. Doing the removal and the move together is the point —
split across two commits the screen is unreachable in between.
2. **Delete the old route, do not leave it.** Two live copies of a screen that both write
is how they drift.
3. **`setBookSelectionState` is not available under `/settings`.** It is set in
`(root)/(library)/+layout.svelte`, which the settings group is not inside. This is
fine — the duplicates page uses `BookImage` directly, not `book-thumbnail.svelte`, and
`MergeBooks` takes its `libraryId` as a prop. **Verify this stays true** if you touch
either component; a `getBookSelectionState()` under settings returns `undefined` and
fails at the first access, not at import.
4. **Keep `depends('app:duplicate-books')`.** Both the dismiss action and `MergeBooks`
call `invalidate('app:duplicate-books')` to make a resolved group leave the screen.
Drop it and the page silently stops refreshing. `MergeBooks` also invalidates
`app:books`, which is a no-op under settings and should stay that way.
5. **The settings shell is height-constrained.** `settings/+layout.svelte` is
`h-[calc(100vh-var(--header-height)-2rem)]` with `overflow-auto` on the content pane.
The review screen is a long list of cards — it must scroll *inside* that pane. Its
current `mx-auto max-w-5xl` wrapper will want revisiting.
6. **Three levels of nav is option B's known cost.** Nav → library → section, and the
pane is narrower than the full-width route the screen was designed against. The group
cards are `w-36` covers in a wrapping flex row, so they reflow, but check a group of
four at a narrow window before calling it done.
7. **`resolve()` must be a direct call in markup** for `svelte/no-navigation-without-resolve`.
Where `nav-main.svelte` computes a url through a variable it carries an
`eslint-disable-next-line`; prefer the direct call over inheriting that.
8. **The loader depends on the `?ids=` fix.** `+page.server.ts` fetches full `Book`
records with `listBooks({ ids, pageSize })` because the merge workbench needs
identifiers, description and publisher, which `DuplicateBookRead` does not carry.
advanced_alchemy's stock id filter types that parameter as `list[str]` regardless of
config, which made Postgres refuse `bigint = character varying`; the override lives in
`backend/src/chitai/services/dependencies.py` (`create_book_filter_dependencies`).
If `GET /books?ids=1&ids=2` 500s, that override is missing — do not work around it in
the loader.
## Out of scope
The **General** and **Danger zone** sections (rename, path template, read-only, consume
directory, delete), and any change to the merge workbench itself. The toolbar entry point
for merge — select 2+ books in the library view — is unrelated and stays where it is.
## Verification
```bash
cd frontend
pnpm check # baseline: 30 errors, 1 warning, 8 files — none of them yours
pnpm lint # not clean either; check the files you touched, not the tree
pnpm build
```
By hand, with a library that has a duplicate group:
- Settings → Libraries lists every library beneath it; clicking one opens its pane.
- Duplicates shows the same groups the old route did, and scrolls inside the settings pane.
- **Not duplicates** removes the group and it stays gone after a reload.
- **Merge…** opens the workbench, merges, and the group leaves the screen.
- The main sidebar no longer has a Duplicates entry, and
`/library/<id>/duplicates` no longer resolves.
- A library with no duplicates shows the empty state, not a blank pane.
+6 -4
View File
@@ -157,10 +157,12 @@ but take a baseline first, because neither is clean (see below).
Observed in the current tree — don't mistake these for intentional patterns to copy: Observed in the current tree — don't mistake these for intentional patterns to copy:
- `src/lib/schema/openapi/schema.d.ts` is **stale** — it predates the `BookProgress` rework and has - `pnpm check` is not clean. **Baseline as of 2026-08-13: 30 errors, 1 warning, 8 files**, most of
no `BookProgressRead`, so `book.progress` types as `{}`. It is the source of most of the errors them in `src/routes/api/[...path]/+server.ts` (see below). Get your own baseline before assuming
`pnpm check` reports; regenerating it should clear them. **Baseline as of 2026-08-11: 37 errors, an error is yours. `src/lib/schema/openapi/schema.d.ts` was regenerated on that date and is
1 warning, 11 files.** Get your own baseline before assuming an error is yours. current; regenerate it again after any backend API change, with
`pnpm exec openapi-typescript http://localhost:8000/schema/openapi.json -o src/lib/schema/openapi/schema.d.ts`
against a backend running **your** branch — a stale server silently writes a stale file.
- `pnpm lint` does not pass either — 131 pre-existing ESLint errors, 35 of them - `pnpm lint` does not pass either — 131 pre-existing ESLint errors, 35 of them
`svelte/no-navigation-without-resolve` on plain `href`s, plus ~59 files Prettier would rewrite `svelte/no-navigation-without-resolve` on plain `href`s, plus ~59 files Prettier would rewrite
(mostly vendored shadcn components). Check the files you touched, not the whole tree. (mostly vendored shadcn components). Check the files you touched, not the whole tree.
+100 -18
View File
@@ -8,12 +8,31 @@ import {
deleteBooksSchema, deleteBooksSchema,
editBookMetadataSchema, editBookMetadataSchema,
updateBookProgressSchema, updateBookProgressSchema,
duplicateDismissalSchema,
bookMergeSchema,
type Book, type Book,
type BooksUploadResult,
type DuplicateBookGroup,
bookFilesUpload bookFilesUpload
} from '$lib/schema/index'; } from '$lib/schema/index';
import { stringCoerce, type PaginatedResponse } from '$lib/schema/common'; import { stringCoerce, type PaginatedResponse } from '$lib/schema/common';
import { createQueryParams } from '$lib/utils'; import { createQueryParams } from '$lib/utils';
/**
* The backend's own message for a failed response, rather than its JSON envelope.
*
* A refused duplicate answers 409 with a `detail` worth reading and the offending
* files in `extra`; passing the body through whole puts JSON in front of the reader.
*/
function detailOf(body: string): string {
try {
const parsed = JSON.parse(body);
return typeof parsed?.detail === 'string' ? parsed.detail : body;
} catch {
return body;
}
}
export const getBook = query(stringCoerce, async (id): Promise<Book> => { export const getBook = query(stringCoerce, async (id): Promise<Book> => {
const { locals } = getRequestEvent(); const { locals } = getRequestEvent();
@@ -68,26 +87,29 @@ export const updateBookCover = form(bookCoverUpload, async ({ book_id, file }) =
return await response.json(); return await response.json();
}); });
export const uploadBooks = form(booksUpload, async ({ library_id, files }) => { export const uploadBooks = form(
const { locals } = getRequestEvent(); booksUpload,
async ({ library_id, files }): Promise<BooksUploadResult> => {
const { locals } = getRequestEvent();
const formData = new FormData(); const formData = new FormData();
files.forEach((file) => { files.forEach((file) => {
formData.append('files', file); formData.append('files', file);
}); });
const response = await locals.api.postMultipart( const response = await locals.api.postMultipart(
`/books/fromFiles?library_id=${library_id}`, `/books/fromFiles?library_id=${library_id}`,
formData formData
); );
if (!response.ok) { if (!response.ok) {
const message = await response.text(); const message = await response.text();
error(response.status, message); error(response.status, message);
}
return await response.json();
} }
);
return await response.json();
});
export const uploadBookFiles = form(bookFilesUpload, async ({ book_id, files }) => { export const uploadBookFiles = form(bookFilesUpload, async ({ book_id, files }) => {
const { locals } = getRequestEvent(); const { locals } = getRequestEvent();
@@ -100,8 +122,9 @@ export const uploadBookFiles = form(bookFilesUpload, async ({ book_id, files })
const response = await locals.api.postMultipart(`/books/${book_id}/files`, formData); const response = await locals.api.postMultipart(`/books/${book_id}/files`, formData);
if (!response.ok) { if (!response.ok) {
const message = await response.text(); // 409 here means the file is already stored under a different book, which is
error(response.status, message); // something the reader can act on — so the message has to survive the trip.
error(response.status, detailOf(await response.text()));
} }
return await response.json(); return await response.json();
@@ -133,6 +156,65 @@ export const deleteBookFiles = command(deleteBookFilesSchema, async ({ book_id,
} }
}); });
/**
* Books already in the library that look like copies of one another.
*
* Metadata only, so every group is a question rather than a verdict — which is why
* the screen it feeds offers a way to disagree.
*/
export const listDuplicateBooks = query(
stringCoerce,
async (libraryId): Promise<DuplicateBookGroup[]> => {
const { locals } = getRequestEvent();
const response = await locals.api.get(`/books/duplicate-books?library_id=${libraryId}`);
if (!response.ok) error(response.status, detailOf(await response.text()));
return await response.json();
}
);
/**
* Fold several books into one and delete the records folded in.
*
* Irreversible, so the caller is expected to have shown what is about to happen.
*/
export const mergeBooks = command(
bookMergeSchema,
async ({ library_id, ...data }): Promise<Book> => {
const { locals } = getRequestEvent();
const response = await locals.api.post(`/books/merge?library_id=${library_id}`, data);
if (!response.ok) error(response.status, detailOf(await response.text()));
return await response.json();
}
);
/** Record that two books are not the same book, so the pair stops being proposed. */
export const dismissDuplicateBooks = command(duplicateDismissalSchema, async (data) => {
const { locals } = getRequestEvent();
const response = await locals.api.post('/books/duplicate-books/dismissals', data);
if (!response.ok) error(response.status, detailOf(await response.text()));
});
/** Undo a dismissal, so the pair is proposed again. */
export const restoreDuplicateBooks = command(duplicateDismissalSchema, async (data) => {
const { locals } = getRequestEvent();
const params = createQueryParams(data);
const response = await locals.api.delete(
`/books/duplicate-books/dismissals?${params.toString()}`
);
if (!response.ok) error(response.status, detailOf(await response.text()));
});
export const updateBookProgress = command( export const updateBookProgress = command(
updateBookProgressSchema, updateBookProgressSchema,
async ({ book_ids, ...data }) => { async ({ book_ids, ...data }) => {
@@ -50,6 +50,22 @@
toast.error(`${file.name} was not added`, { description: reason }); toast.error(`${file.name} was not added`, { description: reason });
}; };
/**
* The API's own words, when it has any.
*
* Adding a file the library already holds under another book is refused with a
* 409 naming it — far more use than "failed to add files". SvelteKit hands an
* `error()` back as an HttpError on the client, so the message sits on `body`.
*/
function apiMessage(error: unknown): string | undefined {
if (typeof error !== 'object' || error === null) return undefined;
const body = (error as { body?: { message?: string } }).body;
if (typeof body?.message === 'string') return body.message;
return error instanceof Error ? error.message : undefined;
}
function confirmDelete(file: BookFile) { function confirmDelete(file: BookFile) {
fileToDelete = file; fileToDelete = file;
confirmOpen = true; confirmOpen = true;
@@ -148,7 +164,8 @@
toast.success('Files added'); toast.success('Files added');
} catch (error) { } catch (error) {
console.error('Failed to add files: ', error); console.error('Failed to add files: ', error);
toast.error('Failed to add files'); toast.error(apiMessage(error) ?? 'Failed to add files');
uploadBookFiles.fields.files.set([]);
} }
})} })}
enctype="multipart/form-data" enctype="multipart/form-data"
@@ -0,0 +1,149 @@
import type { Book } from '$lib/schema';
/**
* What kind of thing a metadata field holds, which is the only thing that decides
* what you can do with it when two records disagree.
*/
export type FieldKind = 'text' | 'number' | 'date' | 'list' | 'keyed' | 'longtext';
/** An action offered on a field, beyond replacing it outright. */
export type FieldAction = 'replace' | 'merge' | 'append';
export interface FieldSpec {
/** The key sent in `BookMetadataUpdate`. */
key: string;
label: string;
kind: FieldKind;
group: string;
/** Extra actions past `replace`, which every field has. */
extra: FieldAction[];
}
/**
* The fields a merge can resolve, in the order and grouping the edit form uses.
*
* Deliberately a spec rather than markup: the merge workbench and, later, the
* provider review screen both render from this, so a field cannot exist in one and
* not the other. `cover` and `files` are absent because they are not choices —
* files always come across and the cover has its own endpoint.
*/
export const MERGE_FIELDS: FieldSpec[] = [
{ key: 'title', label: 'Title', kind: 'text', group: 'Identity', extra: [] },
{ key: 'subtitle', label: 'Subtitle', kind: 'text', group: 'Identity', extra: [] },
{ key: 'edition', label: 'Edition', kind: 'number', group: 'Identity', extra: [] },
{ key: 'series', label: 'Series', kind: 'text', group: 'Identity', extra: [] },
{ key: 'series_position', label: 'No.', kind: 'text', group: 'Identity', extra: [] },
{ key: 'language', label: 'Language', kind: 'text', group: 'Identity', extra: [] },
// Order is meaningful for authors, so the second list is appended rather than
// interleaved; tags are a set, so they merge.
{
key: 'authors',
label: 'Authors',
kind: 'list',
group: 'People and subjects',
extra: ['append']
},
{ key: 'tags', label: 'Tags', kind: 'list', group: 'People and subjects', extra: ['merge'] },
{ key: 'publisher', label: 'Publisher', kind: 'text', group: 'Publication', extra: [] },
{ key: 'published_date', label: 'Published', kind: 'date', group: 'Publication', extra: [] },
{ key: 'pages', label: 'Pages', kind: 'number', group: 'Publication', extra: [] },
{
key: 'identifiers',
label: 'Identifiers',
kind: 'keyed',
group: 'Publication',
extra: ['merge']
},
{
key: 'description',
label: 'Description',
kind: 'longtext',
group: 'Description',
extra: ['append']
}
];
/** A field's value, in the shape `BookMetadataUpdate` expects to receive it. */
export type FieldValue = string | number | string[] | Record<string, string> | null;
/** Read one field off a book, flattening the relations the API returns as objects. */
export function readField(book: Book, key: string): FieldValue {
switch (key) {
case 'authors':
return book.authors.map((author) => author.name);
case 'tags':
return book.tags.map((tag) => tag.name);
case 'publisher':
return book.publisher?.name ?? null;
case 'series':
return book.series?.title ?? null;
case 'identifiers':
return book.identifiers ?? {};
default:
return (book as unknown as Record<string, FieldValue>)[key] ?? null;
}
}
/** Whether a field holds nothing, and so has no decision attached to it. */
export function isEmpty(value: FieldValue): boolean {
if (value === null || value === undefined || value === '') return true;
if (Array.isArray(value)) return value.length === 0;
if (typeof value === 'object') return Object.keys(value).length === 0;
return false;
}
/** Whether two field values say the same thing, order included for lists. */
export function isSame(left: FieldValue, right: FieldValue): boolean {
if (isEmpty(left) && isEmpty(right)) return true;
return JSON.stringify(left ?? null) === JSON.stringify(right ?? null);
}
/**
* Apply an action to a pair of values and return what the target becomes.
*
* `merge` on a keyed collection is per name and the target wins a clash, because
* `Identifier` is unique on `(name, book_id)` — a book cannot hold both its print
* and its ebook ISBN, so the second one has nowhere to go.
*/
export function applyAction(
action: FieldAction,
kind: FieldKind,
target: FieldValue,
incoming: FieldValue
): FieldValue {
if (action === 'replace') return incoming;
if (kind === 'list') {
const current = Array.isArray(target) ? target : [];
const extra = Array.isArray(incoming) ? incoming : [];
// Order preserved, duplicates dropped — works for both append and merge.
return [...new Set([...current, ...extra])];
}
if (kind === 'keyed') {
return { ...(incoming as Record<string, string>), ...(target as Record<string, string>) };
}
if (kind === 'longtext') {
const current = typeof target === 'string' ? target.trim() : '';
const extra = typeof incoming === 'string' ? incoming.trim() : '';
return [current, extra].filter(Boolean).join('\n\n');
}
return incoming;
}
/** How a value reads in the inert reference column. */
export function displayValue(value: FieldValue): string {
if (isEmpty(value)) return '';
if (Array.isArray(value)) return value.join(', ');
if (typeof value === 'object') {
return Object.entries(value as Record<string, string>)
.map(([name, id]) => `${name}: ${id}`)
.join(' · ');
}
return String(value);
}
@@ -0,0 +1,394 @@
<script lang="ts">
import { untrack } from 'svelte';
import { ArrowRight, GitMerge, Plus, Undo2 } from '@lucide/svelte';
import * as Dialog from '$lib/components/ui/dialog/index.js';
import { Button } from '$lib/components/ui/button/index.js';
import { Input } from '$lib/components/ui/input/index.js';
import { Textarea } from '$lib/components/ui/textarea/index.js';
import { Badge } from '$lib/components/ui/badge/index.js';
import BookImage from '$lib/components/view/book-image.svelte';
import GeneratedCover from '$lib/components/view/generated-cover.svelte';
import type { Book } from '$lib/schema';
import { mergeInto } from './merge';
import {
MERGE_FIELDS,
applyAction,
displayValue,
isEmpty,
isSame,
readField,
type FieldAction,
type FieldSpec,
type FieldValue
} from './field-spec';
let {
books,
libraryId,
open = $bindable(),
onmerged
}: {
books: Book[];
libraryId: number | string;
open: boolean;
/** Given the record that survived and the ones folded into it and deleted. */
onmerged?: (survivor: Book, folded: Book[]) => void;
} = $props();
// Seeded once per mount. The dialog is keyed on the group upstream, so a
// different group gets a fresh workbench rather than the previous one's draft.
let survivorId = $state(untrack(() => books[0]?.id));
let candidateId = $state(untrack(() => books[1]?.id));
let draft = $state<Record<string, FieldValue>>({});
let busy = $state(false);
const survivor = $derived(books.find((book) => book.id === survivorId) ?? books[0]);
const candidates = $derived(books.filter((book) => book.id !== survivorId));
/** The records that will be deleted — the same set, named for what happens to them. */
const folded = $derived(candidates);
const candidate = $derived(candidates.find((book) => book.id === candidateId) ?? candidates[0]);
/** The survivor's stored value for a field, or the draft if it has been touched. */
function current(field: FieldSpec): FieldValue {
return field.key in draft ? draft[field.key] : readField(survivor, field.key);
}
function take(field: FieldSpec, action: FieldAction) {
draft[field.key] = applyAction(
action,
field.kind,
current(field),
readField(candidate, field.key)
);
}
function undo(field: FieldSpec) {
delete draft[field.key];
draft = { ...draft };
}
/**
* Fill only the fields the survivor has nothing in.
*
* The safe bulk action, and the one worth reaching for: it cannot overwrite a
* value, so it needs no per-field protection to be pressed without reading.
*/
function fillEmpty() {
for (const field of MERGE_FIELDS) {
const incoming = readField(candidate, field.key);
if (isEmpty(current(field)) && !isEmpty(incoming)) draft[field.key] = incoming;
}
}
const changed = $derived(Object.keys(draft));
const differing = $derived(
candidate
? MERGE_FIELDS.filter((field) => !isSame(current(field), readField(candidate, field.key)))
: []
);
/** Fields shown as a row: anything the two disagree on, plus anything edited. */
const shown = $derived(
MERGE_FIELDS.filter((field) => differing.includes(field) || field.key in draft)
);
const agreed = $derived(MERGE_FIELDS.filter((field) => !shown.includes(field)));
const groups = $derived([...new Set(shown.map((field) => field.group))]);
async function submit() {
if (busy) return;
busy = true;
const merged = await mergeInto(libraryId, survivor, folded, changed.length ? draft : undefined);
busy = false;
if (merged) {
open = false;
onmerged?.(survivor, folded);
}
}
</script>
{#snippet cover(book: Book, size: string)}
<span class="{size} shrink-0 overflow-hidden rounded-sm border bg-muted">
{#if book.cover_image}
<BookImage src="/api/{book.cover_image}" class="h-full w-full object-cover" />
{:else}
<!-- Drawn rather than left blank, so the rail tells two coverless books
apart the same way the shelves do. -->
<GeneratedCover {book} />
{/if}
</span>
{/snippet}
<!-- The inert reference column: the same shape as the control opposite it, with
nothing that invites a click. -->
{#snippet reference(field: FieldSpec, book: Book)}
{@const value = readField(book, field.key)}
<div class="flex min-w-0 flex-col gap-1">
<span class="text-[11px] text-muted-foreground">{field.label}</span>
{#if isEmpty(value)}
<span class="min-h-8 py-1 text-sm text-muted-foreground italic">empty</span>
{:else if field.kind === 'list'}
<span class="flex min-h-8 flex-wrap items-center gap-1 py-0.5">
{#each value as string[] as item (item)}
<Badge variant="secondary" class="font-normal">{item}</Badge>
{/each}
</span>
{:else}
<span class="min-h-8 py-1 text-sm break-words">{displayValue(value)}</span>
{/if}
</div>
{/snippet}
<Dialog.Root bind:open>
<Dialog.Content
class="flex h-[85vh] max-w-[min(72rem,95vw)] flex-col gap-0 overflow-hidden p-0 sm:max-w-[min(72rem,95vw)]"
>
<Dialog.Header class="shrink-0 space-y-0 border-b px-5 py-3 text-left">
<Dialog.Title class="font-serif text-base font-normal">
Merge {books.length} books
</Dialog.Title>
<Dialog.Description class="text-xs">
{candidates.length}
{candidates.length === 1 ? 'record is' : 'records are'} deleted. Their files move onto the book
you keep — nothing is removed from disk.
</Dialog.Description>
</Dialog.Header>
<div class="grid min-h-0 flex-1 grid-cols-1 md:grid-cols-[15rem_1fr]">
<!-- Rail: the books being folded in, one open at a time -->
<aside
class="flex min-w-0 flex-col overflow-y-auto border-b bg-sidebar md:border-r md:border-b-0"
>
<p
class="px-3 pt-3 pb-1 font-mono text-[10px] tracking-widest text-muted-foreground uppercase"
>
Taking from · {candidates.length}
</p>
{#each candidates as book (book.id)}
{@const count = MERGE_FIELDS.filter(
(field) => !isSame(readField(survivor, field.key), readField(book, field.key))
).length}
<button
type="button"
onclick={() => (candidateId = book.id)}
class="flex items-start gap-2 border-l-2 px-3 py-2 text-left transition-colors hover:bg-muted/50 {book.id ===
candidate?.id
? 'border-l-primary bg-background'
: 'border-l-transparent'}"
>
{@render cover(book, 'h-10 w-7')}
<span class="min-w-0 flex-1">
<span class="line-clamp-2 font-serif text-[13px]">{book.title}</span>
<span class="block text-[10px] text-muted-foreground">
#{book.id} · {book.files.length}
{book.files.length === 1 ? 'file' : 'files'}
</span>
{#if count === 0}
<Badge variant="secondary" class="mt-1 text-[10px] font-normal">
nothing to take
</Badge>
{/if}
</span>
</button>
{/each}
</aside>
<div class="flex min-h-0 flex-col">
<!-- Which record survives -->
<div class="flex shrink-0 flex-wrap items-center gap-2 border-b px-5 py-2.5">
<span class="text-[11px] text-muted-foreground">Keeping</span>
{#each books as book (book.id)}
<button
type="button"
onclick={() => (survivorId = book.id)}
class="flex items-center gap-1.5 rounded-md border px-2 py-1 text-xs transition-colors {book.id ===
survivorId
? 'border-primary bg-accent text-accent-foreground'
: 'text-muted-foreground hover:bg-muted'}"
>
{@render cover(book, 'h-6 w-4')}
<span class="max-w-32 truncate">#{book.id}</span>
</button>
{/each}
<div class="ml-auto flex gap-2">
<Button variant="outline" size="sm" onclick={fillEmpty} disabled={!candidate}>
Fill empty fields
</Button>
</div>
</div>
<div class="min-h-0 flex-1 overflow-y-auto px-5 py-4">
{#if !candidate}
<p class="text-sm text-muted-foreground">Nothing left to fold in.</p>
{:else if shown.length === 0}
<p class="text-sm text-muted-foreground">
These records agree on every field. Merging keeps
<span class="font-medium text-foreground">#{survivor.id}</span> and moves the others' files
onto it.
</p>
{:else}
{#each groups as group (group)}
<h3
class="mt-5 border-b pb-1 font-mono text-[10px] tracking-widest text-primary uppercase first:mt-0"
>
{group}
</h3>
{#each shown.filter((field) => field.group === group) as field (field.key)}
{@const value = current(field)}
{@const incoming = readField(candidate, field.key)}
{@const edited = field.key in draft}
<div class="grid grid-cols-[1fr_auto_1fr] items-center gap-3 border-b py-2">
{@render reference(field, candidate)}
<!-- Actions, on the row rather than stacked beside it -->
<div class="flex items-center gap-1">
{#if edited}
<Button
variant="ghost"
size="icon"
class="size-7"
title="Undo"
onclick={() => undo(field)}
>
<Undo2 class="size-3.5" />
<span class="sr-only">Undo {field.label}</span>
</Button>
{/if}
{#if !isSame(value, incoming)}
{#if isEmpty(value)}
<!-- Nothing to weigh, so it is an offer rather than a choice -->
<Button
size="sm"
class="h-7 gap-1 px-2 text-[11px]"
onclick={() => take(field, 'replace')}
>
<ArrowRight class="size-3.5" />
Take
</Button>
{:else}
<Button
variant="outline"
size="icon"
class="size-7"
title="Replace"
onclick={() => take(field, 'replace')}
>
<ArrowRight class="size-3.5" />
<span class="sr-only">Replace {field.label}</span>
</Button>
{#each field.extra as action (action)}
<Button
variant="outline"
size="icon"
class="size-7"
title={action === 'merge' ? 'Merge' : 'Append'}
onclick={() => take(field, action)}
>
{#if action === 'merge'}
<GitMerge class="size-3.5" />
{:else}
<Plus class="size-3.5" />
{/if}
<span class="sr-only">{action} {field.label}</span>
</Button>
{/each}
{/if}
{/if}
</div>
<!-- The survivor's side: the edit form, live -->
<div class="flex min-w-0 flex-col gap-1">
<span class="flex items-center gap-1.5 text-[11px] text-muted-foreground">
{field.label}
{#if edited}
<Badge class="h-4 px-1.5 text-[9px] font-semibold">taken</Badge>
{/if}
</span>
{#if field.kind === 'longtext'}
<Textarea
rows={4}
class="text-sm"
value={(value as string) ?? ''}
oninput={(event) => (draft[field.key] = event.currentTarget.value)}
/>
{:else if field.kind === 'list' || field.kind === 'keyed'}
<!-- Edited through the take actions; typing here would need the
tags and key/value editors, which belong to the edit form. -->
<div
class="flex min-h-8 flex-wrap items-center gap-1 rounded-md border bg-background px-2 py-1"
>
{#if isEmpty(value)}
<span class="text-sm text-muted-foreground"></span>
{:else if field.kind === 'list'}
{#each value as string[] as item (item)}
<Badge variant="secondary" class="font-normal">{item}</Badge>
{/each}
{:else}
{#each Object.entries(value as Record<string, string>) as [name, id] (name)}
<Badge variant="secondary" class="font-mono text-[10px] font-normal">
{name}: {id}
</Badge>
{/each}
{/if}
</div>
{:else}
<Input
type={field.kind === 'number'
? 'number'
: field.kind === 'date'
? 'date'
: 'text'}
class="h-8 text-sm"
value={(value as string | number) ?? ''}
oninput={(event) =>
(draft[field.key] =
field.kind === 'number'
? Number(event.currentTarget.value) || null
: event.currentTarget.value)}
/>
{/if}
</div>
</div>
{/each}
{/each}
{#if agreed.length > 0}
<p class="pt-3 text-center text-[11px] text-muted-foreground italic">
{agreed.map((field) => field.label).join(', ')} — identical in both
</p>
{/if}
{/if}
</div>
</div>
</div>
<Dialog.Footer class="shrink-0 items-center gap-2 border-t px-5 py-3 sm:justify-between">
<p class="text-xs text-muted-foreground">
{#if changed.length}
{changed.length}
{changed.length === 1 ? 'change' : 'changes'} pending · this cannot be undone
{:else}
Metadata is left as #{survivor.id} has it · this cannot be undone
{/if}
</p>
<div class="flex gap-2">
<Button variant="outline" onclick={() => (open = false)}>Cancel</Button>
<Button onclick={submit} disabled={busy || candidates.length === 0}>
Merge into “{survivor.title}
</Button>
</div>
</Dialog.Footer>
</Dialog.Content>
</Dialog.Root>
@@ -0,0 +1,44 @@
import { invalidate } from '$app/navigation';
import { toast } from 'svelte-sonner';
import { mergeBooks } from '$lib/api/book.remote';
import type { Book } from '$lib/schema';
import type { FieldValue } from './field-spec';
/**
* Fold `folded` into `survivor`, and tell the reader how it went.
*
* Shared because a merge is reachable two ways the workbench, where the reader
* resolved the metadata field by field, and the quick action beside it, which
* takes the survivor's metadata as it stands. Only the `metadata` argument
* differs, and the invalidation and the wording should not.
*
* @returns whether the merge went through; the caller decides what to close or
* clear, and has no toast of its own to write either way.
*/
export async function mergeInto(
libraryId: number | string,
survivor: Book,
folded: Book[],
metadata?: Record<string, FieldValue>
): Promise<boolean> {
try {
await mergeBooks({
library_id: libraryId,
survivor_id: survivor.id,
merged_ids: folded.map((book) => book.id),
metadata
});
// Both, because a merge is started from the duplicates review and from the
// library's selection toolbar, and each page depends on a different one.
await Promise.all([invalidate('app:books'), invalidate('app:duplicate-books')]);
toast.success(`Merged into “${survivor.title}`);
return true;
} catch (error) {
console.error('Failed to merge books', error);
toast.error(error instanceof Error ? error.message : 'Could not merge these books');
return false;
}
}
@@ -33,13 +33,15 @@
title: 'Home', title: 'Home',
icon: House, icon: House,
routeId: '/(root)/(library)/library/[libraryId]', routeId: '/(root)/(library)/library/[libraryId]',
path: (id?: number) => resolve('/(root)/(library)/library/[libraryId]', { libraryId: String(id ?? '') }) path: (id?: number) =>
resolve('/(root)/(library)/library/[libraryId]', { libraryId: String(id ?? '') })
}, },
{ {
title: 'Library', title: 'Library',
icon: LibraryBig, icon: LibraryBig,
routeId: '/(root)/(library)/library/[libraryId]/view', routeId: '/(root)/(library)/library/[libraryId]/view',
path: (id?: number) => resolve('/(root)/(library)/library/[libraryId]/view', { libraryId: String(id ?? '') }) path: (id?: number) =>
resolve('/(root)/(library)/library/[libraryId]/view', { libraryId: String(id ?? '') })
}, },
{ title: 'Shelves', icon: Rows3, routeId: null, path: () => '#', shelves: [] } { title: 'Shelves', icon: Rows3, routeId: null, path: () => '#', shelves: [] }
]; ];
@@ -1,11 +1,13 @@
<script lang="ts"> <script lang="ts">
import { fly } from 'svelte/transition'; import { fly } from 'svelte/transition';
import { prefersReducedMotion } from 'svelte/motion'; import { prefersReducedMotion } from 'svelte/motion';
import { ChevronDown, CircleAlert, RotateCcw, X } from '@lucide/svelte'; import { resolve } from '$app/paths';
import { ChevronDown, CircleAlert, Copy, RotateCcw, X } from '@lucide/svelte';
import { Button } from '$lib/components/ui/button/index.js'; import { Button } from '$lib/components/ui/button/index.js';
import { Spinner } from '$lib/components/ui/spinner/index'; import { Spinner } from '$lib/components/ui/spinner/index';
import { getUploadQueueState } from '$lib/state/upload-queue.svelte'; import { getUploadQueueState } from '$lib/state/upload-queue.svelte';
import type { DuplicateBook, DuplicateFile } from '$lib/schema';
import { formatFileSize } from '$lib/utils'; import { formatFileSize } from '$lib/utils';
const queue = getUploadQueueState(); const queue = getUploadQueueState();
@@ -19,10 +21,48 @@
const heading = $derived.by(() => { const heading = $derived.by(() => {
const noun = queue.total === 1 ? 'book' : 'books'; const noun = queue.total === 1 ? 'book' : 'books';
if (queue.active) return `Adding ${queue.settled} of ${queue.total} ${noun}`; if (queue.active) return `Adding ${queue.settled} of ${queue.total} ${noun}`;
if (queue.failed === 0) return `Added ${queue.done} ${noun}`; if (queue.failed === 0 && queue.skipped === 0) return `Added ${queue.done} ${noun}`;
if (queue.done === 0) return `Couldn't add ${queue.failed} ${noun}`; if (queue.done === 0 && queue.skipped === 0) return `Couldn't add ${queue.failed} ${noun}`;
return `Added ${queue.done}, ${queue.failed} failed`;
const parts = [];
if (queue.done > 0) parts.push(`Added ${queue.done}`);
if (queue.skipped > 0) parts.push(`${queue.skipped} already here`);
if (queue.failed > 0) parts.push(`${queue.failed} failed`);
return parts.join(', ');
}); });
/**
* What the reader needs to know about files that were not stored.
*
* One duplicate can name where its bytes already live; several would be a list,
* and the row has no room for one, so they collapse to a count.
*/
function duplicateNote(duplicates: DuplicateFile[]) {
if (duplicates.length > 1) return `${duplicates.length} files are already in your library`;
const [only] = duplicates;
// No book to name: it matched another file in this same upload.
return only.book_title ? `Already in ${only.book_title}` : 'Already added by this upload';
}
/** The book a note can link to, when a single duplicate points at exactly one. */
function noteTarget(duplicates: DuplicateFile[]) {
return duplicates.length === 1 ? duplicates[0].book_id : null;
}
/**
* What the reader needs to know about a book that went in and may already be here.
*
* Worded weaker than the file-level "Already in …" on purpose. That one means the
* library holds these exact bytes; this one means the metadata agrees, which a
* second edition, a translation and a re-scan all do. Nothing was refused.
*/
function possibleNote(candidates: DuplicateBook[]) {
if (candidates.length > 1)
return `Might already be in your library, ${candidates.length} times`;
return `Might already be in your library — ${candidates[0].title}`;
}
</script> </script>
{#if queue.total > 0} {#if queue.total > 0}
@@ -44,6 +84,9 @@
<Spinner class="size-4 shrink-0" /> <Spinner class="size-4 shrink-0" />
{:else if queue.failed > 0} {:else if queue.failed > 0}
<CircleAlert class="size-4 shrink-0 text-destructive" /> <CircleAlert class="size-4 shrink-0 text-destructive" />
{:else if queue.skipped > 0}
<!-- Muted, not alarming: nothing went wrong, the books were already here. -->
<Copy class="size-4 shrink-0 text-muted-foreground" />
{/if} {/if}
<span class="min-w-0 flex-1 truncate text-sm font-medium">{heading}</span> <span class="min-w-0 flex-1 truncate text-sm font-medium">{heading}</span>
@@ -78,12 +121,61 @@
{#if !queue.collapsed} {#if !queue.collapsed}
<ul class="flex max-h-64 flex-col divide-y overflow-y-auto"> <ul class="flex max-h-64 flex-col divide-y overflow-y-auto">
{#each queue.jobs as job (job.id)} {#each queue.jobs as job (job.id)}
{@const duplicates = job.duplicates ?? []}
{@const target = noteTarget(duplicates)}
{@const possible = job.possibleDuplicates ?? []}
<li class="flex min-w-0 items-center gap-2 px-3 py-2"> <li class="flex min-w-0 items-center gap-2 px-3 py-2">
<span class="min-w-0 flex-1"> <span class="min-w-0 flex-1">
<span class="block truncate text-sm" title={job.label}>{job.label}</span> <span class="block truncate text-sm" title={job.label}>{job.label}</span>
<span class="block font-mono text-[10px] text-muted-foreground tabular-nums">
{job.error ?? formatFileSize(job.size)} {#if job.error}
</span> <span class="block font-mono text-[10px] text-muted-foreground tabular-nums">
{job.error}
</span>
{:else if duplicates.length > 0}
<!--
Reported, with no offer to override. Storing the same bytes twice
splits reading progress and shelves across two records that can
never converge, and there is no version of that the reader wants.
`allow_duplicates` stays on the API for when the match is wrong.
-->
<span class="block min-w-0 text-[10px] text-muted-foreground">
{#if target}
<a
href={resolve('/(root)/(library)/book/[bookId]', { bookId: String(target) })}
class="truncate underline underline-offset-2"
>
{duplicateNote(duplicates)}
</a>
{:else}
<span class="truncate">{duplicateNote(duplicates)}</span>
{/if}
</span>
{:else if possible.length > 0}
<!--
A guess, not a fact, so it says so and stops there: the book was
stored, and the library's duplicates screen is where a reader
decides what to do about it.
-->
<span class="block min-w-0 text-[10px] text-muted-foreground">
{#if possible.length === 1}
<a
href={resolve('/(root)/(library)/book/[bookId]', {
bookId: String(possible[0].book_id)
})}
class="truncate underline underline-offset-2"
>
{possibleNote(possible)}
</a>
{:else}
<span class="truncate">{possibleNote(possible)}</span>
{/if}
</span>
{:else}
<span class="block font-mono text-[10px] text-muted-foreground tabular-nums">
{formatFileSize(job.size)}
</span>
{/if}
</span> </span>
<span <span
@@ -96,6 +188,8 @@
Adding Adding
{:else if job.status === 'done'} {:else if job.status === 'done'}
Done Done
{:else if job.status === 'skipped'}
Skipped
{:else if job.status === 'failed'} {:else if job.status === 'failed'}
Failed Failed
{:else} {:else}
@@ -1,15 +1,27 @@
<script lang="ts"> <script lang="ts">
import { buttonVariants } from '$lib/components/ui/button/button.svelte'; import { buttonVariants } from '$lib/components/ui/button/button.svelte';
import * as Tooltip from '$lib/components/ui/tooltip/index'; import * as Tooltip from '$lib/components/ui/tooltip/index';
import { BookOpenCheck, Download, Trash2, SquareCheckBig, X, Album, PlusIcon } from '@lucide/svelte'; import {
BookOpenCheck,
Download,
GitMerge,
Trash2,
SquareCheckBig,
SlidersHorizontal,
X,
Album,
PlusIcon
} from '@lucide/svelte';
import * as DropdownMenu from '$lib/components/ui/dropdown-menu/index'; import * as DropdownMenu from '$lib/components/ui/dropdown-menu/index';
import { getBookOperationsState } from '$lib/state/bookOperations.svelte'; import { getBookOperationsState } from '$lib/state/bookOperations.svelte';
import { getBookSelectionState } from '$lib/state/bookSelection.svelte'; import { getBookSelectionState } from '$lib/state/bookSelection.svelte';
import { getBookCollectionState } from '$lib/state/bookCollection.svelte'; import { getBookCollectionState } from '$lib/state/bookCollection.svelte';
import { getLibraryState } from '$lib/state/library.svelte'; import { getLibraryState } from '$lib/state/library.svelte';
import { getBookshelfState } from '$lib/state/bookshelf.svelte'; import { getBookshelfState } from '$lib/state/bookshelf.svelte';
import { invalidate } from '$app/navigation';
import ShelfCreateDialog from '../forms/shelf-create-dialog.svelte'; import ShelfCreateDialog from '../forms/shelf-create-dialog.svelte';
import MergeBooks from '../forms/merge-books/merge-books.svelte';
import { mergeInto } from '../forms/merge-books/merge';
import type { Book } from '$lib/schema';
const libraryState = getLibraryState(); const libraryState = getLibraryState();
const bookshelfState = getBookshelfState(); const bookshelfState = getBookshelfState();
@@ -17,10 +29,50 @@
const bookOps = getBookOperationsState(); const bookOps = getBookOperationsState();
const collectionState = getBookCollectionState(); const collectionState = getBookCollectionState();
let selectedBooks = $derived(selectionState.getSelectedBooks()) let selectedBooks = $derived(selectionState.getSelectedBooks());
let createShelfDialogOpen = $state(false) let createShelfDialogOpen = $state(false);
// The books the merge dialog opened on. Snapshotted rather than read live from
// the selection, so clearing the selection on success cannot empty the dialog
// underneath itself.
let mergingBooks = $state<Book[] | null>(null);
// A quick merge has no dialog to disable, so the flag is what stops a slow
// round trip being started twice.
let quickMerging = $state(false);
/** Put the library back in step with a merge that went through. */
function afterMerge(folded: Book[]) {
// Only the folded records are gone; the survivor is still in the library.
libraryState.activeLibrary!.total! -= folded.length;
bookshelfState.deletedBooks(folded);
selectionState.deselectAll();
}
/**
* Merge without opening the workbench, keeping the book selected first.
*
* Selection is held in insertion order, so the first record is the one the
* reader started from — the one they were looking at when they decided the
* rest were copies of it.
*/
async function quickMerge() {
if (quickMerging) return;
// Read once: the merge clears the selection, and the bookkeeping afterwards
// still needs the records that went away.
const [survivor, ...folded] = selectedBooks;
if (!survivor || folded.length === 0) return;
quickMerging = true;
const merged = await mergeInto(libraryState.activeLibrary!.id, survivor, folded);
quickMerging = false;
if (merged) afterMerge(folded);
}
</script> </script>
<!-- Mark selected as finished button --> <!-- Mark selected as finished button -->
@@ -95,12 +147,13 @@
{/each} {/each}
</DropdownMenu.Group> </DropdownMenu.Group>
<DropdownMenu.Separator /> <DropdownMenu.Separator />
<DropdownMenu.Item <DropdownMenu.Item
onclick={() => createShelfDialogOpen = true} onclick={() => (createShelfDialogOpen = true)}
class="text-muted-foreground "> class="text-muted-foreground "
<PlusIcon class="size-4" /> >
New Shelf <PlusIcon class="size-4" />
</DropdownMenu.Item> New Shelf
</DropdownMenu.Item>
</DropdownMenu.Content> </DropdownMenu.Content>
</DropdownMenu.Root> </DropdownMenu.Root>
{/snippet} {/snippet}
@@ -128,6 +181,44 @@
</Tooltip.Root> </Tooltip.Root>
</Tooltip.Provider> </Tooltip.Provider>
<!-- Merge selected button. Two books is the smallest thing a merge can mean, so
it appears only once there are two. -->
{#if selectedBooks.length > 1}
<Tooltip.Provider>
<Tooltip.Root ignoreNonKeyboardFocus>
<Tooltip.Trigger>
{#snippet child({ props })}
<!-- A choice rather than a jump straight into the workbench: two
copies of one book have nothing to resolve, and the dialog is a
step in the way of saying so. -->
<DropdownMenu.Root>
<DropdownMenu.Trigger
{...props}
disabled={quickMerging}
class={buttonVariants({ variant: 'ghost', size: 'icon' })}
>
<GitMerge />
</DropdownMenu.Trigger>
<DropdownMenu.Content>
<DropdownMenu.Item onclick={quickMerge}>
<GitMerge class="size-4 shrink-0" />
Merge into first selected
</DropdownMenu.Item>
<DropdownMenu.Item onclick={() => (mergingBooks = selectedBooks)}>
<SlidersHorizontal class="size-4 shrink-0" />
Manual merge…
</DropdownMenu.Item>
</DropdownMenu.Content>
</DropdownMenu.Root>
{/snippet}
</Tooltip.Trigger>
<Tooltip.Content>
<p>Merge {selectedBooks.length} books</p>
</Tooltip.Content>
</Tooltip.Root>
</Tooltip.Provider>
{/if}
<!-- Delete selected button --> <!-- Delete selected button -->
<Tooltip.Provider> <Tooltip.Provider>
<Tooltip.Root ignoreNonKeyboardFocus> <Tooltip.Root ignoreNonKeyboardFocus>
@@ -136,8 +227,8 @@
bookOps.deleteDialogTitle = `Delete ${selectionState.getSelectedIds().length} books?`; bookOps.deleteDialogTitle = `Delete ${selectionState.getSelectedIds().length} books?`;
bookOps.deleteFn = async (deleteFiles: boolean) => { bookOps.deleteFn = async (deleteFiles: boolean) => {
await bookOps.deleteBooks(selectionState.getSelectedIds(), deleteFiles); await bookOps.deleteBooks(selectionState.getSelectedIds(), deleteFiles);
libraryState.activeLibrary!.total! -= selectedBooks.length libraryState.activeLibrary!.total! -= selectedBooks.length;
bookshelfState.deletedBooks(selectedBooks) bookshelfState.deletedBooks(selectedBooks);
selectionState.deselectAll(); selectionState.deselectAll();
}; };
bookOps.deleteDialogOpen = true; bookOps.deleteDialogOpen = true;
@@ -186,14 +277,43 @@
</Tooltip.Root> </Tooltip.Root>
</Tooltip.Provider> </Tooltip.Provider>
<ShelfCreateDialog
<ShelfCreateDialog bind:open={createShelfDialogOpen}
bind:open={createShelfDialogOpen}
onSubmit={async (name: string) => { onSubmit={async (name: string) => {
const bookshelf = await bookshelfState.addBookshelf(
const bookshelf = await bookshelfState.addBookshelf(name, libraryState.activeLibrary!.id, selectionState.getSelectedIds()) name,
selectedBooks.forEach(book => book.lists.push(bookshelf)) libraryState.activeLibrary!.id,
selectionState.deselectAll() selectionState.getSelectedIds()
createShelfDialogOpen = false );
selectedBooks.forEach((book) => book.lists.push(bookshelf));
selectionState.deselectAll();
createShelfDialogOpen = false;
}} }}
/> />
<!--
Keyed on the selection so opening merge on a different pair starts from those
records rather than the previous dialog's draft.
-->
{#if mergingBooks}
{#key mergingBooks.map((book) => book.id).join()}
<MergeBooks
books={mergingBooks}
libraryId={libraryState.activeLibrary!.id}
bind:open={
() => mergingBooks !== null,
(value) => {
// Bound, not passed as a bare `true`: the dialog closes itself on an
// outside click or Escape, and if that never reaches `mergingBooks`
// the snapshot stays set — the menu item then re-assigns the same
// selection, nothing changes, and the dialog never reopens.
if (!value) mergingBooks = null;
}
}
onmerged={(_survivor, folded) => {
afterMerge(folded);
mergingBooks = null;
}}
/>
{/key}
{/if}
@@ -1,7 +1,14 @@
<script lang="ts"> <script lang="ts">
import type { Book } from '$lib/schema'; // Typed by what is drawn rather than by `Book`: the duplicates review holds
// candidates that carry a title and author names without a whole record, and a
let { book, class: className = '' }: { book: Book; class?: string } = $props(); // `Book` satisfies this shape anyway.
let {
book,
class: className = ''
}: {
book: { title: string; authors?: { name: string }[] | null };
class?: string;
} = $props();
/** /**
* Bookcloth tones rather than a point on the hue wheel. * Bookcloth tones rather than a point on the hue wheel.
+45
View File
@@ -6,6 +6,49 @@ export type Book = components['schemas']['BookRead'];
export type BookFile = components['schemas']['FileMetadataRead']; export type BookFile = components['schemas']['FileMetadataRead'];
export type BookProgress = components['schemas']['BookProgressRead']; export type BookProgress = components['schemas']['BookProgressRead'];
/**
* A file the library already held, so the upload did not store it again.
*
* `book_id` is null when the match was another file in the same upload there is
* no book to point at yet.
*/
export type DuplicateFile = components['schemas']['DuplicateFileRead'];
export type BooksUploadResult = components['schemas']['BooksUploadResult'];
/**
* A stored book that may be the same book as another one.
*
* Weaker than `DuplicateFile`, and deliberately so: that one means the library holds
* these exact bytes, this one means the metadata agrees. A second edition and a
* translation both look like this, so nothing is ever refused on the strength of it.
*/
export type DuplicateBook = components['schemas']['DuplicateBookRead'];
/** A book that was imported, together with what it might be a second copy of. */
export type PossibleDuplicate = components['schemas']['PossibleDuplicateRead'];
/** Books the library holds that all look like copies of one book. */
export type DuplicateBookGroup = components['schemas']['DuplicateBookGroupRead'];
/** The metadata a reader resolved while merging, or while reviewing a provider. */
export type BookMetadataUpdate = components['schemas']['BookMetadataUpdate'];
/** Mirrors BookMerge in backend/src/chitai/schemas/book.py */
export const bookMergeSchema = z.object({
library_id: z.coerce.number(),
survivor_id: z.coerce.number(),
merged_ids: z.array(z.coerce.number()).min(1, 'Pick at least one book to fold in'),
// Passed through untouched — the backend validates it as BookMetadataUpdate, and
// duplicating that shape here would be two places to keep in step for no gain.
metadata: z.record(z.string(), z.unknown()).optional()
});
/** Mirrors DuplicateDismissal in backend/src/chitai/schemas/book.py */
export const duplicateDismissalSchema = z.object({
book_a_id: z.coerce.number(),
book_b_id: z.coerce.number()
});
export const bookQuerySchema = commonQuerySchema.extend({ export const bookQuerySchema = commonQuerySchema.extend({
libraries: stringArrayCoerce, libraries: stringArrayCoerce,
authors: stringArrayCoerce, authors: stringArrayCoerce,
@@ -97,4 +140,6 @@ export type BookQuery = z.infer<typeof bookQuerySchema>;
export type BookEditMetadata = z.infer<typeof editBookMetadataSchema>; export type BookEditMetadata = z.infer<typeof editBookMetadataSchema>;
export type DeleteBook = z.infer<typeof deleteBooksSchema>; export type DeleteBook = z.infer<typeof deleteBooksSchema>;
export type DeleteBookFiles = z.infer<typeof deleteBookFilesSchema>; export type DeleteBookFiles = z.infer<typeof deleteBookFilesSchema>;
export type DuplicateDismissal = z.infer<typeof duplicateDismissalSchema>;
export type BookMerge = z.infer<typeof bookMergeSchema>;
export type UpdateBookProgress = z.infer<typeof updateBookProgressSchema>; export type UpdateBookProgress = z.infer<typeof updateBookProgressSchema>;
+319 -11
View File
@@ -22,6 +22,23 @@ export interface paths {
patch?: never; patch?: never;
trace?: never; trace?: never;
}; };
"/books/duplicate-files": {
parameters: {
query?: never;
header?: never;
path?: never;
cookie?: never;
};
get?: never;
put?: never;
/** CheckDuplicateFiles */
post: operations["BooksDuplicateFilesCheckDuplicateFiles"];
delete?: never;
options?: never;
head?: never;
patch?: never;
trace?: never;
};
"/books": { "/books": {
parameters: { parameters: {
query?: never; query?: never;
@@ -58,6 +75,24 @@ export interface paths {
patch?: never; patch?: never;
trace?: never; trace?: never;
}; };
"/books/duplicate-books/dismissals": {
parameters: {
query?: never;
header?: never;
path?: never;
cookie?: never;
};
get?: never;
put?: never;
/** DismissDuplicateBooks */
post: operations["BooksDuplicateBooksDismissalsDismissDuplicateBooks"];
/** RestoreDuplicateBooks */
delete: operations["BooksDuplicateBooksDismissalsRestoreDuplicateBooks"];
options?: never;
head?: never;
patch?: never;
trace?: never;
};
"/books/{book_id}": { "/books/{book_id}": {
parameters: { parameters: {
query?: never; query?: never;
@@ -110,6 +145,40 @@ export interface paths {
patch?: never; patch?: never;
trace?: never; trace?: never;
}; };
"/books/duplicate-books": {
parameters: {
query?: never;
header?: never;
path?: never;
cookie?: never;
};
/** ListDuplicateBooks */
get: operations["BooksDuplicateBooksListDuplicateBooks"];
put?: never;
post?: never;
delete?: never;
options?: never;
head?: never;
patch?: never;
trace?: never;
};
"/books/merge": {
parameters: {
query?: never;
header?: never;
path?: never;
cookie?: never;
};
get?: never;
put?: never;
/** MergeBooks */
post: operations["BooksMergeMergeBooks"];
delete?: never;
options?: never;
head?: never;
patch?: never;
trace?: never;
};
"/books/progress": { "/books/progress": {
parameters: { parameters: {
query?: never; query?: never;
@@ -637,6 +706,12 @@ export interface components {
cover_image?: string | null; cover_image?: string | null;
files?: string[]; files?: string[];
}; };
/** BookMerge */
BookMerge: {
survivor_id: number;
merged_ids: number[];
metadata?: components["schemas"]["BookMetadataUpdate"] | null;
};
/** BookMetadataUpdate */ /** BookMetadataUpdate */
BookMetadataUpdate: { BookMetadataUpdate: {
title?: string | null; title?: string | null;
@@ -710,6 +785,46 @@ export interface components {
BooksCreateFromFiles: { BooksCreateFromFiles: {
files?: string[]; files?: string[];
}; };
/** BooksUploadResult */
BooksUploadResult: {
created: components["schemas"]["BookRead"][];
skipped: components["schemas"]["DuplicateFileRead"][];
possible_duplicates?: components["schemas"]["PossibleDuplicateRead"][];
};
/** DuplicateBookGroupRead */
DuplicateBookGroupRead: {
books: components["schemas"]["DuplicateBookRead"][];
};
/** DuplicateBookRead */
DuplicateBookRead: {
book_id: number;
title: string;
authors: string[];
library_id: number;
cover_image?: string | null;
matched_on: string[];
};
/** DuplicateDismissal */
DuplicateDismissal: {
book_a_id: number;
book_b_id: number;
};
/** DuplicateFileRead */
DuplicateFileRead: {
filename: string;
hash: string;
size: number;
library_id: number;
book_id?: number | null;
book_title?: string | null;
};
/** FileFingerprint */
FileFingerprint: {
hash: string;
size: number;
/** @default */
filename: string;
};
/** FileMetadataRead */ /** FileMetadataRead */
FileMetadataRead: { FileMetadataRead: {
id: number; id: number;
@@ -777,6 +892,12 @@ export interface components {
refresh_token?: string | null; refresh_token?: string | null;
expires_in?: number | null; expires_in?: number | null;
}; };
/** PossibleDuplicateRead */
PossibleDuplicateRead: {
book_id: number;
title: string;
candidates: components["schemas"]["DuplicateBookRead"][];
};
/** PublisherRead */ /** PublisherRead */
PublisherRead: { PublisherRead: {
id: number; id: number;
@@ -828,6 +949,7 @@ export interface operations {
parameters: { parameters: {
query?: { query?: {
library_id?: number | null; library_id?: number | null;
allow_duplicates?: boolean;
}; };
header?: never; header?: never;
path: { path: {
@@ -908,6 +1030,47 @@ export interface operations {
}; };
}; };
}; };
BooksDuplicateFilesCheckDuplicateFiles: {
parameters: {
query?: {
library_id?: number | null;
};
header?: never;
path?: never;
cookie?: never;
};
requestBody: {
content: {
"application/json": components["schemas"]["FileFingerprint"][];
};
};
responses: {
/** @description Request fulfilled, document follows */
200: {
headers: {
[name: string]: unknown;
};
content: {
"application/json": components["schemas"]["DuplicateFileRead"][];
};
};
/** @description Bad request syntax or unsupported method */
400: {
headers: {
[name: string]: unknown;
};
content: {
"application/json": {
status_code: number;
detail: string;
extra?: null | {
[key: string]: unknown;
} | unknown[];
};
};
};
};
};
BooksListBooks: { BooksListBooks: {
parameters: { parameters: {
query?: { query?: {
@@ -917,7 +1080,7 @@ export interface operations {
tags?: number[] | null; tags?: number[] | null;
shelves?: number[] | null; shelves?: number[] | null;
progress?: string[] | null; progress?: string[] | null;
ids?: string[] | null; ids?: number[] | null;
searchString?: string | null; searchString?: string | null;
searchIgnoreCase?: boolean | null; searchIgnoreCase?: boolean | null;
currentPage?: number; currentPage?: number;
@@ -969,6 +1132,7 @@ export interface operations {
parameters: { parameters: {
query?: { query?: {
library_id?: number | null; library_id?: number | null;
allow_duplicates?: boolean;
}; };
header?: never; header?: never;
path?: never; path?: never;
@@ -1047,6 +1211,7 @@ export interface operations {
parameters: { parameters: {
query?: { query?: {
library_id?: number | null; library_id?: number | null;
allow_duplicates?: boolean;
}; };
header?: never; header?: never;
path?: never; path?: never;
@@ -1060,21 +1225,86 @@ export interface operations {
responses: { responses: {
/** @description Document created, URL follows */ /** @description Document created, URL follows */
201: { 201: {
headers: {
[name: string]: unknown;
};
content: {
"application/json": components["schemas"]["BooksUploadResult"];
};
};
/** @description Bad request syntax or unsupported method */
400: {
headers: { headers: {
[name: string]: unknown; [name: string]: unknown;
}; };
content: { content: {
"application/json": { "application/json": {
items?: components["schemas"]["BookRead"][]; status_code: number;
/** @description Maximal number of items to send. */ detail: string;
limit?: number; extra?: null | {
/** @description Offset from the beginning of the query. */ [key: string]: unknown;
offset?: number; } | unknown[];
/** @description Total number of items. */
total?: number;
}; };
}; };
}; };
};
};
BooksDuplicateBooksDismissalsDismissDuplicateBooks: {
parameters: {
query?: never;
header?: never;
path?: never;
cookie?: never;
};
requestBody: {
content: {
"application/json": components["schemas"]["DuplicateDismissal"];
};
};
responses: {
/** @description Request fulfilled, nothing follows */
204: {
headers: {
[name: string]: unknown;
};
content?: never;
};
/** @description Bad request syntax or unsupported method */
400: {
headers: {
[name: string]: unknown;
};
content: {
"application/json": {
status_code: number;
detail: string;
extra?: null | {
[key: string]: unknown;
} | unknown[];
};
};
};
};
};
BooksDuplicateBooksDismissalsRestoreDuplicateBooks: {
parameters: {
query: {
book_a_id: number;
book_b_id: number;
};
header?: never;
path?: never;
cookie?: never;
};
requestBody?: never;
responses: {
/** @description Request fulfilled, nothing follows */
204: {
headers: {
[name: string]: unknown;
};
content?: never;
};
/** @description Bad request syntax or unsupported method */ /** @description Bad request syntax or unsupported method */
400: { 400: {
headers: { headers: {
@@ -1254,6 +1484,84 @@ export interface operations {
}; };
}; };
}; };
BooksDuplicateBooksListDuplicateBooks: {
parameters: {
query?: {
library_id?: number | null;
};
header?: never;
path?: never;
cookie?: never;
};
requestBody?: never;
responses: {
/** @description Request fulfilled, document follows */
200: {
headers: {
[name: string]: unknown;
};
content: {
"application/json": components["schemas"]["DuplicateBookGroupRead"][];
};
};
/** @description Bad request syntax or unsupported method */
400: {
headers: {
[name: string]: unknown;
};
content: {
"application/json": {
status_code: number;
detail: string;
extra?: null | {
[key: string]: unknown;
} | unknown[];
};
};
};
};
};
BooksMergeMergeBooks: {
parameters: {
query?: {
library_id?: number | null;
};
header?: never;
path?: never;
cookie?: never;
};
requestBody: {
content: {
"application/json": components["schemas"]["BookMerge"];
};
};
responses: {
/** @description Document created, URL follows */
201: {
headers: {
[name: string]: unknown;
};
content: {
"application/json": components["schemas"]["BookRead"];
};
};
/** @description Bad request syntax or unsupported method */
400: {
headers: {
[name: string]: unknown;
};
content: {
"application/json": {
status_code: number;
detail: string;
extra?: null | {
[key: string]: unknown;
} | unknown[];
};
};
};
};
};
BooksProgressSetBookProgressBatch: { BooksProgressSetBookProgressBatch: {
parameters: { parameters: {
query: { query: {
@@ -1931,7 +2239,7 @@ export interface operations {
tags?: number[] | null; tags?: number[] | null;
shelves?: number[] | null; shelves?: number[] | null;
progress?: string[] | null; progress?: string[] | null;
ids?: string[] | null; ids?: number[] | null;
searchString?: string | null; searchString?: string | null;
searchIgnoreCase?: boolean | null; searchIgnoreCase?: boolean | null;
currentPage?: number; currentPage?: number;
@@ -2018,7 +2326,7 @@ export interface operations {
OpdsLibraryLibraryIdCollectionTypeGetLibraryCollectionFeed: { OpdsLibraryLibraryIdCollectionTypeGetLibraryCollectionFeed: {
parameters: { parameters: {
query?: { query?: {
ids?: string[] | null; ids?: number[] | null;
searchString?: string | null; searchString?: string | null;
searchIgnoreCase?: boolean | null; searchIgnoreCase?: boolean | null;
currentPage?: number; currentPage?: number;
@@ -2147,7 +2455,7 @@ export interface operations {
tags?: number[] | null; tags?: number[] | null;
shelves?: number[] | null; shelves?: number[] | null;
progress?: string[] | null; progress?: string[] | null;
ids?: string[] | null; ids?: number[] | null;
searchString?: string | null; searchString?: string | null;
searchIgnoreCase?: boolean | null; searchIgnoreCase?: boolean | null;
currentPage?: number; currentPage?: number;
+56 -16
View File
@@ -1,9 +1,9 @@
import { getContext, setContext } from 'svelte'; import { getContext, setContext } from 'svelte';
import { invalidate } from '$app/navigation'; import { invalidate } from '$app/navigation';
import type { Book, PaginatedResponse } from '$lib/schema'; import type { Book, BooksUploadResult, DuplicateBook, DuplicateFile } from '$lib/schema';
export type UploadStatus = 'queued' | 'uploading' | 'done' | 'failed'; export type UploadStatus = 'queued' | 'uploading' | 'done' | 'skipped' | 'failed';
export interface UploadJob { export interface UploadJob {
id: string; id: string;
@@ -15,10 +15,23 @@ export interface UploadJob {
status: UploadStatus; status: UploadStatus;
error?: string; error?: string;
book?: Book; book?: Book;
/**
* Files the library already held, which were not stored again. A job with
* nothing left over lands as `skipped`; one that had something new is `done`
* and still carries these, since the reader asked for those files too.
*/
duplicates?: DuplicateFile[];
/**
* Books already in the library that the one this job created might be a second
* copy of. Much weaker than `duplicates`: the bytes are new and only the metadata
* agrees, which a second edition and a translation both do. The book was stored.
*/
possibleDuplicates?: DuplicateBook[];
} }
export interface UploadSummary { export interface UploadSummary {
created: number; created: number;
skipped: number;
failed: number; failed: number;
firstBook?: Book; firstBook?: Book;
} }
@@ -70,12 +83,30 @@ export class UploadQueueState {
readonly total = $derived(this.jobs.length); readonly total = $derived(this.jobs.length);
readonly done = $derived(this.jobs.filter((job) => job.status === 'done').length); readonly done = $derived(this.jobs.filter((job) => job.status === 'done').length);
readonly skipped = $derived(this.jobs.filter((job) => job.status === 'skipped').length);
readonly failed = $derived(this.jobs.filter((job) => job.status === 'failed').length); readonly failed = $derived(this.jobs.filter((job) => job.status === 'failed').length);
readonly active = $derived( readonly active = $derived(
this.jobs.some((job) => job.status === 'queued' || job.status === 'uploading') this.jobs.some((job) => job.status === 'queued' || job.status === 'uploading')
); );
readonly current = $derived(this.jobs.find((job) => job.status === 'uploading')); readonly current = $derived(this.jobs.find((job) => job.status === 'uploading'));
readonly settled = $derived(this.done + this.failed); readonly settled = $derived(this.done + this.skipped + this.failed);
/**
* Whether the run left something the reader still has to see.
*
* A skipped book is not a failure, but it is the only place that says the file
* was already here and the only place to override it from. A book that may
* already be in the library counts too: it is a note the reader has to actually
* read, and a tray that clears itself after six seconds is one they never will.
*/
readonly needsAttention = $derived(
this.jobs.some(
(job) =>
job.status === 'failed' ||
(job.duplicates?.length ?? 0) > 0 ||
(job.possibleDuplicates?.length ?? 0) > 0
)
);
#running = false; #running = false;
#dismissTimer: ReturnType<typeof setTimeout> | undefined; #dismissTimer: ReturnType<typeof setTimeout> | undefined;
@@ -129,20 +160,20 @@ export class UploadQueueState {
release() { release() {
this.#held = false; this.#held = false;
if (!this.active && this.failed === 0 && this.total > 0) this.#scheduleDismiss(); if (!this.active && !this.needsAttention && this.total > 0) this.#scheduleDismiss();
} }
/** /**
* Clears itself only when everything worked. A run with failures stays until * Clears itself only when everything went in cleanly. A run with failures or
* dismissed it is the only record of what did not make it in, and the only * skipped files stays until dismissed it is the only record of what did not
* place to retry from. * make it in, and the only place to retry or override from.
*/ */
#scheduleDismiss() { #scheduleDismiss() {
clearTimeout(this.#dismissTimer); clearTimeout(this.#dismissTimer);
if (this.#held) return; if (this.#held) return;
this.#dismissTimer = setTimeout(() => { this.#dismissTimer = setTimeout(() => {
if (!this.active && this.failed === 0) this.jobs = []; if (!this.active && !this.needsAttention) this.jobs = [];
}, DISMISS_AFTER_MS); }, DISMISS_AFTER_MS);
} }
@@ -169,26 +200,35 @@ export class UploadQueueState {
const index = this.jobs.findIndex((job) => job.status === 'queued'); const index = this.jobs.findIndex((job) => job.status === 'queued');
if (index === -1) break; if (index === -1) break;
const job = this.jobs[index];
this.#patch(index, { status: 'uploading' }); this.#patch(index, { status: 'uploading' });
try { try {
const body = new FormData(); const body = new FormData();
for (const file of this.jobs[index].files) body.append('files', file); for (const file of job.files) body.append('files', file);
const response = await fetch( const response = await fetch(
`/api/books/fromFiles?library_id=${encodeURIComponent(String(this.jobs[index].libraryId))}`, `/api/books/fromFiles?library_id=${encodeURIComponent(String(job.libraryId))}`,
{ method: 'POST', body } { method: 'POST', body }
); );
if (!response.ok) throw new Error(`The server returned ${response.status}`); if (!response.ok) throw new Error(`The server returned ${response.status}`);
const result: PaginatedResponse<Book> = await response.json(); const result: BooksUploadResult = await response.json();
const book = result.items[0]; const book = result.created[0];
created += result.total ?? result.items.length; created += result.created.length;
firstBook ??= book; firstBook ??= book;
this.#patch(index, { status: 'done', book }); // Nothing created means every file in this folder was already here.
// That is not a failure, and it is not something to hide either.
this.#patch(index, {
status: result.created.length === 0 ? 'skipped' : 'done',
book,
duplicates: result.skipped,
// One job is one book, so it has at most one set of candidates.
possibleDuplicates: result.possible_duplicates?.[0]?.candidates
});
} catch (error) { } catch (error) {
// One bad book must not take the rest of the queue with it. // One bad book must not take the rest of the queue with it.
console.error(`Failed to upload ${this.jobs[index].label}`, error); console.error(`Failed to upload ${this.jobs[index].label}`, error);
@@ -205,9 +245,9 @@ export class UploadQueueState {
if (created > 0) await invalidate('app:books'); if (created > 0) await invalidate('app:books');
if (this.failed === 0) this.#scheduleDismiss(); if (!this.needsAttention) this.#scheduleDismiss();
onFinished?.({ created, failed: this.failed, firstBook }); onFinished?.({ created, skipped: this.skipped, failed: this.failed, firstBook });
} }
#patch(index: number, changes: Partial<UploadJob>) { #patch(index: number, changes: Partial<UploadJob>) {
@@ -1,9 +1,14 @@
<script lang="ts"> <script lang="ts">
import { page } from '$app/state'; import { page } from '$app/state';
import { resolve } from '$app/paths'; import { resolve } from '$app/paths';
import { getLibraryState } from '$lib/state/library.svelte';
let { children } = $props(); let { children } = $props();
// Set in (root)/+layout.svelte, above this group, so listing the libraries in
// the nav needs no load function.
const libraryState = getLibraryState();
// Route ids rather than paths: resolve() is called in the markup so it stays a // Route ids rather than paths: resolve() is called in the markup so it stays a
// direct call the lint rule can see, and the active check compares route ids. // direct call the lint rule can see, and the active check compares route ids.
// A pathname comparison would miss during SSR, where resolve() returns a // A pathname comparison would miss during SSR, where resolve() returns a
@@ -14,12 +19,28 @@
{ title: 'Libraries', routeId: '/(root)/settings/libraries' }, { title: 'Libraries', routeId: '/(root)/settings/libraries' },
{ title: 'Devices', routeId: '/(root)/settings/devices' } { title: 'Devices', routeId: '/(root)/settings/devices' }
] as const; ] as const;
/**
* Whether a nested library item is the one being looked at.
*
* Both halves are needed: the route id alone is shared by every library, so
* matching on it would light all of them up at once.
*/
function isActiveLibrary(id: number) {
return (
page.route.id?.startsWith('/(root)/settings/libraries/[libraryId]') === true &&
page.params.libraryId === String(id)
);
}
</script> </script>
<div class="flex h-[calc(100vh-var(--header-height)-2rem)] flex-col"> <div class="flex h-[calc(100vh-var(--header-height)-2rem)] flex-col">
<h1 class="font-serif text-xl font-medium tracking-tight">Settings</h1> <h1 class="font-serif text-xl font-medium tracking-tight">Settings</h1>
<div class="mt-6 flex flex-1 gap-8 overflow-hidden"> <div class="mt-6 flex flex-1 gap-8 overflow-hidden">
<nav class="flex w-48 shrink-0 flex-col gap-1"> <!-- Scrolls on its own now that it holds a list that grows with the number
of libraries; the row above it is overflow-hidden, so without this a
long list is clipped rather than reachable. -->
<nav class="flex w-48 shrink-0 flex-col gap-1 overflow-y-auto">
{#each items as item} {#each items as item}
<a <a
href={resolve(item.routeId)} href={resolve(item.routeId)}
@@ -30,6 +51,26 @@
> >
{item.title} {item.title}
</a> </a>
<!-- Libraries is the one item with children: a library's own settings
hang off it, so every library and every section it has stay one
click from here. -->
{#if item.routeId === '/(root)/settings/libraries'}
{#each libraryState.libraries as library (library.id)}
<a
href={resolve('/(root)/settings/libraries/[libraryId]/duplicates', {
libraryId: String(library.id)
})}
class="ml-3 truncate rounded-md border-l px-3 py-1.5 text-sm transition-colors hover:bg-muted {isActiveLibrary(
library.id
)
? 'bg-muted font-medium'
: 'text-muted-foreground'}"
>
{library.name}
</a>
{/each}
{/if}
{/each} {/each}
</nav> </nav>
<div class="flex-1 overflow-auto">{@render children()}</div> <div class="flex-1 overflow-auto">{@render children()}</div>
@@ -0,0 +1,48 @@
<script lang="ts">
import { page } from '$app/state';
import { resolve } from '$app/paths';
import { getLibraryState } from '$lib/state/library.svelte';
let { children } = $props();
// Set in (root)/+layout.svelte, above the settings group, so the libraries are
// already here — this pane needs no load function of its own.
const libraryState = getLibraryState();
const library = $derived(
libraryState.libraries.find((lib) => String(lib.id) === page.params.libraryId)
);
// Duplicates is the only section today. The strip exists so General and a
// danger zone have an obvious place to land; neither is built yet, and an
// empty tab is worse than no tab.
//
// Active state is matched on route id, not pathname, for the reason spelled
// out in settings/+layout.svelte.
const sections = [
{ title: 'Duplicates', routeId: '/(root)/settings/libraries/[libraryId]/duplicates' }
] as const;
</script>
<div class="flex flex-col gap-4">
<div>
<h2 class="text-lg font-semibold">{library?.name ?? 'Library'}</h2>
<p class="text-sm text-muted-foreground">Settings for this library</p>
</div>
<nav class="flex gap-1 border-b">
{#each sections as section (section.routeId)}
<a
href={resolve(section.routeId, { libraryId: page.params.libraryId! })}
class="-mb-px border-b-2 px-3 py-2 text-sm font-medium transition-colors hover:text-foreground {page
.route.id === section.routeId
? 'border-foreground'
: 'border-transparent text-muted-foreground'}"
>
{section.title}
</a>
{/each}
</nav>
{@render children()}
</div>
@@ -0,0 +1,10 @@
import { resolve } from '$app/paths';
import { redirect } from '@sveltejs/kit';
export async function load({ params }) {
// The library pane is its sections; land on the first one.
redirect(
303,
resolve('/(root)/settings/libraries/[libraryId]/duplicates', { libraryId: params.libraryId })
);
}
@@ -0,0 +1,18 @@
import { listBooks, listDuplicateBooks } from '$lib/api/book.remote.js';
export async function load({ params, depends }) {
// Dismissing or merging a group re-runs this, so the card leaves the screen.
depends('app:duplicate-books');
const groups = await listDuplicateBooks(params.libraryId);
// The groups carry only enough to render a card. Merging needs the whole record —
// identifiers, description, publisher — so fetch them in one go rather than per
// card, and let the dialog pick out the books for its own group.
const ids = [...new Set(groups.flatMap((group) => group.books.map((book) => book.book_id)))];
const books = ids.length
? await listBooks({ ids, pageSize: ids.length })
: { items: [] };
return { groups, books: books.items };
}
@@ -0,0 +1,249 @@
<script lang="ts">
import { page } from '$app/state';
import { resolve } from '$app/paths';
import { invalidate } from '$app/navigation';
import { toast } from 'svelte-sonner';
import { CopyCheck, Fingerprint, GitMerge, SlidersHorizontal, Type } from '@lucide/svelte';
import * as Card from '$lib/components/ui/card/index.js';
import * as Empty from '$lib/components/ui/empty/index.js';
import * as DropdownMenu from '$lib/components/ui/dropdown-menu/index.js';
import { Badge } from '$lib/components/ui/badge/index.js';
import { Button, buttonVariants } from '$lib/components/ui/button/index.js';
import BookImage from '$lib/components/view/book-image.svelte';
import GeneratedCover from '$lib/components/view/generated-cover.svelte';
import { dismissDuplicateBooks } from '$lib/api/book.remote';
import MergeBooks from '$lib/components/forms/merge-books/merge-books.svelte';
import { mergeInto } from '$lib/components/forms/merge-books/merge';
import type { Book, DuplicateBookGroup } from '$lib/schema';
let { data }: { data: { groups: DuplicateBookGroup[]; books: Book[] } } = $props();
// The group open in the workbench, or null when it is closed.
let merging = $state<DuplicateBookGroup | null>(null);
/** The full records behind a group, in the order the group lists them. */
function recordsFor(group: DuplicateBookGroup): Book[] {
return group.books
.map(({ book_id }) => data.books.find((book) => book.id === book_id))
.filter((book): book is Book => book !== undefined);
}
// Which groups have a request in flight — a dismissal or a quick merge — so a
// slow round trip cannot be started twice.
let dismissing = $state<number[]>([]);
let quickMerging = $state<number[]>([]);
/**
* Merge without opening the workbench, keeping the first book as it stands.
*
* The common case by far: the group is the same book twice, one record is as
* good as the other, and the point is to end up with one. Resolving metadata
* field by field is the other button.
*/
async function quickMerge(group: DuplicateBookGroup) {
const key = keyOf(group);
if (quickMerging.includes(key)) return;
const [survivor, ...folded] = recordsFor(group);
if (!survivor || folded.length === 0) return;
quickMerging = [...quickMerging, key];
try {
await mergeInto(page.params.libraryId!, survivor, folded);
} finally {
quickMerging = quickMerging.filter((id) => id !== key);
}
}
/** A group is named by its lowest book id, which the backend orders it by. */
function keyOf(group: DuplicateBookGroup) {
return group.books[0].book_id;
}
function reasonLabel(reason: string) {
if (reason === 'identifier') return 'Same identifier';
if (reason === 'title-author') return 'Same title and author';
return reason;
}
/**
* Dismiss every pairing in a group at once.
*
* A group is held together pair by pair, so saying "these are not duplicates"
* about three books means saying it about all three pairs — dismissing only the
* first would leave the rest of the group standing and the screen unchanged.
*/
async function notDuplicates(group: DuplicateBookGroup) {
const key = keyOf(group);
if (dismissing.includes(key)) return;
dismissing = [...dismissing, key];
const ids = group.books.map((book) => book.book_id);
const pairs = ids.flatMap((a, index) =>
ids.slice(index + 1).map((b) => ({ book_a_id: a, book_b_id: b }))
);
try {
await Promise.all(pairs.map((pair) => dismissDuplicateBooks(pair)));
await invalidate('app:duplicate-books');
toast.success('Marked as different books');
} catch {
toast.error('Could not mark these as different books');
} finally {
dismissing = dismissing.filter((id) => id !== key);
}
}
</script>
<!-- No max width and no padding of its own: the settings pane sets the width and
is the thing that scrolls, so a wrapper that centres inside it only makes the
cards narrower than they need to be. -->
<div class="flex flex-col gap-6 pb-4">
<p class="text-sm text-muted-foreground">
Books whose metadata matches. A second edition, a translation and a different scan of one book
all look like this, so nothing here has been changed or removed — this is a list to read, not a
problem to fix.
</p>
{#if data.groups.length === 0}
<Empty.Root>
<Empty.Header>
<Empty.Media variant="icon">
<CopyCheck />
</Empty.Media>
<Empty.Title>Nothing looks duplicated</Empty.Title>
<Empty.Description>
No two books in this library share an identifier, or a title and an author.
</Empty.Description>
</Empty.Header>
</Empty.Root>
{:else}
{#each data.groups as group (keyOf(group))}
{@const busy = dismissing.includes(keyOf(group)) || quickMerging.includes(keyOf(group))}
{@const records = recordsFor(group)}
<Card.Root>
<Card.Header>
<Card.Title>{group.books.length} books look like the same book</Card.Title>
<Card.Action class="flex gap-2">
<Button
variant="outline"
size="sm"
disabled={busy}
onclick={() => notDuplicates(group)}
>
Not duplicates
</Button>
<!-- A choice rather than a jump straight into the workbench:
most groups are one book twice, where there is nothing to
resolve and the dialog is a step in the way.
Disabled until every record in the group came back, so
neither action can run on a partial group. -->
<DropdownMenu.Root>
<DropdownMenu.Trigger
class={buttonVariants({ size: 'sm' })}
disabled={busy || records.length !== group.books.length}
>
Merge…
</DropdownMenu.Trigger>
<DropdownMenu.Content align="end">
<DropdownMenu.Item onclick={() => quickMerge(group)}>
<GitMerge class="size-4 shrink-0" />
Merge into first selected
</DropdownMenu.Item>
<DropdownMenu.Item onclick={() => (merging = group)}>
<SlidersHorizontal class="size-4 shrink-0" />
Manual merge…
</DropdownMenu.Item>
</DropdownMenu.Content>
</DropdownMenu.Root>
</Card.Action>
</Card.Header>
<Card.Content>
<ul class="flex flex-wrap gap-4">
{#each group.books as book (book.book_id)}
<li class="w-36">
<a
href={resolve('/(root)/(library)/book/[bookId]', {
bookId: String(book.book_id)
})}
class="group flex flex-col gap-2"
>
<span
class="block aspect-9/12 w-full overflow-hidden rounded-sm bg-muted shadow-md transition-all group-hover:brightness-75"
>
{#if book.cover_image}
<BookImage src="/api/{book.cover_image}" class="h-full w-full object-cover" />
{:else}
<!-- A blank swatch here is worse than anywhere else: this screen
is a side-by-side comparison, and two of them are impossible
to tell apart. The candidate carries author names rather than
records, which is all the cover draws. -->
<GeneratedCover
book={{
title: book.title,
authors: book.authors.map((name) => ({ name }))
}}
/>
{/if}
</span>
<span class="line-clamp-2 font-serif text-sm group-hover:underline">
{book.title}
</span>
</a>
<p class="line-clamp-2 text-xs text-muted-foreground">
{book.authors.join(', ')}
</p>
<!-- Why this book is in the group, so the reader can judge the
evidence rather than take the grouping on trust. -->
<p class="mt-1 flex flex-wrap gap-1">
{#each book.matched_on as reason (reason)}
<Badge variant="secondary" class="gap-1 text-[10px] font-normal">
{#if reason === 'identifier'}
<Fingerprint class="size-3" />
{:else}
<Type class="size-3" />
{/if}
{reasonLabel(reason)}
</Badge>
{/each}
</p>
</li>
{/each}
</ul>
</Card.Content>
</Card.Root>
{/each}
{/if}
</div>
<!--
Keyed on the group so a second merge starts from that group's records rather
than the previous one's draft, the same way the edit dialog keys on the book.
-->
{#if merging}
{#key merging.books[0].book_id}
<MergeBooks
books={recordsFor(merging)}
libraryId={page.params.libraryId!}
bind:open={
() => merging !== null,
(value) => {
// Bound, not passed as a bare `true`: the dialog closes itself on an
// outside click or Escape, and if that never reaches `merging` the
// group stays set — the menu item then re-assigns the same group,
// nothing changes, and the workbench never reopens.
if (!value) merging = null;
}
}
onmerged={() => (merging = null)}
/>
{/key}
{/if}