Data Admin Guide

Creating a dataset

Datasets start as a draft and are published once their resources and metadata are ready. Drafts never appear in search results or dataset listings until published.

Log in to the dashboard
Click “Create Draft” (dropping files also creates a draft and adds them as resources) — the name (URL identifier), organization, and license can be set later
Add resources on the edit page (file upload / external URL) — the pipeline runs while the dataset is still a draft, so you can check previews before publishing
Finalize the metadata and click “Save & Publish”

Basic information

Field	Description	Required to publish
Name (URL slug)	Alphanumeric, hyphens, underscores, periods	Yes
Title	Display title	-
Description	Detailed description of the dataset	-
Organization	The organization that owns the data	Yes
License	Data license	Yes
Public / Private	Private datasets are only visible to organization members	-

Fields marked “Required to publish” may stay unset while the dataset is a draft, but all three must be set before it can be published. While the name is blank, an auto-generated identifier is used as a placeholder (the dataset cannot be published with it).

Managing and deleting drafts

Drafts are managed in the dashboard “Drafts” tab. They can be edited by their creator and site admins, plus the editors of the owning organization once one is set.

Deleting a draft skips the trash and permanently deletes it together with its resource files (it cannot be restored). If the deletion fails partway, the draft stays in the list marked “Deletion incomplete”, and “Retry Delete” completes the permanent deletion.

Managing resources

Resources are files or URLs associated with a dataset.

Adding resources

File upload — Upload directly from the browser (up to 100 MB)
External URL — Specify a URL to an external file

Adding, editing, replacing or deleting a resource is saved right away. Only a new order needs “Save order”.

Grouping resources into sections

When a dataset has many resources, section headings group them. “Add section” in the resource list names a new, empty heading at the end.

A heading is a divider. Drag it into place and the resources from there down to the next heading belong to the section. Moving a heading never changes the order of the resources
Moving a heading down hands the resources it passes to the section above (or to the root); moving it up takes them in. Set down inside another section it splits that section there; set down directly on a heading it takes nothing
A dragged resource joins the section of the resource above it. Dropped on a heading it becomes that section’s first member; dropped at the top of the list it belongs to no section
Rename and dissolve (dissolving keeps the resources) from the buttons on the heading. A section name is used once per dataset
A heading with no members is not saved until a resource enters it. A resource added at the end, or a dropped file, goes into the section the end is in
Like the order, “Save order” commits the change

A section is a display divider on the dataset page and does not change who can see a resource. A / in the name nests the headings on the public page (2024/Tokyo shows “Tokyo” under “2024”, up to three levels). Section names are searchable.

Supported formats

KUKAN supports many formats including CSV, TSV, JSON, GeoJSON, XML, PDF, Excel, and Ichitaro.

Uploaded files are automatically processed through the pipeline:

Text file encoding detection and conversion
Parquet conversion of CSV/TSV, and of JSON/GeoJSON whose records read as a table (for table preview)
ZIP file manifest generation
Recording a version (Version history)
Text extraction for search (so a file’s contents can match a query)
An AI-written description, where the site has it enabled (AI-written descriptions)

Tables a BI tool can open

A resource with a table preview is served as an OData feed as it is (User Guide) — nothing to turn on. It is not served in the cases below, and the OData button on the resource page says which one applies.

The dataset is private or a draft — only resources in a public dataset are served
A heading uses characters an identifier cannot — brackets, spaces, punctuation and dots among them. Fix the heading (Population (people) → Population_people) and replace the file, and the table is served
Two columns share a name — there is no way to say which one is meant. Give them distinct headings
A single row is very large — a table with many columns, or with long text in one cell, can fail to fit in memory partway through a read. The button warns when this is a risk; point those readers at the download instead

With a primary key declared, that column identifies the rows and no RowId column is added (Declaring a primary key).

Column roles

For a tabular resource, each column is given a role — name, category, free text, identifier, number or date, or number written as text — decided automatically from its values and its field name. The role decides how search uses the column: name and category columns are searched by their values, free-text columns as sentences, and the rest are not used for search.

The field list on the public page shows the role alone. The resource editing screen in the dashboard shows it with the reason it was decided (“92% of values are number-shaped” and the like). For a column of numbers written as text, it also lists the values that do not read as numbers — what to fix in the data.

Version history

Every time a resource’s file is replaced, the content it replaced is kept as a version. You can download any past version, and for a table (CSV/TSV, or JSON/GeoJSON read as one) you can see how many rows were added or removed between two versions — and, with a primary key declared, how each row changed.

A version is created only when the content actually changed. Re-fetching an external URL that still serves the same bytes adds no version, and the resource’s processing status says “No new version — the file matches the content already stored”.

Viewing the version history

The resource edit page has a Version history section. Versions are listed newest first, each row showing the version number, creation time, size, origin (Upload, Fetched, or Re-published v○), and the primary key the version was loaded under.

Every version can be downloaded exactly as it was. The version number is added to the filename.

Declaring a primary key

The Primary key control on the resource edit page names the columns used to match a row to the same row in another version (a combination of columns works too). With a key, a diff can count the rows that changed.

The columns you choose — the combination, if more than one — have to be present in the content, hold no empty values, and not repeat across rows. Where they are not met, the reason is shown before you apply
A column inferred as Number (floating-point) cannot be chosen. 1.0 and 1 are the same value there, so two spellings the file keeps apart collapse into one key, and a value written out is not guaranteed to read back as itself. Such a column cannot identify a row — choose an Integer or String column instead
Applying creates one version, and versions from then on are loaded under this key. Diffs of past versions do not change
When content arrives that does not satisfy the key, that version is left out of row-level diffs. The reason (a column is missing, has empty values, or does not identify a row) appears in the version list and on the diff panel
A column that does not keep pointing at the same row — a sequence number reissued on every publication, say — makes a poor key. If that is all there is, declaring no key is the safer choice: editing one row then shows as one addition plus one removal, but two different things are never presented as the same row
The primary key is also what the OData feed keys on. Where it qualifies, that column identifies each row and the RowId column KUKAN would otherwise add disappears; where it does not, RowId is added and the resource page’s OData button says why

Viewing a diff

The Diff control on a version’s row fetches the diff against the preceding version. It is computed when you open it, so simply listing the versions does not run anything.

Shows N rows added / N rows removed, with a few representative rows from each side
With a primary key declared, it also shows “N rows changed” with representative rows before → after. Without one, editing one row appears as one addition plus one removal — KUKAN does not guess a correspondence and present it as change history
When the columns changed (added, removed, retyped, or reordered), the rows cannot be lined up and the panel says so instead

Diffs are available between versions of a table (CSV/TSV, or JSON/GeoJSON read as one) of 100 MB or less. When one is unavailable, the reason is shown.

Message	Meaning
This is the first version, so there is nothing to compare against.	v1 has no predecessor
v○ has no row-level snapshot, so there is nothing to compare against.	A PDF and similar, or a version created before the migration
One of the versions has been deleted, so a diff is not available.	Either side was deleted
A key column is not in v○, so its rows cannot be identified and no diff is available.	A column named in the primary key is missing from that version
A key column of v○ has empty values, so its rows cannot be identified and no diff is available.	The key columns contain empty values
The key of v○ does not identify a row, so no diff is available.	Rows repeat the same key values
Loading v○ for row-level diffs failed repeatedly, so no diff is available.	Loading that version into the diff store kept failing

The three key-related reasons clear up from the next version on once the key is corrected. A version left out is never loaded retroactively.

Stopping and reverting

For when the wrong file was uploaded, there are three operations of increasing destructiveness. Each is chosen explicitly.

Operation	Current content	Versions	Auth
Stop	Left alone	Left alone	Org editor
Stop and revert	Back on the newest saved version’s content	The restored content is issued as a new version	Org editor
Delete This Version	Falls back to the newest remaining version, if this was live	Content erased, the record remains	Sysadmin

Stop and revert puts the live content back on the most recent saved version. The restored content is issued as a new version, so the history gains a row with origin “Re-published v○” and no past version is removed. If there is no earlier version, the resource is left with no file (its metadata remains).

The preview and the search index are discarded as part of the revert and rebuilt automatically from the restored content (the rebuild runs in the background, so it takes a moment to appear). A warning appears only when discarding them, or starting the rebuild, did not succeed — then “Rebuild preview & index” finishes the job.

Deleting a version

Use this when the content itself has to become unobtainable — personal data that should never have been published, for example. Only site administrators can delete a version, a reason is required, and it is recorded in the audit log.

The version becomes unobtainable and its canonical file is deleted. Rows may remain in the version store’s internal storage, and their deletion is not guaranteed (another resource’s history can reference the same file)
Content another version holds is not erased. Each version keeps its own copy, so deleting one leaves the others serving that content — and if deleting the live version falls back onto a version holding the same content, the resource serves it again. The confirmation dialog names those versions; delete them individually if they must go too
Where storage versioning is in use, the set-aside objects disappear once their retention period (30 days by default) elapses
Deleting the live version puts the resource back on the newest remaining one; with none left, the resource is left with no file. Deleting a version that is not live changes nothing about what is served, its preview, or its search index
The history keeps a “Deleted” row and the date. The reason you type is not shown there and is not returned by the API — it goes to the audit log, which today is only readable in the database. Treat it as a note to the operators, and do not restate in it what had to be removed
This cannot be undone

One-time migration

Right after version history is enabled, resources that existed beforehand have no versions yet. A Backfill versions item appears on the site administrator’s dashboard — run it.

Layer 1: records each unversioned resource’s current file as v1 (nothing is re-fetched or re-indexed)
Layer 2: loads the versions of each tabular resource into the diff store, so diffs become available from the next update onwards. It also moves diffs loaded by an earlier release into the new format (a catalog per resource); until moved, those diffs cannot be shown (the diff says the version has no row-level snapshot)
Revert conversion: converts resources still holding versions an earlier revert set aside, so the latest-version label agrees with what is being served

The item disappears once all counts reach zero. Replacing a file before running it loses the previous content — it is never kept as a version.

AI metadata suggestions

Once you add resources and their pipeline processing completes, the “Suggest metadata with AI” button on the edit page can propose a title, notes, tags, categories (and, for a draft, the URL identifier), and per-resource names and descriptions (only when AI is enabled for the site — see the System Administrator Guide).

Prerequisite: at least one resource whose pipeline has completed, and none still processing. Otherwise the button is disabled. When processing finishes, a prompt with a “Generate suggestions” button appears.
How to use:
1. Click “Suggest metadata with AI” on the edit page (generation can take from tens of seconds to a few minutes).
2. In the dialog, compare the current values with the AI proposals, grouped into “Dataset” and “Resources” sections.
3. Turn on the adoption toggle for each field you want to adopt (all toggles start off — overwriting existing values is opt-in).
4. Optionally edit an adopted proposal inline before applying.
5. Click “Adopt selected fields”.
How adoptions apply: adopted fields are saved as soon as you adopt them (this does not publish — a draft stays a draft; if the form has a validation error they stay in the form for you to fix and save). Resource names and descriptions are saved immediately too. Tags and categories are only ever added; adopting never removes an existing one.
What suggestions are based on: CSV/TSV use the columns and the first rows; text files their opening; PDF and Office documents (docx, xlsx, pptx, the doc, xls and ppt from before 2007, and the like) and Ichitaro documents (jtd, jtt) the start of their extracted text; ZIP archives their file list. Anything else is suggested from filename and format. The generated text follows your display language.
Suggestions are never persisted (only adopted values are saved as ordinary updates). A per-user rate limit prevents overuse.

AI-written descriptions

Where the site has it enabled, every resource you register gets a short description written from the file’s contents, shown on the resource page (see the System Administrator Guide). Unlike metadata suggestions, there is nothing to adopt — it is written at the end of the pipeline and published as it is. Replace the file and it is rewritten from the new contents. Resources that predate enabling it get their descriptions from a system administrator, via “Generate AI descriptions in bulk” under AI Management.

The description also feeds search. When someone searches for “newly opened hair salons”, it is often the description that leads them to the right file.

The AI reads the contents and nothing more. It does not decide column types, primary keys, or where the header row is. It writes prose; your data is never rewritten.
How much is read depends on the format — a table’s column layout and first rows, a PDF whole unless it is very long, an image whole, a text file’s opening, the start of the text extracted from an Office document and the like, a ZIP archive’s file names. The resource page always states the extent.
A description a person wrote is never overwritten. This sits in its own block, apart from the resource’s own description field.

When no description is written

If nothing was written, the resource page states why.

Reason	What to do
Over the size limit	Out of scope. Splitting the file into smaller ones will get it a description
More than the AI can read at once	Same as above
A format the AI cannot read	Out of scope for this format
The contents are not the declared format	The source link may be dead. This appears when something registered as a PDF answers with an error page, for instance. Check the URL
No material to describe	The pipeline extracted neither text nor a schema from this file
AI not available	A site-level setting. Contact your system administrator

When the file has moved on

Replacing a file rewrites its description from the new version, but until that finishes the previous version’s description stands. While it does, the resource page says “This file is now at v3; the description has not been rewritten”, so nobody mistakes an old description for a current one.

Organization management

Organizations are the ownership unit for datasets.

Member management

Organization admins can add, remove, and change member roles.

Role	View datasets	Edit datasets	Delete	Restore / Purge	Manage members
Member	Yes	-	-	-	-
Editor	Yes	Yes	Yes	-	-
Admin	Yes	Yes	Yes	Yes	Yes

Deleted datasets are visible to organization members in the dashboard “Deleted” tab. Only admins can restore or permanently purge them.

Deleting, restoring, and purging an organization

The organization management page toggles between “Public” and “Deleted” cards (only site admins can view the “Deleted” set).

Delete (soft delete): Moves the organization to the trash. It must have no active datasets — delete or reassign them to another organization first (the delete button stays disabled while any remain).
Restore: Brings a “Deleted” organization back to active (site admins only).
Purge (permanent delete): Permanently erases the organization and all of its trashed datasets, including their files (irreversible, site admins only). It runs asynchronously as a background job, so the organization may still appear in the list until the job finishes.

Categories (Groups)

Categories classify datasets across organizations. Assign categories when creating a dataset.

Category member management

Categories also have member management with the same role structure as organizations.

Role	View category	Add datasets	Manage members
Member	Yes	-	-
Editor	Yes	Yes	-
Admin	Yes	Yes	Yes