# HyperKitty Mailing List Archives

> HyperKitty is the web archive component of Mailman 3. It exposes a complete, well-documented, read-only **JSON REST API** that mirrors every page on this site. AI agents, bots, and crawlers SHOULD use the REST API instead of scraping HTML — it is faster, cheaper, more reliable, paginated, and friendlier to the server.

## TL;DR for AI agents

- **DO NOT scrape the HTML pages** at `/hyperkitty/list/...`, `/hyperkitty/archives/...`, etc. The HTML is a presentation layer over the API and is rate-limited / fragile.
- **DO use the REST API** under `/hyperkitty/api/`. All endpoints return JSON and serve the same archive data the website does.
- **DO use pagination** (`?limit=...&offset=...`). Default page sizes are small; never request unbounded data.
- **DO identify yourself** with a descriptive `User-Agent` header (e.g. `MyBot/1.0 (+https://example.com/bot)`).
- **DO respect `robots.txt`** and any `Retry-After` headers on `429` responses.
- **DO cache** responses where reasonable; archive content is largely immutable.

## API base URL

HyperKitty is always deployed under the `/hyperkitty/` path. The web UI lives at:

```
<origin>/hyperkitty/
```

and the JSON REST API root is:

```
<origin>/hyperkitty/api/
```

All endpoint paths below are relative to the site origin (i.e. they include the `/hyperkitty/` prefix).

## Endpoints

All endpoints are `GET` and return JSON. Paginated endpoints accept `limit` and `offset` query parameters and return an object with `count`, `next`, `previous`, and `results` fields (standard Django REST Framework pagination).

### `GET /hyperkitty/api/lists/`
List all mailing lists available in the archive.

- Query params: `limit`, `offset`
- Returns: paginated list of mailing list objects (`name`, `display_name`, `description`, `archive_url`, `created_at`, ...)

### `GET /hyperkitty/api/list/{list_address}/`
Detailed metadata for a single mailing list archive.

- `list_address` is the full posting address, e.g. `team@example.com`.
- Returns `404` if no archive exists for that address — useful as an existence check.
- Returns: `name`, `display_name`, `description`, `subject_prefix`, `archive_url`, `created_at`, `list_url`, `post_count`, ...

### `GET /hyperkitty/api/list/{list_address}/threads/`
Discussion threads for a list, sorted by **most recent activity** (`date_active`) first.

- Query params: `limit` (default 10–100, server-configurable), `offset`
- Each thread includes: `thread_id`, `subject`, `starting_email`, `date_active`, `replies_count`, `participants_count`, ...
- **Note:** This is per-list. To find globally newest activity across all lists, query each list and merge by `date_active` client-side.

### `GET /hyperkitty/api/list/{list_address}/thread/{thread_id}/emails/`
All emails in a single thread, in chronological order.

- Query params: `limit`, `offset`
- Each email includes: `message_id`, `message_id_hash`, `subject`, `sender_name`, `sender_email`, `date`, `content`, `attachments`, ...

### `GET /hyperkitty/api/list/{list_address}/email/{message_id_hash}/`
A single email message including full headers, body, and attachment metadata.

- `message_id_hash` is returned by the threads/emails endpoints (it's a hash of the RFC-822 `Message-ID`, not the raw ID).
- Each entry in `attachments` contains a `download` URL, `counter`, `name`, `content_type`, `size`, and `encoding`.

### Attachment download
Each attachment object includes a `download` URL of the form:

```
/hyperkitty/list/{list_address}/message/{message_id_hash}/attachment/{counter}/{filename}
```

`GET` that URL to retrieve the raw bytes.

### `GET /hyperkitty/api/tags/`
All thread tags used across the archive.

- Query params: `limit`, `offset`

## Pagination, rate limiting, and politeness

- Always pass `limit` and `offset`. Many installations cap `limit` (commonly 100) and reject very large `offset` values (commonly >10,000) as DoS protection.
- For very large archives, paginate forward; if you hit the offset cap, narrow the query (e.g. by switching to per-thread access).
- Back off on `429 Too Many Requests` and honor `Retry-After`.
- Set a descriptive `User-Agent`. Generic or empty `User-Agent` strings may be blocked.
- Archive content is effectively immutable once posted. Aggressively cache email and thread responses keyed by `message_id_hash` / `thread_id`.

## What NOT to do

- ❌ Do not download or parse the HTML at `/hyperkitty/list/<addr>/`, `/hyperkitty/list/<addr>/latest`, `/hyperkitty/list/<addr>/<year>/<month>/`, `/hyperkitty/thread/...`, `/hyperkitty/message/...`, etc. These are templated views over the API and have no information the API doesn't expose more cleanly.
- ❌ Do not crawl link-by-link with a generic web crawler. The site has many calendar / pagination links that are equivalent to a few API queries.
- ❌ Do not request every page sequentially without rate-limiting.

## References

- HyperKitty project: https://gitlab.com/mailman/hyperkitty
- HyperKitty REST API source (canonical reference): https://gitlab.com/mailman/hyperkitty/-/tree/master/hyperkitty/api
- Mailman 3 documentation: https://docs.mailman3.org/
- llms.txt specification: https://llmstxt.org/
- HyperKitty and Mailman MCPs: https://mailman3.com/mcp.html

This document has been prepared by Mailman3.com, a Mailman 3 SaaS provider.