continuwuity/docs/DATABASE_SCHEMA.md

Continuwuity Database Schema Documentation

Continuwuity is a Matrix protocol implementation using RocksDB as its storage backend. The database is organized into column families (called "Maps" in the codebase), each serving specific purposes in the Matrix homeserver functionality.

Table Name	Access Pattern	Key Size	Value Size	Description
alias_roomid	RANDOM_SMALL	-	-	Maps room alias to room ID
alias_userid	RANDOM_SMALL	-	-	Maps room alias to user ID
aliasid_alias	RANDOM_SMALL	-	-	Maps alias ID to alias string
backupid_algorithm	RANDOM_SMALL	-	-	Key backup algorithms
backupid_etag	RANDOM_SMALL	-	-	Key backup ETags
backupkeyid_backup	RANDOM_SMALL	-	-	Backed up keys
bannedroomids	RANDOM_SMALL	-	-	Set of banned room IDs
disabledroomids	RANDOM_SMALL	-	-	Set of disabled room IDs
eventid_outlierpdu	RANDOM	48 bytes	1488 bytes	Outlier PDUs (shared cache with pduid_pdu)
eventid_pduid	RANDOM	48 bytes	16 bytes	Event ID to PDU ID mapping
eventid_shorteventid	RANDOM	48 bytes	8 bytes	Event ID to short event ID
global	RANDOM_SMALL	-	-	Global server configuration
id_appserviceregistrations	RANDOM_SMALL	-	-	Application service registrations
keychangeid_userid	RANDOM	-	-	Key change notifications
keyid_key	RANDOM_SMALL	-	-	Cryptographic keys
lazyloadedids	RANDOM_SMALL	-	-	Lazy-loaded member events
mediaid_file	RANDOM_SMALL	-	-	Media file metadata
mediaid_user	RANDOM_SMALL	-	-	Media uploader tracking
onetimekeyid_onetimekeys	RANDOM_SMALL	-	-	One-time keys
pduid_pdu	SEQUENTIAL	16 bytes	1520 bytes	Main PDU storage (shared cache with eventid_outlierpdu)
publicroomids	RANDOM_SMALL	-	-	Public room IDs
pushkey_deviceid	RANDOM_SMALL	-	-	Push key to device mapping
presenceid_presence	SEQUENTIAL_SMALL	-	-	User presence data
readreceiptid_readreceipt	RANDOM	-	-	Read receipts
referencedevents	RANDOM	-	-	Referenced events
roomid_invitedcount	RANDOM_SMALL	-	-	Room invited user count
roomid_inviteviaservers	RANDOM_SMALL	-	-	Room invite via servers
roomid_joinedcount	RANDOM_SMALL	-	-	Room joined user count
roomid_pduleaves	RANDOM_SMALL	-	-	PDU leaves per room
roomid_shortroomid	RANDOM_SMALL	-	8 bytes	Room ID to short room ID
roomid_shortstatehash	RANDOM_SMALL	-	8 bytes	Room ID to state hash
roomserverids	RANDOM_SMALL	-	-	Server IDs per room
roomsynctoken_shortstatehash	SEQUENTIAL	-	8 bytes	Sync token to state hash (special compression)
roomuserdataid_accountdata	RANDOM_SMALL	-	-	Room account data
roomuserid_invitecount	RANDOM_SMALL	-	8 bytes	Room-user invite count
roomuserid_joined	RANDOM_SMALL	-	-	Room-user joined status
roomuserid_lastprivatereadupdate	RANDOM_SMALL	-	-	Last private read update
roomuserid_leftcount	RANDOM	-	8 bytes	Room-user leave count
roomuserid_knockedcount	RANDOM_SMALL	-	8 bytes	Room-user knock count
roomuserid_privateread	RANDOM_SMALL	-	-	Private read markers
roomuseroncejoinedids	RANDOM	-	-	Users who ever joined
roomusertype_roomuserdataid	RANDOM_SMALL	-	-	Account data type mapping
senderkey_pusher	RANDOM_SMALL	-	-	Push notification senders
server_signingkeys	RANDOM	-	-	Server signing keys
servercurrentevent_data	RANDOM_SMALL	-	-	Current server events
servername_destination	RANDOM_SMALL_CACHE	-	-	Server destinations (cached)
servername_educount	RANDOM_SMALL	-	-	EDU counters
servername_override	RANDOM_SMALL_CACHE	-	-	Server name overrides (cached)
servernameevent_data	RANDOM	-	128 bytes	Server event data
serverroomids	RANDOM_SMALL	-	-	Rooms per server
shorteventid_authchain	SEQUENTIAL	8 bytes	-	Event authorization chains
shorteventid_eventid	SEQUENTIAL_SMALL	8 bytes	48 bytes	Short event ID to event ID
shorteventid_shortstatehash	SEQUENTIAL	8 bytes	8 bytes	Event to state hash mapping
shortstatehash_statediff	SEQUENTIAL_SMALL	8 bytes	-	State differences
shortstatekey_statekey	RANDOM_SMALL	8 bytes	1016 bytes	Short state key to state key
softfailedeventids	RANDOM_SMALL	48 bytes	-	Soft-failed events
statehash_shortstatehash	RANDOM	-	8 bytes	State hash to short hash
statekey_shortstatekey	RANDOM	1016 bytes	8 bytes	State key to short key
threadid_userids	SEQUENTIAL_SMALL	-	-	Thread participants
todeviceid_events	RANDOM	-	-	To-device messages
tofrom_relation	RANDOM_SMALL	8 bytes	8 bytes	Event relations
token_userdeviceid	RANDOM_SMALL	-	-	Token to device mapping
tokenids	RANDOM	-	-	Token ID management
url_previews	RANDOM	-	-	URL preview cache
userdeviceid_metadata	RANDOM_SMALL	-	-	Device metadata
userdeviceid_token	RANDOM_SMALL	-	-	Device tokens
userdevicesessionid_uiaainfo	RANDOM_SMALL	-	-	UIAA session info
userdevicetxnid_response	RANDOM_SMALL	-	-	Transaction responses
userfilterid_filter	RANDOM_SMALL	-	-	User sync filters
userid_avatarurl	RANDOM_SMALL	-	-	User avatar URLs
userid_blurhash	RANDOM_SMALL	-	-	Avatar blurhashes
userid_devicelistversion	RANDOM_SMALL	-	-	Device list versions
userid_displayname	RANDOM_SMALL	-	-	User display names
userid_lastonetimekeyupdate	RANDOM_SMALL	-	-	Last OTK update time
userid_masterkeyid	RANDOM_SMALL	-	-	Master signing keys
userid_password	RANDOM	-	-	Password hashes
userid_presenceid	RANDOM_SMALL	-	-	User presence mapping
userid_selfsigningkeyid	RANDOM_SMALL	-	-	Self-signing keys
userid_usersigningkeyid	RANDOM_SMALL	-	-	User-signing keys
useridprofilekey_value	RANDOM_SMALL	-	-	Custom profile fields
openidtoken_expiresatuserid	RANDOM_SMALL	-	-	OpenID tokens
logintoken_expiresatuserid	RANDOM_SMALL	-	-	Login tokens
userroomid_highlightcount	RANDOM	-	-	Highlight counts
userroomid_invitestate	RANDOM_SMALL	-	-	User invite states
userroomid_joined	RANDOM	-	-	User joined rooms
userroomid_leftstate	RANDOM	-	-	User leave states
userroomid_knockedstate	RANDOM_SMALL	-	-	User knock states
userroomid_notificationcount	RANDOM	-	-	Notification counts

Access Pattern Definitions

RANDOM

• Large datasets with random updates across keyspace
• Compaction priority: OldestSmallestSeqFirst
• Write buffer: 32MB
• Cache shards: 128
• Compression: Zstd level -3
• Bottommost compression: level 2

SEQUENTIAL

• Large datasets with append-heavy updates
• Compaction priority: OldestLargestSeqFirst
• Write buffer: 64MB
• Level size: 32MB
• File size: 2MB
• Compression: Zstd level -2

RANDOM_SMALL

• Small datasets with random updates
• Compaction style: Universal
• Write buffer: 16MB
• Level size: 512KB
• File size: 128KB
• Block size: 512 bytes
• Compression: Zstd level -4

SEQUENTIAL_SMALL

• Small datasets with sequential updates
• Compaction style: Universal
• Write buffer: 16MB
• Level size: 1MB
• File size: 512KB
• Compression: Zstd level -4

RANDOM_SMALL_CACHE

• Small persistent caches with TTL
• Compaction style: FIFO
• Size limit: 64MB
• TTL: 14 days
• Unique cache allocation

Special Configurations

Shared Cache Tables

• eventid_outlierpdu and pduid_pdu share cache pool
• Optimizes memory usage for related event data

High-Performance Tables

• roomsynctoken_shortstatehash: Special compression settings for sync performance
• pduid_pdu: Large block size (2KB) for efficient event storage
• eventid_outlierpdu: Optimized for outlier PDU handling

Cache-Only Tables

• servername_destination: FIFO cache for server resolution
• servername_override: FIFO cache for server overrides

Data Types and Sizes

Event IDs

• Full event IDs: 48 bytes (Matrix event ID format)
• Short event IDs: 8 bytes (internal optimization)

Room IDs

• Full room IDs: Variable length Matrix room ID
• Short room IDs: 8 bytes (internal optimization)

PDU Data

• PDU ID: 16 bytes
• PDU content: ~1520 bytes average
• Outlier PDUs: ~1488 bytes average

State Data

• State keys: Up to 1016 bytes
• Short state keys: 8 bytes
• State hashes: 8 bytes (shortened)

This technical reference shows how Continuwuity optimizes storage for different types of Matrix data, using appropriate RocksDB configurations for each access pattern.

Database Architecture

Column Families (Maps)

Room Management

Room Aliases

• alias_roomid: Maps room alias to room ID
• alias_userid: Maps room alias to user ID (for alias management)
• aliasid_alias: Maps alias ID to actual alias string

Room Metadata

• roomid_shortroomid: Maps room ID to short room ID (8-byte identifier)
• roomid_shortstatehash: Maps room ID to current state hash
• roomid_pduleaves: Tracks PDU leaves for each room
• roomid_invitedcount: Count of invited users per room
• roomid_joinedcount: Count of joined users per room
• roomid_inviteviaservers: Via servers for room invites
• publicroomids: Set of public room IDs
• bannedroomids: Set of banned room IDs
• disabledroomids: Set of disabled room IDs

Room State

• shortstatehash_statediff: State differences between state hashes
• statehash_shortstatehash: Maps full state hash to short state hash (8-byte)
• statekey_shortstatekey: Maps state key to short state key (8-byte)
• shortstatekey_statekey: Reverse mapping from short state key to full state key
• roomsynctoken_shortstatehash: Maps room sync tokens to state hashes

Events and Timeline

Event Storage

• eventid_pduid: Maps event ID to PDU ID (16-byte identifier)
• eventid_shorteventid: Maps event ID to short event ID (8-byte)
• eventid_outlierpdu: Stores outlier PDUs (events not yet in timeline)
• pduid_pdu: Main PDU storage (PDU ID to PDU data)
• shorteventid_eventid: Reverse mapping from short event ID to full event ID
• shorteventid_authchain: Authorization chains for events
• shorteventid_shortstatehash: Maps events to their state hashes

Event Relationships

• tofrom_relation: Event relations (replies, edits, reactions)
• threadid_userids: Thread participants tracking
• referencedevents: Referenced events tracking
• softfailedeventids: Events that soft-failed state resolution

User Management

User Identity

• userid_displayname: User display names
• userid_avatarurl: User avatar URLs
• userid_blurhash: Avatar blurhash values
• userid_password: Password hashes
• useridprofilekey_value: Custom profile fields

User Devices and Sessions

• userdeviceid_metadata: Device metadata (name, type, etc.)
• userdeviceid_token: Device access tokens
• token_userdeviceid: Reverse token to device mapping
• userdevicesessionid_uiaainfo: User-Interactive Auth session data
• userdevicetxnid_response: Transaction ID to response caching

User Preferences

• userfilterid_filter: Sync filter definitions
• lazyloadedids: Lazy-loaded member event tracking

Cryptography and Security

Device Keys

• keyid_key: Cryptographic keys storage
• userid_devicelistversion: Device list versions for users
• userid_lastonetimekeyupdate: Last one-time key update timestamps
• onetimekeyid_onetimekeys: One-time keys storage

Cross-Signing

• userid_masterkeyid: Master signing keys
• userid_selfsigningkeyid: Self-signing keys
• userid_usersigningkeyid: User-signing keys
• keychangeid_userid: Key change notifications

Key Backups

• backupid_algorithm: Backup algorithm information
• backupid_etag: Backup ETags for versioning
• backupkeyid_backup: Backed up keys

Room Membership

Membership States

• roomuserid_joined: Current joined room members
• roomuserid_invitecount: Invite counts per room-user
• roomuserid_leftcount: Leave counts per room-user
• roomuserid_knockedcount: Knock counts per room-user
• roomuseroncejoinedids: Users who have ever joined rooms

Membership Events

• userroomid_joined: User's joined rooms
• userroomid_invitestate: Invite state events
• userroomid_leftstate: Leave state events
• userroomid_knockedstate: Knock state events

Push Notifications and Read Receipts

Push Infrastructure

• senderkey_pusher: Push notification endpoints
• pushkey_deviceid: Push key to device mappings

Read Tracking

• readreceiptid_readreceipt: Read receipt storage
• roomuserid_privateread: Private read markers
• roomuserid_lastprivatereadupdate: Last private read updates
• userroomid_highlightcount: Highlight/mention counts
• userroomid_notificationcount: Notification counts per room

Media and Content

Media Storage

• mediaid_file: Media file metadata
• mediaid_user: Media uploader tracking
• url_previews: URL preview cache

Federation and Server-to-Server

Server Management

• server_signingkeys: Server signing keys
• servername_destination: Server destination resolution
• servername_educount: Ephemeral Data Unit counters
• servername_override: Server name overrides for federation
• servernameevent_data: Server event data
• roomserverids: Servers participating in rooms
• serverroomids: Rooms per server
• servercurrentevent_data: Current server event state

Application Services

• id_appserviceregistrations: Application service registrations

Account Data and Presence

Account Data

• roomuserdataid_accountdata: Room-specific account data
• roomusertype_roomuserdataid: Account data type mappings

Presence

• presenceid_presence: User presence information
• userid_presenceid: User to presence ID mapping

To-Device Messages

• todeviceid_events: Direct device-to-device messages

Authentication Tokens

• openidtoken_expiresatuserid: OpenID Connect tokens
• logintoken_expiresatuserid: Login tokens
• tokenids: Token ID management

Global Configuration

• global: Global server settings and state

Key Design Patterns

Short Identifiers

Many tables use "short" versions of identifiers (8-byte integers) to reduce storage overhead:

• shortroomid for room IDs
• shorteventid for event IDs
• shortstatekey for state keys
• shortstatehash for state hashes

Composite Keys

Key naming follows a pattern of {primary}_{secondary} to create efficient lookups:

• roomuserid_* for room-user relationships
• userroomid_* for user-room relationships
• eventid_* for event-related data

Performance Optimizations

• Cache sharing: Related tables share cache pools (e.g., eventid_outlierpdu and pduid_pdu)
• Access patterns: Tables are optimized for their specific usage (RANDOM vs SEQUENTIAL)
• Compression: Different compression levels based on data characteristics
• Block sizes: Tuned based on expected key/value sizes

Storage Efficiency

The schema is designed for efficiency in a Matrix homeserver context:

• Large event data uses sequential storage patterns
• Lookup tables use random access patterns
• Small metadata uses compressed storage
• Caching is strategically shared between related data

This design allows Continuwuity to efficiently handle the complex relationships and high-volume data typical in Matrix federation while maintaining good performance characteristics for both reads and writes.

Column Relationships and Data Flow

Core Event Storage Chain

The heart of the Matrix homeserver is event storage, which uses several interconnected tables:

• eventid_shorteventid ↔ shorteventid_eventid: Bidirectional mapping for event ID compression (48 bytes → 8 bytes)
• eventid_pduid: Maps Matrix event IDs to internal PDU IDs (16 bytes)
• pduid_pdu: Main event storage using PDU IDs as keys (shares cache with eventid_outlierpdu)
• eventid_outlierpdu: Stores events not yet integrated into the timeline
• shorteventid_authchain: Authorization chains using compressed event IDs
• shorteventid_shortstatehash: Links events to room state snapshots

Room State Management

Room state is tracked through multiple interconnected tables:

• statekey_shortstatekey ↔ shortstatekey_statekey: Bidirectional state key compression
• statehash_shortstatehash: Compresses state hashes from full size to 8 bytes
• shortstatehash_statediff: Stores incremental state changes
• roomid_shortstatehash: Current state hash for each room
• roomsynctoken_shortstatehash: Maps sync tokens to state for efficient delta sync

User Identity and Authentication

User management involves several related tables:

• userid_password → authentication base
• token_userdeviceid ↔ userdeviceid_token: Bidirectional token↔device mapping
• userdeviceid_metadata: Device information storage
• userid_displayname, userid_avatarurl, userid_blurhash: Profile data
• openidtoken_expiresatuserid, logintoken_expiresatuserid: Token management

Room Membership Tracking

Membership uses bidirectional indexes for efficient queries:

• roomuserid_joined ↔ userroomid_joined: Current membership from both perspectives
• roomuserid_invitecount ↔ userroomid_invitestate: Invitation tracking
• roomuserid_leftcount ↔ userroomid_leftstate: Leave event tracking
• roomid_joinedcount, roomid_invitedcount: Aggregate room statistics
• roomuseroncejoinedids: Historical membership tracking

Cryptography and Security Chain

End-to-end encryption involves coordinated key management:

• userid_devicelistversion: Tracks when device lists change
• keyid_key: Stores actual cryptographic keys
• userid_masterkeyid, userid_selfsigningkeyid, userid_usersigningkeyid: Cross-signing keys
• onetimekeyid_onetimekeys → userid_lastonetimekeyupdate: One-time key lifecycle
• keychangeid_userid: Key change notifications
• backupid_algorithm, backupid_etag → backupkeyid_backup: Key backup system

Federation and Server Communication

Server-to-server communication requires coordinated tracking:

• roomserverids ↔ serverroomids: Bidirectional room↔server participation
• servername_destination, servername_override: Server resolution (both cached)
• server_signingkeys: Federation authentication
• servername_educount: Ephemeral data unit tracking
• servernameevent_data, servercurrentevent_data: Server event state

Read Tracking and Notifications

Message read tracking involves multiple coordinated updates:

• readreceiptid_readreceipt: Public read receipts
• roomuserid_privateread, roomuserid_lastprivatereadupdate: Private read markers
• userroomid_highlightcount, userroomid_notificationcount: Per-room notification counts
• senderkey_pusher ↔ pushkey_deviceid: Push notification routing

Account Data and Preferences

User preferences and account data use a two-level structure:

• roomusertype_roomuserdataid → roomuserdataid_accountdata: Type index points to actual data
• userid_presenceid → presenceid_presence: Presence data separation
• userfilterid_filter: Sync filter definitions
• lazyloadedids: Lazy loading state tracking

This interconnected design allows Continuwuity to efficiently handle Matrix protocol operations while maintaining data consistency and enabling fast lookups from multiple perspectives.