Audit Logging (Technical)

Audit logging is a first-class subsystem in Eneo for security, compliance, and traceability. It records actions across the platform, stores them in PostgreSQL, and exposes controlled access for admins.

For admin-facing guidance, see the Audit Logging Guide.

High-level architecture

Audit events are created through the AuditService application layer:

1. API routes/services call log_async (preferred) or log (synchronous).
2. log_async enqueues a background job in ARQ (Redis-backed).
3. The worker persists the audit log in PostgreSQL.
4. Retention is enforced by a scheduled purge job.
5. Exports are generated via the AuditExportService (sync or async job).

Audit logging is gated by:

• The audit_logging_enabled feature flag (a kill switch, evaluated per tenant).
• Per-tenant audit configuration (category-level and action-level overrides).

Storage model

Audit logs are stored in PostgreSQL (table audit_logs) with structured JSON metadata.

Key fields include:

• action, entity_type, entity_id
• actor_id (nullable for system actions or deleted users)
• description
• metadata (JSONB)
• outcome, timestamp

Metadata follows a consistent schema via AuditMetadata to prevent drift. It stores actor and target snapshots so log entries remain meaningful even if users or entities are renamed or deleted.

Configuration model

Configuration is layered to allow broad and precise control:

• Global toggle: enable/disable all audit logging for the tenant.
• Category toggles: enable/disable whole groups (admin actions, user actions, security events, file operations, integration events, system actions, audit access).
• Action overrides: enable/disable specific actions within a category.

Changes apply immediately for new audit events; historical logs are not modified.

Access model and justification

Viewing audit logs requires an access justification session, created with POST /api/v1/audit/access-session. The access event itself is logged, including:

• reason category
• justification text
• access time and actor

This provides traceability for who accessed audit logs and why.

The justification is stored server-side (Redis) and referenced by an HTTP-only cookie; it is never carried in URLs. A session lasts one hour, and each user can create at most five sessions per hour (429 when exceeded).

Query and search behavior

Audit queries support filters for:

• actor (actor_id)
• actions (single or multi-select)
• date range

Free-text search matches the log description field. If you want reliable search for a name or identifier, include it in description or metadata.

For data subject access requests, GET /api/v1/audit/logs/user/{user_id} returns every log where the user is the actor or the target (GDPR Article 15). The same user_id filter is accepted by the synchronous export.

Retention

Audit log retention

Audit log retention is tenant-specific. A daily purge job hard-deletes logs older than the configured retention period. This is independent from conversation retention.

The policy is read and updated with GET/PUT /api/v1/audit/retention-policy. Retention is expressed in days: minimum 1, maximum 2555 (about seven years), default 365.

Conversation retention hierarchy

Conversation data (questions and app runs) follows a hierarchical retention policy:

1. Assistant or App data_retention_days (if set)
2. Space data_retention_days
3. null means keep forever

When the purge job deletes a question or app run, it also deletes the files attached to it, such as uploaded documents, audio recordings and their transcriptions, once no other chat, assistant, app or app run uses them. A file that is still used elsewhere is kept until its last use is removed. The stored bytes are then removed by the object content lifecycle; see Files nothing uses anymore.

There is no tenant-level conversation retention setting: it was removed from the retention-policy API (PUT /api/v1/audit/retention-policy accepts only retention_days) to prevent accidental data loss. The purge job still honours the legacy conversation_retention_enabled/conversation_retention_days columns on the tenant’s audit retention policy row if they were set directly in the database, but no API or UI writes them.

In shared and organization spaces, changing or clearing an assistant/app override requires the same space administration permission as changing the space policy. Editors can continue editing other settings but cannot change retention. Personal-space owners retain control of their own assistants and apps.

The existing valid range is 1–2555 days. The space value is a default, not a maximum: administrators can choose a shorter or longer assistant/app override within that range. Setting the override to null restores inheritance; omitting it from an update leaves it unchanged. If neither the assistant/app nor its space has a policy, data is kept indefinitely (unless a legacy tenant policy applies).

Export formats

Synchronous export

Endpoint: GET /api/v1/audit/logs/export

• format=csv (default)
• format=json (JSON Lines/NDJSON)

Synchronous exports are best for small to medium exports. They are capped at 50,000 records by default; max_records raises the cap to at most 100,000. When the cap is hit, the response carries the header X-Records-Truncated: true.

Asynchronous export

Endpoint: POST /api/v1/audit/logs/export/async

• format=csv or jsonl
• status: GET /api/v1/audit/logs/export/{job_id}/status
• download: GET /api/v1/audit/logs/export/{job_id}/download
• cancel: POST /api/v1/audit/logs/export/{job_id}/cancel

Async exports are streamed and resilient for large datasets. Each tenant can have at most two async exports running at once. Export files are retained for 24 hours before cleanup.

UI behavior notes

• Free-text search matches the audit log description. Include names or identifiers in descriptions if you want them to be searchable.
• Access to the audit log view is guarded by an access justification session.

Implementation example

from eneo.audit.domain.audit_metadata_schema import (
    AuditMetadata,
    AuditActor,
    AuditTarget,
    AuditChange,
)
from eneo.audit.domain.action_types import ActionType
from eneo.audit.domain.actor_types import ActorType
from eneo.audit.domain.entity_types import EntityType
 
metadata = AuditMetadata(
    actor=AuditActor(
        id=str(current_user.id),
        name=current_user.username,
        email=current_user.email,
        type=ActorType.USER.value,
    ),
    target=AuditTarget(id=str(space.id), name=space.name),
    changes={
        "name": AuditChange(old=old_name, new=space.name),
    },
).to_dict()
 
await audit_service.log_async(
    tenant_id=current_user.tenant_id,
    actor_id=current_user.id,
    actor_type=ActorType.USER,
    action=ActionType.SPACE_UPDATED,
    entity_type=EntityType.SPACE,
    entity_id=space.id,
    description="Updated space name",
    metadata=metadata,
)

Roadmap

We plan to add optional external audit log storage (e.g., SIEM or analytics sink) to reduce long-term load on the primary database while keeping the same access controls and retention guarantees.