Notification Architecture Before and After

Notification Architecture: Before and After

This document explains, in plain language, how outage notifications worked before and how they work now after the unified notification pipeline.

It is written for product, operations, support, and non-developer readers. It avoids code details and focuses on behavior, data flow, and what changed.

Short Version

Before, each outage monitor sent notifications mostly on its own. Power outage notifications had their own sending path. Scheduled outage notifications had another sending path. Each path decided what to send, how to format it, where to send it, and how to avoid duplicates.

Now, outage monitors still watch the BENECO feeds and save outage data, but they no longer own the whole notification process. They report “something happened” to one shared notification pipeline. The pipeline then decides who should receive it, prepares the message, sends it, and records that it was sent.

The outage history tables still stay in place. They are still important for analytics and future reports.

Main Difference

Area	Previous architecture	New architecture
Who watches BENECO feeds?	Each monitor watches its own feed.	Same. Each monitor still watches its own feed.
Who saves outage data?	Each monitor saves its own data.	Same. Feed data is still saved first.
Who sends notifications?	Each monitor sends directly to ntfy.	A shared notification pipeline sends notifications.
Where are live topics configured?	Topic settings lived under each outage monitor.	Topics are stored as system notification subscriptions.
How are duplicates avoided?	Each monitor had its own flags or tracking tables.	The pipeline has a delivery ledger, plus existing outage tracking is kept.
How easy is it to add user filters later?	Harder, because each monitor would need custom logic.	Easier, because filters are part of the shared pipeline model.
Are outage tables removed?	No.	No. They remain for history and analytics.

What Stayed The Same

The system still polls BENECO. It does not receive instant push updates from BENECO.

The app still saves power outages in power_outages.

The app still saves scheduled outages in scheduled_outages.

The global ntfy topics still exist:

beneco-power-outages
beneco-scheduled-outages

The contact form notification flow is still separate. The new pipeline is mainly for recurring feed-based notifications like power outages and scheduled outages.

Previous Architecture

In the previous design, each monitor had a lot of responsibility.

For example, the power outage monitor did all of this:

Check the BENECO power outage feed.
Save new or updated outage data.
Decide whether an outage was new or restored.
Build the notification message.
Choose the ntfy topic.
Send the notification.
Mark the outage as notified so it would not send again.

The scheduled outage monitor followed a similar idea, but with its own rules for scheduled outage updates, digests, and reminders.

Previous Flow

BENECO feed
    |
    v
Power outage monitor or scheduled outage monitor
    |
    +--> Save outage data
    |
    +--> Decide if a notification is needed
    |
    +--> Build message
    |
    +--> Send directly to ntfy
    |
    +--> Mark as sent

This worked for a small first version, but it had a downside: every notification source needed its own version of the same sending pattern.

Problems With The Previous Design

The old design was not wrong. It was just starting to repeat itself.

The repeated pieces were:

choosing who receives the notification
formatting the notification
sending through ntfy
preventing duplicate sends
tracking delivery history
preparing for future user-specific filters

That repetition makes future changes more risky. If we add user filters, email delivery, or a new feed later, we would have to update multiple places and keep their behavior consistent.

New Architecture

The new design separates the work into three simple parts.

Part 1: Feed Monitors

Feed monitors are still responsible for understanding the BENECO feeds.

They answer questions like:

What did BENECO publish?
Is this outage new?
Was power restored?
Was a scheduled outage updated or cancelled?
Should a digest or reminder be created today?

They also save the feed data first, so the database remains the source for history and analytics.

Part 2: Source Rules

Source rules decide what changed in a specific feed.

They answer questions like:

Is this power outage newly reported?
Did this outage just move from ongoing to restored?
Is this scheduled outage new, updated, or cancelled?
Is today the right time for a digest or reminder?

These rules belong to the feed area because each feed has different business meaning.

Part 3: Notification Pipeline

The notification pipeline is responsible for delivery.

It answers questions like:

Which subscription should receive this event?
Does the event match the subscription’s filter?
Has this already been sent?
What should the message look like?
Which delivery provider should be used?
Was delivery successful?

New Flow

BENECO feed
    |
    v
Feed monitor
    |
    +--> Save outage data for history and analytics
    |
    +--> Source rules decide what changed
    |
    +--> App creates a notification event
              |
              v
        Notification pipeline
              |
              +--> Find matching subscriptions
              |
              +--> Check if already delivered
              |
              +--> Build the message
              |
              +--> Send through ntfy
              |
              +--> Record delivery in the ledger

Code Map In Plain English

This section connects the plain-English workflow to the main parts of the code that were touched.

Code area	Plain-English responsibility
`PowerOutageMonitorWorker`	Wakes up on a timer and starts the power outage check.
`ScheduledOutageMonitorWorker`	Wakes up on a timer and starts the scheduled outage check.
`SyncPowerOutagesAndPublishEvents`	Polls the power outage feed, saves outage rows, asks the power outage rules what changed, creates notification events, and hands them to the pipeline.
`SyncScheduledOutagesAndPublishEvents`	Polls the scheduled outage feed, saves schedule rows, asks the scheduled outage rules what changed or what digest/reminder is due, creates notification events, and hands them to the pipeline.
`PowerOutageChangeDetector`	Decides whether a power outage is new, restored, or already restored when first seen.
`ScheduledOutageChangeDetector`	Decides whether scheduled outage rows are new, updated, or cancelled.
`ScheduledOutageDigestPolicy`	Decides whether the weekly or daily scheduled outage summary is due.
`ScheduledOutageReminderPolicy`	Finds scheduled outages that should get a day-before reminder.
`PowerOutageNotificationEvents`	Turns power outage changes into notification event names like `NewOutage`, `Restoration`, and `RestoredAnnouncement`.
`ScheduledOutageNotificationEvents`	Turns scheduled outage changes into notification event names like `NewEntry`, `Updated`, `Cancelled`, `WeeklyDigest`, `DailyUpdate`, and `DayBeforeReminder`.
`ProcessNotificationEvents`	The shared notification pipeline. It matches subscriptions, checks duplicates, renders messages, delivers, and records successful delivery.
`NotificationEventSourceResolver`	Re-reads the latest saved data before sending, so the message uses fresh data.
`PowerOutageNotificationRenderer`	Builds the final power outage message text, title, tags, priority, and sequence id.
`ScheduledOutageNotificationRenderer`	Builds the final scheduled outage message text, title, tags, priority, and sequence id.
`DatabaseNotificationSubscriptionStore`	Loads active notification subscriptions from the database.
`DatabaseNotificationDeliveryLedgerStore`	Checks and records which subscription already received which event.
`NtfyNotificationDeliveryRouter`	Converts the rendered notification into an ntfy send request.
`NtfyNotificationService`	Sends the HTTP request to the ntfy server.
`NotificationSystemSubscriptionSeeder`	Creates or updates the built-in system subscriptions on startup.
`NotificationSystemSubscriptionHostedService`	Runs the system subscription sync when the app starts.
`NotificationPipelineOptions`	Holds shared notification pipeline settings like enabled state, daily cap, and test topic.
`NotificationSystemSubscriptionsOptions`	Holds configured system topic names for power outages and scheduled outages.
`NotificationPipelineExtensions`	Wires the notification pipeline into dependency injection: options, stores, topic policy, and delivery router.
`DependencyInjection` in Application	Registers the use cases, renderers, pipeline processor, and source resolver.
`DependencyInjection` in Infrastructure	Registers persistence, feed clients, notification providers, and the notification pipeline.
`Program.cs` in the API	Registers startup services and background workers.
`Obaki.ConfigGenerator`	Reads option classes and regenerates `.env.template` and `manifest.json`.
Core tests	Prove source change rules make the right decisions without sending notifications.
Application tests	Prove the monitor and notification use cases still send the expected messages.
Infrastructure tests	Prove system subscription seeding, topic policy behavior, and ledger behavior.

Definitive End-to-End Workflow

This is the full mental model from feed to phone notification.

1. Worker wakes up
2. Use case polls the BENECO feed
3. Feed data is saved to the outage history table
4. Source rules detect what changed
5. Use case maps those changes into notification events
6. Shared pipeline loads matching subscriptions
7. Shared pipeline applies filters
8. Shared pipeline checks cooldown and daily cap
9. Shared pipeline checks the delivery ledger
10. Shared pipeline re-reads the latest saved row
11. Renderer builds title, message, tags, priority, and sequence id
12. Delivery router sends through ntfy
13. Delivery ledger records successful delivery
14. System-specific legacy markers are updated where still needed for analytics

Each step has one main job. That is the point of the new architecture.

Wiring And Configuration Flow

The runtime wiring follows this path:

Program.cs
    |
    +--> AddObakiApplication
    |       |
    |       +--> registers pipeline use cases
    |       +--> registers power outage renderer
    |       +--> registers scheduled outage renderer
    |       +--> registers event source resolver
    |
    +--> AddObakiInfrastructure
    |       |
    |       +--> registers database stores
    |       +--> registers BENECO feed clients
    |       +--> registers ntfy provider
    |       +--> AddNotificationPipeline
    |               |
    |               +--> binds NotificationPipeline options
    |               +--> binds system subscription options
    |               +--> registers subscription store
    |               +--> registers delivery ledger store
    |               +--> registers ntfy delivery router
    |               +--> builds allowed topic policy
    |
    +--> hosted services
            |
            +--> database migration service
            +--> system subscription sync service
            +--> contact outbox worker
            +--> power outage monitor worker
            +--> scheduled outage monitor worker

Configuration follows this path:

Option classes in code
    |
    +--> Obaki.ConfigGenerator
            |
            +--> manifest.json
            +--> .env.template
                    |
                    +--> production environment variables
                            |
                            +--> app startup options

That means the option classes are the source of truth. The generated config files should follow the code, not the other way around.

Workflow Example: App Startup

Before any outage notification can be sent, the app prepares the built-in notification subscriptions.

App starts
    |
    +--> Database migrations may run, depending on environment settings
    |
    +--> NotificationSystemSubscriptionHostedService runs
            |
            +--> NotificationSystemSubscriptionSeeder reads config
            |
            +--> Ensures power outage system subscription exists
            |
            +--> Ensures scheduled outage system subscription exists

The system subscription rows say:

beneco.power-outages -> send to beneco-power-outages
beneco.scheduled-outages -> send to beneco-scheduled-outages

If the topic changes in configuration, startup sync updates the destination in the database. The pipeline then reads the current subscription from the database.

Workflow Example: New Power Outage

Scenario: BENECO publishes a new ongoing power outage with source id 66510.

PowerOutageMonitorWorker
    |
    +--> SyncPowerOutagesAndPublishEvents
            |
            +--> Poll BENECO power outage feed
            |
            +--> See source id 66510
            |
            +--> Save row in power_outages
            |
            +--> PowerOutageNotificationEvents creates NewOutage event
            |
            +--> ProcessNotificationEvents receives the event
                    |
                    +--> Load active subscriptions for beneco.power-outages
                    |
                    +--> Empty system filter matches the event
                    |
                    +--> Ledger says this subscription has not received NewOutage for 66510
                    |
                    +--> NotificationEventSourceResolver loads latest power_outages row
                    |
                    +--> PowerOutageNotificationRenderer builds "Power outage reported"
                    |
                    +--> NtfyNotificationDeliveryRouter sends to beneco-power-outages
                    |
                    +--> DatabaseNotificationDeliveryLedgerStore records delivery
            |
            +--> Mark OutageNotifiedAt for analytics and transition safety

Result: people subscribed to beneco-power-outages receive one “Power outage reported” alert.

Workflow Example: Power Restored

Scenario: BENECO updates the same outage with a restored time.

PowerOutageMonitorWorker
    |
    +--> SyncPowerOutagesAndPublishEvents
            |
            +--> Poll feed
            |
            +--> See source id 66510 now has timerestored
            |
            +--> Update row in power_outages
            |
            +--> PowerOutageNotificationEvents creates Restoration event
            |
            +--> ProcessNotificationEvents handles the event
                    |
                    +--> Find power outage system subscription
                    |
                    +--> Check ledger for subscription + 66510 + Restoration
                    |
                    +--> Render "Power restored"
                    |
                    +--> Send to beneco-power-outages
                    |
                    +--> Record ledger row
            |
            +--> Mark RestoredNotifiedAt

Result: people receive one “Power restored” alert for that feed item.

Workflow Example: Already Restored When First Seen

Sometimes BENECO may publish an outage after it is already restored.

Power outage feed item first appears
    |
    +--> It already has timerestored
    |
    +--> System saves it as first-seen-restored
    |
    +--> Event becomes RestoredAnnouncement instead of NewOutage
    |
    +--> Pipeline sends "Power restored (just announced)"
    |
    +--> RestoredNotifiedAt is marked

This avoids sending a confusing “new outage” alert for something that is already restored.

Workflow Example: Manual Power Outage Announcement

Manual announcements are admin-triggered messages, but they still use the same pipeline.

Admin calls announcement endpoint
    |
    +--> SendPowerOutageAnnouncement validates the message
    |
    +--> Creates ManualAnnouncement event
    |
    +--> ProcessNotificationEvents loads power outage system subscription
    |
    +--> PowerOutageNotificationRenderer builds announcement message
    |
    +--> Ntfy delivery sends to beneco-power-outages
    |
    +--> Ledger records the announcement using a unique source key

The key difference from feed-based events is that the event comes from an admin request, not from BENECO.

Workflow Example: Power Outage Test Notifications

The test endpoint checks notification formatting and delivery without affecting the live system subscription state.

POST /api/power-outages/test-notifications
    |
    +--> SendPowerOutageTestNotifications polls the real feed
    |
    +--> Picks sample ongoing and restored rows
    |
    +--> Creates test events
    |
    +--> Creates an ephemeral test subscription id for this run
    |
    +--> Pipeline renders and sends to NotificationPipeline:TestNtfyTopic
    |
    +--> Ledger records under the ephemeral test id

This matters because test sends should not consume the live power outage subscription’s daily cap or update its last-sent time.

Workflow Example: Scheduled Outage Added, Updated, Or Cancelled

Scenario: BENECO changes the scheduled outage list.

ScheduledOutageMonitorWorker
    |
    +--> SyncScheduledOutagesAndPublishEvents
            |
            +--> Poll BENECO scheduled outage feed
            |
            +--> Compare current feed rows with known saved rows
            |
            +--> Save current rows in scheduled_outages
            |
            +--> ScheduledOutageNotificationEvents creates item events
                    |
                    +--> NewEntry for new rows
                    +--> Updated for changed rows
                    +--> Cancelled for newly cancelled rows
            |
            +--> ProcessNotificationEvents handles those events
                    |
                    +--> Find scheduled outage system subscription
                    |
                    +--> Check filter and ledger
                    |
                    +--> ScheduledOutageNotificationRenderer builds message
                    |
                    +--> Send to beneco-scheduled-outages
                    |
                    +--> Record delivery

Result: scheduled outage changes go through the same delivery path as power outage changes.

Workflow Example: Scheduled Weekly Digest

The weekly digest is slightly different because it is time-based.

ScheduledOutageMonitorWorker
    |
    +--> SyncScheduledOutagesAndPublishEvents
            |
            +--> Convert current time to configured local timezone
            |
            +--> If it is the configured digest hour and weekly day:
                    |
                    +--> Check scheduled_outage_digest_runs
                    |
                    +--> If weekly digest has not been sent today:
                            |
                            +--> Create WeeklyDigest event
                            |
                            +--> Pipeline sends digest to scheduled outage subscription
                            |
                            +--> Mark digest as sent

The digest run table decides whether the digest should exist. The delivery ledger decides whether each subscription already received that digest event.

Workflow Example: Scheduled Daily Update

The daily update summarizes changes for the day.

Scheduled outage feed changed today
    |
    +--> New, updated, or cancelled rows exist
    |
    +--> It is the configured digest hour
    |
    +--> Weekly digest did not already take priority
    |
    +--> Check scheduled_outage_digest_runs for DailyUpdate
    |
    +--> Create DailyUpdate event
    |
    +--> Pipeline sends it
    |
    +--> Mark daily update as sent

This prevents repeated daily update notifications during later polls in the same day.

Workflow Example: Day-Before Reminder

The reminder tells people about scheduled outages happening tomorrow.

Scheduled outage monitor runs at the configured hour
    |
    +--> Look for scheduled outages dated tomorrow
    |
    +--> Ignore cancelled rows
    |
    +--> Check scheduled_outage_reminder_runs
    |
    +--> If no reminder was sent for tomorrow:
            |
            +--> Create DayBeforeReminder event
            |
            +--> Pipeline sends reminder
            |
            +--> Mark reminder as sent

The reminder run table prevents another reminder for the same outage date.

How Dedup Works

Dedup means “do not send the same alert again and again.”

The system uses more than one layer because each layer protects a different part of the process.

Layer 1: Feed-state checks

The monitor first looks at the saved outage data.

For power outages:

OutageNotifiedAt is null -> a new outage alert may be needed
OutageNotifiedAt has a value -> do not create another NewOutage event

RestoredNotifiedAt is null -> a restoration alert may be needed
RestoredNotifiedAt has a value -> do not create another Restoration event

For scheduled outages:

scheduled_outage_digest_runs -> prevents repeated digests
scheduled_outage_reminder_runs -> prevents repeated day-before reminders

This layer decides whether an event should be created at all.

Layer 2: Delivery ledger

The pipeline checks the delivery ledger before sending.

The important key is:

subscription id + feed id + source key + event type

Example:

Subscription: -1
Feed: beneco.power-outages
Source key: 66510
Event type: Restoration

If that exact combination already exists in the ledger, the pipeline skips delivery.

This matters because future user subscriptions may receive the same event differently. One user may receive it, another user may not. The ledger tracks delivery per subscription.

Layer 3: ntfy sequence id

Rendered ntfy messages also include a stable sequence id when useful.

This helps ntfy clients collapse duplicate-looking messages, but the database ledger is the main source of truth.

What happens if delivery fails?

If ntfy delivery fails:

No ledger row is written
No successful delivery is reported
The event can be retried later if the monitor still emits it

That is intentional. A failed send should not be marked as successfully delivered.

What happens if two app instances race?

If two runs try to write the same ledger row at the same time, the database unique rule allows only one row.

The code treats that specific duplicate-ledger case as safe. Other database errors are not hidden.

How Filtering Works

System subscriptions currently use an empty filter.

Empty filter = match all events for that feed

Later, user filters can narrow delivery.

Example future filter:

Feed: beneco.power-outages
Event types: NewOutage, Restoration
Location keywords: Bakakeng
Destination: generated user ntfy topic

The pipeline would handle it like this:

Power outage event arrives
    |
    +--> Load all active subscriptions for beneco.power-outages
    |
    +--> System subscription matches everything
    |
    +--> User subscription matches only if fields contain Bakakeng
    |
    +--> Each matching subscription gets its own ledger check
    |
    +--> Matching subscriptions receive the alert

The monitor does not need to know about user filters. That is one of the main benefits of the new design.

How Message Formatting Works

The monitor does not build the final ntfy message anymore.

Instead:

Notification event
    |
    +--> NotificationRendererRegistry picks a renderer
    |
    +--> PowerOutageNotificationRenderer or ScheduledOutageNotificationRenderer builds:
            |
            +--> title
            +--> body
            +--> tags
            +--> priority
            +--> sequence id

This keeps message wording in one place per feed.

Examples:

Event type	Renderer output
`NewOutage`	`Power outage reported`
`Restoration`	`Power restored`
`RestoredAnnouncement`	`Power restored (just announced)`
`ManualAnnouncement`	Custom admin announcement title and message
`NewEntry`	`Scheduled outage added`
`Updated`	`Scheduled outage updated`
`Cancelled`	`Scheduled outage cancelled`
`WeeklyDigest`	Weekly scheduled outage summary
`DailyUpdate`	Daily scheduled outage update
`DayBeforeReminder`	Tomorrow outage reminder

How Delivery Works

The delivery path is shared.

RenderedNotification
    |
    +--> NtfyNotificationDeliveryRouter
            |
            +--> Check the topic is allowed
            |
            +--> Convert rendered message to ntfy request
            |
            +--> NtfyNotificationService
                    |
                    +--> Send HTTP request to ntfy server
                    |
                    +--> Return success or failure

The topic allowlist is important. It helps prevent the app from sending to random ntfy topics.

System topics are explicitly allowed. Future user topics must use the configured safe prefix.

Key Ideas

Event

An event means “something happened.”

Examples:

A new power outage was reported.
Power was restored.
A scheduled outage was added.
A scheduled outage was cancelled.
A weekly scheduled outage digest is ready.

Subscription

A subscription means “send matching events to this destination.”

There are system subscriptions for the global topics:

Power outage events -> beneco-power-outages
Scheduled outage events -> beneco-scheduled-outages

Later, user subscriptions can be added for more specific alerts, such as “only notify me for this feeder or area.”

Filter

A filter decides whether a subscription cares about an event.

The current system subscriptions use an empty filter, which means “send everything from this feed.”

Future user filters can match things like:

feeder
area keywords
event type
general keywords

Delivery Ledger

The delivery ledger records that a notification was sent.

It helps prevent repeated alerts for the same subscription, feed item, and event type.

For example:

Subscription: beneco-power-outages
Feed item: outage 66510
Event type: Power restored
Result: sent already

If the app sees the same restored outage again, the ledger helps the pipeline avoid sending the same notification again.

Why Outage Tables Still Matter

The notification pipeline is not replacing outage history.

The outage tables answer business and analytics questions:

Question	Data source
How many outages happened this month?	`power_outages`
Which feeders have frequent outages?	`power_outages`
What scheduled outages were published?	`scheduled_outages`
When was an outage first seen?	outage tables
When was a notification sent?	notification delivery ledger

This separation is intentional.

Outage tables store what happened.

Notification tables store who was notified and when.

Why This Is Better

Less duplicate notification logic

The send-and-track behavior is now shared instead of copied into each monitor.

Safer duplicate prevention

The delivery ledger gives the app one shared place to remember what was already sent.

Easier future user filters

The system now has a natural place for user-specific rules, such as area or feeder matching.

Easier new feeds later

If another feed is added later, it can use the same delivery pipeline after it saves data and creates events.

Cleaner responsibility split

Feed monitors focus on feed data.

The pipeline focuses on notification delivery.

What This Does Not Add Yet

This change does not automatically add a full user subscription UI.

It does not make BENECO updates instant. The app still polls.

It does not remove power_outages or scheduled_outages.

It does not move contact form notifications into the pipeline yet.

It does not require email delivery for outage alerts yet.

Operational Notes

The notification pipeline adds two important notification tables:

notification_subscriptions
notification_delivery_ledger

Before deploying this change to production, the database migration that creates those tables must be applied.

System subscriptions are synchronized on startup. That means the app expects those notification tables to exist before outage monitors run.

Simple Mental Model

Use this sentence:

Monitors discover what happened; the pipeline decides who gets told.

That is the core difference between the previous architecture and the new one.