Billing System Architecture

This document provides a comprehensive overview of Tenki Cloud’s billing system, which handles usage-based billing, payment processing, and financial accounting for GitHub Actions runners.

Overview

Tenki Cloud’s billing system is designed to provide accurate, reliable, and scalable billing for compute usage. It integrates multiple systems:

TigerBeetle: High-performance financial database for double-entry bookkeeping
Stripe: Payment processing and invoice generation
Temporal: Workflow orchestration for billing cycles and retry logic
PostgreSQL: Storage for billing metadata and history

System Architecture

┌─────────────────┐     ┌──────────────────┐     ┌─────────────────┐
│  GitHub Actions │────▶│ Runner Service   │────▶│ Usage Events    │
│     Jobs        │     │                  │     │   (Database)    │
└─────────────────┘     └──────────────────┘     └─────────────────┘
                                                           │
                                                           ▼
┌─────────────────┐     ┌──────────────────┐     ┌─────────────────┐
│     Stripe      │◀────│ Billing Service  │◀────│    Temporal     │
│   (Payments)    │     │                  │     │   Workflows     │
└─────────────────┘     └──────────────────┘     └─────────────────┘
                                │
                                ▼
                        ┌─────────────────┐
                        │   TigerBeetle   │
                        │  (Accounting)   │
                        └─────────────────┘

Core Components

1. Data Models

Customer

message Customer {
  string id = 1;
  string identity_id = 2;
  string workspace_id = 3;
  uint64 tb_account_id = 4;        // TigerBeetle account
  string stripe_customer_id = 5;    // Stripe customer
  string default_payment_method = 6;
  bool has_payment_method = 7;
  string payment_method_status = 8;
}

Invoice

message Invoice {
  string id = 1;
  string customer_id = 2;
  string billing_period = 3;        // YYYY-MM format
  string status = 4;                // draft, issued, paid, void
  int64 amount = 5;                 // in cents
  bytes pdf_content = 6;
  string pdf_url = 7;
  string stripe_invoice_id = 8;
  int32 retry_count = 9;
}

UsageEvent

message UsageEvent {
  string id = 1;
  string workspace_id = 2;
  string runner_id = 3;
  google.protobuf.Timestamp started_at = 4;
  google.protobuf.Timestamp finished_at = 5;
  int64 seconds = 6;
  string external_id = 7;           // Idempotency key
  uint64 tb_transfer_id = 8;        // TigerBeetle transfer
}

2. TigerBeetle Accounting

The system uses double-entry bookkeeping with predefined accounts:

Fixed Accounts

1001 - TENKI_RECEIVABLE: Money owed to Tenki
1010 - STRIPE_RECEIVABLE: Money in Stripe
2001 - USER: Customer liability accounts
4001 - REVENUE: Income account
5010 - STRIPE_FEE: Payment processing fees
5020 - MARKETING_EXPENSE: Promotional credits

Transfer Types

1002 - T_StripePayment: Customer payments via Stripe
2001 - T_RunnerCharge: GitHub Actions runner usage charge
2002 - T_RunnerPromoCreditUsage: Promotional credit usage adjustment
2003 - T_UsageReversal: Reversal of negative usage charges
2010 - T_ComputeCharge: Charge for compute resources (future use)
3001 - T_AccountSignup: Initial signup bonus credit
3002 - T_MonthlyFreeCredit: Monthly free credit allowance
3003 - T_PromoCredit: General promotional credit
3004 - T_PromoCreditReversal: Reversal of promotional credits

Example Transactions

Usage Charge (Runner completes job):

Debit:  USER (Customer Account)     $5.00
Credit: REVENUE                     $5.00

Payment Received (Stripe payment):

Debit:  STRIPE_RECEIVABLE          $100.00
Credit: USER (Customer Account)    $100.00

Debit:  STRIPE_FEE                 $2.90
Credit: STRIPE_RECEIVABLE          $2.90

Promotional Credit:

Debit:  MARKETING_EXPENSE          $10.00
Credit: USER (Customer Account)    $10.00

Financial Flow Sequences

The following sequence diagram illustrates the complete financial flows in the Tenki Cloud billing system, showing how money moves between different accounts through various transfer codes:

sequenceDiagram
    participant USER as User/Customer
    participant SIGNUP as Signup Process
    participant GITHUB as GitHub Actions
    participant BILLING as Billing Service
    participant TB as TigerBeetle Ledger
    participant STRIPE as Stripe
    participant CYCLE as Billing Cycle
    participant AUDIT as Audit System

    Note over USER,AUDIT: Tenki Cloud Financial Flow System

    %% Phase 1: Account Creation & Initial Credits
    rect rgb(240, 248, 255)
        Note left of USER: Phase 1: Account Setup & Signup Credits
        USER->>SIGNUP: Create account
        SIGNUP->>BILLING: Create customer account
        BILLING->>TB: Create USER account (ACCOUNT_CODE_USER)

        SIGNUP->>BILLING: Add signup bonus
        BILLING->>TB: Transfer: T_AccountSignup<br/>MARKETING_EXPENSE → USER<br/>($10 signup credit)
    end

    %% Phase 2: Service Usage
    rect rgb(255, 253, 240)
        Note left of GITHUB: Phase 2: Service Usage & Charges
        GITHUB->>BILLING: Job execution event
        BILLING->>BILLING: Calculate usage cost
        BILLING->>TB: Transfer: T_RunnerCharge<br/>USER → REVENUE<br/>(Usage charges)
    end

    %% Phase 3: Payment Processing
    rect rgb(240, 255, 240)
        Note left of CYCLE: Phase 3: Billing Cycle & Payments

        Note over BILLING,TB: Step 1: Promotional credit adjustments
        CYCLE->>BILLING: Start billing cycle
        BILLING->>BILLING: Check promo credit usage for period
        BILLING->>TB: Transfer: T_RunnerPromoCreditUsage<br/>REVENUE → MARKETING_EXPENSE<br/>(Move promo usage from revenue)

        Note over BILLING,STRIPE: Step 2: Invoice generation
        BILLING->>STRIPE: Create Stripe invoice
        STRIPE->>USER: Send payment request
        alt Payment Success
            USER->>STRIPE: Make payment
            STRIPE->>BILLING: Payment webhook

            Note over BILLING,AUDIT: Payment Success Workflow
            BILLING->>TB: Transfer: T_StripePayment<br/>STRIPE_RECEIVABLE → USER<br/>(Payment received)

            BILLING->>TB: Transfer: T_MonthlyFreeCredit<br/>MARKETING_EXPENSE → USER<br/>($10/month free credit reset)

            BILLING->>BILLING: Create payment record in database
            BILLING->>AUDIT: Create billing audit record<br/>(compliance tracking)

        else Payment Failed
            STRIPE->>BILLING: Payment failed webhook
            BILLING->>BILLING: Schedule retry attempts
            BILLING->>BILLING: Start service interruption timer
        end
    end

Complete Transfer Code Reference

The system uses the following transfer codes for different types of financial transactions:

Payment & Withdrawal Operations (1000s)

1001 - T_BankWithdrawal: Cash withdrawal from bank account
1002 - T_StripePayment: Payment received from Stripe (invoice payment)

Service Charges (2000s)

2001 - T_RunnerCharge: Charge for GitHub Actions runner usage
2002 - T_RunnerPromoCreditUsage: Adjustment to move promotional credit usage from revenue to marketing expense
2003 - T_UsageReversal: Reversal of negative usage charges
2010 - T_ComputeCharge: Charge for compute resources (future use)

Credits & Bonuses (3000s)

3001 - T_AccountSignup: Initial signup bonus credit
3002 - T_MonthlyFreeCredit: Monthly free credit allowance (e.g., $10/month)
3003 - T_PromoCredit: General promotional credit (campaigns, support, etc.)
3004 - T_PromoCreditReversal: Reversal of promotional credits (corrections, violations, etc.)

Key Financial Flow Patterns

Customer Onboarding: New users receive signup credits (T_AccountSignup) and monthly free credits (T_MonthlyFreeCredit) from the marketing expense account.
Usage Billing: GitHub Actions runner usage generates charges (T_RunnerCharge) that move money from customer accounts to revenue.
Promotional Credit Accounting: When promotional credits are used for services, the system adjusts by moving the equivalent amount from revenue back to marketing expense (T_RunnerPromoCreditUsage).
Payment Processing: Customer payments through Stripe (T_StripePayment) add funds to customer accounts from the Stripe receivable account.
Administrative Corrections: The system supports reversals for both usage charges (T_UsageReversal) and promotional credits (T_PromoCreditReversal) for corrections and violations.

3. Billing Service

The billing service provides APIs for:

Customer Management: Creating and retrieving billing customers
Balance Operations: Checking workspace credits/debits
Invoice Management: Generating and managing monthly invoices
Usage Tracking: Recording compute usage events
Payment Methods: Managing cards and payment details
Stripe Integration: Setup intents and billing portal

Key service methods:

// Record runner usage
RecordUsage(ctx, workspaceID, runnerID, startTime, endTime)

// Process monthly billing
ProcessInvoiceAndCharge(ctx, workspaceID, billingPeriod)

// Add promotional credits
AddPromotionalCredits(ctx, workspaceID, amount, description)

Workflow Orchestration

1. Billing Cycle Workflow

Runs monthly for each workspace:

flowchart TD
    A[Start Monthly Billing] --> B[Generate Stripe Invoice]
    B --> C[Send Invoice Email]
    C --> D{Amount > 0?}
    D -->|Yes| E[Charge Payment Method]
    D -->|No| F[Complete]
    E --> G{Payment Success?}
    G -->|Yes| H[Payment Succeeded Workflow]
    G -->|No| I[Payment Failed Workflow]
    H --> F
    I --> F

2. Payment Processing Workflows

Payment Succeeded:

Record payment in TigerBeetle
Create payment record in database
Update invoice status

Payment Failed:

Send failure notification
Schedule retry attempts (max 5)
Start service interruption timer

3. Retry Logic

Failed payments are retried with exponential backoff:

Retry 1: 3 days later
Retry 2: 5 days later
Retry 3: 7 days later
Retry 4: 14 days later
Retry 5: 21 days later

If all retries fail by the 9th of the following month, services are suspended on the 10th.

4. Credit Management

Long-running workflow that handles credit operations via signals:

AddPromotionalCredits: Adds credits to workspace
DeductPromotionalCredits: Removes credits
Maintains audit trail in TigerBeetle

Usage Flow

1. Recording Usage

When a GitHub Actions job completes:

sequenceDiagram
    participant Job as GitHub Job
    participant Runner as Runner Service
    participant Billing as Billing Service
    participant TB as TigerBeetle

    Job->>Runner: Job completed
    Runner->>Billing: Record usage event
    Billing->>Billing: Calculate cost
    Billing->>TB: Create usage transfer
    TB->>TB: Debit user account
    TB->>TB: Credit revenue account
    Billing->>Runner: Usage recorded

2. Monthly Billing

At the start of each month:

sequenceDiagram
    participant Temporal
    participant Billing as Billing Service
    participant Stripe
    participant Customer

    Temporal->>Billing: Start billing cycle
    Billing->>Billing: Calculate usage for month
    Billing->>Stripe: Create invoice
    Stripe->>Customer: Send invoice email
    Billing->>Stripe: Charge payment method
    alt Payment successful
        Stripe->>Billing: Payment confirmed
        Billing->>Billing: Record in TigerBeetle
    else Payment failed
        Stripe->>Billing: Payment failed
        Billing->>Temporal: Schedule retry
    end