Virtufin WorkManager

A production-ready, event-driven worker management system built on Dapr that enables you to deploy and manage workers written in different programming languages.

Overview

WorkManager is an event-driven worker system that: - Subscribes to Dapr pub/sub topics - Processes incoming CloudEvents using pluggable execution engines - Supports multiple programming languages (Python, C#, and pre-compiled DLLs) - Provides distributed locking for coordinated multi-worker execution - Persists worker configurations for automatic recovery after restarts

Key Features

• Multi-language Support: Workers can be written in Python, C# source (compiled at runtime), or pre-compiled .NET DLLs
• Dapr Integration: Leverages Dapr for pub/sub messaging, state management, and distributed locking
• Hot Code Reloading: Update worker code without restarting the worker
• Automatic Recovery: Workers are automatically recovered after service restarts
• Group Coordination: Workers can coordinate using distributed locks to ensure mutually exclusive execution
• Dual API Support: REST (HTTP/1.1 and HTTP/2) and gRPC endpoints
• OpenTelemetry Ready: Structured logging with trace context

Architecture

┌─────────────────────────────────────────────────────────────────────────┐
│                           WorkManager Service                           │
│                                                                         │
│  ┌─────────────┐    ┌──────────────┐    ┌─────────────────────────┐  │
│  │  WorkManager │    │ EngineRegistry│    │     WorkerRegistry      │  │
│  │  Component   │    │              │    │                         │  │
│  │              │    │  text/x-python│    │  ┌─────┐ ┌─────┐        │  │
│  │  - Create    │    │  text/x-csharp│    │  │ W1  │ │ W2  │  ...  │  │
│  │  - Start     │◄──►│  ...         │◄──►│  └─────┘ └─────┘        │  │
│  │  - Stop      │    │              │    │                         │  │
│  │  - Delete    │    └──────────────┘    └─────────────────────────┘  │
│  │  - Recover   │                                                        │
│  └──────┬───────┘                                                        │
│         │                                                                  │
│  ┌──────▼───────┐    ┌─────────────────────────────────────────────┐   │
│  │ Dapr Client  │    │              Dapr Subscriptions              │   │
│  │              │◄──►│                                             │   │
│  │  - State     │    │  topic: "time-worker-topic"                 │   │
│  │  - Pub/Sub   │    │    └──► Worker #1 ──► PythonEngine           │   │
│  │  - Pub/Sub   │    │    └──► Worker #2 ──► CSharpSourceEngine    │   │
│  └──────────────┘    └─────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────────────┘

Quick Start

# Build the project
dotnet build

# Run with Dapr
dapr run --app-id workmanager --app-port 5001 -- dotnet run --project src/Virtufin.WorkManager

# Create a Python worker via REST API
curl -X POST http://localhost:5001/v1/workers \
  -H "Content-Type: application/json" \
  -d '{
    "codeSource": {"Content": "ZGVmIFByb2Nlc3MoZXZlbnQpOgogICAgcmV0dXJuIHsiVHlwZSI6ICJjb20uZXhhbXBsZS5vdXRwdXQiLCAiU291cmNlIjogInB5dGhvbi13b3JrZXIiLCAiRGF0YSI6IGV2ZW50WydkYXRhJ119"}
  }'

Documentation

• Introduction - What is WorkManager and why use it?
• Getting Started - Installation and first worker
• Architecture - System design and components
• Engine System - Language-specific execution engines
• Worker Lifecycle - Managing workers from creation to deletion
• API Reference - Complete API documentation
• Python Client - Python client library for WorkManager

Supported Languages

Engine	MIME Type	Execution
PythonEngine	text/x-python	Via python3 subprocess
CSharpSourceEngine	text/x-csharp	Roslyn compilation at runtime
CSharpDllEngine	text/x-dll	Pre-compiled .NET assembly

Project Structure

.
├── src/
│   ├── Virtufin.WorkManager/           # Main application
│   ├── Virtufin.WorkManager.Engines/  # Engine interfaces
│   ├── Virtufin.WorkManager.Engine.Python/
│   ├── Virtufin.WorkManager.Engine.CSharpSource/
│   └── Virtufin.WorkManager.Engine.CSharpDll/
├── docs/                              # Documentation
├── examples/                          # Usage examples
└── tests/                             # Test suite

License

Proprietary - Virtufin