How to Use Worker: Secure Job Execution Made Simple

A complete guide to running isolated workloads with the Worker platform

What is Worker?

Worker is a lightweight job isolation platform that lets you run commands and scripts in secure, resource-controlled
environments. Think of it as a simpler alternative to Docker for job execution – no containers needed, just a single
binary that provides:

Secure isolation using Linux namespaces
Resource limits for CPU, memory, and I/O
Real-time job monitoring and log streaming
gRPC API with authentication
Simple CLI for easy interaction

Whether you’re building a CI/CD system, running user code safely, or need isolated task execution, Worker provides a
clean, production-ready solution.

Why Use Worker? Before vs After Scenarios

Let’s see how Worker transforms common development and operations challenges with real examples:

🔍 System Call Isolation in Action

❌ Without Worker: Direct Host Execution (Dangerous)

# Running ps aux directly on the host shows ALL system processes
$ ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.0  0.3 167744 13132 ?        Ss   Jun26   0:11 /sbin/init
root           2  0.0  0.0      0     0 ?        S    Jun26   0:00 [kthreadd]
systemd+     564  0.0  0.2  90096  5392 ?        Ss   Jun26   0:00 /lib/systemd/systemd-resolved
messagebus   565  0.0  0.1   8808  3840 ?        Ss   Jun26   0:02 /usr/bin/dbus-daemon --system
worker      1234  0.1  0.5 123456 10240 ?        Sl   Jun26   1:23 /opt/worker/worker
postgres    2345  0.0  1.2 456789 25600 ?        S    Jun26   0:45 postgres: main process
mysql       3456  0.2  2.1 789012 43520 ?        Sl   Jun26   2:10 mysqld --datadir=/var/lib/mysql
apache2     4567  0.0  0.8 234567 16384 ?        S    Jun26   0:30 /usr/sbin/apache2 -DFOREGROUND
...
user        9999  0.0  0.0  10072  1608 pts/2    R+   17:37   0:00 ps aux

Security Issues:

Process can see ALL system processes (including sensitive services)
Has access to process details, PIDs, and resource usage
Can potentially interact with or signal other processes
No isolation from host system resources

✅ With Worker: Isolated Job Execution (Secure)

# Running the same command through Worker shows ONLY the job's process
$ worker-cli run ps aux
Job started:
ID: 120
Command: ps aux
Status: RUNNING
StartTime: 2025-01-15T17:34:33Z

$ worker-cli log 120   
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
0              1  0.0  0.0  10044  1580 ?        R    17:34   0:00 ps aux

Security Benefits:

Job sees ONLY its own process (PID 1 in isolated namespace)
Cannot discover or interact with host processes
Complete process isolation from host system
Protected from interference by other jobs

🗂️ Filesystem Isolation Example

❌ Without Worker (Host Access):

$ ls /
bin   dev  home  lib64  mnt  proc  run   srv  tmp  var
boot  etc  lib   media  opt  root  sbin  sys  usr

$ cat /etc/passwd  # Can read sensitive system files!
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
sys:x:3:3:sys:/dev:/usr/sbin/nologin
# ... all system users exposed

✅ With Worker (Isolated Filesystem):

$ worker-cli run ls /
bin  lib  lib64  proc  sys  tmp  usr  work

$ worker-cli run cat /etc/passwd
cat: /etc/passwd: No such file or directory

$ worker-cli run ls /proc
1  cpuinfo  meminfo  mounts  version  # Only job's view

Key Isolation Features:

Isolated Root Filesystem: Each job gets its own / directory
Limited System Access: Only essential directories are available
Private /tmp: Each job has its own temporary space
Controlled /proc: Process information limited to the job’s namespace
Read-only System Binaries: Access to /bin, /usr/bin, etc. but cannot modify

💾 Resource Protection Example

❌ Without Worker (Unrestricted):

# This could consume all system memory and crash the host
$ python3 -c "
data = []
while True:
    data.append('x' * 1024 * 1024)  # Allocate 1MB chunks forever
    print(f'Allocated {len(data)} MB')
"
# Will eventually crash the entire system - NO PROTECTION!

✅ With Worker (Resource Protected):

# Same command but with memory limit - safely contained
$ worker-cli run --max-memory=512 python3 -c "
data = []
while True:
    data.append('x' * 1024 * 1024)
    print(f'Allocated {len(data)} MB')
"

$ worker-cli log <job-id>
Allocated 1 MB
Allocated 2 MB
...
Allocated 500 MB
Allocated 512 MB
Killed  # Job terminated when it hit the 512MB limit

Resource Protection:

Memory Limits: Jobs cannot exceed allocated memory
CPU Limits: Prevents CPU starvation of host system
I/O Limits: Controls disk bandwidth usage
Process Limits: Restricts number of processes per job

🏗️ CI/CD Pipeline Execution

❌ Without Worker:

# Running tests directly on CI server - risky!
npm test                    # Could consume all memory
python tests.py            # Might leave processes running  
./build.sh                 # Could fill up disk space
make integration-tests     # No isolation between test runs

# Manual process tracking
ps aux | grep test          # Find running tests manually
kill $(pgrep -f "npm test") # Hope this works

Problems:

Tests can crash the CI server
Resource leaks between jobs
No job monitoring or log collection
Security risks from untrusted code
Difficult to enforce resource limits
Manual process management

✅ With Worker:

# Safe, isolated execution with limits
worker-cli run --max-memory=512 --max-cpu=100 npm test
worker-cli run --max-memory=256 python tests.py  
worker-cli run --max-cpu=200 ./build.sh
worker-cli run --max-memory=1024 make integration-tests

# Automatic monitoring and control
worker-cli list                 # See all jobs at once
worker-cli log <job-id>        # Stream logs from any job
worker-cli stop <job-id>       # Clean job termination

Benefits:

✅ Complete isolation prevents system crashes
✅ Automatic resource cleanup
✅ Real-time monitoring and logging
✅ Failed jobs don’t affect the system
✅ Easy to track and debug build issues

🎓 Online Code Execution Platform

❌ Without Worker:

# Dangerous: Running user code directly
import subprocess
import sys

# This could do ANYTHING to your server!
user_code = request.json['code']
result = subprocess.run([sys.executable, '-c', user_code],
                        capture_output=True, text=True)

# No protection against:
# - Infinite loops
# - Memory bombs
# - File system access
# - Network attacks
# - Process spawning

Problems:

Users can access your filesystem
Infinite loops can crash your server
No resource limits (memory bombs)
No way to safely kill runaway processes
Security nightmare
No concurrent user support

✅ With Worker:

# Safe execution with Worker API
import worker_client


def execute_user_code(code):
    client = worker_client.JobClient("localhost:50051")

    # Safe execution with limits
    job = client.run_job(
        command="python3",
        args=["-c", code],
        max_cpu=50,  # 50% CPU max
        max_memory=128,  # 128MB max
        max_iobps=1000  # Limit I/O
    )

    # Stream results safely
    return client.get_job_logs(job.id)

Benefits:

✅ Complete filesystem isolation
✅ Resource limits prevent abuse
✅ Automatic job termination
✅ Real-time output streaming
✅ Multiple users can run code simultaneously
✅ No risk to host system

🔄 Data Processing Workflows

❌ Without Worker:

# Manual process management nightmare
./process_batch1.py &
PID1=$!
./process_batch2.py &  
PID2=$!
./process_batch3.py &
PID3=$!

# Manual monitoring - error prone
ps aux | grep process_batch
top | grep python

# Manual cleanup - often fails
kill $PID1 $PID2 $PID3  # Hope they actually die
kill -9 $(pgrep -f "process_batch")  # Nuclear option

# No resource control
# No way to limit memory/CPU per job
# No centralized logging

Problems:

No resource control
Manual process tracking
Difficult to monitor progress
Hard to debug failures
No way to limit resource usage
Cleanup often fails

✅ With Worker:

# Controlled parallel processing
worker-cli run --max-memory=1024 --max-cpu=100 ./process_batch1.py
worker-cli run --max-memory=1024 --max-cpu=100 ./process_batch2.py  
worker-cli run --max-memory=1024 --max-cpu=100 ./process_batch3.py

# Easy monitoring
worker-cli list                    # See all jobs and status
worker-cli log 1 & worker-cli log 2 & worker-cli log 3  # Monitor all logs

# Automatic resource management and cleanup
# Each job gets exactly 1GB RAM and 1 CPU core
# Automatic cleanup when jobs complete

Benefits:

✅ Predictable resource usage
✅ Easy progress monitoring
✅ Automatic cleanup
✅ Centralized logging
✅ Simple job control

Quick Start: Your First Job

Let’s get Worker running in under 5 minutes.

Installation

# Download the latest release
wget https://github.com/ehsaniara/worker/releases/latest/download/worker_1.0.0_amd64.deb

# Install the package
sudo dpkg -i worker_1.0.0_amd64.deb

# Start the service
sudo systemctl start worker
sudo systemctl enable worker

That’s it! Worker is now running as a system service with auto-generated SSL certificates.

Your First Job

# Run a simple command
worker-cli run echo "Hello from Worker!"

# Output:
# Job started:
# ID: 1
# Command: echo Hello from Worker!
# Status: RUNNING
# StartTime: 2024-01-15T10:30:45Z
# Network: host (shared with system)

Congratulations! You just ran your first isolated job.

Essential Commands

Running Jobs

The worker-cli run command is your main interface for job execution:

# Basic command
worker-cli run ls -la

# Python script
worker-cli run python3 -c "print('Hello from Python!')"

# With resource limits
worker-cli run --max-cpu=50 --max-memory=256 stress-ng --cpu 1 --timeout 30s

# Complex shell commands
worker-cli run bash -c "for i in {1..5}; do echo 'Iteration $i'; sleep 1; done"

Resource Limits Explained:

--max-cpu=N – CPU percentage (50 = 50% of one core)
--max-memory=N – Memory limit in MB
--max-iobps=N – I/O operations per second limit

Monitoring Jobs

# List all jobs
worker-cli list

# Output:
# 1 COMPLETED StartTime: 2024-01-15T10:30:45Z Command: echo Hello from Worker!
# 2 RUNNING StartTime: 2024-01-15T10:31:20Z Command: sleep 60
# 3 FAILED StartTime: 2024-01-15T10:32:10Z Command: invalid-command

# Get detailed job status
worker-cli status 2

# Output:
# Id: 2
# Command: sleep 60
# ExitCode: 0
# Started At: 2024-01-15T10:31:20Z
# Ended At: 
# Status: RUNNING
# MaxCPU: 100
# MaxMemory: 512
# MaxIOBPS: 0

Real-time Log Streaming

One of Worker’s killer features is real-time log streaming:

# Stream logs from a running job
worker-cli log 2

# Output streams in real-time:
# Logs for job 2 (Press Ctrl+C to exit if streaming):
# Starting process...
# Processing data...
# Step 1 complete
# Step 2 complete
# ...

The log command automatically follows the output until the job completes or you press Ctrl+C.

Job Control

# Stop a running job
worker-cli stop 2

# Output:
# Job stopped successfully:
# ID: 2
# Status: STOPPED

Real-World Examples

Example 1: Running a Data Processing Script

# Create a Python script
cat > process_data.py << 'EOF'
import time
import sys

print("Starting data processing...")
for i in range(10):
    print(f"Processing batch {i+1}/10")
    time.sleep(2)

print("Data processing complete!")
sys.exit(0)
EOF

# Run with memory limit and stream logs
worker-cli run --max-memory=128 python3 process_data.py &
JOB_ID=$!

# Stream the logs in real-time
worker-cli log $JOB_ID

Example 2: Building a Project

# Run a build with CPU and time limits
worker-cli run --max-cpu=200 bash -c "
    echo 'Starting build process...'
    make clean
    make -j4 all
    echo 'Build completed successfully'
"

Example 3: Testing with Network Access

# Worker allows network access by default
worker-cli run curl -s https://api.github.com/repos/ehsaniara/worker | jq '.stars'

# Or run a simple web server (accessible from host)
worker-cli run python3 -m http.server 8080

Configuration and Customization

Server Configuration

Worker’s behavior can be customized via configuration file at /opt/worker/config/config.yml:

server:
  address: "0.0.0.0"
  port: 50051

worker:
  defaultCpuLimit: 100        # Default CPU limit (100%)
  defaultMemoryLimit: 512     # Default memory limit (512MB)
  defaultIoLimit: 0           # No I/O limit by default
  maxConcurrentJobs: 50       # Max simultaneous jobs
  jobTimeout: "1h"            # Job timeout
  validateCommands: true      # Enable command validation

security:
  serverCertPath: "/opt/worker/certs/server-cert.pem"
  serverKeyPath: "/opt/worker/certs/server-key.pem"
  caCertPath: "/opt/worker/certs/ca-cert.pem"

logging:
  level: "INFO"               # DEBUG, INFO, WARN, ERROR
  format: "text"              # text or json
  output: "stdout"            # stdout or file path

After changing configuration:

sudo systemctl restart worker

Client Configuration

Point the CLI to a different server:

# Connect to remote Worker instance
worker-cli --server 192.168.1.100:50051 run echo "Hello from remote!"

# Or set it as default
export WORKER_SERVER=192.168.1.100:50051
worker-cli run echo "Using environment variable"

Security and Authentication

Certificate Management

Worker uses mutual TLS (mTLS) for security. Certificates are auto-generated during installation, but you can regenerate
them:

# Regenerate all certificates
sudo /usr/local/bin/certs_gen.sh

# This creates:
# - CA certificate
# - Server certificate 
# - Admin client certificate (full access)
# - Viewer client certificate (read-only access)

Role-Based Access

Worker supports two roles via certificate organizational units:

Admin Role (full access):

Run jobs
Stop jobs
View job status and logs
List all jobs

Viewer Role (read-only):

View job status and logs
List all jobs
Cannot run or stop jobs

The role is determined by the OU field in the client certificate.

Monitoring and Troubleshooting

Service Status

# Check Worker service status
sudo systemctl status worker

# View service logs
sudo journalctl -u worker -f

# Check resource usage
sudo systemctl show worker --property=CPUUsageNSec,MemoryCurrent

Job Debugging

# Check job output for failed jobs
worker status <job-id>
worker log <job-id>

# List recent jobs with status
worker list | head -10

# Monitor active jobs
watch -n 2 'worker list | grep RUNNING'

Common Issues

Issue: „Connection refused“

# Check if service is running
sudo systemctl status worker

# Check if port is open
sudo netstat -tlnp | grep 50051

# Restart service
sudo systemctl restart worker

Issue: „Certificate errors“

# Regenerate certificates
sudo /usr/local/bin/certs_gen.sh

# Check certificate permissions
ls -la /opt/worker/certs/

Issue: „Job stuck in RUNNING“

# Stop stuck job
worker-cli stop <job-id>

# Check system resources
top
df -h

Advanced Usage

Programmatic API

Worker provides a gRPC API for integration with other systems. Here’s a simple Go client example:

package main

import (
    "context"
    "fmt"
    "worker/pkg/client"
)

func main() {
    // Connect to Worker
    client, err := client.NewJobClient("localhost:50051")
    if err != nil {
        panic(err)
    }
    defer client.Close()

    // Run a job
    job := &pb.RunJobReq{
        Command:   "echo",
        Args:      []string{"Hello from API!"},
        MaxCPU:    50,
        MaxMemory: 256,
    }

    response, err := client.RunJob(context.Background(), job)
    if err != nil {
        panic(err)
    }

    fmt.Printf("Job %s startedn", response.Id)
}

Batch Job Processing

#!/bin/bash
# Process multiple files in parallel

for file in *.txt; do
    echo "Processing $file..."
    worker-cli run --max-memory=128 python3 process_file.py "$file" &
done

# Wait for all jobs to complete
wait
echo "All files processed!"

Integration with CI/CD

# GitLab CI example
test_job:
  script:
    - worker-cli run --max-cpu=100 --max-memory=256 npm test
    - worker-cli run --max-cpu=50 npm run lint

Performance Tips

Resource Limits: Always set appropriate limits to prevent runaway jobs
Concurrent Jobs: Monitor system resources when running many parallel jobs
Log Management: Long-running jobs can generate large logs – consider log rotation
Cleanup: Worker automatically cleans up completed jobs, but monitor disk space

Comparison with Alternatives

Feature	Worker	Docker	Systemd
Setup Complexity	Simple	Moderate	Complex
Resource Isolation	✅	✅	Limited
Network Isolation	Optional	✅	❌
Filesystem Isolation	✅	✅	❌
Real-time Logs	✅	Manual	Manual
Job API	✅	Manual	❌
Single Binary	✅	❌	❌

Getting Help

Documentation: Check the GitHub repository
Issues: Report bugs and feature requests on GitHub
Logs: Always check sudo journalctl -u worker for service issues
Community: Join discussions in the project’s GitHub issues

Conclusion

Worker provides a clean, simple way to run isolated jobs without the complexity of container orchestration. Whether
you’re processing data, running tests, or executing user code, Worker’s combination of security, simplicity, and
real-time monitoring makes it an excellent choice for job execution.

Key takeaways:

Installation is simple: One package, automatic configuration
Usage is intuitive: Familiar command-line interface
Security is built-in: Namespace isolation and mTLS authentication
Monitoring is real-time: Stream logs and monitor resource usage
Integration is straightforward: gRPC API for programmatic access

Project

https://github.com/ehsaniara/worker

Name	Typ	Größe	Geändert am	Zugriff
📁 .. (Zurück)
🗜️ dxvk-2.7.tar.gz	GZ	9.8 MB	07.07.2025 15:36	-rw-r--r--
📄 vkd3d-proton-2.14.1.tar.zst	ZST	2.77 MB	07.07.2025 15:37	-rw-r--r--

What is Worker?

Why Use Worker? Before vs After Scenarios

🔍 System Call Isolation in Action

🗂️ Filesystem Isolation Example

💾 Resource Protection Example

🏗️ CI/CD Pipeline Execution

🎓 Online Code Execution Platform

🔄 Data Processing Workflows

Quick Start: Your First Job

Installation

Your First Job

Essential Commands

Running Jobs

Monitoring Jobs

Real-time Log Streaming

Job Control

Real-World Examples

Example 1: Running a Data Processing Script

Example 2: Building a Project

Example 3: Testing with Network Access

Configuration and Customization

Server Configuration

Client Configuration

Security and Authentication

Certificate Management

Role-Based Access

Monitoring and Troubleshooting

Service Status

Job Debugging

Common Issues

Advanced Usage

Programmatic API

Batch Job Processing

Integration with CI/CD

Performance Tips

Comparison with Alternatives

Getting Help

Conclusion

Project

Schreibe einen Kommentar Antworten abbrechen