Skip to content

Compute Process Lifecycle

All compute processes — whether workflows, apps, services, or Excel add-ins — go through the same lifecycle stages from deployment to completion. Understanding these stages helps you troubleshoot issues and know what to expect.

Lifecycle Stages

flowchart LR subgraph Deploy["📦 Deploy"] A["Code uploaded"] --> B["Building"] end subgraph Build["🔨 Build"] B --> C{"Success?"} C -->|No| D["Created
fix & retry"] C -->|Yes| E["Ready"] end subgraph Run["🚀 Run"] E --> F["Pending"] F --> G["Starting"] G --> H["Running"] end subgraph Complete["✅ Complete"] H --> I["Completed"] end

Stage 1: Building

What happens: When you deploy code (using the Python SDK or CLI), Datatailr packages it into a Docker container image. This image includes your code, Python packages, and any system dependencies you specified.

What you see: The process shows status Building.

How long: Typically 1-3 minutes, depending on the number of dependencies.

Stage 2: Ready

What happens: The container image is built and stored in Datatailr's registry. Your compute process is ready to run.

What you see: The process shows status Ready.

For scheduled processes: If your process has a schedule (like "run daily at 9 AM"), it transitions to Scheduled and waits for the next scheduled time.

Stage 3: Pending

What happens: Datatailr is allocating compute resources. If all available machines are busy, Datatailr will automatically start a new compute node.

What you see: The process shows status Pending.

How long: Usually a few seconds. If a new node needs to start, this can take 1-2 minutes.

Stage 4: Starting

What happens: A compute node has been assigned. Datatailr is downloading your container image and preparing to run it.

What you see: The process shows status Starting.

Stage 5: Running

What happens: Your code is executing! For workflows, this is when your tasks run — in parallel where possible.

What you see: The process shows status Running. You can view real-time logs.

Stage 6: Completed

What happens: Your compute process finished successfully. For workflows, all tasks completed. Results are saved and available.

What you see: The process shows status Completed with a green indicator.

When Things Go Wrong

Sometimes compute processes don't complete successfully. Here's what the different failure states mean:

Status What It Means What To Do
Created The build failed (bad dependency, syntax error) Check build logs, fix your code, redeploy
Failed Your code threw an unhandled exception Check logs, fix the bug, re-run
Out of Memory Process exceeded its memory allocation Increase memory in config or optimize your code
Expired A scheduled process's end date has passed Update the schedule or redeploy

Reading logs

You can always check what happened by reading the logs:

  • In the UI: Click on the process and select "Logs"
  • Using CLI: dt log read <process_name>
  • In Python: Use the dt.log.read() function

Monitoring Your Compute Processes

From the Dashboard

The Datatailr dashboard shows all your compute processes with their current status. You can:

  • See which processes are running, pending, or completed
  • View execution history and run times
  • Access logs and output
  • Start, stop, or re-run processes

From the CLI

Use the dt job commands to manage compute processes:

# List all your compute processes
dt job ls

# Get details about a specific process
dt job get my_process_name

# View recent runs of a process
dt job runs my_process_name

# Read logs from a process
dt log read my_process_name

Next Steps

Now that you understand the lifecycle, you're ready to: