This content originally appeared on DEV Community and was authored by Ali Cheaib
What is Kestra?
Kestra is an open-source workflow orchestration platform that uses declarative YAML to define workflows. Think of it as a universal automation engine that can orchestrate anything – from simple file processing to complex multi-step business processes.
Unlike traditional CI/CD tools or data pipeline orchestrators, Kestra is designed to be genuinely general-purpose. It can handle DevOps tasks, business processes, data workflows, and basically anything that involves “do X, then Y, then Z.”
Real Example 1: Automated Customer Onboarding
Here’s a workflow that handles our entire customer onboarding process:
id: customer-onboarding
namespace: business-processes
inputs:
- id: customer_email
type: STRING
required: true
- id: customer_name
type: STRING
required: true
- id: plan_type
type: STRING
required: true
tasks:
- id: create-customer-account
type: io.kestra.plugin.scripts.python.Script
beforeCommands:
- pip install requests
script: |
import requests
import json
# Create account in our system
response = requests.post(
"{{ secret('API_BASE_URL') }}/customers",
headers={"Authorization": "Bearer {{ secret('API_TOKEN') }}"},
json={
"email": "{{ inputs.customer_email }}",
"name": "{{ inputs.customer_name }}",
"plan": "{{ inputs.plan_type }}"
}
)
if response.status_code == 201:
customer_data = response.json()
print(f"Created customer: {customer_data['id']}")
else:
raise Exception(f"Failed to create customer: {response.text}")
- id: setup-workspace
type: io.kestra.plugin.scripts.shell.Commands
commands:
- |
# Create customer workspace directory
mkdir -p /workspaces/{{ inputs.customer_email }}
# Copy template files
cp -r /templates/{{ inputs.plan_type }}/* /workspaces/{{ inputs.customer_email }}/
# Set permissions
chmod 755 /workspaces/{{ inputs.customer_email }}
echo "Workspace created for {{ inputs.customer_name }}"
- id: send-welcome-email
type: io.kestra.plugin.notifications.mail.MailSend
from: "welcome@ourcompany.com"
to: ["{{ inputs.customer_email }}"]
subject: "Welcome to Our Platform!"
htmlTextContent: |
<h2>Welcome {{ inputs.customer_name }}!</h2>
<p>Your {{ inputs.plan_type }} account has been created.</p>
<p>You can access your workspace at: https://app.ourcompany.com/{{ inputs.customer_email }}</p>
<p>If you have any questions, just reply to this email.</p>
- id: create-support-ticket
type: io.kestra.plugin.scripts.python.Script
beforeCommands:
- pip install requests
script: |
import requests
# Create welcome ticket in support system
response = requests.post(
"{{ secret('SUPPORT_API_URL') }}/tickets",
headers={"Authorization": "Bearer {{ secret('SUPPORT_TOKEN') }}"},
json={
"customer_email": "{{ inputs.customer_email }}",
"subject": "Welcome & Setup Assistance",
"priority": "low",
"type": "onboarding",
"description": "New customer onboarding completed. Follow up in 3 days."
}
)
print(f"Support ticket created: {response.json()['ticket_id']}")
- id: schedule-followup
type: io.kestra.plugin.core.flow.WorkingDirectory
tasks:
- id: create-followup-reminder
type: io.kestra.plugin.scripts.python.Script
script: |
import datetime
followup_date = datetime.datetime.now() + datetime.timedelta(days=3)
# This would integrate with your calendar/reminder system
print(f"Schedule follow-up for {followup_date.strftime('%Y-%m-%d')}")
print(f"Customer: {{ inputs.customer_name }} ({{ inputs.customer_email }})")
- id: update-crm
type: io.kestra.plugin.scripts.python.Script
beforeCommands:
- pip install requests
script: |
import requests
# Update CRM with onboarding completion
response = requests.patch(
"{{ secret('CRM_API_URL') }}/contacts/{{ inputs.customer_email }}",
headers={"Authorization": "Bearer {{ secret('CRM_TOKEN') }}"},
json={
"onboarding_status": "completed",
"onboarding_date": "{{ now() }}",
"account_type": "{{ inputs.plan_type }}"
}
)
print("CRM updated successfully")
triggers:
- id: webhook-trigger
type: io.kestra.core.models.triggers.types.Webhook
key: "customer-signup"
This workflow gets triggered whenever someone signs up on our website. It creates their account, sets up their workspace, sends welcome emails, creates support tickets, and updates our CRM. What used to take 30 minutes of manual work now happens in under 2 minutes automatically.
Real Example 2: Infrastructure Health Check
Here’s a workflow that monitors our infrastructure and automatically handles common issues:
id: infrastructure-health-check
namespace: devops
tasks:
- id: check-database-connections
type: io.kestra.plugin.scripts.python.Script
beforeCommands:
- pip install psycopg2-binary redis
script: |
import psycopg2
import redis
# Check PostgreSQL
try:
conn = psycopg2.connect(
host="{{ secret('DB_HOST') }}",
database="{{ secret('DB_NAME') }}",
user="{{ secret('DB_USER') }}",
password="{{ secret('DB_PASSWORD') }}"
)
conn.close()
print(" PostgreSQL: Healthy")
postgres_healthy = True
except Exception as e:
print(f" PostgreSQL: {e}")
postgres_healthy = False
# Check Redis
try:
r = redis.Redis(host="{{ secret('REDIS_HOST') }}", port=6379, decode_responses=True)
r.ping()
print(" Redis: Healthy")
redis_healthy = True
except Exception as e:
print(f" Redis: {e}")
redis_healthy = False
- id: check-api-endpoints
type: io.kestra.plugin.scripts.python.Script
beforeCommands:
- pip install requests
script: |
import requests
endpoints = [
"{{ secret('API_BASE_URL') }}/health",
"{{ secret('API_BASE_URL') }}/api/v1/status",
"{{ secret('ADMIN_API_URL') }}/health"
]
failed_endpoints = []
for endpoint in endpoints:
try:
response = requests.get(endpoint, timeout=10)
if response.status_code == 200:
print(f" {endpoint}: Healthy")
else:
print(f" {endpoint}: Status {response.status_code}")
failed_endpoints.append(endpoint)
except Exception as e:
print(f" {endpoint}: {e}")
failed_endpoints.append(endpoint)
if failed_endpoints:
raise Exception(f"Failed endpoints: {failed_endpoints}")
- id: check-disk-space
type: io.kestra.plugin.scripts.shell.Commands
commands:
- |
# Check disk usage on main servers
servers=("web-01" "web-02" "db-01")
for server in "${servers[@]}"; do
usage=$(ssh $server "df -h / | tail -1 | awk '{print \$5}' | sed 's/%//'")
echo "Server $server disk usage: ${usage}%"
if [ "$usage" -gt 85 ]; then
echo " WARNING: $server disk usage is ${usage}%"
fi
done
- id: restart-services-if-needed
type: io.kestra.plugin.scripts.shell.Commands
commands:
- |
# Check if any services are in failed state
failed_services=$(systemctl list-units --state=failed --no-pager | grep -v "0 loaded")
if [ ! -z "$failed_services" ]; then
echo "Found failed services:"
echo "$failed_services"
# Restart common services that sometimes fail
sudo systemctl restart nginx
sudo systemctl restart redis
echo "Services restarted"
else
echo "All services running normally"
fi
triggers:
- id: health-check-schedule
type: io.kestra.core.models.triggers.types.Schedule
cron: "*/15 * * * *" # Every 15 minutes
errors:
- id: alert-on-failure
type: io.kestra.plugin.notifications.slack.SlackIncomingWebhook
url: "{{ secret('SLACK_WEBHOOK') }}"
payload: |
{
"text": " Infrastructure Health Check Failed",
"attachments": [
{
"color": "danger",
"fields": [
{
"title": "Failed Task",
"value": "{{ task.id }}",
"short": true
},
{
"title": "Error",
"value": "{{ task.error }}",
"short": false
}
]
}
]
}
This runs every 15 minutes and checks our databases, APIs, disk space, and system services. If anything fails, it tries to fix common issues automatically and alerts us via Slack.
Real Example 3: Content Publishing Pipeline
Here’s a workflow that handles our blog publishing process:
id: content-publishing
namespace: marketing
inputs:
- id: article_file
type: FILE
required: true
- id: publish_date
type: DATETIME
required: true
- id: author_email
type: STRING
required: true
tasks:
- id: validate-content
type: io.kestra.plugin.scripts.python.Script
beforeCommands:
- pip install markdown beautifulsoup4
script: |
import markdown
from bs4 import BeautifulSoup
import re
# Read the uploaded file
with open("{{ inputs.article_file }}", 'r') as f:
content = f.read()
# Convert markdown to HTML
html = markdown.markdown(content)
soup = BeautifulSoup(html, 'html.parser')
# Validate content
word_count = len(content.split())
if word_count < 500:
raise Exception(f"Article too short: {word_count} words (minimum 500)")
# Check for required elements
if not soup.find('h1'):
raise Exception("Article must have at least one H1 heading")
# Check for images
img_count = len(soup.find_all('img'))
if img_count == 0:
print("Warning: No images found in article")
print(f"Article validated: {word_count} words, {img_count} images")
- id: optimize-images
type: io.kestra.plugin.scripts.shell.Commands
commands:
- |
# Find and optimize images in the article
mkdir -p optimized_images
# This would typically process images referenced in the markdown
find ./images -name "*.jpg" -o -name "*.png" | while read img; do
filename=$(basename "$img")
# Optimize image (example with imagemagick)
convert "$img" -resize "800x600>" -quality 85 "optimized_images/$filename"
echo "Optimized: $filename"
done
- id: generate-social-media-posts
type: io.kestra.plugin.scripts.python.Script
script: |
import re
# Read article content
with open("{{ inputs.article_file }}", 'r') as f:
content = f.read()
# Extract title (first H1)
title_match = re.search(r'^# (.+)$', content, re.MULTILINE)
title = title_match.group(1) if title_match else "New Article"
# Generate social media posts
twitter_post = f" New article: {title}\n\nRead more: https://blog.ourcompany.com/latest"
linkedin_post = f"I just published a new article: {title}\n\nCheck it out and let me know what you think!\n\nhttps://blog.ourcompany.com/latest"
# Save to files
with open('twitter_post.txt', 'w') as f:
f.write(twitter_post)
with open('linkedin_post.txt', 'w') as f:
f.write(linkedin_post)
print("Social media posts generated")
- id: schedule-publication
type: io.kestra.plugin.scripts.python.Script
beforeCommands:
- pip install requests
script: |
import requests
from datetime import datetime
# Upload to CMS
with open("{{ inputs.article_file }}", 'r') as f:
content = f.read()
response = requests.post(
"{{ secret('CMS_API_URL') }}/articles",
headers={"Authorization": "Bearer {{ secret('CMS_TOKEN') }}"},
json={
"content": content,
"author": "{{ inputs.author_email }}",
"publish_date": "{{ inputs.publish_date }}",
"status": "scheduled"
}
)
if response.status_code == 201:
article_id = response.json()['id']
print(f"Article scheduled for publication: {article_id}")
else:
raise Exception(f"Failed to schedule article: {response.text}")
- id: notify-team
type: io.kestra.plugin.notifications.slack.SlackIncomingWebhook
url: "{{ secret('SLACK_WEBHOOK') }}"
payload: |
{
"text": "📄 New article ready for publication",
"attachments": [
{
"color": "good",
"fields": [
{
"title": "Author",
"value": "{{ inputs.author_email }}",
"short": true
},
{
"title": "Publish Date",
"value": "{{ inputs.publish_date }}",
"short": true
}
]
}
]
}
This workflow handles our entire content publishing process: validates articles, optimizes images, generates social media posts, schedules publication, and notifies the team.
Why Kestra Works So Well
1. Truly Declarative
Everything is defined in YAML. No code to maintain, no complex dependencies to manage. Just describe what you want to happen.
2. Built-in Integrations
Kestra comes with plugins for databases, APIs, cloud services, notifications, and more. Most common tasks don’t require custom code.
3. Visual Workflow Editor
The web interface lets you build workflows visually, see execution in real-time, and debug issues easily.
4. Flexible Triggers
Workflows can be triggered by schedules, webhooks, file changes, or other workflows. Mix and match as needed.
5. Resource Management
Tasks run in isolated containers with configurable resources. No more “it works on my machine” issues.
Getting Started
Installation is straightforward:
# Using Docker
docker run -d \
--name kestra \
-p 8080:8080 \
-v /tmp/kestra:/tmp/kestra \
kestra/kestra:latest \
server standalone
Then visit http://localhost:8080 and start building workflows.
Real-World Tips
- Start simple: Begin with basic file processing or notification workflows
- Use secrets: Never hardcode credentials in workflows
- Test incrementally: Build workflows task by task
- Monitor resource usage: Tasks run in containers, so size appropriately
- Use the visual editor: It’s genuinely helpful for building complex workflows
When to Use Kestra
Good for:
- Multi-step business processes
- Integration between different systems
- Automated workflows that need human oversight
- Complex scheduling requirements
- Mixed workloads (APIs, databases, files, notifications)
Maybe not ideal for:
- Simple CI/CD (GitHub Actions might be better)
- Real-time processing (use streaming platforms)
- Very high-frequency tasks (consider event-driven architectures)
The Bottom Line
Kestra fills a gap that I didn’t realize existed. It’s more flexible than traditional CI/CD tools, simpler than complex workflow engines, and more powerful than basic scheduling tools.
This content originally appeared on DEV Community and was authored by Ali Cheaib