This content originally appeared on DEV Community and was authored by raphiki
This is the first article of a series about how to integrate ComfyUI with other tools to build more complex workflows. We’ll move beyond the familiar node-based interface to explore how to connect ComfyUI from code and no-code solutions, using API calls or MCP Servers.
You’ll learn how to use ComfyUI’s API to build custom applications and automate tasks, creating powerful and automated systems for generative AI.
ComfyUI is a powerful, modular interface for Stable Diffusion, allowing users to create complex AI image generation workflows with a node-based editor. Jupyter Notebook, on the other hand, is a popular interactive environment for data analysis, visualization, and prototyping.
By integrating ComfyUI with Jupyter Notebook, you can leverage the flexibility of ComfyUI’s workflows directly within your Python scripts or data science pipelines. This first article focuses on a simple approach using Basic HTTP API calls.
Most of this article is exported from an actual Jupyter Notebook. Both content, Python code and execution results are displayed.
The Use Case
Our goal is to build a high-level generative AI workflow that combines the power of an intelligent agent with the robust image generation capabilities of ComfyUI. The process unfolds in a few simple steps, all orchestrated within a Jupyter Notebook:
- User Input: The workflow begins with a simple, high-level prompt entered directly into the notebook.
- Agent-Powered Expansion: An OpenAI Assistant then takes this basic prompt and transforms it into a detailed, structured JSON Prompt Style Guide. This process enriches the initial idea with specific creative instructions, such as style, composition, and lighting.
- Initiating Generation: This expanded JSON guide is automatically injected into a pre-defined ComfyUI workflow. A single API call to the ComfyUI server starts the image generation process.
- Displaying the Result: Once the generation is complete, we make a second API call to fetch the resulting images. The images are then displayed directly within the Jupyter Notebook, completing our automated pipeline.
Prepare a ComfyUI Workflow
- Create or load a workflow in ComfyUI.
- Save the workflow as a .json file from the “File / Export (API)” menu (e.g.,
t2i-krea.json).
Get initial prompt from user
print("Please enter your prompt")
user_prompt = input()
Please enter your prompt
Hanuman flying over a modern city at night
Generate JSON Prompt Style Guide with an Assistant
import os
from openai import OpenAI
from dotenv import load_dotenv
load_dotenv()
client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))
# Create a thread
thread = client.beta.threads.create()
# Send a message
message = client.beta.threads.messages.create(
thread_id=thread.id,
role="user",
content=user_prompt
)
# Run the assistant
run = client.beta.threads.runs.create(
thread_id=thread.id,
assistant_id="asst_Uj0Qr0rG0bz8NVk1LWiS9UKv"
)
# Wait for completion and retrieve the response
import time
while run.status != "completed":
time.sleep(1)
run = client.beta.threads.runs.retrieve(thread_id=thread.id, run_id=run.id)
# Get the response
messages = client.beta.threads.messages.list(thread_id=thread.id)
json_prompt = messages.data[0].content[0].text.value
print(json_prompt)
{
"style_name": "Urban Deus Ex Hanuman",
"inspiration": [
"Modern Urban Aesthetics",
"Hindu Mythology",
"Superhero Comics",
"Cyberpunk Lighting"
],
"scene": "Hanuman, the Hindu god, flying over a bustling modern city radiating bright lights under the cloak of night sky",
"subjects": [
{
"type": "Hanuman",
"description": "Strong, muscular figure with a monkey face, holding a gada(mace).",
"position": "midground",
"pose": "flying with one hand extended",
"size": "large",
"expression": "determined",
"interaction": "flying over the city"
},
{
"type": "city",
"description": "modern urban skyline with skyscrapers, neon billboards, and busy traffic",
"position": "background",
"size": "expansive"
}
],
"style": "comic-realistic",
"color_palette": {
"primary": "#202020",
"secondary": "#505050",
"highlight": "#ff6a00",
"shadow": "#0d0d0d",
"background_gradient": [
"#0d0d0d",
"#303030"
]
},
"lighting": "Glistening city lights with diffused neon glow and soft moonlight",
"mood": "powerful and captivating",
"background": {
"type": "scenery",
"details": "Modern urban cityscape with skyscrapers, roads, traffic and massive billboards with neon signs"
},
"composition": "Slightly off-center focus with Hanuman taking up prominent space",
"camera": {
"angle": "low angle",
"distance": "medium shot",
"lens": "wide-angle",
"focus": "sharp subject, blurred background"
},
"medium": "Digital Painting",
"textures": [
"smooth skin of Hanuman",
"rough concrete of buildings",
"glossy glass of skyscrapers"
],
"resolution": "4K",
"details": {
"clothing": "Hanuman is dressed in traditional golden and red garment",
"weather": "Night with clear sky and a soft moonlight"
},
"effects": [
"Bokeh effect for city lights",
"Glow effect for neon lights"
],
"themes": [
"Divinity",
"Strength",
"Modernization",
"Contrast",
"Juxtaposition of Tradition with Modernity"
],
"usage_notes": "The style is effective in creating a surprising juxtaposition of traditional divinity with modern landscapes. Use this style for high impact illustrations where contrasts need to be highlighted."
}
Trigger the Workflow from Jupyter Notebook
Use the requests library to send a POST request to the ComfyUI API:
import requests
import json
# ComfyUI server URL
comfy_url = "http://127.0.0.1:8188"
prompt_url = f"{comfy_url}/prompt"
# Load your workflow JSON
with open("t2i-krea.json", "r") as f:
workflow = json.load(f)
# Replace the prompt
workflow["39:6"]["inputs"]["text"] = json_prompt
# Define the payload
payload = {
"prompt": workflow,
"client_id": "jupyter_notebook"
}
# Send the request
response = requests.post(prompt_url, json=payload)
# Get the prompt_id
prompt_id = response.json()['prompt_id']
print(prompt_id)
c1a2ced4-772c-4aeb-ac45-bfa183d03a88
Retrieve the generated images
ComfyUI processes the workflow asynchronously.
To fetch the result, poll the /history endpoint:
import time
from IPython.display import Image, display
# Wait for the workflow to complete
time.sleep(25) # Adjust based on workflow complexity
# Fetch the latest result for our prompt
history_url = f"{comfy_url}/history/{prompt_id}"
history = requests.get(history_url).json()
# Navigate to the list of image outputs and display them
image_outputs = history[prompt_id]["outputs"]["9"]["images"]
for image in image_outputs:
filename = image["filename"]
image_url = f"{comfy_url}/view?filename={filename}"
display(Image(url=image_url, width=200))
In this article, we’ve seen how to leverage the power of ComfyUI directly from a Jupyter Notebook. By making simple API calls, we were able to transform a user’s basic text prompt into a rich, detailed JSON guide using an OpenAI Assistant, and then feed that guide into a ComfyUI workflow to generate images. This approach demonstrates how you can move beyond the graphical interface to build automated, intelligent systems for creative tasks. The combination of Python’s flexibility and ComfyUI’s robust backend opens up a world of possibilities for custom, high-level generative AI workflows.
In the next article, we’ll take our integration a step further by exploring how to use WebSockets for Real-Time Interaction with ComfyUI.
This content originally appeared on DEV Community and was authored by raphiki