Long-Lived Connections and Streaming Push: Standardizing SSE / WebSocket Implementations to Replace Inefficient Polling

In web systems, there is a type of requirement that is especially prone to being implemented by AI as “it runs, but it’s dumb”:

Frontend:
Ask the server every 1 second
"Is the task done?"

Server:
"No."

Ask again 1 second later:
"Is it done?"

This approach is called:

Polling

Querying a status occasionally is fine, but for real-time scenarios such as AI streaming responses, OCR progress, document parsing, knowledge base construction, and background task status, high-frequency polling generates a large number of useless requests.

A more appropriate solution is usually:

SSE (Server-Sent Events)

Or:

WebSocket — a full-duplex, long-lived connection protocol

The core idea boils down to one sentence:

Don’t make the client keep asking “Got any message yet?” — let the server push messages proactively when there is something new.

📌 Technical Profile

Long-lived Connection

After a client and server establish a connection, it is not closed immediately but kept alive for a period of time to support subsequent data transfer.

There are two common solutions.

SSE

SSE (Server-Sent Events) is based on HTTP and is mainly used for:

Server
  ↓
Client

Suitable for:

• AI token streaming output;
• Background task progress;
• Log streams;
• Status notifications.

WebSocket

WebSocket supports:

Client
   ↕
Server

In other words:

Full-Duplex Communication

Better suited for:

• Online chat;
• Multi-user collaboration;
• Real-time games;
• Bidirectional real-time control.

A quick rule of thumb:

If only the server needs to push to the client, prefer SSE; if both sides need to actively send messages at high frequency, consider WebSocket.

💡 A Simple Analogy

Polling is like constantly calling the repair shop:

You: Is it fixed?
Clerk: No.

1 minute later:
You: Is it fixed?
Clerk: Not yet.

While server push is more like:

You: Let me know when it's fixed.

...

Clerk: It's fixed.

So remember:

Polling
Client keeps asking

SSE
Server pushes whenever there's a message

WebSocket
Both sides can speak anytime

1. The Most Common Mistake AI Makes: Simulating Real-Time with High-Frequency Polling

Suppose you ask AI to implement:

Upload a PDF and display processing progress in real time.

It easily generates:

POST /tasks
GET /tasks/{task_id}/status

And the frontend:

setInterval(async () => {
    const result = await getTaskStatus(taskId)
}, 1000)

If 100 users query once per second, the server gets an extra:

100 HTTP Requests

per second. For a task that lasts 60 seconds:

100 × 60 = 6000 requests

But the actual number of state changes may only be a few dozen.

Therefore:

Don’t simulate real-time push with high-frequency polling.

2. Architectural Principles

1. Decouple Task Execution from Streaming

Not recommended:

DocumentService
      ↓
Directly controls StreamingResponse
      ↓
Directly yields SSE

A more reasonable approach:

Business Task
     ↓
Progress Event
     ↓
Async Queue
     ↓
SSE Endpoint
     ↓
Browser

In other words:

The task is responsible for producing events; the SSE endpoint is responsible for sending them.

This way, business logic doesn’t need to know the details of the HTTP protocol.

2. Use a Unified Event Format

Different tasks should not each define their own:

{"percent": 30}

Or:

{"status": "almost_done"}

Instead, unify on something like:

{
  "stage": "embed",
  "message": "Embedding chunks...",
  "progress": 72.5,
  "done": false,
  "error": false,
  "ts": "2026-08-17T10:00:00"
}

Meaning of each field:

stage
Current stage

message
Status description

progress
Progress 0–100

done
Whether completed

error
Whether failed

ts
Event timestamp

This way the frontend can reuse the same progress component.

3. The SSE Lifecycle Must Be Complete

A reliable SSE implementation should consider at least four things:

Create Queue
   ↓
Continuously Send Events
   ↓
Keepalive
   ↓
Close and clean up after done / error

Keepalive

When there is no data for a long time, intermediate proxies may mistakenly assume the connection is dead.

So you can periodically send:

: keepalive

Explicit Termination Conditions

Task completed:

done = true

Task failed:

error = true

Both should end the stream.

Clean Up Resources

After the connection closes, the task queue should be removed; otherwise long-running operations may cause memory to accumulate.

Queues Must Have an Upper Bound

For example:

asyncio.Queue(maxsize=200)

This prevents events from piling up indefinitely when the client consumes too slowly.

This is related to the:

Backpressure

mechanism — when consumers can’t keep up with producers, the system needs a way to limit accumulation.

4. Disable Caching and Proxy Buffering

One common SSE problem:

The server keeps doing:

yield event

but the browser receives nothing for a long time, then suddenly gets a batch of events all at once.

This is usually caused by proxy buffering, so the common response configuration is:

Cache-Control: no-cache
X-Accel-Buffering: no

The purpose is simple:

Deliver events to the client as soon as they are produced, instead of accumulating them and sending later.

5. Separate Task Execution from Task Observation

For long-running tasks such as OCR, document processing, and Embedding, a more reasonable structure is:

Client
   ↓
Start Task
   ↓
task_id

Background Task
   ↓
Continuously produce Progress Events

Client
   ↓
SSE /tasks/{task_id}/progress

That is:

Task execution does the work; SSE observes the progress.

The two are linked via task_id.

6. Why Not Overuse WebSocket

WebSocket is more powerful, but it also means you need to handle extra concerns:

• Heartbeats;
• Reconnection;
• Authentication;
• Connection management;
• Multi-instance routing;
• Message ordering;
• Broadcasting;
• Cleanup on disconnect.

If the business is just:

20%
40%
80%
100%

this kind of one-way progress push, SSE is usually simpler.

So:

An architecture isn’t more advanced because it uses more WebSockets — the protocol should match the communication pattern.

3. What Do Long-Lived Connections and Streaming Push Bring?

1. Fewer Useless Requests

Polling:

Client → Server
Client → Server
Client → Server

SSE:

Client ───────── Server
                 ↓
                20%
                 ↓
                50%
                 ↓
               100%

Business events are pushed only when something actually changes.

2. Lower Perceived Latency

Perceived Latency

Even if an AI response still takes 10 seconds in total:

Traditional mode:

Wait 10 seconds
↓
Complete answer suddenly appears

Streaming mode:

Start seeing content at second 1
↓
Continuous output
↓
Ends at second 10

Users will clearly feel the system is faster.

3. A Natural Fit for AI Systems

AI systems inherently involve:

LLM Token Stream
Document processing progress
Embedding progress
Web Search status
Tool execution process
Agent execution status

So streaming push is usually a key foundational capability of Agent systems.

4. Putting Prompts into Practice: Directly Constraining AI

You can add the following rules to:

Project Rules

## Streaming and Long-lived Connection Rules

1. Do not use high-frequency polling to fetch real-time progress when server push is more appropriate.
2. Prefer SSE for server-to-client streaming:
- AI token streaming
- Background task progress
- OCR/PDF processing progress
- Logs
- Notifications
3. Only use WebSocket when frequent bidirectional communication is required.
4. Separate task execution from streaming.

Recommended architecture: Task → Progress Event → Async Queue → SSE Endpoint → Client

5. Use a unified event schema:
{
    stage,
    message,
    progress,
    done,
    error,
    ts
}
6. SSE must support keepalive.
7. Close the stream when done=true or error=true.
8. Clean up queues and resources after termination.
9. Event queues must have a finite capacity.
10. Disable response buffering for SSE where necessary.
11. Before generating polling code, check whether SSE or WebSocket is more appropriate.

This is much better than saying:

“Help me implement real-time progress.”

because it prevents AI from reflexively generating setInterval().

5. A Positive Example: The Real SSE Implementation in miniagent

miniagent has already implemented a complete task-progress SSE architecture:

Task
 ↓
ProgressTracker
 ↓
asyncio.Queue
 ↓
SSE Endpoint
 ↓
Browser

1. Each Task Has Its Own Event Queue

miniagent’s ProgressTracker uses:

class ProgressTracker:

    _queues: dict[str, asyncio.Queue] = {}

    @classmethod
    def create(cls, task_id: str) -> asyncio.Queue:
        q: asyncio.Queue = asyncio.Queue(maxsize=200)
        cls._queues[task_id] = q
        return q

Each task corresponds to its own Queue:

task_001 → Queue A
task_002 → Queue B
task_003 → Queue C

And uses:

maxsize=200

to limit event backlog.

2. The Business Layer Only Publishes Events

miniagent publishes via:

await ProgressTracker.emit(
    task_id,
    stage="embed",
    message="Embedding chunks...",
    progress=72.5,
)

emitting a unified structure:

{
    "stage": stage,
    "message": message,
    "progress": round(progress, 1),
    "done": done,
    "error": error,
    "ts": datetime.now().isoformat(),
}

The task itself doesn’t need to touch StreamingResponse, achieving decoupling between business logic and the SSE transport layer.

3. The SSE Endpoint Continuously Consumes Events

miniagent provides:

GET /{task_id}/progress

Core logic:

queue = ProgressTracker.get(task_id)

event = await asyncio.wait_for(
    queue.get(),
    timeout=30.0,
)

When new progress arrives, the Queue immediately hands the event to the SSE Endpoint.

4. Send Keepalive After 30 Seconds Without Events

miniagent on timeout:

except asyncio.TimeoutError:
    yield ": keepalive\n\n"

keeping the long-lived connection active even when the task has no new status for a while.

5. Automatically Close and Clean Up on Completion or Failure

Send the event:

yield f"data: {json.dumps(event, ensure_ascii=False)}\n\n"

Then check:

if event.get("done") or event.get("error"):
    break

Finally:

finally:
    ProgressTracker.remove(task_id)

In other words, the complete lifecycle is:

Create Queue
   ↓
Emit Progress
   ↓
SSE Push
   ↓
done / error
   ↓
Close Stream
   ↓
Remove Queue

6. Disable Caching and Proxy Buffering

Finally, miniagent returns:

return StreamingResponse(
    event_generator(),
    media_type="text/event-stream",
    headers={
        "Cache-Control": "no-cache",
        "X-Accel-Buffering": "no",
    },
)

Where:

text/event-stream

indicates the SSE data stream.

And:

Cache-Control: no-cache
X-Accel-Buffering: no

are used to reduce the impact of caching and reverse-proxy buffering on real-time delivery.

6. miniagent’s SSE Architecture

The most important boundary in this structure is:

Task Producer
   ↓
Event Queue
   ↓
SSE Transport Layer
   ↓
Client

rather than letting task code directly control the HTTP Response.

Below is a more detailed data flow diagram of the SSE implementation:

7. A Checklist for AI

Next time you ask AI to implement a real-time feature, you can require it to check:

Streaming Architecture Checklist

□ Is polling really needed here?
□ If it's only Server → Client, should SSE be preferred?
□ Is bidirectional WebSocket really necessary?
□ Is task execution decoupled from Streaming?
□ Is a unified event format used?
□ Does the Queue have a capacity limit?
□ Does the SSE have keepalive?
□ Does it end after done / error?
□ Are resources cleaned up after ending?
□ Are caching and proxy buffering disabled?

Summary

AI easily generates:

setInterval
   ↓
GET /status
   ↓
Not done
   ↓
Request again

For real-time progress, AI output, and long-running tasks, this often produces a flood of useless requests.

A more reasonable design is:

Background Task
   ↓
Progress Event
   ↓
Async Queue
   ↓
SSE
   ↓
Browser

Only when frequent bidirectional communication is genuinely required should you use:

Client
   ↕
WebSocket
   ↕
Server

miniagent already forms a complete SSE task-progress push chain through ProgressTracker, a bounded asyncio.Queue, FastAPI’s StreamingResponse, 30-second Keepalive, done/error termination, and Queue cleanup.

For AI programming, what we need to avoid is not just incorrect code, but also this kind of architectural waste:

When the server could proactively tell you the result, the client still keeps asking: “Is it done yet?”

This is precisely the value of long-lived connections and streaming push.

Open Source Code

• github
• gitee

🪐 Wishing you good luck 🪐