Layer 4 (Transport) - TCP connection pool, ring buffers, and lwIP integration. More...

#include "ServerConfig.h"
#include "freertos/FreeRTOS.h"
#include "freertos/queue.h"
#include "lwip/tcp.h"
#include "network_drivers/network/ip.h"
#include "shared_primitives/ring.h"
#include <Arduino.h>
#include <atomic>

Classes
struct	TcpConn
	A single TCP connection context. More...

struct	TcpEvt
	Event record posted from lwIP callbacks to the session layer. More...

class	DeterministicAsyncTCP
	Static-only facade managing the shared TCP connection pool. More...

Macros
#define	DETWS_PROTO_SLOT_NONE 0xFFu
	Sentinel for TcpConn::proto_slot meaning "no per-protocol session bound".

#define	DETWS_OBS_TRANSITION(slot, olds, news, reason) ((void)0)

#define	DETWS_OBS_NOTICE(slot, st, reason) ((void)0)

Enumerations
enum class	ConnState : uint8_t { CONN_FREE , CONN_ACTIVE , CONN_CLOSING }
	Lifecycle state of a connection pool slot. More...

enum class	EvtType : uint8_t { EVT_CONNECT , EVT_DATA , EVT_DISCONNECT , EVT_ERROR }
	Type of connection event posted to a listener's FreeRTOS queue. More...

Functions
bool	det_conn_send (uint8_t slot, const void *data, u16_t len)
	Send `len` bytes on connection `slot` (copies `data`; TLS-aware).

bool	det_conn_send_flush (uint8_t slot, const void *data, u16_t len)
	Send `len` bytes on `slot` and flush in a single tcpip_thread round-trip.

u16_t	det_conn_sndbuf (uint8_t slot)
	Bytes that can currently be queued for sending on `slot`.

void	det_conn_flush (uint8_t slot)
	Flush queued bytes / finish the send on `slot` (TLS-aware).

void	det_conn_touch_active (uint8_t slot)
	Refresh `slot's` idle-timeout timestamp while a response body is in flight.

void	det_conn_ack_consumed (uint8_t slot)
	Reopen the TCP receive window by however much `slot` has drained.

uint8_t	det_conn_active_count ()
	Number of server connection slots currently in the CONN_ACTIVE state.

bool	det_conn_raw_send (struct tcp_pcb pcb, const void data, u16_t len)
	Write raw bytes straight to `pcb` (no TLS), context-safe.

void	det_conn_close (uint8_t slot)
	Close connection `slot` gracefully (tcp_close), aborting if the FIN cannot be queued. The transport owns the whole teardown: it detaches the pcb from its lwIP callbacks, frees the slot, and (TLS slot) emits close_notify + frees the per-connection TLS context - so callers pass only the slot and never touch the raw tcp_pcb. A no-op if the slot has no live pcb.

void	det_conn_begin_close (uint8_t slot_id)
	Begin a graceful close that dwells in ConnState::CONN_CLOSING until the peer ACKs.

void	det_conn_detach (struct tcp_pcb *pcb)
	Detach `pcb` from its slot's lwIP callbacks before the slot is freed.

void	det_conn_abort (struct tcp_pcb *pcb)
	Hard-abort `pcb` (RST) for a fatal condition; no graceful FIN.

void	det_conn_abort_slot (uint8_t slot)
	Hard-abort connection `slot` (RST) for a fatal condition. The transport owns the teardown order: free the per-connection TLS context (abrupt, no close_notify), detach the pcb from its callbacks, reset the slot, then abort - so callers pass only the slot and never touch the raw tcp_pcb. A no-op if the slot has no live pcb.

uint32_t	det_conn_remote_ip (uint8_t slot)
	Raw source IPv4 of the connection in `slot`, or 0 if the slot has no active pcb (or on host builds). Byte order is irrelevant: this is an identity key (e.g. for the auth lockout), not for display. Keeps the lwIP pcb access inside L4 so callers never reach into the pcb directly.

bool	det_conn_remote_addr (uint8_t slot, DetIp *out)
	The connected peer's address as a family-tagged DetIp (IPv4 or IPv6).

void	det_lwip_to_detip (const ip_addr_t ra, DetIp out)
	A stable per-peer 32-bit identity key for `slot` (the v4 address, or an FNV-1a hash of a v6 address). For rate-limit / auth-lockout buckets, where a v6 peer must not silently share the all-zero v4 bucket. Returns 0 if the slot has no active connection.

err_t	lowlevel_recv_cb (void arg, struct tcp_pcb tpcb, struct pbuf *p, err_t err)
	lwIP receive callback - wired to each new connection by listener_accept_cb.

err_t	lowlevel_sent_cb (void arg, struct tcp_pcb tpcb, u16_t len)
	lwIP sent callback - refreshes the idle-timeout timestamp.

void	lowlevel_err_cb (void *arg, err_t err)
	lwIP error callback - fires when the stack detects a fatal error.

bool	listener_enqueue (uint8_t listener_id, const TcpEvt *evt)
	Post `evt` to the queue owned by listener `listener_id`.

Variables
uint32_t	detws_ap_ip
	softAP IPv4 address (network byte order) for STA/AP interface tagging.

TcpConn	conn_pool [CONN_POOL_SLOTS]
	Static pool of connection contexts. Defined in tcp.cpp. Sized CONN_POOL_SLOTS: MAX_CONNS TCP slots plus any reserved internal dispatch slot(s) (HTTP/3); the TCP accept path only ever uses [0, MAX_CONNS).

Detailed Description

Layer 4 (Transport) - TCP connection pool, ring buffers, and lwIP integration.

Defines the static connection pool and the per-connection event plumbing. Each listener port owns its own FreeRTOS queue (see listener.h); the session layer drains all active queues each tick via server_tick().

Concurrency model

Context	Reads	Writes
lwIP callbacks	rx_head (to check full)	rx_buffer[], rx_head
Arduino loop	rx_buffer[], rx_tail	rx_tail

state, rx_head, and rx_tail are DetAtomic (acquire/release): the single-producer / single-consumer ring buffer is correct without a mutex because the release store of an index publishes the preceding buffer writes and the acquire load observes them, on either core.

Backpressure (lossless) When a whole inbound segment will not fit the free ring space, the recv callback refuses it (returns ERR_MEM without freeing the pbuf); lwIP retains it as refused_data and redelivers once the main loop has drained the ring, so no received byte is dropped. Requires RX_BUF_SIZE > one TCP segment (TCP_MSS).

Author: Douglas Quigg (dstroy0)

Date: 2026

Definition in file tcp.h.

Macro Definition Documentation

◆ DETWS_PROTO_SLOT_NONE

#define DETWS_PROTO_SLOT_NONE 0xFFu

Sentinel for TcpConn::proto_slot meaning "no per-protocol session bound".

Definition at line 111 of file tcp.h.

◆ DETWS_OBS_TRANSITION

#define DETWS_OBS_TRANSITION	(	slot,
		olds,
		news,
		reason
	)	((void)0)

Definition at line 516 of file tcp.h.

◆ DETWS_OBS_NOTICE

#define DETWS_OBS_NOTICE	(	slot,
		st,
		reason
	)	((void)0)

Definition at line 517 of file tcp.h.

Enumeration Type Documentation

◆ ConnState

enum class ConnState : uint8_t

strong

Lifecycle state of a connection pool slot.

Transitions:

ConnState::CONN_FREE → ConnState::CONN_ACTIVE : accept callback fires.
ConnState::CONN_ACTIVE → ConnState::CONN_FREE : graceful close, error, or timeout.
ConnState::CONN_ACTIVE → ConnState::CONN_CLOSING : (reserved for future half-close support).

Enumerator
CONN_FREE	Slot is available; no PCB is attached.
CONN_ACTIVE	Live connection; PCB is valid.
CONN_CLOSING	FIN sent; waiting for final ACK (reserved).

Definition at line 57 of file tcp.h.

◆ EvtType

enum class EvtType : uint8_t

strong

Type of connection event posted to a listener's FreeRTOS queue.

Enumerator
EVT_CONNECT	New connection accepted.
EVT_DATA	Data received; bytes are already in the ring buffer.
EVT_DISCONNECT	Remote peer closed the connection gracefully.
EVT_ERROR	lwIP reported an error (PCB may already be freed).

Definition at line 134 of file tcp.h.

Function Documentation

◆ det_conn_send()

bool det_conn_send	(	uint8_t	slot,
		const void *	data,
		u16_t	len
	)

Send len bytes on connection slot (copies data; TLS-aware).

Returns: true if the bytes were queued; false if the send buffer was full and the write was refused. A streaming producer should pace with det_conn_sndbuf() and resume on a later loop; existing fixed-size senders may ignore the result.

Definition at line 362 of file tcp.cpp.

References conn_pool, and DET_OP_SEND.

Referenced by DetWebServer::send(), DetWebServer::send_chunked(), DetWebServer::send_template(), sse_write(), ssh_conn_accept(), ssh_conn_close_channel(), ssh_conn_open_forwarded(), and ssh_conn_send().

◆ det_conn_send_flush()

bool det_conn_send_flush	(	uint8_t	slot,
		const void *	data,
		u16_t	len
	)

Send len bytes on slot and flush in a single tcpip_thread round-trip.

The terminal-write analogue of det_conn_send(): the tcp_write and its tcp_output() run inside one marshaled op, so a small single-shot response costs one ~23 us on-device marshal instead of the det_conn_send()+det_conn_flush() pair (two). Use for the LAST write of a response whose body is already fully buffered (send / send_empty / redirect); a streaming producer that pages across loops must keep using det_conn_send() + a single trailing det_conn_flush(). TLS-identical to det_conn_send (the record BIO already outputs per record). Same return contract as det_conn_send.

Definition at line 378 of file tcp.cpp.

References conn_pool, and DET_OP_SEND.

Referenced by DetWebServer::redirect(), DetWebServer::send(), and DetWebServer::send_empty().

◆ det_conn_sndbuf()

u16_t det_conn_sndbuf ( uint8_t slot )

Bytes that can currently be queued for sending on slot.

Advisory free space in the TCP send buffer: a producer can send at most this many bytes per handle() loop and resume on the next loop as the window drains (the on_poll hook is the natural resume point). For a TLS slot the usable plaintext is somewhat less (TLS record + cipher overhead). Returns 0 when the slot has no pcb.

Definition at line 398 of file tcp.cpp.

References conn_pool, and TcpConn::pcb.

◆ det_conn_flush()

void det_conn_flush ( uint8_t slot )

Flush queued bytes / finish the send on slot (TLS-aware).

Definition at line 413 of file tcp.cpp.

References conn_pool, and DET_OP_OUTPUT.

Referenced by ssh_conn_accept(), ssh_conn_close_channel(), ssh_conn_open_forwarded(), ssh_conn_send(), and ws_close().

◆ det_conn_touch_active()

void det_conn_touch_active ( uint8_t slot )

Refresh slot's idle-timeout timestamp while a response body is in flight.

The file/chunk send pumps call this each poll they run: a slot still paging out a body is actively streaming (or briefly blocked on a full window / a transient link stall), not idle, so the CONN_TIMEOUT_MS idle sweep must not reap it mid-transfer - that truncates any body larger than one TCP window. A genuinely dead peer is still reclaimed by lwIP's retransmission timers (err callback), not this sweep.

Definition at line 718 of file tcp.cpp.

References CONN_ACTIVE, conn_pool, detws_millis(), TcpConn::last_activity_ms, MAX_CONNS, and TcpConn::state.

◆ det_conn_ack_consumed()

void det_conn_ack_consumed ( uint8_t slot )

Reopen the TCP receive window by however much slot has drained.

Ack-on-consume, owned entirely by the transport layer: recv_cb no longer ACKs on copy. Instead the window tracks how much the application has actually drained from the ring (rx_tail) since the last ACK, so it never advertises more than the ring can hold and the peer is paced to the consumer; a slow sink (e.g. flash writes during a streamed upload) can never overflow the ring and deadlock.

Other layers do not touch the ring indices to manage flow control - the worker just calls this once per owned slot per loop and transport does the rest (computes the delta vs rx_acked, marshals tcp_recved to tcpip_thread, advances rx_acked). A no-op when nothing was drained or slot is not active.

Definition at line 427 of file tcp.cpp.

References CONN_ACTIVE, conn_pool, DET_OP_RECVED, MAX_CONNS, TcpConn::pcb, TcpConn::rx_acked, RX_BUF_SIZE, TcpConn::rx_tail, and TcpConn::state.

Referenced by DetWebServer::service_once().

◆ det_conn_active_count()

uint8_t det_conn_active_count ( )

Number of server connection slots currently in the CONN_ACTIVE state.

The connection pool owns this aggregate: callers (stats / metrics) ask transport for the count instead of sweeping conn_pool[] and testing .state themselves.

Definition at line 646 of file tcp.cpp.

References CONN_ACTIVE, conn_pool, and MAX_CONNS.

◆ det_conn_raw_send()

bool det_conn_raw_send	(	struct tcp_pcb *	pcb,
		const void *	data,
		u16_t	len
	)

Write raw bytes straight to pcb (no TLS), context-safe.

This is the one safe path for the TLS engine's BIO to emit ciphertext: it writes directly when already running inside the lwIP thread (the marshaled app-data send path) and marshals a raw write when called from the main-loop task (the handshake / read pump), so a TLS handshake never does an unsynchronized tcp_write from the main loop. Calls tcp_output() on success.

Returns: true if the bytes were queued; false on a full send buffer (ERR_MEM).

Definition at line 449 of file tcp.cpp.

References DET_OP_RAWSEND.

◆ det_conn_close()

void det_conn_close ( uint8_t slot )

Close connection slot gracefully (tcp_close), aborting if the FIN cannot be queued. The transport owns the whole teardown: it detaches the pcb from its lwIP callbacks, frees the slot, and (TLS slot) emits close_notify + frees the per-connection TLS context - so callers pass only the slot and never touch the raw tcp_pcb. A no-op if the slot has no live pcb.

Definition at line 467 of file tcp.cpp.

References CONN_ACTIVE, CONN_FREE, conn_pool, det_conn_detach(), DET_OP_CLOSE, DETWS_OBS_TRANSITION, MAX_CONNS, TcpConn::pcb, TcpConn::state, and TcpConn::tls.

Referenced by ssh_conn_accept().

◆ det_conn_begin_close()

void det_conn_begin_close ( uint8_t slot_id )

Begin a graceful close that dwells in ConnState::CONN_CLOSING until the peer ACKs.

Unlike det_conn_close() (immediate teardown), this leaves the slot's PCB and callbacks live and moves it ACTIVE -> ConnState::CONN_CLOSING. The slot finalizes (PCB closed, slot freed) from the sent callback once the response has fully drained, or from the idle sweep after DETWS_CLOSING_TIMEOUT_MS if the peer never ACKs. The caller must already have queued (det_conn_send) + flushed the response. A no-op if the slot is not ConnState::CONN_ACTIVE (e.g. an error freed it mid-write).

Definition at line 572 of file tcp.cpp.

References CONN_ACTIVE, CONN_CLOSING, conn_pool, DET_OP_CLOSE_CHECK, detws_millis(), DETWS_OBS_TRANSITION, TcpConn::last_activity_ms, MAX_CONNS, TcpConn::pcb, and TcpConn::state.

Referenced by DetWebServer::http_poll_slot().

◆ det_conn_detach()

void det_conn_detach ( struct tcp_pcb * pcb )

Detach pcb from its slot's lwIP callbacks before the slot is freed.

Definition at line 516 of file tcp.cpp.

References DET_OP_DETACH.

Referenced by DeterministicAsyncTCP::check_timeouts(), det_conn_abort_slot(), det_conn_close(), and DeterministicAsyncTCP::stop().

◆ det_conn_abort()

void det_conn_abort ( struct tcp_pcb * pcb )

Hard-abort pcb (RST) for a fatal condition; no graceful FIN.

Definition at line 527 of file tcp.cpp.

References DET_OP_ABORT.

Referenced by DeterministicAsyncTCP::check_timeouts(), det_conn_abort_slot(), and DeterministicAsyncTCP::stop().

◆ det_conn_abort_slot()

void det_conn_abort_slot ( uint8_t slot )

Hard-abort connection slot (RST) for a fatal condition. The transport owns the teardown order: free the per-connection TLS context (abrupt, no close_notify), detach the pcb from its callbacks, reset the slot, then abort - so callers pass only the slot and never touch the raw tcp_pcb. A no-op if the slot has no live pcb.

Definition at line 496 of file tcp.cpp.

References CONN_ACTIVE, CONN_FREE, conn_pool, det_conn_abort(), det_conn_detach(), DETWS_OBS_TRANSITION, MAX_CONNS, TcpConn::pcb, TcpConn::state, and TcpConn::tls.

Referenced by DetWebServer::http_poll_slot().

◆ det_conn_remote_ip()

uint32_t det_conn_remote_ip ( uint8_t slot )

Raw source IPv4 of the connection in slot, or 0 if the slot has no active pcb (or on host builds). Byte order is irrelevant: this is an identity key (e.g. for the auth lockout), not for display. Keeps the lwIP pcb access inside L4 so callers never reach into the pcb directly.

Definition at line 655 of file tcp.cpp.

References CONN_ACTIVE, conn_pool, MAX_CONNS, TcpConn::pcb, and TcpConn::state.

◆ det_conn_remote_addr()

bool det_conn_remote_addr	(	uint8_t	slot,
		DetIp *	out
	)

The connected peer's address as a family-tagged DetIp (IPv4 or IPv6).

Unlike det_conn_remote_ip() (which flattens to a v4 uint32 and cannot represent a v6 peer), this reports the real address for a dual-stack build (DETWS_ENABLE_IPV6). Format it with det_ip_format() or classify it with det_ip_classify().

Returns: true if slot has an active connection whose address was written to out.

Definition at line 691 of file tcp.cpp.

References CONN_ACTIVE, conn_pool, DET_IP_NONE, det_lwip_to_detip(), DetIp::family, MAX_CONNS, TcpConn::pcb, and TcpConn::state.

◆ det_lwip_to_detip()

void det_lwip_to_detip	(	const ip_addr_t *	ra,
		DetIp *	out
	)

A stable per-peer 32-bit identity key for slot (the v4 address, or an FNV-1a hash of a v6 address). For rate-limit / auth-lockout buckets, where a v6 peer must not silently share the all-zero v4 bucket. Returns 0 if the slot has no active connection.

Convert a raw lwIP address to the portable family-tagged DetIp - for the accept callback, which has the pcb but no connection slot yet. ESP32 only (the parameter is an lwIP type).

Definition at line 673 of file tcp.cpp.

References det_ip_from_v4_octets(), and det_ip_from_v6_bytes().

Referenced by det_conn_remote_addr().

◆ lowlevel_recv_cb()

err_t lowlevel_recv_cb	(	void *	arg,
		struct tcp_pcb *	tpcb,
		struct pbuf *	p,
		err_t	err
	)

lwIP receive callback - wired to each new connection by listener_accept_cb.

See also: tcp.cpp

lwIP receive callback - wired to each new connection by listener_accept_cb.

Copies pbuf chain bytes into the ring buffer; the window is reopened later by det_conn_ack_consumed() as the worker drains (ack-on-consume), not here. If the whole segment will not fit it is refused (ERR_MEM) for lossless backpressure. A null pbuf signals graceful remote close (FIN received).

Definition at line 795 of file tcp.cpp.

◆ lowlevel_sent_cb()

err_t lowlevel_sent_cb	(	void *	arg,
		struct tcp_pcb *	tpcb,
		u16_t	len
	)

lwIP sent callback - refreshes the idle-timeout timestamp.

See also: tcp.cpp

lwIP sent callback - refreshes the idle-timeout timestamp.

Refreshes the idle-timeout timestamp so an active sender is not reaped while its responses are in flight, and - for a slot dwelling in ConnState::CONN_CLOSING - finalizes the close once the response has fully drained (the peer ACKed everything).

Definition at line 896 of file tcp.cpp.

◆ lowlevel_err_cb()

void lowlevel_err_cb	(	void *	arg,
		err_t	err
	)

lwIP error callback - fires when the stack detects a fatal error.

See also: tcp.cpp

By the time this fires the PCB is already gone internally, so we must NOT call tcp_close() or tcp_abort(). Null out the slot's pcb pointer and post EvtType::EVT_ERROR so the session layer resets the protocol state.

Definition at line 922 of file tcp.cpp.

◆ listener_enqueue()

bool listener_enqueue	(	uint8_t	listener_id,
		const TcpEvt *	evt
	)

Post evt to the queue owned by listener listener_id.

Called from tcp.cpp callbacks (running in tcpip_thread context) to deliver connection events to the session layer. Uses xQueueSend with timeout=0 - a full queue means the application is not calling server_tick() fast enough; the dropped event is recoverable via connection timeout.

Parameters

listener_id	Index into listener_pool[]; must be < MAX_LISTENERS.
evt	Event to copy into the queue.

Returns: true if queued; false if dropped (full queue / inactive listener).

Definition at line 291 of file listener.cpp.

References Listener::active, conn_pool, DETWS_WORKER_COUNT, detws_worker_wake(), listener_pool, MAX_LISTENERS, TcpConn::owner, Listener::queue, and TcpEvt::slot_id.

Variable Documentation

◆ detws_ap_ip

uint32_t detws_ap_ip

extern

softAP IPv4 address (network byte order) for STA/AP interface tagging.

Zero when no softAP is configured. Set via DetWebServer::set_ap_ip(); the accept callback tags each connection DetIface::DETIFACE_AP when its local IP equals this, else DetIface::DETIFACE_STA. Used by per-route interface filters.

Definition at line 349 of file tcp.cpp.

Referenced by DetWebServer::set_ap_ip().

◆ conn_pool

TcpConn conn_pool[CONN_POOL_SLOTS]