After ftl.add_host(hostname="web01", ...), the host is immediately addressable as ftl.web01 for attribute-style host-scoped module calls (e.g., ftl.web01.apt(...)).
The disablehostkeychecking=True parameter on ftl.addhost() skips SSH host key verification, essential for newly provisioned cloud VMs whose keys are unknown.
ftl.addhost(name, ansiblehost=, ansible_user=, groups=[]) dynamically registers a host at runtime.
ftl.add_host() accepts a groups parameter to assign the dynamically registered host to inventory groups.
add_host() persists to the state file immediately (not deferred to context exit).
ftl.addhost(hostname, ansiblehost, groups=[], **kwargs) registers hosts at runtime; extra kwargs become custom host variables.
Hosts are registered dynamically via ftl.addhost(hostname=..., ansiblehost=ip, ansibleuser=..., ansiblebecome=True) — this is how scripts add hosts discovered at runtime (e.g., after provisioning a Linode).
Sequential ordering in AI-loop action functions matters — for example, a user must be created before files can be owned by that user — and this is the author's responsibility, not automatically handled.
The AI loop combines resilient observation (shell guards, error suppression, fail-open semantics) with a self-healing contract (observe → condition → action) that supports convergence toward desired state. Together, these mechanisms enable autonomous remediation without requiring human intervention for observed drift.
AI-loop rule conditions can return False when prerequisite state is missing (e.g., IP not yet provisioned), causing the rule to silently skip rather than fail — this handles ordering between dependent rules.
The ftl2-ai-loop takes a declarative goal string as input and reconciles toward it using workspace rules.
AI-loop action functions use the ftl["hostname"].module() accessor pattern with await for sequential module calls (e.g., await ftl["stargate"].user(name="admin")).
The --incremental flag on ftl2-ai-loop enables incremental reconciliation rather than full re-deployment.
Module calls in AI-loop actions (user, file, copy, lineinfile, service) are idempotent — safe to re-run even if the condition gate fails to prevent unnecessary executions.
FTL2 AI-loop rules follow an observe → condition → action pattern: the loop runs observe to gather state, calls condition(state) to decide whether to act, and calls action(ftl) if the condition returns True.
AI-loop observe blocks are lists of dicts with name, module, params, and host keys that run commands on target hosts to gather state.
AI-loop observe steps are resilient by design: shell commands use || true to suppress non-zero exits, 2>&1 captures all output, and failed observations are treated as drift triggers rather than errors.
FTL2 AI-loop rules follow a three-phase contract: observe (gather state), condition(state) (check for drift), and action(ftl) (remediate) — the reconciliation loop calls these in order automatically.
AI-loop rules are not called directly by user code — they are discovered and executed by the FTL2 AI reconciliation loop automatically.
AI-loop rules implement a complete self-healing contract: observation failures trigger action (fail-open), guard clauses defer when prerequisites are missing, and actions apply full convergence regardless of which specific check detected drift.
State and inventory files in ftl2-deployments workspaces are not pre-created — they are generated at runtime by the AI loop.
AI-loop rules define a typed function contract across their three phases: observe is a declarative list of module dicts with name/module/params/host keys, condition receives a structured state dict keyed by observation name, and action receives the full FTL2 runtime with host accessors and module methods.
Each ftl2-deployments workspace has four key components: desired_state.txt (declarative intent), rules/ (AI loop constraints), .env (secrets/config), and runtime-generated state/inventory.
The AI reconciliation loop follows the pattern: observe → decide → act → verify; cost converges toward zero as deterministic rules replace AI reasoning.
Installing Ansible collections into the venv requires setting ANSIBLECOLLECTIONSPATH to the venv's site-packages directory (e.g., .venv/lib/python3.14/site-packages).
Ansible drops hosts as "unreachable" at 25+ hosts; FTL2's persistent gate connections handle them without issues.
FTL2 preserves Ansible familiarity at every interaction layer: CLI flags mirror -m -i -a, variable precedence follows group < host < argument ordering, inventory groups map directly to Ansible groups, and modules support both short names and FQCNs.
FTL2 enables gradual Ansible migration without workflow disruption: familiar CLI/inventory/FQCN patterns reduce learning curve while the universal module system (four addressing syntaxes, dual native/bundled execution modes) lets teams run existing Ansible collections alongside faster FTL2 natives incrementally.
Ansible teams migrating to FTL2 retain familiar patterns while immediately gaining production-grade deployment capabilities: gradual migration preserves CLI, inventory, FQCN, and variable precedence patterns, while state-driven reliability and security-first lifecycle provide enterprise-grade re-runnability and hardening from day one.
The Ansible-to-FTL2 converter maps: with_items to for loops, when to if statements, handlers to explicit service restarts, roles to function calls, hosts to group proxies, vars to Python variables.
Every re-execution in FTL2 is both safe and auditable: state-driven idempotency prevents resource duplication and enables crash recovery, while event streaming and policy audit trails capture exactly what happened in each run — failed re-runs leave evidence, successful ones prove convergence.
SSH public keys are deployed via ftl.host.ansible.posix.authorized_key() using the full FQCN pattern, supporting user/key/state parameters.
The automation() context manager accepts autoinstalldeps=True to automatically install required Ansible collections if they are missing.
The AutomationContext serves as a unified entry point (async context manager) that combines module access through four addressing syntaxes (dot, bracket, FQCN, proxy) with configurable error semantics (continue-on-error or fail-fast with introspectable exceptions), providing a cohesive developer interface for FTL2 automation.
automation(check_mode=True) enables dry-run mode that previews changes without applying them; output is annotated with [CHECK MODE].
automation() defaults to printsummary=True and printerrors=True, printing a per-host task summary with error details when the async context manager exits.
AutomationContext is the primary user-facing class in FTL2's automation framework, providing the async with automation() as ftl: pattern.
The automation() context manager is the entry point for all FTL2 remote operations, used with async with.
async with automation() as ftl: is the entry point for all FTL2 automation scripts, providing module access, inventory, secrets, and result tracking.
FTL2 scripts use async with automation(statefile=..., secretbindings=...) as ftl: as the core entry point — this context manager sets up inventory, state, and secrets.
AutomationContext is the primary user-facing class for FTL2 automation, providing an async context manager entry point, module access via attributes, secret binding and access, categorized result tracking (changed/ok/failed), and dual error handling modes (continue-on-error or fail-fast).
AutomationContext can be used both as a regular object (ctx = AutomationContext(...)) and as an async context manager (async with AutomationContext(...) as ftl).
FTL2's default behavior is fail_fast=False — a failing task does not halt subsequent tasks in the same automation context.
When no inventory is specified, automation() defaults to targeting localhost.
AutomationError has message (str) and result (the failed ExecuteResult) attributes, enabling granular error inspection per host and module.
The automation() function is an async context manager factory that wraps AutomationContext — all real logic lives in AutomationContext.
automation(fail_fast=True) raises AutomationError on first module failure; the exception has .result.module to identify the failing module.
FQCN modules use NamespaceProxy chaining: ftl.amazon.aws.ec2instance(...) chains NamespaceProxy("amazon") → .aws → NamespaceProxy("amazon.aws") → .ec2instance() executes "amazon.aws.ec2_instance".
The inventory parameter on automation() loads a YAML inventory file for host/group definitions, used alongside add_host() for hybrid static+dynamic inventory.
automation(modules=["file", "copy"]) restricts which modules can be called; accessing a disallowed module raises AttributeError, not a runtime module error.
Modules are called as attributes on the automation context: await ftl.module_name(param=value) — no string lookups or dictionaries needed.
automation(modules=["file", "copy"]) restricts which modules are available; calling an unlisted module raises AttributeError.
automation(quiet=True) suppresses all console output during execution. Combined with fail_fast=True, this is the standard pattern for programmatic/embedded usage (TUI, watchdog, CI).
Task results in the automation context are categorized as changed, ok, or failed.
automation(secrets=["KEYNAME", ...]) binds secrets for access via ftl.secrets["KEYNAME"] within module calls.
Accessing a secret not in the requested list raises KeyError: not requested; a requested but unset environment variable raises KeyError: not set in environment — two distinct error cases.
ftl.secrets values are never exposed in print() or repr() — this is a security feature of the secrets proxy.
automation(verbose=True) enables verbose/debug output for the automation session.
AI-loop operations are resilient at every layer: coordinated cross-rule self-healing recovers from drift via observe/condition/action with state-file-mediated communication, while dual error modes (continue-on-error for collection, fail-fast for halt) capture structured errors alongside state persistence for crash recovery.
FTL2 enables autonomous self-healing infrastructure applications: durable AI-loop remediation (observe/condition/action with fail-open, persistent state for crash recovery) operates within a full application framework (long-lived daemons, real-time monitoring via htop, event/gate/SSH pipeline).
Azure authentication uses a service principal configured via four environment variables: AZURECLIENTID, AZURESECRET, AZURESUBSCRIPTIONID, and AZURETENANT.
The Azure examples use .ftl2-state-azure.json as the state file for tracking provisioned resources.
FTL2 achieves operational durability through complementary persistence mechanisms: the state file tracks resource identity and enables crash recovery (control plane), while the backup subsystem preserves file contents before destructive changes (data plane) — together they protect both what exists and what it contains.
FTL2's backup subsystem is comprehensive and safe by default: automatic backups are enabled for destructive operations, two-phase protocol (discover paths, then create backups) ensures correctness, two storage modes (adjacent and central) provide deployment flexibility, consistent naming convention enables discovery, and fail-safe semantics abort operations rather than proceeding without a backup.
If backup creation fails, FTL2 aborts the destructive operation rather than proceeding without a backup (fail-safe behavior).
FTL2 backup files are named {original_path}.ftl2-backup-{YYYYMMDD}-{HHMMSS}.
Backup discovery uses game-type-specific paths: ~/data/minecraft/backups/ for Minecraft/NeoForge and ~/data/terraria/backups/ for Terraria, sorted newest-first by mtime.
FTL2 backups use a two-phase protocol: Phase 1 discovers paths needing backup (ftl2discoverbackups: true), Phase 2 executes with ftl2backupscreated metadata attached.
FTL2 backups support two storage modes: adjacent (default, next to original file) and central (configured via backup.central_dir or --backup-dir).
FTL2 automatic backups are enabled by default for destructive module operations; use --no-backup to disable them.
When privilege escalation is active, copy SFTPs files to /tmp/.ftl2copy* then uses sudo mv to place them at the destination.
FTL2 supports becomeuser parameter on module calls to run operations as a specific unprivileged user rather than root — e.g., becomeuser=SERVICE_USER for uv installs.
Bracket notation ftl["hostname"] supports host names with dashes that cannot be used with dot-access syntax.
FTL2 builtin modules (file, copy, shell, command, service, dnf) run in-process as native implementations; collection modules fall back to subprocess execution.
buildbundle() produces bundles with info.contenthash; stagebundleremote() uses this hash for deduplication — skipping re-upload if the hash matches what's already staged.
Bundled Ansible modules running inside FTL2 can import ftl2.events and emit events using emitprogress(), emitlog(), emit_data() — this is an FTL2-specific extension, not standard Ansible.
The Caddy rule's action phase is idempotent: copy overwrites the Caddyfile, firewalld with state="enabled" is a no-op if already enabled, and service with state="started" is a no-op if already running.
The catbeez-arcade application binds to 127.0.0.1:8000 (localhost only) and is not directly accessible from the internet.
Catbeez-arcade deployments use Caddy as a reverse proxy, terminating TLS on :443 and forwarding to the application on localhost:8000.
Catbeez-arcade Cloudflare DNS records are set to DNS-only (not proxied), so TLS is handled by Caddy/Let's Encrypt rather than Cloudflare.
Catbeez-arcade secrets (OAuth credentials, secret key, allowed emails) are entered interactively at startup rather than stored in automation or config files.
Firewall port 80 must be opened for Let's Encrypt ACME challenges even though traffic is served on port 443.
Catbeez-arcade uses a private Linode image private/37121878 (Fedora 43 with Caddy pre-installed) to skip base configuration during provisioning.
Catbeez demonstrates an FTL2 deployment pattern that combines three production concerns: layered security (localhost binding, Caddy TLS, firewalld drop zone, SSH source-IP restriction, SELinux), DNS-to-TLS automation (Cloudflare DNS-only records enabling Caddy ACME challenges), and hot-reload publishing (HTML5/WASM asset upload with dynamic discovery, no restart required) — illustrating how these concerns compose in a web application hosting scenario.
The catbeez dev deployment automates DNS via community.general.cloudflare_dns module, unlike the prod deployment which uses manual Namecheap DNS.
Catbeez deployments use firewalld's drop zone, which silently discards all uninvited traffic, with explicit allowances for HTTP, HTTPS, and restricted SSH.
Catbeez game assets consist of three file types: .html, .js, and .wasm — these are browser-based HTML5/WASM games.
Catbeez game publishing is a hot-reload pipeline: HTML5/WASM game assets are uploaded to the games directory, and the server discovers new files dynamically without restart, with the same publish script supporting both dev and prod environments via flag selection.
Catbeez deployments implement layered security: application binds localhost-only, Caddy terminates TLS and reverse-proxies, firewalld drop zone silently discards uninvited traffic, SSH is restricted to a source IP range, and SELinux enforces network connect policy.
The catbeez production deployment uses become_user='catbeez' to run application installation as the service user rather than as root.
Game files are uploaded individually (3 files per game × 9 games = 27 copy calls) rather than as an archive.
The catbeez production deployment requires manual DNS A record creation at Namecheap — DNS is not automated in the prod deploy script.
The catbeez production deployment restricts SSH access to 136.56.0.0/16 via firewalld; HTTP/HTTPS are open to all.
publish-games.py supports dev and prod environments: dev uses state.json and hostname arcade; prod (via --prod flag) uses state-prod.json and hostname catbeez-prod.
The catbeez-arcade server discovers new game files dynamically — publishing games is purely a file upload to /home/catbeez/games/ with no application restart required.
The publish-games script reads an existing state file to extract host connection info — the state file must already exist from a prior provisioning run.
Catbeez deployments set the httpdcannetwork_connect SELinux boolean so Caddy can reverse-proxy to localhost:8000 under SELinux enforcing mode.
automation(check_mode=True) enables dry-run mode where operations report what would change without executing.
Not all modules fully support check mode — some may still create files or execute commands despite check mode being enabled.
ftl.results collects results from all operations even in check mode, enabling post-run analysis.
The recommended workflow for critical operations is validate-then-execute: run check mode first, inspect for failures, then execute for real only if validation passes.
The FTL2 CLI provides Ansible-familiar syntax with rich targeting capabilities: the -m -i -a flag pattern mirrors Ansible, shlex parsing handles quoted arguments correctly, the --limit flag supports group names, host names, glob patterns, and ! exclusion, and three run modes (normal, check, teardown) cover the full lifecycle.
FTL2 supports a provision-then-configure workflow within a single run: add_host() dynamically registers a newly created host with connection parameters and group assignment, and a two-phase bootstrap pattern (root → admin re-registration) can harden access before applying configuration. State persistence across these steps supports crash recovery and re-runs.
FTL2 deployments use a consistent Cloudflare+Caddy TLS pattern: Cloudflare DNS records are set to DNS-only (not proxied) so Caddy can perform Let's Encrypt ACME challenges on port 80 and terminate TLS directly, avoiding certificate conflicts between Cloudflare and the origin server.
The Cloudflare DNS module is accessed via ftl.community.general.cloudflare_dns() with parameters: zone, record, type, value, proxied, state.
When using Caddy for automatic TLS via Let's Encrypt, Cloudflare DNS must be set to DNS-only mode (not proxied) so Caddy can perform its own ACME HTTP-01 challenge on port 80.
Cloudflare DNS records are created with TTL 300 seconds and proxied: False (DNS-only), and these settings are hardcoded, not configurable via parameters.
The Cloudflare DNS module uses only Python stdlib (urllib.request) with no external HTTP dependencies and no FTL2 framework imports.
The updatednsrecord function is idempotent: it creates the record if missing, updates it if the IP changed, and skips the API call entirely if the record already matches.
The updatednsrecord function is the single public entry point for Cloudflare DNS management in ftl2-servercraft.
Zone lookup extracts the root domain from the hostname by taking the last two domain parts (e.g., world1.servercraft2.com → servercraft2.com).
Command execution in FTL2 is governed at two independent levels: the policy engine enforces pre-execution access control (deny rules matching module, host, environment, and parameters with first-match semantics), while the command/shell distinction controls injection exposure at runtime (command runs without shell interpretation, shell enables pipes but triggers the same policy deny as command and raw).
FTL2's command execution model balances power with safety: command runs without shell interpretation (injection-safe), shell enables pipes and redirects (more powerful), creates provides idempotency for both, and the policy engine treats command/shell/raw as equivalent — denying any one blocks all three to prevent circumvention.
Community modules are called on the ftl object directly (not on a host) — e.g., ftl.community.general.linodev4(...), ftl.community.general.cloudflaredns(...) — because they target external APIs, not a specific managed host.
Compiled dependencies (C extensions like psutil) cannot be bundled in the gate .pyz package; they must be pre-installed on the remote host via dnf/apt. Pure-Python deps go in the .pyz bundle.
FTL2 provides continuity from development to production: the same automation() context manager, module addressing, and error handling API used during development also drives production deployments — production scripts add parameters like statefile, failfast, and secret_bindings but use the same core API surface, with no separate deployment-specific interface required.
FTL2 provides a three-tier execution feedback pipeline: consistent result objects (success, changed, output, error) provide synchronous control-flow data, configurable display modes (errors-only through verbose timing) shape human-readable output, and parallel event streaming (progress, log, status) delivers structured data for programmatic observability.
FTL2 combines structured error handling with execution feedback and state persistence to support failure recovery: dual error modes (continue-on-error and fail-fast) capture structured error information, a three-tier feedback pipeline (result objects, display modes, event streaming) provides both control-flow data and programmatic observability, and state persistence allows subsequent runs to resume from where a failure occurred rather than starting over.
FTL2 provides a unified pipeline from host selection to module execution: the CLI's rich pattern-based targeting (groups, globs, exclusions) feeds into flexible multi-format inventory resolution, which feeds into four module addressing syntaxes — creating a seamless path from "which hosts" to "what action."
FTL2 achieves comprehensive crash resilience through three independent mechanisms: persistent state files enable re-run recovery from any interruption, automatic backups protect filesystem changes with fail-safe abort, and dual error modes capture failures without halting — every failure path leads to a recoverable state.
FTL2 provides comprehensive operational observability across automation and infrastructure: the event streaming subsystem captures structured module execution data (progress/log/status events always in results), while ftl2-htop delivers real-time system metrics (CPU, memory, network) via SSH-gate-psutil pipeline — together covering both application-level task tracking and system-level resource monitoring.
FTL2 provides layered security from credential storage to network edge: the secrets management spectrum (env bindings, Vault integration, lazy evaluation, defense-in-depth practices like redaction and never-logging) complements a layered deployment security model (two-phase bootstrap, firewalld drop zone, SELinux, SSH restriction, Cloudflare DNS-only, Caddy TLS termination), supporting credential protection across storage, transit, and serving layers.
The statepath, inventorypath, and configdirpath properties all call expanduser(), so ~ is valid in configuration values.
AI-loop rules achieve coordinated multi-rule self-healing: the three-phase contract (observe/condition/action with fail-open and guard clauses) operates on shared persistent state, enabling rules to read each other's outputs, defer when prerequisites are missing, and converge toward a complex desired state that no single rule could reach alone.
The native copy module uses asyncio.gather to transfer files to all hosts in a group concurrently.
FTL2 combines crash recovery and error-resilient AI-loop operations to support autonomous recovery from many failure scenarios without manual intervention: three independent crash recovery mechanisms (persistent state files for re-run recovery, automatic backups with fail-safe abort, and dual error modes for failure capture) complement the AI-loop's self-healing capabilities (observe/condition/action with state-file-mediated coordination and structured error collection), providing multiple recovery paths for both unplanned crashes and runtime errors.
The AI loop achieves durable autonomous remediation: self-healing observation/condition/action with fail-open semantics feeds into persistent state that survives process crashes, enabling the reconciliation loop to resume convergence from where it left off rather than starting over.
FTL2 provides crash-resilient error management: dual error modes (continue-on-error for collection, fail-fast for immediate halt) capture structured error information, while state persistence ensures that after any failure — whether collected or raised — the next run can resume from persisted state rather than starting over.
Declarative resource planning uses topological sort for parallel execution waves and reverse order for teardown.
FTL2's default behavior is continue-on-error: all operations execute regardless of individual failures, with errors collected and inspectable after the run.
Default output mode shows nothing for successful operations — only errors surface.
The default state file path is .ftl2-state.json; it enables crash recovery by persisting dynamically added hosts.
FTL2 deployments achieve defense-in-depth from host bootstrap to network edge: two-phase registration hardens host access before content deployment, layered security (firewalld drop zone, SELinux, SSH restriction) protects the host perimeter, and Cloudflare DNS-only + Caddy TLS secures the network edge with Let's Encrypt ACME.
Credentials are integrated into the security-first deployment lifecycle: defense-in-depth prevents exposure at every surface (bindings, repr, logs) while security-first ordering ensures secrets only flow through hardened, bootstrapped channels — the two-phase registration hardens access before any credentials are injected.
FTL2 deployments follow an incremental verification pattern: each layer (DNS, TLS, services) is confirmed working before proceeding to the next.
FTL2 production deployments can follow a security-first lifecycle from bootstrap to runtime: two-phase host registration hardens access before content deployment, and runtime protection layers (such as localhost binding, reverse proxy TLS termination, firewalld drop zone, SELinux booleans, and SSH IP restrictions) provide defense-in-depth at network boundaries, as demonstrated in the Catbeez deployment pattern.
In the desired-state-as-markdown model, scripts are disposable and regenerated — the state file is ground truth, not the scripts.
FTL2 bridges the complete spectrum from developer onboarding to autonomous production operations: zero-setup scripts with familiar APIs and full observability enable rapid development, which seamlessly transitions to autonomous self-healing production through the same unified API surface — the identical async context manager, module addressing, and state management work at every stage of the journey.
Direct module calls (ftl.file(...)) execute locally and return a dict; ftl.run_on() is required for remote/multi-host execution and returns a list of result objects with .host, .success, .changed, .output.
FTL2 serves as a hosting platform for diverse application types: long-lived daemon contexts with event/gate/SSH infrastructure support game server lifecycle management (servercraft with watchdog, TUI, backup), while the same provisioning pipeline supports hot-reload web applications (catbeez arcade) where publishing is purely file upload with no restart.
FTL2 minimizes learning investment for both new and migrating users simultaneously: newcomers start immediately with PEP 723 single-file scripts that self-bootstrap via uv run with no project scaffolding, while Ansible users recognize CLI flags (-m -i -a), variable precedence (group < host < argument), inventory groups, and FQCN module naming.
FTL2 offers two complementary onramps to production-grade automation: greenfield developers can start with zero-setup PEP 723 scripts and progress to optimized cloud execution without intermediate tooling, while Ansible teams can migrate gradually by retaining familiar patterns (CLI, inventory, FQCN, variable precedence) and immediately gaining state-driven reliability and security-first lifecycle features. Both paths lead to production-capable deployments, though the specific observability and security characteristics available depend on which features each path exercises.
ftl.add_host() provides a complete dynamic host lifecycle: runtime registration with connection parameters and group assignment, immediate state persistence, and instant attribute-style addressing (ftl.hostname) — enabling provisioning scripts to create infrastructure and configure it in a single run.
FTL2's unified inventory model means dynamically provisioned hosts can receive the same security treatment as static infrastructure: add_host registration integrates runtime-discovered hosts into the same addressing model where security-first lifecycle patterns — such as two-phase bootstrap, security-before-content ordering, and layered network hardening — can be applied consistently, rather than requiring separate or reduced security workflows for dynamic hosts.
FTL2 deployments can implement a layered edge-to-application security perimeter. In observed patterns, Cloudflare DNS-only mode delegates TLS to Caddy (enabling Let's Encrypt ACME), Caddy terminates HTTPS and reverse-proxies to a localhost-only application, firewalld's drop zone silently discards uninvited traffic, SSH is restricted to source IP ranges, and SELinux enforces network connect policies. This pattern has been documented in catbeez deployments and the Cloudflare+Caddy TLS approach is used consistently across FTL2 deployments.
FTL2 optimizes remote execution end-to-end from connection to result: the SSH layer pools and reuses connections via asyncssh for async non-blocking I/O, while the gate layer pre-builds modules into cached zipapps that communicate via length-prefixed JSON — eliminating overhead at both the transport and module delivery layers.
FTL2 provides two error handling modes: continue-on-error (default) collects all failures in ftl.errors for post-run analysis, while fail-fast raises AutomationError on first failure with module identification via the .result attribute.
Event callbacks passed via on_event fire synchronously during execution — the callback is a plain function, not a coroutine.
FTL2 supports event-driven application architectures: the complete event streaming subsystem (three event types, always-captured results, structured callbacks) combined with the listen/gather concurrency pattern enables long-running applications to react to infrastructure events in real-time alongside application logic.
Event handlers are registered via proxy.on(event_type, handler); both sync and async handlers are supported, and multiple handlers per event type are allowed.
EventProgressDisplay requires the Rich library; SimpleEventDisplay has no Rich dependency.
FTL2 provides an event streaming subsystem with three event types (progress, log, data) for structured observability. Events are always captured in results regardless of callback registration. Structured dict payloads with fields like event, module, host, success, changed, duration, and timestamp enable programmatic consumption. Handler registration via proxy.on() supports both sync and async handlers, with multiple handlers per event type.
FTL2's event streaming system has three event types: progress (percent/bytes tracking), log (leveled messages: info/warning/error), and data (raw stdout/stderr streams).
FTL2's event streaming serves dual purposes that reinforce each other: operational observability (structured module events captured in results plus ftl2-htop real-time monitoring) AND application-level event-driven control (ftl.listen() with asyncio.gather for concurrent event processing), using the same three event types for both.
Events are always captured in result.events regardless of whether an event_callback is registered — callbacks are optional.
Modules emit events as JSON objects to stderr, one per line. Stdout is reserved for the final result JSON — events on stdout would corrupt the result.
FTL2 provides a dual-mode execution architecture: native in-process modules for speed and Ansible-compatible bundled modules for ecosystem breadth, with the same API surface for both.
Every FTL2 module execution from CLI target selection through result delivery is both secured and fully instrumented: rich pattern-based targeting feeds through policy enforcement, credential injection, and privilege escalation controls, while structured events, consistent result contracts, and flexible output modes provide complete execution telemetry.
By default (failfast=False), errors are collected in ftl.errors and execution continues; failfast=True raises AutomationError on first failure.
automation(fail_fast=True) raises AutomationError immediately on the first module failure, stopping all subsequent operations.
When an AI-loop rule's observe call itself fails (e.g., API error), the condition returns True, treating observation failure as "needs fix" — a defensive pattern.
FTL2 fleet operations are simultaneously fast and resilient: SSH connection pooling with async I/O and gate caching minimize per-host overhead, default parallelism bounds execution to the slowest host, and failure isolation ensures fleet execution continues and collects structured errors even when individual hosts fail.
The FTL2 file module uses state=touch to create files and state=absent to remove them.
FTL2 provides access to the ansible.posix.firewalld module for managing firewall rules (services, ports, zones) via the FQCN pattern.
FQCN module names chain through the host-scoped proxy — e.g., ftl.webservers.ansible.builtin.command(cmd=...) works correctly.
FTL2 supports Ansible-style fully qualified collection names (FQCN) for modules, e.g., ftl.local.ansible.builtin.ping().
FQCN modules (e.g., amazon.aws.ec2_instance) require the corresponding Ansible collection to be installed to function.
Developers get a frictionless full-ecosystem experience: zero-setup PEP 723/uv scripts eliminate project scaffolding while the dual-mode architecture provides native speed and Ansible ecosystem breadth, all accessed through a unified async context manager with four addressing syntaxes and dual error handling.
Ansible module compatibility in FTL2 requires the ftlbuiltinmodules and ftlmoduleutils packages to be installed.
Some Ansible modules (e.g., tempfile.py) shadow Python standard library modules, causing failures outside the full Ansible runtime — this is a known limitation.
buildbundlefromfqcn packages an Ansible module with its dependencies into a transferable bundle with fqcn, contenthash, size, and dependency_count attributes.
The creates parameter on ftl_command skips execution if the specified file already exists, providing idempotency for command execution.
ftlcommand runs commands directly without shell interpretation; ftlshell runs through a shell and supports pipes (|) and environment variable expansion ($HOME).
ftl_copy with backup=True preserves the previous version of the destination file before overwriting.
FTL2 has eight built-in native modules: file, copy, template, command, shell, uri, get_url, and pip.
FTL2 module errors don't raise exceptions — they are captured in result objects with success=False and an error message.
execute_batch provides near-linear speedup over sequential execute() calls by running tasks concurrently via asyncio.
executebatch accepts tasks as a list of (modulename, paramsdict, hostor_none) tuples.
executelocalfqcn(fqcn, params_dict) executes an Ansible module locally by its fully-qualified collection name, returning an ExecutionResult.
ExecutionResult has four fields: success (bool), output (dict), error (str|None), and changed (bool).
The Executor API (execute/run/executeonhosts) is the recommended API for FTL2 module execution, not direct function calls.
The Executor API returns result objects with .success, .changed, and .output attributes, while raw ftl_* functions return plain dicts with a changed key.
ftl.failed is a bool indicating if any operation failed; ftl.errors returns a list of failed ExecuteResult objects; ftl.error_messages returns just the error strings.
The fetch module copies files FROM remote hosts TO the control machine — the reverse of copy. Used for downloading backups and logs.
The get_url module downloads files from URLs to remote hosts, used for fetching server JARs, installers, and other artifacts during provisioning.
Modules are called on hosts via ftl["hostname"].module_name(params...) syntax — e.g., ftl["arcade"].shell(cmd="..."), ftl["arcade"].copy(src=..., dest=...), ftl["arcade"].file(path=..., state="directory").
ftl.listen() blocks and dispatches events; it must run concurrently with application logic via asyncio.gather(ftl.listen(), app_coroutine()).
ftl.local is used for API/cloud modules (equivalent to Ansible's connection: local).
FTL2 applications use ftl.local.community.general.slack() for sending Slack notifications from the control machine, wrapped in try/except to prevent notification failures from aborting infrastructure operations.
LocalHost(name=...) creates a local execution target that enables the same API pattern used for remote hosts (executeonhosts) to work locally.
The execute() and run() functions accept short module names (e.g., 'file', 'command') which map internally to the corresponding ftlfile/ftlcommand functions.
FTL modules run as native Python functions inside the same process, eliminating subprocess overhead (fork/exec per task).
FTL modules are sent to the gate by name (gate already has them); Ansible modules are sent as bundles.
FTL modules achieve ~330x speedup for file operations, ~84x for HTTP/uri operations, and ~7x for command execution compared to subprocess-based execution.
FTL2 modules track idempotency via changed status — operations like touching an existing file or copying identical content return changed=False.
The ftl object is passed as the first argument to all script interface functions — it is the primary handle for FTL2 automation operations within an application.
Remote execution via SSHHost falls back to Ansible module bundling when using the Executor — it does not run FTL functions in-process on the remote side.
resolvefqcn maps dotted Ansible FQCN strings (e.g., ansible.builtin.command) to filesystem paths via findansiblebuiltinpath().
ftl.results provides a list of module results, each with .success and .changed attributes.
The automation context tracks all module executions in ftl.results — each entry has .success, .changed, and .module attributes.
FTL2 provides a swap module for configuring swap space on remote hosts — used in game server provisioning where memory-intensive Java processes need swap.
FTL2 provides two API surfaces: direct synchronous function calls (ftlfile(), ftlcommand(), etc.) and the async Executor API (execute, run, executeonhosts).
ftl.addhost() registers a dynamically created server as an inventory host mid-run with connection parameters (ansiblehost, ansible_user, group membership), enabling provisioning and configuration in the same script.
The AI-loop action(ftl) function receives the FTL2 runtime, providing host accessors and module methods to perform remediation.
The AI-loop condition(state) function receives observed state as a dict keyed by observation step name, where each value contains fields like stdout.
In AI-loop rules, if an observe step fails (e.g., API error), the condition function can treat the failure as drift and return True, triggering the action to re-establish the resource.
AI-loop action functions apply all fixes (e.g., config, firewall, service) regardless of which specific condition check failed — a full convergence approach rather than targeted remediation.
AI-loop condition functions can implement guard clauses that return False when prerequisite state doesn't exist yet, deferring to other rules that establish that state first.
FTL2 AI-loop rules follow a three-part observe/condition/action pattern: observe gathers current state, condition returns True if drift is detected, action converges to desired state.
AI-loop observe steps can use the shell (or command) module with cmd and host parameters to run arbitrary commands for state gathering.
AI-loop rules can be auto-generated by the ftl2-ai-loop system, though they may contain artifacts like duplicate observe blocks.
AI-loop rules can read state populated by other rules via state.statefile.resources, enabling cross-rule data flow (e.g., DNS rule reads IP from Linode provisioning rule's state).
FTL2 serves as a full application framework beyond one-shot automation: long-lived daemon contexts support persistent services, the event/gate/SSH pipeline enables real-time TUIs, and persistent state orchestrates multi-run workflows — demonstrated by servercraft (game lifecycle manager with watchdog and TUI dashboard) and ftl2-htop (real-time infrastructure monitoring).
FTL2 applications can be single-file scripts using PEP 723 inline script metadata for dependency declarations, runnable via uv run with no virtualenv setup.
The automation() context manager accepts autoinstalldeps=True to automatically install missing Ansible collections at runtime.
The core FTL2 usage pattern is the automation() async context manager, which accepts parameters like statefile and secretbindings.
FTL2 inventory always has two built-in groups: all (every host) and ungrouped (hosts not in any explicit group).
FTL2 calls Ansible modules in-process rather than as subprocesses — a core architectural difference from Ansible.
When check_mode=True, FTL2 scripts should skip post-provision configuration (SSH wait, package install, service setup) since resources don't actually exist in dry-run mode.
The FTL2 CLI uses the flag pattern -m <module> -i <inventory> -a "<args>", mirroring Ansible's CLI syntax.
The FTL2 CLI uses Python's shlex module for proper shell-style argument parsing, handling quoted strings like cmd='echo hello'.
When using Caddy for TLS termination, Cloudflare DNS records must be set with proxied=False to avoid certificate conflicts between Caddy and Cloudflare's proxy.
Community modules are called via the ftl.community.general.*() accessor pattern using FQCN-style naming.
FTL2 is a complete production platform: high performance without sacrificing ergonomics (async context manager, flexible addressing, dual error modes) combines with reliable and secure deployments (state-driven crash recovery, security-first lifecycle, two-phase bootstrap).
The destroy script deletes the state file after a successful destroy or when the instance is already gone.
The Linode destroy script finds instances by matching the label against all instances from GET /linode/instances, not by stored instance ID.
The destroy script uses next(iter(resources.values())) and only processes the first resource in the state file, ignoring any others.
Error checking in FTL2 uses ftl.failed (boolean flag) and ftl.errors (structured error objects with .module and .error fields) — not exceptions by default.
In multi-host execution, total execution time approximates the slowest host rather than the sum of all hosts.
ftl.failed is a property on the automation context that indicates whether any module call failed during the session, not a per-call return value.
When some hosts fail during multi-host execution, FTL2 continues executing on the remaining hosts.
FTL2 ships five built-in modules: ping (connectivity), setup (fact gathering), shell (command execution), file (file/directory management), and copy (file copying), located in src/ftl2/modules/.
FTL2's speed advantage derives from three independent optimization sources: in-process native module execution (up to 250x per-call speedup for native modules), gate-based remote delivery (pre-built zipapps cached on remote hosts, JSON-only parameter protocol), and default parallel host targeting (asyncio.gather fan-out, total time bounded by slowest host).
FTL2 scripts achieve fully zero-configuration deployment: inline PEP 723 metadata declares dependencies and uv run bootstraps everything.
The gate zipapp is built once and uploaded to all targeted hosts, not rebuilt per host.
GCP authentication in FTL2 uses three environment variables: GCPPROJECT (project ID), GCPAUTHKIND (auth type, e.g., serviceaccount), and GCPSERVICEACCOUNTFILE (path to service account JSON key).
HCLOUDTOKEN is injected into Hetzner modules via secret bindings (e.g., {"hetzner.hcloud.*": {"apitoken": "HCLOUD_TOKEN"}}), ensuring the token is never logged or visible in scripts.
The Hetzner Cloud collection is installed via ansible-galaxy collection install hetzner.hcloud, not pip.
The Hetzner state file is named .ftl2-state-hetzner.json, following a .ftl2-state-<provider>.json naming pattern.
Host groups in FTL2 correspond to Ansible inventory groups and determine which hosts receive module calls.
During initial provisioning, a host can be registered first as root for user creation, then re-registered as a non-root user with ansible_become=True to bootstrap sudo access.
Hosts are addressed by name via ftl["hostname"] and modules are called as async methods (e.g., .copy(), .shell(), .service(), .file(), .user(), .wait_for()).
ftl2-htop data flow: SSH connection → gate process → SystemMonitor (psutil) → SystemMetrics events → SSH channel → TUI rendering.
The --debug flag in ftl2-htop bypasses the TUI and prints raw gate events to stderr, useful for understanding the event protocol.
Group filtering via -g in ftl2-htop works the same way as in standard FTL2 task execution.
Inventory resolution priority in ftl2-htop: positional hosts → --inventory → --state → fallback to "localhost,".
ftl2-htop supports four distinct output modes for different use cases: native Textual TUI for terminal monitoring, textual-serve for browser-based access, WebSocket broadcasting for custom dashboards and integrations, and debug mode for raw event inspection during development.
ftl2-htop requires only python3-psutil installed on remote hosts; FTL2's gate handles everything else via SSH.
ftl2-htop is a complete real-time infrastructure monitoring stack: SSH→gate→psutil data pipeline, dual-thread architecture (Textual UI + FTL2 daemon), 30-sample sparkline history, group-based host filtering, and browser accessibility via textual-serve — all reusing standard FTL2 inventory and gate subsystems.
ftl2-htop maintains a 30-sample rolling history per host for CPU, memory, and network rates, rendered as Unicode sparkline characters (▁▂▃▄▅▆▇█).
The Textual TUI mode (--tui flag) is compatible with textual-serve for browser-based rendering.
The Textual TUI mode uses a dual-thread architecture: Textual event loop on the main thread, FTL2 automation on a daemon thread with its own asyncio event loop; cross-thread updates use callfromthread.
ftl2-htop supports WebSocket broadcasting via --ws-port; it pushes the full metrics store as JSON to all connected clients every 2 seconds.
FTL2 applications are installed using pip install "package @ git+https://github.com/..." URLs to install directly from GitHub repositories.
FTL2 inventory uses ansiblehost, ansibleport, ansibleuser, ansibleconnection, and ansiblepythoninterpreter as host connection parameters.
FTL2 uses flattened (not nested) group structures in its inventory system.
The --limit flag supports specific groups, individual hosts, glob patterns (web*), exclusion patterns (!db*), and comma-separated lists.
The --limit flag supports group names, host names, glob patterns (e.g., web*), and exclusion with ! prefix (e.g., !databases).
The community.general.linode_v4 module with state: present is idempotent — if the instance already exists and matches the spec, it returns the existing instance without changes.
FTL modules execute as in-process Python, claimed to be 250x faster than subprocess execution used by traditional module invocation.
FTL2 native modules run in-process rather than via subprocess execution, claimed to be 250x faster than subprocess-based execution.
FTL2 inventory variables do NOT use the ansible prefix — use sshprivatekeyfile instead of ansiblesshprivatekeyfile. Only fields starting with ansible_ are treated as direct host attributes; other variables go into a vars dict.
AI-loop observe steps append || true to shell commands to prevent non-zero exit codes from being reported as observation failures, allowing the condition function to interpret missing state gracefully.
FTL2 executes modules concurrently across all targeted hosts by default, with no flag needed to enable parallelism.
FTL2's performance optimizations and developer ergonomics features are complementary aspects of the same system: the three-tier execution stack (in-process, gate, parallel) provides the performance foundation, while the async context manager, flexible module addressing, and dual error modes provide the usability layer. Because both operate within the same architecture, users can benefit from performance optimizations without needing to adopt a different API or workflow for speed-sensitive tasks.
FTL2 action functions can install Python packages from GitHub repositories using pip's git+https:// URL syntax via the .shell() module.
FTL2 requires Python 3.11 or higher.
ftl.results collects all operation outcomes from a session; ftl.errors provides error introspection with module name and error message.
FTL2 example scripts are run with uv run python — FTL2 uses uv as its Python runner.
FTL2's secret_bindings parameter maps module FQCNs to environment variable names, auto-injecting secrets into specific modules at runtime.
FTL2 secret bindings map module FQCN names to environment variable names (e.g., {"community.general.linodev4": {"accesstoken": "LINODE_TOKEN"}}) for automatic secret injection.
Secret bindings support wildcard namespace patterns (e.g., "google.cloud.*") to inject environment variables as parameters to all modules in that namespace, avoiding repetition per call.
FTL2 deployments apply security hardening (firewall, SSH lockdown, SELinux, crypto policies, fail2ban) before any application content is deployed, so failed hardening prevents app exposure.
FTL2 is a self-sufficient production infrastructure platform: autonomous self-healing (AI-loop remediation with persistent state and crash recovery) operates within a complete production framework (high-performance execution, ergonomic API, secure reliable deployments), enabling infrastructure that converges toward desired state without human intervention.
FTL2 deployment scripts use shell guards like which uv || (install uv) and grep -qF ... || echo ... to make shell commands safe to re-run, though heavy shell use reduces idempotency guarantees compared to dedicated modules.
Builtin modules use short names (file, copy, shell); collection modules use fully qualified collection names (e.g., community.general.linode_v4, ansible.posix.firewalld).
FTL2 achieves 2-21x speedup over ansible-playbook, depending on workload and module mix.
FTL2's 2-21x speed advantage derives from three independent sources: in-process module execution (up to 250x per-call for native modules), default parallel host targeting, and gate pre-building that eliminates per-task bundling overhead.
FTL2 achieves 2-21x speedup over Ansible; API workloads show the largest gains (up to ~21x at 25 hosts).
FTL2 modules use state="absent" to delete resources — the same module handles both creation and removal.
ftl.state.add() persists resource metadata to the state file and ftl.state.remove() cleans it up, enabling idempotent re-runs and cross-run resource tracking.
FTL2's state API provides ftl.state.add(key, dict) to persist resource metadata and ftl.state.remove(key) to remove it on teardown.
The state_file parameter on automation() persists run state to a JSON file for crash recovery.
FTL2 can run in long-lived application contexts such as a watchdog daemon, not just one-shot scripts.
Resource teardown must follow reverse dependency order (e.g., instances before networks); FTL2 does not automate this ordering — the user must sequence deletions manually.
Setting FTL2_TELEMETRY=off disables Segment telemetry on FTL2 startup.
FTL2 cloud provisioning scripts support three run modes: normal (provision), --check (dry run), and --teardown (destroy resources).
FTL2 deployments use a two-phase host registration pattern: connect first as root to bootstrap an admin user, then re-register the same hostname with ansibleuser='admin' and ansiblebecome=True for privileged operations.
FTL module Python scripts are executed via uv run python <script>.py, not through the ftl2 CLI.
FTL2 deployment scripts use uv run with PEP 723 inline dependency declarations, requiring no separate install step or virtualenv setup.
FTL2 deployment scripts can use uv run as their shebang with inline PEP 723 script dependencies, requiring no virtualenv setup.
Variable precedence in FTL2 follows group vars < host vars < module arguments (-a flag), with last wins semantics.
ftl.waitforssh() blocks until a newly provisioned server accepts SSH connections before attempting configuration, used in the provision-then-configure pattern.
The wait_for module can block until a TCP port (e.g., SSH on port 22) is reachable with a configurable timeout, acting as a readiness gate for downstream rules.
FTL2 manages the complete cloud lifecycle autonomously from initial provisioning through ongoing remediation: state-driven idempotent provisioning creates and configures hosts, persistent state enables crash recovery, and the AI reconciliation loop continuously observes and heals drift — all without human intervention.
FTL2 achieves fully secure single-file cloud deployments: PEP 723 scripts self-bootstrap dependencies and provide idempotent provisioning with state persistence, while comprehensive security from credential bindings to network edge ensures every deployment surface is protected — all from a single script file.
The gate cache location is ~/.ftl per user.
The gate's BackupManager creates backups, not individual modules — modules only declare backup capability via docstring metadata (Backup-Capable, Backup-Paths, Backup-Trigger).
The gate_dependencies parameter on automation() specifies additional Python packages to bundle into the gate zipapp alongside the modules — needed when modules import libraries not in the Ansible collection (e.g., httpx).
The gate event protocol uses JSON tuples as its message format (e.g., ["CommandName", {params}]), sent as length-prefixed messages over SSH channels.
Using gate_modules="auto" records module usage on the first run to .ftl2-modules.txt and reads from that file on subsequent runs.
Gate modules are pre-built once into a gate package; subsequent tasks send only JSON parameters over SSH, eliminating per-task module upload overhead.
The FTL2 gate protocol uses length-prefixed JSON for communication between the local controller and the remote gate process.
The gate subsystem is a complete remote execution optimization pipeline: modules are pre-built once into a zipapp, uploaded and cached on remote hosts, then subsequent tasks send only JSON parameters over a length-prefixed protocol.
The gate subsystem is enabled by passing gate_subsystem=True to the automation() context manager; gate processes persist for the lifetime of the async with automation() block.
FTL2's gate mechanism packages each module into a Python zipapp (.pyz), uploads it to /tmp/ on the remote host, executes it with the remote Python interpreter, and reads results back via length-prefixed JSON protocol.
FTL2 gate zipapps are cached on the remote host by content hash — once uploaded to /tmp/, a gate is reused on subsequent runs without re-uploading.
GCP automation requires three Python packages: google-auth, google-cloud-compute, and google-api-python-client.
The GCP provisioning example uses .ftl2-gcp-state.json as the state file for idempotent re-runs.
GCP authentication in FTL2 supports two methods via GCPAUTHKIND: serviceaccount (with GCPSERVICEACCOUNT_FILE) and application (via gcloud auth application-default login).
getattr(ftl, group_name) returns a HostScopedProxy that targets all hosts in that group, not a list of hosts. Also works for individual hosts (e.g., ftl.webserver).
Every FTL2 execution is simultaneously governed and fault-tolerant: security enforcement, credential injection, and structured telemetry instrument every module call end-to-end, while three independent crash recovery mechanisms and dual error modes ensure no failure — planned or unplanned — goes unrecoverable.
Grace period and poll interval are per-server configuration in servers.yml, not global settings.
When hosts are added with groups=["minecraft"], all modules for that group are accessible via ftl.minecraft.module() — the group name becomes a proxy attribute on the automation context.
Hetzner Cloud server type prefixes: cx = shared Intel/AMD, cax = shared ARM/Ampere, cpx = dedicated AMD, ccx = dedicated high-memory AMD.
Hetzner Cloud has six datacenter locations: nbg1, fsn1, hel1 (EU), ash (US East), hil (US West), sin (APAC).
The complete FTL2 pipeline from CLI host selection through module execution is performance-optimized at every layer: rich pattern-based targeting (groups, globs, exclusions) feeds into the three-tier performance stack (in-process native execution, gate-cached remote delivery, default parallelism), delivering 2-21x speedup across the entire targeting-to-result path.
Host names containing dashes (e.g., db-primary) are accessed as Python attributes with underscores (ftl.db_primary).
If a host or group name shadows a module name, a UserWarning is emitted at init and the module is accessible via ftl.module.<name>().
Host-scoped execution uses attribute access: ftl.webservers.service(name="nginx", state="restarted") targets a specific host or group.
ftl.<name>.module() proxy syntax targets a host or group by name — e.g., ftl.web01.command(...) targets host web01, ftl.webservers.command(...) fans out to all hosts in the webservers group.
After registering a host with ftl.add_host(), modules can be called against that specific host using bracket syntax: ftl[hostname].module(...).
ftl.hosts["name"] always returns a list of Host objects, whether looking up a host name or a group name.
FTL2 achieves idempotency through multiple complementary mechanisms: shell guards (which X || install X), the creates parameter for file-producing commands, inherently idempotent AI-loop module calls, and Cloudflare DNS create-or-skip logic.
Cloud provisioning is idempotent end-to-end: state persistence enables safe re-runs with crash recovery, multi-layer idempotency (shell guards, creates parameter, module-level) prevents duplicate resources, and the full lifecycle from cloud API through add_host() to two-phase bootstrap can be interrupted and resumed at any point.
FTL2 achieves 3-17x speedup over ansible-playbook by running modules as Python functions in-process rather than as subprocesses.
FTL2's inventory system seamlessly spans static and dynamic infrastructure: multi-format inventory (YAML, JSON, executable, dict) provides static host definitions while add_host's complete lifecycle (runtime registration, immediate state persistence, attribute access, group assignment) dynamically extends it mid-run — supporting traditional fleet management and cloud provisioning within the same scripts and addressing model.
FTL2 inventory is flexible and multi-source: auto-detects three formats (executable scripts, JSON, YAML), accepts Python dicts programmatically, uses Ansible-compatible connection parameters (ansiblehost, ansibleuser, etc.), provides built-in all/ungrouped groups, and uses a flat (non-nested) group hierarchy.
Inventory can be loaded from a YAML file path (automation(inventory="/path/to/inventory.yml")) or from a Python dict passed to AutomationContext(inventory={...}).
The community.general.linode_v4 module with state: present is idempotent — if the instance already exists and matches the spec, it returns the existing instance without changes.
The load_config function returns an empty dict if the config file doesn't exist, rather than raising an error.
load_inventory() auto-detects three inventory formats: executable scripts, JSON (Ansible --list format), and YAML.
FTL2's local execution path bypasses SSH and gate overhead entirely, using direct subprocess/pathlib operations — making it the natural high-performance path for cloud API modules that call external services rather than remote hosts.
FTL2 local execution is triggered by setting ansible_connection: local in inventory, which uses LocalModuleRunner instead of RemoteModuleRunner.
FTL2 optimizes execution across the entire local-to-remote continuum: local operations bypass SSH and gate overhead entirely using direct subprocess/pathlib for cloud API calls, while remote execution uses SSH connection pooling, gate-cached zipapps, and length-prefixed JSON protocol — with the same modules and API working identically in both modes.
All native modules detect local/localhost targets and use direct local operations (subprocess, pathlib) instead of SSH/SFTP.
FTL2 supports four complementary module addressing syntaxes: dot-access for simple names, bracket notation for dashed hostnames, FQCN chains for collection modules, and host-scoped proxy for targeted execution.
FTL2 module calls follow the attribute-chain pattern ftl.<inventorygroup>.<modulename>(param=value).
Module calls like ftl.file(...) are proxied through ModuleProxy, which resolves module names at runtime via attribute access.
The planned Module Discovery API has two functions: ftl.listmodules(category=None) for browsing by category and ftl.describe(modulename) for detailed parameter info.
Every FTL2 module execution passes through three independent security layers: policy enforcement denies unauthorized operations before execution begins, secret bindings auto-inject credentials without code or log exposure, and privilege escalation stages files through safe temp paths with per-user sudoers drop-in files — composing into defense-in-depth at the individual module call level.
Passing modules=["file", "copy"] to automation() restricts available modules; calling any other module raises AttributeError.
Every FTL2 module execution returns a consistent result contract — success, changed, output, and error fields — regardless of whether the module ran locally or remotely, natively or via Ansible compatibility, through direct calls or the Executor API.
Module call results expose an .output dict containing provider-specific data (e.g., selfLink, networkInterfaces) that can be used to chain resource dependencies.
FTL2 provides a universal module system: four addressing syntaxes (dot-access, bracket notation, FQCN chains, host-scoped proxy) cover every naming and targeting pattern, while the dual-mode architecture (native in-process + Ansible-compatible bundled) executes any module transparently with the same local/remote API surface.
Modules are callable attributes on the automation context object, accessible via ftl.modulename() or dynamically via getattr(ftl, modulename).
FTL2 unifies multi-cloud authentication through a consistent env-var and secret-bindings pattern: GCP uses three env vars with two auth methods, Azure uses four service principal vars, Hetzner injects tokens via wildcard secret bindings, and all resolve credentials at module call time without source code exposure.
Multi-cloud operations are fully automated from provisioning through ongoing management: unified credential patterns across GCP, Azure, Hetzner, and Linode combine with provider-specific state isolation and idempotent end-to-end cloud provisioning for completely unattended multi-cloud deployments.
FTL2 enables multi-cloud operations with clean isolation between providers: provider-specific state files (.ftl2-state-<provider>.json) prevent cross-cloud resource conflicts, while the unified env-var and secret-bindings pattern provides consistent credential management across GCP, Azure, Hetzner, and Linode.
Multi-host progress tracking uses display.makecallback(hostname) to create per-host callbacks — not display.handle_event directly.
Accessing an unknown attribute on the automation context returns a NamespaceProxy that builds a dotted FQCN path; calling the final proxy executes the module via context.execute().
Fully qualified collection names are accessed via dot notation: ftl.community.general.slack(channel="#deploy", msg="Done!").
Native FTL modules run in-process and are 250x faster than Ansible's subprocess-based execution.
FTL2 has six native shadowed modules that replace Ansible equivalents: waitforssh, ping, copy, template, fetch, and shell.
Gate modules are NOT used in local execution — gates are a remote execution optimization only.
AI-loop autonomous remediation is fully observable: observe/condition/action cycles emit structured events (progress, log, status) that are always captured in results regardless of display mode, enabling operators to monitor, debug, and audit self-healing operations without interfering with autonomous execution.
FTL2 infrastructure applications achieve both resilience and full observability simultaneously: servercraft demonstrates safe re-provisioning via idempotent cloud operations while every watchdog and AI-loop self-healing cycle emits structured events — autonomous operation without sacrificing operational insight.
Autonomous infrastructure operations are both safe and fully observable: intelligent lifecycle management (warmup delays, consecutive-failure thresholds, player-join grace period resets) operates alongside structured event emission for every observe/condition/action cycle — operators can verify autonomous decisions without intervening.
The observe block in AI-loop rules is a list of dictionaries with name, module, params, and host keys — each entry runs a module call to gather state.
AI-loop rule observe blocks use || true and 2>&1 to prevent observation failures from crashing the reconciliation loop.
automation(on_event=callback) receives structured event dicts with fields: event, module, host, success, changed, duration (float seconds), and timestamp.
FTL2 provides a complete spectrum of output verbosity: default shows only errors with a per-host task summary, verbose adds per-operation execution timing, and quiet with event callbacks enables structured machine-readable capture for production use — covering interactive development through production monitoring without code changes, only parameter changes.
FTL2 executes modules concurrently across all targeted hosts by default with no explicit parallelism configuration required, so total execution time approximates the slowest host rather than the sum of all hosts. asyncio.gather is one mechanism used for concurrent fan-out (e.g., in the copy module), and execute_batch provides near-linear speedup over sequential execute() calls via asyncio.
FTL2 enables a zero-setup single-file deployment model: PEP 723 inline metadata declares dependencies, uv run self-bootstraps them via shebang, and no virtualenv, requirements.txt, or install step is needed.
The FTL2 ping module returns {"changed": false, "ping": "pong"}.
ping() returns {'ping': 'pong'} on success.
ping() tests the full execution pipeline in order: TCP → SSH → Gate setup → Command execution → Response round-trip. A successful ping guarantees module execution will work.
FTL2's ping() tests the entire gate execution pipeline (TCP → SSH → gate → command → response), unlike Ansible's ping which only tests the connection plugin.
Policy audit events are appended as JSON lines immediately after evaluation, making the audit trail crash-safe.
Policy enforcement runs before every module execution; denied actions raise PolicyDeniedError.
FTL2's policy engine provides complete pre-execution access control: enforcement runs before every module call with first-match-deny semantics, rules match on module/environment/host/parameter fields for granular control, and shell/command/raw modules are treated as policy-equivalent to prevent bypass through alternative command execution paths.
Planned policy engine improvements include rule short-circuiting (not just deny), policy validation on load, policy composition from multiple files/directories, and policy summary in audit logs.
The policy engine uses first-match semantics — the first matching deny rule raises PolicyDeniedError, not the most specific rule.
Policy rules match on module, environment, host, and param.* fields; a PolicyDeniedError is raised when a rule blocks an action.
Each game server type in servercraft gets both custom lifecycle behavior through the 8-function script interface (provision, start, verify, backup, restore, stop, getplayercount, reconfigure) and unified resilience monitoring through the watchdog's warmup/threshold/grace-period chain — the polymorphism handles game-specific differences while the watchdog provides game-agnostic operational safety.
Port 80 must be open on the host even when serving only HTTPS, because Let's Encrypt's HTTP-01 ACME challenge requires it.
FTL2 privilege escalation is both safe and flexible: copy operations stage through /tmp with sudo mv to avoid permission races, become_user enables running as specific unprivileged users (not just root), and sudoers configuration uses /etc/sudoers.d/ drop-in files rather than editing /etc/sudoers directly.
FTL2 production deployments are simultaneously reliable and secure: state persistence enables idempotent re-runs and crash recovery while the security-first bootstrap ensures every run (including re-runs) maintains the hardened posture — security is never a one-time setup that drifts.
Production FTL2 deployments are fully hands-off from bootstrap to serving: zero-setup scripts self-bootstrap dependencies via uv run, DNS records are automated via Cloudflare modules, and TLS is auto-provisioned via Caddy's Let's Encrypt integration — the entire deployment lifecycle requires no manual intervention.
FTL2 examples use provider-specific state file names — .ftl2-state-hetzner.json for Hetzner, .ftl2-state-azure.json for Azure, and .ftl2-gcp-state.json for GCP — alongside the default .ftl2-state.json. The Hetzner and Azure names follow a .ftl2-state-<provider>.json pattern, though GCP's .ftl2-gcp-state.json deviates from it. This suggests provider-specific state files can coexist, which could support multi-cloud deployments maintaining independent state per provider.
Production provisioning functions wrap the entire operation in try/except, send a Slack failure notification on error, then re-raise — ensuring the caller still sees the failure while operators get immediate alerts.
FTL2 provisioning is idempotent enough to support "Reconfigure" (re-run on existing server) as a distinct operation from initial "Launch".
FTL2 manages the complete host lifecycle from cloud API call through ongoing real-time monitoring: local execution provisions VMs via cloud API, add_host dynamically registers them with two-phase bootstrap, and the htop monitoring stack (SSH→gate→psutil with sparklines and WebSocket broadcast) provides continuous operational visibility over the provisioned fleet.
FTL2 provides a complete pipeline from cloud provisioning to optimized execution: VMs are provisioned via cloud API, dynamically registered with add_host(), bootstrapped through two-phase registration, then managed through connection-pooled asyncssh with gate-optimized module delivery (pre-built zipapps, length-prefixed JSON, content-hash caching).
quiet=True combined with on_event (silent execution with structured event capture) is the recommended pattern for production scripts.
FTL2 host provisioning is safe from first connection: SSH readiness gating (waitforssh with configurable timeout and wait_for TCP port checks) ensures the host is reachable before two-phase bootstrap (root → admin re-registration → security hardening → content deployment) begins.
FTL2 remote execution uses the asyncssh library for SSH connections, triggered by ansible_connection: ssh in inventory.
Replay matching is positional — action N in the current run matches action N in the replay log; successful replayed actions are skipped with cached output.
FTL2 fleet operations are resilient under partial failure: parallel execution across all hosts continues on remaining hosts when some fail, with dual error modes providing flexible failure policy — continue-on-error collects all failures for fleet-wide analysis, while fail-fast halts on critical operations.
FTL2 module results can be dicts, lists (multi-host), or objects with .output/.success attributes. Production apps use a resultdict() helper to normalize these into plain dicts for reliable field access.
ftl.run_on(target, module, **kwargs) is the older explicit targeting API and remains supported alongside the newer proxy syntax.
ftl.run_on() accepts a group name string, host name string, or list of Host objects as its target parameter.
SSHHost.run_streaming() returns a 4-tuple (stdout, stderr, rc, events), extending the standard run() 3-tuple with parsed events.
Servercraft achieves safe AND intelligent autonomous lifecycle management: intelligent watchdog behavior (10-minute warmup, 3-consecutive-failure threshold, grace-period reset on player join, context-sensitive teardown) safely re-invokes infrastructure operations because FTL2's multi-layer idempotency prevents duplicate resources or repeated destructive side effects on each re-execution cycle.
FTL2 safely scales concurrent operations across large fleets: multi-layer idempotency (shell guards, creates parameter, state-driven re-runs) combined with default parallel execution across all hosts ensures re-runs never create duplicate resources even when targeting dozens of hosts simultaneously.
FTL2 resource lifecycle completion is both orderly and recoverable: state-managed teardown uses label lookup and reverse dependency ordering for clean multi-resource removal with state file cleanup, while the backup subsystem ensures destructive operations are recoverable via the two-phase protocol (discover then backup) with fail-safe abort on backup failure.
FTL2 uses the same automation() context manager, module addressing, and error handling API for both development scripts and production deployments — there is no separate deployment-specific interface.
The same FTL2 modules work identically in both local and remote execution modes — only the connection type and runner change.
The script field in servers.yml selects which Python module under scripts/ handles that server type (e.g., script: neoforge loads scripts/neoforge.py).
The ftl2-servercraft script interface defines exactly 8 functions: provision, startserver, verifyserver, restoreworld, backupworld, getplayercount, saveandstop, destroy.
The secret_bindings parameter of automation() maps environment variables to module parameters without requiring Vault integration.
FTL2 secretbindings map environment variable names to module parameters — e.g., LINODETOKEN maps to community.general.linodev4's accesstoken, so credentials are injected automatically without appearing in rule code.
Secret binding keys are module names or glob patterns (e.g., amazon.aws.*); values are dicts mapping parameter names to environment variable names.
Secret bindings are passed as the secret_bindings parameter to the automation() context manager at initialization time.
Secret bindings ensure credentials are never visible in code or logs, critical for AI agent safety.
Secret bindings resolve environment variables at module call time and inject them into parameters — the script source never contains or sees actual secret values.
FTL2 implements defense-in-depth for credentials: bindings auto-inject secrets without source code exposure, repr/str always redact values, secrets are never logged, and resolution happens at call time so variables never appear in scripts.
ftl.secrets["KEY"] raises KeyError if the environment variable wasn't set or wasn't in the declared secrets list.
FTL2 covers the full secrets management spectrum: environment variable bindings for simple deployments, HashiCorp Vault integration with grouped reads for enterprise key rotation, and defense-in-depth (redaction, never-logged, lazy evaluation) at every tier — teams can start simple and scale to Vault without changing their security posture.
Secrets are declared as a list of environment variable names via automation(secrets=["KEY1", "KEY2"]) and sourced from the process environment.
For modules not covered by secret bindings, ftl.secrets["KEY"] is still available as a fallback mechanism for accessing secrets.
ftl.secrets.get("KEY", default) retrieves a secret with a fallback value if the key is missing.
FTL2 secrets use lazy evaluation: missing env vars don't fail at context creation, errors surface only on access, and get() provides default fallbacks — enabling flexible secret management without upfront validation failures.
ftl.secrets.keys() lists all requested secret names, while ftl.secrets.loaded_keys() lists only secrets actually found in the environment.
Missing environment variables for declared secrets don't cause failure at context creation — the error surfaces on access (or returns a default via .get()).
Secrets are never logged — SecretsProxy provides safe _repr/str_, and secret bindings use fnmatch patterns to auto-inject secrets with pre-injection params sent to audit trails.
FTL2's secrets system achieves both strong security and developer ergonomics: defense-in-depth prevents credential leakage at every surface (bindings, repr, logs), while lazy evaluation avoids premature failures and provides graceful error semantics on access.
str() and repr() on the secrets object redact actual values, preventing accidental exposure in logs or tracebacks.
FTL2 production deployments are simultaneously auditable and credential-safe: every re-execution is tracked through event streaming and state-driven idempotency while defense-in-depth ensures credentials never appear in audit trails, logs, or repr output — full traceability without security exposure.
Autonomous FTL2 operations run within a complete governance framework: self-healing AI-loop remediation with safe lifecycle management (warmup delays, failure thresholds, player-aware teardown) operates under comprehensive credential security from storage to network edge, with full observability at every layer.
Dynamically provisioned cloud hosts receive full security AND observability from first connection: two-phase bootstrap hardens access before content deployment, every module call passes through triple security enforcement (policy, credentials, privilege escalation), and all operations emit structured events for audit.
Every FTL2 module execution is both secured and observable: calls pass through policy enforcement, credential injection, and privilege escalation controls while simultaneously emitting structured events — creating an audit trail where every secured operation has a corresponding observable record.
FTL2 is a secure, self-sufficient production platform: autonomous self-healing with full observability operates within comprehensive security from credential storage to network edge, making it capable of sustained autonomous operation without manual security intervention.
The complete host selection through module execution pipeline is secured end-to-end: CLI targeting (groups, globs, exclusions) feeds into multi-format inventory resolution, then every resolved module call passes through policy enforcement, credential injection, and privilege escalation before execution.
AI-loop autonomous remediation is fully visible without manual monitoring: each observe/condition/action cycle within the error-resilient framework emits structured events through the observability stack, meaning self-healing actions are event-tracked, their failures are captured, and their convergence can be verified by external systems.
The server's name field is used as the resource key in the FTL2 state dict when reading .ftl2-state.json.
The default grace_period for ServerConfig is 1800 seconds (30 minutes) before idle shutdown.
The default poll_interval for ServerConfig is 30 seconds for watchdog polling frequency.
Every servercraft infrastructure command creates an FTL2 automation() context with failfast=True, autoinstalldeps=True, and secretbindings mapping module FQCNs to Vault secret names (LINODETOKEN, LINODEROOTPASS, SLACKTOKEN).
All servercraft commands accept --config path/to/servers.yml with the default being servers.yml.
Running ftl2-servercraft with no subcommand launches the TUI dashboard (equivalent to ftl2-servercraft tui).
Servercraft implements game server polymorphism: the script field in servers.yml selects a Python module implementing the 8-function script interface, with game-specific variations (NeoForge: 7-step provision, 30s backup poll, "minecraft" tmux session; Terraria: 5s backup wait, port 7777, "terraria" host group, regex-based player count) all conforming to the same lifecycle contract.
The launch and reconfigure commands use gatemodules="auto" with gatedependencies=["httpx"] to pre-build modules for remote execution performance.
getplayercount() returns None when the tmux session is down (server not running), 0 when running but no players matched, or the parsed player count as an integer.
Servercraft implements intelligent lifecycle management: automated watchdog paths (warmup grace, consecutive-failure threshold, player-join reset) and manual stop paths adapt teardown behavior based on operational context — watchdog aborts on backup failure to preserve data, while manual stop continues past backup failure because the user explicitly chose to shut down.
ftl2-servercraft is a Minecraft server lifecycle manager built on FTL2, providing CLI subcommands to provision, start, stop, back up, verify, and monitor game servers on Linode infrastructure.
The launch subcommand runs the full lifecycle: provision → optionally restore from backup → start → verify. Verification failure raises RuntimeError and exits with code 1.
The NeoForge backup_world() sends save-all and polls the server log for up to 30 seconds waiting for "Saved the game" confirmation before archiving. Proceeds with a warning if confirmation never appears.
The NeoForge provision() function orchestrates seven sequential steps (infrastructure, minecraft install, utilities, security, swap, ownership, DDNS) plus an optional NeoForge install, each idempotent via ftl.state checks.
The NeoForge Minecraft server script uses the tmux session name constant TMUX_SESSION = "minecraft".
The NeoForge script uses ftl.add_host() to dynamically register the provisioned server into the minecraft group for subsequent module execution.
Servercraft game servers achieve infrastructure-level resilience: watchdog-triggered re-provisioning and reconfiguration are safe because the cloud provisioning pipeline is idempotent end-to-end (state persistence, create-guards, DNS idempotency), and the polymorphic script interface ensures each game type handles its own recovery sequence correctly.
The Terraria script includes a resultdict() helper that normalizes FTL2 action results (which can be dicts, lists, or objects with .output) into plain dicts for extracting stdout.
In the servercraft provisioning scripts, Slack notification failures are silently swallowed via double-wrapped try/except blocks to prevent notification errors from aborting infrastructure operations.
Servercraft provisioning scripts use an SSH bootstrap pattern: initially connect as root, create an admin user, harden SSH (disable password auth, IPv4-only), then re-register the host with the admin user.
The stop subcommand performs save → backup → Slack notification → destroy, where backup and Slack failures are caught and logged but do not abort the destroy step.
Servercraft teardown behavior is context-sensitive: the manual stop subcommand continues past backup failure (user explicitly chose to stop), while watchdog-initiated teardown aborts on backup failure (automated safety demands a higher bar).
The Terraria backup_world() sends save to the console and waits 5 seconds before fetching the .wld file, compared to the NeoForge script which polls for 30 seconds.
The Terraria provisioning script opens TCP port 7777 (Terraria default) via firewalld.
The Terraria script registers provisioned servers into the terraria host group (vs. minecraft for NeoForge), using ftl.add_host().
The Terraria script reads player count by capturing the tmux pane output and regex-matching N players? connected.
The Terraria server script uses the tmux session name constant TMUX_SESSION = "terraria".
Game servers in ftl2-servercraft run in tmux sessions on remote hosts; all monitoring and control happens through tmux commands executed via FTL2 shell modules.
The servercraft TUI dashboard is built on the Textual framework (with Rich for text styling), using Screen, ModalScreen, DataTable, RichLog, Header, Button, and Label widgets.
All FTL2 operations in the TUI run in daemon threads with their own asyncio event loops, keeping the UI responsive while infrastructure operations execute.
The TUI runs checkexisting_state() on startup to read FTL2 state files and detect already-provisioned servers, so the dashboard reflects reality on restart.
The TUI dashboard refreshes every 2 seconds by reading shared runtime_state from a timer callback.
The TUI uses module-level runtime_state dict of ServerState dataclasses as shared mutable state between background threads and the UI, with no locking — relying on the GIL and simple attribute writes.
The verify subcommand exits with code 2 if the tmux session is not found (distinct from code 1 for errors).
Three consecutive unreachable player count polls triggers an emergency backup + teardown sequence.
The watchdog module sleeps 600 seconds (10 minutes) before its first player count poll, allowing the game server time to finish starting.
If backup fails during watchdog-initiated teardown, the teardown is aborted and the server status is set to error, preventing data loss.
If a player joins during the watchdog grace period (countdown to teardown), the empty-since timer resets completely, canceling the pending teardown.
The servercraft watchdog implements a complete resilience chain: 10-minute warmup avoids false positives on slow starts, 3-consecutive-failure threshold absorbs transient errors, grace period resets on player join, and backup failure aborts teardown to prevent data loss.
The watchdog loop exits on one of three conditions: (1) grace period expires and server is torn down, (2) 3 consecutive unreachable polls trigger teardown, or (3) state.watchdog_active is set to False externally by the user.
shell, command, and raw modules are treated as policy-equivalent by FTL2's policy engine — denying one blocks all three, including FQCN variants.
The FTL2 shell module declares Backup-Capable: No because its effects are unpredictable.
The FTL2 shell module returns {"changed": true, "stdout": "...", "stderr": "...", "rc": 0} — it always reports changed: true.
FTL2 enables idempotent cloud deployments from single-file scripts: PEP 723 scripts with integrated credential management (secret bindings, defense-in-depth) orchestrate a complete idempotent provisioning pipeline (state persistence, crash recovery, shell guards, creates parameter) requiring no project structure, virtualenvs, or separate configuration files.
FTL2 enables complete deployment scripts in a single file with integrated credential management: PEP 723 inline metadata self-bootstraps dependencies via uv run, while secret bindings auto-inject credentials without source code exposure, redact values in logs, and lazy-evaluate at call time.
FTL2's SSH layer is designed for connection efficiency: the connection pool reuses connections to identical hosts, asyncssh provides async non-blocking I/O, sshrunon_hosts parallelizes across hosts, and the gate protocol sends only JSON parameters over established connections — eliminating per-task connection overhead.
SSHConnectionPool.get() reuses connections to the same host — calling get() with identical config returns the same SSHHost object.
The SSH user-facing API (SSHHost, SSHConnectionPool, sshrun, sshrunonhosts) is imported from ftl2.ssh.
FTL2 provides comprehensive SSH readiness gating: waitforssh() blocks with configurable timeout and delay until SSH is available, wait_for module checks arbitrary TCP ports as a generic readiness gate, and both serve as blocking preconditions before configuration commands are sent.
sshrunon_hosts() runs commands on multiple hosts in parallel and returns a list of (hostname, stdout, stderr, rc) tuples.
SSHHost is used as an async context manager (async with host:) which handles connect/disconnect automatically.
SSHHost.run() returns a 3-tuple (stdout, stderr, rc) and does not raise an exception on non-zero return code — the caller must check rc.
FTL2 achieves reliable re-runnability through persistent state orchestration combined with multi-layer idempotency: the state file persists between runs, enables crash recovery, and shares context across AI-loop rules, while shell guards, creates parameters, and module-level idempotency ensure repeated execution converges without side effects.
AI-loop rules share state via the FTL2 state file — e.g., a provisioning rule writes resources.arcade.ipv4 and a DNS rule reads it via state["statefile"]["resources"]["arcade"]["ipv4"].
FTL2's state file is .ftl2-state.json; existence is checked with ftl.state.has("name").
The FTL2 state file serves as the orchestration backbone: it persists between runs, enables crash recovery, shares state across AI-loop rules, and provides the resource lookup key for multi-script workflows.
The FTL2 state file (e.g., state.json) persists host IPs and connection info between separate script runs — redeploy reads it to recover the Linode IP without re-querying the API.
ftl.state.has(key) checks if a resource exists in the state file and ftl.state.get(key) retrieves its metadata dict. Used to skip provisioning on re-runs and to look up IPs for re-registration.
FTL2 resource teardown is state-managed and orderly: destroy operations look up resources by label (not ID), state files are cleaned up after successful destroy, modules use state=absent for removal, and teardown must follow reverse dependency order (instances before networks).
All streaming executor functions (executelocalstreaming, executeremotestreaming, executeremotewithstagingstreaming) have a default timeout of 300 seconds.
FTL2's consistent result contract complements its flexible output modes: every module returns success/changed/output/error fields regardless of execution context, while a separate output verbosity spectrum — from errors-only default through verbose per-operation timing to quiet structured event capture — allows tuning presentation for development through production use via parameter changes alone.
FTL2 deployment rules use /etc/sudoers.d/<username> drop-in files rather than editing /etc/sudoers directly for granting sudo access.
Templates are rendered locally via Jinja2 with variables passed as **kwargs, then transferred to the remote host via the copy() method.
Terraform provider integration in FTL2 requires a gRPC protocol bridge from Python to Go because FTL2 is Python and Terraform providers are Go.
State from Terraform providers maps to the same .ftl2-state.json format used by native FTL2 state tracking.
The textual serve -c "<command>" --port <port> command wraps a Textual TUI application for web browser access.
FTL2's two-phase host registration is a complete bootstrap lifecycle: connect as root, create admin user, re-register as the unprivileged admin, then apply security hardening before any application content.
For bootstrapping fresh VMs, a two-phase host registration pattern is used: first register as root for initial user creation, then re-register as the unprivileged user with ansible_become=True for all subsequent operations.
Both pre-existing and dynamically provisioned infrastructure can receive strong security governance and auditability through FTL2's mechanisms: static deployments use state-driven idempotent re-execution with event tracking, while dynamically provisioned hosts receive two-phase bootstrap hardening — and both pass through the same policy, credential, and privilege escalation enforcement layers with structured event emission for audit.
FTL2's universal module system supports multiple addressing patterns (dot-access, bracket notation, FQCN chains, host-scoped proxy) that resolve to modules executable through either native in-process or Ansible-compatible bundled modes. These modules can benefit from FTL2's three-tier optimization stack (in-process native execution, gate-cached remote delivery, default parallel host targeting), though the antecedents do not establish that all addressing patterns achieve identical performance characteristics across all tiers.
FTL2 deployment scripts use uv run with PEP 723 inline script metadata to self-bootstrap dependencies (FTL2, linode-api4) — no separate virtual environment setup is needed.
Cloud provisioning is validation-gated end-to-end: after VM creation via cloud API and dynamic host registration via add_host, SSH readiness gating verifies the host is actually reachable before two-phase bootstrap and configuration begin, preventing wasted operations against unreachable hosts.
Vault integration involves a three-part setup: the hvac optional dependency (pip install ftl2[vault]), two environment variables (VAULTADDR and VAULTTOKEN), and secret reads that are grouped by path to minimize API calls — a layered configuration where each part addresses a different concern (library availability, connection credentials, and API efficiency).
Vault reads are grouped by path to minimize API calls when multiple secrets share the same Vault path.
Vault integration requires VAULTADDR and VAULTTOKEN environment variables.
Vault integration requires pip install ftl2[vault] and the VAULTADDR and VAULTTOKEN environment variables to be set.
Vault integration requires installing the optional hvac dependency via pip install ftl2[vault].
The -vvv flag enables verbose logging in FTL2 for troubleshooting remote execution issues.
automation(verbose=True) shows every operation with execution timing.
In the script interface, verifyserver returns bool and getplayer_count returns int | None where None indicates server unreachable (distinct from zero players).
FTL2 provides a waitfor module (called as ftl.waitfor(host=..., port=22, timeout=300)) that blocks until a TCP port is reachable — used to gate downstream rules on SSH readiness after provisioning.
ftl.<host>.waitforssh(timeout=120, delay=10) polls until SSH is available on a host, with timeout in seconds and delay between retries.
FTL2 automation contexts can be held open for hours in watchdog/monitoring loops. The context manages SSH connections and state for the lifetime of the async with block.
Servercraft's watchdog safely re-invokes infrastructure operations because FTL2's multi-layer idempotency ensures re-execution never creates duplicate state: shell guards skip completed installs, the creates parameter skips existing files, and module-level idempotency handles the rest — so watchdog-triggered re-provisioning after consecutive failures is always safe.
FTL2 enables zero-setup scripts that access the full module ecosystem: PEP 723 inline metadata with uv run eliminates project scaffolding, while dual-mode execution provides both native fast-path modules and Ansible collection compatibility in the same script.
FTL2 supports a developer journey from zero-setup scripting to production execution with autonomous remediation capabilities: frictionless onboarding (PEP 723/uv, dual-mode modules) flows into an optimized cloud-to-execution pipeline, while AI-loop remediation with structured observability enables self-healing actions that are event-tracked and verifiable — reducing the need for platform switches between prototyping and production.
FTL2 delivers a complete developer journey from zero-setup to production execution: frictionless onboarding (PEP 723/uv scripts, dual-mode module ecosystem) flows into an optimized cloud-to-execution pipeline (provision, add_host, gate optimization, parallel execution), enabling developers to go from a single file to running infrastructure with no scaffolding or intermediate tooling.
AI-loop rules are fully trustworthy for autonomous operation: the three-phase contract ensures predictable behavior, observation resilience prevents false negatives, and cross-rule state sharing enables coordinated multi-rule convergence.
All Ansible collection modules work seamlessly in FTL2 via FQCN dot-notation addressing and community accessors.
Teams migrating from Ansible gain FTL2's full compound speed advantage (3-21x: in-process execution, gate caching, default parallelism) while retaining access to their existing module ecosystem through FQCN dot-notation and Ansible-compatible bundled execution — migration preserves investment in existing modules without sacrificing performance.
Teams migrating from Ansible experience zero-friction adoption: familiar CLI flags, variable precedence, inventory groups, and FQCN naming transfer directly while the dual-mode execution architecture runs existing Ansible modules without modification.
AI-loop drift detection accurately determines convergence because every module reports true change status: the remediation system's observe/condition/action cycle trusts the consistent result contract (success, changed, output, error) to distinguish actual corrections from no-ops, enabling reliable convergence detection.
AI-loop autonomous remediation achieves verified convergence: each observe/condition/action cycle uses multi-layer idempotency (shell guards, creates parameter, module-level idempotency) to ensure actions are safe to repeat, and accurate changed/unchanged reporting confirms when desired state has actually been reached, enabling the loop to stop acting.
Check mode provides a reliable dry-run preview: post-provision steps are skipped and results still accumulate for analysis.
FTL2 provides complete change tracking across all operations: every module reports whether it actually changed state via the consistent result contract, and multi-layer idempotency (shell guards, creates parameter, module-level) ensures unchanged operations accurately report no-change — enabling precise audit logs and drift detection.
FTL2 teardown operations handle complete multi-resource deployments: state files track all provisioned resources with provider-specific naming, teardown follows reverse dependency order, and state cleanup removes all traces after successful destruction.
In FTL2 deployments (e.g., ftl2-stargate), secrets such as Google OAuth credentials are entered interactively at startup and are not handled by the deployment automation.
All destructive module operations are recoverable via automatic backups: backups are enabled by default, the two-phase protocol ensures correct backup creation before any modification, and fail-safe semantics abort the operation if backup creation fails.
Dynamically provisioned infrastructure is fully portable across the entire Ansible-compatible module ecosystem: add_host registration feeds into the dual-mode execution architecture supporting both native and bundled modules — unless stdlib shadowing in certain Ansible modules causes import failures in FTL2's in-process runtime.
FTL2 reliably scales to 25+ hosts where Ansible drops unreachable hosts, thanks to persistent gate connections and default parallel execution.
The gate subsystem provides universal module packaging: pre-built zipapps cached by content hash deliver any module to remote hosts with only JSON parameter overhead.
FTL2's multi-layer idempotency strategy provides fully detectable change status: every module execution reports whether it actually changed state, enabling accurate drift detection and convergence verification.
Servercraft is a complete autonomous game hosting system: polymorphic game support via the script interface, intelligent lifecycle management (watchdog warmup, consecutive-failure threshold, grace-period reset, context-sensitive teardown), and a responsive TUI dashboard for monitoring and control.
The servercraft TUI is production-ready: built on Textual with daemon threads for responsive UI, automatic detection of existing state on startup, and web accessibility via textual-serve.