Financial-institution automation fleet

Context

Part of the team (part-time) on a payments platform operating across three countries. My scope was the financial-institution automation and the core architecture — the per-institution crawlers, the isolation model that lets accounts and machines run in parallel, and the central control plane an operator uses to drive the fleet.

Problem

Every institution exposes a different web portal, and each one changes without warning — markup shifts, login flows get reworked, extra verification steps appear. Balances and transaction records had to be collected reliably, at scale, with no human in the loop, and without tripping institution-side bot defenses.

Constraints

These portals change without notice, so selectors and login flows cannot be assumed stable between runs. Anti-bot systems watch for non-human patterns. Many accounts across several machines must run in parallel without their sessions, cookies, or fingerprints bleeding into one another. A single flaky login must never cascade into the rest of the fleet.

Approach

• Analyze first, then automate These portals span different eras and architectures — no two alike — so each is studied to find the most effective, reliable path to the data, rather than forcing one method onto all of them.
• One thin adapter per institution A change at one portal is a contained edit that never touches the others; layered selector strategies (stable attributes and anchored text over brittle DOM paths) ride out routine portal churn.
• Non-intrusive, isolated sessions Human-like interaction mirrors a real user, and each account runs in its own browser profile, storage, and fingerprint — so parallel runs across machines never collide.
• A login-error state machine Every failure is classified as transient (retry with backoff) or hard (disable the slot and flag it), so a bad password disables one account while a network blip just retries — neither takes the fleet offline.
• Central control plane at the core It load-balances work across machines, manages the distributed crawlers as one coordinated fleet, drives the run-state machine, and surfaces live per-institution status and structured logs — so one operator runs the whole thing.

Outcome

Multi-country coverage running in production. Portal changes are absorbed by editing a single adapter rather than firefighting the whole fleet, and one operator has end-to-end visibility and control — scheduling, per-institution/country health, and failure triage — from the central control plane.

Stack

Python · asyncio · Playwright · httpx · cryptography · per-institution adapters · session isolation · state-machine management · load balancing · distributed crawler orchestration · central control plane · YAML-driven config · Linux