Last September, every Irish hosting provider's nightmare became reality. A cascading DNS failure triggered by upstream infrastructure changes started in Dublin at 14:23, reached Cork by 14:47, and within two hours had affected web services across the entire country. For TechFlow Hosting, this presented a unique challenge: their web services team operated from Dublin while their database infrastructure team worked 260 kilometres away in Cork.
What could have been a coordination disaster became their most successful incident response to date. The secret wasn't sophisticated incident management software or enterprise coordination platforms. It was a simple framework they'd developed for managing exactly this scenario.
The Challenge: When Geography Becomes Your Enemy
Most incident response guides assume everyone sits in the same office or at least the same timezone. Reality for distributed teams is messier. When TechFlow's Dublin team started seeing web service timeouts, their first instinct was to restart application servers. Meanwhile, the Cork team was tracking database connection spikes that suggested a completely different root cause.
Without clear communication protocols, both teams would have spent precious minutes working against each other. Dublin's application restarts would have masked the database connection patterns Cork needed to diagnose the real issue.
Dublin Team's Web Services vs Cork Team's Database Infrastructure
The geographic split created an interesting advantage once they learned to use it properly. Dublin team members were seeing the customer-facing impact immediately through support tickets and monitoring alerts. Cork team members had better visibility into the underlying database health and connection pools that supported those services.
This separation of concerns only worked because they'd established clear ownership boundaries beforehand. Dublin owned anything customer-facing: web servers, application performance, user experience metrics. Cork owned the data layer: database performance, backup systems, storage infrastructure.
The key was recognising that during incidents, these boundaries become collaboration points rather than territorial divisions.
Communication Protocols That Actually Work Under Pressure
TechFlow's approach rejected the complexity of most enterprise incident management platforms. Instead, they used a shared text file hosted on a simple internal wiki, combined with a dedicated Slack channel that fed into Server Scout's webhook notifications for critical threshold breaches.
The 15-Minute Check-In Rule
Every 15 minutes, each team lead posted a single-line status update in this format:
"Dublin 14:45 - App servers stable, investigating timeout patterns in zone 3" "Cork 14:45 - DB connections normal, backup processes paused"
This rhythm prevented the chaos of constant interruptions while ensuring nobody worked in isolation for dangerous lengths of time. More importantly, it forced each team to distil their current understanding into something the other team could actually use.
The 15-minute interval wasn't arbitrary. Shorter intervals created update fatigue and prevented deep investigation work. Longer intervals allowed teams to drift too far apart in their understanding of the situation.
Shared Documentation in Real-Time
The shared text file became their single source of truth. Each entry included timestamp, team, current hypothesis, and next action. No complex formatting, no approval processes, just facts and intentions.
This approach scales remarkably well. TechFlow has since used it for incidents involving teams in Dublin, Cork, and their data centre partners in Amsterdam. The simplicity means anyone can contribute without learning new tools under pressure.
Decision-Making Framework: Who Owns What When Everything's Broken
The hardest part of distributed incident response isn't technical - it's avoiding the decision paralysis that comes when multiple teams could reasonably claim ownership of a problem.
TechFlow solved this with a dependency-based hierarchy rather than a seniority-based one. During incidents, whoever owns the most upstream system in the failure chain becomes the primary coordinator.
The Hierarchy of System Dependencies
In their September outage, the failure chain looked like this:
- External DNS providers (outside their control)
- Internal DNS resolution (Cork team)
- Database connection routing (Cork team)
- Application server responses (Dublin team)
- Customer experience (Dublin team)
Since the root cause lived in Cork's infrastructure, Cork team lead became primary coordinator. Dublin team focused on mitigation and customer communication while Cork drove the investigation and recovery decisions.
This framework prevents the common mistake of letting customer-facing impact drive all decisions. Sometimes the fastest path to resolution requires ignoring immediate symptoms to fix underlying causes.
Tools and Workflows That Prevented Duplicate Effort
Server Scout's multi-user dashboard allowed both teams to see the same infrastructure health data without duplicating monitoring systems. Dublin team members could watch database metrics in real-time, while Cork team members tracked application performance metrics.
More importantly, they established "claim and announce" protocols for any investigative work. Before making changes or running diagnostics, team members announced their intentions in the shared channel:
"Dublin investigating zone 3 timeout patterns, ETA 10 minutes" "Cork checking database connection logs from 14:20-14:30"
This prevented two people from running the same diagnostic commands or making conflicting changes to the same systems.
The monitoring system's webhook integrations automatically posted significant threshold breaches to their incident channel, so both teams saw the same alerts without manual relay.
What We'd Do Differently Next Time
TechFlow's success came from preparing for distributed coordination before they needed it. But they identified several improvements for future incidents:
Customer communication during the outage relied too heavily on the Dublin team. Cork team members had valuable insights about recovery timelines that never reached customer-facing communications. They've since created templates that help technical team members contribute to customer updates without requiring customer service training.
Their 15-minute check-in rhythm worked well for a two-hour incident, but they're experimenting with dynamic intervals. Critical phases might need 5-minute updates, while stable investigation periods could extend to 30 minutes.
Most importantly, they realised that geographic distribution during incidents is actually an advantage when properly managed. Different locations often means different perspectives on the same problem, which accelerates both diagnosis and solution validation.
The key insight: distance doesn't make incident coordination harder - lack of preparation does. Teams that establish clear communication protocols, decision-making frameworks, and shared visibility tools before crisis strikes can turn geographic separation into a coordination strength rather than weakness.
For more guidance on building monitoring systems that support effective incident coordination, see our complete implementation guide.
FAQ
How do you prevent teams from accidentally working on the same problem during incidents?
Use "claim and announce" protocols where team members publicly state what they're investigating and for how long. Combined with dependency-based ownership (whoever owns the most upstream failing system coordinates), this prevents duplicate effort without slowing down response.
What's the optimal check-in frequency for distributed incident response?
15-minute intervals work well for most incidents, but consider dynamic timing based on incident phase. Critical recovery moments might need 5-minute updates, while stable investigation periods can extend to 30 minutes. The key is preventing both update fatigue and team isolation.
How do you maintain customer communication when technical expertise is distributed across teams?
Create communication templates that help technical team members contribute timeline and recovery insights without requiring customer service training. The team closest to the root cause often has the most accurate recovery estimates, so include their input in customer-facing updates.