Lessons from a CI/CD Debugging Saga: A Tale from North East India

In the ever-evolving world of software development, setting up a Continuous Integration/Continuous Deployment (CI/CD) pipeline is a significant milestone for developers. However, as our recent experience in North East India demonstrates, the smooth automation dream can sometimes transform into a long night of debugging. This article shares a real-world story of chasing down a mysterious API error and the valuable insights gained along the way.

The Setup: A Modern Tech Stack in a Self-Hosted Environment

Our tech stack consisted of Strapi, Next.js, and Docker for deployment, with a CI/CD pipeline for auto-deployments on a self-hosted Linux server. The application ran smoothly on local development and even on Vercel for testing the frontend alone. However, upon deployment on our self-hosted server, the application broke.

The Error: A Puzzling Timeout Issue

Next.js began throwing an enigmatic error when trying to fetch data from Strapi: "API or validation error: TypeError: fetch failed at async s(.next/server/chunks/507.js:1:806)". The error didn't make sense because it only occurred on the deployed version and not on local or Vercel.

The Investigation: Step-by-Step Troubleshooting

Step 1: Reproduce Locally

To start, I reproduced the issue locally by running the Next.js app with Strapi in Docker. Everything worked fine, even when using Postman or curl to test the API. This ruled out a code issue.

Step 2: Check Docker Networking

Next, I checked Docker networking to see if containers could communicate with each other. I ensured both services were on the same Docker network and updated API calls to use "http://strapi:1337". However, the error still persisted.

Step 3: Look Into the Build Artifacts

I inspected the build artifacts next, looking for any anomalies in the Next.js code. However, everything seemed normal.

Step 4: Increase Timeouts and CORS Configs

Suspecting timeout issues and CORS misconfigurations, I increased the fetch timeout in Next.js, rechecked Strapi's middleware.js for proper CORS headers, but still nothing.

Step 5: Strip Docker from the Equation

Out of desperation, I ran the entire stack without Docker on the same self-hosted server. To my surprise, it worked.

The Real Culprit: Network Restrictions on the Host Server

This confirmed that the issue wasn't with the code, Docker, or even the CI/CD pipeline. It was a network-level issue on the host server, specifically, restrictions that blocked container-to-container communication or DNS resolution for internal Docker service names.

The Fix: Real Domain Names to the Rescue

The solution was simple once I understood the problem: I configured Strapi with a real domain name, like https://api.myapp.com, and updated the frontend to fetch from that instead of http://strapi:1337. After that, I redeployed using the same Docker + CI/CD pipeline, and everything worked.

Key Takeaways

• Docker networking can behave differently across environments.
• What works locally might not work on production.
• Don't overlook your host's firewall, DNS, or networking rules.
• Real domain names can often solve connectivity issues, especially across containers and services.
• Debugging isn't always about code; sometimes, it's about the infrastructure.

This experience serves as a reminder to developers in North East India and beyond that the road to a seamless CI/CD pipeline can sometimes lead to unexpected detours. By staying persistent and adopting a systematic approach to troubleshooting, we can learn valuable lessons that help us grow as developers and deliver better software.