Running Postgres on kubernetes locally

While setting up Kubernetes locally might seem like overkill for one off data ingestion tasks, it provides several advantages:

• Creates a consistent development environment that mirrors production
• Allows testing of Kubernetes configurations before cloud deployment
• Enables development of microservices in isolation
• Provides a foundation for scaling your ML pipeline

For this tutorial, we’ll use Docker for Mac with its built-in Kubernetes support (v1.9.8). This setup offers a straightforward development experience with modern Kubernetes tooling while maintaining compatibility with cloud deployments. Though alternatives like Docker Swarm or Compose are a few years old now, Kubernetes provides a platform for building and managing data pipelines that can help ease the transition from local development to prod deployments.