This is the third story in distributed systems, you can read the others here.

Sharding is the basis of the distribution systems, without sharding DS can simply not exists (for the most part..). Sharding is one of the easiest and trickiest things to explain and understand. It is at the crux of NoSql databases.

When i interview candidates about No-Sql and DS, one of the things i ask in detail is sharding? Many people just blunt out that dividing data is sharding, although that is technically ok, but i can also divide data into multiple Rdbms tables , does that…


So Terraform helps you create infrastructure, the way it works is we define resources that “need” to be created and TF helps you fulfill our requirements. What it does is, internally it maintains a state file and tries to reach the end state defined by us (as in our requirements for the infra) by taking a series of steps that it “figures” out itself and reaches that end/desired state.

How do we create resources in Tf, let's say we want to create an AWS IAM POLICY and attach some policy doc to it. …


I recently got a chance to work on terraform in detail. One of the problems we solved was writing secrets to different K8 namespaces using multiple providers for Kubernetes with terraform.

So the official docs have a lot of information to interact with “Kubernetes providerhere. I have used the host and token approach to interact with K8. Let’s write a very basic provider and use it to copy secrets.

You can decide how to design your code by placing different providers in a provider.tf .(trying to bring some structure in TF)

provider “kubernetes”{

}
resource "kubernetes_secret" "example" {
metadata {…


This is the second story on the topic of DS, if you haven't read the first one, please have a look here.

Replication is at the core of Distributed Systems (DS). We all know what replication is? In its simplest form, is just having a duplicate copy of the original in a case where the original copy is destroyed, we can use the backup one. But in DS replication is not that simple, in fact, it’s much more complicated and it needs to work in order for DS to ‘work’. In a system where storage for example is not distributed…


As the topic says this is distributed systems 101, so I am gonna write some very basic introduction and working of Distributed Systems which I’ll refer to as DS

So What is DS?

Let's go at it word by word. The Word Distributed means to distribute, so what exactly is distributed, it could be anything depending on the use case, for eg

  1. Storage= A case where the processing and storage are separated and your storage is distributed over logical or physical means. Let’s say you are running a query on your laptop which fetches data from different systems and then processes in your system’s…


solace running in docker on the local system

Solace is a queueing mechanism following the pub-sub(publisher-subscriber ) pattern. A producer can publish events/messages to a solace Queue/Topic and the consumer can subscribe to a topic or consume a queue as per the requirement.
The problem was to send events to Solace from HDFS (Hadoop Distributed File System) in a distributed fashion along with adhering to some other business alignments. There are multiple ways to publish and consume data from Solace. One way is via Rest Services which is pretty straight forward if we follow Solace’s Documentation. …

Harry Singh

Data Engineer 3 at Mongo DB

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store