And then a thought hit me. What if I wanted the exact opposite?

What if, instead of scattering keys as far apart as possible, I wanted similar things to land in the same bucket?

That question turns out to have a name: Locality-Sensitive Hashing. And it turns out to be the trick behind some pretty significant systems.

The Problem That Started All of This

Think about what Google had to deal with in, say, 2007. Billions of web pages. A huge chunk of them are duplicates or near-duplicates. Mirror sites, scraped content, syndicated articles, press releases copy-pasted across a hundred news outlets. If you don't catch these, your search index bloats, your rankings get polluted, and users see the same article five times on the first page.

The naive approach is to compare every pair of documents. If you have a million documents, that is about 500 billion comparisons. Even if each comparison takes a microsecond, you are looking at roughly 6 days of compute. For a billion documents, the math just falls apart entirely.

Google needed a way to find near-duplicates without comparing every pair. And they needed it to work on a scale where even O(n log n) felt expensive.

This is also exactly the problem plagiarism detection tools face. Turnitin, Copyscape, and similar services need to take a submitted essay and check it against millions of existing documents. They can't afford to do a full text comparison against every document in their database. They need a shortcut.

The Core Insight

Here is the key idea behind LSH, and I think it is genuinely clever.

A regular hash function tries to make h("cat") != h("bat") even though the inputs differ by one character. That is the whole point of hashing. Uniform distribution, minimal collisions, all of that.

An LSH hash function does the opposite. It is designed so that similar inputs are likely to produce the same hash value. Not guaranteed. Likely. And you can tune exactly how likely.

This gives you a filter. Instead of comparing all pairs, you hash everything, and only compare documents that ended up in the same bucket. Most of the truly dissimilar pairs never even get looked at.

The tradeoff is that you might miss some similar pairs (false negatives) and you might flag some dissimilar pairs (false positives). But you can control both of those error rates by tuning parameters. And you go from O(n^2) comparisons to something much closer to O(n).

Two Flavors of the Same Idea

These are two main algorithms that I studied that fall under the LSH umbrella, and they solve slightly different problems. (Ref: Image generated by Google's Gemini)

MinHash: "Did someone copy-paste this?"

MinHash is built for detecting structural overlap. It works with sets, and it estimates something called Jaccard similarity, which is just "what fraction of elements do these two sets share?"

The setup works like this. You take a document and break it into overlapping character sequences called shingles. The sentence "the cat sat" with shingle size 5 becomes {"the c", "he ca", "e cat", " cat ", "cat s", "at sa", "t sat"}. Now your document is a set, and you can ask set overlap questions.

The problem is that these sets are huge. A typical article might produce tens of thousands of shingles. Comparing two sets directly is fine, but comparing all pairs is not.

MinHash compresses each set into a short signature (say, 128 numbers) with a nice property: the probability that two signatures agree at any position equals the Jaccard similarity between the original sets. So you compare 128 numbers instead of tens of thousands of shingles.

This is not a heuristic. There is a clean theorem behind it. If you pick a random hash function h and look at the minimum hash value across all elements of a set, then:

P( min_hash(A) = min_hash(B) ) = |A ∩ B| / |A ∪ B|

The intuition is straightforward. Pool all shingles from both documents together, randomly order them by h, and look at whichever shingle lands first. That shingle is shared by both documents with probability equal to the fraction of the pool that is shared. That fraction is exactly the Jaccard similarity. Each position in the signature is an independent trial of this experiment, so averaging over 128 positions gives you a tight estimate.

And here is where the LSH part comes in. You don't even compare all pairs of signatures. You split each 128-number signature into bands (say 16 bands of 8 numbers each) and hash each band separately. Two documents become candidates only if they match on at least one band. The math works out so that pairs with high similarity almost always match on at least one band, while pairs with low similarity almost never do.

The result is a system with a sharp threshold. Below some similarity level, pairs are almost never detected. Above it, they almost always are. And you can tune exactly where that cutoff sits by adjusting how many bands you use and how many numbers go in each band.

This is what plagiarism detectors use. If someone copy-pastes a paragraph and changes a few words, the shingle sets still overlap heavily, MinHash catches the overlap, and the LSH index surfaces the pair without having to scan the whole database.

SimHash: "Is this the same article in different words?"

SimHash solves a different problem. It was developed by Moses Charikar, and Google used a variant of it for web-scale deduplication. Where MinHash asks "do these documents share literal text?", SimHash asks "do these documents have similar content?" even if the wording is completely different.

The idea is to compress an entire document into a single 64-bit number (a fingerprint) such that similar documents produce fingerprints that differ in very few bits.

How you get there: take each word in the document, weight it by importance (common words like "the" get low weight, distinctive words get high weight), hash each word to a 64-bit value, and then do a weighted vote across all 64 bit positions. If more weight voted for a 1 in position k, the fingerprint gets a 1 there. Otherwise it gets a 0.

Two documents about the same topic, using similar vocabulary, will have fingerprints that differ in maybe 2 or 3 bits out of 64. Two completely unrelated documents will differ in around 32 bits. You measure similarity by counting the number of differing bits (Hamming distance).

Again, there is a theorem that makes this precise. It comes from a result by Goemans and Williamson, originally in a different context, but the relevant part is this: if you represent two documents as weighted vectors in some high-dimensional feature space, then the probability that a random hyperplane hash disagrees at a single bit position is:

P( bit_i(A) ≠ bit_i(B) ) = θ(A, B) / π

where θ is the angle between the two document vectors. Small angle means similar documents, and similar documents will disagree on very few bits. The expected Hamming distance between two fingerprints is 64 × θ / π. So Hamming distance is not just a rough proxy for similarity. It is a direct, linear function of the angle between the original document representations.

For the index, you use the pigeonhole principle. Split the 64-bit fingerprint into 4 blocks of 16 bits. If two fingerprints differ in at most 3 bits total, then at least one of the four blocks must be identical between them. So you build 4 separate indexes (one per block), and two documents become candidates if they match on any block.

Google used this (and variations of it) to deduplicate their web crawl. When you are indexing billions of pages and a huge fraction of them are just copies of the same content with different ads wrapped around it, SimHash lets you identify the duplicates in time proportional to the number of documents rather than the number of pairs.

Why Not Just Use Embeddings?

This is a fair question. Today, if you wanted to find semantically similar documents, you would probably reach for a sentence embedding model, throw the vectors into a vector database, and call it done.

But consider the constraints Google faced in 2007. Transformer models did not exist. Word2Vec was still six years away. The embedding models we take for granted today simply were not available.

Even today, embeddings are not always the right tool. They are expensive to compute, they require GPU infrastructure, and the vectors are large (768 or 1536 floats per document). SimHash gives you a 64-bit integer. That is 8 bytes. You can fit a billion SimHash fingerprints in about 8 GB of RAM. Try doing that with embedding vectors.

For the specific problem of "is this document a near-copy of something I have already seen?", LSH methods are still hard to beat on cost and simplicity. They don't require a model, they don't require GPUs, and they don't require a vector database. You need a hash function and some hash tables.

Building It for Real

I ended up building both pipelines from scratch to really understand how the pieces fit together. A MinHash-LSH pipeline for plagiarism detection, a SimHash-Pigeonhole pipeline for semantic deduplication, both backed by a custom on-disk hash index (because if you are already going down this rabbit hole, why stop at the algorithm and not also build the storage layer?).

I tested it against 10,000 documents: a mix of real news articles, synthetically mutated copies at various similarity levels, and noise documents. The system picks up copy-paste plagiarism with over 95% recall, and the on-disk index handles single-document queries in under 5ms.

The thing that surprised me most was how well the banded LSH approach works as a filter. Out of the roughly 50 million possible document pairs in a 10k corpus, the index narrows it down to a few thousand candidates. The verification step (which does the expensive exact comparison) only runs on those candidates. That is the real win. Not the hashing itself, but the fact that hashing lets you skip 99.99% of the work.

The Takeaway

I started with a silly question about inverting hash functions and ended up understanding how some of the largest-scale data processing systems actually work.

The core idea is worth remembering: sometimes the best way to solve a search problem is to design a hash function that is deliberately bad at being a hash function. Instead of scattering similar items apart, you scatter them together. And then you just look inside the buckets.

If you want to dig into the code, I have put the full implementation (with the custom disk-based index and all the benchmarks) on GitHub. However, given the coding agents at our disposal, I will rather ask you to giving it a try yourself. If you are still interested my implementation (that I generated using Claude), feel free to comment. Cheers! 🥂

What's This Halting Problem Anyway?

In a nutshell, the halting problem asks: "Can we write a program that can determine whether any arbitrary program will eventually finish running or continue forever?" Sounds reasonable, right? We should be able to analyze code and figure out if it'll terminate, after all both humans and computers are apparently intelligent, we should be able to predict the outcome of what we have written!

Turns out, nope. Not possible. And this isn't just me being pessimistic, this was proven by Alan Turing back in 1936. The proof is beautiful in its simplicity, and we can demonstrate it using Python.

The Python Paradox: Seeing is Believing

Let's imagine we have a magical function called halt(program, input) that can determine if a program will halt when given specific input. It returns True if the program terminates, and False if it runs forever.

Now, watch this paradox unfold:

def paradox(program_code):
    # Assuming we have a magical halt function that works
    if halt(program_code, program_code):
        # If halt predicts the program will terminate...
        # For all practicality purposes for testing, you can just have a timeout here instead.
        while True:  # Run forever!
            pass
    else:
        # If halt predicts the program will run forever...
        return  # Terminate immediately!

# Now what happens when we run: paradox(paradox)?

Do you see the contradiction? If our hypothetical halt function says paradox(paradox) will terminate, then the function enters an infinite loop (making halt wrong). If halt says it won't terminate, then the function returns immediately (also making halt wrong).

This simple example proves that our magical halt function cannot exist. It's not just that we haven't figured out how to implement it - it's mathematically impossible.

Implementation Details

I won’t be going deep into the implementation details, will rather provide you with the test file with which I had mocked the halt function to return the desired output. This helped us to cleverly show the paradox without actually implementing anything in the halt function.

import unittest
from unittest.mock import patch

from paradox import paradox_self_reference
from banner import banner, conclusion

class TestHaltingProblem(unittest.TestCase):

    def test_halting_mock_true(self):
        """
        Simulate halts(paradox) == True → paradox() should loop (i.e., timeout).
        """
        with patch('halt.halt', return_value=True) as mock_halt:
            halted = paradox_self_reference(mock_halt)
            self.assertFalse(halted, "Expected paradox() to infinitely loop. [Due to timeout, returns empty]")

    def test_halting_mock_false(self):
        """
        Simulate halts(paradox) == False → paradox() should halt quickly.
        """
        with patch('halt.halt', return_value=False) as mock_halt:
            halted = paradox_self_reference(mock_halt)
            self.assertTrue(halted, "Expected paradox() to halt, and therefore paradox() returns True")


if __name__ == "__main__":
    unittest.main()

Outcome

Real World Implications:

While the halting problem might seem like an abstract concept, it has serious practical implications:

1. Orchestration systems and job scheduling: You mentioned orchestrators, and you're absolutely right. Since we can't determine in advance whether every program will terminate, we can't precisely calculate how long each job will take. That's why orchestrators use timeouts and resource limits rather than trying to predict exact runtimes.

2. Bug detection tools: Static analysis tools that try to find infinite loops or deadlocks are fundamentally limited. They may find some problems, but they cannot detect all possible non-terminating executions.

3. Compiler optimizations: Some optimizations require knowing if certain code sections terminate. Compilers must use conservative approximations, potentially missing optimization opportunities.

Conclusion

While you won’t frequently come across scenarios where you need to worry about the halting problem, it still shows that we have a far way to go in deterministically define the outcome of problems with the help of mathematics. Surely, long way to go in the evolution of humans!

Resolving conflicting dependencies can be a real headache in python applications. Different parts of the codebase might require different versions of the same package, creating a seemingly impossible situation. While virtual environments help with project-level isolation, they don't solve the problem of conflicting dependencies within a single application. In this article, we'll explore a proof-of-concept approach that uses Python's subprocess mechanism to run specific functions with isolated dependencies. This is purely for exploration and is not at all recommended to be productionized, unless you are ready for late night pagers!

Glossary

Subprocess: A separate process spawned by a parent process, with its own memory space and environment
Runtime Environment: The set of dependencies available to a function when it executes
Dependency Isolation: Running code with specific dependencies regardless of what's installed in the parent process

The Challenge

Consider this scenario: one part of your application needs requests==2.22.0 while another requires requests==2.32.2. How do you satisfy both requirements within the same Python process? The initial thoughts are:

1. Compromise: Choose one version and hope everything works

2. Refactoring: Rewrite code to work with a single version

Let us explore a third option: run specific functions in isolated subprocesses with their own dependencies.

The Subprocess Solution

The core idea is: when a function needs specific dependencies, we:

1. Spawn a new Python subprocess

2. Install the required dependencies in a temporary directory

3. Execute the function code in this subprocess

4. Capture the result and return it to the parent process

This approach leverages the fact that subprocesses have their own memory space and Python environment, allowing for true isolation without affecting the parent process.

Key Components

Our solution consists of two main parts:

1. A decorator (runtime_env) that wraps functions to execute in isolated subprocesses

2. A subprocess runner that handles dependency installation and function execution

Here's how the decorator is used:

from decorator import runtime_env

@runtime_env("requirements.txt")
def process_data(data, factor=1):
    # This function runs in a subprocess with dependencies from requirements.txt
    import requests  # Will use version specified in requirements.txt

    # Process data...
    result = {
        "original": data,
        "processed": {"name": data["name"].upper(), "value": data["value"] * factor},
        "metadata": {"requests_version": requests.__version__}
    }

    return result

How It Works

When the decorated function is called, the following sequence happens:

1. Function identification: The decorator captures the source code of the module containing the function

2. Argument serialization: Function arguments are serialized using Python's pickle module

3. Subprocess creation: A new Python process is started, running our subprocess runner

4. Dependency installation: The subprocess installs required packages in a temporary directory

5. Function execution: The function runs in the isolated environment with its own dependencies

6. Result communication: The result is serialized and passed back to the parent process

7. Cleanup: Temporary directories and files are removed

Example in Action

Let's explore a concrete example. The parent process might have requests==2.26.0 installed, while we want a function to run with requests==2.28.0:

# Parent process
def get_data():
    try:
        import requests
        print(f"Parent process using requests version: {requests.__version__}")
    except ImportError:
        print("Requests not available in parent")
    return {"name": "Test Data", "value": 42}

# This will run in a subprocess with its own dependencies
@runtime_env("requirements.txt")  # Contains requests==2.28.0
def process_data(data):
    import requests
    print(f"Subprocess using requests version: {requests.__version__}")
    return {"result": data, "version_used": requests.__version__}

# Main execution
input_data = get_data()
result = process_data(input_data)
print(f"Process completed with requests version: {result['version_used']}")

When executed, we see different versions of the requests library being used in each process, demonstrating successful isolation. Here’s a screenshot from the PoC that we did:

As you can see in the example above, the parent process PID is 1, where as the subprocess PID is 7, and for both the parent process and subprocess, the requests library version is different.

Behind the Scenes

I won’t be delving in the exact approach here, I would let you take a stab at it, and yes, use LLMs to help you solution! However, here’s a summary of the approach at a very high level:

• The decorator first identifies the module to which the function belongs, and get the module via the sys package, and uses the inspect module to fetch the source file. We need the module in order to ensure that we can also use the imports used for the decorated function.

• Once we have the code information, we use the subprocess module to create a new process, and we create a new temporary directory in which we install the required dependencies

• Now we execute the function and return the output back to the caller process.

Only for fun, not for production!

This approach is only for educational purposes, the primary reason being, this approach is very naive, and there are a lot of aspects to cover even to make it dev ready, for example, the protocol or mechanism used to communicate the results back to the parent process. What about concurrent processes creating new isolated envs? What about common dependencies between the parent and the child process? And what not!

Explore something production ready?

To checkout an implementation that deals with conflicting dependencies you can read about Ray's Runtime Environments feature, which provides a more robust, optimized approach to dependency isolation with additional features like package caching and environment reuse.

Conclusion

Python’s subprocess is a powerful module. The example above reveals the power of process isolation as a solution to conflicting dependencies, a common challenge in modern Python development. While this is a proof-of-concept, nevertheless, understanding the underlying principles of process isolation provides valuable insight into how dependency conflicts can be managed in complex applications.

If you want to explore the source code, feel free to get in touch! However, I strongly recommend doing it yourself, with LLMs at your disposal, should be an hour of effort!

Ever wondered why you are always told that numpy performs better than python lists? Well, it circles back to dynamic + reference based arrays vs static arrays.

In this article, we will understand it by exploring how Python's implementation of dynamic lists affects cache performance and why NumPy arrays can be dramatically faster for numerical operations.

Glossary

• L1 Cache (D1): Smallest, fastest cache directly connected to the CPU

• Last-Level Cache (LL): Larger but slower cache before accessing main memory

• Cache Lines: Data is loaded in fixed-size blocks (typically 64 bytes), not individual values

• Spatial Locality: The principle that after accessing one memory location, programs are likely to access nearby locations soon

Python Lists: Under the Hood

Python's lists are implemented as dynamic arrays of references. This means that when you create a list like [1, 2, 3], what you're actually getting is an array of pointers, each pointing to a separate Python object stored somewhere else in memory:

List in memory:
[ptr] → PyObject(1)
[ptr] → PyObject(2)
[ptr] → PyObject(3)

Each PyObject carries type information, reference counts, and the actual value, requiring 16+ bytes per integer on 64-bit systems. When the list grows beyond its capacity, Python allocates a new, larger array (typically 1.125x the size until 9 elements, then 2x thereafter), copies over the pointers, and frees the old array.

NumPy Arrays: The Contiguous Approach

In contrast, NumPy arrays store values directly in a contiguous memory block:

NumPy array in memory:
[1][2][3][4][5]...

This approach sacrifices Python's flexibility (mixing types) for dramatic performance improvements with homogeneous data.

Two Approaches Compared

Let's implement Python-like and NumPy-like structures in C++ to demonstrate the difference.

Python Lists (Dynamic Arrays)

// Python-like list implementation (simplified)

class PythonList {
private:
    T* array;
    int capacity;
    int size;

    void resize() {
        capacity *= 2;
        T* newArray = new T[capacity];
        for (int i = 0; i < size; i++) {
            newArray[i] = array[i];
        }
        delete[] array;
        array = newArray;
    }

public:
    PythonList() : capacity(1), size(0) {
        array = new T[capacity];
    }

    ~PythonList() {
        delete[] array;
    }

    void append(T value) {
        if (size == capacity) {
            resize();
        }
        array[size++] = value;
    }

    T get(int index) const {
        if (index < 0 || index >= size) {
            throw std::out_of_range("Index out of range");
        }
        return array[index];
    }

    int getSize() const {
        return size;
    }

    int getCapacity() const {
        return capacity;
    }
};

Dynamic lists resize when the initial defined size, and then subsequently, double their size. In simple words, [capacity reached → create new array double the size → copy elements → repeat]

Extras!

When a Python list runs out of capacity, it needs to resize. This resizing operation requires:
1. Allocating a new, larger array (typically 2× the previous size)
2. Copying all existing elements to the new array
3. Deallocating the old array

This resize operation is expensive—O(n) where n is the number of elements. However, it occurs infrequently. If we've performed n append operations, resizing has only happened approximately log₂(n) times. The total cost after n operations is approximately: O(n) + O(n·log₂(n)) = O(n)

When divided by n operations, the amortized cost per operation becomes: O(n)/n = O(1)

Numpy (Similar to Primitive Arrays)

This approach sacrifices flexibility for performance, as the size and the data type cannot be changed.

// NumPy-like array implementation (simplified)
template <typename T>
class NumpyArray {
private:
    T* array;
    int size;

public:
    NumpyArray(int n) : size(n) {
        array = new T[size];
    }

    ~NumpyArray() {
        delete[] array;
    }

    void set(int index, T value) {
        if (index < 0 || index >= size) {
            throw std::out_of_range("Index out of range");
        }
        array[index] = value;
    }

    T get(int index) const {
        if (index < 0 || index >= size) {
            throw std::out_of_range("Index out of range");
        }
        return array[index];
    }

    int getSize() const {
        return size;
    }
};

Both serve similar purposes but have fundamentally different memory layouts.

Analyzing Cache Performance

Let's examine what happens when we iterate through both structures to calculate a sum:

// PythonList iteration
long long sum = 0;
for (int i = 0; i < pyList.getSize(); i++) {
    sum += pyList.get(i);  // Each access may cause a cache miss
}

// NumpyArray iteration
long long sum = 0;
for (int i = 0; i < npArray.getSize(); i++) {
    sum += npArray.get(i);  // Sequential access has better locality
}

Cache Performance Results

Using Valgrind's cachegrind tool, we measured cache misses for both implementations across different sizes and performance:

1. Numpy’s execution time is less than list as the dataset size increases.

   

2. D1 and LLd cache misses are more for python’s dynamic list implementation

   

   

The Spatial Locality Advantage

NumPy outperforms Python lists because:

Contiguous Storage: When a cache line is loaded for array[i], it also contains array[i+1], array[i+2], etc., which will be used immediately afterward.

Fewer Memory Indirections: Python lists require a pointer dereference for each access, potentially bringing in a cache line that contains only one useful value.

Better Prefetching: Modern CPUs can predict sequential access patterns and prefetch data, but struggle with the scattered memory access of pointer chasing.

Cache Misses Explained

When iterating through a Python list, each element access follows this pattern:

1. Load pointer from list array (potential cache miss #1)

2. Follow pointer to actual object (potential cache miss #2)

3. Extract value from object (potential cache miss #3)

4. Move to next element (which may be in a completely different memory area)

For NumPy arrays, the access pattern is:

1. Calculate position (base address + i * element_size)

2. Access value directly (adjacent to previous value)

3. Move to next element (likely in the same cache line)

With Python lists generating significantly more cache misses, the performance impact is enormous.

Conclusion

The post is not about how lists are internally implemented in python, but rather it is an attempt to spark your curiosity to understand how things work understand the hood, and understand the trade-offs. For the ease of use, and to provide flexibility to have different types of elements in the list, python has traded it off with performance. Hope you found this insightful!

Computers use a hierarchy of memory systems to balance speed and capacity. The fastest memory (cache) sits closest to the CPU but is limited in size. When data isn't found in cache, it must be fetched from slower memory, creating a "cache miss" that significantly slows execution. In this article we will explore how spatial locality can have significant impact on the performance of your algorithm leveraging dynamic programming techniques.

Glossary

• L1 Cache (D1): Smallest, fastest cache directly connected to the CPU

• Last-Level Cache (LL): Larger but slower cache before accessing main memory

• Cache Lines: Data is loaded in fixed-size blocks (typically 64 bytes), not individual values

• Spatial Locality means that after accessing one memory location, programs are likely to access nearby locations soon, and therefore, having these locations in the cache can speed up the operations.

The Two DP Approaches

Our example calculates Fibonacci numbers using two approaches:

1. Memoization: Stores previously calculated values in an array but accesses them in a somewhat random pattern through recursion.

    long long fib_memoization_array(int n, std::vector<long long>& memo) {
        if (n <= 1) return n;
   
        if (memo[n] != -1) {
            return memo[n];
        }
   
        memo[n] = fib_memoization_array(n-1, memo) + fib_memoization_array(n-2, memo);
        return memo[n];
    }
   

2. Tabulation: Builds the solution incrementally, accessing memory in a highly sequential pattern

    long long fib_tabulation(int n) {
        if (n <= 1) return n;
   
        std::vector<long long> dp(n+1, 0);
        dp[0] = 0;
        dp[1] = 1;
   
        // Sequential access pattern - excellent spatial locality
        for (int i = 2; i <= n; i++) {
            dp[i] = dp[i-1] + dp[i-2];
        }
   
        return dp[n];
    }
   

Both solve the same problem, but their memory access patterns differ dramatically.

Setup

For our cache performance experiments, we're using Docker to run Valgrind's cachegrind tool on macOS with an M3 chip. Since Valgrind does not support Apple Silicon chips, therefore, we used Docker to spin up a linux instance.

Analyzing Cache Performance

Memoization

root@62969aff3a5f:/workspace# valgrind --main-stacksize=10000000000 --tool=cachegrind \
                                ./access_test memoization

Time taken for 1 runs: 20404 ms
==224== 
==224== I   refs:      4,251,823,292
==224== I1  misses:            2,032
==224== LLi misses:            1,674
==224== I1  miss rate:          0.00%
==224== LLi miss rate:          0.00%
==224== 
==224== D   refs:      2,950,712,351  (1,600,511,544 rd   + 1,350,200,807 wr)
==224== D1  misses:      187,518,436  (   87,515,735 rd   +   100,002,701 wr)
==224== LLd misses:      187,505,766  (   87,504,607 rd   +   100,001,159 wr)
==224== D1  miss rate:           6.4% (          5.5%     +           7.4%  )
==224== LLd miss rate:           6.4% (          5.5%     +           7.4%  )
==224== 
==224== LL refs:         187,520,468  (   87,517,767 rd   +   100,002,701 wr)
==224== LL misses:       187,507,440  (   87,506,281 rd   +   100,001,159 wr)
==224== LL miss rate:            2.6% (          1.5%     +           7.4%  )

Tabulation

root@62969aff3a5f:/workspace# valgrind --tool=cachegrind ./access_test tabulation
Time taken for 1 runs: 1620 ms
==225== 
==225== I   refs:      751,823,239
==225== I1  misses:          2,031
==225== LLi misses:          1,673
==225== I1  miss rate:        0.00%
==225== LLi miss rate:        0.00%
==225== 
==225== D   refs:      250,712,360  (100,511,549 rd   + 150,200,811 wr)
==225== D1  misses:     25,018,742  (     15,990 rd   +  25,002,752 wr)
==225== LLd misses:     25,010,403  (      8,689 rd   +  25,001,714 wr)
==225== D1  miss rate:        10.0% (        0.0%     +        16.6%  )
==225== LLd miss rate:        10.0% (        0.0%     +        16.6%  )
==225== 
==225== LL refs:        25,020,773  (     18,021 rd   +  25,002,752 wr)
==225== LL misses:      25,012,076  (     10,362 rd   +  25,001,714 wr)
==225== LL miss rate:          2.5% (        0.0%     +        16.6%  )

Let's examine the cachegrind output:

Execution Time Comparison

• Memoization: 20,404 ms

• Tabulation: 1,620 ms (12.6× faster!)

Cache Statistics Breakdown

While tabulation shows a higher D1 miss rate percentage (10.0% vs 6.4%), but look at the absolute number of cache misses is dramatically lower because it performs far fewer operations overall. There is a ~7.5x increased in the D1 cache misses in the memoization approach, which can create a big difference highly latency sensitive applications.

The Spatial Locality Advantage

Tabulation outperforms memoization because:

1. Sequential Access Pattern: Tabulation accesses array elements in order (dp[0], dp[1], dp[2]...), which perfectly aligns with how cache lines are loaded and thus eliminates redundant calculations, requiring far fewer instructions and memory accesses.

2. More Efficient Cache Usage: When a cache line is loaded for dp[i], it also contains dp[i+1], dp[i+2], etc., which will be used immediately afterward, whereas in memoization, this does not hold true, because of the recurrence.

Cache Misses Explained

In the memoization approach, the recursive calls create a less predictable access pattern. When calculating fib(n), the function might jump to calculating fib(n-2) before fib(n-1) is fully resolved, causing scattered memory access that defeats the cache's prefetching mechanisms.

Each cache miss can cost 100-300 CPU cycles compared to 1-3 cycles for a cache hit. With memoization generating 7.5× more cache misses, the performance impact is enormous.

Conclusion

This example perfectly demonstrates why understanding spatial locality matters for performance-critical code. Though both algorithms have the same mathematical complexity, the memoization approach's inferior cache utilisation and extensive call stack usage makes it over 12 times slower.

When designing algorithms, especially for dynamic programming problems, consider not just the time complexity but also how your data access patterns interact with the cache. Sequential access patterns will almost always outperform scattered access, particularly for large datasets that exceed cache size.

P.S. For those wondering why the cover photo of the article features lilies, here’s an extra byte for you: Lilies have a Fibonacci number of petals!

Isn’t it very common to hear about the worst-case time complexity of an algorithm? Well, let’s hear the math behind that!

Let’s say we have two functions: f(x) and g(x). Big-O notation is a way to describe how fast functions grow, especially as x becomes really, really large.

Set Theory

To put it up in simple words, the notation O(g(x)) represents a family of functions that grow no faster than g(x), up to constant factors and for large enough x.

So, when we write:

$$|f(x)| \in O(g(x)) \text{ we are saying that } f(x) \text{ belongs to this family, such that } $$

$$f(x) \text{ doesn’t outgrow } g(x) \text{ as } x \to \infty$$

Calculus

Now let’s back that up with some actual math.

We say f(x) ∈ O(g(x)) if there exist positive constants C and x₀ such that:

$$|f(x)| ≤ C\cdot|g(x)| \text{ for all } x > x_0$$

Or, in terms of limits:

$$\lim_{x \to \infty} \frac{|f(x)|}{|g(x)|} < \infty$$

This tells us that the ratio of f(x) to g(x) stays bounded. In other words, f(x) may grow, but it’s never going to grow faster than some constant multiple of g(x) beyond a certain point. For sufficiently large values of x, f(x) will always be smaller than g(x).

Example

Let us use the following example to understand it better:

$$\text{Prove that: } 2^{n} \text{ is upper bounded by } e^n \text{ i.e. } 2^n \in O(e^n)$$

Let us test with the limit, as n → infinity,

$$\lim_{n \to \infty} \frac{f(n)}{g(n)} = \lim_{n \to \infty} \frac{2^n}{e^n} = \lim_{n \to \infty} (\frac{2}{e})^n [\text{ By power of quotient rule}]$$

$$\text{ Since } e\sim2.718 > 2 \text{, therefore, } \frac{2}{e} < 1. \text{ Therefore as } n \to \infty \text{ , } ({\frac{2}{e}})^n \to 0$$

$$ \text{Therefore, } \lim_{n \to \infty} \frac{f(n)}{g(n)} = 0 < \infty. \text{ Hence, } 2^n \in O(e^n)$$

From this, it becomes clear that 2^n belongs to the set O(e^n), which means it is one of the functions in the family of functions that do not grow faster than e^n as n → infinity.

Conclusion

That’s all Big-O is: a mathematical way to group functions into families based on their long-term growth rates. It gives us a high-level idea of performance, complexity, or behaviour—without sweating the exact details. Similarly, you can later explore Big-Omega and Big-Theta to complete the picture!

Introduction

I am sure that if you use Kubernetes for container orchestration, you must have used either minikube or kind for running k8s locally. When first faced with the challenge of testing a few loading balancing scenarios, I chose minikube, as I had used it while learning kubernetes. However, this time, I found installation issues on my apple silicon chip laptop. Therefore, I started to look for alternatives, and there’s where I came across kind.

FYI, I haven’t extensively used both minikube and kind. The following are just my observations based on my usage and research.

In essence, Kubernetes cluster consists of a control plane (master node), and a set of worker nodes. The control plane consists of scheduler, controller, api-server, etc. The worker node consists of, CRE (container runtime engine), that manages and runs containers. kube-proxy helps manage service networking, while a Container Network Interface (CNI) plugin (such as Flannel or Calico) enables inter-node communication. Kubelet, the agent that ensures the node is running the assigned workload.

As you can see for Kubernetes itself to function we need to have quite a few components working in sync. While majority of the cloud providers already provide Kubernetes setup with ease, testing kubernetes can be challenging locally.

Here’s where kind comes into the picture, Kind enables you to create a Kubernetes cluster where each node is a separate container running inside Docker. While Kind uses Docker to launch the cluster, each Kubernetes node inside the cluster runs its own container runtime (usually containerd). Kind manages their networking and lifecycle to simulate a real Kubernetes cluster.

Kind

As you can find in the documentation, kind stands for Kubernetes in Docker. To put in simple words, kind is a tool that let’s you run Kubernetes inside container. How ironical! A container to run a tool whose sole purpose is to orchestrate other containers! The master node and worker nodes are provisioned as separate docker containers when a cluster is created via kind.

If you look through the code base of kind, you will find that kind at its core, internally, just executes docker commands (if docker provider is used).

func createContainer(name string, args []string) error {
    return exec.Command("docker", append([]string{"run", "--name", name}, args...)...).Run()
}

This is what makes kind an interesting lightweight tool as it simplifies cluster management by using Docker to create nodes, but still manages cluster configuration, networking, and lifecycle using Kubernetes tools like kubeadm. The image below is a rough illustration of what kind does, it does not show the true working, and is only for representational purposes.

Code Highlights

Since I don’t want this article to be a tutorial, I am attaching the references to the code bits that I found interesting wrt Kind.

1. Provider Interface (provider.go)

   At the time of writing this article, Kind support three types of CREs, docker, nerdctl and podman. Kind uses the provider pattern (or the strategy pattern for the OO folks). Simple, yet powerful design pattern that makes any provider pluggable.

2. Docker Provider (docker/provider.go)

   This is the concrete implementation of provider interface for docker. In this file, you can find how nodes are created from a base image.

3. Kind CLI (cmd/kind/root.go)

   The kind CLI tool internally figures out which provider to use (docker, podman, etc.) and goes on the run the commands against the concrete implementation for that provider. CLI supports the following top level subcommands:

    // add all top level subcommands: build, completion, create, delete, export, get,
    // version and load
    cmd.AddCommand(build.NewCommand(logger, streams))
    cmd.AddCommand(completion.NewCommand(logger, streams))
    cmd.AddCommand(create.NewCommand(logger, streams))
    cmd.AddCommand(delete.NewCommand(logger, streams))
    cmd.AddCommand(export.NewCommand(logger, streams))
    cmd.AddCommand(get.NewCommand(logger, streams))
    cmd.AddCommand(version.NewCommand(logger, streams))
    cmd.AddCommand(load.NewCommand(logger, streams))
   

Another interesting thing to explore is how kubeadm initialisation happens, but I will leave that as a homework for the readers!

Images in the kind cluster

Kind is unable to automatically access to locally built images because it does not have direct access to the Docker daemon on your host. But the interesting thing is that using the following command, you can load your locally built image into the kind cluster:

kind load docker-image kind.local/image-name:image-tag --name kind-cluster-name

where kind.local/ is a custom namespace for local images in the kind cluster. The kind load command allows users to push images from the host’s Docker daemon to the Kind cluster, making them accessible to nodes inside the containers

Conclusion

Overall, I feel that kind successfully caters to the following needs: fast cluster spin-up, lightweight resource usage, and easy CI/CD integration, and without a doubt can be used for these use-cases.

Regardless of your choice of tool, kind or minikube, understanding how Kubernetes works internally, especially concepts like kubeadm, container runtimes, and networking, is crucial for effectively managing clusters in production. Here is where understanding such tools and having a look at the source code can help you develop better understanding. I also find the code of kind to be clean and can be used to learn a couple of design patterns. So the next time you want to test Kubernetes locally, do have a look at kind, not just at kind but its source code as well!

This also addresses one of the most common questions that I get, "Can I recover data from a hard reset?"

Disaster Recovery

Let us consider the following as your commit history:

a (first-commit) <- b <- c <- d <- e <- f (latest-commit)

Now you realise that the commits, d, e, f, were incorrect, and you no longer want them. So, you have two options now:

1. Revert these commits individually. The commit history will look something like this:

    (first-commit) <- b <- c <- d <- e <- f <- ~f <- ~e <- ~d (latest-commit),
   
    Where, ~f is the revert of f
           ~e is the revert of e
           ~d is the revert of d.
   

   You must have noticed here that you have to revert the changes from the latest to the oldest. So, you should always first revert the latest commit, and then move to the older ones. The command to revert the git commit is:

    git revert <commit-id>
   

   The advantage of this is that the history is preserved, and if at some later point in time, you realise that you needed the commit, you can simply refer to the commit ID and get back the changes that were made in those commits.

2. Delete the commits from the timeline itself, which deletes them from the history itself, as if they never existed. This is the one we will be looking at in detail in the next section.

Git RESET

As mentioned above, we have the option of deleting commits itself. There are two types of RESET that git supports:

1. Soft Reset:

   • git reset --soft <commit>

   • A soft reset does not modify the working directory or staging area. It only moves the branch pointer to the specified commit, effectively "undoing" the commits that came after that commit.

   • The changes from the undone commits are retained in the staging area. You can then make further modifications and create a new commit that incorporates both the retained changes and your new modifications.

Example:

    ➜  time-travel-with-git git:(main) git log --oneline
    892665b (HEAD -> main) main-branch commit to merge
    9393d41 Add merge.txt
    b4f560a (soft-delete) Second commit to time travel!
    cc3a96b First stop in time-travel

    ➜  time-travel-with-git git:(main) git reset --soft b4f560a
    //ON SOFT RESET, THE CHANGES MADE AFTER THE b4f560a COMMIT
    //ARE MOVED IN THE STAGING AREA.
    ➜  time-travel-with-git git:(main) ✗ git status
    On branch main
    Changes to be committed:
      (use "git restore --staged <file>..." to unstage)
            new file:   merge.txt

    //THE TOP TWO COMMITS ARE COMPLETELY REMOVED FROM THE HISTORY.
    ➜  time-travel-with-git git:(main) git log --oneline
    b4f560a (HEAD -> main) Second commit to time travel!
    cc3a96b First stop in time-travel

1. Hard Reset:

   • git reset --hard <commit>

   • A hard reset, on the other hand, not only moves the branch pointer but also modifies the working directory and staging area to match the specified commit. It discards changes in both the working directory and the staging area, effectively resetting the branch to the specified commit.

Example:

    ➜  time-travel-with-git git:(hard-reset) git log --oneline
    892665b (HEAD -> hard-reset, main) main-branch commit to merge
    9393d41 Add merge.txt
    b4f560a (soft-delete) Second commit to time travel!
    cc3a96b First stop in time-travel

    ➜  time-travel-with-git git:(hard-reset) git reset --hard b4f560a
    HEAD is now at b4f560a Second commit to time travel!

    //HARD RESET COMPLETELY DELETES THE COMMITS FROM THE HISTORY. 
    //THE CHANGES ARE LOST FOREVER
    ➜  time-travel-with-git git:(hard-reset) git status
    On branch hard-reset
    nothing to commit, working tree clean

    //THE COMMITS ARE LOST FOREVER
    ➜  time-travel-with-git git:(hard-reset) git log --oneline
    b4f560a (HEAD -> hard-reset, soft-delete) Second commit to time travel!
    cc3a96b First stop in time-travel

It is evident from the above explanation that hard reset is an irrecoverable damage. But is it really for someone who is a time-traveller? Aren't we a Git time-travellers?

Reflog

"reflog" (reference log) is a mechanism that records the history of changes to Git references in the local repository. References in Git include branches (refs/heads/), remote branches (refs/remotes/), and tags (refs/tags/). The reflog is particularly useful for recovering lost commits, branches, or other changes. The reflog entries have a limited lifetime, and older entries are eventually pruned by Git's garbage collection mechanism. The default expiration time is 90 days, but it can be configured.

Let us see what does reflog show us:

➜  time-travel-with-git git:(hard-reset) 
b4f560a (HEAD -> hard-reset, soft-delete) HEAD@{0}: reset: moving to b4f560a
892665b (main) HEAD@{1}: checkout: moving from main to hard-reset
892665b (main) HEAD@{2}: checkout: moving from soft-delete to main
b4f560a (HEAD -> hard-reset, soft-delete) HEAD@{3}: reset: moving to HEAD
b4f560a (HEAD -> hard-reset, soft-delete) HEAD@{4}: reset: moving to b4f560a
892665b (main) HEAD@{5}: checkout: moving from main to soft-delete
892665b (main) HEAD@{6}: checkout: moving from soft-delete to main
b4f560a (HEAD -> hard-reset, soft-delete) HEAD@{7}: reset: moving to HEAD
b4f560a (HEAD -> hard-reset, soft-delete) HEAD@{8}: reset: moving to b4f560a
892665b (main) HEAD@{9}: checkout: moving from main to soft-delete
892665b (main) HEAD@{10}: commit: main-branch commit to merge
9393d41 HEAD@{11}: checkout: moving from merge-branch to main
adb0db6 (merge-branch) HEAD@{12}: commit: Merge-branch commit to merge
9393d41 HEAD@{13}: checkout: moving from main to merge-branch
9393d41 HEAD@{14}: commit: Add merge.txt
b4f560a (HEAD -> hard-reset, soft-delete) HEAD@{15}: checkout: moving from timeline1 to main
0c8e8d9 (timeline1) HEAD@{16}: checkout: moving from main to timeline1
b4f560a (HEAD -> hard-reset, soft-delete) HEAD@{17}: checkout: moving from timeline1 to main
0c8e8d9 (timeline1) HEAD@{18}: checkout: moving from 01b3b247a53a68b399494b392003c81d0ea63087 to timeline1
01b3b24 HEAD@{19}: checkout: moving from timeline1 to 01b3b247a53a68b399494b392003c81d0ea63087
0c8e8d9 (timeline1) HEAD@{20}: checkout: moving from main to timeline1
b4f560a (HEAD -> hard-reset, soft-delete) HEAD@{21}: checkout: moving from timeline1 to main
0c8e8d9 (timeline1) HEAD@{22}: commit: Second commit on timeline1
01b3b24 HEAD@{23}: commit: First commit on timeline1
b4f560a (HEAD -> hard-reset, soft-delete) HEAD@{24}: checkout: moving from main to timeline1
b4f560a (HEAD -> hard-reset, soft-delete) HEAD@{25}: commit: Second commit to time travel!
cc3a96b HEAD@{26}: commit (initial): First stop in time-travel

Woah! There's a lot in here. But what we are particularly interested in is the latest commit that we had, which we deleted. It is 892665b. Here comes the interesting recovery step:

➜  time-travel-with-git git:(hard-reset) git checkout -b recover-from-hard-reset 892665b
Switched to a new branch 'recover-from-hard-reset'
➜  time-travel-with-git git:(recover-from-hard-reset) git status
On branch recover-from-hard-reset
nothing to commit, working tree clean
➜  time-travel-with-git git:(recover-from-hard-reset) git log --oneline
892665b (HEAD -> recovered-from-hard-reset, main) main-branch commit to merge
9393d41 Add merge.txt
b4f560a (soft-delete, hard-reset) Second commit to time travel!
cc3a96b First stop in time-travel

Congratulations, we have magically recovered everything!

Conclusion

However, do remember that git reflog is only available locally, i.e. only on your machine, and has a limited lifetime. So it is always better to be cautious about hard resets. But, just in case, if you ever mess up, remember, that you are an expert git time-traveller. Recover, travel and build!

Note: In this article we are not going to cover the types of merging like fast-forward merge, three-way merge, etc. We also won't cover concepts like squash merge or rebasing. There are plenty of great resources availble for these concepts.

What we do cover in this article is the rationale behind how git merges branches and indentifies what files have changed.

Before getting started with how git does merging. Let us start with something much simpler, how does git identify what files have changed when we stage changes?

Index

Git identifies changes in the working directory by maintaining an index, also known as the staging area. The index acts as an intermediary between our working directory and the repository, and it helps Git determine which changes should be included in the next commit. When we modify a file in our working directory, Git detects these changes, but they are not automatically committed. Instead, Git relies on the index to keep track of the staged changes. The index essentially serves as a snapshot of the working directory, containing information about the state of each file at the time of the last commit.

Git recognizes files that have changed even when they are not staged by utilizing the information stored in the index file within the .git directory. The index maintains a record of the files and their respective states, helping Git efficiently track modifications. When you we commands like git status or git diff, Git consults the index to compare the current state of the files in your working directory with the last committed state. This allows Git to highlight any differences and provide insights into which files have been modified but not yet staged. By leveraging the index, Git ensures a systematic and organized approach to managing changes, allowing developers to selectively stage and commit modifications.

Let us have a look at our index file present in the .git folder.

➜  time-travel-with-git git:(main) cat .git/index
DIRe??[
?k?e??[
?k?'????︚A?
g?Ͷ??imW?first_file.txte??82V
                             6e??82V
                                    6'???????WLJ?o96?
                                                     ???Bt+folder1/fourth_file.txte??m?gGe??m?gG'??????TK????A2C?n?-[V??%folder1/sub-folder11/seventh_file.txte??&12`e??&12`'?h???rA?#z?cYM?P!^??#folder1/sub-folder11/third_file.txte??MZ)De??MZ)D'??????????;gv?[[?;??1E?folder2/fifth_file.txte??\6|??e??\6|??'??????R8???ÉM{??$u?5?ׯ?folder3/sixth_file.txte?CZ    ??Ve?CZ    ??V(?b????ez??xe?<}?+n?n?x    merge.txte??5Ԇ8e??5Ԇ8'???????z?0?s???I?,_h*?sDsecond_file.txtTREE?8 3
???
   ??݋?`g#??fI?\cfolder13 1
D?@n4I???*N????sub-folder112 0
FJ"^?????6}???$?\?T?folder21 0
?2)A????s
         ?֞??32?folder31 0
 C?)?jƦz???%

The contents of the index file are not human-readable, and there isn't a direct way to decode its contents. However, we can see the current files being tracked in the index by using the following command: git ls-files --stage.

➜  time-travel-with-git git:(main) git ls-files --stage
100644 efb89ab841a90e0a67fbcdb6cad8696d0357e715 0    first_file.txt
100644 c6c3571f4c4a10c36f3936bc0be783d2f742742b 0    folder1/fourth_file.txt
100644 84cfff544bbf9ea894413243bc6e882d5b56afb5 0    folder1/sub-folder11/seventh_file.txt
100644 72418f23007afa1d63594d8e50215ed51b84ec03 0    folder1/sub-folder11/third_file.txt
100644 f5a688aed83b1c6776885b5b05c13ba6ca3145d1 0    folder2/fifth_file.txt
100644 a952388abaf3c3894d7bbab02475a0359dd7afed 0    folder3/sixth_file.txt
100644 e965047ad7c57865823c7d992b1d046ea66edf78 0    merge.txt
100644 e7e57ac6308173dec7dc1749b72c5f682aa67344 0    second_file.txt

As you see, the current index contains all the files of our working repository. Now let us examine any of the object IDs that are shown in the output.

➜  time-travel-with-git git:(main) git cat-file -t efb89ab841a90e0a67fbcdb6cad8696d0357e715
blob
➜  time-travel-with-git git:(main) git cat-file -p efb89ab841a90e0a67fbcdb6cad8696d0357e715
Hello, for the first time!

Each of the objects shown in the previous output was a reference to the blobs of the files in the working directory. Using this as a reference, git can identify if the files have been updated or not.

Bonus:
Shouldn't there be a way to see the changes introducted in commits if there is a way to compare unstaged / staged changes?
Well, there is another command in git, git show, that displays the information about a specific commit, including the changes introduced in that commit. When we run git show followed by a commit hash or a branch name, Git internally compares the specified commit with its parent commit (the previous state). It then shows the differences, or changes, between the two commits.

Now let us move on to the merging of branches.

Merging

The process of combining the changes from different timelines into a single timeline is called merging. We know that git knows what changes were introduced in commits in the given timeline. Now let us understand, how git understands which files to update, and what happens in case of the same files being updated in both timelines, or worse, if the same content is updated in both timelines.

When merging two branches in Git, we identify the changes made on each branch since they diverged. Git uses a three-way merge algorithm to combine changes from the common ancestor (base commit) to the tips of the two branches being merged. Let us try to understand step by step how a three-way merge takes place:

1. Git identifies the common ancestor commit of the branches being merged. This commit represents the last point where the branches share the same commit history.

2. Now git identifies the changes made on each branch independently. This involves comparing the common ancestor with the tips of the two branches to find the modifications made on each side.

3. The three-way merge combines the changes from the common ancestor to the branch tips. It considers the changes made on both branches and attempts to automatically reconcile them.

   1. If the changes made to the files are the same on both branches (even though the commits are different), Git recognizes this as a non-conflicting scenario.

   2. Git performs a content-level merge, applying the shared changes to the files from both branches.

   3. If Git encounters conflicting changes (i.e., changes made on the same lines of the same file), it marks those files as conflicted. Git requires manual intervention to resolve these conflicts. In this case, git can no longer be in the autopilot mode. Here's where the experience of the time-traveller is required.

4. The result of the merge is a new commit that represents the combination of changes from both branches. This new commit will have multiple parent commits, indicating the branches that were merged.

During this process, Git doesn't necessarily identify "changed files" directly or compare the differences in the content of the files. Instead, it looks at changes in terms of commits and the differences introduced in those commits. The changes are then applied to the files, and the resulting state is what you see after completing the merge.

Summary

So, now, if you come across scenarios of why some changes are unintentionally showing up in the PRs that you have raised with your parent / other branches, do remember, that git is not only about comparing the content, it is also about comparing the commits and the where the commits are placed in the history of our timeline!

In the next article, we will be having a look at how to recover from the disastrous event of resetting our timeline, and why the recovery is possible!

But, wait! Did I mention that we can traverse through different timelines of code in the grand scheme of the multiverse of your code? My bad, I didn't! So let us get started with it!

Note: You might already be aware of git branches and the related operations, however, again, the intent of this article is not to impart knowledge on the basics of branching, rather on how git internally mantains these branches!

git init, The start of everything...

When git is initialized in the working directory, git automatically creates a default timeline for you whose history is maintained by git. The default timeline is known as the main timeline, formerly, it was known as the master timeline. All the snapshots of your code are saved in this timeline, and you can move back in time in this default timeline. So what does this timeline do?

Now, enter the fascinating world of Git branches. A branch, in essence, is a divergent timeline in the vast multiverse of your code. It allows developers to create alternate realities where they can experiment, innovate, and make changes without affecting the main timeline. Picture each branch as a parallel universe, coexisting independently until the developer decides to merge them back into the main timeline.

To bring this in perspective, when you initialize git in your working directory, by default, git creates a default branch, called main. This branch consists of all your commits (code snapshots). From this main branch, you can cut off other timelines for development purposes and then merge them into other timelines. When you merge two timelines, git automatically brings commits from both timelines, you might need to intervene in case of conflicts!

So how does git maintain the branches internally?

refs: Where timelines are defined

A branch is nothing but a reference to the head of a sequence of commits. Since you can be working on multiple features simultaneously, you may have multiple different branches, and the branch itself is not a code snapshot, but just a reference to the latest commit of that branch. Then how can we move back in time in that branch? Well, if you remember, each commit object has a parent commit in it, which enables git for easy traversal internally in the specific timeline as well.

So where does git store these references? Yes, you guessed it right, it stores it in the .git/refs folder. Let us inspect the contents of this folder.

➜  .git git:(main) cd refs 
➜  refs git:(main) tree
.
├── heads
│   └── main
└── tags

3 directories, 1 file

Continuing the example from our previous article, we have only one branch, i.e. the main branch. As shown above, the reference of branches is stored in the .git/refs/heads folder. The name of the branch is the name of the file containing the reference to the top commit of that branch. Let us inspect the contents of the main file in the folder mentioned above.

➜  refs git:(main) cat heads/main
b4f560a353a2202e9bb18939c7c3ccbf6e479639
➜  refs git:(main) git cat-file -p b4f560a353a2202e9bb18939c7c3ccbf6e479639
tree 408c5d6e7750f028dd67dd5b25cf98076e400638
parent cc3a96b5a05c666fca63d7b4acd29ce779ad0ce5
author Aman <dummy-email@gmail.com> 1703971554 +0530
committer Aman <dummy-email@gmail.com> 1703971554 +0530

Second commit to time travel!

As seen above, the main file contains only one commit, which on inspecting, turns out to be the latest commit on the main branch. Now, let us create a new branch.

➜  time-travel-with-git git:(main) git checkout -b timeline1 main
Switched to a new branch 'timeline1'

Now let us examine the contents of the refs folder.

➜  time-travel-with-git git:(timeline1) ✗ cd .git/refs                             
➜  refs git:(timeline1) tree
.
├── heads
│   ├── main
│   └── timeline1
└── tags

3 directories, 2 files

As seen above, a new file gets added to the heads folder, which has the same name as that of the new branch name. Now let us have a look at timeline1's content.

➜  refs git:(timeline1) cat heads/timeline1
b4f560a353a2202e9bb18939c7c3ccbf6e479639
➜  refs git:(timeline1) git cat-file -p b4f560a353a2202e9bb18939c7c3ccbf6e479639
tree 408c5d6e7750f028dd67dd5b25cf98076e400638
parent cc3a96b5a05c666fca63d7b4acd29ce779ad0ce5
author Aman <dummy-email@gmail.com> 1703971554 +0530
committer Aman <dummy-email@gmail.com> 1703971554 +0530

Second commit to time travel!

The above commit message is the same as that of the latest commit on the main branch. Why is that so?
If you carefully look at how we created our branch: git checkout -b timeline1 main, we have created it from the main branch. So it means that our new timeline starts from the latest timeline of the main and will contain all the commits of the main timeline till the point from where the timeline1 was cut off. From this point onwards, the timelines are separate and can have separate commits till the time they are merged.

Let us add a few more commits to the timeline1 branch.

➜  time-travel-with-git git:(timeline1) touch timeline1-file.txt
➜  time-travel-with-git git:(timeline1) ✗ echo "Hello from timeline1! " >> file2-timeline1.txt 
➜  time-travel-with-git git:(timeline1) ✗ git add .
➜  time-travel-with-git git:(timeline1) ✗ git commit -m "First commit on timeline1"
[timeline1 01b3b24] First commit on timeline1
 1 file changed, 1 insertion(+)
 create mode 100644 timeline1-file.txt
➜  time-travel-with-git git:(timeline1) touch file2-timeline1.txt
➜  time-travel-with-git git:(timeline1) ✗ echo "Hello from file2 timeline1! " >> file2-timeline1.txt 
➜  time-travel-with-git git:(timeline1) ✗ git add .
➜  time-travel-with-git git:(timeline1) ✗ git commit -m "Second commit on timeline1"          
[timeline1 0c8e8d9] Second commit on timeline1
 1 file changed, 1 insertion(+)
 create mode 100644 file2-timeline1.txt

Let us inspect the contents of the timeline1 branch file in the refs folder.

➜  time-travel-with-git git:(timeline1) cat .git/refs/heads/timeline1 
0c8e8d91dce2222d669776aea7af55a8c27685fa
➜  time-travel-with-git git:(timeline1) git cat-file -p 0c8e8d91dce2222d669776aea7af55a8c27685fa
tree 71fa913d1f22a71e1718b0df4dae7ca63759a088
parent 01b3b247a53a68b399494b392003c81d0ea63087
author Aman <dummy-email@gmail.com> 1704010558 +0530
committer Aman <dummy-email@gmail.com> 1704010558 +0530

Second commit on timeline1

As seen here, the head is now pointing to the latest commit on the timeline1 branch. Now if you have to move to another branch, git will simply look up the branch name in this refs folder and move to the commit objectId present in that file.

Good progress! Down goes the theory of timelines in git!

logs: The registry holding all commits refs

What if you want to have a look at all the commits made to date in a single go? Well, this can be useful in a variety of use cases. So if we continue with our current model, we will need to traverse backwards from the current commit up to the first commit (since the commit object has parent commits as well). In theory, this sounds great, but as the complexity of the project increases, looking up all the commits becomes a bottleneck, because it becomes a sequential operation. So what do we do now?

Inverted Indexes are useful in such scenarios, and git has its flavour of inverted index. The term "inverted index" here refers to Git's approach of holding a reference to all the commits in a way that can be looked up in constant time! This is a crucial optimization that enhances the efficiency of operations related to commit history. So where does Git store this information?

It resides inside the .git/logs folder. Let us examine the contents of the logs folder.

➜  time-travel-with-git git:(timeline1) cd .git/logs 
➜  logs git:(timeline1) tree
.
├── HEAD
└── refs
    └── heads
        ├── main
        └── timeline1

3 directories, 3 files

As seen above, we have the HEAD file and the refs folder. Let us first have a look at the heads folder. From the file name, it is clear that the file name represents the branch names. Let us inspect the contents of these files.

I have modified the content to match our needs for the time being:

➜  logs git:(timeline1) cat refs/heads/main
0000 cc3a commit (initial): First stop in time-travel
cc3a b4f5 commit: Second commit to time travel!
➜  logs git:(timeline1) cat refs/heads/timeline1 
0000 b4f5 branch: Created from main
b4f5 01b3 commit: First commit on timeline1
01b3 0c8e commit: Second commit on timeline1

As seen from the above excerpt, the files contain the entire history of the given timeline (branch). The main branch has only two commits, whereas the timeline1 branch, clearly shows that the branch has been cut off from the main branch, and after that two commits were made. Git provides a command to check the history of the current timeline, git log.

➜  logs git:(timeline1) git log 

commit 0c8e8d91dce2222d669776aea7af55a8c27685fa (HEAD -> timeline1)
Author: Aman
Date:   Sun Dec 31 13:45:58 2023 +0530

    Second commit on timeline1

commit 01b3b247a53a68b399494b392003c81d0ea63087
Author: Aman
Date:   Sun Dec 31 13:45:09 2023 +0530

    First commit on timeline1

commit b4f560a353a2202e9bb18939c7c3ccbf6e479639 (main)
Author: Aman
Date:   Sun Dec 31 02:55:54 2023 +0530

    Second commit to time travel!

commit cc3a96b5a05c666fca63d7b4acd29ce779ad0ce5
Author: Aman
Date:   Sun Dec 31 02:26:23 2023 +0530

    First stop in time-travel

Now let us move back to the main timeline, i.e. the main branch and see what git log has in store for us.

➜  logs git:(main) git log

commit b4f560a353a2202e9bb18939c7c3ccbf6e479639 (HEAD -> main)
Author: AmanMulani <amanmulani369@gmail.com>
Date:   Sun Dec 31 02:55:54 2023 +0530

    Second commit to time travel!

commit cc3a96b5a05c666fca63d7b4acd29ce779ad0ce5
Author: AmanMulani <amanmulani369@gmail.com>
Date:   Sun Dec 31 02:26:23 2023 +0530

    First stop in time-travel
(END)

As seen here, git shows the history of the current timeline. In the first line, you see that HEAD is pointing to the main branch. Git maintains a HEAD reference to the latest commit of the current timeline. This reference is maintained in the .git/HEAD file. Let us examine the contents of the HEAD file:

➜  time-travel-with-git git:(main) cat .git/HEAD
ref: refs/heads/main
➜  time-travel-with-git git:(main) git checkout timeline1
Switched to branch 'timeline1'
➜  time-travel-with-git git:(timeline1) cat .git/HEAD
ref: refs/heads/timeline1

As seen from the snippet above, the HEAD file always contains the reference to the current branch. (Although in a detached state, HEAD can also point to a commit ID). Using the combination of techniques mentioned above and some other optimizations, git can perform a look-up of the history of any timeline efficiently.

Summary

Ahhh! Finally! We have understood how timelines aka branches are maintained internally in Git, and how Git looks up the history of any given timeline. This is particularly useful as you get into situations where basic git commands are no longer able to save your day!

With this knowledge in the arsenal, in the upcoming articles, we will focus on how the timelines are merged, resulting in the creation of something beautiful, or sometimes, something very very terrible!

Note: Although we delve into the basics of git internal, it is expected that the reader is familiar with the basic git concepts. The intend of the article is not to teach the basics of git, but to understand how things work behind the scene, and to find rationale behind why git is the way it is.

Three-Tiered Architecture of Git

Let us first have a look at the workflow that git follows to save the code snapshots. Let us suppose that you have a couple of hundred files in the folder of your concern (the folder where your code lies), also known as the working directory. Git is already initialized in this working directory. Git creates its database inside the working directory itself in the .git folder. This folder holds the code snapshots and is also called the git repository or simply the repository.

Out of these ~200 files, you have updated thirty files. So ideally, one will expect git to take the snapshot of the current state of code by just executing one command. In our context, the process of taking the snapshot of code is known as committing the code to the repository. Straightforward, right?

Now think of a scenario where although you have made changes to 30 files, you are sure only of 14 files, while the remaining 16 files you want to continue working on. You want to have the snapshot of code where only those 14 files have changed, and the remaining 16 files unchanged. This can be a tricky problem to solve if there is no intermediary layer between the changes that the developer has made in the working directory and committing the changes to create a snapshot.

To address this problem, Git uses a three-tiered architecture. The first step remains the same, i.e., the developer makes changes in the working directory. Once the changes are made, now instead of directly committing the changes (i.e. taking the snapshot of all the files changed), the developer first selects only those files that he/she wants to commit out of all the files that are changed. This stage is known as staging. Once the files that the developer intends to take a snapshot of are staged, then those files can be committed. So here's what the three-tiered architecture looks like:

Demystifying the .git folder

Now we are aware of how the git workflow to store a code snapshot works, i.e. the workflow to commit code. Where is the snapshot stored? Yes, you guessed it right, in the .git folder. The entire mystery lies in that folder. It is where the magic happens!

Objects

git init is used to initialize git in any folder. Once git is initialized in that folder (repository), a new folder .git is added to the root of the folder where the git init command was executed.

Now, let us go to the place where magic happens, the .git folder

The .git folder has a bunch of sub-folders, out of which the one that we are focused on in this article will be the objects folder. All the code snapshots are stored in the objects folder. The entire code-related data in git is stored in the form of objects. In the subsequent sections, we discuss in detail the different types of objects in git and their significance.

Blobs

Now let us create a new file in our working directory.

➜  time-travel-with-git git:(main) touch first_file.txt                                                                                    
➜  time-travel-with-git git:(main) ✗ echo "Hello" >> first_file.txt

We see that by simply adding a file in the working directory, nothing changes in the git repository.

➜  time-travel-with-git git:(main) ✗ cd .git/objects 
➜  objects git:(main) tree
.
├── info
└── pack

3 directories, 0 files

Therefore, we can safely conclude that no changes are automatically synced up in the git repository. Let us use the git status command (use this command in the root folder, not in the .git folder) to derive some more insights.

➜  time-travel-with-git git:(main) ✗ git status
On branch main

No commits yet

Untracked files:
  (use "git add <file>..." to include in what will be committed)
    first_file.txt

nothing added to commit but untracked files present 
(use "git add" to track)

As seen in the message above, to track the file via git, we need to stage the file. Once we stage the file using the git add first_file.txt command, let us examine the contents of the objects folder once again. Here's the output:

➜  time-travel-with-git git:(main) ✗ git add first_file.txt 
➜  time-travel-with-git git:(main) ✗ cd .git/objects 
➜  objects git:(main) tree
.
├── a6
│   └── cb6ee3f97c51226a1e86bda81f48021ad27584
├── info
└── pack

4 directories, 1 file

We see a new folder here, which contains a file with a weird long name. What exactly is this?

If you recall, git stores the content in the compressed form and then calculates the SHA-1 (also known as the objectId) of the compressed content to use it as the file name. So it means that once we stage the file, git creates a snapshot of the file, and stores it in the .git/objects folder. An interesting thing to note is that git does not directly store the file name in the objects folder. Rather, it takes the first two letters of the calculated objectId and creates a folder with the same name as those two letters. The subsequent letters of the objectId are the name of the file containing the compressed contents. Therefore, the objectId is the folder name + file name. This enables quicker look-up of objects.

Let us inspect the contents of the file containing the compressed contents

➜  objects git:(main) cd a6
➜  a6 git:(main) cat cb6ee3f97c51226a1e86bda81f48021ad27584 
??W??!Qc?x??%??V^?R????ṃk?X?C51i???.???R{?prə?,???d??Fc????

As seen above, we are not able to infer any meaning from the compressed content. However, git provides us with a command to see the raw content from the file. The command is: git cat-file . This command takes the object ID as input.

➜  a6 git:(main) git cat-file -p a6cb6ee3f97c51226a1e86bda81f48021ad27584
Hello

Voila! We are now able to get the raw content back from Git!
One question that arises is what type of object the current file in question is. By using the -t flag we can find out the type of object it is.

➜  a6 git:(main) git cat-file -t a6cb6ee3f97c51226a1e86bda81f48021ad27584
blob

As seen above, the contents of our files are stored in compressed form in objects known as blobs.

However, as discussed earlier, to store the current snapshot of code, we need to commit the changes to the git repository. Before moving ahead, let us make a few random changes to the existing file and add one more file.

If you inspect the contents of the .git/objects folder, it hasn't changed yet. However, once you stage the files by using the git add command, the content of the objects folder will have changed again.

➜  time-travel-with-git git:(main) ✗ git add .
➜  time-travel-with-git git:(main) ✗ cd .git/objects 
➜  objects git:(main) tree
.
├── a6
│   └── cb6ee3f97c51226a1e86bda81f48021ad27584
├── e7
│   └── e57ac6308173dec7dc1749b72c5f682aa67344
├── ef
│   └── b89ab841a90e0a67fbcdb6cad8696d0357e715
├── info
└── pack

6 directories, 3 files

One thing that you will quickly notice is that although we have two files, three objects are present in the objects directory. Any idea why?
This is because whenever you stage files, git takes the snapshot of all the files that have changed. So when you staged first_file.txt initially, git had created one object of type blob. The next time when we staged two files, git creates two blob objects, containing the compressed content of the staged files.

Commit

Let us try committing the code.

Once the changes have been committed, git successfully creates a snapshot of the current state of the code. Let us inspect the objects folder. We see two additional objects:

➜  objects git:(main) tree
.
├── 0d
│   └── 643279d1e62540cb18cb003d9b5e12b46bd992
├── a6
│   └── cb6ee3f97c51226a1e86bda81f48021ad27584
├── cc
│   └── 3a96b5a05c666fca63d7b4acd29ce779ad0ce5
├── e7
│   └── e57ac6308173dec7dc1749b72c5f682aa67344
├── ef
│   └── b89ab841a90e0a67fbcdb6cad8696d0357e715
├── info
└── pack

8 directories, 5 files

On inspecting the type of object present in the cc folder, we find a new object type: commit

➜  objects git:(main) cd cc
➜  cc git:(main) git cat-file -t cc3a96b5a05c666fca63d7b4acd29ce779ad0ce5
commit

Let us try to get more details about this object by using the -p flag:

➜  objects git:(main) git cat-file -p cc3a96b5a05c666fca63d7b4acd29ce779ad0ce5
tree 0d643279d1e62540cb18cb003d9b5e12b46bd992
author Aman <dummy-author-email@gmail.com> 1703969783 +0530
committer Aman <dummy-author-email@gmail.com> 1703969783 +0530

First stop in time-travel

You can see that the commit object contains a reference to another object known as a tree. The commit object also contains the commit message and the details of the author of the commit.

Tree

If you carefully observe, the reference of the tree object provided in the commit object is already present in the objects folder (the 0d folder in objects/). Let us examine the object.

➜  objects git:(main) git cat-file -p 0d643279d1e62540cb18cb003d9b5e12b46bd992
100644 blob efb89ab841a90e0a67fbcdb6cad8696d0357e715    first_file.txt
100644 blob e7e57ac6308173dec7dc1749b72c5f682aa67344    second_file.txt

So the tree object contains the reference to the blob objects of all the files in the working directory. But then a question arises as to how git maintains the folder hierarchy of the working directory if the tree only consists of blob objects of individual files. For example, if I add another file, third_file.txt in a new folder called notes inside my working directory, how will git know that this file belongs to the notes folder and not the root folder?

Let us add a few more files and directories to our working directory and figure out how git handles this use case.

➜  time-travel-with-git git:(main) tree
.
├── first_file.txt
├── folder1
│   ├── fourth_file.txt
│   └── sub-folder11
│       ├── seventh_file.txt
│       └── third_file.txt
├── folder2
│   └── fifth_file.txt
├── folder3
│   └── sixth_file.txt
└── second_file.txt

5 directories, 7 files

After we stage all the files and commit these changes to the git repository, there will be a lot of objects present in the objects repository. To keep it simple, let us directly focus on the commit object

➜  objects git:(main) cd b4
➜  b4 git:(main) git cat-file -p b4f560a353a2202e9bb18939c7c3ccbf6e479639
tree 408c5d6e7750f028dd67dd5b25cf98076e400638
parent cc3a96b5a05c666fca63d7b4acd29ce779ad0ce5
author AmanMulani <amanmulani369@gmail.com> 1703971554 +0530
committer AmanMulani <amanmulani369@gmail.com> 1703971554 +0530

Second commit to time travel!

This commit object seems to be a bit different from the previous one. We have a parent object here. We will come back to that later, but let us first focus on the tree object. Let us further inspect the tree object:

➜  objects git:(main) git cat-file -p 408c5d6e7750f028dd67dd5b25cf98076e400638
100644 blob efb89ab841a90e0a67fbcdb6cad8696d0357e715    first_file.txt
040000 tree 44f140066e3411499a3f1dda2a4e1c1fb1fda7e4    folder1
040000 tree a6322941fa81fde0730c1bb6cdd69edfe23332b1    folder2
040000 tree 2028d42939fff173dece6e7050245b7f8e70a919    folder3
100644 blob e7e57ac6308173dec7dc1749b72c5f682aa67344    second_file.txt

Initially, we thought that the tree only consisted of the blob objects, which is partly true. But it also consists of other tree objects. So if we inspect the rest of the tree objects, we find that the tree object maintains the folder hierarchy via the tree objects itself.

# folder1
➜  objects git:(main) git cat-file -p 44f140066e3411499a3f1dda2a4e1c1fb1fda7e4
100644 blob c6c3571f4c4a10c36f3936bc0be783d2f742742b    fourth_file.txt
040000 tree 464a225e9d96b59aba367df8e9d124c95ce4549e    sub-folder11

# subfolder11
➜  objects git:(main) git cat-file -p 464a225e9d96b59aba367df8e9d124c95ce4549e
100644 blob 84cfff544bbf9ea894413243bc6e882d5b56afb5    seventh_file.txt
100644 blob 72418f23007afa1d63594d8e50215ed51b84ec03    third_file.txt

# folder2
➜  objects git:(main) git cat-file -p a6322941fa81fde0730c1bb6cdd69edfe23332b1
100644 blob f5a688aed83b1c6776885b5b05c13ba6ca3145d1    fifth_file.txt

# folder3
➜  objects git:(main) git cat-file -p 2028d42939fff173dece6e7050245b7f8e70a919
100644 blob a952388abaf3c3894d7bbab02475a0359dd7afed    sixth_file.txt

Therefore the tree object in essence is the code snapshot we talk about. So after we commit our changes, git creates a new tree object, which has references to all the blob objects. In the tree, only those blobs are changed that have been modified. For the files that have not changed, the old references remain as is in the tree.

Parent (object type is commit)

If you recall from the example above, we also have the parent object. If you inspect the parent object, you will find that it is of type commit, and it simply points to the previous commit. Having the parent commit as a part of the commit object helps in tracking history, and has a bunch of other use cases as well which are not discussed in this article.

➜  objects git:(main) git cat-file -t cc3a96b5a05c666fca63d7b4acd29ce779ad0ce5
commit
➜  objects git:(main) git cat-file -p cc3a96b5a05c666fca63d7b4acd29ce779ad0ce5
tree 0d643279d1e62540cb18cb003d9b5e12b46bd992
author AmanMulani <amanmulani369@gmail.com> 1703969783 +0530
committer AmanMulani <amanmulani369@gmail.com> 1703969783 +0530

First stop in time-travel

Summary

By now, you are equipped with a strong foundational and fundamental understanding of git works. With the help of a simple repository, we have understood how git stores the snapshots of our data, and how to inspect the objects ourselves.

In the upcoming articles, we will be delving into real-time time travel from the present to the past. There will also be surprises as we learn more about how we can travel from one timeline to another via branching in the madness of this multiverse!

Problem Statement

First, let us try to understand what the time machine should be capable of:

• The tool should be able to save code snapshots of the given repository (folder).

• The tool should be able to traverse these code snapshots seamlessly.

By snapshots, we mean the current state of the code.

A Stab at Design

We can easily conclude that each snapshot needs to have a unique identifier. So whenever we save a snapshot, we should get a unique identifier associated with the snapshot, which can used to travel back to that snapshot.

Compressing Files

Since, we are working with files, over time, the amount of disk space used to store code snapshots can increase drastically. Although the disk space isn't a big concern, however along with the disk space, the time to retrieve those snapshots also increases due to the increased size of the history. To solve this problem, let us compress the content of our files when we store the snapshot. We can easily decompress the content of our snapshot while retrieving it.

Hashing the compressed contents

Compressing files solves the problem of storage, but what about the unique identifiers? The first thing that usually strikes us when we think about a unique identifier is hashing. We will use the same concept here. To generate a unique code identifier, we will simply hash the compressed content using SHA1 (since git uses the same algo). The SHA1 generated can be used to travel back to this code snapshot.

Storage Mechanism

Now that we have somewhat solved the problem of uniquely identifying code snapshots and optimally saving the code, let us delve one layer beneath on how to save this data and keep track of it. So basically, we now have to design how the contents given out by our time machine should be saved and retrieved from our client-side storage.

A few constraints to keep in perspective are that the client-side tool has to be lightweight with a minimal number of extra dependencies and should be highly portable. This non-functional requirement somewhat rules out the possibility of using client-side databases. Can we think of something simpler, one that is already available out of the box by the operating system?

Ahh! Can we store the compressed file contents in files and maintain the SHA1 identifiers in the other files for quick look-up of hashes? Well, this seems a good deal. Or, at least this is what Linus Torvalds, the inventor of the time machine git thought. He leveraged the Unix-like file directory structure to store the compressed content.

Final Outcome

So we first compress the contents of the file and store them in a different file. The name of the file is the SHA1 of the file's content. To enable easier look-up for reference, this SHA1 is also stored in a different file. If you are aware of the term inverted index, this might sound something similar in principle. This enables easy storage and retrieval of code snapshots via the VCS tool!

Summary

Congratulations! You have designed a very basic version of git, the time machine! Without a doubt, git uses more sophisticated ways to store and keep track of the timeline. But worry not! In the next article, we will equip ourselves with how git is implemented. The power of commit, blob and tree!

Fundamentals

As always, let us try to understand the fundamentals. To do so, we need to understand why git came into existence in the first place. Git is a distributed version control tool, which means that developers can make changes to the code in their local systems, and once done, they can push the changes to a central collaborative repository.

This is the major problem that git solves, it enables each developer to maintain several different versions of the code locally in their systems, while being able to get new changes (pull changes) from the remote repository in the central server and also push their changes to the remote repository in the central server from their local systems.

Problem Statement

To have a better understanding, let us imagine that you are tasked with designing a tool, that can move you back to any given code snapshot and also give you the ability to share it with other developers as well.

By snapshot, we mean any particular version of code that you want your tool to be able to go back to. Consider a website development scenario where post the initial model (v1), code modifications occur (v2), necessitating a return to the previous snapshot (v1). You should be able to do this with your tool. You should also be able to share the snapshot of any state of your code with other developers.

Solution 1

Now let us try to understand the components needed to design such a tool. Since we have to maintain the code snapshots, it is obvious that we need some storage mechanism, i.e. we will need a database to store our code changes. We want to enable code sharing and, therefore we want the storage to be a remote central storage to save our content (don't even think about peer-to-peer connections!).

Just having central remote storage won't help, because to maintain the connections with multiple clients, making the client-side tool lightweight, ensuring a secure connection and maintaining role-based access control, we need to have a remote server itself, which interfaces between the client and the central storage.

Workflow

• The development flow entails a developer making changes, capturing a code snapshot, and automatically syncing with the remote server while compressing the snapshot.

• Every time a new snapshot is created by the developer, it is directly saved to the remote server.

• The client side maintains the current code version available on the remote server, excluding local modifications.

Even after having this structured approach, scalability concerns arise with the accumulation of data from numerous changes and snapshots, especially with a sizable number of developers.

The performance bottlenecks are due to the storage of compressed data for all files in each snapshot. For example, if our tool takes the first snapshot of the code having 1000 files as v1. It sends all of the compressed data of these 1000 files to the central repository. Now, in the second version, even if we have updated only 2 lines in all the 1000 files, it will send the compressed data of all these 1000 files. However, we have only modified 2 lines.

So a better solution would be to just store what has changed from v1 till any number of changes in the file. So for example, the first snapshot v1 will have all the file contents, however the subsequent snapshots: v2, v3, v4, etc. will be the delta, i.e. only the contents that have changed. This way we have optimized our performance and storage as well. Let's introduce some mathematical terms:

• Initial Snapshot (v1):

  $$S(v1)​=C(1)​+C(2)​+…+C(1000)$$

  Where S(v1) is the snapshot of version 1, and C(n) refers to the compressed data of the given file number

• Subsequent Snapshots (v2, v3, ...):

  $$Δ (v1,v2)=C(modified)$$

  Where Δ is the change of file contents from v1 to v2.

By storing only the delta, representing what has changed in each file from v1 onward, we optimize both performance and storage. See, how simple it is to time travel from one version to another!

Congratulations! You have just designed a tool similar to SVN.

Problems

This seems to be ideal but it is a nightmare for teams working on a decent-sized project. Problems?

1. Internet Dependency
   The developer always needs to be connected to the internet. To save the current snapshot, developers often struggle when they are offline or have a slow network connection since snapshots can only be saved on the remote server. This hampers the developer's productivity.

2. Linear Snapshot Retrieval
   Since only the delta changes are stored, accessing a specific snapshot requires reconstructing the code's state sequentially from the initial version. This is necessary because each subsequent snapshot builds upon the previous one. Retrieving snapshots linearly ensures the integrity of the code history, but it may become less efficient as the project grows in size and complexity.

3. Single point of failure

   If the remote server goes offline, then all the developers will be stuck in the current local copy, and won't be able to save any snapshots.

Although our system is optimized in terms of computer resources, it fares very poorly in terms of developer experience and productivity.

In our analogy, it means that if you have to travel in the past, you must always start from the beginning of time itself, the very first big bang! Not a pleasant experience for the time-travellers!

Solution 2

Developer time is more expensive than local disk storage.

If we think in terms of developer time being more expensive than the local disk storage, we can think of having client-side storage. Having client-side storage means that we need an internet connection only when we have to update or fetch the data from the remote repository.

Since we consider storage to be cheaper as compared to productive time, we can start storing the entire snapshot of the code instead of delta. At any given instance of time, the developer can push the snapshots to the remote server, which in return will maintain these snapshots, which can later be pulled by some other developer. Using some smart compression techniques and retrieval methods, we can make the system performant. So here we go, we have the following three components:

Client-side storage, client tool, remote server.

Workflow

• The developer changes the code stored in a folder (repository). This folder is somehow registered in the tool that we use, and thus there is a remote copy of this repository.

• Multiple developers take a copy of this repository in their local machines. Each developer can push their snapshot of code to the remote repository. Developers can make changes and store the required snapshots in the client-side storage. Internet connectivity is not required for creating snapshots. An Internet connection is only required to push the snapshots to the remote server.

• Since we store all the contents of the code with a given snapshot, it becomes easier to jump back to any version, without the need to calculate it linearly from the initial version.

• Multiple developers can continue to work on their local copies even if the central remote server is down.

Since there are multiple copies of snapshots present with each local machine, which can be eventually synced back with the remote server, we can call this system a loosely coupled distributed version control system that empowers the developer and can result in more developer productivity.

Congratulations, you have designed a very raw version of git!

Summary

In principle, we have designed a time machine, which can take us back in time to any snapshot by just knowing where we want to land (the version number). This is what git is. A time machine which can take us back to the dinosaur age of our code, and then bring us efficiently back to the current modern age (even if it is no better than the Stone Age. Pun intended!)

There are quite a few drawbacks in git as well, which are easily available on the internet if you want to read through them! Although in this post we have designed the most basic and scrappy version of git, in the upcoming posts, we will delve into the details of the internal workings of git and build upon these foundations! Looking forward to meeting you again, time-traveller!