terminal

Siddhartha Venkatayogi

Projects for CS 378: Cloud Computing aka Applied Distributed Systems with Dr. Kia Teymourian.


Unfortunately can’t share the code publicly, but if you’d like to see the code please reach out!


External Sort

Sorting a huge database that exceeds the capacity of RAM on the machine. My solution implements k-way merge, parallel/concurrent processing, and data processing pipelines. Ran on Google Cloud GCE. Sorted the FOIL NYC Taxi Dataset.

Map Reduce

Implementing Map and Reduce over huge database. Map and Reduce run on 2 separate machines connected via socket connection. Ran on Google Cloud GCE. Foil NYC Taxi Dataset again.

Distributed MapReduce

Implementing MapReduce across five machines connected via sockets, with two mappers, two reducers, and a final aggregation node. Ran concurrent jobs to find the top earning hours and drivers using multithreading and batched data transfers. GCP

Hadoop MapReduce

Analyzing taxi data using Apache Hadoop. Implemented chained MapReduce jobs to count GPS errors by hour, identify taxis with the highest GPS error rates, and rank taxis by average speed. Used min-heaps for efficient top-K selection. Ran on a 3 machine Google Cloud cluster.

Spark kNN Text Classification

Classifying Wikipedia documents with a PySpark k-nearest neighbors classifier using TF-IDF vectors. Processes a corpus of one million pages and uses Spark DataFrames to summarize category statistics, rank common categories, and identify pages with the most categories.

We also got to take a field trip to TACC!!! (pictured)


TACC field trip photo 1 TACC field trip photo 2 TACC field trip photo 3 TACC field trip photo 4