Projects for CS 378: Cloud Computing aka Applied Distributed Systems with Dr. Kia Teymourian.
Unfortunately can’t share the code publicly, but if you’d like to see the code please reach out!
External Sort
Sorting a huge database that exceeds the capacity of RAM on the machine. My solution implements k-way merge, parallel/concurrent processing, and data processing pipelines. Ran on Google Cloud GCE. Sorted the FOIL NYC Taxi Dataset.
Map Reduce
Implementing Map and Reduce over huge database. Map and Reduce run on 2 separate machines connected via socket connection. Ran on Google Cloud GCE. Foil NYC Taxi Dataset again.
Distributed MapReduce
Implementing MapReduce across five machines connected via sockets, with two mappers, two reducers, and a final aggregation node. Ran concurrent jobs to find the top earning hours and drivers using multithreading and batched data transfers. GCP
Hadoop MapReduce
Analyzing taxi data using Apache Hadoop. Implemented chained MapReduce jobs to count GPS errors by hour, identify taxis with the highest GPS error rates, and rank taxis by average speed. Used min-heaps for efficient top-K selection. Ran on a 3 machine Google Cloud cluster.
Spark kNN Text Classification
Classifying Wikipedia documents with a PySpark k-nearest neighbors classifier using TF-IDF vectors. Processes a corpus of one million pages and uses Spark DataFrames to summarize category statistics, rank common categories, and identify pages with the most categories.
We also got to take a field trip to TACC!!! (pictured)