Evaluating Data Processing Datacenter Applications in Serverless Environments

Master's Thesis, Systems Group ETH Zurich, February 2023 - September 2023

Skills: Bash, Python, AWS Lambda, Amazon EC2, Apache Spark, Apache Drill, Docker

Serverless platforms provide a high degree of elasticity and fine-grained billing. Even though the serverless programming model is very popular for short-lived, event-driven applications, it is ill-suited for distributed data center applications because of restrictions like the lack of direct communication between individual serverless functions. Boxer is a system developed to overcome these limitations and provides an environment in real-world serverless platforms that is compatible with existing data center applications. Boxer enables the use of existing unmodified query engines in commercially available serverless offerings (e.g., AWS Lambda), and thus, allowing us to take advantage of the elasticity properties of serverless platforms without the need to re-design the applications to fit the serverless programming model. This offers the ability to instantiate data center clusters on a per-request granularity and therefore reducing resource over-provisioning.

In this thesis, we study the requirements, feasibility and performance of using Boxer to run existing data processing applications in a serverless environment. We show how to instantiate whole query engine clusters in serverless functions on a per-request granularity and present experimental results of the TPC-H benchmark using Apache Spark and Apache Drill query engines running in AWS Lambda serverless functions. We evaluate the query engine cluster initialization time, query execution time, as well as monetary costs, and compare them to results gathered from the same experiments performed in Amazon EC2 virtual machine clusters.

Final Report

Update (October 2023):
Our paper Off-the-shelf Data Analytics on Serverless has been accepted at the Conference on Innovative Data Systems Research (CIDR), 2024.

Computer Graphics Project

Computer Graphics course project, ETH Zurich, 2022

Skills: C++, Python

During this course, we implemented a renderer to create physically-accurate images.
We extended the Nori v2 ray tracer by adding the following features:

• Images as Textures
• Normal/Bump Mapping
• Textured Area Emitter
• NL-means Denoising
• Heterogeneous Participating Media
• Directional Light
• Motion Blur
• Procedural Volume
• Environment Map
• Advanced Camera (Depth of Field, Lens Distortion, Bokeh)
• Disney BSDF

For our final image, we wanted to show the beauty of space. To fit the "Out of Place" theme, we wanted to put something random into the scene: For example a car "driving" in space.

Final Report Source Code

Hybris - The Rise and Fall of Atlantis

Game Programming Laboratory course project, ETH Zurich, 2022

Skills: C# (MonoGame)

Hybris (from Ancient Greek húbris) refers to excessive pride or dangerous overconfidence. It was due to these traits that Atlantis, an island whose naval power besieged Athens in times long past, fell out of the Greek Gods' favour, which lead to its downfall and submergence in the Atlantic Ocean.

"Hybris - The Rise and Fall of Atlantis" is a 3D city builder where the player must employ finesse and strategy to expand their budding nation, all the while taking care not to draw the wrath of the gods upon themselves and ultimately succumbing to the same fate as the Atlanteans before them.

Hybris was developed by Benjamin Simmonds, Gianluca Moro, Eric Enzler, Fabian Bösiger and Kate Gadola during the 2022 Game Programming Lab course at ETH Zurich. The game was implemented in C# using MonoGame and 3D models were designed in Blender.

Download Trailer

Wildfire Visualization

Scientific Visualization course project, ETH Zurich, 2022

Skills: Python

In this project, we wrote an interactive program to visualize the spread of wildfires using real-life data. We can use our visualization to analyze the effects of wind and terrain on the spread and temperature of the fires.

Final Report Source Code

"Seeing" Temperature - Integration of IR Camera into the HoloLens 2

Mixed Reality course project, ETH Zurich, 2021

Skills: C# (Unity), Python

The goal of this project is to display the thermal information of people to the user by integrating an infra-red camera connected to a Raspberry Pi computer into the HoloLens 2 system. In the end, the HoloLens will have all the information needed to detect potential temperature hazards and make them visible. The idea behind the project is to develop an application that helps detecting potential Covid-19 threads, e.g., in hospitals, airports, workplaces and other places with a high infection risk.

Final Report Source Code

Storage Format for Almost-Homogeneous Data Sets

Bachelors's Thesis, Systems Group ETH Zurich, March 2020 - July 2020

Skills: Python, Java, JSON, Apache Parquet

JSON is a popular data format which is very flexible since no schema needs to be defined and therefore the data can also be heterogeneous. But this flexiblity comes at the price of performance. However, in practice, most data sets do not use the flexibilty of JSON to its full extend and are mostly homogeneous. We call such data sets almost-homogeneous. For almost-homogeneous data sets, the trade-off between the flexibilty and performance loss of JSON is not justified.

Thus, we propose a new storage format for almost-homogeneous data sets which allows for faster processing by storing the data in the Parquet file format. Parquet is optimized for structured, homogeneous data sets and allows for significantly faster data processing compared to unstructured JSON.

In this thesis, we first cover the background on Parquet and also analyse the almost-homogeneity in real-life JSON data sets. We then show how to preprocess and convert data sets from JSON to the Parquet format. Finally, we evaluate the processing performance of our new storage format.

Final Report