Recent Projects

Project Enzian: A New Research Computer - Heterogeneous Compute with Cache Coherence


As a research computer, Enzian is designed for computer systems software research, rather than any particular commercial workload. An Enzian node has a big server-class Marvell CPU closely coupled to a large Xilinx FPGA, in cache coherence with ample main memory and network bandwidth on both sides. Prof. Mothy Roscoe gave a seminal presentation that motivates the need for Enzian at the ARM Research Summit in September 2017. It can be viewed at the 28 minute mark here - For the latest info on Enzian, including the presentation at the 2020 Conference on Innovative Data Systems Research (CIDR), please see

Traditional systems software research is facing a new challenge to its relevance.  Modern hardware CAD systems, the drive to lower power and "dark silicon", FPGAs, and other factors have made it easy, quick, and cheap for system vendors to build custom hardware platforms.  Almost any function can now be put into silicon or reconfigurable logic: the choice of exactly what “should” be built is a short-term business decision based on markets and workloads.  

Such hardware qualitatively changes how systems, including system software, should be conceived and designed. However, most published OS research in rack-scale, embedded, or datacenter computing only uses affordable commodity platforms for which documentation is available to researchers. Academia and industry practice are diverging.

Enzian is a new research computer designed and built at ETH Zurich (with help from Marvell and Xilinx) to bridge this gap in a way not possible with commodity hardware or simulation.  Enzian nodes closely couple a server-class CPU SoC with a large FPGA in the same coherence domain, with abundant network bandwidth to both chips.  It is designed for maximum research flexibility, and can be used in many ways: a high-end server with FPGA-based acceleration, a multiport 200Gb/s NIC supporting custom protocols and cache access, a platform for runtime verification and auditing of system code, to name but three.

Data Center Networking


2017 Publication: IncBricks: Toward In-Network Computation with an In-Network Cache Ming Liu, L. Luo, J. Nelson (Microsoft), Luis Ceze, Arvind Krishnamurthy, Kishore Atreya (Cavium)

IncBricks is an in-network caching fabric with basic computing primitives. IncBricks is a hardware-software co-designed system that supports caching in the network using a programmable network middlebox. As a key-value store accelerator, the prototype lowers request latency by over 30% and doubles throughput for 1024 byte values in a common cluster configuration.

This collaborative groundbreaking work was presented at ASPLOS 2017: 22nd Intl. Conference on Architectural Support for Programming Languages and Operating Systems, Xi'an, China

Prof. Arvind Krishnamurthy (right) describes innovations in programmable switches at the 2017 UW Cloud Workshop. The emergence of programmable network devices and the increasing data traffic of datacenters motivate the idea of in-network computation. By offloading compute operations onto intermediate networking devices (e.g., switches, net- work accelerators, middleboxes), one can (1) serve network requests on the fly with low latency; (2) reduce datacenter traffic and mitigate network congestion; and (3) save energy by running servers in a low-power mode. However, since (1) existing switch technology doesn't provide general computing capabilities, and (2) commodity datacenter networks are complex (e.g., hierarchical fat-tree topologies, multipath communication), enabling in-network computation inside a datacenter is a challenging task that IncBricks tackles.

Data Center Compute: ARM-based Cloud Servers

2017 Publication: SWAP: Effective fine-grain management of shared last-level caches with minimum hardware support, X. Wang, S. Chen, J. Setter and J.F. Martínez


Performance isolation is an important goal in server-class environments. Partitioning the last-level cache of a chip multiprocessor (CMP) across co-running applications has proven useful in this regard. Two popular approaches are (a) hardware support for way partitioning, or (b) operating system support for set partitioning through page coloring. Unfortunately, neither approach by itself is scalable beyond a handful of cores without incurring significant performance overheads.

Cornell proposed SWAP, a scalable and fine-grained cache management technique that seamlessly combines set and way partitioning. By cooperatively managing cache ways and sets, SWAP (“Set and WAy Partitioning”) can successfully provide hundreds of fine-grained cache partitions for the manycore era.

SWAP was presented at the Intl. Symposium on High Performance Computer Architecture (HPCA), Austin, TX, Feb. 2017.

SWAP requires no additional hardware beyond way partitioning. In fact, SWAP can be readily implemented in existing commercial servers whose processors support hardware way partitioning. In this work, Cornell graduate students Xiaodong Wang, Shuang Chen and Professor José F. Martínez are working with Cavium Engineers Shay GalOn and Bobbie Manne to prototype SWAP on a 48-core Cavium ThunderX platform running Linux. They achieved average speedups over no cache partitioning that are twice as large as those attained with ThunderX’s hardware way partitioning alone.  


Cornell PhD Student Xiaodong Wang in his lab at Cornell with Cavium board.

OpenStack Projects

"Thank you to the IAP, CMU and Cornell teams for their hard work and attention to detail.
Outstanding collaborations with fantastic results!"
– Dr. Brad Topol, IBM Distinguished Engineer, OpenStack

Openstack Carnegie Mellon University IBM

2016 Cornell OpenStack Team (left to right): Qin Jia, Dr. Weijia Song, Vishnu Govindaraj,
Mengcheng Qi, Xiao (Alex) Tan, Samir Borle, Richard Quan, and Prof. Robbert van Renesse.

“This was a great opportunity for us to learn how to work in the open source community. Very complementary to our coursework, teaching us industry best practices for test-driven software development, and being held accountable for high quality contributions to OpenStack.”
– Alex Tan, Cornell Computer Science, Master of Engineering

The Cornell OpenStack team was formed after the IAP Cornell Cloud Workshop in October 2015, where Kiloeyes was described in a presentation to the attendees by IBM Senior Engineer and OpenStack Contributor Tong Li. A special class was created for the students to earn credit in the Spring 2016 semester, led by Professor Robbert van Renesse.  Prof. van Renesse is an expert in distributed systems, particularly in their fault tolerance and scalability aspects, chair of ACM SIGOPS, editor for ACM Computing Surveys, and an ACM Fellow.  The team contributed to the Kiloeyes service, an open-source multi-tenant, highly scalable, performant, fault-tolerant monitoring-as-a-service solution. It uses a REST API for high-speed metrics processing and querying, and has streaming alarm and notification engines.

Tong Li (left), IBM, Senior Software Engineer and OpenStack Contributor

describes Kiloeyes at the IAP Cornell Cloud Workshop in October 2015. Tong pioneered and leads the development of Kiloeyes, the OpenStack cloud monitoring solution, and is currently building out new micro services for Kiloeyes to reliably analyze real time message streams of several hundred thousand per second.  Tong met informally with several students at the Workshop during the breaks and networking session to share his perspectives and experiences as a software engineer at IBM, focused on open source solutions.

Kiloeyes uses Kafka as its message queue, to reliably consume over >>100,000 messages per second, that are distributed to data processing engines for alarms, anomaly prediction and notifications. Elasticsearch is the back end data store, providing fast and durable storage of all incoming messages at the pace delivered by Kafka. The basic architecture is shown in the figure below.

Kiloeyes Architecture: The Analyzer/Transformer/Aggregator represents multiple mico-service engines that operate independently, such that new engines for specific data processing and analytics tasks can be easily added.

The Cornell team collaborated together on three important tasks specified and mentored by IBM Senior Engineer and OpenStack Contributor Tong Li: (1) Security, (2) Metrics visualization, and (3) Kiloeyes integration with the other OpenStack services via support of the OpenStack Ceilometer API.

Security: The integration of authentication and security features into Kiloeyes required changes to the Kiloeyes server pipeline. The approach was to authenticate incoming requests against OpenStack's Keystone service, by integrating the Keystone Middleware project into the server pipeline. Then, user authentication information was appended to the Kiloeyes metrics, so every metric contains data about which tenant, service, and user posted it. Finally, the added authentication data was used to filter queries to Kiloeyes such that only metrics that a user is authorized to view would be presented.

Ceilometer API: Ceilometer is a legacy Openstack monitoring service that was recognized not to scale well for large clusters; indeed, this limitation was the motivation for the development of Kiloeyes. However, the Ceilometer API is a natural and convenient means to interface Kiloeyes with the other Openstack services, hence the goal was to the implement the Ceilometer API on Kiloeyes based on Ceilometer API's documentation. The team successfully implemented Meters API including Get Meters, Get Meters by Name, Get Meter Statistics and Post. Additionally, they built a parameter search utility in Meters that supports all Ceilometer query and aggregation to enable users to query the metrics they desire.

Visualization: Kiloeyes lacked any Visual Representation of the aggregated metrics. This limitation was overcome with the help of the open source tool Kibana. The team successfully integrated Kibana with the elastic search host and created Graphs and Charts representing the Kiloeyes metrics. A dashboard plugin was created for Horizon, the Openstack dashboard application and the visualizations were added to the dashboard. The team also made sure that the details represented in visualization can change according to the project selected by the end-user. Below is a screen shot of Network packet activity collected by Kiloeyes and displayed on the Horizon Dashboard.

Kiloeyes Visualization: Network packet traffic vs. time is aggregated by Kiloeyes and plotted in the Horizon dashboard shown above. The tabs in the panel on the left of the screen shot ("Load Avg", "Process Thread Count", etc.) have several standard line and pie charts. New metrics and charts can be easily customized by an OpenStack user, and added as a tab.

The Kiloeyes code is available on github (

The Cornell project built on top of the outstanding work performed at Carnegie Mellon in 2015. The CMU project culminated in the first availability of Kiloeyes in the OpenStack Liberty release in October 2015. 

Graduate students Han Chen, Shaunak Shatmanyu, and Yihan Wang and CMU research scientist Dr. Ipek Ozkaya at Carnegie Mellon University in Pittsburgh collaborated with IBM Senior Engineer and OpenStack Contributor Tong Li on a two-semester project to build out the python-monasca project, including new data processing engines for alarms, anomaly prediction and notifications. The students merged their first production code to Stackforge at the mid-point of the project in May 2015, and final code in August 2015. Python-monasca was migrated to become OpenStack Kiloeyes in August 2015, and released with Liberty in October 2015. Hence, this project is a significant achievement for the students and IBM, and a noteworthy milestone for OpenStack.

OpenDaylight Projects

OpenDaylight (ODL) is an open platform for network programmability to enable software defined networking (SDN) for networks at any size and scale. The “Internet of Things” IoT Data Management (IoTDM) project uses the ODL infrastructure to provide a data-centric middleware that acts as an IoT data broker, and enables authorized applications to post and retrieve IoT data uploaded by any device -

The IoT data is organized in a massive resource tree (potentially millions of nodes). This tree contains measurements from devices (the things) and its associated attributes.  The attributes represent metadata about the resource, for example access rights, creation time, children list, owner, size, quota, etc.  IoTDM complies with the oneM2M standards for machine-to-machine and IoT communications -

Boston University (BU) students built apps for Cisco’s Smart Cities projects to provide visualizations of mobile device positions on interactive maps of both indoor facilities and aerial/satellite imagery by combining signals acquired from disparate “things” including Bluetooth low energy beacons, internet routers, and GPS.

Using the BU apps, mobile users can (a) search for other registered users on their mobile devices, (b) view a “heat map” that displays real­time “people traffic” and historical data, and (c) manage the Tree data using a unique data visualization tool to search, add, edit or delete data (“leaf”) nodes for users, devices, locations and maps visually displayed on a browser.

Students who contributed to the apps include Mark Barrasso, Jose Bautista, Justina Choi, Sean Liu, Nehal Odedra, Niklas Kunkel, Qingqing Li, and Yingchao Zhu.  The students were supervised by Prof. Orran Krieger and Dr. Ata Turk of the BU Department of Electrical and Computer Engineering, and Cisco Principal Engineer Lionel Florit, who scoped the projects and guided the students in weekly meetings.

Mark Barrasso won the Top Hacker Award for the Smart Cities projects, for leadership in application design and coding. Mark also acted as scrum leader for one of the BU student teams, and represented BU and his colleagues at Cisco Live in San Diego in June 2015.

Project demos presented at Cisco Live are here.

Mark Barrasso (center) at Cisco Live in San Diego on June 10, 2015, is presented the Top Hacker
Award by Lionel Florit, Cisco Principal Engineer and Kristen Wright, Director, Cisco Research Center