ERC - HAMMER Digital Twin CAD modelling

Digital twin creation Spring 2023-Spring 2024.

• Made accurate CAD model from physical diagrams and images (Rhino3d).
• Programmed a virtual UR5e robotic arm to follow multiple toolpaths (Rhino3d, Urscript, C# and Grasshopper).
• Integrated an AI-powered Digital Twin into a real-time Unity3d environment for VR and interactivity.
• Developed an automated data processing pipeline using hosted, secure servers and Python scripts.
• Integrated fast real-time physics interactivity within the Digital Twin.
• Took initiative for a team of four, assigned tasks among 2 undergrads (including myself) and 2 PhD students.
• Worked in a larger team of 20 people, recorded feedback and worked with an interdisciplinary team across timezones.

Future work:

• Adding educational functionality to the digital twin.
• Making the digital twin easy to use for members of industry.
• Adding digital twin control through voice commands and hand gestures.

ERC - HAMMER Digital Twin Networking

Creating a digital twin backend Fall 2023 - Spring 2025

• Used Golang for high quality, fast data processing.
• Created an online geometry viewer that communicates with a cloud server and visualizes data present. Click here for a simplified beta!
• Single-handedly discovered and listed cybersecurity concerns, kept a running list so multiple University administrators could be on the same page.
• Implemented many networking features, like routing without a framework (I followed many online tutorials, but made something for our specific use case)
• Created python scripts for automated uploads/downloads/processing.
• Utilized Docker to create containerized applications (containerization helps with performance and storage)
• Used Github actions for automated CI/CD.
• Fun fact: This portfolio you are looking at uses github actions for automated CI/CD too :)
• Implemented secure access through two-factor authentication.
• Created a self-hosting solution using tunneling protocols to connect two universities together.
• Led a team of eight, delegated tasks between two undergrads (including myself), two masters students, two faculty members and two PhD students
• Organized my team in an organization with over 30 members across four Universities.
• Used multiple cloud technologies, such as AWS, Google Cloud, and Microsoft Azure.
• Created multiple benchmarks to compare pricing, security concerns and performance to stakeholders.

Using my self-hosting tunneling solution, latencies ranged from 60ms to 100ms.

Mutli Panel Display VR development

Applications created for Head-mounted displays and Multi-Panel* displays

*Basically, think combining 30 monitors & TVs together, and having them all synced as one display. Like those big Nasa ones.

Summer 2021 - Fall 2024

• Maintained over 500,000$ worth of VR equipment, was on-call to make sure that it worked.
• I handled 100% of the programming, bugtesting, and implementation of over 40 high quality data science applications.
• Frequently used software like CloudCompare for optimization of pointclouds.
• Taught over 10 students how to develop these applications .
• Created documentation and training materials for my lab for over 30 students, many of them without technical backgrounds.
• Presented over 60 times unscripted, with live technical demos. Presented to industry leaders (Rockwell Automation, Meta, Google, University faculty from across America).

Examples of applications developed (Feel free to ask about more!)

• Created interactive 3d graphs that users could
• Used a Downtown Greensboro pointcloud scan to create an immersive tour.
• Used generalized Pointcloud scans for accurate real-time distance tracking in VR.
• Created a realistic human skeletal and digestive system demo with animations.
• Created real time physics simulations of concrete mixing with fluid dynamics.
• Created an educational human anatomy demo with text-to-speech that could dynamically translate languages in real-time.
• Helped develop a farm VR tour application. It eventually won an award at a small business conference.
Read more here (click me)

Google maps and tiles integration

Worked with geometric data & scans and visualized them on a curved screen.

2023

• Took 3d datasets of locations from the internet (not involved with scanning) and used data science tools to convert and display them into an interactive VR environment.
• Worked with multiple formats (.ply, .txt, .obj, .fbx, .stl and more) and converted them using meshlab to optimized .obj files.
• Wrote custom LOD/Occlusion systems for display on the curved screen. I had to do this for performance and visual fidelity.
• Used multiple engines and technologies to represent this in VR (Unity3d, Unreal Engine, Meshlab). I wanted to do something more bespoke with a C/Rust based visualization engine and Vulkan - but I didn't feel confident enough to devote time to it.
• Fun fact: Epic Games released a massive bustling city as a part of their Matrix Reloaded promotion... Getting this onto the screen was absurdly difficult - ask me how I did it! Read more here (click me)

Self driving bot research

Assisted with debugging and programming small self-driving vehicles.

2023

• My research department was heavily involved in self driving cars, computer vision and embedded systems. NCAT regularly participated in competitions and collaborated with the mechanical engineering department to send students across the country to participate in competitions.
• Every now and again, I would use my software engineering/data science background to help assist in adding new features, optimizing performance and fixing bugs.
• We had to do a lot of work with the embedded systems (Arduino, Rasberry Pi systems - sometimes multiple microcontrollers in one car) to fit our AI models/Vision systems.
• Hardware stack: (Handled by other )
• Tech stack: Python and it's assorted ML libraries (pytorch, Jupyter notebooks, Tensorflow, pandas [especially for debugging/datasets]). We specifically used a LOT of OpenCV and YOLO in our work.
• Fun fact: I heavily reduced inference times (from around 100ms to around 300ms) on multiple machines. I did this by:
  1. Changing models to smaller/more lightweight versions.
  2. Reducing resolution (We used a 1080p depth camera and reduced it down to a square aspect ratio, and heavily compressed the image for inference)
  3. Changing how we performed sensor fusion. We had a LIDAR scanner and a camera, and fusing the two of them together was difficult, and required us to do some advanced embeddings to get it working. We had to follow some scientific papers, and there was one released in 2023 that helped us a TON!

High performance computing for AI systems

A small project attempting to fill in performance gaps.

Currently, many AI systems use slower technologies and methods for AI development.

Examples:

Python is seen as the go-to for many data processing steps. This is fine, however many analyses and studies have shown that Python is slow compared to other programming languages. This is due to multiple things - one of the most significant being the GIL. To put it simply, it makes developing robust multithreaded programs in python much more difficult.

After looking at the literature, many steps of the AI development cycle is spent on extremely expensive computation - particularly when it comes to lengthy matrix multiplication and keeping track of all of the different mathematical steps (I'm oversimplifying, but it's not important). What if there's a way to speed up these steps?

Many AI models use reduced-order modelling to speed up computation at the cost of accuracy. This is infeasible in areas that require high accuracy AND speed (autonomous driving vehicles, medical devices, stock trading).

• Began learning rust, a mathematically focused programming language focused on performance and memory safety.
• Came up with an ambitious plan to develop an image processing library that's able to implement something similar to YOLO for Python in Rust.
• Identified a need for fast AI models to be used on weaker hardware (portable systems, such as self-driving cars).

Status:

Shelved during Fall 2024. This was due to the complexity of the process involved. Python

I really want to go back and start on this once again, but I don't have the skills just yet... This is an area of important Business and scientific need.

Publications as first author

Problems and Solutions of Point Cloud Mapping for VR and CAVE Environments for Data Visualization and Physics Simulation

Publications as co-author

On the feasibility of an integrated English wheel system

Initial framework design of a digital twin mixed-reality-application on human-robot bi-directional collaboration...

12/15/2024 last updated. (4 papers awaiting review)

Ask me about my in-progress work!