Log Search

While I was interning at Open Systems, a hackathon within the company took place. For the two days, I joined some people from the data lake team. Around that time, they were working on ingesting logs from all the servers the company maintains into ElasticSearch. For the hackathon, we wanted to make this data accessible to engineers within the company through an easy to use interface. They should be able to search for things like IP addresses, domains etc. and find servers for which there hits. Further, the results would be linked to the various internal tools that generated these logs, along with appropriately set filters.

After the two days, the project was in a quite decent state - thanks to an excellent team constellation where all necessaries areas where covered: Design, frontend, backend, tweaking ElasticSearch queries and presenting. Apparently, the project has even appealed to the higher ups. So we decided that I would continue my internship with the data lake team, and, as one of my duties, continue working on our hackathon project.

For me, this was an awesome project. I got to work on a clean slate (of course throwing away all the code we wrote on the hackathon, barely held in place by band-aid) and develop the project with some more time and proper processes, from start to finish.

The backbone of the search engine that would tie everything together was a Go application. It would expose a REST API that parsed the input and formed ElasticSearch queries, custom for each of the different log sources (and of course in parallel). The frontend was written with TypeScript/React and the in-house developed component library pyrene - I was told that I was the first outside of the team that developed the library to use it! With such an array of technologies, multiple build tools would be necessary. However, at that time, the team was transitioning to use Bazel as a unified build tool, which came in handy for this project: From builidng the backend and frontend applications to packing all into containers, everything has been done using this one build tool. Last but not least, the entire enchilada would also need to be deployed - and this happened with Ansible.

I’m not sure if the search engine is still running, but it was an amazing project to work on where I learned a lot in a multitude of areas and was able to bring it from start to end.