How 8 Software Engineers Solved Their Biggest Technical Challenges

For software engineers, the pressure is on to get it right. 

After all, issues in the code can potentially affect every other facet of a product. This is especially true in the fields of education and financial technology — when people’s money or learning is at stake, it becomes even more important to approach software programming carefully and thoughtfully.

“Not being able to trade is bad,” said Wendy M, a software engineer for trading firm DRW. “But trading blindly without knowing the firms’ position can be disastrous. A bad system can bring down the entire firm.”

Roadblocks are inevitable with any project, and code is notoriously finicky. A good engineer expects these problems, and knows the true test of their mettle is how they address technical challenges when they appear. To be a true problem solver, a software engineer doesn’t need to know everything — instead, they need creativity, persistence and the ability to use the tools at their disposal. 

“In search of a solution, I resorted to the instincts carried by any child with a LEGO set: I did some serious playing and tinkering around,” said Hubert Shon, a full stack engineer at KPI Sense.

Built In Chicago spoke to software engineers at eight different tech companies and discussed the biggest challenges they’ve tackled.  Each of them showed ingenuity, resourcefulness and the understanding that their code doesn’t just affect one person or team, but the whole of the product. Read on to hear about every challenge they faced, and how each one was overcome.

Sunder Rajan

Sr Manager Full Stack Engineering

What’s the biggest technical challenge you’ve faced recently in your work? What made this particular challenge so tricky?

Digital Collaboration Hub is one of Wealth Management’s digital initiatives to modernize clients’ experiences interacting with Northern Trust and to serve those who seek digitally-forward experiences. It is fun, innovative and fast-paced — but we also have to focus on security and data privacy. One of our biggest challenges is solutioning and hosting sensitive data in the cloud due to data privacy and compliance regulations. 

In order to utilize the myriad of benefits — both for the bank and our clients — that come from using the cloud, we have to demonstrate that our client’s data is secure. The kind of data — personally identifiable information (PII) — we need to store is sensitive because it could identify an individual. We must ensure that the technology solutions we build have best-in-class information security controls built in to make sure it is both preventative and detective.

How did you and your team overcome this challenge in the end? What were some of the specific tools or technologies you used?

We invited external experts to deliver workshops and training for everyone involved in the initiative — these sessions focused on information security vulnerability and the implications of using cloud-based solutions on data privacy. We also leveraged the experts from Northern Trust’s info and cyber security teams to understand current-state, applicable regulations and other factors specific to Northern Trust. Gaining an understanding of industry-leading practices, the needs of our clients and our current infrastructure allows us to put the necessary controls in place for handling data stored in the cloud. Some of the security features include encryption in transport and at rest, multi factor authentication, IP restrictions and robust access controls. We started with a proof of concept, limiting the scope of implementation and conducted rigorous testing — then we shared the design and results with various internal governance committees.

We must ensure that the technology solutions we build have best-in-class information security controls built in.”

How did this technical challenge help you grow as an engineer or help you strengthen a specific skill?

Our experience with the Digital Collaboration Hub increased our level of understanding and the organization’s comfort with cloud solutions. We learned about the constantly evolving fields of information security and data privacy, which is important for all of the work we do. At Northern Trust, we bring expertise to our clients, so we are constantly seeking to enhance our own knowledge around issues that impact them and our service. This initiative also underscored the importance of seeking external and internal expertise. This type of implementation wouldn’t work without understanding the impacts of the solution, our clients and Northern’s specific information security and data policies and perspectives. In bringing together these two pieces, we were able to think outside the box and provide cloud-based options for our most digitally-savvy clients.

Wendy M

Software Engineer

What’s the biggest technical challenge you’ve faced recently in your work? What made this particular challenge so tricky?

Building an application is easy. Data comes in, data is processed and an output is received. Voila! How hard can it be, right? 

It’s easy when data comes in as expected, but building an application to prepare for unforeseen issues can be a bit more complex. A big part of my job is making sure that our trading platform stays up 99.99 percent of the time — however, it can be challenging to keep our system up through heavy volume days, free of network interruptions and responsive to the mandated compliance checks.

Designing a scalable distributed system that is capable of handling all of the different kinds of unexpected scenarios is particularly challenging and tricky, but it’s a part of my job that I enjoy. 

How did you and your team overcome this challenge in the end? What were some of the specific tools or technologies you used?

To ensure that we are prepared for any unforeseen issues to our platform, it is important that we understand all limitations — from the operation system to the overall network, from our own code to all the third-party softwares. The best approach is to avoid reaching these constraints. 

Redundancy is also one of the best resolutions to deal with unexpected failures. In the current era, anything can happen — pandemics, lockdowns, power outages or extreme weather. Sometimes unexpected scenarios are unavoidable, so being able to recover from failure and shorten the down time becomes the best mitigation to combat the unexpected. 

Lastly, collaboration and coordination are key tools in mitigating risks. Integration testing, regression testing and stress testing are important to help identify the points of failure and prevent unexpected outcomes. Before we roll out any product, we ensure that is passes thorough internal testing.

Sometimes unexpected scenarios are unavoidable, so being able to recover from failure and shorten the down time becomes the best mitigation to combat the unexpected.”

How did this technical challenge help you grow as an engineer or help you strengthen a specific skill?

I have learned to always think of the impossible and prepare for the unexpected — not to design for what you know, but design to make ways for what you don’t know. 

When I first started working in the financial industry, I never believed that cryptocurrency would one day take off. The cryptocurrency market has evolved significantly and gained momentum in recent years which has largely impacted my role and how I think about risks. At DRW, the high standards and coding best practices create an environment for me to not just focus on designing applications for 99.99 percent availability, but to spend time developing software that can handle the 0.01 percent of defective scenarios.

Kali

Software Engineer

What’s the biggest technical challenge you’ve faced recently in your work? What made this particular challenge so tricky?

The biggest technical challenge I faced recently was also one of the most interesting! What’s wonderful about Screencastify is that I always feel like my opinions are heard and I am given the space to explore, experiment and learn. Recently, we were working on a bug fix and I noticed something about some data that just seemed off. We keep a registry of all the videos users create and use the registry to enable features such as interactive questions and our watch page — there seemed to be some users that had odd patterns in their entries, as if they were creating a very high number of videos per day. Not just super users, but super duper users! 

Our initial data showed numbers that seemed somewhat benign, but because of how the incoming data was structured it was misleading. After digging a little further and aggregating more appropriately, I found a shocking number of affected entries that were resulting in unnecessary writes to our database that polluted data and would eventually have disrupted one of our newest features getting ready for wider release. This bug was lying in wait and the effects had the potential to be explosive.

How did you and your team overcome this challenge in the end? What were some of the specific tools or technologies you used?

Initially, the work was heavily centered around using SQL with Google’s Big Query and Firestore to search and sort the data. Ultimately, I figured out that we were seeing duplicate entries that had unique IDs and unique reference paths, and all were marked as pending uploads. At this point I pulled in a staff engineer on my team. Together, we traced the issue back to a quirk in how our system was interacting with Google Drive’s upload API: Failed uploads, on retry, were creating unique entries — and this was happening several hundred times a day on a single video. Multiply that by several thousand users and the numbers got very big, very quickly. 

Soon after these discoveries were made, the collaboration process kicked into high gear and a cross-team working group was assembled to craft a fix. This bug would have affected a major feature from another team, so it was very important we had eyes on this from them as well. Working together, we created new routes and checks to ensure it wouldn’t continue to happen. It was so great to have people in from other teams and gain insight into their features and working styles. 

Even if my hunch ended up not being correct, I would have still learned so much just engaging in the process and seeing it through.”

How did this technical challenge help you grow as an engineer or help you strengthen a specific skill?

I think as a newer engineer it is easy to second guess yourself and assume the more experienced folks around you always know exactly what is going on — but often, the fresh eyes see things others overlook. It’s important to trust your gut, be persistent and never be afraid to ask questions. 

Most importantly, know when to ask for help and to communicate your findings with everyone — not just your team. This bug was found by me, but the ultimate impact would have been on another team’s work. Without collaboration, it may still be a lingering issue. The fix ultimately ended up being fairly straightforward, but something that I had never done with our stack. Being able to see the bug through from discovery to solution was a very rewarding learning experience and the opportunities to do this at Screencastify are plentiful. Big picture, even if my hunch ended up not being correct, I would have still learned so much just engaging in the process and seeing it through. If there’s something to learn along the way, it’s always worth doing!

Hubert Shon

Full Stack Engineer

What’s the biggest technical challenge you’ve faced recently in your work? What made this particular challenge so tricky?

Our team was tasked with creating a dynamic dashboard builder. The builder would allow users to create and edit their own charts from their chosen data, modify display properties on the fly and arrange charts into custom dashboards. 

My experience thus far as a software developer had been handling a few inputs at a time — names, dates, checkboxes, all quite manageable. Displaying multiple types of charts meant numberless possibilities of what could happen, and what could go wrong. We had also introduced a new chart library that we were still getting to know — I wasn’t sure how these new components would behave under certain conditions, or how the system would communicate. Briefly put, I was trying to build something, while still learning how it actually works. That would be concerning coming from a heart surgeon, but as my senior developer put it, “that’s pretty much all of programming!”

Online searches were the go-to resources when we didn’t have immediate answers, but the examples we found were far less dynamic and customizable than what we were trying to achieve. Little was helpful to our specific use case — we were seemingly alone in what we were trying to accomplish.

How did you and your team overcome this challenge in the end? What were some of the specific tools or technologies you used?

Angular projects usually revolve around predefined templates. However, the fluid nature of these dashboards pushed us to find a more flexible route: Instead of templates, we learned that these components had to be created dynamically. While examples had guided our early setup, we were finding problems. For example, we would often find a chart disappearing or seeing display settings intended for one chart get applied to all charts.

I methodically combed through the code piece by piece and line by line. All the while I asked: “How does this piece change what we are building? What possibilities does this addition now provide?”

With a deliberate process and a watchful eye on our browser’s developer tools, our breakthrough came through utilizing angular instances. While creating components dynamically, we should have also realized that we had access to the instance of that component. These instances can be stored, referenced and updated just like any other object. The discovery helped tie all the pieces of the dashboard builder together.

When building our solution, I didn’t know how to build the entire structure — but I did know how to stack up the bricks.”

How did this technical challenge help you grow as an engineer or help you strengthen a specific skill?

If you could forgive the illustration, some of the most compelling stories of Tony Stark — the regular person behind the superhero, Ironman — are when he must face his challenges while stripped of his weaponized suit, advanced gadgetry and AI companion. Even without these powers, he still surmounts his challenges because his character is not one that relies on his privileges, but on his engineering ingenuity to piece together and build a solution. 

As an early-experience engineer, this challenge had me draw from the fundamental abilities and characteristics of a programmer. When building our solution, I didn’t know how to build the entire structure — but I did know how to stack up the bricks. I knew how these building blocks of code are supposed to behave, even if they did surprise me at times.  With this assurance and methodical approach, the application came together piece by piece. 

Technology will change and advance, but these developer qualities seem to always be a necessity. Problem solving, learning to adapt, adapting to learn and a curious itch — these are the traits that will stick with you wherever you go. And I can rely on them to keep me undaunted when the next formidable problem comes our way.

Robert Hahn

Senior Software Engineer

What’s the biggest technical challenge you’ve faced recently in your work? What made this particular challenge so tricky?

One of the biggest technical challenges we’re facing is the continuous scaling of Optiver’s machine learning (ML) capabilities in such a fast-paced environment. We have to execute extremely quickly in order to trade more effectively and efficiently in the markets, and when we’re constantly increasing the complexity of our ML models — adding more market data and signals and trading more financial instruments — it can be a challenge to keep up.

How did you and your team overcome this challenge in the end? What were some of the specific tools or technologies you used?

We tackled the problem in two ways. To start, we dramatically increased our machine learning pipeline’s compute and data capabilities by building a first-class compute cluster in house. This was not only to meet the immediate demand, but to also facilitate future growth. A number of different departments had to collaborate and build out new areas of expertise to meet this ambitious goal. 

At the same time, we invested in building an entirely new — and considerably more capable — framework for all of our research needs. This new fundamental building block allows us to operate more incrementally and provide more visibility and understanding of problems we encounter, while also enabling the exploration of entirely new research areas and approaches to the challenges we face.

Working on this project has exposed me to a new set of challenges, particularly involving the intersection of science and engineering.”

How did this technical challenge help you grow as an engineer or help you strengthen a specific skill?

This project really allowed me to develop my knowledge and technical skills for building a highly-scalable machine learning platform. Prior to joining Optiver, I spent most of my career working on microkernels and low-level embedded systems, such as satellites. Working on this project has exposed me to a new set of challenges, particularly involving the intersection of science and engineering. It’s been an exciting task to overcome and I’m continuously learning along the way.

Pawlos Campbel

Software Engineer

What’s the biggest technical challenge you’ve faced recently in your work? What made this particular challenge so tricky?

Recently we had to support the integration of varying precision fractional quantities into our existing integer quantity workflow for routing orders. This code was written in C++ using a lot of memory-efficient practices. All of our high-speed routing code was set up to only handle whole number quantities — but with the introduction of cryptocurrencies and a desire to work with more vendors, we had to pivot to handle fractional quantities as well without sacrificing performance or backward compatibility. 

Complexity mainly lay in permitting large decimal quantities of upwards of 18 digits of precision, while making sure integer quantity functionality functioned the same. JSON integers were used to represent order quantities, however much of the software was written well before the expectation of the addition of cryptocurrencies. With this in mind, we had to ensure back end code was capable of receiving both integer JSON values, as well as floating point numbers. Another issue presented with floating point numbers was the size of available data types and what we were able to store without dramatically changing our code.

How did you and your team overcome this challenge in the end? What were some of the specific tools or technologies you used?

Due to limitations with other pieces of our backend, we decided to forego using the JSON floating point object and instead accept both integers and string representation of numbers. The code responsible for compiling order information from JSON into serializable data basically had to implement a go-between functionality based on the type of data received. Strings are automatically parsed to determine whether there is a decimal component or not. We then split the string into two separate 32-bit integers to guarantee we have enough room to store the whole and fractional portion — allowing up to 19 digits of precision. This way, we could move forward with the original whole number routing functionality, with an optional storage for the fractional component. The value is preserved using a constant to ensure the original fractional number is returned to the backend on a response from a vendor containing the fully constructed fractional number.

Every decision should be made with purpose — and it’s important to keep in mind future improvements.”

How did this technical challenge help you grow as an engineer or help you strengthen a specific skill?

Our C++ codebase is rather large and had been written before I started at this company a couple of years ago. As a result, to solve this particular problem I had to dig into a lot of the existing code to gain a deeper understanding of the decisions made and why they were made before willfully changing code to fit a new feature. I’d argue that this skill is the core of any decision you have to make as a software engineer. It helped me realize that every decision should be made with purpose — and it’s important to keep in mind future improvements. This makes your job easier down the line and will lead to less “hacky” design.

Another practice that our company emphasizes is test-driven development, something I wasn’t used to in the C++ realm coming from previous positions. Having tests to verify object and method behavior is invaluable when designing a new feature such as this, as it gives us a baseline to always come back to basic program functionality. Working on this challenge definitely improved my understanding and adeptness at TDD.

Chris Guevara

Sr Software Engineer

What’s the biggest technical challenge you’ve faced recently in your work? What made this particular challenge so tricky?

Our team was tasked with expanding upon the core functionality and makeup of our course content and creation. This involved a deep dive of the workings of the creation, persistence and consumption of our core course components. Extending functionality at this level is tricky due to the importance of keeping existing processes and user experiences unchanged and unaffected by the new feature set.

How did you and your team overcome this challenge in the end? What were some of the specific tools or technologies you used?

We took care to leverage the input of many stakeholders and experts across different areas of our platform and software stack. We created entity relationship diagrams to better illustrate ideas and uncover potential issues — these were paired with potential designs where we discussed the UI/UX process and how it related. When needed, we took a step back to reevaluate parts of the approach and altered based on newly discovered product needs.

I was able to reach out to experts in varying areas and pull ideas together to make solid, confident decisions.”

How did this technical challenge help you grow as an engineer or help you strengthen a specific skill?

As an engineer and the lead on the main team working on these features, this process strengthened my ability to navigate through different departments. I was able to reach out to experts in varying areas and pull ideas together to make solid, confident decisions about our direction. It helped me grow in my ability to communicate ideas and issues with multiple stakeholders and balance needs and concerns.

Cody Fayolle

Lead Software Engineer

What’s the biggest technical challenge you’ve faced recently in your work? What made this particular challenge so tricky?

NextCapital manages tens of thousands of accounts that need to be regularly maintained and serviced over time. Currently, many service requests are handled manually by our portfolio ops team — which as we continue to grow will be very difficult to scale. To solve this, my team is building a framework to generalize how we implement service requests. Each request varies in nature, so our framework needs to handle connecting to multiple parties and partners across a variety of APIs. The framework requires significant state management to ensure that these actions are accurately executed in a timely manner.

How did you and your team overcome this challenge in the end? What were some of the specific tools or technologies you used?

Our framework is primarily built with Ruby on Rails using a combination of models and specialized classes. One key decision we made was to intentionally rely on plain classes for all business logic regarding specific service requests as opposed to single table inheritance within our modeling. We wanted our models to serve purely as data storage and not be concerned with request-specific logic or actually running the framework.

NextCapital has several partners who often have a different set of use cases. In order to accommodate this, we have an internal Go Serverless app using AWS technologies that stores partner-specific config and is integrated across our architecture. Our new framework takes advantage of this tooling to enable partner-specific service request implementations.

Working on a project of this scale has reaffirmed that you need to have a strong foundation of trust within your team in order to succeed.”

How did this technical challenge help you grow as an engineer or help you strengthen a specific skill?

This has been my first experience in designing and implementing a complex system from scratch, so there has been plenty of growth. Before diving into any code, our team spent several weeks workshopping the design and architecture for this new system. We worked closely with our product owners and stakeholders to ensure our framework would cover all of the required use cases. This experience taught me the importance of having a strong line of communication between engineers and the product team.

Furthermore, as the primary tech lead for this framework, I have collaborated closely with the developers on my team to make sure we are aligned. This has greatly strengthened my ability to discuss technical issues in a more practical context. Working on a project of this scale has also reaffirmed that you need to have a strong foundation of trust within your team in order to succeed.