DataChef's Multi-Account Journey: Part 1

July 27, 2022
Author:
Milad Beigi



Introduction

AWS has been the leader in the cloud computing industry for many years. As many companies move their infrastructure to the AWS cloud, they think of an AWS account as the logical equivalent of their on-premise infrastructure. However, AWS recommends that companies use multiple accounts to run different environments and workloads. We call this practice a multi-account architecture. We believe this practice is not for every organization and can add a lot of engineering work to manage your AWS environments. We are here to share our experience with you on this topic.

DataChef’s AWS multi-account journey had a lot of ups and downs. So we decided to share our experiences with the community through a series of articles. Hopefully, these posts will be helpful for many companies and individuals like us. In the first part, we start with why “Multi-account ” architecture is an industry-standard right now and why DataChef needed this approach. In part two of this series, we talk about different challenges faced during development ranging from technical to cultural or even wrong decisions made along the way. Finally, in the last article, we explain how DataChef now manages its AWS accounts and why this approach is selected.

Why AWS Multi-Account architecture?

AWS strongly recommends multi-account architecture to its customers, and we can hear this argument from different channels. From AWS’s regular blog posts to the AWS Re-invent, the multi-account architecture is always one of the topics. AWS believes:

“Using multiple AWS accounts helps you isolate and manage your business applications and data. This approach optimizes most of the AWS Well-Architected Framework pillars, including operational excellence, security, reliability, and cost optimization.”

AWS suggests Multi-account architecture for various reasons. In general, AWS accounts serve as a security boundary and a resource container that provides a level of isolation for the organization. By separating your resources into separate accounts:

Multi-Account for Machine Learning, Data Science, and Data Mesh
  1. You can define separate security profiles or control policies on your applications.
  2. Isolate one account from others. So potential security threats and risks would only affect one account.
  3. Teams within your organization can have different accounts separating their resources and responsibilities. You can define budgets for your business units inside your organization.
  4. Data stores also can be isolated, which can help limit the number of people that can access and manage that data store.

And the list goes on, as AWS states in this article.

AWS has built many services to encourage this idea and reduce the complexity of managing multiple environments from the customer’s point of view. One example would be AWS Landing Zone, which later turned into AWS Control Tower. The AWS Control Tower has been built on top of services like AWS Organizations, CloudFormation, and Service Catalog, to provide a central interface for the governance of a multi-account setup.

DataChef’s requirements

At DataChef, we had some requirements for a multi-account architecture, and this demand became more significant over time. Our company, like many others, has different areas like internal applications or tools, websites, and storage. We refer to these different areas as workloads. Moreover, developers inside the company should have access to AWS easily for experiments and learning purposes. We also needed to provide temporary access to potential candidates interviewing with DataChef, so that they could work on their take-home assignments. Again, for some trainees in our company, we needed to give them access to AWS accounts.

So anyone can imagine combining these production and non-production workloads in one account can be problematic in many ways. Without enough guardrails and budget monitoring, a bad practice like this can also be turned into a huge bill, as it did for us. This incident was a huge learning experience for us, pushing us further toward a multi-account architecture.

As a consultancy company, we always want to amplify our presence in the project’s early days, demonstrating our expertise not by hours but with the impact we bring to the customer. So getting our hands dirty with these services and pragmatically using them was also an essential factor for us. With good implementation, we could bring up as many AWS accounts as we want. Additionally, we could set up separate workloads for these accounts in various areas like Machine Learning, Data Analytics, and Data Engineering when we enter a new project. This idea evolved, became a recipe later, and eventually the starting point for our multi-account journey. In the next post, we will provide detail on the specifics of Data Analytics and Machine Learning accounts.

DataChef’s multi-account recipe for ML/DS

What is a recipe? In short, in our recipes, we try to provide clear implementations for simple business requirements. Given our new need for a multi-account implementation specifically for Machine Learning and Data Science workloads, we decided to design a recipe for it. The idea for this recipe came from the actual need of our customers. They wanted to create many pre-configured (Focused on ML/DS) AWS accounts at once without having someone create these accounts one by one.

The idea of having multiple accounts was also aligned with the notion of self-serving domains, splitting the accounts based on domains and data products, having federated governance, and central data discovery. All of these concepts come from the foundation of Data Mesh principles. This was another proof for us that this is the right recipe to work on it.

Multi-Account for Machine Learning, Data Science, and Data Mesh

Data Mesh is an analytical data architecture and operating model where data is treated as a product and owned by teams that most intimately know and consume the data.

At DataChef, we all believe that Data Mesh is the future of Data Analytics and Machine Learning, and having the proper domain structure is an essential step toward this goal.

So we started the development, assuming we can apply this to our internal usage and deliver something valuable to our customers.

We had a clear goal in mind, and the recipe and its significance were promising. But that wasn’t enough, there were many challenges along the way and many lessons to be learned. In the following article, we are going to share everything with the community and explain why things didn’t go as we expected and how we had to stop and think again about this recipe from the start.

Conclusion

As we discussed, the AWS multi-account architecture is industry-standard right now and is promoted by AWS for various reasons. We also pointed out how companies like DataChef can benefit from this architecture in their internal usage and offering consultancy to other companies. But we also believe there are some considerations on this topic. After all, AWS multi-account architecture, like all cloud solutions, may not be a good fit for all organizations. We will discuss the last case further in subsequent posts in this series.

Made with + in Amsterdam

Back to top