ADI WoW — Intro — The Why and the What

The Why

We have been on a journey for the last ten years working with a bunch of organisations and teams crafting ways of combining patterns from the agile and product domains with patterns from the data domain, into new AgileData Ways of Working.

These new ways of working and the related patterns have added value to each of the data teams and organisations we have worked with and we want to share this knowledge and experience with others, so hopefully they can get similar value from them.

For too long data teams have worked in traditional ways that have caused a myriad of problems for the data work they do.

Let’s start with a story that may be familiar to you

The Department of Magic is a government agency tasked with regulating and overseeing the use of magic within the United Kingdom. Located in Somerset, a rural county in the southwest of the United Kingdom, the department is made up of a diverse group of magic users and non-magic users who work together to ensure the safe and responsible use of magic.

As a government agency, the Department of Magic has faced its fair share of challenges in its long history. However, its dedicated staff and commitment to upholding the highest standards of integrity and professionalism have helped it earn the respect of both magic users and non-magic users alike.

The department’s main duties include issuing licenses to magic users, enforcing magical laws and regulations, and responding to magical emergencies. Its staff consists of a range of specialists, including magical investigators, spellcasters, and magical creature experts.

Despite its important role in maintaining order and safety in the magical world, the Department of Magic is often misunderstood and misrepresented by those who are unaware of its true purpose. But for those who work within its walls, the department represents a rare opportunity to make a positive impact on the world through the responsible use of magic.

The Department of Magic’s legacy data warehouse was a problem that had been plaguing the agency for years. Based on an outdated technology called Soothsayer, the warehouse was 20 years old and no longer fit for purpose.

The warehouse was prone to frequent glitches and outages, which made it difficult for staff to access important data and information. This caused delays and frustration for magic users and non-magic users alike and made it hard for the department to keep up with its workload. In addition, the Soothsayer technology was cumbersome and time-consuming to use, which slowed down the department’s operations and made it difficult to adapt to changing needs and demands.

As if these problems weren’t enough, the legacy data warehouse was also a security risk. The outdated technology made it vulnerable to magic attacks, and the lack of proper safeguards put the department’s sensitive data at risk. This created a potential liability for the agency and jeopardized its reputation and credibility.

The legacy data warehouse was also a drain on resources. The outdated technology required frequent maintenance and repairs, and the cost of keeping it running was becoming unsustainable. This put a strain on the department’s budget and made it difficult to allocate funds for other important initiatives.

Despite these challenges, the Department of Magic had been slow to address the problem. Many within the agency saw the legacy data warehouse as an untouchable sacred cow, and any attempts to modernise or replace it were met with resistance.

But as the problems with the warehouse grew more pressing, it became clear that something needed to be done. The Department of Magic could no longer afford to ignore the issues with its legacy data warehouse, and it was time to take action.

The Department of Magic had long been aware of the problems and it had made several attempts to modernise and improve it. As part of its organisation-wide digital transformation programme, the department had brought in a consulting company called Pyramids for Hire (PFH) to help implement the latest big data technology Babar.

The department was under pressure to deliver new operational reporting for the modern systems that the digital transformation program was implementing, and it hoped that the Babar technology would be the solution. However, things did not go as planned. The PFH team was inexperienced and unprepared for the challenges of the project, and the Babar technology was not fit for purpose. The project quickly ran into problems and eventually had to be paused.

The failure of the project was a major blow to the department. Not only had it wasted valuable time and resources, but it had also damaged the agency’s reputation and credibility. The department was now left without a reliable way to generate the operational reporting it needed, and it was falling behind on its digital transformation goals. Many within the department were disillusioned and frustrated, and morale plummeted.

To make matters worse, the legacy data warehouse was still causing problems. The outdated technology was becoming increasingly unreliable and costly to maintain, and the department was no closer to finding a solution. It was clear that something needed to be done, and the Department of Magic couldn’t afford to wait any longer.

As the problems with the Department of Magic’s data warehouse project continued to mount, it became clear that something drastic needed to be done. That’s when a project troubleshooter named Laszlo arrived on the scene.

Laszlo was a seasoned veteran with a reputation for rescuing data projects in trouble. With a quick assessment, he was able to get a handle on the problems facing the Department of Magic’s previous data warehouse project.

According to Laszlo, the typical causes of data warehouse projects going off the rails included inadequate planning, unrealistic expectations, poor communication, and a lack of understanding of the business needs. These issues were often compounded by a lack of resources, unrealistic deadlines, and a lack of clear ownership and accountability.

Let’s take a pause for a second

In our experience as consultants in the data domain, we have seen data projects that have been deemed a success and data projects that have been deemed a failure.

As with all things AgileData the definition of success or failure is always dependent on the context of your data teams and your organisation.

We often see what we call the 3,2,1 data projects. 3 Million dollars, 2 years and 1 report in production. To some this is a success, to most it is not.

It is widely recognised that in both the traditional data warehousing and subsequent big data waves, 80% of the data projects didn’t manage to get the one report into production or deliver the outcomes they promised when the projects were first started.

You may have experienced these scenario’s, your data project being cancelled before it managed to fully replace and decommission the previous data platform, leaving you with two platforms to maintain, or your data project not delivering the features and outcomes that were promised in the initial business case used to get permission to undertake the work, leaving the Business as Usual (BAU) team to somehow find the time to add these features now the project has been deemed to have been “delivered”.

Often the first cost estimates are revised upwards during multiple project resets, while at the same time the list of features and benefits which were promised to be delivered are constantly reduced.

Ralph Hughes describes it in his 2008 Agile Data Warehousing book as the “Disappointment Cycle”.

This variation of the diagram from Ralph Hughes book tracks the value vs cost as the data project constantly undergoes resets and replanning.

It seems during the subsequent big data, data science and AI waves post 2008, nothing has changed, apart from the technology used. Surveys still outline a problem of over promising and under delivering in the data domain.

It is a problem with the way we are working, not with the technologies we are working with.

Let’s continue with the story

Laszlo had seen projects fail because the scope of the project was not well defined, leading to scope creep and unrealistic expectations. He had also seen projects fail due to poor communication between team members, resulting in misunderstandings and misalignment. In other cases, he had seen projects fail because the technology being used was not fit for purpose, or because the team did not have the necessary skills and expertise to use it effectively.

After completing his assessment of the Department of Magic’s previous data warehouse project, Laszlo was convinced that the best way to rescue the project was to use an agile approach. He had used agile in the past to successfully resolve data project issues, and he was confident that it could work for the Department of Magic as well.

Laszlo knew that the programme leadership team was out of time and money, and he knew that they would be hesitant to try something new. But he also knew that the agile approach had a number of benefits that could help the department get its data warehouse project back on track.

He presented the benefits of agile to the leadership team, explaining how it could help them deliver value earlier, reduce risk, and improve collaboration and communication. He also explained how agile could help the department adapt to changing needs and demands, and how it could help them build a culture of continuous improvement.

To the leadership team’s surprise, they were persuaded by Laszlo’s arguments. Despite their reservations, they agreed to try the agile way of working, recognising that it was their best hope for rescuing the data warehouse project. With the leadership team on board, Laszlo was ready to get to work.

Laszlo knew that he would need some help. That’s when he remembered Shane, an AgileData coach he had worked with at a previous organisation.

Laszlo reached out to Shane and the two of them agreed to meet at a local bar to discuss the situation further. Over a bottle of mead, Laszlo outlined the problems facing the data warehouse project and explained how he thought an agile approach could help.

Shane listened carefully and asked a few questions to get a better understanding of the project. “How does the team currently work?” he asked. “And are they onboard for the change to agile?”

Laszlo explained that the team was used to working in a more traditional, hierarchical way, and that they were hesitant to try something new. He also mentioned that there was some resistance to the idea of agile, and that he was worried about getting buy-in from the team.

Shane nodded. “And what about support from the leadership team?” he asked. “Do they have the resources and commitment to see this through?”

Laszlo admitted that the leadership team was stretched thin and that they were under a lot of pressure to deliver results. He also mentioned that they were worried about the cost and time involved in switching to agile.

Despite these challenges, Shane agreed to help. Shane was an experienced AgileData coach with a proven track record of helping organisations improve their data projects. He understood the challenges that data projects often faced and knew how to use agile techniques to overcome them.

Laszlo was confident that Shane could help the Department of Magic get its data warehouse project back on track. With Shane’s expertise and guidance, he was confident that they could turn things around and deliver the value that the department needed.

After agreeing to help Laszlo rescue the Department of Magic’s data warehouse project, Shane began his work at the department. He started by observing the current data team and trying to get a sense of the things that were working and the things that were not.

One of the first things Shane noticed was that the team was trying to use a traditional, waterfall approach to deliver the data work. This was not surprising, as many data teams were accustomed to using a traditional way of working. However, Shane also knew that waterfall was often not the best approach for data projects, and he was concerned that the team was setting itself up for failure.

As he watched the team work, Shane identified several problems that were hindering their progress. One of the biggest issues was the fact that the consulting company, Pyramids for Hire (PFH), was operating in a silo and not providing any visibility of the work they were doing to the team. Despite supposedly reporting the status to the senior executive who was their internal sponsor, the team had no idea what PFH was doing and had no way to track their progress.

Shane knew that these problems could be addressed by helping the data teams to combine patterns from the agile, product and data domains into a new AgileData Way of Working, and he was determined to help the team make the transition.

Shane knew that the consulting company PFH was a major contributor to the problems facing the Department of Magic’s data warehouse project. He also knew that if they were going to rescue the project, they would need to get PFH on board.

He discussed this issue with Laszlo and suggested that they try to get PFH to change the way they worked. Specifically, Shane suggested that they ask PFH to embed their data team into the same team as the department’s data team.

Laszlo was skeptical, but he agreed to try. He approached the partner from PFH, Marcus, and made the suggestion. To his disappointment, Marcus flatly refused.

“That’s not how we work,” Marcus said. “We prefer to operate independently and maintain our own best practice processes and methods.”

Laszlo knew from experience that PFH was using a model where they were relying on inexperienced graduates to do the work, and where the practises and methods were being constantly reinvented and reimagined and he suspected that they did not want to expose this to the department. He also knew that this was a major part of the problem and that it would be difficult to move forward without addressing it.

After his unsuccessful attempt to get PFH to change the way they worked, Laszlo knew that he needed to come up with a new plan. He decided to have a talk with the head of data and analytics at the Department of Magic, Maria.

Laszlo explained to Maria the problems with PFH and how they were hindering the data warehouse delivery. He also outlined his idea for building an internal, permanent team to take over the work.

Maria listened carefully and considered Laszlo’s proposal. After some discussion, she agreed that it was worth a try. She recognised the value of having a dedicated, in-house team to work on the data warehouse project and saw the potential benefits of using an agile approach.

Pleased by the news, Laszlo immediately contacted Shane and explained the plan. Shane was glad to hear that they had the support of the leadership team and was eager to get started. Together, Laszlo and Shane set to work building the internal, permanent team and preparing for the transition to a new way of working.

As part of the transition to agile, Shane knew that it was important to provide some overview training for the data team. He wanted to make sure that everyone understood what agile was and how it worked in the data domain.

Shane started by giving a presentation on the core principles of agile and how they applied to data work. He emphasised the importance of flexibility, adaptability, transparency, and collaboration, and explained how these principles could help the team deliver value earlier and more effectively. He followed up with some of the challenges that typically arose when applying agile principles to data work, and some of the patterns that helped resolve these challenges. He then listened and answered the myriad of questions the team had.

After the presentation, Shane led the team through a series of exercises to establish some foundational AgileData patterns. He helped them define a team agreement, which outlined their shared values and expectations. He also facilitated the creation of an initial definition of ready, which described the criteria that work items needed to meet before they could be considered ready for development. Finally, he helped the team define an initial definition of done, which described the acceptance criteria that work items needed to meet before they could be considered complete.

By working through these foundational patterns, Shane was able to establish a common understanding and set the stage for the team’s transition to agile. He was confident that with this foundation in place, the team would be well-equipped to succeed.

As part of the transition to agile, Shane knew that it was important to get a clear understanding of the data team’s current processes and identify any problems. He decided to start by having the team document their current data value chain, which described the way they moved data from source systems through to the end consumers and systems that needed to use the data.

Shane led the team through the process of documenting their data value chain, and he encouraged them to be as detailed and specific as possible. He also asked them to identify any problems or bottlenecks that they encountered along the way.

The team was able to identify several issues with their current data value chain. For example, they had difficulty getting timely access to data from certain source systems, and they struggled with data quality issues that often arose when data was transformed or migrated from one system to another. They also had trouble coordinating with the end consumers of the data, which led to delays and misunderstandings.

Shane knew that these problems would need to be addressed if the team was going to succeed with their new AgileData Way of Working. He was grateful for the team’s insights and was confident that they would be able to work together to overcome these challenges, many he had already experienced when working with other data teams.

Shane also knew that it was important to understand the skills and capabilities of the data team. He decided to start by having the team document their T-shaped skills, which described the depth and breadth of their expertise.

Shane led the team through the process of documenting their T-shaped skills and helped them identify any gaps in their knowledge or experience. As they overlaid their skills, it became clear that there were significant gaps in the areas of data platforms and DataOps. These were areas that were currently being handled by the PFH team, and Shane knew that they would need to be addressed if the team was going to succeed with their agile transition.

Shane was confident that with support from Laszlo and Maria they would be able to work together to fill these gaps. He knew that it would take time and effort, but with the right training and support, he was confident that they would be able to overcome these challenges and deliver the value that the department needed.

The data team had significant gaps in their knowledge and experience when it came to data platforms and DataOps. Shane knew that these were critical areas that needed to be addressed if the team was going to succeed with their transition to a new way of working.

That’s when he thought of Nigel “the plumber”, an expert in implementing data platforms and automating processes using DataOps patterns. Shane had worked with Nigel at a previous organisation and knew that he would be the perfect person to help the team fill these gaps.

Shane reached out to Nigel and the two of them agreed to meet at a local bar to discuss the situation further. Over a cider, Shane outlined the problems facing the data team and explained how he wanted Nigel to come on as a DataOps coach to teach the team how to do the work, rather than doing it in isolation like PFH had done. The goal was to make the team self-sufficient and capable of handling these tasks on their own.

Nigel listened carefully and asked a couple of questions to clarify his understanding of the situation. He wanted to know more about the team’s current skills and experience, as well as the overall goals and objectives of the new data platform.

Once he had a better understanding of the context, Nigel agreed to help. He was excited by the opportunity to work with the data team and was confident that he could help them succeed. Shane was grateful for Nigel’s willingness to help and was confident that with his guidance and support, the team would be able to overcome their challenges and deliver the value that the department needed.

So how does the story end?

Well as with all things AgileData that depends.

We have worked with organsiations where the story ended with what everybody viewed as a massive success. We have also worked with organisations where value was delivered, but not as much value as we had hoped when we started.

It depends on the context surrounding the data teams, the data leaders and the organisations themselves on how much value an AgileData Way of Working delivers.

In this book we provide a set of patterns that you can choose to adopt and which have proven to add value to the data teams and organisations we have worked with.

The What

As part of that journey I have become fixated on the concept of patterns.

The Big Book of AgileData Ways of Working brings patterns from the agile, product and data domains and applies them to the specific challenges data teams have when managing complex data in complex organisations.

These patterns span the ways teams are organised, the processes and practises the teams follow and the data tasks the teams undertake.

What is a pattern?

“Patterns are a widely used concepts to describe good solutions to recurring problems using a common language.”

Patterns can be found in many different domains, including art, architecture, music, nature, and human behavior. In art and design, patterns are often used to create visual interest and repetition, while in nature, patterns can be observed in the arrangement of plants and animals, as well as in physical phenomena such as weather and climate.

Overall, patterns are a useful tool for understanding and organising the world around us. They can help us to identify regularities and repetitions in complex systems, and can provide a starting point for solving design problems. By studying and applying patterns, we can gain insight and create more effective and efficient solutions.

The first person recognised to formally document and name a pattern in the context of software engineering was Christopher Alexander, an architect and design theorist.

In 1977, Alexander published a book titled “A Pattern Language,” in which he documented and named 253 patterns that he had observed in the design of buildings and towns. These patterns were organised into a hierarchical structure, with each pattern building on the ones below it to create a comprehensive system for designing and building environments.

Alexander’s work on patterns was highly influential, and has had a lasting impact on the field of software engineering. His book was the first to document and name patterns in a systematic way, and his approach to organising and applying patterns has been widely adopted by software developers and designers.

One example of a pattern from Alexander’s book is for a porch that is named “The Deep Porch” pattern. This pattern is intended to deliver a comfortable, sheltered outdoor space that is connected to the house, but also open to the surrounding environment.

To create a deep porch using this pattern, the following elements are recommended:

• The porch should be at least six feet deep, with enough space for several people to sit and relax.
• The porch should be covered by a roof that extends at least six feet beyond the edge of the house. This will provide shade and shelter from the elements.
• The porch should be surrounded by large windows or screens that allow for a clear view of the surrounding environment. This will help to create a sense of connection to the outdoors.
• The porch should be furnished with comfortable seating and other amenities, such as a table and chairs, a swing, or a fireplace. This will make it a desirable place to spend time.

Overall, the deep porch pattern is intended to create a comfortable, sheltered outdoor space that is connected to the house, but also open to the surrounding environment. By following the elements of this pattern, architects and designers can create porches that are functional, inviting, and enjoyable to use.

This example of a building architecture pattern outlines the goal that needs to be achieved, the context within which this goal needs to be achieved, the things that can be implemented to achieve this goal and the way these things move us closer to achieving our goal.

In the context of AgileData, a pattern is a recurring solution to a common problem when working with data. These patterns are documented and shared among members of the community, and can be used as a starting point for solving similar data problems.

Why use patterns?

Patterns are useful because they:

• represent proven solutions to common problems;
• organise solutions into a standard and easily referenced format;
• can be adopted and implemented by most practitioners who work in that pattern domain;
• can be used to ensure consistency in how systems are designed and built;
• reduce the level of effort required to solve specific problems;
• can be selected prior to the implementation of the system;
• provide a shared language for specific problems and potential solutions.

AgileData Patterns

As part of working with these organisations and teams I have discovered and refined a toolkit of AgileData patterns that I regularly reference and reuse.

The majority of these patterns are not new and are not invented by me, they are patterns that have been around in other domains for a while, and I have tailored them to deal with the intricacies that come with the data domain.

I see this as “standing on the shoulders of giants” and I would like to thank the original inventors of these patterns. Where possible I have referenced these people as I describe the patterns.

A couple of examples of these patterns are:

Persona’s

<<< Description >>>

Information Products

An Information Product is an approach to describe a boundary for data, analytical and visualisation requirements in a way that the business stakeholders can agree what they will get and the team can understand and deliver it in small iterations.

A Pyramid of Patterns

In Alexander’ “A Pattern Language,” book he describes 253 patterns. The patterns are organised into a hierarchy, with larger-scale patterns at the top and smaller-scale patterns at the bottom.

Here is a list of the patterns at each level of the hierarchy:

• Metaphors: Patterns that describe the fundamental qualities and characteristics of a place or space. There are 10 metaphors in total.
• Regions: Patterns that describe the larger-scale patterns of towns and neighbourhoods. There are 31 regions in total.
• Districts: Patterns that describe the patterns of intermediate-scale neighbourhoods and districts. There are 31 districts in total.
• Neighbourhoods: Patterns that describe the patterns of smaller-scale neighbourhoods and communities. There are 35 neighbourhoods in total.
• Houses: Patterns that describe the patterns of individual houses and residential buildings. There are 50 houses in total.
• Building: Patterns that describe the patterns of individual buildings and spaces within buildings. There are 41 building patterns in total.
• Rooms: Patterns that describe the patterns of individual rooms and spaces within rooms. There are 45 room patterns in total.
• Furniture: Patterns that describe the patterns of individual pieces of furniture and how they fit into a space. There are 21 furniture patterns in total.

The pattern hierarchy he uses makes sense as it is based on the physical reality of a building. A building physically has rooms, in each room is physical furniture.

In the data domain we can see hierarchies in the relationally oriented data repositories we utilise, for example databases have schemas, schemas hold tables, tables have columns.

But the patterns we describe cover more than just technical patterns, they cover ways of working as well. It is often difficult to provide a conceptual hierarchy for these patterns.

DORO — Define Once, Reuse Often

I often apply the same pattern multiple times in different ways to achieve different outcomes.

For example I use the Persona patterns as an input to the Information Product Canvas, as an input into Data Blueprints, as an input into Data Technology Evaluations, as input to defining the Team Popology, as an attribute within the Data Supply Chain, as an attribute for the teams skills matrix, and I also use the pattern as part of my courses to increase the agile literacy of data teams.

The Information Product Canvas is used to assist with the definition of a delivery roadmap, with the prioritization process, as a short form requirements document and finally as a light as built document.

Hence the concept of DORO, Define Once, Reuse Often. The benefit of this approach is that each pattern has been iterated multiple times over multiple years as it was discovered, refined and reused based on multiple industries, use cases and context.

The big book of agile data patterns

These conceptual rather than physical hierarchies and focus on reuse makes it difficult to decide the information architecture for a book that describes the patterns.

Does the book just provide a list of patterns?

While that would be useful it would not expose the context on when the pattern provides the most value and would leave the reader to work out when to combine patterns together.

Does the book tell a story?

While a story is useful to understand context, each output for me is a different story, and I would struggle to tell just one.

Choose your own Adventure

When I was growing up I remember a series of books based on adventure stories with the requisite monsters and hero’s. The exciting thing about these books is they were base don a pick your own path pattern. You started at the beginning of the book but as you read through the hero would come to a fork in the road, or to a room with two doors and you had to choose which path would they take. If they take the left path you would flip tp page 2o in the book to continue the hero’s journey bit if they take the right path you would flip tp page 25 and continue on a slightly different journey.

Adopting the DORO chose your own adventure patterns I have decided to describe each pattern first then, describe the patterns again in the context of a specific way of working output, where relevant.

These outputs being:

• Data Blueprint;
• Data Value Chain;
• Team Topology;
• Data Literacy adventure;
• Course Content.

My hope is that one of these will resonate with the problem you are trying to resolve and help you find and reuse the patterns that are valuable in solving that problem. Effectively allowing you to pick your own AgileData adventure.

If any of the outputs I combine the patterns together to deliver don’t resonate with your particular context, then feel free to treat the patterns like a big box of lego and build your own tower of useful patterns.

That is enough reminiscing about the patterns of my childhood, let’s start our AgileData adventure.