Documentation has a cost, some documentation has a high level of value, some has a low level, some has no value.

#AgileDataWoW

The best way to tell which is which is to track who uses the documentation, and what they use it for.

Understand if it is worth the effort to maintain the documentation that already exists.

If there is nobody using it, why would you maintain it, remove that wasted effort.

Then track all the questions you and your team get asked on a regular basis, and then understand why there is no documentation to answer it.

Once you understand those things, you can understand if it’s worth the effort to create the missing documentation.

If you’re spending time explaining the same thing to multiple people, document it, remove that repeating wasted effort.

When I start coaching a new Data and Analytics team, or start as a virtual CxO for an organisation I always want to know what System of Capture we collect data from, and what systems and people Consume the data that is produced.

No I don’t want to trace your mad person knitting lineage graph, I don’t need that level of detail.

Hell no, I don’t want to look at your DAG’s or read your plethora of SQL, Python code or your 1,000 YAML files.

I want a simple table that tells me:

System name

Method of Collection or Consumption

Reason for Collection or Consumption

System of Capture example:

Salesforce | Data is pulled via API using Google Cloud Transfer | Customers and their contact details are collected.

System of Consumption example:

Marketo | Data is pushed to a Cloud File Bucket, rest of the collection process is managed by vendor | Customers and their email addresses are used to send marketing communications.

Don’t boil the ocean, keep it simple, I don’t need a novel. I just need to be able to scan the entire data domain to get a sense of its depth and breadth, the level of complexity in place and help focus on what should be worked on next.

Make sure you update this documentation when you make changes, out of date documentation is worse than no documentation. People will trust documentation is true, but if no documentation is available then they will go and find the truth.

And lastly if your documentation is behind an internal sign-up-wall, for example in Confluence, but to get access you have to go through a 2 week approval processes, and that is if you know it exists there and what approval process to go through, then make some of your documentation available outside of this hidden area. Not all of it, revert to the first two paragraphs above to see how to decide what to publish.

So if I popped into your organisation tomorrow, and asked for a list of Systems you collect data from, and the system or people who consume your data, how quickly and easily could I find that information?

Have you even got it documented anywhere to start with?