SDS 487: Fixing Dirty Data

With a decade of experience fixing your dirty data, Susan Walsh is the Classification Guru. Susan is Founder and Managing Director of The Classification Guru Ltd, a specialist data classification, taxonomy customisation and data cleansing consultancy. She is an industry thought leader, TEDx speaker and author of the soon-to-be-published ‘Between the Spreadsheets: Classifying and Fixing Dirty Data’. She’s also the founder of COAT. Susan has developed a methodology to accurately and efficiently classify, cleanse and check data for errors which will help prevent costly mistakes. This could save days of laborious cleansing and classifying and can help your business find cost savings through spend and time management – supporting better, more informed business decisions.

We kicked off the conversation talking about some of the things that don’t necessarily translate well from UK English to US English in Susan’s Data Den videos. But, I came across Susan’s work at DataScienceGO, as a highly requested speaker. She spoke at the conference about her “COAT system” a way of imagining the data “has its coat on” which comes from the UK saying “get your coat, you’ve pulled”, a term referring to picking someone up at the bar. COAT stands for consistent, organized, accurate, and trustworthy, outlining all things that data must have in order to be ready for actionability. The trick, of course, is data maintenance to ensure the data continues to “have its coat on” as Susan says. But she believes in teaching organizations to look after their own data after her work is done.

Susan’s company works to get coats on the data for companies and clients. She works a lot with procurement data, the data on everything bought for the company to allow it to function (such as cleaning costs, rent, utilities, IT, legal services, and so on). This data is famously messy. There are often misspellings on invoices, lack of consistency between titling items, lack of organization and categories, and more. Susan’s team is non-automated, doing their cleaning work manually to normalize and classify data. One of the tools they use is Omniscope which allows for live data editing into visualization. Many of Susan’s case studies involve supplier data, digging into which can reveal her clients are spending more money than they thought with certain companies. This allows them to streamline their suppliers, negotiate better rates, or cut ties with suppliers they don’t need. The data, of course, can be very messy considering the companies and their suppliers come from all over the world. It can take days to normalize without AI. Another case study involved Susan building a customized taxonomy for a client when she found over $30 million sitting in the wrong area of their reporting. It raised an important issue: everyone is interested in refresh but ignores the data that is already there and potentially faulty.

Ultimately, Susan wrote a book about her work and how companies and practitioners can learn to be more efficient with their data. She emphasizes that this is a real problem for companies that many want to ignore. Many of the folks who pay the bills for services are not data people and Susan aims to get them interested in the topics she covers in work and the importance of services like hers. You can’t make the right decision with bad data. Data modeling is always the focus but clean, useable data is more important. You can’t model on faulty data.