Como você está melhorando a qualidade de seus dados?

Sharing the findings of the assessments is also important so that data users are well aware about the usability of the data. It also informs where checks are needed during the data production.

Measuring the data quality on data sources is the primary step for any data quality initiative. Identify the data elements, collect all business rules around them, and convert into technical rules are initial steps. The profiling of data elements can be automated. Completeness, conformity, consistency, integrity, duplicates, validity and timeliness are the industry-standard data quality dimensions that can be used for assessing the data quality.

It's useful to ask the Steward or consumer of the data what "good quality" looks like. Does every single row of every single column in a data set need to be 100% accurate, all of the time? If not, focus efforts on the fields that do. Identifying critical data - or what has the most impact on operational or strategic work when of poor quality - will ensure that you are having the most impact into the data that really matters to your organisation.

Data profiling plays a crucial role in data quality assessment. Sometimes, you need to figure out where to start, and having a profile report will help to define data quality requirements. It all starts with the assessment, but you need to understand the relevance of each finding. For example, you can find outliers in a sales table, but the subject matter expert can help you understand that it was due to seasonality or special offers on the website. Another essential aspect is to store the assessment results. Most data catalog solutions can attach data quality measures alongside the data source so all stakeholders can access that information. For example, a data analyst can access that while building a dashboard.

Showing the data users and producers how the data quality can influence business outcomes by connecting them to quantifiable metrics wherever possible is a very important point to start with. And for this, defining a uniform and clear data standard across the org keeping in mind the industry wide standards and best practices is very important (We are currently undertaking this task in our org).

Identify the root causes of data quality issues. Ideally, the remediation process should occur at the source systems, addressing issues at their origin. Data Quality(DQ) tools should offer an intuitive user interface for effortless review and updating of DQ error data. Once errors are resolved and approved, transactions should seamlessly reintegrate into the process. Automation and the utilization of AI capabilities should be central to DQ tools, streamlining remediation processes. These tools should possess the ability to prioritize DQ issues based on their business impact, facilitating effective remediation planning. Lastly, DQ tools should provide lightweight and high-performance integration capabilities with data platforms.

Here is my approach: Data Validation: This ensures that your data meets certain criteria or constraints. Data Standardization: This makes data consistent. Converting all text to lowercase or uppercase. Standardizing date formats across the dataset. Converting currency values to a standard currency. Data Deduplication: This is about removing duplicate entries. Data Enrichment: This involves adding additional data or attributes to enhance the data's value. Data Transformation: This involves changing the format, structure, or values of data to meet specific criteria. Converting categorical data into numerical data (one-hot encoding). Lastly, documentation and backup are vital.

Establish data stewardship process, with the help from data governance. This should help move forward in the cleansing efforts. Cleansing or fixing data quality issues could be very simple and at times, it would need longer time for complex issues. Having a workflow system would definitely be recommended and handy in handling the data quality issues shared with data owners.

Always, always, go back to the source. If you are transforming data in its second or third use, when the data comes from elsewhere, you are only moving the problem. If you fix it at source (e.g. input), you are fixing it for anyone who uses it after you. I cannot count the number of examples where multiple teams are cleaning the same data in the same way - but in their own static spreadsheets. If we take the time to investigate the root cause, it saves oodles of time in the long run.

The task that fall into assessment and before cleansing is known as data profiling, with data quality objectives or domains. Assessment / Profiling with a score of the data quality will provide a clear picture on the cleansing tasks and anticipated effort. I strongly believe that such cleansing effort should be reflected back into the System of Record (SoR, if possible) and also work on ways to enhance the SoR data validations at time of capture so the data quality can be push back to the point of capture. This should be a principle of data quality: Data Quality start at the point of capture!

One thing to add here, you also need to perform health checks on your databases periodically to ensure the database is working in excellent conditions. This will also ensure that the fetching time is faster.

Data contracts are a great way to ensure communication is flowing between producers and consumers and ensure that changes are appropriately discussed and approved. For data contracts, you have a cultural and a technological component where you need to place the contracts for different datasets. Ideally, it would be best if you integrated them into your tech stack. For example, if you create a pull request for a code change, analyze that to see if the change is breaking the contract.

It will take time, but identifying the single source of truth will save a world of pain. If you can direct all of your colleagues to using the same initial dataset, you are preventing duplication and reducing the risk of manual error. It is a cultural change to get people to "let go" of their curated copies, but by showing them the value in being consistent and the time it will save them goes a long way.

In managing data sources, it's crucial to establish a robust ETL (Extract, Transform, Load) pipeline to ensure a seamless flow of data from diverse sources. Prioritize data quality by implementing thorough validation and cleansing processes to maintain accuracy. Additionally, consider implementing a comprehensive metadata management system to track lineage, dependencies, and usage, facilitating transparency and traceability across the organization.

Knowing your authoritative data source and documenting the same which is readily accessible is important here. The lineage tree and the complete metadata information should be reviewed, tested and approved regularly.

Data Quality dashboard is highly recommended and daily profiling should be needed to reflect the dashboard. Dashboard needs not only to focus on reporting issues but also alert the business team before it becomes an issue. A validated, by data consumers, the data quality dashboard would be very helpful to control the data quality.

Data quality improvement can be measured using data quality KPIs. You can calculate the data quality KPI as a percentage of passed data quality checks. You can define dedicated KPIs for different dimensions, for example, the data freshness checks can use a maximum age of data as a threshold for the KPI calculation. Once you define high-level KPI dashboards, you will need operational level dashboards with intuitive drill-down filters to pinpoint the issue. The issues that are reviewed should be converted as incidents and tracked like any other incident in the organization.

It's essential to prioritize your data quality controls and alerts. The data team could suffer from notification fatigue and unintentionally ignore crucial problems. You also need to define the severity of different issues. Is that check a reason to stop the data pipeline? Should we move the data to a "quarantine" table or schema to analyze what happened? Should we ignore an issue if it's below a certain threshold? All these questions can help you to define different policies and actions.

DQ controls should be designed through collaboration with business users, so that the key data elements are recognised and standards of those elements comprehended. Knowing and understanding these aspects from business users will help to create checks that compliment the KPIs. DQ checks should also be reviewed to gauge updations and redundancies if required. The checks should be framed like the eye of the needle, so that any abberation are recognised at the earliest.

First, establish data validation rules to ensure incoming data meets set standards, like format and range checks. Second, use automated tools for continuous data monitoring and cleansing, catching issues in real-time. Third, implement regular data audits to systematically review and correct inaccuracies. Fourth, enforce data governance policies to maintain quality across different departments.

Community engagement is key here. If a lot of colleagues are frustrated by a certain data set, bring them together to discuss their frustrations - someone will have a solution. It may be that someone is inadvertently changing data that affects others, or misinterpreting how to use it. Co-creation and collaboration are just as important as more formal training (though we need that too!)

Educating data users is paramount for extracting maximum value from our analytics efforts. I would advocate for tailored training sessions and workshops, covering topics like data exploration, visualization, and interpretation. Additionally, fostering a data-driven culture by emphasizing the importance of data integrity, privacy, and compliance will empower users to make informed decisions based on reliable insights. Regular knowledge-sharing forums and documentation can also serve as valuable resources for continuous learning.

Educating data users is complicated when you don't have a central repository with documentation and quality assessments. I highly recommend integrated solutions like data catalogs, where you can access multiple perspectives of the dataset in one place. It can start with something else like a spreadsheet, but whatever it's used, it must be maintained and shared between stakeholders.

First, provide regular training on data handling best practices, ensuring everyone understands the importance of accuracy and consistency. Second, create easy-to-follow guidelines and cheat sheets for data entry and validation. Third, use real-world examples in training to show the impact of poor data quality on business outcomes. Finally, establish a feedback loop where users can report data issues and suggest improvements, fostering a pre-emptive data quality culture.

Continuously improving data quality is not only a proactive approach, but also a strategic move to keep pace with rapidly changing business needs. In today's data-driven world, it is critical to view data as an evolving asset that requires ongoing maintenance and adaptation. Implementing a continuous data quality improvement process enables companies to respond to change in an agile way and use their data as a strategic asset that provides a clear competitive advantage. Overall, data quality should be viewed as a dynamic aspect that is not only focused on the past, but also includes a clear vision for the future use and value enhancement of data.

A data quality dashboard can come in handy to track the data quality metrics for datasets. Once initial tests for different data quality dimensions have been placed in a pipeline, they should be monitored using these data quality dimensions and then measures should be taken to address those data quality dimensions which are not up to par by introducing relevant processes during the data collection process.

Improving data quality is like tending to a garden - it requires ongoing care and attention. Implementing automated validation checks, regular audits, and feedback loops with data producers will help weed out inconsistencies. Additionally, encouraging a culture of data stewardship and providing tools for self-service data cleaning will ensure that our data garden stays lush and thriving. Remember, a well-maintained dataset yields the ripest insights!

While there are many tools to help explore, analyze, cleanse, and load various other tools, I find it useful to start with basic data profiling to understand the big picture of any dataset I am asked to process. Whether it is small or large, I like to understand what dimensions or attributes might have a small or large impact on any operation I might perform using that data. It allows me to limit the impact of surprises that might show up as I begin to process the data.

It's essential to have a robust data governance operating model in place to facilitate and support the issue remediation process. This includes clearly defined roles and responsibilities, standardized procedures and workflows, effective communication channels, and mechanisms for monitoring and oversight. By establishing a structured and systematic approach to data governance, organizations can ensure the timely identification and resolution of data quality issues, ultimately enhancing the overall quality and reliability of their data assets.

There could many. For example, dirty data, like incorrect customer addresses in a database. Duplicates, such as repeated entries in a patient's medical records. Inconsistent data formats. Irrelevant data, like including a person's favorite color in a financial risk assessment model. Incomplete data, such as missing income details in loan applications. Adherence to data standards is critical; for instance, neglecting GDPR compliance in data handling can result in legal issues. Misinterpreting seasonal sales data as a sign of market demand. There could be many more. A deep understanding of data, including its limitations and context, is critical.