Larry Ladd’s Post

With budgets being cut and cost savings top of many organisations priority list. Why bother investing in Data Quality? The truth is that Data Quality issues are increasing in most organisations and, on average, take up 34% of a Data Engineers time, preventing them from creating any kind of value. This increases to 50% of the time in large Data Platforms. All these Data Quality issues add up, with bad Data Quality costing organisations, on average $15 million a year. Data Quality investment is also an investment in high-quality AI and ML, as you’ll get more accurate AI and ML results from improving Data Quality than changing your AI model and code. In our latest blog, Jake Watson looks at the reasons you should invest in Data Quality, together with the latest trends and technology. #data #dataquality #datareliability #datagovernance #datacontracts #dataobservability #datablog https://lnkd.in/eD7jmZdw

The Dirty Little Secret about current data architectures is that they all lack data integrity between datasets. While each dataset is typically designed with internal referential integrity, each dataset lacks any enforced data integrity between them. Because of this lack of data integrity, tedious and expensive data management projects, such as data integration, are used as a Band-Aid to the problem. We have found that when data integrity is enforced between datasets, the datasets become compatible and directly interoperable. All your data from all your compatible datasets is now seamlessly accessible on demand. Your compatible datasets spontaneously form a data fabric capable of readily supporting advanced technologies such as AI/ML. #Data #DataIntegration #DataCompatibility #AI https://lnkd.in/eh4YKT9p

🤔 Interesting Read 🚀 Re-imagining Data Stewardship in the Age of Rapid Datafication and AI ➡️ As to navigate the ever-evolving digital landscape, Courtney Cameron’s article, “Data Stewardship: The Way Forward in the New Digital Data Landscape,” highlights the critical need to rethink how we manage data. This resonates deeply with our efforts to ensure systematic, sustainable, and responsible data access for reuse by reimagining data stewardship. 👉 Cameron calls for a re-imagination of data stewardship frameworks to keep pace with rapid digitalization, and growing data asymmetries. In particular, I made the following takeaways: 🔄 Integrated Decision-Making: Consulting data stewards in decisions (including decisions regarding AI) to ensure data is used in the public interest; 🏢 System Integration: Embedding data stewardship across the data and AI lifecycle and into systems development methodologies (leveraging Gather/Guard/Grow/Give principles). 🌐 Public Engagement: Maintaining public trust, and obtaining the social license for data re-use, through transparent processes. ✅ By fostering data stewardship, we can build robust data ecosystems that drive innovation, support policy-making, and maintain public trust. 💻 Read Courtney Cameron's article at Centre for International Governance Innovation (CIGI): https://lnkd.in/e3JcSYAj ➡️ Data Stewardship Re-Imagined — Capacities and Competencies: https://lnkd.in/ewPXMA5U #DataStewardship #DigitalTransformation #AI #DataGovernance #Innovation #PublicPolicy #Data #Artificialintelligence

🚀 Leveraging RAG-Based LLMs in 2024: A New Era for ESG and Data Intelligence 🚀 In "The Rise of RAG-Based LLMs in 2024" by Kyle Kirwan, the spotlight is on the emerging trend of Retrieval-Augmented Generation (RAG) within Large Language Models (LLMs). These models are shaping up to be a game-changer in addressing challenges like model hallucinations and training limitations. Key Insights: 1. Overcoming Data Challenges: RAG models provide accurate, up-to-date external information, enhancing data reliability crucial for informed decision-making. 2. Sophisticated Data Linking: Expertise is housed outside the model in databases or knowledge graphs, creating an advanced connection between data stores and users. 3. Data Observability: With the increased reliance on RAG models, robust auditing processes are essential to ensure accuracy. Applications in ESG and Internal Data: 1. Sustainable Investment Analysis: Clients can leverage RAG-based LLMs to analyze vast datasets on sustainability metrics, aiding in identifying environmentally and socially responsible investment opportunities. 2. ESG Reporting: Enhance ESG reporting by integrating RAG models to pull the latest regulatory updates and benchmarks, ensuring accurate, current data in reports. 3. Risk Management: Utilize RAG-based LLMs to monitor and analyze internal data alongside external ESG risk factors, providing a comprehensive risk profile for strategic decision-making. 4. Supply Chain Transparency: Clients can use these models to scrutinize their supply chain, ensuring compliance with environmental and social standards by accessing the latest data on suppliers. 5. Stakeholder Engagement: Improve communication with stakeholders by providing data-driven insights on ESG performance, using RAG models to gather and interpret relevant data. As we integrate these advancements into our ESG open-source intelligence engine (EDIE), it becomes evident that RAG-based LLMs are a critical evolutionary step in AI. They are set to revolutionize data processing, analysis, and decision-making in various sectors. Let's harness the power of RAG models to unlock new possibilities in ESG and data intelligence! Reach out to us to find out more about how Primacy can help your ESG journey. You can read the full article here: https://lnkd.in/gY793QaA #AI #ESG #DataIntelligence #RAGmodels #LLMs #InnovationInESG Alaina Franklin Jirat Boomuang Andrew Tucker, Ph.D. Anthony Dixon Mettle Capital Wyser Primacy Technology

“Mistakes in data governance that affect the working of such (AI) systems can have profound legal, financial and social implications on the organizations involved, citizens and businesses, and society at large. Such mistakes can result in systemic bias, unlawful decisions, large financial exposures, political crises, lives lost or any combination thereof. In the interconnected world, where data is collected by (and about) governments, businesses and citizens, and is processed by different entities using various algorithms, dependencies grow, mistakes accumulate, and accountability is gradually lost in the process. [30]” [30] Janssen, Marijn, et al. "Data governance: Organizing data for trustworthy Artificial Intelligence." Government Information Quarterly 37.3 (2020): 101493. https://lnkd.in/e4D_-3aX

Came across a paper that does a nice job of communicating data governance, which may help understand why LLMs are such high-risk, and why the precision data management system is the core of our KOS. AI is data-dependent. Without precision high quality data, it's algorithmic chaos. The underlying commonality causing the risks and cost in socioeconomics, environment, safety, and security in LLMs is the lack of robust data governance in the data sets LLMs are trained on. The majority of data sets they train on simply lack sufficient data structure and provenance necessary to mitigate significant risks and costs to customers and society, including catastrophic risks. It's also conveniently very difficult to nearly impossible to determine ownership without provenance (automatically). --> Data Governance: “Organizations and their personnel defining, applying and monitoring the patterns of rules and authorities for directing the proper functioning of, and ensuring the accountability for, the entire life-cycle of data and algorithms within and across organizations.” “Mistakes in data governance that affect the working of such systems can have profound legal, financial and social implications on the organizations involved, citizens and businesses, and society at large. Such mistakes can result in systemic bias, unlawful decisions, large financial exposures, political crises, lives lost or any combination thereof. In the interconnected world, where data is collected by (and about) governments, businesses and citizens, and is processed by different entities using various algorithms, dependencies grow, mistakes accumulate, and accountability is gradually lost in the process [30].” [30] Janssen, Marijn, et al. "Data governance: Organizing data for trustworthy Artificial Intelligence." Government Information Quarterly 37.3 (2020): 101493. https://lnkd.in/eGTG8z96

I was going through some old posts I did from back in 2019. You know, way before AI or LLM were even acronyms that were actionable or had serious meaning. Back then, I was talking about 'data.' Having it, managing it, controlling it, stewardship, loving it! Today, I read more 'up-to-date' articles on AI and LLM and about the need for 'data management,' or 'a CDO.' So, what has happened in the past 4 years to the concept of data ownership since I wrote that original article in 2019? It would appear 'not much' as we are now having the same discussions today only now it is about the hottest tech that we have all seen in a very long time. 2024 the year?

Unstructured data represents 80-90% of the majority of data by multiple estimates. However, Deloitte found only 18% of organizations are able to take advantage of that data. It’s a huge opportunity for organizations, especially those with real-time data. Recent improvements in AI/ML will drive monetization efforts in the unstructured data. Data integrity is required for monetization, placing pressure on data governance, hygiene, and protection initiatives. Keep in mind growth is 50% a year, doubling every 2 years, sprawling across multiple repositories. Any management problems will compound with linearity.

I've been quiet for a while on LinkedIn, we've been delivering a major project. One of the powers of DeepMiner is that we take large amounts of raw unstructured data and make it usable, helping deliver real insights into what that data contains. Data Governance is an important, and all too often, an ignored part of the data journey. With our customers, we help them move along this Governance journey, delivering solutions and navigating Data Governance issues through insights and practical suggestions. This great article goes some way to explaining its importance: https://lnkd.in/eZMcbJ6H #deepminer #ai #GovTech #dataunification #governement

Can semantic graph technology truly solve the data management problem, or is it just another point solution? Simply deploying the technology won't magically banish data silos. The real challenge for businesses is building a strong data foundation with clear ownership, access controls, and consistent data structures. This is where the the unglamorous but essential work of cleaning, organising, and connecting data before reaping the benefits of AI. Will businesses prioritise this data housekeeping, or will siloed data continue to be the Achilles' heel of AI projects? #DataManagement #AIandData #FutureofBusiness https://lnkd.in/gtS_uUmu