Szehon Ho

Cloud agnostic and built for the customer’s preferred language, Dataiku gives Doug Liming, Sr. Data and Analytics Engineer at Corios, collaboration and validation in one place. It makes moving SaaS source models into open source easier, keeps documentation within each project, and tracks who did what, when, delivering complete insight from an audit perspective. Learn how Corios and Dataiku help enterprises migrate from SAS: https://lnkd.in/ej-ZgGQ7 #EnterpriseAI #AI

40

This article by Mukul Singh caught my attention because it challenges an assumption I see more lately ("MCP Connecting agents to tools is enough"). It also addresses the new Tokenomics architectural constraints we all need to master. MCP is important. It standardizes how agents reach tools, data, and systems... That's transport. But production agentic systems need more than connectivity. They need a structured way to express business intent, manage context, enforce policy, and execute safely across multi-step workflows... That's orchestration layered over transport. That is where Code DSLs can become interesting layered over MCP transport. Agents are easy. Agentic systems are not. P.S. I find it fascinating that we may use Agentic AI to abstract away traditional coding and end up creating more languages (DSLs) so agentic solutions can reason, orchestrate, and operate safely and efficiently. https://lnkd.in/ePAxmGhN

18

1 Comment

Thinking about Apache Iceberg™? Already on Databricks or planning to migrate? Our CTO Eddie Edgeworth, explores three paths to success with Apache Iceberg™ on Databricks, and why open formats paired with open platforms are the future. Read the full blog: https://lnkd.in/gAU7s8em #DataGovernance #Databricks #DataLakehouse #DataInnovation #Koantek #UnityCatalog #DataTransformation #AIInnovation

34

1 Comment

Databricks engineers have transformed database management by moving from reactive monitoring to proactive scoring, driving efficiency, reliability, and scalability across thousands of databases. Key innovations: 🔹 Client-Side Query Tracing: Connect queries to tenants, features, and APIs for rapid incident triage 🔹 Query & Schema Scorer in CI: Flags anti-patterns before deployment, catching inefficient queries and risky schema changes early 🔹 Database Usage Scorecard: Unified AI/BI dashboards consolidate metrics across CI and production, giving a 360° view of queries, schemas, traffic, and resource use Read on for more: https://lnkd.in/e8MPBqYs #Databricks #DatabaseReliability #DataEngineering #AI

1

Building with LLMs isn’t just about flashy demos, it’s about real applications that work. Hamel Husain walks us through fine-tuning, model evaluation, and optimization strategies to get your AI projects from proof-of-concept to production. Learn the tools, pitfalls, and techniques developers actually use. Listen here: https://lnkd.in/eCP9yn8a #LLM #AIEngineering #MachineLearning

1

Last week, dbt Labs launched the public beta of dbt Fusion, a Rust based engine rewrite that brings a series of neat new features to our favorite data transformation tool: 🛠️ Real-time SQL validation and autocompletion 👀 Live previews of CTEs and compiled code 🔍 Rich model lineage at the colomn and table levels ⚡️ 30x faster project parse time (thanks Rust 🦀) 💻 Deep VS Code / Cursor integration At Opsima, dbt is a key component of our data stack (along with Prefect and Snowflake). dbt powers all data transformations across our cost and usage models and commitment automation engine. And being Cursor power users, the idea of a faster, smarter model development workflow within our IDE is quite exciting! That said, the reaction hasn’t been all positive. Reddit’s on fire with concerns about what Fusion means for the future of dbt Core and open source. https://lnkd.in/e9WSFARf dbt Labs insists Core stays under the Apache 2.0 license and fully supported, but Fusion is only “source-available”, not entirely open source, and licensed to limit hosting it as a service which raises some concerns. https://lnkd.in/ekVzgxcW For teams like ours though, we’re confident Fusion will help us ship faster and smarter while keeping core flows portable. What is your own take on the launch? #dbt #dbtFusion #FinOps #DataEngineering #OpenSource

13

Join the Databricks Data and AI Experts: Steve Sobel --Global Leader, Startup & Venture Programs; Kevin Rasmussen -- Field CTO, Early Stage Programs, and Sriharsha Tikkireddy -- Technical Lead, Early Stage Programs, for the next Omaxn event: Workshop: #VibeCoding Unlocked: Effortless App Building with Databricks Details" TechNexus Venture Collaborative on Aug 26 What you'll learn: #FasterDevelopment and #Iteration: Databrick's "#Vibecoding" allows developers to use natural language prompts to generate code and build applications, cutting the time from idea to deployment from weeks to minutes or hours. Engineers can test, refine, and redeploy applications in a highly efficient, conversational way—enabling tight feedback loops. #FullCustomization and #CodeOwnership: Engineers get complete control over the generated code instead of being limited by drag-and-drop low-code templates. You can modify, extend, and host the application anywhere, ensuring flexibility for enterprise needs and long-term maintainability. #PatternExtraction and #Maintainable Architecture: The AI focuses on extracting repetitive code patterns and parameterizing them, producing clean, documented configurations. This ensures that applications are not only quick to build, but also easy for future engineers to understand and maintain. #Integration with #DatabricksInfrastructure: Applications inherit benefits from Databricks’ robust governance and security: access controls, audit trails, RBAC, and deployment on the trusted Lakehouse platform. #TurningIdeas into #Apps: Fast: Vibe coding enables business stakeholders, analysts, and subject matter experts to describe their needs in plain English (or any natural language). The system translates these ideas into working applications, so non-techies can rapidly bring their insights to life—no computer science degree required.

10

1 Comment

Your data team enriches IP addresses by exporting them from Snowflake, running them through an external API, parsing the JSON response, and importing the results back. For every batch. Every time. That workflow just became unnecessary. IPGeolocation.io is now on the Snowflake Marketplace. Six IP intelligence databases shared directly into your Snowflake account. No ETL. No API calls. No data leaving your environment. Install a listing and the data appears immediately. Each database includes a built-in lookup function. Pass any IPv4 or IPv6 address, get the full record back. Join it against any table with an IP column for batch enrichment. 100 rows or 100 million. Same SQL workflow. What is available: 1. IP to Location - city, country, ZIP, coordinates, timezone, currency, language, connection type, and 20+ fields per record 2. IP Security - VPN, proxy, Tor, bot, attacker detection with threat score (0-100), provider names, confidence levels, and last-seen dates 3. IP to Company - 30.6 million records mapping IPs to company name, type (business, ISP, hosting), and domain 4. IP to ASN - every IP mapped to its ASN, owning organization, domain, country, and network type 5. IP Abuse Contact - name, email, phone, and address of the abuse team responsible for any IP range 6. Residential Proxy - IPs on residential proxy networks with specific provider name and last-seen date Five workflows teams are running with this: 1. Fraud detection - join security data against transaction logs. Flag high-risk connections using threat scores and provider names. Layer residential proxy data to catch traffic routed through real home connections. 2. Audience segmentation - enrich analytics with city-level geolocation, timezone, currency, and DMA data. Results piped into BI dashboards without leaving Snowflake. 3. Incident response - abuse contact data returns the responsible team's name, email, and phone for every offending IP. ASN adds network ownership. Entire incident log enriched in one query. 4. B2B visitor intelligence - IP to Company turns anonymous traffic into named business visitors. Company type filters out ISP and hosting so only real accounts surface. 5. Geo-fencing - filter by country for licensing, GDPR, or pricing rules. Combine with security data to catch anonymizers bypassing restrictions. Free trial on all six. No credit card. Updates automatic via Snowflake's secure data sharing. #Snowflake #DataEngineering #IPIntelligence #CyberSecurity #Analytics

7

1 Comment

Autonomous AI is now running your data pipelines. This week’s standout: **Agentic AI is transforming data engineering workflows, automating everything from schema evolution to pipeline optimization**. Major platforms like AWS and Azure, plus lakehouse leaders like Databricks, are deploying these capabilities—and the impact is real. AWS cites a 45% drop in pipeline maintenance overhead, while Databricks users are launching pipelines 60% faster thanks to GenAI-powered assistants[1][3]. Why does this matter? For data engineers and ML teams, it means less firefighting and more time focused on high-value analytics. Onboarding new data sources, handling schema drift, and maintaining data reliability are being handled autonomously, accelerating innovation and keeping tech debt in check[1][3]. **Takeaway:** Now is the time to explore integrating agentic AI tools into your stack—they’re not just a trend, but a new baseline for data ops. How will this shift affect your team’s priorities—and what would you automate first? #DataEngineering #AIinData #Databricks #MLOps #CloudData

2

🦈📡 Connecting Megaladata with Apache Kafka: Real-time data streaming with purpose. In a world where data moves fast, legacy batch pipelines fall behind. That’s where Apache Kafka comes in. As a distributed event streaming platform, Kafka acts as the "central nervous system" for your data: ingesting, buffering, and distributing continuous streams of events from producers to consumers without delay. The Shift to Real-Time In practical terms, the Kafka integration allows organizations to: ✨Instantly Capture: Ingest live transactions, sensor logs, and user events the moment they occur. ✨Decouple Systems: Allow producers and consumers to scale independently, removing architectural bottlenecks. ✨Slash Latency: Move from "yesterday’s reports" to "this second’s insights." The Megaladata + Kafka Advantage: By integrating Megaladata with Kafka, you bridge the gap between high-speed data transport and sophisticated analytics. Megaladata acts as both a consumer (reading streams for analysis) and a producer (sending transformed data back to Kafka topics). Why this integration changes the game: 🚀 Eliminate "Batch Lag": Megaladata ingests data from Kafka topics as it is generated, replacing stale nightly uploads with a live, consistent data flow. 📊 Low-code stream processing: Skip the complex Java/Scala boilerplate. Use Megaladata’s low-code interface to apply complex transformations and deliver up-to-the-minute dashboards. 🤝 Flexible, unified pipelines: Connect multiple data sources without custom coding. Kafka serves as the backbone, and Megaladata serves as your intelligent analytics engine. 🔁 Resilience at scale: Use Kafka’s industry-leading fault tolerance alongside Megaladata’s orchestration to ensure your pipelines scale with your growth. Turning Events into Action This integration is the gold standard for use cases where every second counts: fraud detection, IoT telemetry, fleet tracking, and financial transaction processing. Megaladata + Kafka turns raw event streams into actionable insight, closing the gap between observation and decision. If your business depends on responsiveness, streaming is no longer optional—it’s the new baseline. 🔥 🧠 Ready to bring streaming data into your analytics workflows? Let’s talk about how Megaladata + Kafka can transform your real-time data strategy. #Megaladata #ApacheKafka #StreamingAnalytics #DataEngineering #RealTime #DataPipelines #LowCode #Automation

2

The Data Management team at an asset management firm faced a time-consuming, error-prone process with 30-40 manual steps to integrate Snowflake pipelines with AWS. Using Terraform, we automated the setup with reusable code, saving time, minimizing errors, and creating a more efficient, maintainable workflow.

2

1 Comment

We just migrated our default license from MIT to The Apache Software Foundation Apache 2.0 across the Omni ecosystem. Why Apache 2.0? - Explicit patent grant protects contributors and users - Recommended default by Cloud Native Computing Foundation (CNCF) for cloud-native projects - Trusted by the foundations behind Kubernetes (Official), The Rust Programming Language, TypeScript, and Android Nothing changes for how you use our software. Apache 2.0 is equally permissive, while offering stronger legal protections for everyone involved. If you're building on Omni, you're protected. If you're using Omni, you're protected. No ambiguity. https://lnkd.in/ee-QeQic

2

AI is moving fast. Most data architectures… aren’t. If your AI strategy still relies on batch data, you’re already behind. In this webinar, Confluent Field CTO Adam Tybor joins CRN to break down why data streaming is the foundation modern AI actually needs and how to become truly data ready. Watch now -> https://cnfl.io/3NSFaV5

16

1 Comment