Amazon Redshift RG instances powered by AWS Graviton

"Started exploring Snowflake today — a cloud data warehouse that's widely used in modern data engineering! Key thing I learned — unlike traditional databases, Snowflake separates storage and compute. This means teams can query data independently without slowing each other down! #DataEngineering #Snowflake #LearningInPublic #CloudData"

Most enterprise data platform builds start with the vendor question. AWS or Azure? Snowflake or Databricks? The cloud and the engine are variables. The architecture is the constant. Bronze. Silver. Gold. Source data preserved as-is. Cleaned and conformed across systems. Curated and ready for serving. Runs on any cloud, with any engine, swappable layer by layer. 𝗪𝗼𝗿𝗸𝘀𝗵𝗼𝗽. Inventory the top 3 use cases. Pick one. 𝗣𝗿𝗼𝘃𝗲. Land Bronze. Promote one Gold use case end to end. 𝗦𝗰𝗮𝗹𝗲. Decide the compute engine when scope is real, not before. 90 days to first Gold. Not 12 months to a half-finished warehouse. The architecture stays the same. The platform choices are the variables. None of these decisions are one-way doors. 👉 If your team is six months into a data platform build with no shipped use case, the constraint is usually less about budget and more about sequence. 🔗 More on how we approach it → https://lnkd.in/eX7m8jHP #EnterpriseData #DataEngineering #DataFoundation #SmartData

Most data platform conversations start in the wrong place. This is a better way to sequence it. #Snowflake #Databricks #enterprisedataplatforms #SmartData

A great way to learn about the end to end AI/ML capabilities in BigQuery. A codelab to go along - so that you can build it yourself! It was a great time staffing this booth along with Nivedita Kumari, Larry Henderson & Jithin S L Thank you Alicia Williams & Jeff Nelson for a great demo!

BigQuery does more than query structured tables. Alicia Williams and I show how to use it for raw multimodal data, vector search, and predictive pricing on the show floor at Next 2026. The full step-by-step Codelab (with credits to try at no-cost) in the comments. Shout out to Jithin S L, Tarak Parekh, Larry Henderson, Nivedita Kumari for their work representing this!

Our AWS bill jumped overnight even though nothing new was deployed. Same pipelines, same dashboards, same schedules. Everything looked normal on the surface, which made the cost spike even more confusing. When we dug into it, the issue came down to one Athena query scanning way more data than expected. Nothing changed in the query logic itself, but a partition filter wasn’t being applied properly. That small miss meant the query started reading the entire S3 dataset instead of just the required slice. The fix was simple—correct the partition usage and make sure filters were actually getting pushed down. Once that was done, the cost dropped back to normal. It was a good reminder that cloud cost issues don’t always come from big architectural problems; sometimes it’s just one small thing that quietly grows into a big bill. What’s the smallest issue you’ve seen turn into a surprisingly large cloud cost? #CloudComputing #AWS #DataEngineering #FinOps #SQL #BigData

Manifest files really help especially when migrating data across cloud environments. Recently, I leveraged this approach in a pipeline I built. Instead of directly copying datasets, I generated a manifest file from the source cloud containing metadata and file references. This manifest was then transferred to the target cloud and used to seamlessly register the data in Databricks. This approach changed a few things for me: • No need for heavy data copying or duplication • Faster data availability in the target system • Clear separation between data movement and metadata handling • Better scalability for large datasets In short, the manifest acted as a bridge making the migration lightweight, efficient, and reliable. Curious are others using manifest driven approaches for cloud migrations, or still relying on traditional copy methods? #DataEngineering #CloudMigration #BigData #DataPipelines #ETL #ELT #Databricks #ApacheSpark #DataArchitecture #DataEngineeringLife #DataOps #CloudComputing #DataPlatform #AnalyticsEngineering #ModernDataStack #DataLake #DataLakehouse #DeltaLake #Metadata #DataManagement #ScalableSystems #DistributedSystems #CloudData #DataIntegration #PipelineEngineering #TechInnovation #AIEngineering #MachineLearning #DataScience #EngineeringLife #Automation #DataStrategy #DataInfrastructure #CloudArchitecture #DataGovernance #TechCommunity #LinkedInTech #LearnInPublic #BuildInPublic #TechPost

If your team is dealing with fragmented data environments, runaway BigQuery costs, or pipelines that underperform at scale, this session is worth your time. On May 29, Jellyfish Training (Google's North American Cloud Training Partner of the Year) is running a full-day virtual session: 𝗢𝗽𝘁𝗶𝗺𝗶𝘇𝗶𝗻𝗴 𝗔𝗴𝗲𝗻𝘁𝗶𝗰 𝗗𝗮𝘁𝗮 𝗪𝗼𝗿𝗸𝗹𝗼𝗮𝗱𝘀 𝗼𝗻 Google Cloud. Built for data architects and engineers, the day covers Knowledge Catalog (formerly Dataplex) and Data Mesh architecture, BigQuery workload management and pricing, Dataflow and batch pipeline tuning, and a FinOps module on budgets and alerting. Two hands-on labs are built into the schedule. Participants who complete the training earn a digital Credly "Enterprise Data Efficiency" badge. 📅 May 29, 2026 | 9:00 AM – 5:00 PM CDT | Virtual Participation is limited to keep the experience hands-on and high-impact. 🔗 Register here ➡ https://lnkd.in/e_H6UAJu #GoogleCloud #DataEngineering

We cut cloud costs by 80%!! 💸📉 The surprising part? The biggest saving wasn't from better code. 🙅♀️ It was from reading the bill properly. 📑🔍 We found: → Clusters running all night with no jobs on them. 🌙💤 → Data stored in CSV (on a distributed system, in 2022!). 📁🚫 → Jobs running one by one that could run in parallel. 🐢➡️🐎 Migrated to Databricks . Converted to Delta Lake. Fixed the cluster config. 🛠️✨ The Result: ✅ 80% cost reduction. ✅ 85% faster processing. Sometimes the best engineering is just paying attention. 🧠💡 What's the most surprising inefficiency you've found in a data platform? 👇 #Databricks #DataEngineering #CloudCost #BigData #ApacheSpark #DeltaLake #CloudOptimization #TorontoTech #TorontoDataEngineer