Identifying Trends and Patterns in Data

A lot of us still rely on simple trend lines or linear regression when analyzing how user behavior changes over time. But in recent years, the tools available to us have evolved significantly. For behavioral and UX data - especially when it's noisy, nonlinear, or limited - there are now better methods to uncover meaningful patterns. Machine learning models like LSTMs can be incredibly useful when you’re trying to understand patterns that unfold across time. They’re good at picking up both short-term shifts and long-term dependencies, like how early frustration might affect engagement later in a session. If you want to go further, newer models that combine graph structures with time series - like graph-based recurrent networks - help make sense of how different behaviors influence each other. Transformers, originally built for language processing, are also being used to model behavior over time. They’re especially effective when user interactions don’t follow a neat, regular rhythm. What’s interesting about transformers is their ability to highlight which time windows matter most, which makes them easier to interpret in UX research. Not every trend is smooth or gradual. Sometimes we’re more interested in when something changes - like a sudden drop in satisfaction after a feature rollout. This is where change point detection comes in. Methods like Bayesian Online Change Point Detection or PELT can find those key turning points, even in noisy data or with few observations. When trends don’t follow a straight line, generalized additive models (GAMs) can help. Instead of fitting one global line, they let you capture smooth curves and more realistic patterns. For example, users might improve quickly at first but plateau later - GAMs are built to capture that shape. If you’re tracking behavior across time and across users or teams, mixed-effects models come into play. These models account for repeated measures or nested structures in your data, like individual users within groups or cohorts. The Bayesian versions are especially helpful when your dataset is small or uneven, which happens often in UX research. Some researchers go a step further by treating behavior over time as continuous functions. This lets you compare entire curves rather than just time points. Others use matrix factorization methods that simplify high-dimensional behavioral data - like attention logs or biometric signals - into just a few evolving patterns. Understanding not just what changed, but why, is becoming more feasible too. Techniques like Gaussian graphical models and dynamic Bayesian networks are now used to map how one behavior might influence another over time, offering deeper insights than simple correlations. And for those working with small samples, new Bayesian approaches are built exactly for that. Some use filtering to maintain accuracy with limited data, and ensemble models are proving useful for increasing robustness when datasets are sparse or messy.

In a sea of possible insights, how do you know which are worth reporting? As a data analyst, there are two types of insights you will report: 1) Ones that are directly aligned to a business question or priority 2) Ones that nobody is asking for… but should be 90% of the time, you should be focusing on the first one. But when done right, the second can be very powerful. So… how do you find those hidden insights? How do you know which ones truly matter? ➤ Explore high-level trends Scan dashboards, reports, or raw data for unexpected patterns. Look for sudden spikes, dips, or emerging trends that don’t have an obvious explanation. ➤ Slice the data by different dimensions Break data down by different categories (customer segments, time periods, product lines, etc.). Where are things changing the most? Which groups are behaving unlike the others? ➤  Identify outliers Look at the extremes. What’s happening with your best customers? Worst-performing regions? Most productive employees? Outliers often reveal inefficiencies or hidden opportunities. ➤ Tie insights to business impact Before reporting, ask: Would knowing this change a decision? If it doesn’t, it’s probably not worth surfacing. ➤ Pressure-test with stakeholders Run your findings by a manager or friendly stakeholder. Ask them if the finding resonates with other trends they've seen, whether they see potential value, and whether it could influence strategy. In other words: - Start broad - Dig deep - Sense-check —-— 👋🏼 I’m Morgan. I share my favorite data viz and data storytelling tips to help other analysts (and academics) better communicate their work.

What is Time Series Decomposition—and Why Does It Matter in Quantitative Finance? In Quantitative Finance, we rarely model raw data directly. Instead, we decompose it into components that help us understand what’s driving market behaviour. These four components—Trend, Seasonality, Cyclical Movement, and Irregular Fluctuations—form the backbone of time-based financial modelling. Let’s break them down and see why they matter in the real world. 1. Trend: The Long-Term Direction Trend refers to the sustained upward or downward movement in data over time. In finance, this could be structural economic growth, persistent inflation, or long-run shifts in interest rates. → Portfolio managers use trend models to calibrate expected returns → Risk teams align stress scenarios with long-term market drift → Trend filtering helps isolate genuine alpha signals from temporary noise Without accounting for trend, any model risks misattributing long-term movement as short-term volatility. 2. Seasonality: Recurring Patterns Within the Year Seasonality is about predictable, time-bound repetition—think quarter-end flows, earnings cycles, or holiday-driven consumer spending. → Seasonal volatility impacts options pricing ahead of earnings or economic releases → In fixed income, coupon schedules affect reinvestment flows → Adjusting for seasonality improves forecast accuracy and reduces overfitting Seasonal effects aren’t noise—they’re structured and repeatable. Ignoring them can skew your model. 3. Cyclical Movements: Economic Ups and Downs Cyclicality captures non-fixed, but systematic swings tied to broader economic conditions—interest rate cycles, credit expansions, inflation regimes. → Asset allocation shifts as macro cycles unfold → Risk exposure changes as we move through different volatility regimes → Cyclical adjustments help dynamic models adapt to economic shifts Unlike seasonality, cycles are not tied to a calendar—they evolve with the market itself. 4. Irregular Fluctuations: The Unexpected Residual These are the outliers—the black swans, sudden news events, and random noise. → Irregular spikes must be managed, not modelled → Scenario design and tail-risk management rely on recognising what cannot be predicted → Robust models separate structural effects from residual shocks No matter how advanced the model, separating noise from pattern is the hallmark of clean forecasting. So Why Does All This Matter in Quant Finance? Because time series isn’t just a chart—it’s the story of how financial data evolves. By decomposing it, we move from raw data to insight, from chaos to structure, and from noise to signal. This decomposition powers everything from volatility modelling to stress testing, yield curve simulations, asset pricing, and beyond. #QuantFinance #TimeSeriesAnalysis #FinancialModelling #StochasticProcesses #RiskManagement #SignalExtraction #FinancialEngineering #QuantitativeFinance

Ever tried finding patterns like 𝗵𝗲𝗮𝗱 𝗮𝗻𝗱 𝘀𝗵𝗼𝘂𝗹𝗱𝗲𝗿𝘀 or 𝗱𝗼𝘂𝗯𝗹𝗲 𝗯𝗼𝘁𝘁𝗼𝗺𝘀 in millions of data points? It’s slow, tedious, and often doesn't scale well with traditional methods. That’s the exact problem I tackled with T𝗲𝗺𝗽𝗼𝗿𝗮𝗹 𝗦𝗶𝗺𝗶𝗹𝗮𝗿𝗶𝘁𝘆 𝗦𝗲𝗮𝗿𝗰𝗵 (𝗧𝗦𝗦), a direct pattern matching approach that scales to millions of time-series data points—fast. I ran a test on 10 𝗺𝗶𝗹𝗹𝗶𝗼𝗻 𝘀𝘆𝗻𝘁𝗵𝗲𝘁𝗶𝗰 𝗺𝗮𝗿𝗸𝗲𝘁 𝗱𝗮𝘁𝗮 𝗽𝗼𝗶𝗻𝘁𝘀 and identified classic patterns like 𝗰𝘂𝗽 𝗮𝗻𝗱 𝗵𝗮𝗻𝗱𝗹𝗲 in well under a second. No heavy feature engineering, no ML models—just direct comparison between time-series vectors. This method saves hours of manual work and speeds up everything from backtesting to real-time signal detection. I was able to detect any synthetic pattern I wanted, no matter how complex, simply by defining an example. Here’s what stood out: • 𝗠𝗮𝘀𝘀𝗶𝘃𝗲 𝘀𝗰𝗮𝗹𝗮𝗯𝗶𝗹𝗶𝘁𝘆: TSS processes millions of data points without bottlenecks, ideal for large datasets and real-time market analysis. • 𝗖𝘂𝘀𝘁𝗼𝗺 𝗽𝗮𝘁𝘁𝗲𝗿𝗻 𝗺𝗮𝘁𝗰𝗵𝗶𝗻𝗴: You can define and search for any pattern—traditional or custom—across huge datasets. • 𝗜𝗺𝗺𝗲𝗱𝗶𝗮𝘁𝗲 𝘀𝗶𝗴𝗻𝗮𝗹 𝗱𝗲𝘁𝗲𝗰𝘁𝗶𝗼𝗻: Use it in live trading environments to spot emerging patterns instantly, without the lag of machine learning pipelines. Curious about the implementation or how it fits into your workflow? Check out the link to my article on using TSS for technical analysis in the comments!

Most businesses panic when they see their average order value (AOV) drop 25%. They then… - Slash prices - Rush promotions - Question their premium products But smart retailers know better — they investigate patterns first. Here are a few to get you started: 1. Sales data Your 6-month trends reveal the first signs of change: - Did price changes affect order value? - Which products are selling more or less? - What's the pattern in shopping cart composition? - What does purchase frequency tell us? - What's hiding in abandoned carts? - Are premium products getting abandoned? 🧩 Let’s say you see premium items getting abandoned at checkout repeatedly. Looking deeper, you might find a specific price threshold — leading to an opportunity for strategic bundling. 2. Website behavior Tools like CrazyEgg, LuckyOrange, Hotjar, and FullStory show complete interaction patterns: - Most visited pages - Heat map patterns - Premium product engagement 🧩 Are customers spending time on review sections but leaving? You might need stronger social proof and not necessarily lower prices. 3. Customer voices Data tells half the story, and your customers tell the other half. Direct fact-finding reveals… - Customer sentiments on new premium products - Views on popular vs. unpopular items - Feedback on existing products Social media conversations add another layer of insight. 🧩 Suppose your focus groups reveal confusion about premium features. This could signal you need better education — not different products. 4. Competitive landscape A comprehensive look at your market reveals if competitors… - Launched promotions that coincided with the change - Introduced new products during your AOV drop - Brought innovative solutions to the market - Lowered their existing product prices 🧩 Did you notice your AOV drop right when a competitor introduced similar products at lower prices? This is a direct connection between market changes and your sales patterns. 5. Long-term trends Customer surveys help you identify shifts in popularity before they hurt your bottom line. 🧩 If they show customers gradually losing interest in a once-popular product category… You’ve spotted a trend that explains your dropping order value (and suggests you should act accordingly). 💡 Remember this: Numbers don't drop without reason. Patterns don't form by accident. Solutions don't come from guessing. Understanding your customers' behavior is the difference between reacting and leading.

I've analyzed 100+ datasets in the last 3 years. Every single time I skipped proper EDA, my model failed in production. EDA isn't just "making charts." It's detective work. It's interrogating your data before you trust it. 𝐖𝐡𝐚𝐭 𝐦𝐨𝐬𝐭 𝐩𝐞𝐨𝐩𝐥𝐞 𝐭𝐡𝐢𝐧𝐤 𝐄𝐃𝐀 𝐢𝐬: → Plot a few histograms → Check for nulls → Move to modeling That's not EDA. That's checking boxes. 𝐖𝐡𝐚𝐭 𝐄𝐃𝐀 𝐚𝐜𝐭𝐮𝐚𝐥𝐥𝐲 𝐢𝐬: It's asking your data questions it doesn't want to answer. → Why is this column 40% null? → Why does revenue spike every third Tuesday? → Why are these two variables perfectly correlated? → Why does this outlier exist? EDA is where you find the story behind the numbers. 𝐓𝐡𝐞 𝟒 𝐨𝐛𝐣𝐞𝐜𝐭𝐢𝐯𝐞𝐬 𝐨𝐟 𝐄𝐃𝐀: 𝟏. 𝐒𝐩𝐨𝐭 𝐀𝐧𝐨𝐦𝐚𝐥𝐢𝐞𝐬 Find outliers, errors, data entry mistakes. That $10M transaction? Data entry error. The 200-year-old customer? System bug. 𝟐. 𝐃𝐢𝐬𝐜𝐨𝐯𝐞𝐫 𝐏𝐚𝐭𝐭𝐞𝐫𝐧𝐬 See trends, seasonality, correlations. Sales drop every December? Holiday season. Churn spikes after 90 days? Onboarding problem. 𝟑. 𝐓𝐞𝐬𝐭 𝐀𝐬𝐬𝐮𝐦𝐩𝐭𝐢𝐨𝐧𝐬 Verify what you think you know. "Age predicts spending"—does it? Check. "Gender doesn't matter"—sure about that? Verify. 𝟒. 𝐏𝐫𝐞𝐩𝐚𝐫𝐞 𝐟𝐨𝐫 𝐌𝐨𝐝𝐞𝐥𝐢𝐧𝐠 Select features, handle nulls, encode categories, scale data. EDA tells you what transformations you need. 𝐓𝐡𝐞 𝐄𝐃𝐀 𝐰𝐨𝐫𝐤𝐟𝐥𝐨𝐰: 𝟏. 𝐈𝐦𝐩𝐨𝐫𝐭 - Get raw data 𝟐. 𝐂𝐥𝐞𝐚𝐧 - Handle missing values, duplicates 𝟑. 𝐔𝐧𝐢𝐯𝐚𝐫𝐢𝐚𝐭𝐞 - Analyze one variable at a time (distribution) 𝟒. 𝐁𝐢𝐯𝐚𝐫𝐢𝐚𝐭𝐞 - Relationships between two variables (correlation) 𝟓. 𝐌𝐮𝐥𝐭𝐢𝐯𝐚𝐫𝐢𝐚𝐭𝐞 - Interactions between many variables 𝐓𝐡𝐞 𝟒 𝐯𝐢𝐬𝐮𝐚𝐥𝐢𝐳𝐚𝐭𝐢𝐨𝐧𝐬 𝐲𝐨𝐮 𝐦𝐮𝐬𝐭 𝐤𝐧𝐨𝐰: 𝐇𝐢𝐬𝐭𝐨𝐠𝐫𝐚𝐦 - See distribution and skew. Is data normal? Skewed left? Bimodal? 𝐁𝐨𝐱 𝐏𝐥𝐨𝐭 - Spot outliers. See quartiles. Identify extreme values that need investigation. 𝐒𝐜𝐚𝐭𝐭𝐞𝐫 𝐏𝐥𝐨𝐭 - Find relationships. Linear? Exponential? Clustered? No pattern? 𝐇𝐞𝐚𝐭𝐦𝐚𝐩 - See correlations across all variables. Red = strong positive. Blue = strong negative. 𝐁𝐮𝐭 𝐡𝐞𝐫𝐞'𝐬 𝐰𝐡𝐚𝐭 𝐧𝐨 𝐨𝐧𝐞 𝐭𝐞𝐥𝐥𝐬 𝐲𝐨𝐮: EDA is where 80% of your insights come from. Not from fancy models. From asking the right questions early. I've seen: → Teams build complex neural networks when simple linear regression would work → Models trained on data with 60% duplicates (because no one checked) → Predictions ruined by outliers no one noticed → Features with zero variance included in models All preventable. With proper EDA. 𝐓𝐨𝐨𝐥𝐬: Python: Pandas, Matplotlib, Seaborn| R: ggplot2, dplyr | Others: Tableau, Excel Pick your tool. Master the process. EDA is not optional. It's foundational. Get to know your data before you model it. Can you explain the difference between univariate and bivariate analysis? ♻️ Repost if EDA is underrated in data science

How do you know if your medical affairs efforts are moving the needle? Here’s how to track the true impact of your work. ------------ Measuring success in medical affairs isn’t just about tracking surface-level metrics—it’s about understanding the trends and nuances within themes, topics, and key insight questions (KIQs) over time. But how do you get the full picture? Part of measuring success in medical affairs efforts across field and med info is tracking the result of your efforts over time. Imagine you’re monitoring a drug like Perolax, indicated for type 2 diabetes. You want to track HCP opinions on nausea as a side effect. Traditionally, you might: - Manually sift through field notes and med info inquiries for keywords like “nausea.” - Read through all the notes - Write up a report, possibly on a quarterly basis. - Manually compare sentiments across periods. This approach is time-consuming and often limited to a single channel, which means you could be missing out on a comprehensive view of what’s really happening. Traditional NLP tools will typically show word clouds or trends over time with specific keywords which can help track changes in keyword mentions over time. However, keyword tracking lacks the richness to uncover the full story behind theme (nausea in this case). So, how can you enhance this process today? Emerging AI technologies can automate and deepen your analysis across multiple channels: ☑️ Smart Tagging: AI can automate the tagging of relevant data points—something we've been able to do, but now with more sophistication. ☑️ Deeper Insights: Generative AI combined with NLP and other qualitative analysis techniques can automatically identify trends and sub-themes in your data with much more granularity. ☑️ Continuous Monitoring: AI keeps track of these trends over time, alerting you to any significant changes or emerging patterns. By shifting from manual keyword tracking to AI-driven analysis, you gain richer, more actionable insights—giving you a comprehensive understanding of how side effects like nausea might impact patient adherence and strategy.

What can you do with Python in Excel for FP&A and #finance? I have received this question many times since the launch! Python in Excel can be a game changer for FP&A and finance professionals. If you learn how and for what to use it. You can now do: ✅Cohort Analysis with Heatmaps ✅Time Series Forecasting Using ARIMA ✅Outliers Identification ✅Statistical Advanced Outliers Identification ✅Headcount Analysis ✅Monte Carlo Simulations I created this cheat sheet to help you. But if you want to learn how to leverage AI and Python for Finance, Nicolas Boucher and I have a course coming up: https://lnkd.in/e4FugWeY Comment "Python in Excel is here" and I can send you the Excel file with all the code in the examples! And a bit more detail on the examples: 1) Cohort Analysis with Heatmaps Easily track and visualize customer retention or employee performance trends over time with beautiful, interactive heatmaps. 2) Time Series Forecasting Using ARIMA Predict future financial outcomes like revenue or expenses using advanced ARIMA models that can capture patterns in historical data. 3) Outliers Identification Quickly spot unusual data points (e.g., abnormally high expenses or revenues) with scatter plots and advanced visuals. 4) Statistical Advanced Outliers Identification Go deeper with statistical methods to identify outliers based on standard deviation or interquartile range, providing a robust analysis of deviations from the norm. 5) Headcount Analysis Analyze workforce trends across departments or time periods using visually engaging box plots and scatter diagrams, highlighting fluctuations and unusual spikes. 6) Monte Carlo Simulations Simulate thousands of financial scenarios to model risk and uncertainty, providing a data-driven approach to decision-making and forecasting.

Most Excel users stop at formulas and PivotTables. But that’s just the surface. Would you like to stand out from the crowd? You need to start thinking like an analyst. Here are 4 data analysis techniques that will take your Excel skills to the next level. Just to be clear, PivotTables are great for summarizing data. But they're limited in helping you analyze it. Here's why. Data tables, including PivotTables, are good at two things: Looking up exact values. Comparing exact values. Quite frankly, this is more reporting than analysis. 1) Visual Analysis > Data Tables Tables summarize. Charts reveal. Visuals like: Histograms (for distributions) Scatter plots (for relationships) Line charts (for trends) ...make patterns jump out. Good luck seeing these patterns in a monster PivotTable. Instead, PivotTables feed your charts. 2) RFM Analysis: This is a simple but powerful analysis technique to evaluate customers: (R)ecency: How recently they purchased. (F)requency: How often they purchase. (M)onetary: How much they spend. RFM analysis is super simple to implement in Excel. **AND** It's not just for customers. At its core, RFM analysis is about analyzing data based on behaviors. You can define the analysis however you would like. Take healthcare as an example. Analyzing patients: (A)ge (B)lood pressure (W)eight (E)xercise minutes per week The possibilities are endless! 3) Cluster Analysis Sometimes, patterns aren’t apparent until you group the data. Two examples: Segment users by behavior Classify patients by characteristics Start with a scatter plot of two columns. Look for any clusters. Then, figure out what defines each cluster. Better yet... Use Python in Excel for cluster analysis. Python in Excel is included in Microsoft 365 subscriptions. It's your gateway to battle-tested analytics like k-means clustering. This will allow you to scale to using many columns to find hidden patterns. It's the future of Excel. 4) Logistic Regression This one’s for when you want to predict something like yes/no, true/false, approve/deny, etc. It helps answer questions like: Approve this application? Will the customer churn? Is this claim fraudulent? You can implement logistic regression using Solver. Better yet... Use Python in Excel. People have implemented logistic regression using Solver for years. But here's the problem. It's error-prone and doesn't scale. Python in Excel eliminates these problems and gives you way more insights. It's the future of Excel.

Advanced Analysis with Python in Copilot: How to work with time series data https://lnkd.in/gaXFYYd9 Time series analysis enables analysts to identify patterns, trends, and cyclic fluctuations over time. These insights are crucial for accurate forecasting, strategic planning, and informed decision-making. Despite its importance, working with dates and times in a sophisticated manner is often limited by Excel, which has become the butt of many internet jokes due to its handling of dates and times. Integrating Python within Excel has significantly enhanced its capabilities. The popular Pandas package, widely used in this environment, is even named in part after “Panel Data,” a type of time series data. To make this process even more user-friendly, we can now leverage Copilot’s Advanced Analysis features—a generative AI-assisted tool—to run various time series analyses on our data with ease. In this post, we'll explore how to derive time series insights in Excel with the help of Python using a well-known sales dataset, doing everything from basic data resampling and visualization to creating forecasts and predictive models. You can follow along with the free exercise files on my website. Integrating Python into Excel opens up a wide range of possibilities for time series analysis, far beyond what Excel can traditionally handle on its own. With tools like Copilot, Excel users can now easily perform complex tasks such as resampling data, checking for stationarity, and building advanced forecasting models like ARIMA. What questions do you have about working with time series data in Python within Excel or using Copilot’s Advanced Analysis features? Let me know below.