An HR analyst company, pulled just one-month Microsoft Teams data to draw a heat-map of internal chat from about 1200 employees. A simple analysis of the chat messages revealed that a drop of daily messages by >30% preceded resignations by five to six weeks. Managers got an early warning ‘retention heat map’ every Monday to take action. Result; it cut down attrition by 18% in two quarters. This is a classic examples of when good enough data, or small data meets the business requirements, without having to boil the ocean with big data analytic.
According to MyDataModels, a pioneering French start-up, small data accounts for 85% of all data collected. In fact, modern AI technologies are now adopting capabilities to encapsulate knowledge-based intelligence from data and information that is: small, specific and feature-rich.
However, small data is not always solving small challenges, in fact smart data is useful to handle quite complex challenges. Imagine planning a subway line beneath Manhattan. Conventional wisdom says you’d need to drill test sites across hundreds of city blocks, spending millions on geological studies before choosing a route. But what if you could guarantee the optimal path with just a handful of strategically chosen tests? MIT researchers just proved this isn’t wishful thinking—it’s mathematically possible.
The Data Collection Trap We’ve All Fallen Into
We’ve been conditioned to believe more data equals better decisions. Companies spend fortunes gathering exhaustive datasets, training massive AI models, and conducting endless field studies. In supply chain management, firms routinely collect shipment data across thousands of routes “just to be safe”. Transportation planners face similar pressure, often working with limited budgets while relying on outdated surveys that are “rarely updated” and fail to capture granular geographic insights.
In India, this challenge is particularly acute. The country’s supply chain market is projected to grow at 11.1% annually, reaching $6,433.24 million by 2030. Yet Indian businesses grapple with high logistics costs, inventory mismanagement, and demand fluctuations—problems that traditional data collection approaches struggle to solve efficiently. A multinational FMCG company expanding into India faced exactly these challenges, with logistics costs spiraling due to inadequate data on optimal routes.
This “collect everything” approach creates a vicious cycle: limited budgets force planners to use stale data, which leads to poor decisions, which makes it harder to justify future funding for better data collection. The MIT team asked a radical question: what if we’re measuring the wrong thing? Instead of estimating every parameter accurately, we only need data that can discriminate between competing optimal solutions.
The Genius of “Optimality Regions”
Here’s where the math gets elegant. Every possible set of costs—construction expenses, travel times, energy prices—makes one specific decision optimal. These “optimality regions” partition the decision space like neighborhoods on a map. Your dataset is sufficient not when it’s comprehensive, but when it can pinpoint which region contains the true cost.
Think of it like finding the best dosa place in Bengaluru. You don’t need to try every single restaurant. You just need enough data to rule out all but one neighborhood. The MIT algorithm works iteratively, constantly asking: “Is there any scenario that would change the optimal decision in a way my current data can’t detect?” If yes, it adds one precise measurement. If no, you’re done—your dataset is provably sufficient.
Supply Chains: From Billions to Millions
Let’s ground this in Indian supply chain reality. When an automotive parts manufacturer struggled with high raw material procurement costs, they didn’t need to analyze every supplier transaction. By focusing data collection on bottleneck processes and strategic sourcing decisions, they achieved significant cost reductions. Tecnova, a consulting firm helping businesses navigate India’s complex supply chain landscape, uses similar principles—targeting data collection at critical decision points rather than gathering everything.
The MIT framework explains why this worked. The company didn’t need perfect cost estimates for every supplier—they just needed enough data to distinguish between procurement strategies. As Deere & Company demonstrated globally, strategic focus on “merge centers” and cross-dock locations slashed inventory by $1 billion and cut delivery times in half.
Infrastructure Planning: Digging Smarter in Indian Cities
India’s metro rail boom makes this framework urgently relevant. Bengaluru Metro Phase 1 faced significant challenges during construction, with excavation and tunnelling difficulties requiring “many permutations and combinations of tunnelling methods”. Planners had to choose between tunnel boring machines and the new Austrian tunnelling method without complete geological data.
Traditional approaches would mandate extensive borehole studies across the entire route. But using MIT’s framework, planners could identify the specific segments where geological uncertainty might actually change the optimal construction method. WRI India researchers note that metro rail operators currently don’t use data-driven methods for planning last-mile connectivity, instead relying on intuition rather than “data that could be collected by metro rail last-mile planning departments”.
The MIT algorithm would pinpoint exactly where to collect geological data to guarantee the lowest-cost route. Early design and planning investments significantly impact risk management, so getting this right matters. Instead of drilling everywhere, you’d drill only where the data would definitively tell you, “This tunnelling method beats that one.
Why This Matters for Your Career
For Indian data science and management students, this framework is liberating. It challenges the misconception that small data means approximate solutions. The Indian government is already exploring operations research for budget planning, with initiatives to “establish a dedicated team of OR experts to work alongside policymakers”. You don’t need massive datasets to provide exact, provably optimal answers. You need smarter questions.
The researchers have shown that careful data selection can guarantee optimal solutions with mathematical certainty—not probability. This means you can walk into a meeting and say, “We need only these five measurements, and here’s the proof they’ll give us the right answer.” That’s career-changing credibility, especially as Indian businesses increasingly adopt data-driven decision-making.
Stay connected with us to explore endless opportunities at Praxis Business School!
Visit our website at https://praxis.ac.in/ to learn more about our programs, admissions, and campus life. For any queries, feel free to reach out to us at https://praxis.ac.in/contact-us.
Follow us for the latest updates, insights, and success stories.
We look forward to connecting with you!