New Algorithm from MBZUAI Reveals Hidden Insights in Complex Data
The ability to uncover such insights from limited, high-impact data could be especially valuable in fields where rare events carry outsized consequences.
Topics
News
- New Algorithm from MBZUAI Reveals Hidden Insights in Complex Data
- AI Phishing Is Rising in the UAE—How Can Professionals and Businesses Stay Safe?
- CFOs Rethink ROI as AI Moves to Center Stage in Enterprise Strategy
- Core42 Deploys OpenAI GPT-OSS Models for Sovereign AI Access
- UAE Sets Global Standard for Human-Centric Digital Transformation
- Cybersecurity Consolidation Gains Momentum Across the Middle East
[Image source: Krishna Prasad/MITSMR Middle East]
In a recent breakthrough, a team of researchers including scholars from Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) has developed a novel statistical method that addresses a major blind spot in current data analysis techniques: the detection of rare dependence — subtle, localized relationships between variables that can be easily missed by conventional models.
From public health to economic modeling and AI systems, the ability to identify these hidden patterns is increasingly vital. As Yiqing Li, a research associate at MBZUAI and co-author of the study, explains, “People want to know why and how economic crises happen and how to prevent them, but the number of data points for financial crises is small. Normal testing methods fail in these scenarios, and this is one of the areas where our method can provide value.”
The research, conducted in collaboration with institutions including Carnegie Mellon University and The University of Queensland, was presented at the International Conference on Machine Learning (ICML) in Vancouver. It centers on a method that uses kernel-based conditional independence testing via sample importance reweighting. In simpler terms, the approach reassigns more importance to the parts of a dataset where unusual dependencies may exist — rather than treating all data points equally.
Li and her co-authors — Yewei Xia, Xiaofei Wang, Zhengming Chen, Liuhua Peng, Mingming Gong, and Kun Zhang, describe their strategy as paying more “attention to the dependent sub-samples, successfully detecting rare dependence.”
The technique was validated using both synthetic and real-world datasets. On the synthetic side, the method outperformed established tools such as the Hilbert-Schmidt Independence Criterion (HSIC), which gives equal weight to every sample. In contrast, the new test controlled for type I errors (false positives), while demonstrating higher testing power. “We have to solve an optimization problem to get the best reweighting function and then amplify the informative subsamples to detect the rare dependence,” Li noted. “Existing testing methods can test independence directly, which is faster, and testing the whole data can also provide important insights.”
In a real-world application, the team analyzed data from the U.S. Federal Reserve covering the years 1990 to 2010. They examined the relationship between the exchange rate of the Japanese Yen to the U.S. dollar and the U.S. federal funds rate. Their method identified dependencies during key crisis years, from 2001 and 2008, while traditional methods like HSIC failed to flag any meaningful relationships.
The ability to uncover such insights from limited, high-impact data could be especially valuable in fields where rare events carry outsized consequences. For example, in medicine, drugs can have unintended effects in small subsets of patients. “Opioids have been found to actually make the pain worse in a subset of patients,” the researchers noted a classic case of rare dependence that standard tests might overlook.
Beyond testing for simple dependence, the team also derived a conditional independence test and incorporated it into the PC algorithm, a widely used tool for causal discovery in AI and data science. The enhanced algorithm, dubbed Rare Dependence PC (RDPC), successfully identified the correct causal relationships in datasets where rare dependence exists.
Looking ahead, the research team sees a range of possibilities for further development. “One direction would be to try to figure out the exact mechanism that’s behind rare dependence in certain datasets and under what conditions these relationships happen,” said Li.
In the meantime, the team hopes their method will prove useful across disciplines. “Whenever you encounter a dataset that might contain rare dependence relationships, please try our method,” Li encouraged.
