|
| | | | |
| | | |
| Stat |
Members: 3645
Articles: 2'504'928
Articles rated: 2609
25 April 2024 |
|
| | | | |
|
Article overview
| |
|
|
Sketch and Scale: Geo-distributed tSNE and UMAP | | Viska Wei
; Nikita Ivkin
; Vladimir Braverman
; Alexander Szalay
; | | Date: |
11 Nov 2020 | | Abstract: | Running machine learning analytics over geographically distributed datasets
is a rapidly arising problem in the world of data management policies ensuring
privacy and data security. Visualizing high dimensional data using tools such
as t-distributed Stochastic Neighbor Embedding (tSNE) and Uniform Manifold
Approximation and Projection (UMAP) became common practice for data scientists.
Both tools scale poorly in time and memory. While recent optimizations showed
successful handling of 10,000 data points, scaling beyond million points is
still challenging. We introduce a novel framework: Sketch and Scale (SnS). It
leverages a Count Sketch data structure to compress the data on the edge nodes,
aggregates the reduced size sketches on the master node, and runs vanilla tSNE
or UMAP on the summary, representing the densest areas, extracted from the
aggregated sketch. We show this technique to be fully parallel, scale linearly
in time, logarithmically in memory, and communication, making it possible to
analyze datasets with many millions, potentially billions of data points,
spread across several data centers around the globe. We demonstrate the power
of our method on two mid-size datasets: cancer data with 52 million 35-band
pixels from multiple images of tumor biopsies; and astrophysics data of 100
million stars with multi-color photometry from the Sloan Digital Sky Survey
(SDSS). | | Source: | arXiv, 2011.06103 | | Services: | Forum | Review | PDF | Favorites | |
|
|
No review found.
Did you like this article?
Note: answers to reviews or questions about the article must be posted in the forum section.
Authors are not allowed to review their own article. They can use the forum section.
browser Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)
|
| |
|
|
|
|
News, job offers and information for researchers and scientists:
| |
REGISTER