Categories
AI

Databricks unifies data science and engineering with a federated data mesh

Elevate your enterprise data technology and strategy at Transform 2021.


During its online Data + AI Summit conference, Databricks today unveiled Databricks Machine Learning, a platform that lets data science teams build AI models based on the AutoML framework.

The offering follows yesterday’s launch of an open source Delta Sharing project that lets organizations employ a protocol to securely share data across disparate data warehouses and data lakes in real time.

The Delta Sharing project has been donated to the Linux Foundation and is being incorporated with Delta Lake, an open source data lake platform Databricks previously made available. Organizations that have pledged to support the Delta Sharing project include Nasdaq, ICE, S&P, Precisely, Factset, Foursquare, SafeGraph, Amazon Web Services (AWS), Microsoft, Google Cloud, and Tableau.

Delta Sharing can already be applied to share data across Azure Data Share, Azure Purview, Big Query, AtScale, Collibra, Dremio, Immuta, Looker, Privacera, Qlik, Power BI, and Tableau platforms.

A slew of updates

Delta Sharing will in effect establish a common standard for sharing all data types using an open protocol that can be invoked using SQL, visual analytics tools, and programming languages such as Python and R. Delta Sharing also enables data stored in Apache Parquet and Delta Lake formats to be shared in real time without employing copy management tools. And it provides built-in security controls and permissions to address privacy and compliance requirements.

This week, Databricks also unveiled Unity Catalog, a unified data catalog for Delta Lake that makes it easier to discover and apply controls at a more granular level in order to govern data assets using capabilities enabled in part by Delta Sharing. Alation, Collibra, Immuta, and Privacera have pledged to support Unity Catalog.

Finally, Databricks has added a cloud service dubbed Delta Live Tables to simplify the development and management of data pipelines using a set of simpler extract, transform, and load (ETL) capabilities that automates that process. Delta Live Tables abstracts away the low-level instructions data engineers previously had to code, which reduces the opportunity for errors. Delta Live Tables then automatically creates the instructions for both the data transformations and the data validations, as well as implementing error handling. Any dependencies are automatically executed downstream whenever a table is modified. Delta Live Tables will also make it simpler to identify the root cause of errors and restart pipelines when necessary.

Unity

Collectively, these offerings are part of an effort to unify data science and data engineering and reduce friction by creating a federated data mesh, Databricks CEO Ali Ghodsi told VentureBeat. As part of that effort, Delta sharing provides a standard mechanism through which data can be migrated from legacy platforms into any data lake without requiring data engineers to employ cumbersome processes using copy data management tools, Ghodsi noted.

IT organizations can also choose to not employ specific data lakes and warehouses as they see fit instead of being forced to standardize on a single platform just to simplify sharing and accessing data, Ghodsi noted. That’s especially critical for organizations that need to share data with other entities because the odds that those organizations will have adopted the same data warehouse or data lake are slim to none, Ghodsi added. “The data is always going to be out of sync,” he said.

AI models constructed with data aggregated using these tools are also extensible, which Ghodsi said is a capability unique to the Databricks platform. That approach enables organizations to construct and train AI models with either a user interface or application programming interface (API) in a way that allows experiments to be automatically generated without compromising transparency, Ghodsi added.

Those AutoML experiments are integrated with the rest of the Databricks Lakehouse Platform, which makes it possible to employ open source MLflow software Databricks has developed to track parameters, metrics, artifacts, and even entire models, Ghodsi said. The Databricks Machine Learning platform is designed from the ground up to unify best data science and engineering practices, Ghodsi added.

It remains to be seen to what degree data science and engineering will converge, but today most organizations find data management the biggest obstacle to implementing AI.

VentureBeat

VentureBeat’s mission is to be a digital town square for technical decision-makers to gain knowledge about transformative technology and transact.

Our site delivers essential information on data technologies and strategies to guide you as you lead your organizations. We invite you to become a member of our community, to access:

  • up-to-date information on the subjects of interest to you
  • our newsletters
  • gated thought-leader content and discounted access to our prized events, such as Transform 2021: Learn More
  • networking features, and more

Become a member

Repost: Original Source and Author Link

Categories
AI

Aunalytics unifies siloed bank customer data with AI-driven data mart and NLP

Join Transform 2021 for the most important themes in enterprise AI & Data. Learn more.


Aunalytics announced an update to its Daybreak for Financial Services platform that employs machine learning algorithms to enable midrange banks and credit unions to more easily analyze data.

The latest update adds a data mart that automatically discovers and aggregates customer data residing in siloed lending, mobile banking, automated teller machine (ATM), customer relationship management (CRM), wealth management, and trust applications. The platform has also added support for a natural language processing (NLP) engine that eliminates the need to know SQL to query data. Companies can automatically create visualizations of those query results as well.

Finally, Aunalytics made it simpler to access external data via connectors and added a “smart features” capability that will, for example, automatically generate alerts anytime a customer’s credit score changes.

Midrange banks and credit unions are at a distinct AI disadvantage compared to larger financial services rivals that can afford to build their own AI models with specialists who know how to program in Python or R programming languages, Aunalytics president Rich Carlton said. “They can’t afford to hire a team of data scientists,” he added.

Aunalytics is making a case for a platform that automates low-level data science tasks in a way that enables either end users or a small team of data scientists to maximize the value of the data any midrange bank or credit union routinely collects, Carlton said. The Daybreak for Financial Services platform is based on cloud-native technologies such as Hadoop, containers, and Kubernetes clusters that he said enable it to be deployed in the cloud or an on-premises IT environment.

Midrange financial services providers have realized they are losing touch with customers in the wake of the COVID-19 pandemic as they rely more on digital services. The number of banking customers that visit their local bank has sharply declined as reliance on web and mobile applications increases. The challenge midrange financial services face today is that they already rely on a disjointed suite of applications to manage their business. Mobile applications in particular have added yet another silo that makes it difficult for financial services providers to correlate customer activity across a portfolio of services.

Awareness of AI and data science has never been higher. The issue organizations are trying to come to terms with is to what degree they are now at a competitive disadvantage because they lack these capabilities. Platforms and applications that embed AI capabilities may provide a way to close that gap at a time when many smaller financial services firms need to operate as efficiently as possible just to stay afloat.

As data science and AI continue to evolve, organizations will soon need to decide when it makes sense to employ advanced analytics that are baked into a platform such as Daybreak for Financial Services versus building and maintaining their own AI models. Given the general shortage of data science professionals, it’s especially difficult for smaller organizations to hire and retain in-house talent.

At the same time, it usually takes a data science team several months to successfully deploy an AI model in a production environment. Providers of applications and platforms may very well have added similar capabilities to their offerings before that custom AI project ever comes to fruition. In many cases, organizations will find they are gaining access to advanced analytics capabilities at no extra cost as new updates are made available under a subscription license.

Most end users, of course, are a lot more interested in the business outcomes AI models and data science enable than they are in the processes employed to build them. The fact that an independent provider of a platform or application is willing to vouch for the accuracy of those AI models adds yet another perceived level of comfort.

VentureBeat

VentureBeat’s mission is to be a digital town square for technical decision-makers to gain knowledge about transformative technology and transact.

Our site delivers essential information on data technologies and strategies to guide you as you lead your organizations. We invite you to become a member of our community, to access:

  • up-to-date information on the subjects of interest to you
  • our newsletters
  • gated thought-leader content and discounted access to our prized events, such as Transform 2021: Learn More
  • networking features, and more

Become a member

Repost: Original Source and Author Link

Categories
AI

Torch.AI raises $30M for AI that unifies disparate enterprise data

Join Transform 2021 for the most important themes in enterprise AI & Data. Learn more.


Torch.AI, a startup developing “network-centric” AI to deliver big data insights, announced that it raised $30 million. The company plans to put the proceeds toward growth as it acquires new customers, particularly U.S. federal agencies in the national security realm, including those operating in high-risk environments.

Most enterprises have to wrangle countless data buckets, some of which inevitably become underused or forgotten. A Forrester survey found that between 60% and 73% of all data within corporations is never analyzed for insights or larger trends. The opportunity cost of this unused data is substantial, with a Veritas report pegging it at $3.3 trillion by 2020. That’s perhaps why the corporate sector has taken an interest in solutions that ingest, understand, organize, and act on digital content from multiple digital sources.

Leawood, Kansas-based Torch.AI, which was founded in 2017, offers one such solution in a platform that connects disparate apps, systems, cloud services, and databases to enable data reconciliation and processing. Torch.AI provides domain-specific pretrained machine learning models for optical character recognition, natural language processing, sentiment analysis, and more that enrich existing data, even if the data is incomplete, flawed, or unstructured. Companies can bring their own models to automate data engineering tasks and workflows ranging from notifying an analyst of an anomaly and its potential impact to recalculating risk or complexity scores, according to Torch.AI.

Torch.AI’s platform functions as an enterprise communication system that provides AI-enhanced data transformations, ingesting data from any source. It decomposes data into smaller, normalized bits with visibility and transparency into data lineage and source integrity, delivering “tagging on ingest” capabilities to help users organize and correlate information. In addition, Torch.AI provides a suite that maintains a hardened cybersecurity posture and tools that make the implementation of compliance regulations and policies ostensibly easier.

“Most data enablement implementations and analytics suffered from internal data engineering challenges. When we engaged with companies across the U.S., we heard the same thing: doing almost anything meaningful with data was too complicated and took too long,” CEO Brian Weaver told VentureBeat via email. “We discovered we could use the efficiency of an advanced application of machine learning to instantly understand and describe data with atomic detail, in memory and while it is still in motion. We patented the concept and started developing our platform, which intelligently connects all a company’s applications and business systems by overlaying a ‘synaptic mesh.’”

The global big data and business analytics market was valued at $169 billion in 2018 and is expected to grow to $274 billion in 2022, according to Statista. While Torch.AI’s AI-powered data reconciliation is more holistic than most, it has a number of competitors, including BackboneAI, which last year emerged from stealth with a product designed to unify enterprise data sets with AI. There’s also Tamr, a Cambridge, Massachusetts-based startup that uses machine learning to speed up data analytics workflows. And there’s Quantexa, which last July raised $64.7 million to further develop its AI platform that extracts insights from big data.

But Torch.AI claims to have quickly established itself in the private sector with a roster of blue-chip private clients including Microsoft, H&R Block, and General Electric. The company also says it’s “moving strongly” into the government sector, where its technology has been deployed across over more than a dozen U.S. federal agencies.

“Today, our customer base spans Fortune 100 companies to mission critical elements of the U.S. government. Typically, our clients come to us after suffering poor analytic outcomes or failing decision systems,” Weaver said. “One of the main benefits and differentiator of our platform is that our clients don’t need to change their internal IT infrastructure; rather, the platform overlays existing data and systems.”

WestCap Group led the series A funding, which is 50-employee Torch.AI’s first public funding round.

VentureBeat

VentureBeat’s mission is to be a digital town square for technical decision-makers to gain knowledge about transformative technology and transact.

Our site delivers essential information on data technologies and strategies to guide you as you lead your organizations. We invite you to become a member of our community, to access:

  • up-to-date information on the subjects of interest to you
  • our newsletters
  • gated thought-leader content and discounted access to our prized events, such as Transform 2021: Learn More
  • networking features, and more

Become a member

Repost: Original Source and Author Link