Primer Series: MemVerge

Do you ever attend meetings where the conversation turns to an unfamiliar technology? Suddenly a bunch of acronyms are being thrown around and you have no idea what they mean. Confused, you scan the room and notice that everyone else is nodding as if they know exactly what's being discussed.

We've all been there before. To help our valued customers make sense of it all, the engineers at WWT offer a continuing series of 'primer' articles to provide a basic understanding of emerging products and technologies.

This article covers the fundamentals of MemVerge, a software-defined memory service that virtualizes DRAM and persistent memory to cost-effectively eliminate I/O bottlenecks — by eliminating the need for I/O altogether.

Why MemVerge "Big Memory" software is important

More and more data is being generated faster than ever, and IT infrastructures are hard-pressed to capture, analyze and store it quickly, let alone in real time. Additionally, compute-intensive workloads — analytics, simulation, modeling, 3D visualization and more — are increasingly in demand. By 2024, IDC estimates that 25 percent of all data will be real-time data.

In the traditional data center, memory and storage are bonded closely as an integrated system that must be kept in balance, and the need for moving data from one to the other can result in I/O bottlenecks. To process data on demand, large enterprise applications incorporating high performance computing (HPC), artificial intelligence (AI), machine learning (ML), deep learning (DL) and real-time analytics are increasingly common, and it's creating a burgeoning need for massive amounts of data "in-memory." 

But IT organizations have been stymied by the inability of leading-edge in-memory infrastructures to keep pace. What's more, scaling dynamic random access memory (DRAM) into terabytes is prohibitively expensive.

The quest for better real-time workflows got a boost with the introduction of Intel® Optane™ persistent memory, a storage-class memory that actually combines both tiers, bringing together the speed of DRAM with the persistence of storage to make storage much faster and memory more scalable.

But, in order for today's enterprise applications to use the features of Intel Optane persistent memory, a new software layer was necessary — one that would make the API compatible with existing applications and deliver the necessary data services on top of the memory to maximize its usefulness. That software has now been developed by MemVerge, a technology partner of WWT and Intel.

The elements of MemVerge       

MemVerge has transformed the memory/storage paradigm by combining these essential assets:

• Big Memory is a new class of computing in which big applications and data reside in byte-addressable persistent memory. This memory can scale out massively in a cluster and is protected by next-gen memory data services that provide snapshots, replication and ultra-fast recovery, as well as making immense memory pools highly available.
• Memory Machine™ is the flagship offering: enterprise-class Big Memory software that virtualizes DRAM as a "fast tier" and persistent memory as a "storage tier," answering the need for a modern in-memory computing model. It can scale out memory in a cluster to form the great memory lakes required for real-time analytics, artificial intelligence and machine learning applications.
• ZeroIO™ Snapshot is a checkpointing feature of Memory Machine that greatly simplifies the process of crash recovery. ZeroOI can be automated to instantly persist the memory state into storage as often as every minute. When disaster strikes, IT can use the last snapshot to restore applications in seconds.
• Enabling technology from Intel. MemVerge leverages Intel Optane persistent memory and the Intel chipset. The MemVerge solution also relies on the Apache Spark unified analytics engine for big data, as well as a robust compute and network infrastructure.

Transparent memory services enable the MemVerge solution to be quickly deployed, and get up and running easily without having to modify application codes. Because memory and storage exist as one, MemVerge simply cannot experience I/O bottlenecks, resulting in extremely low latency.

The result: abundant, persistent, available memory for a range of new capabilities.

The benefits of MemVerge: Scalability, fast recovery and the end of bottlenecks 

• Scalability – Organizations have easy access to terabytes of low-cost DRAM-speed memory available for memory-intensive applications on an as-needed basis.
• Faster crash recovery – Memory-intensive applications used to take forever to restart after a crash or shutdown. The MemVerge ZeroIO Snapshot now enables lightning-fast data restoration from the last snapshot, recovering hundreds of gigabits in seconds.
• Compatibility – No need for time-consuming application changes or rewrites. MemVerge is plug-in compatible with existing applications.
• Rapid cloning – Quickly create a new application instance from any snapshot without memory copy.
• Elimination of complexity – MemVerge helps simplify the data center, reducing the need for complicated and redundant storage backups.
• The end of I/O bottlenecks — When storage and memory are joined, organizations can accomplish new classes of computing workflows at greater scale. In fact, MemVerge often improves DRAM performance overall.

Use cases with real-time workloads are made for MemVerge

Perhaps the most logical applications for MemVerge's bottomless memory pool lie in the financial industry. MemVerge technology is ideal for latency-sensitive transactional workloads such as trading applications. The company is collaborating with financial clients on real-time market data analysis to deliver insights for high-frequency trades. 

From healthcare and retail to energy industries, applications that employ HPC, AI, ML/DL and real-time analytics to process copious volumes of data are rapidly growing — and so is the need to process that data in-memory.

If data is the new oil, then memory is the new gold. WWT's long-term vision for MemVerge is that the memory layer will continue to play an expanding role in the data center, minimizing complexity while improving performance. By expanding in-memory capabilities and instant recovery features, data centers can eliminate the need to send data to storage, disk or SSD. It will always be there in Big Memory.

Learn more

Have questions about MemVerge? Our subject matter experts are here to help. Contact your WWT account team or reach out to us directly to start the discussion.