Fun with Neutrix Cloud – From Rendering to Dev&Test to Spark Tasks

I am writing this blog post while flying to Japan to meet with some of our great customers. All I see out the window is the cloud-filled horizon. These endless clouds as far as the eye can see make everything looking white and fluffy…

There is a similar view in the IT industry as well – the endless compute clouds covering the world, and we can see a growing demand among our customers to move some or all of their IT operations to these public clouds. While they appreciate the simplicity, agility and endless capacity, they are not willing to give up the functionality and reliability they enjoy in their private data centers. This is especially true for data storage, a critical component of any infrastructure deployment, whether it happens on-premises, in the clouds, or some mixture of both.

At INFINIDAT, we certainly care about data storage. We see tremendous growth in our on-premises deployed capacity, now exceeding 4 exabytes, and we are talking to customers every day about how to help them with their public cloud journey. Our solution for storage in the public cloud is Neutrix Cloud – a cloud-adjacent storage-as-a-service offering, powered by our proven InfiniBox storage.

There are three major usage models for Neutrix:

    • Replicating data from on-premises environments to Neutrix Cloud, potentially as a DR target
    • Consuming data over NFS and/or iSCSI interfaces from virtual machines running in one or more public clouds, as a feature-rich alternative to native cloud storage solutions
    • A combination of the above – replicating data between on-premises and Neutrix, and accessing it from one or more public clouds.

Over the next few blog posts, I’d like to review a few real use cases for Neutrix Cloud. We’ll use the Neutrix Command Line Interface in the examples below, but the same operations can also be done using the Neutrix API and our new GUI, both of which will be on display and available for demonstrations at this year’s AWS re:Invent, booth #1503. Let’s start with…

Rendering a movie

A design company is planning to run a rendering farm on the cloud. They have limited compute capacity in the data center, with enough resources only to perform interactive work. However, rendering requires much more compute power – and it might be very expensive. The public cloud is a well-known solution for this kind of challenge.

All major public clouds offer an option to buy excess compute capacity at a lower price, what they call “spot instances”. At scale, it makes sense to compare pricing between different public clouds and find the cheapest offering.

However, data sharing among clouds becomes a gating factor. If your input files are stored in a Cloud A, transferring them to a Cloud B might be costly due to egress charges applied by every cloud vendor.

Neutrix Cloud comes to the rescue.

Let’s look at a simple example – using blender tool to render frames in the public cloud.

The first step would be to create a blender input file. This is an interactive process, which can be done within a data center – or even on a laptop by a designer.

Let’s assume a movie designer has an NFS file system on their on-premises InfiniBox, where the input files are stored. The InfiniBox name is MyMovieIbox and the file system name is NeutrixBlenderDemo.

First, this file system should be replicated to Neutrix Cloud to make it accessible for the public cloud. Let’s assume the replication link has been already established between the customer’s data center and Neutrix Cloud. The next step would be to create a replica between an on-premises file system and the Neutrix environment:

awsdemo@ims-dc > 
replica create 
--source MyMovieIbox
--target US-EAST 
--dataset NeutrixBlenderDemo 
--sync_interval 20 
--description BlenderDemo

Create replica for NeutrixBlenderDemo succeeded. 
Target dataset ID: ds_4b4ae144ee1d4623982784c05bb30335_tgt

While the blend file can be pretty big, the changes within it are often very small on each iteration. The very efficient InfiniBox replication mechanism will transfer only the changed blocks between the source and the target to keep them in sync.

So a replica of the InfiniBox file system in Neutrix Cloud will contain all content of the source file system, including the input data – in close proximity to the public cloud regions.

Now, the designer may turn the Neutrix file system into a master copy and the on-premises file system into a target, reversing the replication direction:

awsdemo@ims-dc > replica change_role 
--remote_system MyMovieIbox 
--remote_dataset NeutrixBlenderDemo

Change role replica for NeutrixBlenderDemo succeeded, 
source is now in the cloud

The file system in Neutrix can now be accessed from any adjacent public cloud or clouds. As of today, we have direct integration with Amazon Web Services (AWS), Microsoft Azure, Google Cloud Platform, IBM Cloud (SoftLayer), and VMware Cloud on AWS (VMC) environments. We’re willing to consider other public cloud connectivity per request.

Assuming the designer has an account in each of these clouds, they can access the Neutrix file system using NFS over a dedicated low-latency, high-bandwidth connection from any of these clouds:

sudo mkdir -p /mnt/NeutrixBlenderDemo
sudo mount $PRIVATE_NFS_ENDPOINT:$PRIVATE_NFS_EXPORT /mnt/NeutrixBlenderDemo

They can dynamically shift the rendering workload among public clouds to achieve the lowest cost based on the best spot instances’ pricing available. All their virtual machines will have concurrent read-write access to the same file system from all supported clouds.

Running a separate blender process on each of the participating virtual machines will distribute the workload and speed up the overall rendering process, while storing all generated frames on the same Neutrix NFS file system.

blender -b /mnt/NeutrixBlenderDemo/infinidat_cube.blend 
-o /mnt/NeutrixBlenderDemo/${CLOUD}_`hostname`/frame_ 
-s $START -e $END -a

The Neutrix file system is being replicated back to the on-premises InfiniBox, and by the time all blender processes finish, the frames will already be available back in the datacenter.

At this stage the designer can deprovision compute resources in the cloud and reverse the replication direction once again, making the on-premises file system a source: > replica change_role 
--remote_system MyMovieIbox 
--remote_dataset NeutrixBlenderDemo

Change role replica for NeutrixBlenderDemo succeeded, 
source is now on-premises

From here it’s possible to glue all the frames into a movie:

$ ffmpeg -framerate 25 
-i frame_%04d.png 
-c:v libx264 -profile:v high 
-crf 20 
-pix_fmt yuv420p output.mp4

This is just one example of how to leverage Neutrix Cloud in a multicloud scenario. I’ll cover Spark tasks on Amazon Elastic MapReduce and spawning multiple database instances in the next couple of posts…

About Gregory:
Gregory Touretsky (@gregnsk) is a Senior Director, Product Management at INFINIDAT. He drives the company’s roadmap around NAS, cloud and containers topics. Before that Gregory was a Solutions Architect with Intel, focused on distributed computing and storage solutions, data sharing and the cloud. He has over twenty years of practical experience with distributed computing and storage. Gregory has an M.S. in Computer Science from Novosibirsk State Technical University and an MBA from Tel-Aviv University.


Request a Demo