Sometimes we come across unique, specialized use cases where solutions must be found outside of the box.  This particular use case for me involved industry physical security video recording for hundreds of cameras across many different geographic locations -- with a caveat -- the vendor software can ONLY write to a locally presented disk (no NAS locations, no cloud possibility).  Conditions were also right in the storage sphere where a new large-capacity all flash array came to market with incredible pricing, so it was more attractive than using HCI storage for this solution (we needed ~2PB of storage space).  We had never used iSCSI in this fashion before, so it was a journey of discovery for us.  This is that journey.

Prep work:
1) carve out a dedicated network for the iSCSI traffic and make sure the storage and the VM's can access it.
2) on each VM, add a vNIC to be used exclusively for iSCSI traffix
3) on each VM, modify the registry according to vendor best practices.  In our case that meant modifying two: HKLM TcpAckFrequency and HKLM ​TcpNoDelay.

Configuration & Presentation:
1) Get the IQN from the VM by opening iSCSI initiator

2) Register the VM initiator IQN with the storage array.  Typically you will create a host/VM entry in the storage array, and connect the initiator IQN to it.

3) On the VM, ) install MPIO if it’s not already installed on the server you are setting up iSCSI on:
https://support.purestorage.com/Solutions/Microsoft_Platform_Guide/Multipath-IO_and_Storage_Settings/Installing_Multipath-IO
​
3a) Configure MPIO to mask the iSCSI volumes in Windows Server per the vendor specifications.  This involves inputting a string specific to the vendor into the Device Hardware ID field in the MPIO properties window

3b) ​You need to add iSCSI support on the MPIO Discover Multi-paths tab as well:

4) Add the iSCSI discovery portals on the VM (server).  In our case, we wanted 8 paths, 4 per controller, so each controller needs to be added to the target portals: 

5) Go into ISCSI on the VM and connect to the discovered target.  Be sure to enable Multipath.

6) the new disk(s) should now show up in disk management on your VM and can be formatted and used.

Great news for those using ACI in their environments, who might be used to ACI having visibility into their virtualized VMWare environments -- this functionality is coming to Nutanix! 

​From Cisco's release notes: "You can now integrate the Nutanix Acropolis Hypervisor (AHV) with Cisco Application Centric Infrastructure (ACI). The Cisco APIC integrates with Nutanix AHV and enhances the network management capabilities. The integration provides virtual and physical network automation and
VM endpoints visibility in Cisco ACI." ​https://www.cisco.com/c/en/us/td/docs/dcn/aci/apic/kb/cisco-aci-nutanix-integration.pdf

You need to be running:
Cisco APIC 6.0(3) -- (released August 2023) or newer
Nutanix: To use current vSwitches without creating a NEW vSwitch (no vSwitch creation):

• AOS 6.5
• Prism Central 2022.6

To be able to create vSwitches

• AOS 6.6+
• Prism Central 2022.9+

Benefits of the integration are:

• visibility into the cluster from the Cisco APIC
• network statistics collection into APIC
• support for Cisco EPG's (endpoint groups) to configure networking support on Nutanix
• support for intra-EPG policies (intra-EPG contract is not supported) 

The image above shows a sample topology. The Cisco APIC manages the ACI fabric and policies. The VMs connected to the virtual switch are managed by the Nutanix Prism. ACI fabric provides network connectivity for VMs in AHV Hosts with the uplinks (Uplink1 and 2).

How it works:
Cisco ACI uses EPGs, or end-point groups, to deliver policy to sets of defined endpoints (i.e. VMs) within
ACI to NCP AHV with Flow Network Security as a VMM solution.

Documentation from Cisco here:
Inclusion on Cisco’s ACI VMM solution webpage:
https://www.cisco.com/c/dam/en/us/td/docs/Website/datacenter/aci/virtualization/matrix/virtmatrix.html
. Cisco ACI 6.0.3 release notes mention the new Nutanix integration:
https://www.cisco.com/c/en/us/td/docs/dcn/aci/apic/6x/release-notes/cisco-apic-release-notes-603.html
. Cisco ACI and Nutanix AHV VMM Integration Guide/Tech Note:
https://www.cisco.com/c/en/us/td/docs/dcn/aci/apic/kb/cisco-aci-nutanixintegration.
html?dtid=osscdc000283

Big announcements aplenty at the .NEXT 2023 conference. 
In no particular order:
1. Nutanix Central: A SaaS based cloud-delivered solution that will provide a single interface for managing ALL of your Prism Central deployments, whether on-prem, cloud or MSP.  In early-access now(ish) with features set to roll out going forward.
​https://www.nutanix.com/blog/introducing-nutanix-central

2. Multicloud Snapshot Technology (MST) - This allows snapshots directly to cloud-native datastores in S3.  

• Ability to protect and migrate stateful Kubernetes applications and data across cloud infrastructures with NDK
• Disaster recovery and backup for containerized virtualized applications
• Ability to create a snapshot with instant recovery anywhere
• Cross-cloud migration
• Ability to share data for workflows like test/dev, long-term retention for compliance, etc.

https://www.nutanix.com/blog/nutanix-reimagines-business-continuity-for-hybrid-multicloud-users

3. Project Beacon - This initiative takes aim to provide a portfolio of data-centric PaaS services available natively in the cloud.  It's intent is to decouple applications and their data from the underlying infrastructure, giving developers greater flexibility to build applications once and run them across various environments natively on Nutanix or in the public cloud.
https://www.nutanix.com/blog/introducing-project-beacon

4. Nutanix Data Services for Kubernetes (NDK) - in early access.  NDK is designed to deliver enterprise-class storage, snapshots and DR capability to Kubernetes.  
https://www.nutanix.com/blog/nutanix-announces-early-access-of-ndk

5. NCP enahancements which include the ability to dynamically apply security policies across multiclouds enabling secure, seamless virtual networks using Flow Virtual Networking.  Available now in NC2/Azure, and in development for AWS.

https://www.nutanix.com/blog/dynamic-approach-to-hybrid-multicloud-network-securityNCP 

It was also a chance to meet up with old friends and make some new ones.  NUG Chapter Champions in attendance met in the Solutions Expo for some hangout time and a group photo:

Shout out to all the Nutanix Technology Champions in the Day 2 General Session:

Great conference, nice venue.  It was nice having the conference so close to home base.  Hope to see everybody in 2024!

History:
We had been running Nutanix successfully for years on the ESXi platform.  We initially landed on ESXi because that's what we were using when we decided to move to HCI-first.  In an effort to streamline operations, lifecycle management, we took a serious look at AHV as an option -- after all, it was included with the Nutanix licenses we were already paying for.  We added an AHV cluster to our test lab and put it through the paces for a few months and then we decided it was ready for primetime, so we kicked off this project.

Our goals: 

• reduce overhead wherever possible and improve efficiencies
• remove the extra management task of keeping up with frequent security vulnerabilities on the VMWare platform and having to manage 2 different lifecycle management processes. 
• reduce our redundant licensing costs for VMWare

Challenges to overcome:

• Existing backup solution not compatible with AHV (ended up pivoting to Hycu for AHV workloads)
• Existing BCDR solution not compatible with AHV (ended up selecting Nutanix Leap for AHV workloads)
• end-user access/rights via Prism Central not like-for-like compared to vSphere.  Our migration took place prior to custom RBAC being available, so we used categories to control what users would see when logged in.
• end-user training on Prism Central would be needed
• Some vendor-supplied workloads not certified/approved for AHV, vendor "FUD"  These were simply added to an exemption list for future analysis/migration.

Phase: Planning

• spreadsheet from all active VM's with identified owners, work through support and migration plans to AHV.
• Work with vendors, where applicable, to get buy-in to run workloads on AHV vs ESXi
• Making sure passwords with administrative access were in place was key.  AD connected machines were easy, Linux took some coordinated effort with our Linux team to make sure we could seed the MOVE jobs.  

Phase: Buildout

• We were lucky enough to have the cluster refresh cycle align with our move plans.  3 new clusters were sized, ordered and deployed.
• Most of our production workloads were coming from 3 large general-purpose ESXi clusters, which made things easier: we would just move ESX1->AHV1, and so on.  We had a few other special use-case clusters which also needed new AHV target clusters. 

​Here are the steps to move virtual machines from ESXi to Nutanix AHV using Nutanix Move:

1. Download and install Nutanix Move on a server or virtual machine that has network connectivity to both the source ESXi host and the target Nutanix AHV cluster.
2. Configure Nutanix Move to connect to both the source and target environments (for us the vCenter and the target Prism Element cluster)
3. Create a migration plan by selecting the virtual machines to be migrated and specifying the migration settings, such as the target cluster and storage container.
4. Validate the migration plan to ensure that it meets the requirements of the target environment and does not have any errors or warnings.
5. Start the migration process and monitor the progress of each virtual machine as it is migrated to the target environment.
6. Verify that the migrated virtual machines are running correctly on the Nutanix AHV cluster.
7. If necessary, perform post-migration tasks such as updating the virtual machine configuration or modifying the network settings.

The goal: move an entire production Citrix & VDI platform from ESXi to AHV.  Below is our before (left) and after (right)

What is delivered:
•EMR sessions daily: 10000+
•VDI sessions daily: 4800+ 
•delivers 900+ general apps
•13000+ Citrix users
What makes it up:
•Published Apps (1155 servers)​
•Infrastructure (69 Server)​
•primary data center hosts- 110​
•secondary data center hosts- 72

The Citrix migration

1. Drain old ESXi cluster
2. Convert 3 hosts to initially seed the new AHV cluster
3. Foundationed the remainder of the hosts and added them to the AHV clusters
4. Redeploy Citrix servers (power off on ESXi, redeploy on AHV via Citrix)

Entire process was done in less than a month
​

The migration was fairly painless and we actually saw improved density on the clusters running AHV instead of ESXi on top of AOS -- about a 10% improvement.  The transition for our Citrix support staff was seamless.

The success of this project paved the way for our next project; converting our general production workloads from ESXi to AHV.