Veeam Support troubleshooting series: Linux FLR appliance deploy fail

Source: Veeam

Hello from Veeam Support!

Veeam Support Team is excited to announce a brand-new way of sharing useful and important information with you. We have prepared a series of advanced-level tech articles based on real support cases. We hope you will like these articles as much as we like to help our customers.

Issue of the week: Linux FLR appliance deploy failed

Linux FLR appliance deploy failed: Module ‘MonitorLoop’ power on failed.

Identifying the issue: Houston, we have a problem

First of all, we need to identify the issue. The main trick is that it hits during a very special phase of OtherOS FLR — only deployment of the appliance is affected. The error happens right when we attempt to mount a restore point to FLR appliance. In the GUI, the error looks like this:

Linux FLR appliance deploy fail
Fig 1. Error in UI

We can see the failed module ‘MonitorLoop’ in the log for the relevant FLR session (the name looks like “year_month_day_hour_minute_second.log”):

[05.07.2017 17:16:49] <06> Info     Mounting restore point. VM: [fileserver], BackupDate: [09.01.2017 18:31:12], Oib: [aa6038d3-bf68-42d6-86c0-de3a48784066]

[05.07.2017 17:17:49] <06> Error    Failed to mount oib “aa6038d3-bf68-42d6-86c0-de3a48784066”

[05.07.2017 17:17:49] <06> Error    Linux FLR appliance deploy failed: Module ‘MonitorLoop’ power on failed.  (Veeam.Backup.Common.CAppException)

Keep in mind that the FLR appliance is deployed by the mount service, which also logs similar issue in Svc.VeeamMount log. However, it doesn’t show us the failing module:

[05.07.2017 17:16:49] <23> Error    Recreating WCF proxy…

[05.07.2017 17:16:49] <23> Error    Linux FLR appliance deploy failed (System.ServiceModel.FaultException`1[Veeam.Backup.Interaction.MountService.CRemoteInvokeExceptionInfo])

Solving the issue: Resources are key

If we dig deeper, we notice that according to the VMware KB article, Module ‘MonitorLoop’ controls resources assigned to VMs. The error itself is thrown by VMkernel and can be traced in the relevant VMkernel log:

Linux FLR appliance deploy fail

Fig 2. VMware log

The root cause of our issue is that ESXi host doesn’t have enough resources to start the appliance, and naturally FLR fails. Without scanning through VMkernel logs, we cannot directly point which of the resources are missing, but we can easily estimate that.

This is either CPU/RAM available for VMs on the hosts, or free space for swap file storage. The latter is much more plausible, so if vSphere environment does not face obvious lack of CPU/RAM — the issues narrows down to lack of free space for storing FLR appliance and its swap file. To find out what is happening and fix it, run the FLR restore and double-check that FLR appliance configuration meets following criteria.

Linux FLR appliance deploy fail

Fig 3. Configuration setup

The selected Host should have enough CPU/RAM to store the appliance. Their consumption will usually be really low, so normally something more than 0 Mb of RAM and at least some CPU will do to start the FLR appliance. If needed, change Host to the one which has enough resources.

By default, Veeam stores swap file of our appliance in the NFS datastore which is just a Windows folder on the mount server. However, that is not always the case. On the picture below, you see the location of the setting which regulates where swap file is stored by default. Sometimes you may want to store it not in the VM directory, but on a specific datastore. That datastore may be full, so deployment of the FLR appliance — as well as new VMs — just fails. Check that it’s not your case.

Linux FLR appliance deploy fail

Fig 4. Host configuration

The similar issue may hit the appliance during SureBackup as well, as it’s related to the allocation of the resources.

Did you know?

All detailed information about Veeam products is provided in the help section. If you click F1 in any product window, you will be taken to the online Help Center.

Clicking F1 when working with any wizard or dialog window (including the main window) will bring up the corresponding topic of the online Help Center, opened with your default internet browser.

Linux FLR appliance deploy fail

Fig 5. Help window

In our next series

In our next series, we are going to cover the most common misconfigurations and issues we see in Veeam Support every day. Join our investigation trip and stay tuned!

The post Veeam Support troubleshooting series: Linux FLR appliance deploy fail appeared first on Veeam Software Official Blog.

Veeam Support troubleshooting series: Linux FLR appliance deploy fail

The Network. Intuitive. The Next Chapter with Peter Dinklage

Source: Cisco
It’s been just over two months since Cisco unveiled our vision for the new era of networking, The Network. Intuitive. The response has been amazing. Employees, partners, customers, and influencers are all excited about what happens when a network constantly, learns, adapts and protects. And the more people learn what Cisco’s intent-based network can do, […]The Network. Intuitive. The Next Chapter with Peter Dinklage

Cisco Does Compute

Source: Cisco
Cisco will be a Platinum sponsor at this years’ VMWorld conference occupying Booth 614 In the booth you will see the latest M5 UCS server, HyperFlex appliance, the latest analytics solutions from Cisco called Tetration and AppDynamics, the latest security and automation solutions, and Cisco’s Cloud Center offerings.  There will be demo stations associated with […]Cisco Does Compute

The VCSP opportunity with Veeam Agent for Windows

Source: Veeam

For a good part of the last 18 months of my previous role as Lead Architect at a leading Veeam Cloud & Service Provider (VCSP) partner I was involved in a project to try and come up with an easy way for our clients to perform in-guest backups of their IaaS virtual machines. At that time it involved clumsy and complex methods of performing file-level or application-aware backups to an external location. Not only were those methods problematic, they often led customers to consuming storage that wasn’t part of our own service offerings.

There had to be a better way to build out that new service offering in such a way that we could give our clients a more streamlined approach to offsite backups that also generated income for us in the form of clients using our own storage as targets for the offsite backups to be stored in. At the time, I knew about Veeam Endpoint Backup but it did not have an option to back up externally unless we exposed our Veeam Backup & Replication server over the internet, which in many ways defeated the requirement of the project for simplicity in the service offering.

Cloud Connect comes through again!

NEW Veeam Agent for Microsoft Windows 2.0 has the ability to back up directly to Veeam Cloud Connect repositories without any additional investments from the VCSP. It just works as part of any existing Cloud Connect Backup infrastructure built on Veeam Backup & Replication 9.5 Update 2. As we did with Cloud Connect Backup and then Cloud Connect Replication in resolving the complexity around getting virtual machine backups, backup copy jobs and replicas up into a cloud platform, we have now resolved the complexity of getting physical servers and workstations as well as cloud-based Windows instances into cloud environments operated by our VCSP partners.

The Availability challenge

As mentioned, there exists a challenge in being able to provide robust backup for physical servers and workstations that cannot be virtualized due to complex hardware configurations or regulatory compliance regulations. There is also a challenge in being able to provide low RPOs for users on workstations, laptops and tablets whether in corporate, remote or home offices and there is also a big challenge around being able to back up and recover Windows instances that reside in public cloud environments as well as those workloads that sit on alternative hypervisors to VMware and Hyper-V.

The agent solution

Veeam Agent for Microsoft Windows solves all those challenges listed above and would have certainly been the answer to the project in my previous role. All VCSPs running Cloud Connect can now extend their Backup as a Service (BaaS) offerings with the capabilities provided by Veeam Agent for Microsoft Windows in a combination with Veeam Backup & Replication 9.5 Update 2.

VCSPs can now offer their partners and clients Veeam Agent for Microsoft Windows licensing through the VCSP program and also provide new and existing tenants with the ability to create sub-tenants and consume storage using Cloud Connect repositories as the offsite backup target. With Veeam Agent for Microsoft Windows allowing to back up directly to Cloud Connect repositories, we have opened the way to back up offsite physical servers, workstations and endpoints as well as workloads running in Azure, AWS or any other public cloud.

The VCSP agent opportunity

Let this opportunity sink in – the ability to offer offsite backup services beyond virtual machines sitting on VMware or Hyper-V to both on-premises and remote offices, physical workloads as well as workloads residing in public clouds. The physical market opportunity is truly open for business by Veeam with the release of Veeam Agent for Microsoft Windows and VCSPs should be, if not already, looking to this new Veeam Agent offering to deliver value to customers and increase adoption of Cloud Connect Backup services.

With over one million downloads of Veeam Endpoint Backup, we are already seeing a great number of users upgrading to Veeam Agent for Microsoft Windows 2.0. With all the Veeam Endpoint Backup users upgrading to the newer version, imagine one million endpoints, which you as a VSCP can help to protect by backing up offsite to a Cloud Connect repository… think about that!

So, if you are a VCSP looking for new opportunities to offer your customers backup services around physical servers, workstations and Windows-based cloud workloads, don’t wait! Make sure you look at Veeam Agent for Microsoft Windows 2.0 and get in position to offer your partners and clients service offerings that take advantage of Veeam’s enhanced features for VCSPs in Veeam Backup & Replication 9.5 Update 2.

The post The VCSP opportunity with Veeam Agent for Windows appeared first on Veeam Software Official Blog.

The VCSP opportunity with Veeam Agent for Windows

ACI Anywhere!

Source: Cisco
Evolving beyond multi-pod and multi-site deployment models, ACI will soon be available within public cloud environments, including Amazon Web Services (AWS), Google Cloud Platform and Microsoft Azure At Cisco, we are proud that over 4,000 customers have chosen Cisco Application Centric Infrastructure (ACI), the industry’s #1 Software Defined Networking (SDN) solution, to increase operational efficiencies, […]ACI Anywhere!

Cisco’s Intent to Acquire Springpath – Taking a Leap Forward in Hyperconvergence

Source: Cisco
I am proud to share that today Cisco announced its intent to acquire Springpath, Inc., a leader in hyperconvergence software based in Sunnyvale, Calif. Cisco first made an investment in Springpath in 2015. A year later, we jointly co-engineered and developed Cisco HyperFlex Systems, a fully-integrated hyperconverged infrastructure (HCI) solution. Combining Springpath’s distributed file system […]Cisco’s Intent to Acquire Springpath – Taking a Leap Forward in Hyperconvergence

Three Musts for 21st Century Services

Source: Cisco
Remember the lonely Maytag repairman, just sitting around waiting for something to break?  Those days of break-fix service models are over. Today, technical and professional services have become central to every aspect of most companies’ operations. That’s because businesses today rely more and more on complex technologies that power digital transformation through every aspect of […]Three Musts for 21st Century Services

Why snapshots alone are not backups

Source: Veeam

Having a clear picture of what VM snapshots and backups can do for you is critical when your data is at stake. To dispel any doubts, snapshots are NOT backups. They are two different processes designed to address different needs. Today, I’m going to explain the discrepancy between VM snapshots and backups and provide you with a few scenarios where each of them best fits.

While it is true that many Veeam products will use a snapshot as part of a backup — a snapshot by itself is not a backup. This logic applies to VMware VM snapshots, Hyper-V checkpoints and storage snapshots as well.

How do snapshots work?

In a nutshell, a VM snapshot is the process of saving the data state of a VM with the possibility to revert to that point in time. The VM can be powered off, powered on or suspended when snapshots are taken. Multiple snapshots are organized in a parent-child hierarchy.

Usually, snapshots are used to test software updates or for unsafe operations on a VM, and then returned to the initial state if needed — think about it as a bookmark or an undo button. Snapshots are not a full copy of the base disk, therefore, they are not sufficient to restore a VM in case of storage failure.

VMware snapshots

In VMware VMs, the virtual disk is a .vmdk file residing on a data store (LUN). When a snapshot is created in Snapshot Manager, the original disk becomes read-only, and all the new data changes are written into a temporary .vmdk delta disk, pointing to the original one. The delta disk is the difference between the state at the moment when the snapshot was taken and the current state of the virtual disk. The process of taking a VMware snapshot also involves the creation of two additional files: snapshots and metadata information (.vmsd) and the running state information (.vmsn). After a snapshot is deleted (committed), all the changes are merged to the original .vmdk file, and it returns to read-write mode.

Why snapshots alone are not backups

Figure 1. Snapshots in VMware vSphere client

To ensure a healthy use of snapshots in vSphere virtualized environments, VMware provides several best practices:

  • Use a maximum of 32 snapshots in a chain, but for better performance, use only two to three snapshots
  • Don’t leave a snapshot running for more than 24 – 72 hours. It will increase in size, and your storage can run out of space
  • Do not use snapshots as backups. If the original virtual disk is deleted, you cannot restore a VM from snapshots

Hyper-V checkpoints

Things are slightly different with Hyper-V snapshots (renamed Hyper-V checkpoints starting with Windows Server 2012 R2). When a Hyper-V checkpoint is created in Hyper-V Manager, the running VM is paused and an .avhd(x) differencing disk is created in the same folder as the parent virtual disk (.vhd /.vhdx) to stock the changes, along with an .xml configuration file copy. The original virtual disk is set as read-only and, if the VM is running, there will be two more associated files — the VM saved state (.bin) and memory information (.vsv), then the VM is resumed.

Why snapshots alone are not backups

Figure 2. Checkpoints in Hyper-V Manager

Microsoft recommendations for Hyper-V checkpoints include:

  • Don’t use snapshots on VMs that host time-sensitive services, such as Microsoft Exchange Server or Active Directory Domain Services
  • Don’t expand existing virtual storage of a VM when there are snapshots on it, as they will get compromised
  • Use Hyper-V Manager to delete .avhd(x) files from the snapshots tree, instead of deleting them manually

What about storage snapshots?

Storage snapshots are a great framework to leverage as part of a backup job. Veeam Backup & Replication supports many storage arrays for both Backup from Storage Snapshots and Veeam Explorer for Storage Snapshots. There are a few points to cover here as well:

  • Even with a support array to leverage Veeam Explorer for Storage Snapshots, you still need to take a backup to go to different storage. Veeam Explorer for Storage Snapshots is a recovery-only technique from the source array that took the snapshot.
  • Backup from Storage Snapshots is a great way to take a backup using the power of the storage array and move the data to different storage.

One of Veeam’s evangelists, Rick Vanover, likes to say that: “We have nothing but evidence in the form of customer success stories that good arrays do indeed fail — so please take your backups onto different storage and follow the 3-2-1 Rule.”

When should I use snapshots?

Snapshots are a short-term solution to be used mostly in testing and development environments for patching, updates, or to test things quickly and rollback in case of failure. They are less recommended in production. However, there are certain scenarios where snapshots really come in handy for the production environment. For example, if you take risky actions, such as an OS update or configuration changes that could harm your system, then snapshots are a good idea.

Why aren’t snapshots recommended for production environments? Mainly for data integrity reasons. With snapshots, you’re not making a copy of the virtual hard disk. There is the VM virtual disk and the delta disk, which means that if the VM disk volume gets damaged, then your snapshots are gone as well because they can’t be merged on the base disk. Snapshots don’t protect you against disk breakdowns, and you’ll still have a single point of failure.

Another reason is performance-based. Snapshots can impact the performance of VMs. This does not happen very often – this occurs only in some particular situations, but it can happen. For example, running highly loaded VMs on aged (and thus increased in size) snapshots would definitely worsen the performance of those VMs, especially if they use dynamic disks. It is a common mistake to keep a VM running on a snapshot for a long time — the snapshot will increase in size because it will absorb all the changes, instead of the source disk. Consequently, committing will take much longer and it could even stun the VM during the merge process.

Wait, doesn’t Veeam use snapshots?

Good observation! Veeam Backup & Replication does indeed use snapshots as part of a backup job. It can use a VMware VM snapshot, a Hyper-V checkpoint or a storage snapshot. It is important to note that the snapshot by itself is not a backup – but it can be used as a critical part of the backup process. This is because the snapshot is used as part of the data movement process to a backup file or a replicated VM. The snapshot is removed when the backup job is complete.

How are backups different than snapshots?

A backup is a consistent VM copy that gives you the possibility to restore it in case the original files are compromised by a disaster or a human mistake. Unlike snapshots, backups are independent of the VM, and they can easily be exported and stored off premises (in the cloud, on tape or other remote storage). Read about the golden 3-2-1 Rule.

Veeam Backup & Replication leverages VSS technology (Volume Shadow Copy Service) and application-aware image processing to create image-level VM backups. Image-level VM backups allow you to protect an entire workload — virtual disk, operating system, software applications and system configuration files. All of those are stored in a single image-level VM backup file, which provides multiple restore options for your business-critical applications — from full VM recovery to granular, application-item recovery.

Furthermore, Veeam Backup & Replication is designed with numerous technologies for optimized backup traffic and backup file size reduction, such as deduplication, compression or WAN acceleration, and it gives you the ability to test your backup recoverability with SureBackup. Also, Veeam provides you with an alternative way to quickly test and troubleshoot your VM, the Virtual Lab. Here you can create an isolated virtual environment that doesn’t impact your production and perform different operations, like testing software updates or running trainings.

What if my environment cannot tolerate snapshots of any type?

This is a real possibility today. One way to go about the backup process is to use Veeam Agent for Microsoft Windows or Veeam Agent for Linux. These new Veeam backup products don’t use any infrastructure snapshots at all below the operating system. For Windows, the VSS framework is used to make an image-based backup, and for Linux, the veeamsnap is used to have an image of the file system.

Additionally, coming in Veeam Backup & Replication v10, the Veeam CDP capability will provide a replication engine for VMware virtual machines that doesn’t use VMware snapshots. This is leveraging the vSphere APIs for I/O Filtering or VAIO that work in the storage path of the VM.

Conclusion

VM snapshots alone cannot be used as a reliable way to protect your data and restore it in case of a failure, but they’re very handy for quick testing and troubleshooting. Additionally, a VM snapshot can be used if it is part of a comprehensive sequence of events to do a backup or a replication job. Remember, however, to keep an eye on snapshot amounts and properly manage them in order to avoid any storage and performance issues. On the other hand, image-level VM backups provide a high level of applications and data protection, allow for low RPO and support virtually any recovery scenario — from a full-VM to application-item restore.

Useful resources

The post Why snapshots alone are not backups appeared first on Veeam Software Official Blog.

Why snapshots alone are not backups