Application-level monitoring for your workloads

Source: Veeam

If you haven’t noticed, Veeam ONE has really taken on an incredible amount of capabilities with the 9.5 Update 4 release.

One capability that can be a difference-maker is application-level monitoring. This is a big deal for keeping applications available and is part of a bigger Availability story. Putting this together with incredible backup capabilities from Veeam Backup & Replication, application-level monitoring can extend your Availability to the applications on the workloads where you need the most Availability. What’s more, you can combine this with actions in Veeam ONE Monitor to put in the handling you want when applications don’t behave as expected.

Let’s take a look at application-level monitoring in Veeam ONE. This capability is inside of Veeam ONE Monitor, which is my personal favorite “part” of Veeam ONE. I’ve always said with Veeam ONE, “I guarantee that Veeam ONE will tell you something about your environment that you didn’t know, but need to fix.” And with application-level monitoring, the story is stronger than ever. Let’s start with both the processes and services inside of a running virtual machine in Veeam ONE Monitor:

 

I’ve selected the SQL Server service, which for any system with this service, is likely important. Veeam ONE Monitor can use a number of handling options for this service. The first are simple start, stop and restart service options that can be passed to the service control manager. But we also can set up some alarms based on the service:

 

The alarm capability for the services being monitored will allow a very explicit handling you can provide. Additionally, you can make it match the SLA or expectation that your stakeholders have. Take how this alarm is configured, if the service is not running for 5 minutes, the alarm will be triggered as an error. I’ll get to what happens next in a moment, but this 5-minute window (which is configurable) can be what you set as a reasonable amount of time for something to go through most routine maintenance. But if this time exceeds 5 minutes, something may not be operating as expected, and chances are the service should be restarted. This is especially true if you have a fiddlesome application that constantly or even occasionally requires manual intervention. This 5-minute threshold example may even be quick enough to avoid being paged in the middle of the night! The alarm rules are shown below:

 

The alarm by itself is good, but we need more sometimes. That’s where a different Veeam ONE capability can help out with remediation actions. I frequently equate, and it’s natural to do so, the remediation actions with the base capability. So, the base capability is the application-level monitoring, but the means to the end of how to fully leverage this capability comes from the remediation actions.

With the remediation actions, the proper handling can be applied for this application. In the screenshot below, I’ve put in a specific PowerShell script that can be automatically run when the alarm is triggered. Let your ideas go crazy here, it can be as simple as restarting the service — but you also may want to notify application owners that the application was remediated if they are not using Veeam ONE. This alone may be the motivation needed to setup read-only access to the application team for their applications. The configuration to run the script to automatically resolve that alarm is shown below:

 

Another piece of intelligence regarding services, application-level monitoring in Veeam ONE will also allow you to set an alarm based on the number of services changing. For example, if one or more services are added; an alarm would be triggered. This would be a possible indicator of an unauthorized software install or possibly a ransomware service.

Don’t let your creativity stop simply at service state, that’s one example, but application-level monitoring can be used for so many other use cases. Processes for example, can have alarms built on many criteria (including resource utilization) as shown below:

 

If we look closer at the process CPU, we can see that alarms can be built on if a process CPU usage (as well as other metrics) go beyond specified thresholds. As in the previous example, we can also put in handling with remediation actions to sort the situation based on pre-defined conditions. These warning and error thresholds are shown below:

 

As you can see, application-level monitoring used in conjunction with other new Veeam ONE capabilities can really set the bar high for MORE Availability. The backup, the application and more can be looked after with the exact amount of care you want to provide. Have you seen this new capability in Veeam ONE? If you haven’t, check it out!

You can find more information on Veeam Availability Suite 9.5 Update 4 here.

More on new Veeam ONE capabilities:

The post Application-level monitoring for your workloads appeared first on Veeam Software Official Blog.


Application-level monitoring for your workloads

Backup infrastructure at your fingertips with Heatmaps

Source: Veeam

One of the best things an organization can do is have a well-performing backup infrastructure. This is usually done by fine-tuning backup proxies, sizing repositories, having specific conversations with business stakeholders about backup windows and more. Getting that set up and running is a great milestone, but there is a problem. Things change. Workloads grow, new workloads are introduced, storage consumption increases and more challenges come into the mix every day.

Veeam Availability Suite 9.5 Update 4 introduced a new capability that can help organizations adjust to the changes:

Heatmaps!

Heatmaps are part of Veeam ONE Reporter and do an outstanding job of giving an at-a-glance view of the backup infrastructure that can help you quickly view if the environment is performing both as expected AND as you designed it. Let’s dig into the new heatmaps.

The heatmaps are available on Veeam ONE Reporter in the web user interface and are very easy to get started. In the course of showing heatmaps, I’m going to show you two different environments. One that I’ve intentionally set to be performing in a non-optimized fashion and one that is in good shape and balanced so that the visual element of the heatmap can be seen easily.

Let’s first look at the heatmap of the environment that is well balanced:

Here you can see a number of things, the repositories are getting a bit low on free space, including one that is rather small. The proxies carry a nice green color scheme and do not show too much variation in their work during their backup windows. Conversely if we see a backup proxy is dark green, that indicates it is not in use, which is not a good thing.

We can click on the backup proxies to get a much more detailed view, and you can see that the proxy has a small amount of work during the backup window in this environment in the mid-day timeframe and carries a 50% busy load:

When we look at the environment that is not so balanced, the proxies tell a different story:

You can see that first of all there are three proxies, but one of them seems to be doing much more work than the rest due to the color changes. This clearly tells me the proxies are not balanced, and this proxy selected is doing a lot more work than the others during the overnight backup window — which stretches out the backup window.

One of the coolest parts of the heatmap capability is that we can drill into a timeframe in the grid (this timeline can have a set observation window) that will tell us which backup jobs are causing the proxies to be so busy during this time, shown below:

In the details of the proxy usage, you can see the specific jobs that are set which are taking the CPU cycles are shown.

How can this help me tune my environment?

This is very useful as it may indicate a number of things, such as backup jobs being configured to not use the correct proxies, proxies not having the connectivity they need to perform the correct type of backup job. An example of this would be if one or more proxies are configured for only Hot-Add mode and they are physical machines, which makes that impossible. The proxy would never be selected for a job and the remaining proxies would be in charge of doing the backup job. This is all visible in the heatmap yet the backup jobs would complete successfully, but this type of situation would extend the backup window. How cool is that?

Beyond proxy usage, repositories are also very well reported with the heatmaps. This includes the Scale-Out Backup Repositories as well. This will allow you to view the underlying storage free space. The following animation will show this in action:

Show me the heatmaps!

As you can see, the heatmaps add an incredible visibility element to your backup infrastructure. You can see how it is performing, including if things are successful yet not as expected. You can find more information on Veeam Availability Suite 9.5 Update 4 here.

 

Helpful resources:

The post Backup infrastructure at your fingertips with Heatmaps appeared first on Veeam Software Official Blog.


Backup infrastructure at your fingertips with Heatmaps

New era of Intelligent Diagnostics

Source: Veeam

Veeam Availability Suite 9.5 U4 is full of fantastic new features to enable Veeam customers to intelligently manage their data like never before. A big component of Veeam Availability Suite 9.5 U4 is of course, Veeam ONE.

Veeam ONE is all about providing unprecedented visibility into customers’ IT environments, and this update is full of fantastic enhancements, so stay tuned for more blogs all about new features like Veeam Agent monitoring and reporting enactments, Heatmaps, and more!

One of the most interesting new features of Veeam ONE 9.5 U4 is Veeam Intelligent Diagnostics. Think of Veeam Intelligent Diagnostics as the entity watching over your Veeam Backup & Replication environment so you don’t have to. Wouldn’t it be great for potential issues to fix themselves before they cause problems? Veeam Intelligent Diagnostics enables just that.

How it works

Veeam Intelligent Diagnostics works by comparing the logs from your Veeam Backup & Replication environment to a known list of issue signatures. These signatures are downloaded from Veeam directly, so think of it as a reverse call home. All log parsing and analysis is done by Veeam ONE with the help of a Veeam Intelligent Diagnostics agent which is installed on every Veeam Backup & Replication server you want to keep an eye on.

These signatures can easily be updated by Veeam Support as needed. This allows Veeam Support to be proactive if they are noticing a number of cases with the same characteristics or cause, and fix issues before customers even encounter them. Think of things like common misconfigurations that Veeam customers spend time troubleshooting. These items can easily be avoided by Veeam customers leveraging Veeam Intelligent Diagnostics.

When an issue is detected, Veeam Intelligent Diagnostics can fix things in one of two ways, automatically or semi-automatically. The semi-automatic method requires manual approval to remediate the issue. Veeam calls these fixes Remediation Actions. In either case, Veeam ONE alarms will be triggered when an issue is detected. Veeam Intelligent Diagnostics will also include handy Veeam Knowledge Base articles in the alarms to allow customers to understand the issues they are avoiding.

Management made easier

One of the things that has always attracted me to Veeam products is how easy they are to use and get started with. Veeam Intelligent Diagnostics takes things a step further by eliminating potential issues before they even cause problems, making Veeam Backup & Replication easier to use and manage. Reducing operational complexity is a win for any organization, and Veeam Intelligent Diagnostics helps do this.

The Veeam Availability Suite, which is made up of Veeam Backup & Replication and Veeam ONE is available for a completely free 30-day trial. Be sure to try it out to take advantage of Veeam Intelligent Diagnostics and the rest of the powerful features released in Veeam Availability Suite 9.5 U4.

The post New era of Intelligent Diagnostics appeared first on Veeam Software Official Blog.


New era of Intelligent Diagnostics

Smart Home Devices

Source: SANS security tip
Now adays most of us have numerous devices in our homes connect to the Internet. From thermostats and gaming consoles to baby monitors, door locks or even your car. Ensure you change the default passwords on these devices and enable automatic updating.
Smart Home Devices

Tips on managing, testing and recovering your backups and replicas

Source: Veeam

Several months ago, I wrote a blog post about some common and easily avoidable misconfigurations that we, support engineers, keep seeing in customers’ infrastructures. It was met with much attention and hopefully helped administrators improve their Veeam Backup & Replication setups. In this blog post, I would like to share with you several other important topics. I invite you to reflect on them to make your experience with Veeam even smoother.

Backups are only half of the deal — think about restores!

Every now and then we get calls from customers who catch themselves in a very bad situation. They needed a restore, but at a certain point hit an obstacle they could not circumvent. And I’m not talking about lost backups, CryptoLocker or something! It’s just that their focus was on creating a backup or replica. They never considered that data recovery is a whole different process that must be examined and tested separately. I’ll give you several examples to get the taste of it:

  1. The customer had a critical 20-terabyte VM that failed. Nobody wants downtime, so they started the VM in instant recovery and had it working in five minutes. However, instant recovery is a temporary state and must be finalized by migration to the production datastore. As it turned out, the infrastructure did not allow it to copy 20 TB of data in any reasonable time. And since instant recovery was started with an option to write changes to the C: drive of Veeam Backup & Replication (as opposed to using a vSphere snapshot), it was quickly filling up without any possibility for sufficient extension. As some time passed before the customer approached support, the VM had already accumulated some changes that could not be discarded. With critical data at risk, there’s no way to finalize instant recovery in a sufficiently short time and imminent failure approaching. Quite a pickle, huh?
  2. The customer had a single domain controller in the infrastructure and everything added in Veeam Backup & Replication using DNS. I know, I know. It could have gone wrong in a hundred ways, but here is what happened: The customer planned some maintenance and decided to fail over to the replica of that DC. They used planned failover, which is ideal for such situations. The first phase went fine, however during the second phase, the original VM was turned off to transfer the last bits of data. Of course, at that moment the job failed because DNS went down. Luckily, here we could simply turn on the replica VM manually from vSphere (this is not something we recommend, see the next advice). However, it disrupted and delayed the maintenance process. Plus, we had to manually add host names to the C:WindowsSystem32driversetchosts file on Veeam Backup & Replication to allow a proper failback.
  3. The customer based backup infrastructure around tapes and maintained only a very short backup chain on disks. When they had to restore some guest files from a large file server, it turned out there was simply not sufficient space to be found on any machine to act as a staging repository.

I think in all these situations the clients fell into the same trap they simply assumed that if a backup is successful, then restore should be as well! Learn about restore, just as you learn about backups. A good way to start is our user guide. This section contains information on all the major types of restores. In the “Before you begin” section of each restore option, you can find initial considerations and prerequisites. Information on other types of restores such as restore from tapes or from storage snapshots can be found in their respective sections. Apart from the main user guide, be sure to check out the Veeam Explorers guide too. Each Veeam Explorer has a “Planning and preparation” section — this will help you prepare your system for restore beforehand.

Do not manage replicas from vSphere console

Veeam replicas are essentially normal virtual machines. As such, they can be managed using usual vSphere management tools, mainly vSphere client. It can, but should not be used. Replica failover in Veeam Backup & Replication is a sophisticated process, which allows you to carefully go one step at a time (with the possibility to roll back if something goes wrong) and finalize failover in a proper way. Take a look at the scheme below:

If instead of using the Veeam Backup & Replication console, you simply start a replica in vSphere client or start a failover from Veeam Backup & Replication. But if you switch to managing from the vSphere client later, you get a number of serious consequences:

  1. The failover mechanism in Veeam Backup & Replication will no longer be usable for this VM, as all that flexibility described above will no longer be available.
  2. You will have data in the Veeam Backup & Replication database that does not represent the actual state of the VM. In worst cases, fixing it requires database edits.
  3. You can lose data. Consider this example: A customer started a replica manually in vSphere client and decided to simply stick with it. Some time passed, and they noticed that the replica was still present in the Veeam Backup & Replication console. The customer decided to clean it up a little, right-clicked on the replica and chose “Delete from disk.” Veeam Backup & Replication did exactly what was told — deleted the replica, which unbeknownst to the software, had become a production VM with data.

There are situations when starting the replicas from the vSphere client is necessary (mainly, if the Veeam Backup & Replication server is down as well and replicas must be started without delay). However, if the Veeam Backup & Replication server is operational, it should be the management point from start to finish.

It is also not recommended to delete the replica VMs from vSphere client. Veeam Backup & Replication will not be aware of such changes, which can lead to failures and stale data in the console. If you do not need a replica anymore, delete it from the console and not from the vSphere client as a VM. That way your list of replicas will contain only the actual data.

Careful with updates!

I’m speaking about updates for hypervisors and various applications backed up by Veeam. From a Veeam Backup & Replication perspective, such updates can be roughly divided into two categories — major updates that bring a lot of changes and minor updates.

Let’s speak about major updates first. The most important ones are hypervisor updates. Before installing them, it is necessary to confirm that Veeam Backup & Replication supports them. These updates bring a lot of changes to the libraries and APIs that Veeam Backup & Replication uses, so updating Veeam Backup & Replication code and rigorous testing from QA is necessary before a new version is officially supported. Unfortunately, as of now VMware does not provide any preliminary access to the new vSphere versions for the vendors. So Veeam’s R&D gets access together with the rest of the world, which means that there is always a lag between a new version release and official support. The magnitude of changes also does not allow R&D to fit everything in a hotfix, so official support is typically added with the new Veeam Backup & Replication versions. This puts support and our customers in a tricky situation. Usually after a new vSphere release, the amount of cases increases because administrators start installing updates, only to find out that their backups are failing with weird issues. This forces us, support, to ask the customers to perform a rollback (if possible) or to propose workarounds that we cannot officially support, due to lack of testing. So please check the version compatibility before updates!

The same applies to backed up applications. Veeam Explorers also has a list of supported versions and new versions are added to this list with Veeam Backup & Replication updates. So once again, be sure to check the Veeam Explorers user guide before passing to a new version.

In the minor updates’ category, I put things like cumulative updates for Exchange, new VMware Tools versions, security updates for vSphere, etc. Typically, they do not contain major changes and in most situations Veeam Backup & Replication does not experience any issues. That’s why QA does not release official statements as with major updates. However, in our experience there were situations where minor updates changed workflow enough to cause issues with Veeam Backup & Replication. In these cases, once the presence of an issue is confirmed, R&D develops a hotfix as soon as possible.

How should you stay up to date on the recent developments? My advice is to register on https://forums.veeam.com/. You will be subscribed to a weekly “Word from Gostev” newsletter from our Senior Vice President Anton Gostev. It contains information on discovered issues (and not limited to Veeam products), release plans and interesting IT news. If you do not find what you are looking for in the newsletter, I recommend checking the forum. Due to the sheer number of Veeam clients, if any update breaks something, a related thread appears soon after.

Now backups are not the only thing that patches and updates can break. In reality, they can break a lot of stuff, the application itself included. And here Veeam has something to offer — Veeam DataLabs. Maybe you heard about SureBackup — our ultimate tool for verifying the consistency of backups. SureBackup is based on DataLabs, which allows you to create an isolated environment where you can test updates, before bringing them to production. If you want to save yourself some gray hair, be sure to check it out. I recommend starting with this post.

Advice to those planning to buy Veeam Backup & Replication or switching from another solution

Sometimes in technical support we get cases that go like this: “We have designed our backup strategy like this, we acquired Veeam Backup & Replication, however we can’t seem to find a way to do X. Can you help with it?” (Most commonly such requests are about unusual retention policies or tape management). We are happy to help, but at times we have to explain that Veeam Backup & Replication works differently and they will need to change their design. Sure enough, customers are not happy to hear that. However, I believe they are following an incorrect approach.

Veeam Backup & Replication is very robust and flexible and in its current form it can satisfy the absolute majority of the companies. But it is important to understand that it was designed with certain ideas in mind and to make the product really shine, it is necessary to follow these ideas. Unfortunately, sometimes the reality is quite different. Here is what I imagine happens with some of the customers: They decide that they need a backup solution. So they sit down in a room and meticulously design each element of their strategy. Once done, they move to choosing a backup solution, where Veeam Backup & Replication seems to be an obvious choice. In another scenario, the customer already has a backup solution and a developed backup strategy. However, for some reason their solution does not meet their expectations. So they decide to switch to Veeam and wish to carry their backup strategy in Veeam Backup & Replication unchanged. My firm belief is that this process should go vice versa.

These days Veeam Backup & Replication has become a de-facto standard for backup solution, so probably any administrator would like to have a glance at it. However, if you are serious about implementation, Veeam Backup & Replication needs to be studied and tested. Once you know the capabilities and know this is what you are looking for, build your backup strategy specifically for Veeam Backup & Replication. You will be able to use the functionality to the maximum, reduce the risks and support will have an easier time understanding the setup.

And that’s what I have for today’s episode. I hope that gave you something to consider.

 

The post Tips on managing, testing and recovering your backups and replicas appeared first on Veeam Software Official Blog.


Tips on managing, testing and recovering your backups and replicas