Edit

Data collection rules in Azure Monitor

  • Article

Data collection rules (DCRs) are sets of instructions supporting data collection in Azure Monitor. They provide a consistent and centralized way to define and customize different data collection scenarios. Depending on the scenario, DCRs specify such details as what data should be collected, how to transform that data, and where to send it.

DCRs are stored in Azure so that you can centrally manage them. Different components of a data collection workflow will access the DCR for particular information that it requires. In some cases, you can use the Azure portal to configure data collection, and Azure Monitor will create and manage the DCR for you. Other scenarios will require you to create your own DCR. You may also choose to customize an existing DCR to meet your required functionality.

For example, the following diagram illustrates data collection for the Azure Monitor agent running on a virtual machine. In this scenario, the DCR specifies events and performance data, which the agent uses to determine what data to collect from the machine and send to Azure Monitor. Once the data is delivered, the data pipeline runs the transformation specified in the DCR to filter and modify the data and then sends the data to the specified workspace and table. DCRs for other data collection scenarios may contain different information.

Diagram that shows basic operation for DCR using Azure Monitor Agent.

Data collection in Azure Monitor

DCRs are part of a new ETL-like data collection pipeline being implemented by Azure Monitor that improves on legacy data collection methods. This process uses a common data ingestion pipeline for all data sources and provides a standard method of configuration that's more manageable and scalable than current methods. Specific advantages of the new data collection include the following:

  • Common set of destinations for different data sources.
  • Ability to apply a transformation to filter or modify incoming data before it's stored.
  • Consistent method for configuration of different data sources.
  • Scalable configuration options supporting infrastructure as code and DevOps processes.

When implementation is complete, all data collected by Azure Monitor will use the new data collection process and be managed by DCRs. Currently, only certain data collection methods support the ingestion pipeline, and they may have limited configuration options. There's no difference between data collected with the new ingestion pipeline and data collected using other methods. The data is all stored together as Logs and Metrics, supporting Azure Monitor features such as log queries, alerts, and workbooks. The only difference is in the method of collection.

View data collection rules

There are multiple ways to view the DCRs in your subscription.

To view your DCRs in the Azure portal, select Data Collection Rules under Settings on the Monitor menu.

Screenshot that shows DCRs in the Azure portal.

Select a DCR to view its details. For DCRs supporting VMs, you can view and modify its associations and the data that it collects. For other DCRs, use the JSON view to view the details of the DCR. See Create and edit data collection rules (DCRs) in Azure Monitor for details on how you can modify them.

Note

Although this view shows all DCRs in the specified subscriptions, selecting the Create button will create a data collection for Azure Monitor Agent. Similarly, this page will only allow you to modify DCRs for Azure Monitor Agent. For guidance on how to create and update DCRs for other workflows, see Create and edit data collection rules (DCRs) in Azure Monitor.

Data collection rule associations

Some data collection scenarios will use data collection rule associations (DCRAs), which associate a DCR with an object being monitored. A single object can be associated with multiple DCRs, and a single DCR can be associated with multiple objects. This allows you to manage a single DCR for a group of objects.

For example, the diagram above illustrates data collection for the Azure Monitor agent. When the agent is installed, it connects to Azure Monitor to retrieve any DCRs that are associated with it. You can create an association with to the same DCRs for multiple VMs.

Data collection scenarios

The following table describes the data collection scenarios that are currently supported using DCR and the new data ingestion pipeline. See the links in each entry for details.

Scenario Description
Virtual machines Install the Azure Monitor agent on a VM and associate it with one or more DCRs that define the events and performance data to collect from the client operating system. You can perform this configuration using the Azure portal so you don't have to directly edit the DCR.

See Collect events and performance counters from virtual machines with Azure Monitor Agent.
When you enable VM insights on a virtual machine, it deploys the Azure Monitor agent to telemetry from the VM client. The DCR is created for you automatically to collect a predefined set of performance data.

See Enable VM Insights overview.
Container insights When you enable Container insights on your Kubernetes cluster, it deploys a containerized version of the Azure Monitor agent to send logs from the cluster to a Log Analytics workspace. The DCR is created for you automatically, but you may need to modify it to customize your collection settings.

See Configure data collection in Container insights using data collection rule.
Log ingestion API The Logs ingestion API allows you to send data to a Log Analytics workspace from any REST client. The API call specifies the DCR to accept its data and specifies the DCR's endpoint. The DCR understands the structure of the incoming data, includes a transformation that ensures that the data is in the format of the target table, and specifies a workspace and table to send the transformed data.

See Logs Ingestion API in Azure Monitor.
Azure Event Hubs Send data to a Log Analytics workspace from Azure Event Hubs. The DCR defines the incoming stream and defines the transformation to format the data for its destination workspace and table.

See Tutorial: Ingest events from Azure Event Hubs into Azure Monitor Logs (Public Preview).
Workspace transformation DCR The workspace transformation DCR is a special DCR that's associated with a Log Analytics workspace and allows you to perform transformations on data being collected using other methods. You create a single DCR for the workspace and add a transformation to one or more tables. The transformation is applied to any data sent to those tables through a method that doesn't use a DCR.

See Workspace transformation DCR in Azure Monitor.

Supported regions

Data collection rules are available in all public regions where Log Analytics workspaces and the Azure Government and China clouds are supported. Air-gapped clouds aren't yet supported.

Single region data residency is a preview feature to enable storing customer data in a single region and is currently only available in the Southeast Asia Region (Singapore) of the Asia Pacific Geo and the Brazil South (Sao Paulo State) Region of the Brazil Geo. Single-region residency is enabled by default in these regions.

Data resiliency and high availability

A DCR gets created and stored in a particular region and is backed up to the paired-region within the same geography. The service is deployed to all three availability zones within the region. For this reason, it's a zone-redundant service, which further increases availability.

Next steps

See the following articles for additional information on how to work with DCRs.