Post last updated: June 3, 2020 (minor name updates May 2022)

This post discusses two methods of receiving an alert when a tenant setting has changed in the Power BI Service: one using Microsoft Defender for Cloud Apps (formerly called Cloud App Security), and the other using the M365 Security & Compliance Center.

Tenant settings in the Power BI Service are among the most important things to get right. Once they are set the way you want, the objective is to ensure all changes are well-controlled.

There are two aspects, used together, for managing tenant settings:

Activity log data. The ‘UpdatedAdminFeatureSwitch’ activity in the Power BI activity log (originating from the M365 unified audit log) indicates when a tenant setting change has occurred. Tracking this is important from a logging and auditing perspective.
Alerts. Alerting of when a tenant setting change has occurred is a good companion to #1, as it can help you know within a couple of hours when something changed.

Also, I recommend documenting your tenant settings for the broader Power BI user community for the purpose of:

Documenting what can and cannot be done. This helps users understand when something doesn’t work and the reasoning/rationale for it, versus what they might be reading online.
Documenting what group to request to be a member of if a user wants to be able to perform some activity.

Receiving an Alert When a Tenant Setting Changes

Without rolling your own alerting or using a 3rd party product, I’m aware of two ways to get alerted when a specific audit log entry occurs:

Microsoft Defender for Cloud Apps (formerly Cloud App Security). Discussed in the first section below.
M365 Security & Compliance Center. Discussed in the second section below.

Note that using Microsoft Defender for Cloud Apps does require some separate licensing. Both options require permissions completely independent from Power BI administration.

Using Microsoft Defender for Cloud Apps for Alerting When a Power BI Tenant Setting Changes

Microsoft Defender for Cloud Apps (formerly called Microsoft Cloud App Security) is a CASB - a Cloud Access Security Broker. Defender for Cloud Apps does several things to help secure and oversee applications, both Microsoft and non-Microsoft, like finding suspicious or atypical activities, and tracking cloud app activities. For purposes of this post, we are focused on its functionality to manage cloud applications with policies.

Screenshot of policies in Cloud App Security

The following screenshot shows the types of things you can configure for an activity policy:

Screenshot of activity policy options in Cloud App Security

Note in the above screenshot that we have specified the App as being Power BI, and that Administrative Activity is true.

Also, as you can see above, you don’t have to send an e-mail alert. You could instead do a text message or even incorporate a flow in Power Automate.

Here is an example of the e-mail alert that you will receive from Defender for Cloud Apps:

Screenshot of the e-mail received from the activity policy in Cloud App Security

The e-mail alert does tell you which property changed, by whom, and when.

I do prefer using Defender for Cloud Apps. However, it does require licensing which you may not have. Therefore, let’s discuss another secondary option next.

Using Microsoft Security & Compliance Center for Alerting When a Power BI Tenant Setting Changes

At the time of this writing (early June 2020 — and it’s still true mid-2022), a conversion is still happening from “activity alerts” to “alert policies.” Unfortunately, the new alert policies do not support Power BI audit log activities yet. Therefore, this section does show using the older activity alerts which will become deprecated at some point, in order for this functionality to work. URLs for reference:

Newer alert policies: https://protection.office.com/alertpolicies
Older activity alerts: https://protection.office.com/managealerts (we’re using this one)

Disadvantages of this approach:

Need to use the deprecated activity alerts to get the activity we want. (I’m guessing this one is just a matter of time.)
Recipients can only be a user - not a group. (The newer alert policies do accept groups, so hopefully this limitation will be removed at some point.)
The alerts in the Security & Compliance Center currently do basic e-mail alerting only. It doesn’t integrate with Power Automate like Defender for Cloud Apps does.
The e-mail received from the Security & Compliance Center aren’t quite as nice as what’s received from Defender for Cloud Apps. (Probably also just a matter of time.)

The following screenshot shows the types of things you can configure for an activity alert in the Security & Compliance Center:

Screenshot of an activity alert from the Security & Compliance Center

Note in the above activity alert, we searched for the “Updated organization’s Power BI settings” activity.

Here is an example of the e-mail alert that you will receive from the Security & Compliance Center:

Screenshot of email alert received from the Security & Compliance Center

Limitations for Both Options Discussed Above

The alert is not real-time (the unified audit log isn’t either). It’ll take an hour or two, sometimes more, for the e-mail to arrive.
We do *not* have information about the before/after setting (neither does the unified audit log). Nor do we know which group(s) were added or removed. Basically, we just know that something changed and a save occurred. This is another reason to have your tenant settings well-documented…had to say it!

Even with the limitations, I think it’s very helpful to have a heads-up that something changed. Just in case.