Azure DevOps tests for Maester
· 10 min read
Azure DevOps tests are now available in Maester!
Overview
Maester now includes an optional suite of Azure DevOps security and resource limit tests. The tests are bundled with Maester and are automatically discovered when you run Invoke-Maester. However, they are only executed if you have an active connection to an Azure DevOps organization using the community ADOPS PowerShell module. Without a connection, the tests are skipped.
Each test includes a markdown document with rationale and remediation guidance, and a PowerShell script containing the implementation. The tests are inspired by Learn - Azure DevOps Security Best Practices and cover a subset of the available settings.
To get started:
Install-Module Maester, ADOPS
Connect-ADOPS -Organization <your-organization>
Invoke-Maester
That's it — Maester detects the Azure DevOps connection and runs the tests automatically alongside any other configured tests.
Some tests use unsupported Azure DevOps REST API endpoints that may change without notice. These tests use the
-Forceflag internally to bypass the unsupported API warning.
Permissions
- At least a Basic access level license in Azure DevOps
- Certain tests require organization-level permissions such as "Project Collection Administrator" (e.g., AZDO.1030) or tenant-level permissions such as "Azure DevOps Administrator" (e.g., AZDO.1032-1036).
Available tests
| Test ID | Severity | Description | Link |
|---|---|---|---|
| AZDO.1000 | High | Third-party application access via OAuth should be disabled. | Learn more |
| AZDO.1001 | High | Connecting to Azure DevOps using SSH should be disabled. | Learn more |
| AZDO.1002 | High | Auditing should be enabled. | Learn more |
| AZDO.1003 | High | Public projects should be disabled. | Learn more |
| AZDO.1004 | High | Externally sourced package versions should be manually approved for internal use to prevent malicious packages from a public registry being inadvertently consumed. | Learn more |
| AZDO.1005 | High | Conditional Access Policies should be configured for Microsoft Entra ID-backed organizations. | Learn more |
| AZDO.1006 | High | External guest access to Azure DevOps should be a controlled process. | Learn more |
| AZDO.1007 | High | Access to Azure DevOps should be a controlled process managed by the IAM team or the appropriate Azure DevOps administrator roles. | Learn more |
| AZDO.1008 | Medium | Request access to Azure DevOps by email notifications to administrators should be disabled. | Learn more |
| AZDO.1009 | Info | Providing or collecting customer feedback to the product team for Azure DevOps should be enabled. | Learn more |
| AZDO.1010 | High | Audit logs should be retained according to your organization's needs and protected from purging. | Learn more |
| AZDO.1011 | Info | Azure DevOps supports up to 1,000 projects within an organization. | Learn more |
| AZDO.1012 | Info | Azure DevOps supports up to 150,000 tag definitions per organization or collection. | Learn more |
| AZDO.1013 | High | Azure DevOps organization owner should not be assigned to a regular user. | Learn more |
| AZDO.1014 | Medium | Status badges in Azure DevOps should be disabled. | Learn more |
| AZDO.1015 | High | User-defined variables should not be able to override system variables or variables not defined by the pipeline author. | Learn more |
| AZDO.1016 | High | YAML & build pipelines should have restricted access to only those repositories in the same project as the pipeline. | Learn more |
| AZDO.1017 | High | Release pipelines should have restricted access to only those repositories in the same project as the pipeline. | Learn more |
| AZDO.1018 | High | Access to repositories in YAML pipelines should apply checks and approval before granting access. | Learn more |
| AZDO.1019 | Medium | Users should not be able to skip stages defined by the pipeline author. | Learn more |
| AZDO.1020 | Medium | Creating classic build pipelines should be disabled. | Learn more |
| AZDO.1021 | Medium | Creating classic release pipelines should be disabled. | Learn more |
| AZDO.1022 | High | Azure DevOps pipelines should not automatically build on every pull request and commit from a GitHub repository. | Learn more |
| AZDO.1023 | High | Disable the ability to install and run tasks from the Marketplace, which gives you greater control over the code that executes in a pipeline. | Learn more |
| AZDO.1024 | Medium | Disable Node 6 tasks. | Learn more |
| AZDO.1025 | High | Enable Shell Task Validation to prevent code injection. | Learn more |
| AZDO.1026 | Medium | GitHub Advanced Security for Azure DevOps should be automatically enabled for new projects. | Learn more |
| AZDO.1027 | Medium | Gravatar images should not be exposed for users outside your enterprise. | Learn more |
| AZDO.1028 | Medium | Creation of Team Foundation Version Control (TFVC) repositories should be disabled. | Learn more |
| AZDO.1029 | Medium | Azure Artifacts storage limit should not be reached. | Learn more |
| AZDO.1030 | Critical | Project Collection Administrator is a highly privileged role, and membership should be restricted and regularly reviewed. | Learn more |
| AZDO.1031 | High | Validation of SSH key expiration date should be enabled. | Learn more |
| AZDO.1032 | High | Restrict creation of global Personal Access Tokens (PATs) should be enabled. | Learn more |
| AZDO.1033 | Critical | Automatic revocation of leaked Personal Access Tokens should be enabled. | Learn more |
| AZDO.1034 | High | Restrict creation of new Azure DevOps organizations should be enabled. | Learn more |
| AZDO.1035 | High | Restrict setting a maximum Personal Access Token (PAT) lifespan should be enabled. | Learn more |
| AZDO.1036 | High | Restrict creation of full-scoped Personal Access Tokens (PATs) should be enabled. | Learn more |
Quick Stats
- 🔢 37 tests in total
- 🔴 2 Critical | 🟠 22 High | 🟡 10 Medium | 🔵 3 Info
Azure DevOps Pipeline Example
For automated monitoring, the following pipeline runs Maester tests (including the Azure DevOps tests) on a schedule and publishes the results to an Azure Web App. The pipeline connects to both Microsoft Graph (for Entra ID tests) and Azure DevOps.
Prerequisites:
- An app registration with the required Maester permissions
- A workload identity federation service connection in Azure DevOps
- An Azure Web App to host the report (see Maester results on Azure Web App)
Update the following variables in the pipeline to match your environment:
| Variable | Description | Where to find it |
|---|---|---|
ServiceConnection | Name of your Azure DevOps service connection (workload identity federation) | Azure DevOps > Project Settings > Service connections |
WebAppSubscriptionId | Azure subscription ID where the Web App is hosted | Azure Portal > Subscriptions |
WebAppResourceGroup | Resource group containing the Web App | Azure Portal > Resource groups |
WebAppName | Name of the Azure Web App for the report | Azure Portal > App Services |
TenantId | Your Microsoft Entra tenant ID | Azure Portal > Microsoft Entra ID > Overview |
ClientId | Application (client) ID of the app registration | Azure Portal > App registrations > Overview |
DevOpsOrganization | Your Azure DevOps organization name (as it appears in dev.azure.com/<name>) | Azure DevOps > Organization Settings |
trigger: none
variables:
ServiceConnection: <your-service-connection>
WebAppSubscriptionId: <your-subscription-id>
WebAppResourceGroup: <your-resource-group>
WebAppName: <your-web-app-name>
TenantId: <your-tenant-id>
ClientId: <your-client-id>
DevOpsOrganization: <your-devops-organization>
schedules:
- cron: "0 6 * * *"
displayName: Daily at 06:00
always: true
branches:
include:
- main
jobs:
- job: maester
pool:
vmImage: ubuntu-latest
steps:
- checkout: self
fetchDepth: 1
- task: AzurePowerShell@5
inputs:
azureSubscription: '$(ServiceConnection)'
ScriptType: 'InlineScript'
pwsh: true
azurePowerShellVersion: latestVersion
Inline: |
# Install modules
Install-Module 'Maester', 'Pester', 'Microsoft.Graph.Authentication', 'ADOPS' -SkipPublisherCheck -Confirm:$false -Force
# Connect to Microsoft Graph
$graphToken = Get-AzAccessToken -ResourceUrl 'https://graph.microsoft.com' -AsSecureString
Connect-MgGraph -AccessToken $graphToken.Token -NoWelcome
# Connect to Azure DevOps
$DevOpsToken = ConvertFrom-SecureString -SecureString (Get-AzAccessToken -AsSecureString -TenantId '$(TenantId)').Token -AsPlainText
Connect-ADOPS -Organization '$(DevOpsOrganization)' -OAuthToken $DevOpsToken
# Prepare output folder
$date = (Get-Date).ToString("yyyyMMdd-HHmm")
$TempOutputFolder = "$PWD/temp$date"
New-Item -ItemType Directory -Force -Path $TempOutputFolder
New-Item -ItemType File -Force -Path "$TempOutputFolder/index.html"
# Install and run Maester tests
mkdir maester-tests
cd maester-tests
Install-MaesterTests .\tests
Invoke-Maester -OutputHtmlFile "$TempOutputFolder/index.html" -Verbosity Normal
# Publish to Azure Web App
$FileName = "$PWD/MaesterReport$date.zip"
Compress-Archive -Path "$TempOutputFolder/*" -DestinationPath $FileName
Select-AzSubscription -Subscription '$(WebAppSubscriptionId)'
Publish-AzWebApp -ResourceGroupName '$(WebAppResourceGroup)' -Name '$(WebAppName)' -ArchivePath $FileName -Force
displayName: 'Run Maester tests and publish report'
This pipeline only includes Microsoft Graph and Azure DevOps connections. To add Exchange Online, Teams, or Security & Compliance, see the advanced connection guide.
Documentation
- Setting up Maester in Azure DevOps Pipelines
- Maester results on Azure Web App
- Deploy Maester Web App with Bicep