This jumpstart deploys an end-to-end On-Premises Data Gateway observability stack into your Microsoft Fabric workspace β two Eventstreams that receive heartbeat and report telemetry from a PowerShell uploader running on the gateway host, an Eventhouse-backed Gateway Monitoring KQL database that durably stores every event, and a DirectQuery Power BI semantic model + report with six built-in pages (Jobs, Job Details, Queries, Running Jobs, Gateways, System Counters) for live operational insight.
β±οΈ Deploy time: ~2 minutes. Plan another ~5 minutes inside Fabric for the walkthrough β plus a separate one-time install of the PowerShell uploader on each On-Premises Data Gateway machine you want to monitor.
What Gets Deployed
| Item | Type | Role |
|---|---|---|
Gateway Monitoring Setup | Notebook | The entry point. Installs ms-fabric-cli, looks up the two Eventstreams' CustomEndpoint source connection strings via the Fabric CLI, and writes ./builtin/config.json β the configuration file the gateway-side PowerShell uploader consumes. Six user-editable variables seed the config: HeartbeatEnable, HeartbeatInterval, ReportSendInterval, ReportRetention, TenatId, AppId. |
GatewayMonitoringHeartbeat | Eventstream | CustomEndpoint source named Heartbeat. Single destination: the Eventhouse, writing to the GatewaysHeartbeat table via DirectIngestion with mapping GatewaysHeartbeat_mapping. |
GatewayMonitoringReports | Eventstream | CustomEndpoint source named Reports. Single destination: the Eventhouse, writing to the GatewayReports-Raw table via DirectIngestion with mapping GatewayReports-Raw_mapping. |
Fabric Platform Monitoring | Eventhouse | Container for the KQL database below. |
Gateway Monitoring | KQLDatabase | All telemetry lives here. GatewaysHeartbeat and GatewayReports-Raw are the ingest tables; downstream KQL functions and views (GatewayNodeStatusWithInfo, QueryExecutionUnified, QueryConnections, SystemCounters) are what the semantic model queries. |
Gateway Monitoring | SemanticModel | DirectQuery Power BI model over the KQL database. Six query-group parameters β KustoURL, KustoDB, DaysBack, DateToSearch, SelectedGateways, SelectedClusters β drive four DirectQuery tables (Gateway Node Info, Query Execution, Query Connections, System Counters) plus import tables for Calendar, Period, Execution Measures, and the counter measure parameter. Execution Measures contains 18 measures for jobs, queries, durations, and disk I/O. |
Gateway Monitoring | Report | Power BI report bound to the semantic model. Six pages: Jobs, Job Details (drillthrough), Queries, Running Jobs, Gateways, System Counters. Page refresh is enabled on Jobs, Queries, and System Counters at 5 minutes, Running Jobs at 10 seconds, and Gateways at 30 seconds. |
How It Works
Setup β generating the gateway config
The Setup notebook is the entry point. It does five things:
%pip install ms-fabric-cli --quiet.- Grabs a
pbitoken fromnotebookutils.credentialsand exports it asFAB_TOKEN/FAB_TOKEN_ONELAKEso the Fabric CLI (fab) inherits the caller's identity. - Resolves the IDs of
GatewayMonitoringHeartbeat.EventstreamandGatewayMonitoringReports.Eventstreamin the current workspace. - Calls the Fabric REST API (
/workspaces/{id}/eventstreams/{id}/topologyand/sources/{sid}/connection) for each stream to read eachCustomEndpointsource'sprimaryConnectionString(an EventHub-compatible connection string). - Writes
./builtin/config.jsonβ aGatewayLogUploadConfigdocument containing the two connection strings, yourHeartbeatEnable / HeartbeatInterval / ReportSendInterval / ReportRetentionchoices, a service-principal block (TennatId,AppId,SecretText), and a hard-coded gateway log path (C:\Windows\ServiceProfiles\PBIEgwService\AppData\Local\Microsoft\On-premises data gateway).
Service principal requirements
The uploader authenticates to Fabric with a service principal that must satisfy all three conditions called out in the setup notebook:
- Fabric Admin API access (tenant-level setting).
- Member of the Fabric workspace that holds this accelerator.
- Administrator of the On-Premises Data Gateway itself.
AppId goes into config.json. SecretText is left empty in the generated file β you populate it on the gateway machine, never check it in.
Runtime β gateway uploader to Eventstreams
The PowerShell uploader runs on the gateway host (under an identity that can read C:\Windows\ServiceProfiles\PBIEgwService\AppData\Local\Microsoft\On-premises data gateway). On each tick:
- Every
HeartbeatIntervalseconds it pushes a heartbeat message to theGatewayMonitoringHeartbeatEventstream'sHeartbeatCustomEndpoint. - Every
ReportSendIntervalseconds it tails the gateway log files, batches reports, and pushes them to theGatewayMonitoringReportsEventstream'sReportsCustomEndpoint. - Reports older than
ReportRetentiondays are aged out locally.
Ingest β Eventstream to Eventhouse
Each Eventstream has a single Eventhouse destination configured for DirectIngestion. Heartbeats land in Fabric Platform Monitoring β Gateway Monitoring β GatewaysHeartbeat within seconds; reports land in GatewayReports-Raw. Update policies and KQL functions inside the database derive the curated views the semantic model consumes (GatewayNodeStatusWithInfo, QueryExecutionUnified, QueryConnections, SystemCounters).
Consume β the Power BI semantic model and report
The Gateway Monitoring semantic model uses DirectQuery against the KQL DB over the KustoURL parameter. Four DirectQuery tables map to KQL views / functions:
| Table | Source | What's in it |
|---|---|---|
Gateway Node Info | View GatewayNodeStatusWithInfo | One row per gateway node β cluster, machine, OS, cores, memory, status, last-seen times. |
Query Execution | Function QueryExecutionUnified(DateToSearch, DaysBack, ...) | Per-activity execution detail β start/end, duration, success, query text, spooling I/O, data source. |
Query Connections | Function QueryConnections(DateToSearch, DaysBack, ...) | Per-connection metadata β data source path joined by QueryTrackingId to Query Execution. |
System Counters | Function SystemCounters(DateToSearch, DaysBack, ...) | Per-counter min/avg/max aggregations β including the divide-by-cores CPU correction in Avg./Max/Min Counter Value measures. |
Calendar | Static M-query | Standard date dimension for time-intelligence. |
Execution Measures ships 18 measures: # Activities, # Nodes, # Queries, # Queries (DirectQuery), # Queries (Refresh), # Queries (success), # Queries (errors), # Queries (running), # Jobs, Job Success, Duration (s), Perc. 0.9 Query Duration (s), Max. Query Duration (s), Query Avg. Execution - DirectQuery (ms), Query Avg. Execution - Refresh (ms), Disk Read Avg. (byte/sec), Disk Write Avg. (byte/sec), Workspace ID.
The report has six pages:
- Jobs β landing page, auto-refresh every 5 minutes, slicer-driven day range.
- Job Details β drillthrough target. Select a row in Jobs and click Get Job Details to navigate filtered by
RootActivityId. - Queries β per-query detail across the chosen window.
- Running Jobs β jobs whose success status is "R" (computed by the
Job Successmeasure). - Gateways β node-level inventory from
Gateway Node Info. - System Counters β counter trends;
gatewaycpupercentis auto-divided byLogical Coresso values are normalized.
Try It β A 5-Minute Walkthrough
- Install β
jumpstart.install("fpm-gateway-monitoring")and wait for the ~2-minute deploy. - Generate the config β open
Gateway Monitoring Setup, setHeartbeatInterval,ReportSendInterval,ReportRetention,TenatId,AppIdin cell 2, then run all cells. The notebook writes./builtin/config.jsonwith the two Eventstream connection strings embedded. - Install the uploader on the gateway β copy
config.jsonto the On-Premises Data Gateway machine, fill inServicePrincipal.SecretText, and start the PowerShell uploader as documented in the setup notebook's checklist. - Watch ingestion β back in Fabric, open the
Gateway MonitoringKQL database and runGatewaysHeartbeat | top 10 by ingestion_time() descand['GatewayReports-Raw'] | countto confirm rows are landing. - Open the report β open the
Gateway MonitoringPower BI report. The Jobs page auto-refreshes every 5 minutes. Use the Days Back slicer to widen the window if you've just started ingesting, and verify the semantic model'sKustoURLparameter if the visuals do not show your newly ingested rows. - Drill through β select any row in the Jobs visual and click Get Job Details for the per-
RootActivityIdview.
Extending the Accelerator
| Extension | How |
|---|---|
| Monitor more than one gateway cluster | Install the PowerShell uploader on each gateway machine with the same config.json β GatewayObjectId discriminates nodes in the data. Gateway Node Info and the SelectedGateways / SelectedClusters slicers light up automatically. |
| Tune ingestion cadence | Re-run Gateway Monitoring Setup with different HeartbeatInterval / ReportSendInterval values and redeploy config.json to the gateway host. The defaults are 1 s heartbeat / 5 s reports / 10-day retention. |
| Alert on gateway health | Add an Activator (Reflex) on a KQL query that fires when GatewaysHeartbeat rows stop arriving for a node, or when # Queries (errors) crosses a threshold over the last hour. |
| Build a custom dashboard | Clone the report or build a KQL Dashboard directly on Gateway Monitoring β QueryExecutionUnified, GatewayNodeStatusWithInfo, and SystemCounters are the three workhorse query functions. |
| Ship telemetry elsewhere | Add a second destination to either Eventstream (Lakehouse, Custom Endpoint, Activator) without changing the gateway-side uploader. |
| Stream verbose logs too | The generated config.json includes a VerboseLogSendInterval: 600 knob and a Lakehouse block (UploadReports, UploadLogs, WorkspaceName, LakehouseName) β set them to also archive raw log files into a Lakehouse for forensic analysis. |
Cost Notes
The semantic model uses DirectQuery, so every report page render hits the Eventhouse. Several pages also have automatic refresh enabled: Jobs, Queries, and System Counters refresh every 5 minutes, Running Jobs refreshes every 10 seconds, and Gateways refreshes every 30 seconds. If many users will leave the report open, consider switching to a scheduled-refresh import model on the high-traffic pages.
Requirements
- A Microsoft Fabric workspace with at least F2 capacity (Eventstream + Eventhouse + Real-Time Dashboard require it).
- Permissions to create Notebook, Eventstream, Eventhouse, KQL Database, Semantic Model, and Report items.
- A service principal with: Fabric Admin API access; Member of the deployment workspace; Administrator of the On-Premises Data Gateway being monitored.
- Access to the On-Premises Data Gateway machine β to install the PowerShell uploader, drop
config.json(and the SPSecretText) on disk, and run the uploader under an identity that can readC:\Windows\ServiceProfiles\PBIEgwService\AppData\Local\Microsoft\On-premises data gateway. - Network egress from the gateway host to the EventHub-compatible endpoint URIs surfaced by each Eventstream's CustomEndpoint.
Resources
- Source folder β ecotte/fabric-platform-monitoring @ v0.2.3 (gateway-monitoring)
- Fabric Platform Monitoring landing page
- On-premises data gateway β overview
- On-premises data gateway β service principal admin
- Fabric Eventstream β CustomEndpoint source
- Eventhouse β Direct Ingestion from Eventstream
- On-premises data gateway monitoring β CPU counters (referenced by the System Counters measure)