Operations

Running the platform end-to-end — build and run automation, observability, and the closed loop of collect, correlate, remediate.

Operations is the domain of keeping things running: build/run automation, observability, the closed loop that turns telemetry into remediation, plus the MLOps model stack and the Day-2 lifecycle. It spans roughly twenty-five years, and its oldest discipline is still its core one. Model-driven network management and the collect→correlate→remediate loop, inherited from OSS/BSS and TMN, are exactly what AIOps is today — the vocabulary changed more than the problem did. As with the other domains, what follows leads with the newest work.

Convergence

Operating the platform now means running GPU-dense AI infrastructure as the default workload. That means standing up and scheduling clusters through control planes such as BCM, Slurm, and Kubernetes, sharpened by GPU-aware orchestration and fractional scheduling (Run:ai). It is inseparable from serving the models: inference stacks such as Triton, vLLM, and NIM, optimized down to TensorRT.

Day-2 discipline has shifted from dashboards to declarative closed loops. Intent-based networking, Nephio, and O-RAN’s O2 interface pushed GitOps-style reconciliation into the telecom edge, while fleet governance (Red Hat ACM) and self-healing controllers made declare the desired state and let it converge the de-facto operating model.

Observability remains the non-negotiable substrate — metrics, logs, and traces feeding OpenTelemetry and Prometheus — but it increasingly feeds ML-driven operations, where anomaly detection, failure prediction, and digital-twin simulation mature the long-standing AIOps ambition of proactive, self-remediating systems. That is now fronted by LLM copilots such as OpenShift Lightspeed for natural-language Day-2 troubleshooting. Multi-tenancy is assumed as the baseline for sharing expensive infrastructure, while telecom operating frameworks (TM Forum eTOM and Open APIs) persist as standardizing scaffolding beneath the newer cloud-native automation.

Transition to AI

Operating the platform meant declaring it rather than touching it. The de-facto backbone became Git-driven, version-controlled infrastructure: infrastructure as code (Terraform, Pulumi, Crossplane) and GitOps reconciliation loops made the repo the single source of truth, while zero-touch provisioning pushed the same hands-off, first-boot automation down to switches and routers in the field.

The hardest work carried that discipline into carrier-grade domains, where telco-cloud architecture demanded distributed multi-site footprints, deterministic low-latency networking, and hardware acceleration (SR-IOV, DPDK, SmartNICs) that ordinary IT clouds never had to solve. A parallel model stack matured fast: MLOps brought DevOps-style CI/CD, monitoring, and retraining to production models, with MLflow handling experiment tracking and registry-based promotion.

Closed-loop assurance

The platform ran as a closed-loop assurance engine. AIOps sat at the centre, applying machine learning over telemetry, logs, and alarm streams to collapse alert noise into root-caused incidents and drive automated remediation.

In the telco domain this loop was concretized by ONAP: its DCAE analytics subsystem collected VNF and PNF events and threshold-crossings, then triggered policy-driven control across the OSS/BSS stack. Operating end-to-end meant wiring orchestration of the underlying services and containers into ServiceNow’s ITSM workflows, so that detection, ticketing, and fulfillment moved as one — while longer-horizon work quietly retired legacy systems onto these modern platforms.

Cloud-native operations

The platform was operated as Kubernetes-native declarative infrastructure. The centre of gravity was the operator pattern: CRDs, admission webhooks, reconciliation controllers, and operator frameworks (Operator SDK, Kubebuilder) that encode install, upgrade, and heal knowledge as code — extended to the clusters themselves through Cluster API and kubeadm.

Delivery ran on GitOps and Kubernetes-native pipelines: Argo CD reconciling to Git, Helm and Kustomize for packaged manifests, Tekton for containerized builds, atop Ansible for agentless, idempotent configuration management. Run-time operations paired Prometheus-led observability with production incident management, distributed-systems debugging, and multi-tenant Day-2 support, closing the loop from metric to page to reconciled fix. On the telco edge, the same machinery carried the NFV MANO stack and its TOSCA descriptors, plus the ONAP/ECOMP orchestration lineage, into containerized clouds via OpenStack-Helm. MLOps and multi-cluster threads (Kubeflow, Anthos) were the early signals of where model-serving and fleet ops would head next.

Distributed systems

Operating the platform meant the DevOps convergence of development and operations taking hold. Fleets were kept in a declared desired state by pull-based configuration management such as Puppet, while lifecycle management matured into automated onboarding, healing, upgrade, and scaling atop the emerging distributed-systems substrate.

On the telecom side, day-to-day operation still ran through the TMN hierarchy, with vendor-specific EMS handling per-element fault, configuration, performance, and inventory beneath the NMS. Early self-driving ambitions surfaced as 3GPP SON — self-configuration, self-optimization, self-healing — alongside high-volume usage-data pipelines feeding billing and analytics.

Network modeling

Operating the network meant running it from a central console. A Network Management System collected alarms, performance metrics, and topology across vendor equipment so that operators could detect faults and provision devices end-to-end. This is the direct root of today’s model-driven, telemetry-fed operations.

Interoperability came from CORBA-era common information models, anchored by TM Forum’s TMF-814 for multi-vendor optical and transport OSS integration, while carrier control extended to the data plane through traffic-policy management — the lineage that became PCRF and PCF.

Control loop

The earliest roots trace to the OSS/BSS split: the foundational division between running the infrastructure (inventory, provisioning, fault and performance management) and monetizing the subscriber (billing, rating, CRM). That separation of concerns still shapes platform operations today.

Managing the elements ran through two competing lineages: the heavyweight OSI/ITU-T TMN stack (CMIS and CMIP over the Q3 interface, modeling network elements as managed objects) and the lighter-weight IETF SNMP, whose agent and MIB poll-and-trap loop proved durable while the TMN stack faded. Even the first closed-loop automation appears here in embryonic form: declarative business-rules engines externalized operational decisions from application code, prefiguring the policy-driven, event-triggered control loops that later operations eras were built on.

Operations runs on infrastructure, consumes what the software lifecycle delivers, and is where agentic transformation is landing fastest — the collect→correlate→remediate loop is exactly the shape an agent fits into.