Business-relevant workflow
The demo models a digital onboarding or loan eligibility journey instead of a generic hello-world endpoint, keeping the story close to banking operations.
Client demo strategy
Recommended demo app
A compact enterprise application that shows Open Liberty as a production-ready MicroProfile runtime behind an NGINX web tier. The app demonstrates canary routing, operational health, JVM metrics, edge hardening, mesh telemetry, and end-to-end traceability in one business-relevant workflow.
Capability story
The demo models a digital onboarding or loan eligibility journey instead of a generic hello-world endpoint, keeping the story close to banking operations.
Stable users receive the current scoring response from Instance A. Beta users receive the experimental risk model response from Instance B.
Target architecture
OpenShift target platform
All runtime, observability, GitOps, and mesh design targets the
spoke-dc cluster only.
NGINX remains the L7 control point for header canary routing, hardening headers, rate limiting, and custom error pages.
The project namespace is brac-poc-openliberty-v5, matching the repo,
docs site, and client demo identity.
MicroProfile health and metrics are exposed through NGINX. User workload monitoring, OpenShift Logging, Tempo, OpenTelemetry, and Perses are available on the target cluster.
Current platform status
| Area | Current state | Impact for the POC |
|---|---|---|
| Target cluster | spoke-dc API reachable at api.spoke-dc.ocp.comptech-lab.com. |
This remains the only deployment and observability target. |
| OpenShift base | OpenShift 4.20.18 cluster operators are Available, not Progressing, and not Degraded. |
The platform is healthy for the current demo slice. |
| Project namespace | brac-poc-openliberty-v5 exists and contains the app runtime. |
The namespace now hosts Liberty A, Liberty B, NGINX, services, route, and ServiceMonitor. |
| Application route | https://brac-poc-openliberty-v5.apps.spoke-dc.ocp.comptech-lab.com |
Public demo route for canary, health, metrics, and hardening proof. |
| Runtime pods | liberty-a, liberty-b, and nginx are 1/1 available. |
The client can see two Liberty instances behind one NGINX reverse proxy. |
| OpenShift GitOps | openshift-gitops running. Two Applications on spoke-dc: labops Synced/Healthy, spoke-dc-cluster-config OutOfSync/Healthy. No brac-poc-openliberty-v5 Application yet. |
Architectural choice made: Option C — the workload now lives at lab-workloads.git/components/apps/brac-poc-openliberty-v5/ and is activated by clusters/spoke-dc/kustomization.yaml (lab-workloads MR !2 merged 2026-05-07). Liberty instances are OpenLibertyApplication CRs (apps.openliberty.io/v1), reconciled by the Open Liberty Operator v1.6.1 already installed on spoke-dc. The Argo Application yaml in this repo (PR #60 merged) targets path: clusters/spoke-dc; applying it is Phase 5 step 3, gated on Phase 4 (Nexus image publish). |
| OSSM3 | Istio is Healthy with ZTunnel v1.28.5; Kiali and istio-cni pods are running. |
Mesh visibility and policy are available; namespace enrollment + waypoint + AuthorizationPolicy are drafted in PR #48. |
| Ambient mode | ZTunnel/default is Healthy on spoke-dc with 4 ztunnel pods Running across worker nodes; istio-waypoint GatewayClass is Accepted. |
Platform side is ready. Namespace enrollment (istio.io/dataplane-mode: ambient) and the waypoint Gateway are drafted in PR #48 and apply via Argo sync. |
| Gateway API | istio, istio-remote, and istio-waypoint GatewayClasses are accepted. |
The app can later use Gateway API patterns if the owner chooses that route; istio-waypoint is consumed by PR #48. |
| Observability | User workload monitoring, Loki, OpenTelemetry Collector, Tempo, and Perses resources are present. | Health, metrics, and Tempo traces are verified for the deployed app. |
| Trace pipeline | The previous collector Kafka bridge exporter issue is resolved for the demo trace path. | The active collector path is traces to Tempo. A namespace-scoped fanout collector that preserves Tempo/Loki/UWM and adds Splunk + SigNoZ + Kafka exporters is drafted in PR #51. |
| External Secrets Operator | ESO v1.1.0 healthy; SecretStore/rke2-vault Ready=True. Namespace-scoped SecretStore/brac-poc-vault Ready=True and ExternalSecret/nexus-docker-pull SecretSynced=True (PR #45 applied directly to brac-poc-openliberty-v5 namespace 2026-05-06). |
Vault → nexus-docker-pull sync verified end-to-end. Same shape reused by PR #51 (observability-export) and PR #56 (runtime-sso) once their Vault KV paths are populated. |
| Vault state (RKE2) | HA active (3 raft nodes), v1.18.2, transit auto-unseal. Auth mount kubernetes-spoke-dc/ enabled with role brac-poc-openliberty-v5 bound to SA eso-vault-auth. Policies brac-poc-openliberty-v5-read and brac-poc-openliberty-v5-jenkins-ci exist. KV paths secret/brac-poc-openliberty-v5/ci/{nexus,source} populated. |
Phase 2 of the rollout is effectively complete. Outstanding KV writes: runtime/observability-export (PR #51 destinations) and runtime/sso (PR #56 mpJwt) — pending real destination/IDP credentials. |
| Jenkins build worker | RKE2 PVCs jenkins-buildah-cache (30Gi) and jenkins-maven-cache (10Gi) on longhorn are Bound and match the PR #46 drafts. JCasC ConfigMap does not yet declare a Kubernetes clouds: block — pod-agent cloud config is the remaining gap before a release rehearsal. |
Phase 3 step 1 (PVCs) done. Step 2 (JCasC clouds + smoke run with buildah container) is the next piece of real work before Phase 4 image publish to Nexus. |
Ambient mesh and traceability
The target design uses ambient mode for the POC namespace. ztunnel provides L4 mTLS, while waypoint proxies provide L7 policy, telemetry, retries, timeouts, and circuit breaking for NGINX-to-Liberty calls.
NGINX and Open Liberty must preserve W3C trace headers such as
traceparent and tracestate so each demo request can be
followed across proxy, mesh, and app logs.
ztunnelL4 service identity and mTLS for ambient workloads.waypointL7 policy point for retries, timeouts, circuit breaking, and richer telemetry.NGINXHeader-based canary routing remains in the web tier; ztunnel is not used for L7 header steering.KialiTopology view for NGINX, Liberty A, and Liberty B traffic flow.traceparentRequired request correlation header propagated from edge to application.request-idApplication-visible fallback correlation ID for logs and support workflows.
Verified on May 6, 2026: OSSM3 is Healthy on spoke-dc, Kiali is running,
istio-cni is running, and the platform-side ambient runtime (ZTunnel/default
Healthy with 4 ztunnel pods, istio-waypoint GatewayClass
Accepted) is ready. Namespace enrollment, waypoint Gateway, and a baseline
AuthorizationPolicy are drafted in
PR #48 with
the runbook at docs/handoff/ossm3-ambient-runbook.md; owner applies via
Argo sync after PR #47, or via the standalone-apply path before.
CI/CD and GitOps
lab-workloads.git.
Jenkins should not deploy directly to OpenShift in the final flow. It should publish artifacts to Nexus and create or prepare a GitOps promotion change for owner review, leaving OpenShift GitOps to reconcile the cluster after the approved merge.
Verified on May 6, 2026: Jenkins build brac-poc-openliberty-v5-maven-smoke #4
completed successfully. The job read source configuration and a GitHub read-only
deploy key from Vault through Kubernetes auth, cloned the private GitHub repository
over SSH, compiled 13 Java sources with Java 17, resolved dependencies through Nexus
maven-public, and produced brac-poc-openliberty-v5.war.
Build speed policy is cache-first and IPv4-preferred. Use Nexus-hosted or proxied
base images, keep Jenkins Maven/container caches, and let the owner pre-pull demo
images with crictl on approved nodes before rehearsals. For RKE2 or
other containerd targets, use owner-run scp + ctr image archive imports
when LAN transfer is faster than registry pulls. Jenkins can also publish an
app-specific Liberty feature base image so later runtime builds can skip repeated
features.sh execution.
Repository ownership
brac-poc-openliberty-v5 owns the Java application source, Open Liberty
runtime config, NGINX source config, Dockerfiles, Jenkinsfile, and this Cloudflare
Pages wiki.
lab-workloads.git owns the OpenShift desired state: namespace,
Deployment, Service, Route or Ingress, ConfigMap references, and promoted image tags.
| Asset | Source of truth | Reason |
|---|---|---|
| Open Liberty app code | GitHub | Owned by the application delivery lifecycle and built by Jenkins. |
Open Liberty server.xml |
GitHub | Versioned with the runtime image and MicroProfile feature set. |
| NGINX reverse proxy config | GitHub | Part of the app web tier: canary routing, HSTS, rate limiting, and custom errors. |
| NGINX runtime deployment | GitLab GitOps | OpenShift desired state references the built NGINX image and runtime objects. |
| OpenShift manifests | GitLab GitOps | Already consumed by OpenShift GitOps through lab-workloads.git. |
| Built artifacts and images | Nexus | Stores versioned WAR/Maven artifacts and container images for promotion. |
Future enterprise integrations
Verified 2026-05-07: every external destination this section refers to is
already running on the lab RKE2 cluster. Splunk
(splunk.apps.sub.comptech-lab.com), SigNoZ
(signoz.apps.sub.comptech-lab.com), Strimzi/Kafka
(kafka namespace, brokers 0/1/2 + bootstrap), Keycloak / SSO IDP
(auth.apps.sub.comptech-lab.com), and WSO2 IS / APIM (API gateway
candidates) are all up. Per-destination enablement is now a Vault-write +
collector-uncomment task, not a provisioning task.
Logs, metrics, and traces forward through the namespace fanout collector to a
Splunk HEC endpoint. Drafted in
PR #51;
owner enables by writing splunk_hec_* keys to Vault and uncommenting
the exporter in pipelines.
OTLP gRPC export to SigNoZ via the same fanout collector. Drafted in
PR #51;
owner enables by writing signoz_otlp_endpoint (and optional access
token) to Vault.
OTLP-proto traces, logs, and metrics on per-signal Kafka topics for a deferred consumer. Drafted in PR #51; Strimzi/AMQ Streams provisioning is owner-managed.
Identity contract codified: API gateway owns the trust boundary, NGINX scrubs
client-supplied identity headers, Liberty re-verifies JWTs via MicroProfile JWT.
See
PR #50 /
docs/adr/0006-sso-and-api-gateway-integration-points.md.
Vault is the source of truth whenever applicable. The ESO pattern operationalised
in PR #45
synthesizes nexus-docker-pull from
secret/brac-poc-openliberty-v5/ci/nexus; the same shape is reused by
PR #51 (observability-export) and the SSO/gateway paths in ADR-0006. GitHub PAT
migration to Vault is drafted in
PR #49.
Infrastructure governance
The assistant may inspect cluster state with read-only commands, draft manifests, document required actions, update application or documentation files, and publish this Cloudflare Pages wiki.
The project owner manually performs all OpenShift, RKE2, GitLab GitOps, Jenkins, Nexus, Kafka, Redis, and platform service changes.
Direct cluster mutations are normally avoided. The current OpenShift deployment was done under an explicit owner-approved exception; future promotion should return to Jenkins, Nexus, and GitLab/OpenShift GitOps.
Secret handling is Vault-first. Kubernetes Secrets, OpenShift pull secrets, Jenkins credentials, ExternalSecrets, and CSI mounts are delivery mechanisms only and should be backed by Vault wherever the platform supports it.
Manual owner actions
All design PRs (#45–#58) are merged to main. The
architectural choice for the GitOps tree (Option C — OpenLibertyApplication
CRs + components/apps/<base> + cluster overlay) is also merged
(this repo PR #60, lab-workloads.git MR !2) on 2026-05-07.
The remaining work is operational rollout, sequenced in
docs/handoff/rollout-sequence.md.
Cluster-side state on 2026-05-07:
spoke-dc.lab-workloads.git/main: done (MR !2 merged).clouds: block: pending (real work).lab-workloads.git: pending, owner-applied with PAT.Application + manual sync: pending on Phase 4 (so pods don't ImagePullBackOff).docs/handoff/gitops-promotion-runbook.md.
ztunnel and waypoint readiness.
docs/handoff/ossm3-ambient-runbook.md.
lab-workloads.git path for clusters/spoke-dc/brac-poc-openliberty-v5.
docs/handoff/gitops-promotion-runbook.md.
nexus-docker-pull image pull secret before full image promotion.
nexus-docker-pull drafted in
PR #45 · Jenkins K8s pod agent (maven + buildah + persistent caches) drafted in
PR #46 · runbooks docs/handoff/nexus-vault-bootstrap-runbook.md and docs/handoff/jenkins-k8s-agent-runbook.md.
/home/ze/secrets/github-token; Jenkins source checkout should continue through the Vault-backed read-only deploy key.
docs/handoff/github-token-rotation-runbook.md · helper scripts/secrets/refresh-gh-token.sh.
docs/adr/0006-sso-and-api-gateway-integration-points.md.
docs/handoff/observability-fanout-runbook.md.
Client requirement map
| Requirement | Demo implementation | Proof point |
|---|---|---|
| L7 Load Balancing | Two Open Liberty workloads behind one NGINX reverse proxy on OpenShift. | Verified on the OpenShift route; all external proof uses NGINX. |
| Header-based Canary | NGINX routes version: beta to Instance B. |
Verified response body shows instance: B, beta flags, and route decision. |
| MicroProfile Health | Liberty MicroProfile Health exposes /health/live through NGINX. |
Verified /health/live returns UP. |
| MicroProfile Metrics | Liberty MicroProfile Metrics exposes /metrics through NGINX. |
Verified thread_count{mp_scope="base"} is exposed. |
| Hardening | NGINX config includes HSTS, custom errors, and rate limiting. | Verified /app/login returns custom 429 under burst load. |
| Enterprise delivery | Jenkins smoke build and Nexus Docker Registry API are verified; Vault-backed credentials and GitOps promotion remain open. | Evidence pack separates current smoke proof from pending image push and GitOps sync proof. |
| Traceability and mesh | OSSM3 ambient mode, waypoint L7 policy, and propagated trace headers. | Tempo traces are verified; Kiali/ambient proof remains a later phase. |
Application scope
/appLanding page showing the active backend instance./app/api/applicationsMock onboarding application records./app/api/eligibilityLoan eligibility scoring endpoint./app/api/instanceReturns instance, version, hostname, and feature flags./app/loginProtected context for NGINX rate-limit demonstration./health/liveMicroProfile liveness endpoint exposed via NGINX./health/readyMicroProfile readiness endpoint exposed via NGINX./metricsMicroProfile metrics endpoint for JVM thread count./openapiOpenAPI document for API discovery.Traffic steering
curl -k https://brac-poc-openliberty-v5.apps.spoke-dc.ocp.comptech-lab.com/app/api/instance
curl -k -H "version: beta" https://brac-poc-openliberty-v5.apps.spoke-dc.ocp.comptech-lab.com/app/api/instance
The beta response should visibly include instance: B,
version: beta, and a feature flag such as
new-risk-model.
The browser page at /app shows instance,
channel, hostname, request_id,
trace_id, span_id, latency_ms, and
route_decision so the client can see routing and traceability without
reading logs first.
MicroProfile observability
Enable Liberty MicroProfile Health, expose /health/live through
NGINX, and show an aggregate green liveness status in the demo.
Enable Liberty MicroProfile Metrics, read /metrics, and highlight the
JVM thread count as the required runtime signal.
ServiceMonitorScrapes the Open Liberty metrics endpoint for user workload monitoring.Perses dashboardShows liveness, JVM thread count, canary split, rate-limit responses, and upstream errors.Loki logsReceives structured app and NGINX logs with request and trace correlation fields.Tempo tracesReceives OpenTelemetry spans from the deployed Liberty workloads.MicroProfile TelemetryEnabled in Liberty with otel.sdk.disabled=false; OTLP points to otel-collector.openshift-tracing.svc:4317.End-to-end traceability contract
| Signal | Required fields | Where it appears |
|---|---|---|
| Trace headers | traceparent, tracestate, and fallback x-request-id. |
NGINX access logs, Liberty request logs, mesh telemetry, and traces. |
| Application logs | trace_id, span_id, request_id, instance, channel, status, duration_ms. |
OpenShift Logging, Loki, future Splunk forwarding. |
| Metrics labels | app, component, instance, channel, namespace, route. |
Prometheus user workload monitoring and Perses dashboards. |
| Trace spans | service.name, service.version, deployment.environment, http.route, net.peer.name. |
OpenTelemetry Collector, Tempo, and future SigNoZ export. |
NGINX hardening
/app/login and high-risk app contexts.
Proof checks
docker compose up --build
BASE_URL=https://brac-poc-openliberty-v5.apps.spoke-dc.ocp.comptech-lab.com tests/nginx/proof.sh
Dockerfile.libertyBuilds the Maven WAR and runs it on Open Liberty Java 17 for local/container builds.Dockerfile.liberty.runtimeBuilds the Liberty runtime image from the Jenkins-packaged WAR.Dockerfile.nginxRuns the hardened NGINX reverse proxy on port 8080.docker-compose.ymlStarts Liberty A, Liberty B, and NGINX for local validation.scripts/image-cache/prepare-ctr-cache.shBuilds image archives and an owner-run scp + ctr import script for RKE2/containerd cache warm-up.tests/nginx/proof.shValidates routing, HSTS, liveness, JVM thread metrics, trace headers, UI visibility, and rate limiting.Client evidence pack
| Evidence | Repository template | Purpose |
|---|---|---|
| GitHub commit | docs/evidence/01-github-commit.md |
Shows source control provenance. |
| Jenkins build and Nexus artifact | 02-jenkins-build.md, 03-nexus-artifact.md |
Shows CI and artifact lifecycle. |
| GitOps sync | 04-gitops-sync.md |
Shows deployment through OpenShift GitOps. |
| Canary, health, metrics, and rate limit | 05 through 08 |
Shows the client web-tier requirements directly. |
| Kiali and Perses | 09-kiali-topology.md, 10-perses-dashboard.md |
Shows ambient mesh and observability proof after platform readiness. |
| Tempo trace search | 11-tempo-trace-search.md |
Shows the fixed OpenTelemetry trace path from Liberty to Tempo. |
Platform handoffs
docs/handoff/rollout-sequence.mdPhased order, hard gates, and parallel paths for the seven open draft PRs (#45–#51) that close all wiki Manual Actions. Owner reference for taking the drafts live.docs/handoff/jenkins-release-worker-handoff.mdCurrent Jenkins release blocker: source Vault access works, ci/nexus returns 403, and a container-capable build worker is still required.docs/handoff/nexus-maintainer-handoff.mdNexus readiness status covering Maven repositories, Docker Registry API readiness, Jenkins credentials, and OpenShift pull-secret actions.docs/handoff/image-cache-speedup-runbook.mdOwner-run cache warm-up, crictl pre-pull, scp + ctr archive import for containerd targets, IPv4 validation, and Jenkins cache guidance for slow networks.docs/handoff/openshift-observability-maintainer-handoff.mdVerified OpenShift collector trace path and future forwarding notes for Splunk, SigNoZ, or Kafka.docs/adr/0004-vault-first-secret-policy.mdVault-first policy for CI/CD credentials, runtime secrets, pull secrets, TLS material, and future integration tokens.docs/adr/0005-build-speed-and-image-cache-policy.mdNexus-backed image cache, IPv4-preferred egress, owner-controlled crictl pre-pull, and scp + ctr policy for containerd targets.docs/adr/0006-sso-and-api-gateway-integration-points.mdIdentity contract across Client → API gateway → NGINX → Liberty: gateway is the trust boundary, NGINX scrubs client-supplied identity headers, Liberty re-verifies JWTs via MicroProfile JWT. Service identity (mesh mTLS) and user identity (JWT) are orthogonal.
Maven dependency resolution is verified through Nexus, and the Docker Registry API
now returns the expected unauthenticated 401. Maven and image publishing
should wait until Nexus credentials are readable through Vault by the Jenkins release
role, a container-capable Jenkins worker is available, and the OpenShift pull secret
is created or synchronized by the owner.
Demo flow
/app and show request_id, trace_id, span_id, latency, and route decision on the page.version: beta and show the request is served by Instance B./health/live through NGINX and show liveness status./metrics and highlight the JVM thread count metric./app/login to show rate limiting.Delivery plan
brac-poc-openliberty-v5/
|-- app/ Open Liberty MicroProfile application
|-- liberty/ server.xml and runtime config
|-- nginx/
| |-- nginx.conf
| |-- conf.d/
| | `-- brac-poc.conf
| `-- error-pages/
| |-- 404.html
| |-- 429.html
| `-- 50x.html
|-- jenkins/ Jenkinsfile and pipeline helper scripts
|-- deploy/openshift/ current OpenShift manifest used for the direct exception
|-- gitops-drafts/ GitOps-ready draft resources for lab-workloads.git
|-- docs/ Cloudflare Pages wiki
| |-- adr/ Architecture decisions and change-control rules
| |-- evidence/ client demo evidence templates
| `-- handoff/ platform maintainer handoff notes
|-- observability/
| |-- trace-context.md
| |-- splunk-export.md
| |-- perses-dashboard.md
| |-- otel-collector-pipeline.md
| `-- signoz-export.md
|-- security/
| |-- sso-integration.md
| |-- api-gateway.md
| `-- vault-integration.md
|-- Dockerfile.liberty
|-- Dockerfile.liberty.runtime
|-- Dockerfile.nginx
|-- docker-compose.yml two Liberty instances plus NGINX
`-- README.md local runbook
lab-workloads.git/
|-- components/apps/
| `-- brac-poc-openliberty-v5/
| |-- kustomization.yaml
| |-- namespace.yaml
| |-- liberty-sa.yaml # per-component SAs
| |-- nginx-sa.yaml
| |-- eso-vault-auth-sa.yaml
| |-- secretstore.yaml # ESO -> Vault
| |-- eso-tokenrequest-rbac.yaml
| |-- externalsecret-nexus-docker-pull.yaml
| |-- liberty-a.yaml # OpenLibertyApplication CR
| |-- liberty-b.yaml # OpenLibertyApplication CR
| |-- nginx-deployment.yaml # plain Deployment
| |-- services.yaml # nginx Service only
| |-- route.yaml
| |-- servicemonitor.yaml
| |-- prometheusrule.yaml
| |-- networkpolicy.yaml
| |-- waypoint.yaml # OSSM3 ambient
| |-- authorizationpolicy.yaml
| |-- requestauthentication.yaml # mpJwt
| |-- externalsecret-runtime-sso.yaml
| |-- externalsecret-observability-export.yaml
| |-- otel-fanout-collector.yaml
| `-- otel-fanout-queue-pvc.yaml
`-- clusters/
`-- spoke-dc/kustomization.yaml # `- ../../components/apps/brac-poc-openliberty-v5`
Decision: C (verified 2026-05-07). Liberty instances are
OpenLibertyApplication CRs (apps.openliberty.io/v1),
reconciled by the Open Liberty Operator v1.6.1 installed on spoke-dc
via lab-gitops. The two CRs share applicationName: brac-poc-openliberty-v5
and split on applicationVersion: stable / beta, which the
operator surfaces as app.kubernetes.io/version labels for canary
telemetry. NGINX is still a plain Deployment (the operator only manages
OpenLibertyApplication). Trade-off discussion is preserved in
docs/handoff/gitops-promotion-runbook.md.