TIBCO BW6 Administration

A developer's complete guide to administering TIBCO BusinessWorks 6 and Container Edition in enterprise production — from day one operations to deep debugging.

1. Admin Mindset for Developers

Developers transitioning into admin roles carry a significant advantage: they understand what the applications are supposed to do. Most infrastructure admins do not. When something breaks, a developer-turned-admin can reason about application behavior, data flow, and integration logic — not just infrastructure health. This perspective is the foundation of effective BW6 administration.

2. BW6 Architecture — What You're Administering

Understanding the runtime architecture is the foundation of everything else. BW6 is not the same as BW5 — it's a fundamentally different OSGI-based runtime.

Core Components

Key Runtime Concepts

The OSGI Runtime

BW6 runs on an OSGI (OSGi Alliance) container — every BW process is a bundle within a Felix OSGI framework. This matters for admin because:

Process Instance Lifecycle

# A BW6 process instance goes through these states:
Created → Running → [Suspended] → [Checkpoint] → Completed
                                                 ↘ Faulted → Dead Letter

# Admin actions per state:
Running  → can kill/suspend via TEA
Suspended → can resume or kill
Faulted  → investigate via TEA process monitor, re-run or kill

3a. Docker & Container Fundamentals for BWCE

Before you can administer BWCE, you need to own Docker. As a BWCE admin you're not just running containers — you're building them, inspecting them, and diagnosing them. This section gives you everything you need.

Why Containers Change Everything for BW6

BWCE Docker Image Architecture

A BWCE image is built in layers. Understanding this is critical for diagnosing build failures and optimizing image size.

BWCE Dockerfile — Anatomy of Every Build

# Standard BWCE Dockerfile — understand every line

# Stage 1: Start from TIBCO's official BWCE base image
FROM tibco/bwce:2.8.0 AS bwce-base
# This image contains: BW6 engine, OSGI runtime, startup scripts
# Get exact version from: https://hub.docker.com/r/tibco/bwce/tags

# Stage 2: Install required connector plugins
# Plugins NOT in base image must be explicitly added
COPY plugins/com.tibco.bw.palette.ems-7.x.x.zip /bw/connectors/
COPY plugins/com.tibco.bw.palette.rest-7.x.x.zip /bw/connectors/
COPY plugins/com.tibco.bw.palette.jdbc-7.x.x.zip /bw/connectors/

# Stage 3: Copy 3rd party libraries (JDBC drivers, EMS client)
COPY lib/postgresql-42.x.x.jar /bw/lib/
COPY lib/tibjms.jar /bw/lib/

# Stage 4: Copy your application EAR
COPY target/MyIntegrationApp-1.0.0.ear /bw/apps/

# Stage 5: Copy certificates (if needed for TLS)
COPY certs/truststore.jks /bw/config/certs/

# Stage 6: Expose the port(s) the BW app listens on
EXPOSE 8080
EXPOSE 8443

# Stage 7: Environment variables with defaults (override at runtime)
ENV BW_PROFILE=default
ENV BW_APP_PROP_DB_URL=jdbc:postgresql://localhost:5432/mydb
ENV BW_LOGLEVEL=INFO

# Stage 8: Health check (Kubernetes uses this for readiness)
HEALTHCHECK --interval=30s --timeout=10s --retries=3 \
  CMD curl -f http://localhost:8080/health || exit 1

# Entrypoint is defined in the TIBCO base image
# It runs startup.sh which initializes the BW runtime

Building BWCE Images — Full Command Reference

## ─── BUILD ───────────────────────────────────────────────────

# Basic build
docker build -t myregistry/myapp:1.0.0 .

# Build with specific Dockerfile name
docker build -f Dockerfile.prod -t myregistry/myapp:1.0.0 .

# Build with build arguments (version inject)
docker build \
  --build-arg APP_VERSION=1.0.0 \
  --build-arg BUILD_DATE=$(date -u +%Y-%m-%dT%H:%M:%SZ) \
  -t myregistry/myapp:1.0.0 .

# Build with no cache (fresh build — use when debugging image issues)
docker build --no-cache -t myregistry/myapp:1.0.0 .

# Multi-platform build (build AMD64 image on ARM Mac or vice versa)
docker buildx build --platform linux/amd64 -t myregistry/myapp:1.0.0 .

# Build and see ALL output (no quiet mode — see every layer)
docker build --progress=plain -t myregistry/myapp:1.0.0 . 2>&1 | tee build.log

## ─── TAG & PUSH ──────────────────────────────────────────────

# Tag image with multiple tags
docker tag myregistry/myapp:1.0.0 myregistry/myapp:latest
docker tag myregistry/myapp:1.0.0 myregistry/myapp:prod

# Push to registry
docker push myregistry/myapp:1.0.0
docker push myregistry/myapp:latest

# Login to private registry first
docker login myregistry.company.com
docker login myregistry.company.com -u username -p password

Running & Testing BWCE Containers Locally

## ─── RUN LOCALLY (before pushing to Kubernetes) ─────────────

# Basic run with environment variable overrides
docker run -d \
  --name bwce-myapp \
  -p 8080:8080 \
  -e BW_APP_PROP_DB_URL="jdbc:postgresql://host.docker.internal:5432/mydb" \
  -e BW_APP_PROP_DB_USER="myuser" \
  -e BW_APP_PROP_DB_PASSWORD="mypassword" \
  -e BW_LOGLEVEL=DEBUG \
  myregistry/myapp:1.0.0

# Run with volume mount (for checkpoint persistence or config files)
docker run -d \
  --name bwce-myapp \
  -p 8080:8080 \
  -v /local/checkpoints:/bw/checkpoint \
  -v /local/certs:/bw/config/certs:ro \
  -e BW_APP_PROP_DB_URL="jdbc:postgresql://dbhost:5432/mydb" \
  myregistry/myapp:1.0.0

# Run with env file (cleaner for many variables)
cat > app.env << 'EOF'
BW_APP_PROP_DB_URL=jdbc:postgresql://dbhost:5432/mydb
BW_APP_PROP_DB_USER=myuser
BW_APP_PROP_DB_PASSWORD=mypassword
BW_APP_PROP_EMS_URL=tcp://emshost:7222
BW_LOGLEVEL=INFO
EOF
docker run -d --name bwce-myapp -p 8080:8080 --env-file app.env myregistry/myapp:1.0.0

# Run interactively — see startup logs in real time
docker run -it \
  --name bwce-debug \
  -p 8080:8080 \
  -e BW_LOGLEVEL=DEBUG \
  myregistry/myapp:1.0.0

# Run and override entrypoint — get shell instead of starting BW
docker run -it --entrypoint /bin/bash myregistry/myapp:1.0.0
# Now you can inspect the container filesystem before BW starts
# ls /bw/apps/    ← check EAR is there
# ls /bw/lib/     ← check JDBC drivers
# cat /bw/config/appconfig.ini  ← check default config

Analyzing & Inspecting BWCE Containers — Full Reference

## ─── INSPECT RUNNING CONTAINER ───────────────────────────────

# View all container details (JSON)
docker inspect bwce-myapp

# Extract specific fields
docker inspect bwce-myapp --format='{{.State.Status}}'
docker inspect bwce-myapp --format='{{.State.StartedAt}}'
docker inspect bwce-myapp --format='{{range .Mounts}}{{.Source}} → {{.Destination}}{{"\n"}}{{end}}'
docker inspect bwce-myapp --format='{{range .Config.Env}}{{.}}{{"\n"}}{{end}}'
# ↑ Lists ALL environment variables — verify overrides took effect

# Check resource usage of running container
docker stats bwce-myapp
docker stats bwce-myapp --no-stream  # One snapshot, then exit
# Output: CPU%, MEM USAGE/LIMIT, MEM%, NET I/O, BLOCK I/O

# Check exposed ports and actual port bindings
docker port bwce-myapp

## ─── ANALYZE IMAGE (before running) ─────────────────────────

# List all layers and their sizes
docker history myregistry/myapp:1.0.0
docker history myregistry/myapp:1.0.0 --no-trunc  # full commands
# Use this to identify which layer is bloating the image

# Inspect image metadata
docker inspect myregistry/myapp:1.0.0
docker inspect myregistry/myapp:1.0.0 --format='{{.Config.Env}}'
docker inspect myregistry/myapp:1.0.0 --format='{{.Config.ExposedPorts}}'
docker inspect myregistry/myapp:1.0.0 --format='{{.Config.Labels}}'

# Get image size
docker images myregistry/myapp:1.0.0 --format "{{.Size}}"

# Dive deeper — use 'dive' tool for interactive layer analysis
dive myregistry/myapp:1.0.0
# 'dive' shows each layer and what files changed — great for optimization
# Install: brew install dive OR docker run -it wagoodman/dive myregistry/myapp:1.0.0

## ─── LOGS ────────────────────────────────────────────────────

# Follow live logs
docker logs -f bwce-myapp

# Last N lines
docker logs --tail=200 bwce-myapp

# Logs since timestamp
docker logs --since="2024-01-15T10:00:00" bwce-myapp
docker logs --since=1h bwce-myapp   # last 1 hour

# Filter logs for errors
docker logs bwce-myapp 2>&1 | grep -i "ERROR\|FATAL\|Exception"
# 2>&1 combines stdout and stderr — BW logs to both

## ─── EXEC INTO CONTAINER ─────────────────────────────────────

# Get interactive shell
docker exec -it bwce-myapp /bin/bash
docker exec -it bwce-myapp /bin/sh   # if bash not available (Alpine)

# Run single command without interactive shell
docker exec bwce-myapp ls -la /bw/apps/
docker exec bwce-myapp cat /bw/config/appconfig.ini
docker exec bwce-myapp ps aux | grep java

# Check environment variables inside running container
docker exec bwce-myapp env | sort
# Verify all BW_APP_PROP_* overrides are present

# Check filesystem inside container
docker exec bwce-myapp ls -la /bw/
docker exec bwce-myapp ls -la /bw/apps/      # EAR files
docker exec bwce-myapp ls -la /bw/lib/       # 3rd party jars
docker exec bwce-myapp ls -la /bw/connectors/ # connector plugins
docker exec bwce-myapp ls -la /bw/log/        # BW log files inside container

# Copy files from container to host (get logs, config, heap dump)
docker cp bwce-myapp:/bw/log/bwappnode.log ./bwappnode.log
docker cp bwce-myapp:/bw/log/ ./container-logs/

# Copy files into running container
docker cp new-cert.crt bwce-myapp:/bw/config/certs/new-cert.crt

BWCE Container Network Diagnostic Commands

## ─── NETWORK DIAGNOSTICS ─────────────────────────────────────

# List all networks
docker network ls

# Inspect network containers are attached to
docker network inspect bridge
docker network inspect bwce-network

# Test connectivity FROM inside container
docker exec bwce-myapp curl -v http://downstream-api/health
docker exec bwce-myapp ping -c 3 dbhost
docker exec bwce-myapp telnet emshost 7222

# Check DNS resolution inside container
docker exec bwce-myapp nslookup downstream-api
docker exec bwce-myapp cat /etc/resolv.conf

# Port connectivity test from inside container
docker exec bwce-myapp bash -c "timeout 3 bash -c 'cat </dev/null > /dev/tcp/dbhost/5432' && echo 'OPEN' || echo 'CLOSED'"

BWCE Multi-Container Architecture with Docker Compose

For local development and testing, Docker Compose lets you run the full BWCE architecture stack together:

# docker-compose.yml — full BWCE local stack
version: '3.8'

services:

  # BWCE Application
  bwce-myapp:
    image: myregistry/myapp:1.0.0
    ports:
      - "8080:8080"
    environment:
      - BW_APP_PROP_DB_URL=jdbc:postgresql://postgres:5432/mydb
      - BW_APP_PROP_EMS_URL=tcp://ems:7222
      - BW_LOGLEVEL=DEBUG
    depends_on:
      postgres:
        condition: service_healthy
      ems:
        condition: service_started
    volumes:
      - ./checkpoints:/bw/checkpoint
    networks:
      - bwce-net

  # PostgreSQL database
  postgres:
    image: postgres:14
    environment:
      - POSTGRES_DB=mydb
      - POSTGRES_USER=myuser
      - POSTGRES_PASSWORD=mypassword
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U myuser"]
      interval: 10s
      retries: 5
    networks:
      - bwce-net

  # TIBCO EMS (if licensed — or use ActiveMQ for testing)
  ems:
    image: myregistry/tibco-ems:8.x.x
    ports:
      - "7222:7222"
    networks:
      - bwce-net

networks:
  bwce-net:
    driver: bridge

# Commands
docker-compose up -d         # start all services
docker-compose logs -f bwce-myapp  # follow app logs
docker-compose ps            # status of all services
docker-compose down          # stop and remove containers
docker-compose down -v       # also remove volumes

How TIBCO Architecture is Solutioned Using Containers

Enterprise BWCE deployments follow one of these reference architectures depending on scale and operational requirements:

Pattern 1 — Single Namespace (Small/Medium Enterprise)

Pattern 2 — Multi-Namespace (Large Enterprise, Team Isolation)

Pattern 3 — Sidecar Pattern (Enhanced Observability)

# Pod with BW app + observability sidecars
spec:
  containers:

  # Main BW application container
  - name: bwce-myapp
    image: myregistry/myapp:1.0.0
    ports:
    - containerPort: 8080

  # Sidecar 1: Log shipper (ships BW logs to ELK/Splunk)
  - name: filebeat
    image: elastic/filebeat:8.x
    volumeMounts:
    - name: bw-logs
      mountPath: /bw/log

  # Sidecar 2: Metrics exporter (exposes JVM metrics to Prometheus)
  - name: jmx-exporter
    image: bitnami/jmx-exporter:latest
    ports:
    - containerPort: 9090

  volumes:
  - name: bw-logs
    emptyDir: {}   # shared between main + filebeat sidecar

Container Image Maintenance — Production Admin Tasks

## ─── IMAGE HOUSEKEEPING ──────────────────────────────────────

# List all BWCE images on host
docker images | grep bwce
docker images --format "{{.Repository}}:{{.Tag}}\t{{.Size}}\t{{.CreatedAt}}"

# Remove old images (dangling = untagged after build)
docker image prune              # remove dangling only
docker image prune -a --filter "until=720h"  # remove all older than 30 days

# Pull latest base image (for security patches)
docker pull tibco/bwce:2.8.0

# Check image for known CVE vulnerabilities
docker scout cves myregistry/myapp:1.0.0      # Docker Scout (built-in)
trivy image myregistry/myapp:1.0.0             # Trivy (recommended)
grype myregistry/myapp:1.0.0                   # Grype (alternative)

# Compare two image versions (what changed between releases)
docker diff bwce-myapp-old   # shows files changed in running container

# Export image to tar (for air-gapped environments)
docker save myregistry/myapp:1.0.0 -o myapp-1.0.0.tar
docker load -i myapp-1.0.0.tar

## ─── SYSTEM CLEANUP ──────────────────────────────────────────
docker system df              # show disk usage by Docker
docker system prune           # remove stopped containers, unused networks, dangling images
docker system prune -a        # include all unused images
# ⚠️ Be careful with prune -a in production — confirm what will be removed first

Debugging a BWCE Build Failure

## When 'docker build' fails — step-by-step diagnosis

Step 1: Read the exact error in build output
docker build --progress=plain -t myregistry/myapp:1.0.0 . 2>&1 | tee build.log
grep -A5 "ERROR\|failed\|not found" build.log

Step 2: Identify which layer failed
# Each 'Step N/M' in output is one Dockerfile instruction
# Failed step tells you exactly which COPY/RUN/FROM failed

Step 3: Debug a specific layer by building up to that point
# Add --target if using multi-stage, or manually comment out lines below failure

Step 4: Inspect the last successfully built layer
docker build -t debug-layer . 2>&1 | grep "sha256" | tail -5
docker run -it sha256:<last-good-layer-hash> /bin/bash
# Now you're inside the last good state — inspect filesystem, test commands

Common BWCE build failures:
COPY failed: file not found in build context
  → EAR file path wrong, or mvn build didn't run before docker build

FROM tibco/bwce:2.x.x: manifest unknown
  → Wrong BWCE version tag; check docker.io/tibco/bwce for valid tags

unauthorized: authentication required
  → docker login to private registry before build

no space left on device
  → docker system prune on build host

BWCE is BW6 packaged as Docker containers, typically orchestrated by Kubernetes or Docker Swarm. The application logic is identical to traditional BW6 — only the runtime infrastructure differs dramatically.

BWCE Image Anatomy

# BWCE Docker image layers:
TIBCO Base OS Image (RHEL/CentOS)
  └─ TIBCO Runtime Layer (BW6 engine + OSGI bundles)
       └─ Connector Layer (EMS, JDBC, HTTP connectors)
            └─ Application Layer (your EAR/AAR + appconfig.ini)
                 └─ Environment overrides (via env vars at runtime)

BWCE Configuration Override Pattern

# appconfig.ini — base config baked into image
ConnectionTimeout=30
MaxRetries=3
DBUrl=jdbc:postgresql://placeholder:5432/mydb

# Kubernetes ConfigMap — overrides per environment
apiVersion: v1
kind: ConfigMap
metadata:
  name: bwce-app-config
data:
  BW_APP_PROP_DB_URL: "jdbc:postgresql://prod-db:5432/mydb"
  BW_APP_PROP_MAX_RETRIES: "5"

# Kubernetes Secret — for credentials
apiVersion: v1
kind: Secret
type: Opaque
data:
  BW_APP_PROP_DB_PASSWORD: base64encodedvalue

Key kubectl Commands for BWCE Admin

# Pod status
kubectl get pods -n <namespace> -l app=<bwce-app>
kubectl describe pod <pod-name> -n <namespace>

# Logs
kubectl logs <pod-name> -n <namespace> --tail=200 -f
kubectl logs <pod-name> -n <namespace> --previous  # crashed pod logs

# Exec into running pod
kubectl exec -it <pod-name> -n <namespace> -- /bin/bash

# Scale
kubectl scale deployment <deployment-name> --replicas=3 -n <namespace>

# ConfigMap/Secret refresh (after editing)
kubectl rollout restart deployment <deployment-name> -n <namespace>

# Check resource usage
kubectl top pods -n <namespace>
kubectl top nodes

4. Domains, AppSpaces, AppNodes & Deployment

Domain Structure Best Practices

Deployment via tibcoamx CLI

# Connect to domain
tibcoamx connect --domain MyDomain --host adminhost --port 9091 --user admin --password ****

# List deployed apps
tibcoamx app list --domain MyDomain

# Deploy new application
tibcoamx app deploy --file MyApp-1.0.ear --appSpace MyAppSpace --domain MyDomain

# Start application
tibcoamx app start --name MyApp --appSpace MyAppSpace --domain MyDomain

# Stop application
tibcoamx app stop --name MyApp --appSpace MyAppSpace --domain MyDomain

# Undeploy
tibcoamx app undeploy --name MyApp --appSpace MyAppSpace --domain MyDomain

# Check AppNode status
tibcoamx node list --appSpace MyAppSpace --domain MyDomain

# Kill and restart AppNode
tibcoamx node stop --name MyAppNode --appSpace MyAppSpace --domain MyDomain
tibcoamx node start --name MyAppNode --appSpace MyAppSpace --domain MyDomain

Deployment Checklist

5. TEA & Admin UI — Your Control Panel

TIBCO Enterprise Administrator (TEA) is your primary management interface for traditional BW6. Understanding every panel is essential.

TEA Navigation Map

Process Monitor — Critical Admin Skill

Process Monitor shows every running BW6 process instance. As an admin you'll use it to:

6. Configuration Management

BW6 Configuration Hierarchy

# Priority order (highest to lowest):
1. Environment Variables (OS level or container env vars)
2. Application Properties set via TEA / Admin UI
3. appconfig.ini (bundled in EAR)
4. Default values in BW module descriptors

# Rule: always use TEA / env vars for production — never bake creds into EAR

Property Files — Know These Locations

Modifying JVM Arguments (TRA Files)

# bwagent.tra — sets JVM args for the BW Agent
tibco.env.HEAP_SIZE=1024m
tibco.env.MAX_HEAP_SIZE=2048m
java.property.Djava.io.tmpdir=/data/tmp
java.property.Djavax.net.ssl.trustStore=/certs/truststore.jks
java.property.Djavax.net.ssl.trustStorePassword=changeit

# AppNode JVM — configured via TEA > AppNode > JVM Settings
# Or via config.ini in the AppNode config directory
-Xms512m -Xmx2048m -XX:+UseG1GC -XX:MaxGCPauseMillis=200

Connection Pool Configuration

# JDBC connection pool — critical for database-heavy apps
Minimum Pool Size: 5        # Never set to 0 (cold start latency spike)
Maximum Pool Size: 20       # Match your DB max_connections / num_apps
Connection Timeout: 30000   # 30 seconds — fail fast
Idle Timeout: 600000        # 10 minutes — return idle connections
Validation Query: SELECT 1  # Validates connection before use

# EMS connection factory — for JMS-heavy apps
ConnAttemptCount: 3
ConnAttemptDelay: 1000
ConnAttemptTimeout: 5000
ReconnAttemptCount: -1      # -1 = infinite reconnect (recommended for prod)

7. Day-One Operations Checklist

Before you do anything in a production environment, run through this checklist every morning or after any change.

Morning Health Check (15 minutes)

## STEP 1: Check all BW Agents are running
ps aux | grep bwagent | grep -v grep
# Expected: one bwagent process per machine

## STEP 2: Check AppNode JVM processes
ps aux | grep AppNode | grep -v grep
# One JVM process per AppNode — if missing, an AppNode has crashed

## STEP 3: Check TEA connectivity
curl -s http://adminhost:9091/tea/api/v1/domains | python3 -m json.tool

## STEP 4: Check all applications are STARTED
tibcoamx app list --domain ProdDomain | grep -v STARTED
# Any non-STARTED app needs investigation

## STEP 5: Check error logs from last 24 hours
grep -i "ERROR\|FATAL\|Exception" /opt/tibco/bw6/domains/Prod/appspaces/*/logs/*.log \
  --include="*.log" | tail -100

## STEP 6: Check disk space (logs fill fast)
df -h /opt/tibco /var/log /data
# Alert threshold: 80% — action threshold: 90%

## STEP 7: Check JVM heap usage via TEA or JMX
# TEA > AppNode > Memory — if heap is consistently >80%, tune or scale

Know Your Environment Map

On day one, document these before anything else. You cannot admin what you don't know exists:

8. Monitoring & Alerting

What to Monitor — BW6 Key Metrics

JMX Monitoring (Advanced)

# Enable JMX in AppNode TRA / config.ini
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9999
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false

# Connect with JConsole or jvisualvm
jconsole <host>:9999

# Key MBeans to watch:
java.lang:type=Memory              → HeapMemoryUsage (used/max)
java.lang:type=GarbageCollector    → CollectionCount, CollectionTime
java.lang:type=Threading           → ThreadCount, PeakThreadCount
com.tibco.bw:*                     → BW-specific MBeans (process counts, rates)

Log Aggregation Setup

In production, never hunt individual log files manually. Set up centralized logging:

9. Log Management & Analysis

BW6 Log Locations

# BW Agent log
BW_HOME/domains/<domain>/logs/bwagent.log

# AppNode logs (one directory per AppNode)
BW_HOME/domains/<domain>/appspaces/<appspace>/appnodes/<appnode>/logs/
  ├── bwappnode.log         ← main AppNode log
  ├── bwappnode.log.1       ← rolled previous file
  ├── gc.log                ← GC log (if enabled)
  └── stderr.log            ← JVM startup errors, OOM messages

# Application-level logs (if app uses logback file appender)
BW_HOME/domains/<domain>/appspaces/<appspace>/appnodes/<appnode>/logs/<appname>.log

Log Configuration (logback.xml)

<!-- logback.xml — control log verbosity per package -->
<configuration>
  <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <file>${bw.home}/logs/app.log</file>
    <rollingPolicy class="...TimeBasedRollingPolicy">
      <fileNamePattern>app.%d{yyyy-MM-dd}.log</fileNamePattern>
      <maxHistory>30</maxHistory>   <!-- keep 30 days -->
    </rollingPolicy>
  </appender>

  <!-- Production: INFO level. Never DEBUG in prod — fills disk fast -->
  <root level="INFO"><appender-ref ref="FILE"/></root>

  <!-- Specific package — DEBUG for targeted troubleshooting -->
  <logger name="com.mycompany.integration" level="DEBUG"/>
</configuration>

Log Analysis — Common Patterns to Search

# Find all ERRORs in last hour
grep "ERROR" bwappnode.log | awk -v d="$(date -d '1 hour ago' '+%Y-%m-%d %H:%M')" '$0 >= d'

# Find OutOfMemoryError
grep -i "OutOfMemory\|java.lang.OOM" *.log stderr.log

# Find connection failures
grep -i "connection refused\|connection timeout\|could not connect" bwappnode.log | tail -50

# Find slow processes (BW logs process duration)
grep "process.duration" bwappnode.log | awk -F'=' '{if($2 > 30000) print}' | tail -20

# Count errors per minute (identify spike time)
grep "ERROR" bwappnode.log | awk '{print $1, $2}' | cut -c1-16 | sort | uniq -c

# Find specific correlation ID / process ID
grep "processId=abc123" bwappnode.log

Log Rotation — Prevent Disk Full

# If logback rolling is not configured, set up OS-level logrotate
# /etc/logrotate.d/tibco-bw
/opt/tibco/bw6/domains/*/appspaces/*/appnodes/*/logs/*.log {
    daily
    rotate 14
    compress
    missingok
    notifempty
    copytruncate    # Non-disruptive — copies then truncates live file
}

10. Performance Tuning

JVM Tuning for BW6 AppNodes

# Recommended JVM flags for BW6 production AppNode
-Xms1g -Xmx2g                      # Set min=max to avoid heap resize pauses
-XX:+UseG1GC                        # G1 GC — best for BW6 mixed workloads
-XX:MaxGCPauseMillis=200            # Target max GC pause — tune per SLA
-XX:G1HeapRegionSize=16m            # Larger regions for large object avoidance
-XX:+HeapDumpOnOutOfMemoryError     # Capture heap dump on OOM
-XX:HeapDumpPath=/data/dumps/       # Save dumps — need 2x heap free space!
-Xlog:gc*:file=/opt/tibco/gc.log:time,uptime:filecount=5,filesize=20m
                                    # GC logging — essential for tuning

Thread Pool Sizing

# BW6 thread pool — configured in AppNode config
# Too few → throttled throughput; Too many → context switch overhead

Min Threads: 10      # Always keep threads warm
Max Threads: 100     # Formula: (num CPUs × 2) + I/O wait threads

# For I/O-heavy apps (DB, EMS, HTTP):
Max Threads = num_CPUs × 10   # I/O releases threads during wait

# For CPU-heavy apps (transformation, encryption):
Max Threads = num_CPUs × 2    # CPU-bound — more threads = more contention

EMS Tuning (if using TIBCO EMS)

# tibemsd.conf — key parameters
max_msg_memory = 512mb           # In-memory queue size before overflow to disk
msg_swapping = enabled           # Allow disk overflow (slower but safer)
max_connections = 1000           # Max client connections
max_msg_size = 10mb              # Reject messages larger than this
flow_control = enabled           # Back-pressure — slow producers when queues fill

# Check EMS server status
tibemsadmin -server tcp://emshost:7222 -user admin -password ****
show server
show queues
show consumers

11. High Availability & Failover

BW6 HA Topologies

Fault Tolerance Configuration

# Enable fault tolerance in AppSpace
# TEA > AppSpace > Fault Tolerance Settings
Fault Tolerance Mode: Active-Active
Activation Interval:  2000ms     # How often nodes coordinate
Heartbeat Interval:   1000ms     # Health check frequency
Activation Timeout:   30000ms    # How long to wait for peer before taking over

# For EMS-based HA — use EMS fault-tolerant pairs
# Primary EMS + Backup EMS — automatic failover
Primary Server:  tcp://ems-primary:7222
Backup Server:   tcp://ems-backup:7222

Graceful Shutdown Procedure

# NEVER hard-kill an AppNode without following this sequence

Step 1: Stop new message intake
# Suspend all listeners (EMS, HTTP) via TEA
# Allow in-flight processes to complete

Step 2: Wait for active processes to finish
# TEA > Process Monitor — wait until Active Process Count = 0
# Set a timeout — if not zero in N minutes, escalate

Step 3: Graceful stop via TEA/CLI
tibcoamx app stop --name MyApp --appSpace MyAppSpace --domain MyDomain --graceful true

Step 4: Verify AppNode stopped cleanly
ps aux | grep AppNode
# Should show no process

12. Debug Scenarios & Runbook

The following scenarios are the most common production situations you will face. Each follows the same structure: symptom, diagnosis approach, resolution.

Step 1: Check AppNode log immediately after deploy attempt
tail -200 bwappnode.log | grep -A5 "ERROR\|Exception\|FAILED\|Bundle"

Step 2: Look for OSGI bundle activation errors specifically
grep -i "BundleException\|could not be resolved\|missing required" bwappnode.log
# Missing bundle = missing Shared Module or connector plugin

Step 3: Check Shared Modules deployed to domain
tibcoamx sharedmodule list --domain MyDomain
# Compare against what the app's MANIFEST.MF requires

Step 4: Check plugin versions in TEA > Plugin Manager
# Developer should provide the list of required plugins + versions

Step 5: Check application configuration for missing required properties
tibcoamx app getconfig --name MyApp --appSpace MyAppSpace --domain MyDomain

Step 1: Find and read the crash evidence
grep -i "OutOfMemory\|OOM\|heap space\|GC overhead" stderr.log bwappnode.log

Step 2: Check if heap dump was captured
ls -lh /data/dumps/*.hprof
# If exists → share with developer for analysis in Eclipse MAT

Step 3: Check GC log for warning signs leading up to crash
grep "GC" gc.log | tail -100
# Look for: increasing GC pause times, decreasing heap free after GC

Step 4: Check active process count before crash (from logs)
grep "active.process.count" bwappnode.log | tail -50

Step 5: Immediate recovery
tibcoamx node start --name MyAppNode --appSpace MyAppSpace --domain MyDomain

Step 6: Temporary mitigation while investigating
# Increase heap (if safe headroom on host)
# Or add a second AppNode to distribute load

Step 1: Check if consumers are active on the queue
tibemsadmin -server tcp://ems:7222 -user admin -password ****
show queue MyQueue
# Look at: Consumer Count, Messages Pending, Messages/Second
# If consumer count = 0 → BW app not connected → restart app
# If consumer count > 0 but rate = 0 → consumer is stuck

Step 2: Check for stuck/long-running processes in TEA Process Monitor
# Filter by application, sort by start time → oldest first
# Oldest process = potentially processing the bad message

Step 3: Check BW logs for repeated errors on same message
grep "redelivered\|retry\|failed to process" bwappnode.log | tail -50

Step 4: Check downstream dependency health
curl -v http://downstream-api/health
psql -h dbhost -U user -c "SELECT count(*) FROM pg_stat_activity WHERE state='active'"

Step 5: If bad message — move to dead letter queue to unblock
# In EMS Admin:
move queue MyQueue MyQueue.deadletter    # moves ALL messages — careful!
# Or use EMS browser to selectively move specific message IDs

Step 1: Check current pool state via JMX or TEA
# Look for: Active Connections, Idle Connections, Awaiting connections
# If Active = Max → pool exhausted

Step 2: Check DB side — active connections from BW host
psql -h dbhost -c "SELECT count(*), state, application_name FROM pg_stat_activity \
  WHERE application_name LIKE '%tibco%' GROUP BY state, application_name"

Step 3: Test basic DB connectivity from BW host
telnet dbhost 5432
psql -h dbhost -U tibcouser -d mydb -c "SELECT 1"

Step 4: Check for stale connections (firewall timeout)
# If firewall closes idle connections after N minutes:
# → Add validation query to connection pool: SELECT 1
# → Set idle connection timeout < firewall timeout

Step 5: Temporary fix — restart AppNode to recycle pool
tibcoamx node stop --name MyAppNode --appSpace MyAppSpace --domain MyDomain
tibcoamx node start --name MyAppNode --appSpace MyAppSpace --domain MyDomain

Step 1: Get crash reason immediately
kubectl describe pod <pod-name> -n <namespace>
# Look at: Last State, Exit Code, Reason
# Exit code 137 = OOM killed by kernel (increase memory limit)
# Exit code 1 = app error (check logs)

Step 2: Read logs from the crashed pod
kubectl logs <pod-name> -n <namespace> --previous
kubectl logs <pod-name> -n <namespace> --previous | grep -i "error\|fatal\|exception"

Step 3: Check ConfigMap has all required properties
kubectl get configmap bwce-app-config -n <namespace> -o yaml
# Compare with appconfig.ini from the EAR artifact

Step 4: Check resource limits
kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.containers[0].resources}'

Step 5: Rollback deployment if recent change caused it
kubectl rollout undo deployment <deployment-name> -n <namespace>
kubectl rollout status deployment <deployment-name> -n <namespace>

Step 1: Establish where the time is going
# Check BW process duration in logs
grep "duration" bwappnode.log | awk -F'duration=' '{print $2}' | sort -n | tail -20

Step 2: Check GC logs for pause correlation
grep "Pause\|GC" gc.log | awk '{print $1, $NF}' | tail -50
# If GC pauses > 500ms correlate with slow response times → tune GC or increase heap

Step 3: Check thread pool — is it saturated?
# Via JMX: java.lang:type=Threading → ThreadCount vs PeakThreadCount
# If ThreadCount = Max configured → requests are queuing

Step 4: Measure each downstream dependency independently
time curl -o /dev/null -s http://downstream-api/endpoint
time psql -h dbhost -c "EXPLAIN ANALYZE SELECT * FROM orders WHERE id=1"

Step 5: Check network latency
ping -c 20 dbhost | tail -5
traceroute dbhost

Step 1: Check if BW Agent process is running
ps aux | grep bwagent | grep -v grep
# If no process → agent crashed → check bwagent.log for cause → restart

Step 2: Check TEA server process
ps aux | grep tibco.admin | grep -v grep

Step 3: Restart BW Agent
cd BW_HOME/bin
./bwagent.sh start
# Or via systemd:
sudo systemctl start tibco-bwagent

Step 4: Check agent log for connection errors
tail -100 BW_HOME/domains/<domain>/logs/bwagent.log
# Look for: port conflict, certificate error, cannot reach TEA

Step 5: If agent is running but TEA can't connect → check port/firewall
telnet bwagent-host 5555    # default agent port
netstat -tlnp | grep 5555

13. Security Administration

Certificate Management

# Check certificate expiry — schedule this as monthly cron
keytool -list -keystore /certs/server.jks -storepass changeit | grep "Valid from"
openssl s_client -connect downstream-api:443 -servername api.company.com 2>/dev/null \
  | openssl x509 -noout -dates

# Add new certificate to truststore
keytool -import -alias new-cert -file new-cert.crt \
  -keystore /certs/truststore.jks -storepass changeit

# List all certs in truststore
keytool -list -keystore /certs/truststore.jks -storepass changeit

# After cert update — restart AppNode to pick up new truststore
tibcoamx node stop ... && tibcoamx node start ...

User & Role Management in TEA

Secrets Management

# NEVER store passwords in plain text in TRA files or properties
# BW6 built-in obfuscation (weak — for display only, not real security)
tibcoamx obfuscate --password mysecretpassword

# Better: use external vault
# BWCE: Kubernetes Secrets (base64, not encryption — use Sealed Secrets or Vault)
# Traditional: HashiCorp Vault with BW6 plugin
# Or: OS-level environment variables set by secure configuration management (Ansible, Chef)

14. CI/CD & Deployment Automation

Traditional BW6 — Maven + tibcoamx

# Maven build (Developer responsibility — triggers from CI)
mvn clean package -DskipTests
# Produces: target/MyApp-1.0.0.ear

# Deployment script template (Admin responsibility)
#!/bin/bash
APP_NAME="MyApp"
APP_VERSION="1.0.0"
DOMAIN="ProdDomain"
APPSPACE="ProdAppSpace"
EAR_PATH="/deploy/artifacts/MyApp-${APP_VERSION}.ear"

# Stop existing
tibcoamx app stop --name $APP_NAME --appSpace $APPSPACE --domain $DOMAIN

# Undeploy old version
tibcoamx app undeploy --name $APP_NAME --appSpace $APPSPACE --domain $DOMAIN

# Deploy new version
tibcoamx app deploy --file $EAR_PATH --appSpace $APPSPACE --domain $DOMAIN

# Apply properties
tibcoamx app setproperty --name $APP_NAME --appSpace $APPSPACE --domain $DOMAIN \
  --property "DBUrl" --value "jdbc:postgresql://prod-db:5432/mydb"

# Start
tibcoamx app start --name $APP_NAME --appSpace $APPSPACE --domain $DOMAIN
echo "Deployment complete: ${APP_NAME} v${APP_VERSION}"

BWCE — Kubernetes Rolling Deployment

# Build and push new image
docker build -t myregistry/bwce-myapp:1.0.1 .
docker push myregistry/bwce-myapp:1.0.1

# Update deployment image (triggers rolling update)
kubectl set image deployment/bwce-myapp bwce-myapp=myregistry/bwce-myapp:1.0.1 -n prod

# Watch rolling update progress
kubectl rollout status deployment/bwce-myapp -n prod

# Rollback if needed
kubectl rollout undo deployment/bwce-myapp -n prod

# BWCE deployment YAML — key sections
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1        # Add 1 new pod before removing old
      maxUnavailable: 0  # Never reduce below desired count

BW5 → BW6 Transition Guide

BW5 and BW6 share the TIBCO name but are fundamentally different products. A BW5 developer stepping into BW6/CE will encounter a platform that shares almost none of the same runtime, tooling, or deployment model. This section maps the conceptual differences, identifies what prior knowledge transfers, and outlines the topics that require learning from scratch — including where MuleSoft platform experience provides a meaningful head start.

Concept Mapping — BW5 → BW6 → MuleSoft

What Transfers from BW5 (directly useful)

What Transfers from MuleSoft

Training Roadmap — Topic by Topic

Each topic below includes a summary of what to focus on and the best free resource available. Work through them in order — each builds on the previous.

Recommended Learning Sequence

The topics below are ordered by dependency — each builds the foundation for the next. Work through them in sequence rather than in parallel for the most efficient path to operational readiness.

When you hit a problem you've never seen before, this framework prevents panic and gets you to resolution systematically.

The DREAD Methodology (for unknown issues)

TIBCO Support Resources

Information to Gather Before Escalating

# Collect before contacting TIBCO Support or senior engineer

1. BW version:    tibcoamx --version
2. OS + version:  cat /etc/os-release
3. Java version:  java -version (from AppNode process)
4. Exact error:   Full stack trace from log
5. Timeline:      When did it start? What changed?
6. Reproducible:  Does it happen every time or intermittent?
7. Scope:         One app? One AppNode? All apps? All environments?
8. Logs:          bwappnode.log, bwagent.log, stderr.log (last 24hr)
9. Thread dump:   kill -3 <appnode-pid> → stderr.log
10. Heap dump:    If OOM → .hprof file
11. Config:       Current application configuration (anonymize passwords)