Scaling the meshIQ Platform

The meshIQ platform is a robust, flexible system built for management, observability, and tracking across various messaging and stream-processing environments. Whether you're using Kafka, IBM MQ, or other technologies, meshIQ can be tuned to meet your resource needs—small-scale or enterprise-grade.

This guide walks you through every scaling method, configuration file, and practical command needed to manage workloads smoothly.

General Guidelines

meshIQ is a Java-based platform, making it more RAM-intensive than CPU-heavy. So when provisioning nodes in your Kubernetes cluster, prioritize RAM over CPU. For a standard deployment (e.g., meshIQ Simple on OKE), it's recommended to start with:

• 2 CPUs
• 24 GB RAM

As workload increases, RAM should be scaled first. CPU scaling is secondary unless you're seeing throttling.

Pod Resource Limits and Requests (values.yaml)

Each pod's resource use can be customized in values.yaml under:

limits.cpu
limits.memory
requests.cpu
requests.memory

These are commented out by default, but you can tweak them based on your needs. If you're using auto-scaling node pools, it's highly recommended to set RAM limits, so no pod hogs resources unexpectedly.

Horizontal Scaling Support

Some meshIQ components support horizontal scaling (increasing the number of replicas):

Connection Managers

You can scale any connection manager based on your environment (e.g., Kafka, IBM MQ). Example for scaling Kafka Connection Manager:

Steps to Scale a Kafka Connection Manager:

1. Scale via Kubernetes:

   kubectl scale deployment nsqcmkafka --replicas=2

2. Manually Register New Pod in Workgroup Server (WGS):
   • Go to Manage > Node viewlet > Click Add Node button. The Node Create Window opens.
   • Enter the details:
     • Name: Must be unique (e.g., anything but REMOTE_KAFKA)
     • Hostname: Use the pod name from kubectl get pods
     • Use DNS: Keep unchecked
     • IP Address: Use internal pod IP from kubectl get pods -o wide
     • Port: Use same port as original (e.g., 5024)
     • Node Type: Select appropriate connection type (e.g., Kafka Node)

Scaling Core Services: Manage, Track, and Observe

These components can be horizontally scaled without extra configuration.

Commands to Scale:

kubectl scale deployment manage --replicas=2
kubectl scale deployment track --replicas=2
kubectl scale deployment observe --replicas=2

• No manual registration required.
• Load balancers will automatically distribute traffic to all running pods.
• Ideal when Track or Observe starts handling a large number of monitors or historical lookups.

Control Center and Gateway

Technically, you can scale these too, but:

Control Center:

• Used by administrators for managing users, roles, and generating audit reports.
• Low concurrency. Even with many Track/Manage users, this component rarely experiences load issues.

Gate way:

• Handles authentication across all meshIQ apps.
• Even with multiple users, the resource demand remains low. So for most use cases, scaling these services is unnecessary.