Importing / exporting remote Kafka clusters with nsqmigr

See the below sections for information on how to import or export remote Kafka clusters (nsqcmkafka.jar) in bulk using the nsqmigr utility.

Export (get) remote Kafka clusters

Perform the following to bulk export remote Kafka clusters:

1. Run a command similar to the following to create an information (.info) containing a list of all remote Kafka clusters. Please note that the -K parameter must be uppercase. 

   nsqmigr -g -K -f<filename>.info

   For a full list of commands, enter the command: nsqmigr -?
   Example:
   The following command will generate the file, remote_kafka.info, and will write all remote Kafka clusters from the workgroup server database to it. The file will be created in C:\nastel\apwmq\config\groups\remote_kafka.info.

   C:\nastel\apwmq\bin>nsqmigr -g -K -fC:\nastel\apwmq\config\groups\remote_kafka.info

2. The file is now created. The default export directory is [APWMQ_HOME]/config/groups, but can be overwritten by using the -O parameter.

Exported fields

All remote Kafka clusters and their associated fields will be written to the information file created. The fields are as follows:

Kafka::ManagerName
RemoteNode
Cluster
Bootstrap
GroupId
Config1::Entry
Config1::Value
Config2::Entry
Config2::Value
Config3::Entry
Config3::Value
Config4::Entry
Config4::Value
Config5::Entry
Config5::Value
Config6::Entry
Config6::Value
Config7::Entry
Config7::Value
Config8::Entry
Config8::Value
Config9::Entry
Config9::Value
Config10::Entry
Config10::Value

Example file

The following is an example of an information (.info) file.

Import (put) remote Kafka clusters

Perform the following to bulk import remote Kafka clusters:

1. You will need to create an information (.info) file to define each remote Kafka cluster being imported. To create this file, either:
   • manually create this file to define each remote Kafka cluster being imported. Add the fields listed in the Exported fields section above for each remote Kafka cluster. See Example file above for a sample; or
   • follow the steps in the Export (get) remote Kafka clusters section above to export the remote Kafka clusters. Update the file accordingly.
2. Place the file in:

   [APWMQ_HOME]/config/groups

3. Run a command similar to the following to import the file. Please note that the -K parameter must be uppercase.

   nsqmigr -p -K -f<filename>.info

   Example:
   The following command will import the remote Kafka clusters from the remote_kafka.info file located in C:\nastel\apwmq\config\groups. 

   C:\nastel\apwmq\bin>nsqmigr -p -K -fC:\nastel\apwmq\config\groups\remote_kafka.info

4. The remote Kafka clusters are now imported in to the WGS database. The info file is automatically backed up in [APWMQ_HOME]/config/groups/workgroup/history and is not deleted.