This article describes how to migrate Data Services data from Solr to Pinot/SQL and how to migrate Track dictionary data from the Solr Dictionary collection to the Track SQL database.

Data migration is optional and should only be performed if existing data must be preserved and migrated to the new environment.

Data Service Data Migration

Exporting Data

Use Query Browser from Data Services 11.x to export the required data.

Export Administrative Objects

Perform this step only if you are not using the default Organization, Repository, and Access Tokens.

• Export the Organization:
  1. In the item list, right-click Organization.
  2. Then select Export.
• Export the Repository:
  1. In the item list, right-click Repository.
  2. Then select Export.
• Export the AccessToken:
  1. In the item list, right-click AccessToken.
  2. Then select Export.

Export Actions

1. Select File > Backup Repository > Absolute Dates....
2. Clear Include all repo items.
3. Select Include repository, organization, IDs.
4. Select Action from the second group of items.
5. Complete the export.

Export Repository Data

1. Select File > Backup Repository > Absolute Dates...
2. Clear Include all repo items.
3. Select Include repository, organization, IDs.
4. Select the following item types:
   • Activity
   • Dataset
   • Event
   • Relative
   • Resource
   • Set
   • Snapshot
   • Source
   • View
5. Complete the export.

If you plan to restream data, simply select Set and View.

To export additional repositories, switch to the desired repository and repeat the repository export procedure.

Export Considerations

• Deprecated items may be included in the export.
• During import into Data Services 12.x, deprecated items are identified, logged, and skipped automatically.
• Dashboard, Viewlet, and Settings information previously stored in the Dictionary table is now stored in a dedicated Track SQL schema and must be migrated separately using the Track Data Migration procedure.

Importing Data

Before starting Query Browser, ensure that Zookeeper, Kafka, Pinot, and the Domain Server are running, as Query Browser requires the Domain Server for authentication and Kafka for writing data to Pinot.

1. Start Query Browser.
2. Log in as Administrator.
3. Import the exported administrative objects in the following order:
   • Organization
   • Repository
   • AccessToken
4. Import the remaining repository data exports.
5. Verify that the imported data is accessible and functioning as expected.

Track UI Data Migration

Dictionary data, including Dashboards, Viewlets, Settings, and related configuration, must be migrated from the Data Services (Solr) Dictionary collection to the Track SQL database.

Exporting Dictionary Data

Before exporting dictionary data, ensure that all Dashboards and Viewlets are saved.

1. Log in to Query Browser as Administrator.
2. Execute the following query:

Get Dictionaries fields all

3. Select Results > Export.
4. Save the file as dictionaries.csv.

Converting Dictionary Data

Use the track-migration.jar utility to convert the exported CSV file into SQL migration scripts.

Run the following command:

java -jar track-migration.jar -f dictionaries.csv --dialect postgres

The utility generates SQL scripts required to recreate the dictionary data in the Track database.

Generated SQL Scripts

Do not import scripts generated for the Global or Sample repositories.

Generated Review Files

Importing Dictionary Data

Import the generated SQL scripts into the Track database using the jdbcmk utility.

Configure the Database Connection

DB_URL="jdbc:postgresql://localhost:5432/meshiq-db?user=${DB_USERNAME}&password=${DB_PASSWORD}&currentSchema=meshiq_track"

Import Settings

java -cp "lib/*" com.nastel.jdbcmk \
-runsql "$DB_URL" \
dictionaries_migration_settings_postgres.sql \
org.postgresql.Driver

Import Repository Data

java -cp "lib/*" com.nastel.jdbcmk \
-runsql "$DB_URL" \
dictionaries_migration_DefaultRepo_meshIQ_postgres.sql \
org.postgresql.Driver

Repeat the repository import step for each repository migration file that must be migrated.

Verification

1. Log in to Track.
2. Verify that dashboards are available.
3. Verify that viewlets are present and functioning correctly.
4. Verify that settings have been migrated successfully.
5. Review the import logs for warnings or errors.