Home › Forums › Miria forums – EN › Miria : How To ? › How Miria distributes Jobs to Datamovers
- This topic has 0 replies, 1 voice, and was last updated 1 year, 6 months ago by Eric Collart.
5 November 2021 at 9 h 53 min #10591Eric CollartKeymaster
Typical Miria configuration uses 1 database server and 1 or more datamovers (or agents).
Pools of 1 or more datamovers can be created
Excluding database server from pools avoids the risk of overloading it as the database server is also an agent (datamover).
The database server records all required information to keep track of migrated files but also orchestrates the migration task(s) and their related job(s)
A migration task transfers files from source storage to destination storage (many types supported by Miria)
Per default, 1 migration task is using 1 job that is using 1 thread.
But you can configure the task to run more than 1 job.
And/or you can configure jobs to run more than 1 thread.
Job load balancing
Starting the migration task will start scanning the source and the first job starts transferring data soon after.
Then, if desired, more jobs can be fired based on criteria below that can be all defined:
- Total size of files scanned
- Time the scan job is running
- Number of files scanned
As soon one of the criteria is reached, a new job is fired and Miria DB server assigns that new job to the datamover in the pool having the least number of jobs already running.
Miria load job balancing criteria is based on the number of job(s) each datamover is running.
Single thread or Multithread
Per default, 1 job is single thread consuming 1 CPU core and some memory.
But 1 job can also be multithread in what case each thread uses 1 CPU core and some memory.
The number of threads a job can run is a task’s configuration parameter.
Based on that explanation, you can partially understand how a Miria configuration is sized.
Take an example:
- Imagine a configuration with 1 Miria DB server and 1 Miria datamover where only the datamover will be used to migrate data (by using pool).
- The datamover server has a single 8 cores CPU. Let’s say we have enough RAM for this example.
- You can then run 8 single threaded jobs or 4 dual threaded jobs or 2 jobs with each 4 threads or 1 job with 8 threads.
Trying to run more jobs or more threads per job will cause performance problem!
If you need more jobs/threads to complete the migration in the timeframe foreseen, you can add more datamover(s) so more jobs/threads can be spread across all of them.
Performance will also be impacted by many other factors like the source and the destination storage’s load capabilities and the network path performance for instance.
The first migration’s run last the longest because :
- the first scan job must scan the entire source storage
- the first run of the migration task(s) must transfer all files to the destination
The next run of the migration task will be quicker as only modified files must be treated.
If the source is still used by users in production during migration, more migration runs will be needed to transfer files modified by users since previous run to the source.
The next runs of the scanning can also be quicker if Miria supports FastScan on the source storage; it is the case for several storage models.
FastScan is based on Snapdiff techniques.
- You must be logged in to reply to this topic.
Sujets / Topics
- Tina 4.8.1 Controlled Release is available
- PowerShell for Lina 6.0
- Advisory ID : LINA/ADE-2023-0002
- Advisory ID : LINA/ADE-2023-0001
- Miria 4.0 is available
- Lina 6.0 is available !
- Tina 4.7.1 GA is available
- Recent Tina versions require CentOS 7 minimum
- Lina 5.3.4 requires CentOS 7 minimum
- How to remove the HTTP port in Miria
Tous les articles
Tous les sujets (forums)
All Topics (forums)