Angular

Parallel Processing with PowerShell

Introduction

Before we jump on to the Parallel Processing concept in powershell, Let’s understand What is Powershell and Parallel(Asynchronous) processing? PowerShell is a Windows command-line shell designed especially for system administrators. While it may look similar to the Command Prompt app in Windows, It is a scalable way for corporate IT managers to automate business-critical tasks on every Windows PC across a wide area network.

What is Synchronous?
Each next task only will start, once a task which is started before it gets completed.

What is Asynchronous?
When you run something asynchronously it means it is non-blocking, you execute it without waiting for it to complete and carry on with other tasks.

What is Parallelism?
Parallelism means to run multiple tasks at the same time, in parallel. Parallelism works well when you can separate tasks into independent pieces of work.

Description

When you think about PowerShell script, you’re probably thinking of synchronous execution. This means that PowerShell executes code, one line at a time. It starts executing one command, waits for it to complete, and then starts the next one.

Synchronous execution is fine for scripts In which execution of the next line depends on the execution of a line before it.When you are working with the script that can take many minutes or an hour, You can instead choose to execute code asynchronously with any of the below PS concepts.

Jobs (PowerShell 3.0)

Job is a piece of code that is executed in the background, creates n multiple background processes to execute n no. of jobs , each job creates a separate powershell instance to complete execution. (You can find out no. of instances created while running jobs , inside task manager)

Different cmdlets to work with PS Jobs.

Start-Job: Create and execute job.

1..5 | % {Start-Job  { "Hello" }  }

Get-Job: Get all jobs that are started with Start-Job cmd.
Wait-Job: Wait for all jobs to complete.

Get-Job | Wait-Job

Receive-Job: To print output of job to console.

Get-Job | Receive-Job

Remove-Job: To delete all jobs that were created with Start-Job command.
*Jobs created must be removed with this command.

Get-Job | Remove-Job

 

ThreadJob (PowerShell 6.0)

This is a thread based job. This is a lighter weight solution compared to Jobs. Unlike traditional PS Jobs which spawn a whole new host process for each running job, PS ThreadJobs run in multiple threads on the same process which vastly increases performance by lowering overhead.

There are a few drawbacks to using a ThreadJob over a background job. If a background job hangs, only that process hangs. All other jobs keep chugging away. If you have a job that hangs with ThreadJob the entire queue is affected.

Measure-Command {1..5 | % {Start-Job {Start-Sleep 1}} | Wait-Job} | Select-Object TotalSeconds
Measure-Command {1..5 | % {Start-ThreadJob {Start-Sleep 1}} | Wait-Job} | Select-Object TotalSeconds

TotalSeconds
------------
   5.7665849
   1.5735008

Syntax is quite similar to PSJobs , Job string is replaced with ThreadJob. One parameter is there to set no of jobs you want to start concurrently. (i.e. throttle limit , default value is 5)

 

Parallel foreach (PowerShell 7.0)

Each iteration of ForEach-Object that is passed in via the Parallel scriptblock input, will run in it’s own thread.This is faster than both the above methods. You can run all scripts in parallel for each piped input object.

If your script is crunching a lot of data over a significant period of time and if the machine you are running on has multiple cores that can host the script block threads. In this case the -ThrottleLimit parameter should be set approximately to the number of available cores. If you are running on a VM with a single core, then it makes little sense to run high compute script blocks in parallel since the system must serialize them anyway to run on the single core.

Scripts that do a lot of file operations, or perform operations on external machines can benefit by running in parallel. Since the running script cannot use all of the machine cores, it makes sense to set the -ThrottleLimit parameter to something greater than the number of cores. If one script execution waits many minutes to complete, you may want to allow tens or hundreds of scripts to run in parallel..

1..5 | ForEach-Object -Parallel { "Hello $_"; sleep 1; } -ThrottleLimit 5 
Hello 1 
Hello 3 
Hello 2 
Hello 4 
Hello 5

Parallel processing is an ideal solution when you want to run the jobs that are independent of each other.

 

Performance test

#% -> ForEach-Object
Measure-Command {1..5 | % {Start-Sleep 1} } | Select-Object TotalSeconds
#Job
Measure-Command {1..5 | % {Start-Job {Start-Sleep 1}} | Wait-Job} | Select-Object TotalSeconds
#Thread Job
Measure-Command {1..5 | % {Start-ThreadJob -ThrottleLimit 5 {Start-Sleep 1}} | Wait-Job} | Select-Object TotalSeconds
#ForEach-Object Parallel
Measure-Command {1..5 | ForEach-Object -Parallel {Start-Sleep 1} -ThrottleLimit 5} | Select-Object TotalSeconds

% represents forech

  • Regular foreach command took almost 5 sec to run sequentially, each iteration took one second to complete the execution.
  • With PS Job it took 7 secs. (2s Overhead of starting jobs assigning runspace etc.)
  • With PS ThreadJob it took 1 sec, all executed asynchronously and executed within 1 sec. (background job created and we will need to remove it manually)
  • Parallel execution of foreach also completed within a second as runs based on throttle limit which should be set as per the CPU cores.

Scripts attached (Executed with VS code)

  1. Folder Copy (with Thread Job vs Regular way)
  2. API call with parallel foreach
References:

 

Team Triveni (Technical)
About Triveni

Triveni is such strong and trustworthy name in the field of customised IT solution provider which is earnest name in Indian IT sector.

Read More

Leave a Reply

Your email address will not be published. Required fields are marked *

Privacy Settings
We use cookies to enhance your experience while using our website. If you are using our Services via a browser you can restrict, block or remove cookies through your web browser settings. We also use content and scripts from third parties that may use tracking technologies. You can selectively provide your consent below to allow such third party embeds. For complete information about the cookies we use, data we collect and how we process them, please check our Privacy Policy
Youtube
Consent to display content from Youtube
Vimeo
Consent to display content from Vimeo
Google Maps
Consent to display content from Google
Spotify
Consent to display content from Spotify
Sound Cloud
Consent to display content from Sound