Встроенный антивирус Exchange mail flow breaks (Disable AntiMalwareScanning)

Настройка встроенного антивируса

Для того, чтобы встроенный антивирус забирал обновления с правильного URL по HTTPS, нужно выполнить команду ниже. При необходимости можно так же настроить частоту обновления и время ожидания. Не забудьте предоставить соответствующие сетевые доступы.

Set-MalwareFilteringServer EXCH19 -PrimaryUpdatePath https://forefrontdl.microsoft.com/server/scanengineupdate/ -UpdateFrequency 120 -DeferWaitTime 10

В случае необходимости отключения встроенного антивируса – можно для начала проверить, что встроенный агент включен (Malware Agent  - True), командой:

Get-TransportAgent

При необходимости выключить скриптом:

C:\Program Files\Microsoft\Exchange Server\V15\scripts>.\Disable-AntimalwareScanning.ps1

https://serverfault.com/questions/1085840/exchange-2019-antimalware-engine-updates-download-but-dont-get-applied 

https://techcommunity.microsoft.com/discussions/exchange_general/exchange-20162019-filtering-engine-updates-failing-since-1272021/3037138

https://www.alitajran.com/exchange-mail-flow-breaks/

Exchange mail flow breaks (Disable AntiMalwareScanning)

• Updated on June 27, 2023
 
• Exchange
 
• 142 Comments

Mail flow seems not to be working and the mail queue is getting bigger. If you see these issues, disable AntiMalWareScanning for now on the on-premises Exchange Servers and restart the Microsoft Exchange Server Transport Service.

Update: Microsoft releases a patch. See the part Solution to Exchange mail flow breaks.

Table of contents

• Introduction
• Get mail queue
• Fix for Exchange mail flow breaks
  • 1a. Disable AntiMalWareScanning
  • 1b. Bypass anti malware filtering
  • 2a. Verify AntiMalwareScanning is disabled
  • 2b. Verify bypass filtering is disabled
  • 3. Restart Exchange Server Transport Service
• Solution to Exchange mail flow breaks
  • 1a. Automated Solution
  • 1b. Manual Solution
  • 2. Verify engine update info
  • 3. Re-enable anti malware scanning
  • 4. Verify mail flow and FIPFS error events not present
• Latest updates

Introduction

The mail flow is not working and you can check the error in Event Viewer.

Error
The FIP-FS “Microsoft” Scan Engine failed to load. PID: 9244, Error Code: 0x80004005. Error Description: Can’t convert “2201010005” to long.

Get mail queue

Run Exchange Management Shell as administrator and check the queue.

[PS] C:\>Get-ExchangeServer | Get-Queue

MICROSOFT RELEASED A PATCH FOR THE EMAIL STUCK IN TRANSPORT QUEUE. SKIP TO THE SOLUTION DOWN BELOW!!

Fix for Exchange mail flow breaks

There are two ways to fix the Exchange mail flow:

1. Disable anti malware scanning (1a+2a)
2. Bypass anti malware filtering (2a+2b)

Both methods will work. Choose one of these options and follow the steps below.

1a. Disable AntiMalWareScanning

In Exchange Management Shell, run the command to disable AntiMalware Scanning on the Exchange Server on-premises.

Note: This command can take a minute or more. Leave it running till it’s successfully run.

[PS] C:\>& $env:ExchangeInstallPath\Scripts\Disable-AntimalwareScanning.ps1

WARNING: The following service restart is required for the change(s) to take effect : MSExchangeTransport
Anti-malware scanning is successfully disabled. Please restart MSExchangeTransport for the changes to take effect.

Note: To re-enable malware filtering, use Enable-Antimalwarescanning.ps1 instead of Disable-Antimalwarescanning.ps1.

You can find the script Disable-AntiMalWareScanning.ps1 in File Explorer.

1b. Bypass anti malware filtering

In Exchange Management Shell, run the command to bypass anti malware filtering on the Exchange Servers on-premises.

[PS] C:\>Get-ExchangeServer | % {Set-MalwareFilteringServer -BypassFiltering $true -Identity $_.Name}

2a. Verify AntiMalwareScanning is disabled

To verify that malware filtering is disabled, run the Get-TransportAgent cmdlet and confirm that it returns a value of False.

[PS] C:\>Get-ExchangeServer | % {Get-TransportAgent "Malware Agent"}

Identity       Enabled  Priority
--------       -------  --------
Malware Agent  False    5

2b. Verify bypass filtering is disabled

To verify that malware bypass filtering is disabled, run the Get-MalwareFilteringServer cmdlet and confirm that it returns a value of True.

[PS] C:\>Get-ExchangeServer | % {Get-MalwareFilteringServer}

Name      ForceRescan BypassFiltering DeferWaitTime DeferAttempts UpdateFrequency PrimaryUpdatePath
----      ----------- --------------- ------------- ------------- --------------- -----------------
EX01-2019 False       True            5             3             30              http://amupdatedl.microsoft.com/server/amupdate

3. Restart Exchange Server Transport Service

Restart the Microsoft Exchange Server Transport service.

[PS] C:\>Restart-Service MSExchangeTransport

Restart the Microsoft Exchange Server Transport service on all Exchange Servers.

[PS] C:\>Get-ExchangeServer | % {Get-Service -ServiceName MSExchangeTransport | Restart-Service -Force}

Read more: Restart Exchange Server services through PowerShell »

The queue will start to process the mail.

Solution to Exchange mail flow breaks

Microsoft did release a patch and you can run that in your environment.

Implementation of the solution requires customer actions. These actions can be automated with the scan engine reset script from https://aka.ms/ResetScanEngineVersion or they can be performed manually.

Important: Whether you perform the steps automatically or manually, they must be performed on every Exchange 2016 and Exchange 2019 server in your organization.

If you use the automated script, you can run it on multiple servers in parallel. We recommend starting with the servers in your environment that have the largest queues. You can use Get-Queue to view queue size and determine which queues are the largest.

Note: If you previously disabled or bypassed antimalware scanning as a mitigation for this issue, we recommend that you re-enable it after performing the steps below.

1a. Automated Solution

Download the ResetScanEngineVersion.ps1 PowerShell script and place it on the Exchange Server C:\scripts folder. If you don’t have a scripts folder, create one.

1. Run Exchange Management Shell as administrator
2. Run Set-ExecutionPolicy -ExecutionPolicy RemoteSigned
3. Change the path to the scripts folder
4. Run the script .\ResetScanEngineVersion.ps1

Important: Run the .\ResetScanEngineVersion.ps1 PowerShell script on each Exchange Server.

Run the script on each Exchange server in your organization. You can run this script on multiple servers in parallel. After the script has completed, you will see the following output:

[PS] C:\>Set-ExecutionPolicy -ExecutionPolicy RemoteSigned
[PS] C:\>cd C:\scripts
[PS] C:\scripts>.\Reset-ScanEngineVersion.ps1
EX01-2019 Stopping services...
EX01-2019 Removing Microsoft engine folder...
EX01-2019 Emptying metadata folder...
EX01-2019 Starting services...
WARNING: Waiting for service 'Microsoft Filtering Management Service (FMS)' to start...
WARNING: Waiting for service 'Microsoft Filtering Management Service (FMS)' to start...
WARNING: Waiting for service 'Microsoft Filtering Management Service (FMS)' to start...
WARNING: Waiting for service 'Microsoft Exchange Transport (MSExchangeTransport)' to start...
EX01-2019 Starting engine update...
Running as EXOIP\administrator.
--------
Connecting to EX01-2019.exoip.local.
Dispatched remote command. Start-EngineUpdate -UpdatePath http://amupdatedl.microsoft.com/server/amupdate
--------

Note: Do you get the error “Cannot stop process” in relation to the updateservice.exe located in the FIP-FS? Ensure that the account you sign in to the Exchange Server has Debug programs user rights under the Local Group Policy. Otherwise, the process will not terminate, and the script fails to run.

1b. Manual Solution

You can can also manually perform steps to resolve the issue and restore service. To manually resolve this issue, you must perform the following steps on each Exchange server in your organization:

Remove existing engine and metadata

1. Stop the Microsoft Filtering Management service. When prompted to also stop the Microsoft Exchange Transport service, click Yes.
2. Use Task Manager to ensure that updateservice.exe is not running.
3. Delete the following folder: %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\amd64\Microsoft.
4. Remove all files from the following folder: %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\metadata.

Update to latest engine

1. Start the Microsoft Filtering Management service and the Microsoft Exchange Transport service.
2. Open the Exchange Management Shell.
3. Navigate to the Scripts folder (%ProgramFiles%\Microsoft\Exchange Server\V15\Scripts).
4. Run .\Update-MalwareFilteringServer.ps1 <server FQDN>

[PS] C:\>cd "$env:ProgramFiles\Microsoft\Exchange Server\V15\Scripts"
[PS] C:\Program Files\Microsoft\Exchange Server\V15\Scripts>.\Update-MalwareFilteringServer.ps1 EX01-2019

Running as EXOIP\administrator.
--------
Connecting to EX01-2019.
Dispatched remote command. Start-EngineUpdate -UpdatePath http://amupdatedl.microsoft.com/server/amupdate
--------

2. Verify engine update info

1. Start Exchange Management Shell.
2. Run Add-PSSnapin Microsoft.Forefront.Filtering.Management.Powershell.
3. Run Get-EngineUpdateInformation and verify the UpdateVersion information is 2112330001 or higher.

Note: Run the Get-EngineUpdateInformation cmdlet and verify the UpdateVersion information is 2112330001 or higher.

[PS] C:\>Add-PSSnapin Microsoft.Forefront.Filtering.Management.Powershell

[PS] C:\>Get-EngineUpdateInformation


Engine            : Microsoft
LastChecked       : 01/02/2022 08:22:33 AM +01:00
LastUpdated       : 01/02/2022 08:22:41 AM +01:00
EngineVersion     : 1.1.18800.4
SignatureVersion  : 1.355.1227.0
SignatureDateTime : 01/01/2022 12:29:06 PM +01:00
UpdateVersion     : 2112330001
UpdateStatus      : UpdateAttemptSuccessful

Filter the Application log on the Event sources: FIPFS.

3. Re-enable anti malware scanning

If you did apply the fix before that Microsoft published a solution, enable anti malware scanning. If you did not apply any fix and you immediately applied the solution, you don’t have to re-enable anti malware scanning because it’s enabled by default.

Did you disable anti malware scanning? Run the command below and restart the Microsoft Exchange Transport Service.

[PS] C:\>& $env:ExchangeInstallPath\Scripts\Enable-AntimalwareScanning.ps1

Anti-malware engines are updating. This may take a few minutes.
Checking for engines updated after 12/26/2021 8:31:11 AM.
Updating Microsoft. Last updated : 1/2/2022 8:22:41 AM
WARNING: The following service restart is required for the change(s) to take effect : MSExchangeTransport
Anti-malware scanning is successfully enabled. Please restart MSExchangeTransport for the changes to take effect.

Did you bypass anti malware scanning? Run the command below and restart the Microsoft Exchange Transport Service.

[PS] C:\>Get-ExchangeServer | % {Set-MalwareFilteringServer -BypassFiltering $false -Identity $_.Name}

4. Verify mail flow and FIPFS error events not present

After updating the engine, we also recommend that you verify that mail flow is working and that FIPFS error events are not present in the Application event log.

Test and verify that mail flow works:

• Test internal mail flow in Exchange Server
• Test inbound mail flow in Exchange Server
• Test outbound mail flow in Exchange Server

Important: Do you see FIPFS errors in the event log and the email are still stuck in the queue/mail flow is not working after running the ResetScanEngineVersion.ps1 PowerShell script? RESTART THE EXCHANGE SERVER.

Latest updates

Update: Microsoft pushed an agent update, but it’s not yet working. So keep anti-malware scanning disabled.

Update1: Updated screenshot with how to check the event viewer for the issue.

Update2: Downloaded the latest MS Filtering Engine Update: Update Version:2201010006, it’s still not working.

MS Filtering Engine Update process has successfully committed and handed off updates for Microsoft
Last Checked:2022-01-01T12:05:10Z
Last Updated:2022-01-01T12:05:18Z
Engine Version:1.1.18800.4
Signature Version:”1.355.1234.0″
Update Version:2201010006
Last Definition Update:‎2022‎-‎01‎-‎01T05:20:29.000Z
Update Path:http://amupdatedl.microsoft.com/server/amupdate

Update3: Added both the options on how to disable anti malware scanning or bypass anti malware filtering.

Update4: MS Filtering Engine update will not fix this issue. You don’t have to check for engine updates. Implement this fix and wait for a proper fix release by Microsoft.

Update5: Microsoft Product Manager sent me a message, and they are working on a fix. The fix will be released with how-to instructions. Stay tuned.

Update6: You can wait for the Microsoft patch if you have enough disk space and don’t mind that Exchange Server will deliver the messages later. If not, apply the fix and patch later when Microsoft releases.

Update7: Microsoft Exchange Server expiration message is by default 48 hours (2 days). If a message remains in the queue for longer than this period of time, the message is returned to the sender as a permanent failure.

You can change that value between 1-90 days. The problem is that the longer days you set, the more messages will stay, the larger the mail queue will become. It means you have to keep adding more disk space.

I recommend running the above fix. But, if you do not want to disable the anti malware scanning, it’s better to expand the message expiration timeout in the Transport Server. Just to be safe.

In the example below, it’s 2 days.

[PS] C:\>Get-TransportServer | select Name,MessageExpirationTimeout

Name      MessageExpirationTimeout
----      ------------------------
EX01-2019 2.00:00:00

Change it to 4 days (or another value).

[PS] C:\>Set-TransportServer -Identity EX01-2019 -MessageExpirationTimeout 4.00:00:00

Update8: Microsoft patch is OUT! Run the automated or manual solution. I recommend the automated solution first.

Update9: If you applied the fix before the solution, don’t forget to re-enable antimalware scanning and restart the Microsoft Exchange Transport Service.

Update10: Updated screenshot with how to check the event viewer after applying the solution.

Update11: The mail.que database got larger in size, and it will not shrink automatically. This happened because the Exchange Server kept the mail in the queue, and the messages didn’t get released before the fix/solution. It can take a lot of space, and I recommend making it smaller by removing the mail.que database. After that, Exchange Server will automatically create a new mail.que database.

Read more: Mail.que file large in size Exchange Server »

Update12: If you didn’t have enough disk space on the OS drive or ran out of disk space because of the mail.que database was filling up, it’s now an excellent time to move the mail.que database to another drive. This way, the OS drive will stay active and not give you performance issues.

Read more: Move mail queue to another location in Exchange Server »

Update13: Added articles to test mail flow internal/inbound/outbound.

Update14: Some admins mention that an Exchange Server reboot is required after applying the Solution. After that, the mail queue gets empty/mail flow starts working.

Update15: Running the Update-MalwareFilteringServer.ps1 cmdlet shows: Error starting the anti-malware engine update. To fix this error, run the steps below:

1. Run Exchange Management Shell as administrator
2. Run Add-PSSnapin Microsoft.Forefront.Filtering.Management.Powershell
3. Run Start-EngineUpdate
4. Run Get-EngineUpdateInformation

Update 16: Only Exchange Server 2016 and Exchange Server 2019 are affected. The Exchange Edge Transport server is not affected by this Microsoft Exchange Y2K22 bug.

Update 17: UpdateVersion is now on 2112330003. Added to the article that it needs to show UpdateVersion 2112330001 or higher.

Update 18: Some admins mention that Exchange Server 2013 is also affected and some say it’s not. This is not true and the mail transport will work. But, you have to run the same script automatically or manually.

If your Exchange 2013 server took the antimalware update and it is now on version starting with “22…” you should use the automated or manual steps in this blog post to get your server on an engine version “21…” to continue getting the antimalware updates. Without taking action your server will not get any future antimalware updates.

Update19: Added to run Set-ExecutionPolicy -ExecutionPolicy RemoteSigned in Exchange Management Shell before running the script.

Update20: The definition download takes 30-45 minutes (maybe slower depending on Internet speed), so allow for some time.

Update21: Added solution to updateservice.exe access denied error when running the script.

Subscribe to the newsletter and stay up to date with the latest IT tips and tricks.

Did this article help you? Share this article with your network and help them too.