How to fix 'The extension package size exceeds the maximum package size' error in your Azure Devops Server

First check that there are no junk files in your vsix (like .git folder). If Found - delete it. If this didn’t help, Run this sql in your Configuration DB:

USE "Tfs_Configuration";
EXEC dbo.prc_SetRegistryValue @partitionId = 1, @key = '#\Configuration\Service\Gallery\LargeExtensionUpload\MaxPackageSizeMB\', @value = 128, @identityName = null;

But change this value carefully, it can affect performance.