How to with Powershell:

Copy this code and paste into a .txt file and rename it to .ps1:

$gm=New-Object -com FSRM.FSRMGlobalStoreManager

$s=[xml]$gm.GetStoreData("Settings", "ReportSettings")

$s.Save("c:\temp\fsrm-reports-backup.xml")

$s.root.MaxFiles="50000"

$gm.SetStoreData("Settings", "ReportSettings",$s.get_InnerXml())

Change MaxFiles to then number you want to be the limit.  CAUTION: 50,000 is very high but I understand the risks.  Be sure you have enough memory and space on the hard drive for the report.  Each line in the report is 2k so on computers now days you should be fine but be sure first.

Next open Powershell and browse to the file. (Example: C:\Temp\IncreaseFSRM.ps1)

Use this command to run the script:

./IncreaseFSRM.ps1

If you get an error then refer to this post to change the exection policy:

http://fabozzi.net/powershell-set-execution-policy/

How to with vbscript:

Copy this code and paste into a .txt file and rename it to .vbs:

const scriptName = "fsrmReportLimit"

DIM limitNames
limitNames = Array("MaxFiles", "MaxFileGroups", "MaxFileOwners", "MaxFilesPerFileGroup", "MaxFilesPerFileOwner", "MaxFilesPerDuplGroup", "MaxDuplicateGroups", "MaxQuotas", "MaxFileScreenEvents")

const optLimit = "/limit"
const optValue = "/value"

DIM objArgs, fsrm, strLimitName, strLimitValue

set objArgs = wscript.Arguments
if objArgs.count = 0 then
PrintUsage()
wscript.quit
end if

if objArgs.count = 1 then
if objArgs(0) = "/?" then
PrintUsage()
wscript.quit
end if
end if

DIM i, j
DIM strOption, strNewOption
DIM nModifyProperties
nModifyProperties = 0
for i = 0 to objArgs.count-1
if (LCase(objArgs(i)) = optLimit) then
strLimitName = objArgs(i+1)
i = i + 1
elseif (LCase(objArgs(i)) = optValue) then
strLimitValue = objArgs(i+1)
i = i + 1
else
wscript.echo "Error: invalid argument, " & objArgs(i)
PrintUsage()
wscript.quit
end if
next

DIM limitNameCode
limitNameCode = -1
for i = LBound(limitNames) to UBound(limitNames)
if (LCase(strLimitName) = LCase(limitNames(i))) then
limitNameCode = i + 1
exit for
end if
next

if (limitNameCode = -1) then
wscript.echo "Error: invalid limit name, " & strLimitName
PrintUsage()
wscript.quit
end if

set fsrm = WScript.createobject("fsrm.FsrmReportManager")

DIM newLimit
call fsrm.SetReportSizeLimit(limitNameCode, strLimitValue)
newLimit = fsrm.GetReportSizeLimit(limitNameCode)

if (Int(newLimit) = Int(strLimitValue)) then
wscript.echo "Report size limit " & limitNames(limitNameCode - 1) & " was changed to " & strLimitValue
else
wscript.echo "unable to change limit " & limitNames(limitNameCode - 1) & ". Limit is set to " & newLimit
end if

function PrintUsage()
wscript.echo ""
wscript.echo scriptName & " /limit [/value "
wscript.echo " - name of the report size limit to modify"
wscript.echo " - new value for the size limit"
wscript.echo ""
wscript.echo "Report limit values:"
for i = LBound(limitNames) to UBound(limitNames)
wscript.echo " " & limitNames(i)
next
end function

Next Open Command Prompt and browse to the location of the saved .vbs file. (Example: C:\Temp\IncreaseFSRM.vbs)

Run this command to change the limit:

wscript increasefsrm.vbs /limit MaxFiles /Value 50000

options for limit values are:

a. MaxFiles
b. MaxFileGroups
c. MaxFileOwners
d. MaxFilesPerFileGroup
e. MaxFilesPerFileOwner
f. MaxFilesPerDuplGroup
g. MaxDuplicateGroups
h. MaxQuotas
i. MaxFileScreenEvents

References:
https://social.technet.microsoft.com/Forums/en-US/ed7dd6af-3fa7-476e-8ac7-87666b3e8f70/how-to-extent-fsrm-storage-reports-limit-of-100-files-to-more-on-2008-r2
https://gallery.technet.microsoft.com/FSRM-Increase-FSRM-report-9caa545e#content
https://social.technet.microsoft.com/Forums/windowsserver/en-US/fe7854c8-52f7-4471-9da3-0b246a97365f/windows-2008-fsrm-quota-limit
https://docs.microsoft.com/en-us/archive/blogs/adioltean/script-recipe-how-to-increase-the-maximum-number-of-quotas-in-the-fsrm-quota-report-w2k3-r2