Skip to main content
Name:urldedupe
Category:Utilities
Publisher:trickest
Created:6/23/2021
Container:quay.io/trickest/urldedupe:cc9b25a
Output Type:
License:Unknown

Parameters

​
mode
string
-mThe mode/filters to be enabled (can be 1 or more, comma separated). Default is none, available options are the other flags (--mode r,s,qs,ne)
​
urls-file
file
required
-uFile containing urls
​
regex-parse
boolean
-rThis is significantly slower than normal parsing, but may be more thorough or accurate
​
no-extensions
boolean
-neDo not include URLs if they have an extension (i.e. .png, .jpg, .woff, .js, .html)
​
query-strings-only
boolean
-qsOnly include URLs if they have query strings
​
remove-similar-urls
boolean
-sRemove similar URLs (based on integers and image/font files) - i.e. /api/user/1 & /api/user/2 deduplicated