Problems robots Crawler traps!! 2 Sites

The sites jt-autobekleding.nl and dutchautomotiveparts.nl are currently using a lot of server memory. This is causing by crawler traps. How can I fix this? Every couple days the sites are offline because of internal 500 errors.

See logging from the server

# dutchautomotiveparts.nl.log
66.249.73.199 – – [26/Sep/2025:00:35:03 +0200] “GET /de/shop/?filter_materiaal=stoff-anthranzit&filter_type-stoel=beifahrer-sitz,fahrer-sitz-de&filter_bouwjaar=2003-de,2005-de,2006-de,2007-de,2013-de,2014-de,2017-de,2021-de HTTP/1.1” 503 506 “-” “Mozilla/5.0 (Linux; And
roid 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0.7258.127 Mobile Safari/537.36 (compatible; GoogleOther)”
65.108.99.45 – – [26/Sep/2025:00:35:09 +0200] “GET /product-categorie/stoelhoezen/?filter_merk=volkswagen&add-to-cart=11776 HTTP/1.1” 200 171084 “https://www.dutchautomotiveparts.nl/product-categorie/stoelhoezen/?filter_merk=volkswagen” “Mozilla/5.0 (compatible; Seekpor
tBot; +https://bot.seekport.com)”
66.249.73.199 – – [26/Sep/2025:00:35:07 +0200] “GET /de/shop/?filter_type-stoel=beifahrer-sitz,fahrer-sitz-de&filter_bouwjaar=2004-de,2005-de,2006-de,2008-de,2010-de,2012-de,2014-de,2015-de,2019-de,2020-de,2022-de HTTP/1.1” 503 506 “-” “Mozilla/5.0 (Linux; Android 6.0.1
; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0.7258.127 Mobile Safari/537.36 (compatible; GoogleOther)”
2a03:2880:f800:35:: – – [26/Sep/2025:00:35:11 +0200] “GET /winkel/?filter_type-stoel=bestuurdersstoel&filter_bouwjaar=2001,2002,2004,2005,2006,2007,2012&filter_merk=opel HTTP/2.0” 503 506 “-” “meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmast
ers/crawler)”
66.249.73.199 – – [26/Sep/2025:00:35:13 +0200] “GET /de/shop/?filter_type-stoel=fahrer-sitz-de&filter_bouwjaar=2002-de%2C2003-de%2C2007-de%2C2013-de&add-to-cart=14026 HTTP/1.1” 503 506 “-” “Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KH
TML, like Gecko) Chrome/139.0.7258.127 Mobile Safari/537.36 (compatible; GoogleOther)”
65.108.99.45 – – [26/Sep/2025:00:35:13 +0200] “GET /product-categorie/stoelhoezen/?filter_merk=volkswagen&add-to-cart=11770 HTTP/1.1” 200 171090 “https://www.dutchautomotiveparts.nl/product-categorie/stoelhoezen/?filter_merk=volkswagen” “Mozilla/5.0 (compatible; Seekpor
tBot; +https://bot.seekport.com)”
114.119.132.59 – – [26/Sep/2025:00:35:14 +0200] “GET /en/product-categorie/seat-covers/?filter_materiaal=artificial-leather&filter_bouwjaar=2010-en,2012-en,2014-en,2015-en,2016-en,2017-en,2020-en,2022-en HTTP/1.1” 503 506 “https://www.dutchautomotiveparts.nl/en/product-
categorie/seat-covers/?filter_materiaal=artificial-leather&filter_bouwjaar=2012-en%2C2014-en%2C2015-en%2C2016-en%2C2017-en%2C2020-en%2C2022-en” “Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://
webmaster.petalsearch.com/site/petalbot)”
66.249.73.199 – – [26/Sep/2025:00:35:16 +0200] “GET /de/shop/?filter_bouwjaar=2005-de%2C2007-de%2C2013-de%2C2014-de&filter_type-stoel=fahrer-sitz-de&filter_merk=nissan-de&add-to-cart=13734 HTTP/1.1” 503 506 “-” “Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0.7258.127 Mobile Safari/537.36 (compatible; GoogleOther)”
65.108.99.45 – – [26/Sep/2025:00:35:20 +0200] “GET /product-categorie/stoelhoezen/?filter_merk=volkswagen&add-to-cart=10601 HTTP/1.1” 200 171087 “https://www.dutchautomotiveparts.nl/product-categorie/stoelhoezen/?filter_merk=volkswagen” “Mozilla/5.0 (compatible; Seekpor
tBot; +https://bot.seekport.com)”

# jt-autobekleding.nl
6.249.74.14 – – [26/Sep/2025:00:35:14 +0200] “GET /?taxonomy=product_shipping_class&term=nl&filter_bouwjaar=1993,1996,2002,2004,2007,2008,2009,2010,2013,2014,2015,2016&filter_type-auto=movano,t5,t6 HTTP/1.1” 200 22114 “-” “Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0.7258.127 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”
66.249.74.14 – – [26/Sep/2025:00:35:17 +0200] “GET /product-categorie/autobekleding/?filter_type-auto=5-serie,t5,t5-gp,vivaro&filter_bouwjaar=1993,2003,2005,2008,2010,2012,2013 HTTP/1.1” 200 22070 “-” “Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0.7258.127 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”
114.119.135.98 – – [26/Sep/2025:00:35:22 +0200] “GET /de/product-categorie/autopolsterung-de/sitzueberzug/?filter_type-auto=movano-de,t4-de&filter_bouwjaar=1995-de,1996-de,2002-de,2010-de,2015-de HTTP/1.1” 200 22276 “https://jt-autobekleding.nl/de/product-categorie/autopolsterung-de/sitzueberzug/?filter_type-auto=movano-de%2Ct4-de&filter_bouwjaar=1995-de%2C1996-de%2C2010-de%2C2015-de” “Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot)”
66.249.74.13 – – [26/Sep/2025:00:35:29 +0200] “GET /product-categorie/autobekleding/?filter_type-auto=5-serie,t5-gp&filter_bouwjaar=1994,1995,2000,2003,2010,2012,2013 HTTP/1.1” 200 21980 “-” “Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0.7258.127 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”
66.249.74.13 – – [26/Sep/2025:00:35:31 +0200] “GET /?taxonomy=product_shipping_class&term=nl&filter_bouwjaar=1994,1995,1996,2004,2006,2007,2011,2013,2016&filter_type-auto=daily,lt,primastar,schuim,sprinter,t5,t6,transit,vito,xf-cf HTTP/1.1” 200 22249 “-” “Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0.7258.127 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”
66.249.74.13 – – [26/Sep/2025:00:35:31 +0200] “GET /?taxonomy=product_shipping_class&term=nl&filter_bouwjaar=1993,1995,2002,2007,2008,2013,2014,2015,2016,2017&filter_type-auto=daily,r-serie,t4 HTTP/1.1” 200 22050 “-” “Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0.7258.127 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”
114.119.137.64 – – [26/Sep/2025:00:35:35 +0200] “GET /de/product-categorie/autopolsterung-de/sitzueberzug/?filter_type-auto=movano-de,sprinter-de,trafic-de,xf-cf-de&filter_bouwjaar=1993-de,2000-de,2009-de,2017-de HTTP/1.1” 200 22320 “https://jt-autobekleding.nl/de/product-categorie/autopolsterung-de/sitzueberzug/?filter_bouwjaar=1993-de%2C2000-de%2C2009-de&filter_type-auto=movano-de%2Csprinter-de%2Ctrafic-de%2Cxf-cf-de” “Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot)”
66.249.74.13 – – [26/Sep/2025:00:35:44 +0200] “GET /?taxonomy=product_shipping_class&term=nl&filter_bouwjaar=1994,1995,1997,1998,1999,2000,2001,2002,2003,2004,2008,2009,2010,2013,2014,2017&filter_type-auto=6-serie,ducato,master,t5,xf-cf HTTP/1.1” 200 22249 “-” “Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0.7258.127 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”
66.249.74.13 – – [26/Sep/2025:00:35:49 +0200] “GET /product-categorie/autobekleding/?filter_type-auto=5-serie,t5,t5-gp,vivaro&filter_bouwjaar=1993,2003,2008,2012,2013 HTTP/1.1” 200 21988 “-” “Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0.7258.127 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”
114.119.150.249 – – [26/Sep/2025:00:35:54 +0200] “GET /de/product-categorie/autopolsterung-de/sitzueberzug/?filter_type-auto=movano-de,schaum,sprinter-de,t4-de&filter_bouwjaar=2004-de,2010-de HTTP/1.1” 200 22224 “https://jt-autobekleding.nl/de/product-categorie/autopolsterung-de/sitzueberzug/?filter_type-auto=movano-de%2Cschaum%2Csprinter-de%2Ct4-de&filter_bouwjaar=1995-de%2C2004-de%2C2010-de” “Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot)”

Hello

The issue you are experiencing with crawler traps causing high server memory usage and 500 errors is not related to the WoodMart theme itself. These crawler traps occur due to bots repeatedly accessing URLs with many filter combinations, which can overload your server.

To mitigate this problem, we recommend implementing bot management and protection services such as Cloudflare or similar CDN and firewall solutions. These services can help identify and block abusive crawlers, limit request rates, and reduce server load.

regards

[Author]

Posts