The current failure tracking mechanism will never request anything that
didn't respond with a success, 403, 404, or 5xx codes. This is causing
issues when using in real fediverse because of weird status codes
some software has and timeouts being frequent. This patch changes
failure tracking mechanism to only never request the url again if it
responded with 400, 204, or the body is too large, otherwise it can be
re-requested in 60 seconds.
