Failed jobs do not become expired to allow you to re-queue them without any
time pressure. You should re-queue or delete them manually, or apply AutomaticRetry(OnAttemptsExceeded = AttemptsExceededAction.Delete)
attribute to delete them automatically.
An exception occurred during performance of the job. Fewer details...
07/04/2024 01:09:06
System.Threading.Tasks.TaskCanceledException
The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
---> System.TimeoutException: A task was canceled.
---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationTokencancellationToken)
at System.Net.Http.HttpConnectionPool.HttpConnectionWaiter`1.WaitForConnectionAsync(Booleanasync, CancellationTokenrequestCancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessagerequest, Booleanasync, BooleandoRequestAuth, CancellationTokencancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessagerequest, Booleanasync, CancellationTokencancellationToken)
at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessagerequest, HttpCompletionOptioncompletionOption, CancellationTokenSourcects, BooleandisposeCts, CancellationTokenSourcependingRequestsCts, CancellationTokenoriginalCancellationToken)
--- End of inner exception stack trace ---
--- End of inner exception stack trace ---
at System.Net.Http.HttpClient.HandleFailure(Exceptione, BooleantelemetryStarted, HttpResponseMessageresponse, CancellationTokenSourcects, CancellationTokencancellationToken, CancellationTokenSourcependingRequestsCts)
at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessagerequest, HttpCompletionOptioncompletionOption, CancellationTokenSourcects, BooleandisposeCts, CancellationTokenSourcependingRequestsCts, CancellationTokenoriginalCancellationToken)
at MR.Schedules.DeleteDraftJobsTask.StartDeleteDraftJobs() in /var/www/ManageRight/src/BackEnd/MR.Schedules/Tasks/DeleteDraftJobsTask.cs:line 37
at InvokeStub_TaskAwaiter.GetResult(Object,Object, IntPtr*)
at System.Reflection.MethodInvoker.Invoke(Objectobj, IntPtr*args, BindingFlagsinvokeAttr)
An exception occurred during performance of the job. More details...
07/04/2024 01:07:15
System.Threading.Tasks.TaskCanceledException
The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
---> System.TimeoutException: A task was canceled.
---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationTokencancellationToken)
at System.Net.Http.HttpConnectionPool.HttpConnectionWaiter`1.WaitForConnectionAsync(Booleanasync, CancellationTokenrequestCancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessagerequest, Booleanasync, BooleandoRequestAuth, CancellationTokencancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessagerequest, Booleanasync, CancellationTokencancellationToken)
at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessagerequest, HttpCompletionOptioncompletionOption, CancellationTokenSourcects, BooleandisposeCts, CancellationTokenSourcependingRequestsCts, CancellationTokenoriginalCancellationToken)
--- End of inner exception stack trace ---
--- End of inner exception stack trace ---
at System.Net.Http.HttpClient.HandleFailure(Exceptione, BooleantelemetryStarted, HttpResponseMessageresponse, CancellationTokenSourcects, CancellationTokencancellationToken, CancellationTokenSourcependingRequestsCts)
at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessagerequest, HttpCompletionOptioncompletionOption, CancellationTokenSourcects, BooleandisposeCts, CancellationTokenSourcependingRequestsCts, CancellationTokenoriginalCancellationToken)
at MR.Schedules.SendNotificationsTask.StartJobSendNotifications() in /var/www/ManageRight/src/BackEnd/MR.Schedules/Tasks/SendNotificationsTask.cs:line 36
at InvokeStub_TaskAwaiter.GetResult(Object,Object, IntPtr*)
at System.Reflection.MethodInvoker.Invoke(Objectobj, IntPtr*args, BindingFlagsinvokeAttr)
An exception occurred during performance of the job. More details...
07/04/2024 01:06:14
System.Net.Http.HttpRequestException
Response status code does not indicate success: 504 (Gateway Time-out).
System.Net.Http.HttpRequestException: Response status code does not indicate success: 504 (Gateway Time-out).
at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
at MR.Schedules.SendNotificationsTask.StartJobSendNotifications() in /var/www/ManageRight/src/BackEnd/MR.Schedules/Tasks/SendNotificationsTask.cs:line 37
at InvokeStub_TaskAwaiter.GetResult(Object,Object, IntPtr*)
at System.Reflection.MethodInvoker.Invoke(Objectobj, IntPtr*args, BindingFlagsinvokeAttr)
An exception occurred during performance of the job. More details...
07/04/2024 01:02:32
System.Net.Http.HttpRequestException
Response status code does not indicate success: 504 (Gateway Time-out).
System.Net.Http.HttpRequestException: Response status code does not indicate success: 504 (Gateway Time-out).
at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
at MR.Schedules.SendNotificationsTask.StartJobSendNotifications() in /var/www/ManageRight/src/BackEnd/MR.Schedules/Tasks/SendNotificationsTask.cs:line 37
at InvokeStub_TaskAwaiter.GetResult(Object,Object, IntPtr*)
at System.Reflection.MethodInvoker.Invoke(Objectobj, IntPtr*args, BindingFlagsinvokeAttr)
An exception occurred during performance of the job. More details...
07/04/2024 01:01:28
System.Net.Http.HttpRequestException
An error occurred while sending the request.
System.Net.Http.HttpRequestException: An error occurred while sending the request.
---> System.IO.IOException: Unable to write data to the transport connection: Broken pipe.
---> System.Net.Sockets.SocketException (32): Broken pipe
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.CreateException(SocketErrorerror, BooleanforAsyncThrow)
at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.SendAsyncForNetworkStream(Socketsocket, CancellationTokencancellationToken)
at System.Net.Sockets.NetworkStream.WriteAsync(ReadOnlyMemory`1buffer, CancellationTokencancellationToken)
at System.Net.Security.SslStream.WriteSingleChunk[TIOAdapter](ReadOnlyMemory`1buffer, CancellationTokencancellationToken)
at System.Net.Security.SslStream.WriteAsyncInternal[TIOAdapter](ReadOnlyMemory`1buffer, CancellationTokencancellationToken)
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine&stateMachine)
at System.Net.Security.SslStream.WriteAsync(ReadOnlyMemory`1buffer, CancellationTokencancellationToken)
at System.Net.Http.HttpConnection.WriteToStreamAsync(ReadOnlyMemory`1source, Booleanasync)
at System.Net.Http.HttpConnection.FlushAsync(Booleanasync)
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessagerequest, Booleanasync, CancellationTokencancellationToken)
at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine&stateMachine)
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessagerequest, Booleanasync, CancellationTokencancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessagerequest, Booleanasync, BooleandoRequestAuth, CancellationTokencancellationToken)
at System.Threading.ExecutionContext.RunInternal(ExecutionContextexecutionContext, ContextCallbackcallback, Objectstate)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(ThreadthreadPoolThread)
at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBoxbox, BooleanallowInlining)
at System.Threading.Tasks.Task.RunContinuations(ObjectcontinuationObject)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1task, TResultresult)
at System.Net.Http.HttpConnectionPool.HttpConnectionWaiter`1.WaitForConnectionAsync(Booleanasync, CancellationTokenrequestCancellationToken)
at System.Threading.ExecutionContext.RunInternal(ExecutionContextexecutionContext, ContextCallbackcallback, Objectstate)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(ThreadthreadPoolThread)
at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBoxbox, BooleanallowInlining)
at System.Threading.Tasks.Task.RunContinuations(ObjectcontinuationObject)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetExistingTaskResult(Task`1task, TResultresult)
at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationTokencancellationToken)
at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(ThreadthreadPoolThread, ExecutionContextexecutionContext, ContextCallbackcallback, Objectstate)
at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(ThreadthreadPoolThread)
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
--- End of stack trace from previous location ---
--- End of inner exception stack trace ---
at System.Net.Security.SslStream.<WriteSingleChunk>g__CompleteWriteAsync|153_1[TIOAdapter](ValueTaskwriteTask, Byte[]bufferToReturn)
at System.Net.Security.SslStream.WriteAsyncInternal[TIOAdapter](ReadOnlyMemory`1buffer, CancellationTokencancellationToken)
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessagerequest, Booleanasync, CancellationTokencancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessagerequest, Booleanasync, CancellationTokencancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessagerequest, Booleanasync, BooleandoRequestAuth, CancellationTokencancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessagerequest, Booleanasync, CancellationTokencancellationToken)
at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessagerequest, HttpCompletionOptioncompletionOption, CancellationTokenSourcects, BooleandisposeCts, CancellationTokenSourcependingRequestsCts, CancellationTokenoriginalCancellationToken)
at MR.Schedules.SendNotificationsTask.StartJobSendNotifications() in /var/www/ManageRight/src/BackEnd/MR.Schedules/Tasks/SendNotificationsTask.cs:line 36
at InvokeStub_TaskAwaiter.GetResult(Object,Object, IntPtr*)
at System.Reflection.MethodInvoker.Invoke(Objectobj, IntPtr*args, BindingFlagsinvokeAttr)
An exception occurred during performance of the job. More details...
07/04/2024 01:01:26
System.Threading.Tasks.TaskCanceledException
The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
---> System.TimeoutException: A task was canceled.
---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationTokencancellationToken)
at System.Net.Http.HttpConnectionPool.HttpConnectionWaiter`1.WaitForConnectionAsync(Booleanasync, CancellationTokenrequestCancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessagerequest, Booleanasync, BooleandoRequestAuth, CancellationTokencancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessagerequest, Booleanasync, CancellationTokencancellationToken)
at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessagerequest, HttpCompletionOptioncompletionOption, CancellationTokenSourcects, BooleandisposeCts, CancellationTokenSourcependingRequestsCts, CancellationTokenoriginalCancellationToken)
--- End of inner exception stack trace ---
--- End of inner exception stack trace ---
at System.Net.Http.HttpClient.HandleFailure(Exceptione, BooleantelemetryStarted, HttpResponseMessageresponse, CancellationTokenSourcects, CancellationTokencancellationToken, CancellationTokenSourcependingRequestsCts)
at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessagerequest, HttpCompletionOptioncompletionOption, CancellationTokenSourcects, BooleandisposeCts, CancellationTokenSourcependingRequestsCts, CancellationTokenoriginalCancellationToken)
at MR.Schedules.DeleteDraftJobsTask.StartDeleteDraftJobs() in /var/www/ManageRight/src/BackEnd/MR.Schedules/Tasks/DeleteDraftJobsTask.cs:line 37
at InvokeStub_TaskAwaiter.GetResult(Object,Object, IntPtr*)
at System.Reflection.MethodInvoker.Invoke(Objectobj, IntPtr*args, BindingFlagsinvokeAttr)
An exception occurred during performance of the job. More details...
07/04/2024 01:01:26
System.Threading.Tasks.TaskCanceledException
The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
---> System.TimeoutException: A task was canceled.
---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationTokencancellationToken)
at System.Net.Http.HttpConnectionPool.HttpConnectionWaiter`1.WaitForConnectionAsync(Booleanasync, CancellationTokenrequestCancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessagerequest, Booleanasync, BooleandoRequestAuth, CancellationTokencancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessagerequest, Booleanasync, CancellationTokencancellationToken)
at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessagerequest, HttpCompletionOptioncompletionOption, CancellationTokenSourcects, BooleandisposeCts, CancellationTokenSourcependingRequestsCts, CancellationTokenoriginalCancellationToken)
--- End of inner exception stack trace ---
--- End of inner exception stack trace ---
at System.Net.Http.HttpClient.HandleFailure(Exceptione, BooleantelemetryStarted, HttpResponseMessageresponse, CancellationTokenSourcects, CancellationTokencancellationToken, CancellationTokenSourcependingRequestsCts)
at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessagerequest, HttpCompletionOptioncompletionOption, CancellationTokenSourcects, BooleandisposeCts, CancellationTokenSourcependingRequestsCts, CancellationTokenoriginalCancellationToken)
at MR.Schedules.SendNotificationsTask.StartJobSendNotifications() in /var/www/ManageRight/src/BackEnd/MR.Schedules/Tasks/SendNotificationsTask.cs:line 36
at InvokeStub_TaskAwaiter.GetResult(Object,Object, IntPtr*)
at System.Reflection.MethodInvoker.Invoke(Objectobj, IntPtr*args, BindingFlagsinvokeAttr)
An exception occurred during performance of the job. More details...
07/04/2024 01:01:26
System.Threading.Tasks.TaskCanceledException
The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
---> System.TimeoutException: A task was canceled.
---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationTokencancellationToken)
at System.Net.Http.HttpConnectionPool.HttpConnectionWaiter`1.WaitForConnectionAsync(Booleanasync, CancellationTokenrequestCancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessagerequest, Booleanasync, BooleandoRequestAuth, CancellationTokencancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessagerequest, Booleanasync, CancellationTokencancellationToken)
at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessagerequest, HttpCompletionOptioncompletionOption, CancellationTokenSourcects, BooleandisposeCts, CancellationTokenSourcependingRequestsCts, CancellationTokenoriginalCancellationToken)
--- End of inner exception stack trace ---
--- End of inner exception stack trace ---
at System.Net.Http.HttpClient.HandleFailure(Exceptione, BooleantelemetryStarted, HttpResponseMessageresponse, CancellationTokenSourcects, CancellationTokencancellationToken, CancellationTokenSourcependingRequestsCts)
at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessagerequest, HttpCompletionOptioncompletionOption, CancellationTokenSourcects, BooleandisposeCts, CancellationTokenSourcependingRequestsCts, CancellationTokenoriginalCancellationToken)
at MR.Schedules.DeleteDraftJobsTask.StartDeleteDraftJobs() in /var/www/ManageRight/src/BackEnd/MR.Schedules/Tasks/DeleteDraftJobsTask.cs:line 37
at InvokeStub_TaskAwaiter.GetResult(Object,Object, IntPtr*)
at System.Reflection.MethodInvoker.Invoke(Objectobj, IntPtr*args, BindingFlagsinvokeAttr)
An exception occurred during performance of the job. More details...
07/04/2024 01:01:26
System.Threading.Tasks.TaskCanceledException
The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
---> System.TimeoutException: A task was canceled.
---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationTokencancellationToken)
at System.Net.Http.HttpConnectionPool.HttpConnectionWaiter`1.WaitForConnectionAsync(Booleanasync, CancellationTokenrequestCancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessagerequest, Booleanasync, BooleandoRequestAuth, CancellationTokencancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessagerequest, Booleanasync, CancellationTokencancellationToken)
at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessagerequest, HttpCompletionOptioncompletionOption, CancellationTokenSourcects, BooleandisposeCts, CancellationTokenSourcependingRequestsCts, CancellationTokenoriginalCancellationToken)
--- End of inner exception stack trace ---
--- End of inner exception stack trace ---
at System.Net.Http.HttpClient.HandleFailure(Exceptione, BooleantelemetryStarted, HttpResponseMessageresponse, CancellationTokenSourcects, CancellationTokencancellationToken, CancellationTokenSourcependingRequestsCts)
at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessagerequest, HttpCompletionOptioncompletionOption, CancellationTokenSourcects, BooleandisposeCts, CancellationTokenSourcependingRequestsCts, CancellationTokenoriginalCancellationToken)
at MR.Schedules.SendNotificationsTask.StartJobSendNotifications() in /var/www/ManageRight/src/BackEnd/MR.Schedules/Tasks/SendNotificationsTask.cs:line 36
at InvokeStub_TaskAwaiter.GetResult(Object,Object, IntPtr*)
at System.Reflection.MethodInvoker.Invoke(Objectobj, IntPtr*args, BindingFlagsinvokeAttr)
An exception occurred during performance of the job. More details...
07/04/2024 00:58:19
System.Threading.Tasks.TaskCanceledException
The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
---> System.TimeoutException: A task was canceled.
---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationTokencancellationToken)
at System.Net.Http.HttpConnectionPool.HttpConnectionWaiter`1.WaitForConnectionAsync(Booleanasync, CancellationTokenrequestCancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessagerequest, Booleanasync, BooleandoRequestAuth, CancellationTokencancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessagerequest, Booleanasync, CancellationTokencancellationToken)
at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessagerequest, HttpCompletionOptioncompletionOption, CancellationTokenSourcects, BooleandisposeCts, CancellationTokenSourcependingRequestsCts, CancellationTokenoriginalCancellationToken)
--- End of inner exception stack trace ---
--- End of inner exception stack trace ---
at System.Net.Http.HttpClient.HandleFailure(Exceptione, BooleantelemetryStarted, HttpResponseMessageresponse, CancellationTokenSourcects, CancellationTokencancellationToken, CancellationTokenSourcependingRequestsCts)
at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessagerequest, HttpCompletionOptioncompletionOption, CancellationTokenSourcects, BooleandisposeCts, CancellationTokenSourcependingRequestsCts, CancellationTokenoriginalCancellationToken)
at MR.Schedules.DeleteDraftJobsTask.StartDeleteDraftJobs() in /var/www/ManageRight/src/BackEnd/MR.Schedules/Tasks/DeleteDraftJobsTask.cs:line 37
at InvokeStub_TaskAwaiter.GetResult(Object,Object, IntPtr*)
at System.Reflection.MethodInvoker.Invoke(Objectobj, IntPtr*args, BindingFlagsinvokeAttr)