{"@t":"2025-11-12T02:01:14.5656355Z","@mt":"{State:l}","@r":["Execution DelayedJobScheduler is in the Failed state now due to an exception, execution will be retried no more than in 00:00:09"],"@l":"Error","@x":"Microsoft.Data.SqlClient.SqlException (0x80131904): A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)\n   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)\n   at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)\n   at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)\n   at Microsoft.Data.SqlClient.SqlConnection.Open(SqlConnectionOverrides overrides)\n   at Hangfire.SqlServer.SqlServerStorage.CreateAndOpenConnection()\n   at Hangfire.SqlServer.SqlServerConnection.AcquireLock(String resource, TimeSpan timeout)\n   at Hangfire.Server.DelayedJobScheduler.UseConnectionDistributedLock[T](JobStorage storage, Func`2 action)\n   at Hangfire.Server.DelayedJobScheduler.Execute(BackgroundProcessContext context)\n   at Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(Guid executionId, Object state)\n   at Hangfire.Processing.BackgroundExecution.Run(Action`2 callback, Object state)\nClientConnectionId:00000000-0000-0000-0000-000000000000","State":"Execution DelayedJobScheduler is in the Failed state now due to an exception, execution will be retried no more than in 00:00:09","SourceContext":"Hangfire.Processing.BackgroundExecution","ThreadId":40,"EnvironmentName":"Production","ProcessId":2557180,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-12T02:01:14.5656354Z","@mt":"{State:l}","@r":["Execution RecurringJobScheduler is in the Failed state now due to an exception, execution will be retried no more than in 00:00:09"],"@l":"Error","@x":"Microsoft.Data.SqlClient.SqlException (0x80131904): A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)\n   at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)\n   at Microsoft.Data.SqlClient.TdsParser.Connect(ServerInfo serverInfo, SqlInternalConnectionTds connHandler, Boolean ignoreSniOpenTimeout, Int64 timerExpire, SqlConnectionString connectionOptions, Boolean withFailover)\n   at Microsoft.Data.SqlClient.SqlInternalConnectionTds.LoginNoFailover(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString connectionOptions, SqlCredential credential, TimeoutTimer timeout)\n   at Microsoft.Data.SqlClient.SqlInternalConnectionTds.OpenLoginEnlist(TimeoutTimer timeout, SqlConnectionString connectionOptions, SqlCredential credential, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance)\n   at Microsoft.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, Boolean applyTransientFaultHandling, String accessToken, DbConnectionPool pool)\n   at Microsoft.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions)\n   at Microsoft.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)\n   at Microsoft.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)\n   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)\n   at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)\n   at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)\n   at Microsoft.Data.SqlClient.SqlConnection.Open(SqlConnectionOverrides overrides)\n   at Hangfire.SqlServer.SqlServerStorage.CreateAndOpenConnection()\n   at Hangfire.SqlServer.SqlServerConnection.AcquireLock(String resource, TimeSpan timeout)\n   at Hangfire.Server.RecurringJobScheduler.UseConnectionDistributedLock[T](JobStorage storage, Func`2 action)\n   at Hangfire.Server.RecurringJobScheduler.Execute(BackgroundProcessContext context)\n   at Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(Guid executionId, Object state)\n   at Hangfire.Processing.BackgroundExecution.Run(Action`2 callback, Object state)\nClientConnectionId:00000000-0000-0000-0000-000000000000","State":"Execution RecurringJobScheduler is in the Failed state now due to an exception, execution will be retried no more than in 00:00:09","SourceContext":"Hangfire.Processing.BackgroundExecution","ThreadId":41,"EnvironmentName":"Production","ProcessId":2557180,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-12T02:01:17.2218719Z","@mt":"{State:l}","@r":["Server cc923dec-9adf-4cc6-9b48-04479174f16b encountered an exception while sending heartbeat"],"@l":"Warning","@x":"Microsoft.Data.SqlClient.SqlException (0x80131904): A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)\n   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)\n   at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)\n   at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)\n   at Microsoft.Data.SqlClient.SqlConnection.Open(SqlConnectionOverrides overrides)\n   at Hangfire.SqlServer.SqlServerStorage.CreateAndOpenConnection()\n   at Hangfire.SqlServer.SqlServerStorage.UseConnection[T](DbConnection dedicatedConnection, Func`2 func)\n   at Hangfire.Server.ServerHeartbeatProcess.Execute(BackgroundProcessContext context)\nClientConnectionId:00000000-0000-0000-0000-000000000000","State":"Server cc923dec-9adf-4cc6-9b48-04479174f16b encountered an exception while sending heartbeat","SourceContext":"Hangfire.Server.ServerHeartbeatProcess","ThreadId":14,"EnvironmentName":"Production","ProcessId":2557180,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-12T02:01:17.2217281Z","@mt":"{State:l}","@r":["Execution Worker is in the Failed state now due to an exception, execution will be retried no more than in 00:05:00"],"@l":"Error","@x":"Microsoft.Data.SqlClient.SqlException (0x80131904): A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)\n   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)\n   at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)\n   at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)\n   at Microsoft.Data.SqlClient.SqlConnection.Open(SqlConnectionOverrides overrides)\n   at Hangfire.SqlServer.SqlServerStorage.CreateAndOpenConnection()\n   at Hangfire.SqlServer.SqlServerStorage.UseConnection[T](DbConnection dedicatedConnection, Func`2 func)\n   at Hangfire.SqlServer.SqlServerJobQueue.DequeueUsingSlidingInvisibilityTimeout(String[] queues, CancellationToken cancellationToken)\n   at Hangfire.SqlServer.SqlServerJobQueue.Dequeue(String[] queues, CancellationToken cancellationToken)\n   at Hangfire.SqlServer.SqlServerConnection.FetchNextJob(String[] queues, CancellationToken cancellationToken)\n   at Hangfire.Server.Worker.Execute(BackgroundProcessContext context)\n   at Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(Guid executionId, Object state)\n   at Hangfire.Processing.BackgroundExecution.Run(Action`2 callback, Object state)\nClientConnectionId:00000000-0000-0000-0000-000000000000","State":"Execution Worker is in the Failed state now due to an exception, execution will be retried no more than in 00:05:00","SourceContext":"Hangfire.Processing.BackgroundExecution","ThreadId":26,"EnvironmentName":"Production","ProcessId":2557180,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-12T02:01:46.2101700Z","@mt":"{State:l}","@r":["Server cc923dec-9adf-4cc6-9b48-04479174f16b caught stopping signal..."],"State":"Server cc923dec-9adf-4cc6-9b48-04479174f16b caught stopping signal...","SourceContext":"Hangfire.Server.BackgroundServerProcess","ThreadId":130,"EnvironmentName":"Production","ProcessId":2557180,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-12T02:01:46.2108205Z","@mt":"{State:l}","@r":["Execution loop Worker:fd261bb8 was unable to wait for '00:04:16' delay due to an exception. Execution will be stopped."],"@l":"Fatal","@x":"System.OperationCanceledException: The operation was canceled.\n   at System.Threading.CancellationToken.ThrowOperationCanceledException()\n   at System.Threading.CancellationToken.ThrowIfCancellationRequested()\n   at Hangfire.Processing.TaskExtensions.WaitOne(WaitHandle waitHandle, TimeSpan timeout, CancellationToken token)\n   at Hangfire.Processing.BackgroundExecution.HandleDelay(Guid executionId, TimeSpan delay)","State":"Execution loop Worker:fd261bb8 was unable to wait for '00:04:16' delay due to an exception. Execution will be stopped.","SourceContext":"Hangfire.Processing.BackgroundExecution","ThreadId":36,"EnvironmentName":"Production","ProcessId":2557180,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-12T02:01:46.2109268Z","@mt":"{State:l}","@r":["Execution loop Worker:cb0349a3 was unable to wait for '00:05:00' delay due to an exception. Execution will be stopped."],"@l":"Fatal","@x":"System.OperationCanceledException: The operation was canceled.\n   at System.Threading.CancellationToken.ThrowOperationCanceledException()\n   at System.Threading.CancellationToken.ThrowIfCancellationRequested()\n   at Hangfire.Processing.TaskExtensions.WaitOne(WaitHandle waitHandle, TimeSpan timeout, CancellationToken token)\n   at Hangfire.Processing.BackgroundExecution.HandleDelay(Guid executionId, TimeSpan delay)","State":"Execution loop Worker:cb0349a3 was unable to wait for '00:05:00' delay due to an exception. Execution will be stopped.","SourceContext":"Hangfire.Processing.BackgroundExecution","ThreadId":26,"EnvironmentName":"Production","ProcessId":2557180,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-12T02:01:46.2109676Z","@mt":"{State:l}","@r":["Execution loop Worker:2922fdc1 was unable to wait for '00:05:00' delay due to an exception. Execution will be stopped."],"@l":"Fatal","@x":"System.OperationCanceledException: The operation was canceled.\n   at System.Threading.CancellationToken.ThrowOperationCanceledException()\n   at System.Threading.CancellationToken.ThrowIfCancellationRequested()\n   at Hangfire.Processing.TaskExtensions.WaitOne(WaitHandle waitHandle, TimeSpan timeout, CancellationToken token)\n   at Hangfire.Processing.BackgroundExecution.HandleDelay(Guid executionId, TimeSpan delay)","State":"Execution loop Worker:2922fdc1 was unable to wait for '00:05:00' delay due to an exception. Execution will be stopped.","SourceContext":"Hangfire.Processing.BackgroundExecution","ThreadId":21,"EnvironmentName":"Production","ProcessId":2557180,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-12T02:01:46.2107202Z","@mt":"{State:l}","@r":["Execution loop Worker:4469ab04 was unable to wait for '00:05:00' delay due to an exception. Execution will be stopped."],"@l":"Fatal","@x":"System.OperationCanceledException: The operation was canceled.\n   at System.Threading.CancellationToken.ThrowOperationCanceledException()\n   at System.Threading.CancellationToken.ThrowIfCancellationRequested()\n   at Hangfire.Processing.TaskExtensions.WaitOne(WaitHandle waitHandle, TimeSpan timeout, CancellationToken token)\n   at Hangfire.Processing.BackgroundExecution.HandleDelay(Guid executionId, TimeSpan delay)","State":"Execution loop Worker:4469ab04 was unable to wait for '00:05:00' delay due to an exception. Execution will be stopped.","SourceContext":"Hangfire.Processing.BackgroundExecution","ThreadId":24,"EnvironmentName":"Production","ProcessId":2557180,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-12T02:01:46.2112208Z","@mt":"{State:l}","@r":["Execution loop Worker:36fabb1c was unable to wait for '00:02:01' delay due to an exception. Execution will be stopped."],"@l":"Fatal","@x":"System.OperationCanceledException: The operation was canceled.\n   at System.Threading.CancellationToken.ThrowOperationCanceledException()\n   at System.Threading.CancellationToken.ThrowIfCancellationRequested()\n   at Hangfire.Processing.TaskExtensions.WaitOne(WaitHandle waitHandle, TimeSpan timeout, CancellationToken token)\n   at Hangfire.Processing.BackgroundExecution.HandleDelay(Guid executionId, TimeSpan delay)","State":"Execution loop Worker:36fabb1c was unable to wait for '00:02:01' delay due to an exception. Execution will be stopped.","SourceContext":"Hangfire.Processing.BackgroundExecution","ThreadId":31,"EnvironmentName":"Production","ProcessId":2557180,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-12T02:01:46.2113493Z","@mt":"{State:l}","@r":["Execution loop Worker:9b703ef3 was unable to wait for '00:02:24' delay due to an exception. Execution will be stopped."],"@l":"Fatal","@x":"System.OperationCanceledException: The operation was canceled.\n   at System.Threading.CancellationToken.ThrowOperationCanceledException()\n   at System.Threading.CancellationToken.ThrowIfCancellationRequested()\n   at Hangfire.Processing.TaskExtensions.WaitOne(WaitHandle waitHandle, TimeSpan timeout, CancellationToken token)\n   at Hangfire.Processing.BackgroundExecution.HandleDelay(Guid executionId, TimeSpan delay)","State":"Execution loop Worker:9b703ef3 was unable to wait for '00:02:24' delay due to an exception. Execution will be stopped.","SourceContext":"Hangfire.Processing.BackgroundExecution","ThreadId":32,"EnvironmentName":"Production","ProcessId":2557180,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-12T02:01:46.2114688Z","@mt":"{State:l}","@r":["Execution loop Worker:c94f9c9b was unable to wait for '00:01:04' delay due to an exception. Execution will be stopped."],"@l":"Fatal","@x":"System.OperationCanceledException: The operation was canceled.\n   at System.Threading.CancellationToken.ThrowOperationCanceledException()\n   at System.Threading.CancellationToken.ThrowIfCancellationRequested()\n   at Hangfire.Processing.TaskExtensions.WaitOne(WaitHandle waitHandle, TimeSpan timeout, CancellationToken token)\n   at Hangfire.Processing.BackgroundExecution.HandleDelay(Guid executionId, TimeSpan delay)","State":"Execution loop Worker:c94f9c9b was unable to wait for '00:01:04' delay due to an exception. Execution will be stopped.","SourceContext":"Hangfire.Processing.BackgroundExecution","ThreadId":29,"EnvironmentName":"Production","ProcessId":2557180,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-12T02:01:46.2106998Z","@mt":"{State:l}","@r":["Execution loop RecurringJobScheduler:478ea0bc was unable to wait for '00:00:25' delay due to an exception. Execution will be stopped."],"@l":"Fatal","@x":"System.OperationCanceledException: The operation was canceled.\n   at System.Threading.CancellationToken.ThrowOperationCanceledException()\n   at System.Threading.CancellationToken.ThrowIfCancellationRequested()\n   at Hangfire.Processing.TaskExtensions.WaitOne(WaitHandle waitHandle, TimeSpan timeout, CancellationToken token)\n   at Hangfire.Processing.BackgroundExecution.HandleDelay(Guid executionId, TimeSpan delay)","State":"Execution loop RecurringJobScheduler:478ea0bc was unable to wait for '00:00:25' delay due to an exception. Execution will be stopped.","SourceContext":"Hangfire.Processing.BackgroundExecution","ThreadId":41,"EnvironmentName":"Production","ProcessId":2557180,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-12T02:01:46.2115760Z","@mt":"{State:l}","@r":["Execution loop Worker:77eaa872 was unable to wait for '00:05:00' delay due to an exception. Execution will be stopped."],"@l":"Fatal","@x":"System.OperationCanceledException: The operation was canceled.\n   at System.Threading.CancellationToken.ThrowOperationCanceledException()\n   at System.Threading.CancellationToken.ThrowIfCancellationRequested()\n   at Hangfire.Processing.TaskExtensions.WaitOne(WaitHandle waitHandle, TimeSpan timeout, CancellationToken token)\n   at Hangfire.Processing.BackgroundExecution.HandleDelay(Guid executionId, TimeSpan delay)","State":"Execution loop Worker:77eaa872 was unable to wait for '00:05:00' delay due to an exception. Execution will be stopped.","SourceContext":"Hangfire.Processing.BackgroundExecution","ThreadId":28,"EnvironmentName":"Production","ProcessId":2557180,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-12T02:01:46.2107599Z","@mt":"{State:l}","@r":["Execution loop Worker:51c030db was unable to wait for '00:05:00' delay due to an exception. Execution will be stopped."],"@l":"Fatal","@x":"System.OperationCanceledException: The operation was canceled.\n   at System.Threading.CancellationToken.ThrowOperationCanceledException()\n   at System.Threading.CancellationToken.ThrowIfCancellationRequested()\n   at Hangfire.Processing.TaskExtensions.WaitOne(WaitHandle waitHandle, TimeSpan timeout, CancellationToken token)\n   at Hangfire.Processing.BackgroundExecution.HandleDelay(Guid executionId, TimeSpan delay)","State":"Execution loop Worker:51c030db was unable to wait for '00:05:00' delay due to an exception. Execution will be stopped.","SourceContext":"Hangfire.Processing.BackgroundExecution","ThreadId":39,"EnvironmentName":"Production","ProcessId":2557180,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-12T02:01:46.2117424Z","@mt":"{State:l}","@r":["Execution loop Worker:f2fc4707 was unable to wait for '00:05:00' delay due to an exception. Execution will be stopped."],"@l":"Fatal","@x":"System.OperationCanceledException: The operation was canceled.\n   at System.Threading.CancellationToken.ThrowOperationCanceledException()\n   at System.Threading.CancellationToken.ThrowIfCancellationRequested()\n   at Hangfire.Processing.TaskExtensions.WaitOne(WaitHandle waitHandle, TimeSpan timeout, CancellationToken token)\n   at Hangfire.Processing.BackgroundExecution.HandleDelay(Guid executionId, TimeSpan delay)","State":"Execution loop Worker:f2fc4707 was unable to wait for '00:05:00' delay due to an exception. Execution will be stopped.","SourceContext":"Hangfire.Processing.BackgroundExecution","ThreadId":25,"EnvironmentName":"Production","ProcessId":2557180,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-12T02:01:46.2119119Z","@mt":"{State:l}","@r":["Execution loop Worker:eb98c1d2 was unable to wait for '00:05:00' delay due to an exception. Execution will be stopped."],"@l":"Fatal","@x":"System.OperationCanceledException: The operation was canceled.\n   at System.Threading.CancellationToken.ThrowOperationCanceledException()\n   at System.Threading.CancellationToken.ThrowIfCancellationRequested()\n   at Hangfire.Processing.TaskExtensions.WaitOne(WaitHandle waitHandle, TimeSpan timeout, CancellationToken token)\n   at Hangfire.Processing.BackgroundExecution.HandleDelay(Guid executionId, TimeSpan delay)","State":"Execution loop Worker:eb98c1d2 was unable to wait for '00:05:00' delay due to an exception. Execution will be stopped.","SourceContext":"Hangfire.Processing.BackgroundExecution","ThreadId":20,"EnvironmentName":"Production","ProcessId":2557180,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-12T02:01:46.2107158Z","@mt":"{State:l}","@r":["Execution loop Worker:58b9871d was unable to wait for '00:01:21' delay due to an exception. Execution will be stopped."],"@l":"Fatal","@x":"System.OperationCanceledException: The operation was canceled.\n   at System.Threading.CancellationToken.ThrowOperationCanceledException()\n   at System.Threading.CancellationToken.ThrowIfCancellationRequested()\n   at Hangfire.Processing.TaskExtensions.WaitOne(WaitHandle waitHandle, TimeSpan timeout, CancellationToken token)\n   at Hangfire.Processing.BackgroundExecution.HandleDelay(Guid executionId, TimeSpan delay)","State":"Execution loop Worker:58b9871d was unable to wait for '00:01:21' delay due to an exception. Execution will be stopped.","SourceContext":"Hangfire.Processing.BackgroundExecution","ThreadId":27,"EnvironmentName":"Production","ProcessId":2557180,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-12T02:01:46.2107158Z","@mt":"{State:l}","@r":["Execution loop Worker:3406b4e0 was unable to wait for '00:03:16' delay due to an exception. Execution will be stopped."],"@l":"Fatal","@x":"System.OperationCanceledException: The operation was canceled.\n   at System.Threading.CancellationToken.ThrowOperationCanceledException()\n   at System.Threading.CancellationToken.ThrowIfCancellationRequested()\n   at Hangfire.Processing.TaskExtensions.WaitOne(WaitHandle waitHandle, TimeSpan timeout, CancellationToken token)\n   at Hangfire.Processing.BackgroundExecution.HandleDelay(Guid executionId, TimeSpan delay)","State":"Execution loop Worker:3406b4e0 was unable to wait for '00:03:16' delay due to an exception. Execution will be stopped.","SourceContext":"Hangfire.Processing.BackgroundExecution","ThreadId":34,"EnvironmentName":"Production","ProcessId":2557180,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-12T02:01:46.2112326Z","@mt":"{State:l}","@r":["Execution loop Worker:7e579b9d was unable to wait for '00:01:40' delay due to an exception. Execution will be stopped."],"@l":"Fatal","@x":"System.OperationCanceledException: The operation was canceled.\n   at System.Threading.CancellationToken.ThrowOperationCanceledException()\n   at System.Threading.CancellationToken.ThrowIfCancellationRequested()\n   at Hangfire.Processing.TaskExtensions.WaitOne(WaitHandle waitHandle, TimeSpan timeout, CancellationToken token)\n   at Hangfire.Processing.BackgroundExecution.HandleDelay(Guid executionId, TimeSpan delay)","State":"Execution loop Worker:7e579b9d was unable to wait for '00:01:40' delay due to an exception. Execution will be stopped.","SourceContext":"Hangfire.Processing.BackgroundExecution","ThreadId":30,"EnvironmentName":"Production","ProcessId":2557180,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-12T02:01:46.2113504Z","@mt":"{State:l}","@r":["Execution loop Worker:1c191b6d was unable to wait for '00:04:49' delay due to an exception. Execution will be stopped."],"@l":"Fatal","@x":"System.OperationCanceledException: The operation was canceled.\n   at System.Threading.CancellationToken.ThrowOperationCanceledException()\n   at System.Threading.CancellationToken.ThrowIfCancellationRequested()\n   at Hangfire.Processing.TaskExtensions.WaitOne(WaitHandle waitHandle, TimeSpan timeout, CancellationToken token)\n   at Hangfire.Processing.BackgroundExecution.HandleDelay(Guid executionId, TimeSpan delay)","State":"Execution loop Worker:1c191b6d was unable to wait for '00:04:49' delay due to an exception. Execution will be stopped.","SourceContext":"Hangfire.Processing.BackgroundExecution","ThreadId":37,"EnvironmentName":"Production","ProcessId":2557180,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-12T02:01:46.2107593Z","@mt":"{State:l}","@r":["Execution loop Worker:cf7c3f9d was unable to wait for '00:05:00' delay due to an exception. Execution will be stopped."],"@l":"Fatal","@x":"System.OperationCanceledException: The operation was canceled.\n   at System.Threading.CancellationToken.ThrowOperationCanceledException()\n   at System.Threading.CancellationToken.ThrowIfCancellationRequested()\n   at Hangfire.Processing.TaskExtensions.WaitOne(WaitHandle waitHandle, TimeSpan timeout, CancellationToken token)\n   at Hangfire.Processing.BackgroundExecution.HandleDelay(Guid executionId, TimeSpan delay)","State":"Execution loop Worker:cf7c3f9d was unable to wait for '00:05:00' delay due to an exception. Execution will be stopped.","SourceContext":"Hangfire.Processing.BackgroundExecution","ThreadId":38,"EnvironmentName":"Production","ProcessId":2557180,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-12T02:01:46.2116505Z","@mt":"{State:l}","@r":["Execution loop Worker:43515dc2 was unable to wait for '00:03:45' delay due to an exception. Execution will be stopped."],"@l":"Fatal","@x":"System.OperationCanceledException: The operation was canceled.\n   at System.Threading.CancellationToken.ThrowOperationCanceledException()\n   at System.Threading.CancellationToken.ThrowIfCancellationRequested()\n   at Hangfire.Processing.TaskExtensions.WaitOne(WaitHandle waitHandle, TimeSpan timeout, CancellationToken token)\n   at Hangfire.Processing.BackgroundExecution.HandleDelay(Guid executionId, TimeSpan delay)","State":"Execution loop Worker:43515dc2 was unable to wait for '00:03:45' delay due to an exception. Execution will be stopped.","SourceContext":"Hangfire.Processing.BackgroundExecution","ThreadId":35,"EnvironmentName":"Production","ProcessId":2557180,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-12T02:01:46.2122551Z","@mt":"{State:l}","@r":["Execution loop Worker:934bdcd6 was unable to wait for '00:05:00' delay due to an exception. Execution will be stopped."],"@l":"Fatal","@x":"System.OperationCanceledException: The operation was canceled.\n   at System.Threading.CancellationToken.ThrowOperationCanceledException()\n   at System.Threading.CancellationToken.ThrowIfCancellationRequested()\n   at Hangfire.Processing.TaskExtensions.WaitOne(WaitHandle waitHandle, TimeSpan timeout, CancellationToken token)\n   at Hangfire.Processing.BackgroundExecution.HandleDelay(Guid executionId, TimeSpan delay)","State":"Execution loop Worker:934bdcd6 was unable to wait for '00:05:00' delay due to an exception. Execution will be stopped.","SourceContext":"Hangfire.Processing.BackgroundExecution","ThreadId":23,"EnvironmentName":"Production","ProcessId":2557180,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-12T02:01:46.2107367Z","@mt":"{State:l}","@r":["Execution loop Worker:e564ab3f was unable to wait for '00:05:00' delay due to an exception. Execution will be stopped."],"@l":"Fatal","@x":"System.OperationCanceledException: The operation was canceled.\n   at System.Threading.CancellationToken.ThrowOperationCanceledException()\n   at System.Threading.CancellationToken.ThrowIfCancellationRequested()\n   at Hangfire.Processing.TaskExtensions.WaitOne(WaitHandle waitHandle, TimeSpan timeout, CancellationToken token)\n   at Hangfire.Processing.BackgroundExecution.HandleDelay(Guid executionId, TimeSpan delay)","State":"Execution loop Worker:e564ab3f was unable to wait for '00:05:00' delay due to an exception. Execution will be stopped.","SourceContext":"Hangfire.Processing.BackgroundExecution","ThreadId":22,"EnvironmentName":"Production","ProcessId":2557180,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-12T02:01:46.2127468Z","@mt":"{State:l}","@r":["Execution loop Worker:b4d4da24 was unable to wait for '00:02:49' delay due to an exception. Execution will be stopped."],"@l":"Fatal","@x":"System.OperationCanceledException: The operation was canceled.\n   at System.Threading.CancellationToken.ThrowOperationCanceledException()\n   at System.Threading.CancellationToken.ThrowIfCancellationRequested()\n   at Hangfire.Processing.TaskExtensions.WaitOne(WaitHandle waitHandle, TimeSpan timeout, CancellationToken token)\n   at Hangfire.Processing.BackgroundExecution.HandleDelay(Guid executionId, TimeSpan delay)","State":"Execution loop Worker:b4d4da24 was unable to wait for '00:02:49' delay due to an exception. Execution will be stopped.","SourceContext":"Hangfire.Processing.BackgroundExecution","ThreadId":33,"EnvironmentName":"Production","ProcessId":2557180,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-12T02:01:46.2107593Z","@mt":"{State:l}","@r":["Execution loop DelayedJobScheduler:9e1dbf4a was unable to wait for '00:00:25' delay due to an exception. Execution will be stopped."],"@l":"Fatal","@x":"System.OperationCanceledException: The operation was canceled.\n   at System.Threading.CancellationToken.ThrowOperationCanceledException()\n   at System.Threading.CancellationToken.ThrowIfCancellationRequested()\n   at Hangfire.Processing.TaskExtensions.WaitOne(WaitHandle waitHandle, TimeSpan timeout, CancellationToken token)\n   at Hangfire.Processing.BackgroundExecution.HandleDelay(Guid executionId, TimeSpan delay)","State":"Execution loop DelayedJobScheduler:9e1dbf4a was unable to wait for '00:00:25' delay due to an exception. Execution will be stopped.","SourceContext":"Hangfire.Processing.BackgroundExecution","ThreadId":40,"EnvironmentName":"Production","ProcessId":2557180,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-12T02:01:46.2169828Z","@mt":"{State:l}","@r":["Server cc923dec-9adf-4cc6-9b48-04479174f16b All dispatchers stopped"],"State":"Server cc923dec-9adf-4cc6-9b48-04479174f16b All dispatchers stopped","SourceContext":"Hangfire.Server.BackgroundServerProcess","ThreadId":13,"EnvironmentName":"Production","ProcessId":2557180,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-12T02:01:46.2193849Z","@mt":"{State:l}","@r":["Server cc923dec-9adf-4cc6-9b48-04479174f16b there was an exception, server may not be removed"],"@l":"Warning","@x":"Microsoft.Data.SqlClient.SqlException (0x80131904): Login failed for user 'sa'. Reason: Server is in script upgrade mode. Only administrator can connect at this time.\n   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)\n   at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)\n   at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)\n   at Microsoft.Data.SqlClient.SqlConnection.Open(SqlConnectionOverrides overrides)\n   at Hangfire.SqlServer.SqlServerStorage.CreateAndOpenConnection()\n   at Hangfire.SqlServer.SqlServerStorage.UseConnection[T](DbConnection dedicatedConnection, Func`2 func)\n   at Hangfire.SqlServer.SqlServerStorage.UseConnection(DbConnection dedicatedConnection, Action`1 action)\n   at Hangfire.SqlServer.SqlServerConnection.RemoveServer(String serverId)\n   at Hangfire.Server.BackgroundServerProcess.ServerDelete(BackgroundServerContext context, Stopwatch stoppedAt)\nClientConnectionId:3c393150-cf18-4868-87d1-dd3a75868007\nError Number:18401,State:1,Class:14","State":"Server cc923dec-9adf-4cc6-9b48-04479174f16b there was an exception, server may not be removed","SourceContext":"Hangfire.Server.BackgroundServerProcess","ThreadId":13,"EnvironmentName":"Production","ProcessId":2557180,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-12T02:02:10.0004248Z","@mt":"{State:l}","@r":["Start installing Hangfire SQL objects..."],"State":"Start installing Hangfire SQL objects...","SourceContext":"Hangfire.SqlServer.SqlServerObjectsInstaller","ThreadId":1,"EnvironmentName":"Production","ProcessId":2594223,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-12T02:02:10.2403186Z","@mt":"{State:l}","@r":["Hangfire SQL objects installed."],"State":"Hangfire SQL objects installed.","SourceContext":"Hangfire.SqlServer.SqlServerObjectsInstaller","ThreadId":1,"EnvironmentName":"Production","ProcessId":2594223,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-12T02:02:10.3904752Z","@mt":"Application starting..............","ThreadId":1,"EnvironmentName":"Production","ProcessId":2594223,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-12T02:02:10.4672246Z","@mt":"{State:l}","@r":["Starting Hangfire Server using job storage: 'SQL Server: .@ez-prms-staging'"],"State":"Starting Hangfire Server using job storage: 'SQL Server: .@ez-prms-staging'","SourceContext":"Hangfire.BackgroundJobServer","ThreadId":1,"EnvironmentName":"Production","ProcessId":2594223,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-12T02:02:10.4674832Z","@mt":"{State:l}","@r":["Using the following options for SQL Server job storage: Queue poll interval: 00:00:00."],"State":"Using the following options for SQL Server job storage: Queue poll interval: 00:00:00.","SourceContext":"Hangfire.BackgroundJobServer","ThreadId":1,"EnvironmentName":"Production","ProcessId":2594223,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-12T02:02:10.4677984Z","@mt":"{State:l}","@r":["Using the following options for Hangfire Server:\r\n    Worker count: 20\r\n    Listening queues: 'default'\r\n    Shutdown timeout: 00:00:15\r\n    Schedule polling interval: 00:00:15"],"State":"Using the following options for Hangfire Server:\r\n    Worker count: 20\r\n    Listening queues: 'default'\r\n    Shutdown timeout: 00:00:15\r\n    Schedule polling interval: 00:00:15","SourceContext":"Hangfire.BackgroundJobServer","ThreadId":1,"EnvironmentName":"Production","ProcessId":2594223,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-12T02:02:10.5718660Z","@mt":"{State:l}","@r":["Server 31a061ca-33b2-4051-950c-f0c862d6ba2c successfully announced in 91.7283 ms"],"State":"Server 31a061ca-33b2-4051-950c-f0c862d6ba2c successfully announced in 91.7283 ms","SourceContext":"Hangfire.Server.BackgroundServerProcess","ThreadId":12,"EnvironmentName":"Production","ProcessId":2594223,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-12T02:02:10.5741887Z","@mt":"{State:l}","@r":["Server 31a061ca-33b2-4051-950c-f0c862d6ba2c is starting the registered dispatchers: ServerWatchdog, ServerJobCancellationWatcher, ExpirationManager, CountersAggregator, SqlServerHeartbeatProcess, Worker, DelayedJobScheduler, RecurringJobScheduler..."],"State":"Server 31a061ca-33b2-4051-950c-f0c862d6ba2c is starting the registered dispatchers: ServerWatchdog, ServerJobCancellationWatcher, ExpirationManager, CountersAggregator, SqlServerHeartbeatProcess, Worker, DelayedJobScheduler, RecurringJobScheduler...","SourceContext":"Hangfire.Server.BackgroundServerProcess","ThreadId":12,"EnvironmentName":"Production","ProcessId":2594223,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-12T02:02:10.6160097Z","@mt":"{State:l}","@r":["Server 31a061ca-33b2-4051-950c-f0c862d6ba2c all the dispatchers started"],"State":"Server 31a061ca-33b2-4051-950c-f0c862d6ba2c all the dispatchers started","SourceContext":"Hangfire.Server.BackgroundServerProcess","ThreadId":12,"EnvironmentName":"Production","ProcessId":2594223,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-12T02:07:10.5828133Z","@mt":"{State:l}","@r":["1 servers were removed due to timeout"],"State":"1 servers were removed due to timeout","SourceContext":"Hangfire.Server.ServerWatchdog","ThreadId":14,"EnvironmentName":"Production","ProcessId":2594223,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-12T03:38:08.2876372Z","@mt":"Failed to determine the https port for redirect.","@l":"Warning","EventId":{"Id":3,"Name":"FailedToDeterminePort"},"SourceContext":"Microsoft.AspNetCore.HttpsPolicy.HttpsRedirectionMiddleware","RequestId":"0HNH1JJIO5UPA:00000001","RequestPath":"/.well-known/acme-challenge/6Y0M8EXUZQQK7I2U6BSVT9XVIAURZN2X","ConnectionId":"0HNH1JJIO5UPA","ThreadId":190,"EnvironmentName":"Production","ProcessId":2594223,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-12T10:54:03.7845273Z","@mt":"{State:l}","@r":["Server 31a061ca-33b2-4051-950c-f0c862d6ba2c caught stopping signal..."],"State":"Server 31a061ca-33b2-4051-950c-f0c862d6ba2c caught stopping signal...","SourceContext":"Hangfire.Server.BackgroundServerProcess","ThreadId":312,"EnvironmentName":"Production","ProcessId":2594223,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-12T10:54:03.7863476Z","@mt":"{State:l}","@r":["Server 31a061ca-33b2-4051-950c-f0c862d6ba2c All dispatchers stopped"],"State":"Server 31a061ca-33b2-4051-950c-f0c862d6ba2c All dispatchers stopped","SourceContext":"Hangfire.Server.BackgroundServerProcess","ThreadId":12,"EnvironmentName":"Production","ProcessId":2594223,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-12T10:54:03.7901852Z","@mt":"{State:l}","@r":["Server 31a061ca-33b2-4051-950c-f0c862d6ba2c successfully reported itself as stopped in 2.9038 ms"],"State":"Server 31a061ca-33b2-4051-950c-f0c862d6ba2c successfully reported itself as stopped in 2.9038 ms","SourceContext":"Hangfire.Server.BackgroundServerProcess","ThreadId":12,"EnvironmentName":"Production","ProcessId":2594223,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-12T10:54:03.7902610Z","@mt":"{State:l}","@r":["Server 31a061ca-33b2-4051-950c-f0c862d6ba2c has been stopped in total 4.8466 ms"],"State":"Server 31a061ca-33b2-4051-950c-f0c862d6ba2c has been stopped in total 4.8466 ms","SourceContext":"Hangfire.Server.BackgroundServerProcess","ThreadId":12,"EnvironmentName":"Production","ProcessId":2594223,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-12T10:55:17.6918645Z","@mt":"{State:l}","@r":["Start installing Hangfire SQL objects..."],"State":"Start installing Hangfire SQL objects...","SourceContext":"Hangfire.SqlServer.SqlServerObjectsInstaller","ThreadId":1,"EnvironmentName":"Production","ProcessId":2794283,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-12T10:55:17.8208079Z","@mt":"{State:l}","@r":["Hangfire SQL objects installed."],"State":"Hangfire SQL objects installed.","SourceContext":"Hangfire.SqlServer.SqlServerObjectsInstaller","ThreadId":1,"EnvironmentName":"Production","ProcessId":2794283,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-12T10:55:17.9206512Z","@mt":"Application starting..............","ThreadId":1,"EnvironmentName":"Production","ProcessId":2794283,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-12T10:55:17.9631510Z","@mt":"{State:l}","@r":["Starting Hangfire Server using job storage: 'SQL Server: .@ez-prms-staging'"],"State":"Starting Hangfire Server using job storage: 'SQL Server: .@ez-prms-staging'","SourceContext":"Hangfire.BackgroundJobServer","ThreadId":1,"EnvironmentName":"Production","ProcessId":2794283,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-12T10:55:17.9633038Z","@mt":"{State:l}","@r":["Using the following options for SQL Server job storage: Queue poll interval: 00:00:00."],"State":"Using the following options for SQL Server job storage: Queue poll interval: 00:00:00.","SourceContext":"Hangfire.BackgroundJobServer","ThreadId":1,"EnvironmentName":"Production","ProcessId":2794283,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-12T10:55:17.9634590Z","@mt":"{State:l}","@r":["Using the following options for Hangfire Server:\r\n    Worker count: 20\r\n    Listening queues: 'default'\r\n    Shutdown timeout: 00:00:15\r\n    Schedule polling interval: 00:00:15"],"State":"Using the following options for Hangfire Server:\r\n    Worker count: 20\r\n    Listening queues: 'default'\r\n    Shutdown timeout: 00:00:15\r\n    Schedule polling interval: 00:00:15","SourceContext":"Hangfire.BackgroundJobServer","ThreadId":1,"EnvironmentName":"Production","ProcessId":2794283,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-12T10:55:18.0228820Z","@mt":"{State:l}","@r":["Server 29282bcc-6236-46bf-86b5-b0b332ce38bd successfully announced in 51.8361 ms"],"State":"Server 29282bcc-6236-46bf-86b5-b0b332ce38bd successfully announced in 51.8361 ms","SourceContext":"Hangfire.Server.BackgroundServerProcess","ThreadId":13,"EnvironmentName":"Production","ProcessId":2794283,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-12T10:55:18.0246519Z","@mt":"{State:l}","@r":["Server 29282bcc-6236-46bf-86b5-b0b332ce38bd is starting the registered dispatchers: ServerWatchdog, ServerJobCancellationWatcher, ExpirationManager, CountersAggregator, SqlServerHeartbeatProcess, Worker, DelayedJobScheduler, RecurringJobScheduler..."],"State":"Server 29282bcc-6236-46bf-86b5-b0b332ce38bd is starting the registered dispatchers: ServerWatchdog, ServerJobCancellationWatcher, ExpirationManager, CountersAggregator, SqlServerHeartbeatProcess, Worker, DelayedJobScheduler, RecurringJobScheduler...","SourceContext":"Hangfire.Server.BackgroundServerProcess","ThreadId":13,"EnvironmentName":"Production","ProcessId":2794283,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-12T10:55:18.0292957Z","@mt":"{State:l}","@r":["Server 29282bcc-6236-46bf-86b5-b0b332ce38bd all the dispatchers started"],"State":"Server 29282bcc-6236-46bf-86b5-b0b332ce38bd all the dispatchers started","SourceContext":"Hangfire.Server.BackgroundServerProcess","ThreadId":13,"EnvironmentName":"Production","ProcessId":2794283,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-12T10:55:31.4216609Z","@mt":"Failed to determine the https port for redirect.","@l":"Warning","EventId":{"Id":3,"Name":"FailedToDeterminePort"},"SourceContext":"Microsoft.AspNetCore.HttpsPolicy.HttpsRedirectionMiddleware","RequestId":"0HNH1R7VQ6QLQ:00000001","RequestPath":"/","ConnectionId":"0HNH1R7VQ6QLQ","ThreadId":12,"EnvironmentName":"Production","ProcessId":2794283,"MachineName":"nc-ph-4626"}