{"@t":"2025-11-14T02:01:11.8067028Z","@mt":"{State:l}","@r":["Server 29282bcc-6236-46bf-86b5-b0b332ce38bd encountered an exception while sending heartbeat"],"@l":"Warning","@x":"Microsoft.Data.SqlClient.SqlException (0x80131904): A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: TCP Provider, error: 40 - Could not open a connection to SQL Server: Could not open a connection to SQL Server)\n   at Microsoft.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)\n   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)\n   at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)\n   at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)\n   at Microsoft.Data.SqlClient.SqlConnection.Open(SqlConnectionOverrides overrides)\n   at Hangfire.SqlServer.SqlServerStorage.CreateAndOpenConnection()\n   at Hangfire.SqlServer.SqlServerStorage.UseConnection[T](DbConnection dedicatedConnection, Func`2 func)\n   at Hangfire.Server.ServerHeartbeatProcess.Execute(BackgroundProcessContext context)\nClientConnectionId:00000000-0000-0000-0000-000000000000","State":"Server 29282bcc-6236-46bf-86b5-b0b332ce38bd encountered an exception while sending heartbeat","SourceContext":"Hangfire.Server.ServerHeartbeatProcess","ThreadId":14,"EnvironmentName":"Production","ProcessId":2794283,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-14T02:01:29.3814399Z","@mt":"{State:l}","@r":["Execution DelayedJobScheduler is in the Failed state now due to an exception, execution will be retried no more than in 00:00:09"],"@l":"Error","@x":"Microsoft.Data.SqlClient.SqlException (0x80131904): Login failed for user 'sa'. Reason: Server is in script upgrade mode. Only administrator can connect at this time.\n   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)\n   at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)\n   at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)\n   at Microsoft.Data.SqlClient.SqlConnection.Open(SqlConnectionOverrides overrides)\n   at Hangfire.SqlServer.SqlServerStorage.CreateAndOpenConnection()\n   at Hangfire.SqlServer.SqlServerConnection.AcquireLock(String resource, TimeSpan timeout)\n   at Hangfire.Server.DelayedJobScheduler.UseConnectionDistributedLock[T](JobStorage storage, Func`2 action)\n   at Hangfire.Server.DelayedJobScheduler.Execute(BackgroundProcessContext context)\n   at Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(Guid executionId, Object state)\n   at Hangfire.Processing.BackgroundExecution.Run(Action`2 callback, Object state)\nClientConnectionId:8e92d00a-d7d6-413a-abd6-807482161a37\nError Number:18401,State:1,Class:14","State":"Execution DelayedJobScheduler is in the Failed state now due to an exception, execution will be retried no more than in 00:00:09","SourceContext":"Hangfire.Processing.BackgroundExecution","ThreadId":40,"EnvironmentName":"Production","ProcessId":2794283,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-14T02:01:29.3825191Z","@mt":"{State:l}","@r":["Execution Worker is in the Failed state now due to an exception, execution will be retried no more than in 00:05:00"],"@l":"Error","@x":"Microsoft.Data.SqlClient.SqlException (0x80131904): Login failed for user 'sa'. Reason: Server is in script upgrade mode. Only administrator can connect at this time.\n   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)\n   at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)\n   at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)\n   at Microsoft.Data.SqlClient.SqlConnection.Open(SqlConnectionOverrides overrides)\n   at Hangfire.SqlServer.SqlServerStorage.CreateAndOpenConnection()\n   at Hangfire.SqlServer.SqlServerStorage.UseConnection[T](DbConnection dedicatedConnection, Func`2 func)\n   at Hangfire.SqlServer.SqlServerJobQueue.DequeueUsingSlidingInvisibilityTimeout(String[] queues, CancellationToken cancellationToken)\n   at Hangfire.SqlServer.SqlServerJobQueue.Dequeue(String[] queues, CancellationToken cancellationToken)\n   at Hangfire.SqlServer.SqlServerConnection.FetchNextJob(String[] queues, CancellationToken cancellationToken)\n   at Hangfire.Server.Worker.Execute(BackgroundProcessContext context)\n   at Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(Guid executionId, Object state)\n   at Hangfire.Processing.BackgroundExecution.Run(Action`2 callback, Object state)\nClientConnectionId:8e92d00a-d7d6-413a-abd6-807482161a37\nError Number:18401,State:1,Class:14","State":"Execution Worker is in the Failed state now due to an exception, execution will be retried no more than in 00:05:00","SourceContext":"Hangfire.Processing.BackgroundExecution","ThreadId":37,"EnvironmentName":"Production","ProcessId":2794283,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-14T02:01:29.3825480Z","@mt":"{State:l}","@r":["Execution RecurringJobScheduler is in the Failed state now due to an exception, execution will be retried no more than in 00:00:09"],"@l":"Error","@x":"Microsoft.Data.SqlClient.SqlException (0x80131904): Login failed for user 'sa'. Reason: Server is in script upgrade mode. Only administrator can connect at this time.\n   at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)\n   at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)\n   at Microsoft.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)\n   at Microsoft.Data.SqlClient.SqlInternalConnectionTds.CompleteLogin(Boolean enlistOK)\n   at Microsoft.Data.SqlClient.SqlInternalConnectionTds.LoginNoFailover(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString connectionOptions, SqlCredential credential, TimeoutTimer timeout)\n   at Microsoft.Data.SqlClient.SqlInternalConnectionTds.OpenLoginEnlist(TimeoutTimer timeout, SqlConnectionString connectionOptions, SqlCredential credential, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance)\n   at Microsoft.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, Boolean applyTransientFaultHandling, String accessToken, DbConnectionPool pool)\n   at Microsoft.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions)\n   at Microsoft.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)\n   at Microsoft.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)\n   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)\n   at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)\n   at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)\n   at Microsoft.Data.SqlClient.SqlConnection.Open(SqlConnectionOverrides overrides)\n   at Hangfire.SqlServer.SqlServerStorage.CreateAndOpenConnection()\n   at Hangfire.SqlServer.SqlServerConnection.AcquireLock(String resource, TimeSpan timeout)\n   at Hangfire.Server.RecurringJobScheduler.UseConnectionDistributedLock[T](JobStorage storage, Func`2 action)\n   at Hangfire.Server.RecurringJobScheduler.Execute(BackgroundProcessContext context)\n   at Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(Guid executionId, Object state)\n   at Hangfire.Processing.BackgroundExecution.Run(Action`2 callback, Object state)\nClientConnectionId:8e92d00a-d7d6-413a-abd6-807482161a37\nError Number:18401,State:1,Class:14","State":"Execution RecurringJobScheduler is in the Failed state now due to an exception, execution will be retried no more than in 00:00:09","SourceContext":"Hangfire.Processing.BackgroundExecution","ThreadId":41,"EnvironmentName":"Production","ProcessId":2794283,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-14T02:01:41.8409016Z","@mt":"{State:l}","@r":["Server 29282bcc-6236-46bf-86b5-b0b332ce38bd encountered an exception while sending heartbeat"],"@l":"Warning","@x":"Microsoft.Data.SqlClient.SqlException (0x80131904): Login failed for user 'sa'. Reason: Server is in script upgrade mode. Only administrator can connect at this time.\n   at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)\n   at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)\n   at Microsoft.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)\n   at Microsoft.Data.SqlClient.SqlInternalConnectionTds.CompleteLogin(Boolean enlistOK)\n   at Microsoft.Data.SqlClient.SqlInternalConnectionTds.LoginNoFailover(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString connectionOptions, SqlCredential credential, TimeoutTimer timeout)\n   at Microsoft.Data.SqlClient.SqlInternalConnectionTds.OpenLoginEnlist(TimeoutTimer timeout, SqlConnectionString connectionOptions, SqlCredential credential, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance)\n   at Microsoft.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, Boolean applyTransientFaultHandling, String accessToken, DbConnectionPool pool)\n   at Microsoft.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions)\n   at Microsoft.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)\n   at Microsoft.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)\n   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)\n   at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)\n   at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)\n   at Microsoft.Data.SqlClient.SqlConnection.Open(SqlConnectionOverrides overrides)\n   at Hangfire.SqlServer.SqlServerStorage.CreateAndOpenConnection()\n   at Hangfire.SqlServer.SqlServerStorage.UseConnection[T](DbConnection dedicatedConnection, Func`2 func)\n   at Hangfire.Server.ServerHeartbeatProcess.Execute(BackgroundProcessContext context)\nClientConnectionId:50ef9be2-c428-41a9-9c21-17fb1201f3ad\nError Number:18401,State:1,Class:14","State":"Server 29282bcc-6236-46bf-86b5-b0b332ce38bd encountered an exception while sending heartbeat","SourceContext":"Hangfire.Server.ServerHeartbeatProcess","ThreadId":14,"EnvironmentName":"Production","ProcessId":2794283,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-14T02:02:11.8603489Z","@mt":"{State:l}","@r":["Server 29282bcc-6236-46bf-86b5-b0b332ce38bd encountered an exception while sending heartbeat"],"@l":"Warning","@x":"Microsoft.Data.SqlClient.SqlException (0x80131904): Login failed for user 'sa'. Reason: Server is in script upgrade mode. Only administrator can connect at this time.\n   at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)\n   at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)\n   at Microsoft.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)\n   at Microsoft.Data.SqlClient.SqlInternalConnectionTds.CompleteLogin(Boolean enlistOK)\n   at Microsoft.Data.SqlClient.SqlInternalConnectionTds.LoginNoFailover(ServerInfo serverInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString connectionOptions, SqlCredential credential, TimeoutTimer timeout)\n   at Microsoft.Data.SqlClient.SqlInternalConnectionTds.OpenLoginEnlist(TimeoutTimer timeout, SqlConnectionString connectionOptions, SqlCredential credential, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance)\n   at Microsoft.Data.SqlClient.SqlInternalConnectionTds..ctor(DbConnectionPoolIdentity identity, SqlConnectionString connectionOptions, SqlCredential credential, Object providerInfo, String newPassword, SecureString newSecurePassword, Boolean redirectedUserInstance, SqlConnectionString userConnectionOptions, SessionData reconnectSessionData, Boolean applyTransientFaultHandling, String accessToken, DbConnectionPool pool)\n   at Microsoft.Data.SqlClient.SqlConnectionFactory.CreateConnection(DbConnectionOptions options, DbConnectionPoolKey poolKey, Object poolGroupProviderInfo, DbConnectionPool pool, DbConnection owningConnection, DbConnectionOptions userOptions)\n   at Microsoft.Data.ProviderBase.DbConnectionPool.CreateObject(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)\n   at Microsoft.Data.ProviderBase.DbConnectionPool.UserCreateRequest(DbConnection owningObject, DbConnectionOptions userOptions, DbConnectionInternal oldConnection)\n   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)\n   at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)\n   at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)\n   at Microsoft.Data.SqlClient.SqlConnection.Open(SqlConnectionOverrides overrides)\n   at Hangfire.SqlServer.SqlServerStorage.CreateAndOpenConnection()\n   at Hangfire.SqlServer.SqlServerStorage.UseConnection[T](DbConnection dedicatedConnection, Func`2 func)\n   at Hangfire.Server.ServerHeartbeatProcess.Execute(BackgroundProcessContext context)\nClientConnectionId:d80b7244-3319-4ad5-8c3a-a7a54c3a52af\nError Number:18401,State:1,Class:14","State":"Server 29282bcc-6236-46bf-86b5-b0b332ce38bd encountered an exception while sending heartbeat","SourceContext":"Hangfire.Server.ServerHeartbeatProcess","ThreadId":14,"EnvironmentName":"Production","ProcessId":2794283,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-14T02:02:41.8613130Z","@mt":"{State:l}","@r":["Server 29282bcc-6236-46bf-86b5-b0b332ce38bd encountered an exception while sending heartbeat"],"@l":"Warning","@x":"Microsoft.Data.SqlClient.SqlException (0x80131904): Login failed for user 'sa'. Reason: Server is in script upgrade mode. Only administrator can connect at this time.\n   at Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection(DbConnection owningObject, UInt32 waitForMultipleObjectsTimeout, Boolean allowCreate, Boolean onlyOneCheckConnection, DbConnectionOptions userOptions, DbConnectionInternal& connection)\n   at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)\n   at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)\n   at Microsoft.Data.SqlClient.SqlConnection.Open(SqlConnectionOverrides overrides)\n   at Hangfire.SqlServer.SqlServerStorage.CreateAndOpenConnection()\n   at Hangfire.SqlServer.SqlServerStorage.UseConnection[T](DbConnection dedicatedConnection, Func`2 func)\n   at Hangfire.Server.ServerHeartbeatProcess.Execute(BackgroundProcessContext context)\nClientConnectionId:d80b7244-3319-4ad5-8c3a-a7a54c3a52af\nError Number:18401,State:1,Class:14","State":"Server 29282bcc-6236-46bf-86b5-b0b332ce38bd encountered an exception while sending heartbeat","SourceContext":"Hangfire.Server.ServerHeartbeatProcess","ThreadId":14,"EnvironmentName":"Production","ProcessId":2794283,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-14T02:03:11.0239073Z","@mt":"{State:l}","@r":["Execution DelayedJobScheduler recovered from the Failed state after 00:01:41.6417597 and is in the Running state now"],"State":"Execution DelayedJobScheduler recovered from the Failed state after 00:01:41.6417597 and is in the Running state now","SourceContext":"Hangfire.Processing.BackgroundExecution","ThreadId":40,"EnvironmentName":"Production","ProcessId":2794283,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-14T02:03:11.0283408Z","@mt":"{State:l}","@r":["Execution RecurringJobScheduler recovered from the Failed state after 00:01:41.6424524 and is in the Running state now"],"State":"Execution RecurringJobScheduler recovered from the Failed state after 00:01:41.6424524 and is in the Running state now","SourceContext":"Hangfire.Processing.BackgroundExecution","ThreadId":41,"EnvironmentName":"Production","ProcessId":2794283,"MachineName":"nc-ph-4626"}
{"@t":"2025-11-14T02:03:11.8664216Z","@mt":"{State:l}","@r":["Server 29282bcc-6236-46bf-86b5-b0b332ce38bd is now able to continue sending heartbeats"],"State":"Server 29282bcc-6236-46bf-86b5-b0b332ce38bd is now able to continue sending heartbeats","SourceContext":"Hangfire.Server.ServerHeartbeatProcess","ThreadId":14,"EnvironmentName":"Production","ProcessId":2794283,"MachineName":"nc-ph-4626"}