EU Cloud Production Incident (21/10 - 31/10/2019) - now available via Flexera Community
NA Cloud Production Incident (1/10 - 5/11/2019) - now available via Flexera Community
Posted Nov 12, 2019 - 21:30 PST
Resolved
Due to a number optimizations and workarounds, support have greatly minimized the impact and frequency of the networking issue causing NA production batch failures. This incident is now resolved.
Further improvements will be delivered when enhanced batch servers and 2019 R2 are deployed.
Posted Nov 03, 2019 - 23:23 PST
Update
.
Posted Nov 03, 2019 - 23:01 PST
Update
Customers using the US Production environment may have been experiencing intermittent batch job and reconciliation failures. Support have identified the issue and deployed a hot-fix to remediate it.
Health checks and monitoring are currently being performed to ensure services are stable.
Posted Oct 31, 2019 - 16:21 PDT
Update
Support have restored the Inventory beacons policy download's functionality in the EU Production environment.
Batch processing in the EU production environment has been further optimized to improve import and processing performance.
Health checks and monitoring are currently being performed to ensure services are stable.
Posted Oct 31, 2019 - 04:53 PDT
Update
Batch processing in the EU production environment is currently processing normally, however some EU customers may notice inventory beacons policy download's occasionally timing out. This process will auto-retry every 15 minutes and will not impact customer licensing positions. Support teams are currently investigating this issue.
Posted Oct 30, 2019 - 21:17 PDT
Update
Batch processing in the EU production environment is currently performing normally. Once import tasks are complete, Customers impacted by the recent issues, will be able to see their current licensing position.
Posted Oct 30, 2019 - 05:38 PDT
Update
A performance issue was introduced into EU Production with 2019 R2. This performance issue was part of the License Reconciliation Transparency feature which was causing license reconciliations to fail, run excessively slowly or not run at all.
A hotfix was deployed at around 5am PDT / 12pm GDT October 29th to disable this feature. We are currently terminating tasks that are still running with the pre-hotfix code which will to allow subsequent tasks to be run with greatly improved performance.
Posted Oct 29, 2019 - 08:13 PDT
Update
As part of our remediation efforts, enhanced Batch processing has been deployed to the EU production environment. This has resulted in a backlog of tasks building up. This backlog may prevent some customers from viewing the latest data in FNMS while it completes processing.
Posted Oct 24, 2019 - 08:45 PDT
Update
Support teams have completed deployment of our enhanced Batch processing servers to the EU production environment. Deployment to the US production environment is currently planned to occur next week.
Posted Oct 24, 2019 - 02:50 PDT
Update
Support teams have completed development and testing of our enhanced Batch processing servers and will be deploying them to the EU production environment on October 23rd. Deployment to the US production environment is currently planned to occur the following week.
Posted Oct 21, 2019 - 22:50 PDT
Update
Support teams have applied additional performance tuning which has eased the pressure on Batch processing. This has resulted in minimal batch failures over the last 2 days. Support teams are currently working on a permanent solution to fix the underlying issue and we will be progressively rolling out the upgrade to UAT and then PROD over the next two weeks.
Posted Oct 08, 2019 - 20:02 PDT
Update
As part of our remediation efforts the Batch processor had been paused to allow further investigation - this has resulted in a significant backlog of tasks building up. This backlog may prevent customers from viewing the latest data in FNMS.
We are now resuming the Batch processor in order to process the backlog.
Posted Oct 07, 2019 - 10:32 PDT
Update
Support teams have identified the performance bottleneck responsible for the batch task failures and are currently planning a significant upgrade to our batch processing capability to remediate the issue. The upgrade is being planned for implementation within the next 7 days.
Posted Oct 03, 2019 - 16:54 PDT
Identified
Support teams have identified a performance bottleneck responsible for the batch task failures and are currently investigating potential solutions to remediate the issue.
Posted Oct 02, 2019 - 17:25 PDT
Update
Support teams have installed additional logging tools to assist diagnosing this issue and we are continuing to monitor the environment.
Posted Oct 01, 2019 - 21:41 PDT
Investigating
An infrastructure issue has been identified which has been intermittently impacting batch processing tasks. This may cause some customers licensing positions to unavailable.
Support are actively investigating this issue.
Posted Oct 01, 2019 - 18:42 PDT
This incident affected: Flexera One - IT Asset Management - North America (IT Asset Management - US Beacon Communication, IT Asset Management - US Inventory Upload) and Flexera One - IT Asset Management - Europe (IT Asset Management - EU Beacon Communication, IT Asset Management - EU Inventory Upload).