Access Server 2013 Robustification Project

The Access/License server has been running fine for several years but it has become apparent that the system is vulnerable to some “single point of failure” conditions.

There are many potential problems with the current Access server architecture. In a rough order of severity they are:

• Complete region failure (rare, but something close to this happened in 2010 or 2011). Fortunately the S3 buckets are not specific to a region.
• RDS or EC-2 instance failure (has happened in 2011 and 2012, about once per year)
• AWS fabric failure (at least once per year), e.g. S3, network, virtual host failure
• Storage and retrieval failures, mostly experienced by the Agtek Access Java Client
• Lack of Track redundancy due to them being stored in instance specific storage.
• Potential black hat attacks (mainly on the AccessWeb application)
• Through put of operations, appears to be DB related.
• Client failures (losing keys)

AWS Tools

• RDS snapshots, currently being done and retained for last 3 days. (50Gib) Can restore from a snapshot.
• EC2 snaps shots done once per day, keeping last two days (100 GiB) Can restore from snapshot.
• EBS Snapshots can be moved between regions.
• AWS console operations (new instance, snapshot, etc) can be automated.

Virtual Machine failure recovery strategies

• Quick reboot - Available via the AWS management console.
• Quick instance recreation - Either:
  • scripted in command lines tools on dev.agtek.com or
  • built into the AccessSupport tool (NOT keyforge)
  • Scripting is easy, but not easy to transfer the skills to another person
  • Building into the support tool makes them easy to use, but not as easy to adapt for future changes.

Recreate RDS, EC2 constellation in new region

• Quick instance recreation (as in last section): allow region specifier
• Migrate EBS / RDS instances : cross region migration/snapshot.

Security issues:

• Implement https for web application
• Add a security analyzer to look for anomalies and send alerts
• Include failure (404, 501, bad login) attempts in auto security analysis

Monitoring:

• Increase real time monitor goals to include:
  • Real time connection monitoring
  • Operation duration

Storage problems:

• Most storage issues appear to be related to the Access Java Client, fix it.
• Track storage can be moved to S3, increasing the safety of track storage.

• TBD: Identify robustness goals, recovery speed, etc. to guide solution selection.
• Attend April 30 AWS conference to get briefed on more AWS tech.
• Document modern (2013) system architecture
• Document failover process (for manual recovery), recovery procedures for failure modes.
• Modify AccessSupport tool to automate instance creation (from existing AMI-create snap of existing AMI, reattach EBS), recover EBS from snaps, repopulate DB from backup snap,
• Modify AccessSupport tool to copy snaps to another region to prep for region failover process.
• Consider auto copy snaps to another region for backup.

Likely only make these modifications when we rework a track product

• Move track storage to S3, integrate with Access Files.
• Integrate track api with regular API?
• Drop support for firmware loads on devices (old grey boxes).
• Drop support for TSMAdmin client
• Drop support for TSMAdmin server
  • SQL customer tables; assetid, association, gps, rtk, rtktrack, track, vehicle
  • SQL customer tables; tsm, tmm, device
• Remove SupportTool tabs for Devices, Trackwork Modules, Trackwork Servers (associated tables if not already present).

The following areas are routine maintenance items and/or feature requests that need to be done. The timing is right to do these at the same time as the other efforts.

• Upgrade the server infrastructure to the latest Java 7
• Add wildcard search to admin api for users
• Routine update of AMI Linux server upgrades (security)
• Possible update of entire Linux AMI (2013-03 variant released).
• Performance improvements: add index to problematic tables (licence, licenseuser, licenselog).
• Add licenselog pruning.
• File/Folder level permissions.
• Complete remove of the Box stuff.