Table of Contents
Journal File Storage
Project journal items are obtained from field data and may either post processed from KMZ data or may be stored proactively by field (mobile) applications. As noted in the Access Web 2.0 story, project journals are composed of the four types of data entries:
- Photos
- User Notes (text)
- User Tracks
- User Areas (measure)
Field applications such as SmartDirt/SmartPlan/SmartGrade currently capture this information and save it the Access server by bundling them into KMZ files. There is much value to being able to access this data in the web server or other applications via the Access server API. The Access server will provide the capability to store and retrieve these items.
The intention to export these items to KMZ files requires that the original LLA coordinate information be retained. It is not appropriate to convert these items into an arbitrary NEZ coordinate space. Note that this still might be done be an application for display purposes, but it is not intended to do so within the file.
The existing server provides native storage for Photos (via files), but does not for Notes/Tracks/Measures. The server database server is inappropriate to store large items such as tracks and measures. The current scaleabale storage mechanism available is the file system. This proposal describes how to extend ADF to enable storage for these new data types.
Server Structure
In order to understand how the entire system will work, the server structure must be described. Journal items will either be extracted from KMZ files or stored piecemeal from field applications. The journal entries are maintained in an SQL table (per customer) which looks like:
| Field | Type | Null | Key | Default | Extra | Description |
|---|---|---|---|---|---|---|
| handle | int(16) | NO | PRI | NULL | auto_increment | Auto generated id |
| time | bigint(20) unsigned | NO | 0 | Std milliSecond date | ||
| project | int(16) | NO | -1 | Handle of the referenced project | ||
| user | int(16) | NO | -1 | Handle of creating user | ||
| codelist | int(16) | NO | 0 | Handle of the CodeList, 0 if not used | ||
| code | int(16) | NO | 0 | Handle of CodeList item, 0 if not used | ||
| type | varchar(16) | NO | unknown | String value indicating entry type | ||
| file | int(16) | NO | -1 | Handle of StoredFile for object | ||
| object | text | YES | NULL | JSON Representation |
This table allows user queries against the journal to obtain views of the journal limited by type, user, date, etc. The actual journal data will be stored in the Access filestore and the entry's 'file' attribute points to the data. The 'object' attribute is a very simple JSON object which contains more meta-data for the object. The attribute 'object' will be describe in a later section.
Entry Storage
Each entry type will be represent and stored as follows:
- Photo → <Project>/Photos/FOO.jpg
- Measures → <Project>/UserData/Measure1.adf
- Note → [No file] test is represented in the entry's object JSON.
- UserTrack → <Project>/UserData/Track1.adf
The directory “UserData” is a peer of other project folders. Because of the issue of data consistency, this folder is not visible to users, nor is it directly manipulated by Access clients. Directories in the AccessServer will be extended to include a “hidden” flag (boolean) which will accomplish this task. Because photos are stored in the <Project>/Photos directory and are user accessible, the Journal code must deal with possible dangling journal entries.
Entry Object JSON Schema
Each entry in the SQL table must be supplied with a corresponding JSON object.
In addition to the SQL table attributes, each entry has the following attributes in common:
- Latitude
- Longitude
- Altitude
The JSON object for all entries will minimally look like:
{
"latitude": number, // LLA
"longitude": number, // LLA
"altitude": number, // LLA
"note": "string" // A comment about the entry.
}
The location contained within the entry's JSON is referred to as the reference point. This point will be used by display applications when drawing a corresponding icon or glyph.
Photos, Tracks, and Areas all have their data stored within a corresponding ADF file (see section on ADF format). Textual notes do not have an ADF storage, the text of the note is stored as “note”.
ADF Structure
The ADF structure will support all of the currently defined Journal entry types. It should be noted that the the Photo Note entry type will not create ADF files in the server. The definition exists to support an ADF journal export use case.
All Journal entry's are stored under first Construction Model of the ADF. If a brand new ADF is bring created the Construction Model is the same name as the enclosing project.
The path to the Journal entry's are as follows:
/CM/JournalData/Photos/ /CM/JournalData/Areas/ /CM/JournalData/Note/ /CM/JournalData/Tracks/
Each section has a JSON descriptor, with corresponding data files. For instance, Photos, might look like:
/CM/JournalData/Photos/photos.json /CM/JournalData/Photos/Photo1.jpg /CM/JournalData/Photos/Photo2.jpg
Because notes can be entirely contained in the JSON, the notes structure looks like:
/CM/JournalData/Photos/notes.json
Tracks will look like
/CM/JournalData/UserTracks/usertracks.json /CM/JournalData/UserTracks/track1.track /CM/JournalData/UserTracks/tracl2.track
Areas will look like
/CM/JournalData/Areas/areas.json /CM/JournalData/Areas/Measure1.area /CM/JournalData/Areas/Measure2.area
While the structure is defined to support multiple data instances, the server may only create single ADF file per entry.
ADF JSON
Each entry type has a json descriptor which describes the rest of the data. Examples of the JSON for each type are:
photos.jpg:
{ "photos" : [
{ "latitude": nnn.nn,
"longitude": mmm.mmm,
"altitude": aaa.aa,
"time": tttttttttttt, /* Std mSec time */
"customer": "custid", /* Access customer ID string */
"user": "user@example.com" /* Access user ID string */
"comment": "text",
"file": "Photo1.jpg"
},
{ "latitude": nnn.nn,
"longitude": mmm.mmm,
"altitude": aaa.aa,
"time": tttttttttttt, /* Std mSec time */
"customer": "custid", /* Access customer ID string */
"user": "user@example.com", /* Access user ID string */
"comment": "text",
"file": "Photo2.jpg"
}
] }
Notes:
{ "notes" : [
{ "latitude": nnn.nn,
"longitude": mmm.mmm,
"altitude": aaa.aa,
"time": tttttttttttt, /* Std mSec time */
"customer": "custid", /* Access customer ID string */
"user": "user@example.com", /* Access user ID string */
"comment": "Text of note 1"
},
{ "latitude": nnn.nn,
"longitude": mmm.mmm,
"altitude": aaa.aa,
"time": tttttttttttt, /* Std mSec time */
"customer": "custid", /* Access customer ID string */
"user": "user@example.com", /* Access user ID string */
"comment": "Text of note 2"
}
] }
Tracks:
{ "notes" : [
{ "latitude": nnn.nn,
"longitude": mmm.mmm,
"altitude": aaa.aa,
"time": tttttttttttt, /* Std mSec time */
"customer": "custid", /* Access customer ID string */
"user": "user@example.com", /* Access user ID string */
"comment": "comment",
"file": "Track1.trk"
},
{ "latitude": nnn.nn,
"longitude": mmm.mmm,
"altitude": aaa.aa,
"time": tttttttttttt, /* Std mSec time */
"customer": "custid", /* Access customer ID string */
"user": "user@example.com", /* Access user ID string */
"comment": "COmment about track 2"
"file": "Track2.trk"
}
] }
Areas: Tracks:
{ "notes" : [
{ "latitude": nnn.nn,
"longitude": mmm.mmm,
"altitude": aaa.aa,
"time": tttttttttttt, /* Std mSec time */
"customer": "custid", /* Access customer ID string */
"user": "user@example.com", /* Access user ID string */
"comment": "An area",
"file": "Area1.trk"
},
{ "latitude": nnn.nn,
"longitude": mmm.mmm,
"altitude": aaa.aa,
"time": tttttttttttt, /* Std mSec time */
"customer": "custid", /* Access customer ID string */
"user": "user@example.com", /* Access user ID string */
"comment": "Measure 2 name"
"file": "Measure2.trk"
}
] }
ADF Block Types
Photos and Notes have defined data storage types, but Tracks and Areas do not. This section details two new binary ADF block types.
User Track Block (.track)
The Track block is a zipfile entry, which is a series of track points. A track is defined to have a starting point (the very first point) and an end point (the very last point).
Block Header:
- Block Code 2001 : Int32
- Block Size : Int32 = 40 for this definition
- Number of points : Int32
The series of points are as follows:
- latitude - Double 64 (degrees)
- longitude - Double 64 (degrees)
- altitude - Double 64 (meters above MSL)
- time - Int64 (std mSec time)
- compass - Double 64 (0..359.9999)
Area Block (.area)
The Area block is a zipfile entry, which is a series of points. A area is defined to be a closed region, e.g. the last point connects to the first point. There is no duplicated points to “close” the area, and it is expected that applications perform whatever operations needed to represent it as such. It is considered an error to close an area by duplicating a point for the first and last in this block.
Block Header:
- Block Code 2002 : Int32
- Block Size : Int32 = 24 bytes for this definition
- Number of points : Int32
The series of points are as follows:
- latitude - Double 64 (degrees)
- longitude - Double 64 (degrees)
- altitude - Double 64 (meters above MSL)
