Some normalization issues.

While shaping out code for the pilot data set analysis I've figured out browsing motifs and plots that normalization to the zero mean and unit of energy failed to produce "proper" results in the next two cases:
[1] when only single value in the sub-series
[2] when all values are the same, i.e. deviation = 0
which subsequently lead to some weird patterns.

Fixed!

Also still a bit confused by appearance of some "interesting" motifs like here:


The raw values look a bit unaligned to me, last symbol of motif is "c" for both, but this "c" in the red trajectory is less than "a" on blue. But I guess it's all about scale.

Pilot data set.

I've made plots of pilot data

raw	and Z-normalized

The PDF versions are here: The raw and normalized data.

Working on the Pilot data set.

Aiming the dissertation thesis proposal I'm working on the Trajectory code right now. The main change here is that the data comes now from the single project and categorized by users instead of simply navigating "anonymous" streams before. This change in the analysis flow pushed me to change a database schema and it looks now as follows:

Also, changes in schema and analysis forced me to rewrite bunch of the iBATIS queries, the coolest query so far is like that:
SELECT sm.id AS motif_id, sm.substring AS motif, sme.id AS entry_id,
(SELECT COUNT(*) FROM sax_motif_offset
JOIN sax_motif_entry ON sax_motif_offset.sax_motif_entry = sax_motif_entry.id
JOIN sax_motif ON sax_motif.id = sax_motif_entry.sax_motif
WHERE sax_motif.id = motif_id) AS entry_frequency
FROM sax_motif sm
JOIN sax_motif_entry sme ON sme.sax_motif=sm.id
JOIN chart ON chart.id = sme.chart
WHERE sm.sax_index = #value#
GROUP BY motif
HAVING entry_frequency > 1
ORDER BY entry_frequency DESC;
and retrieves all motifs for the specific index sorted by frequency.

litreview: almost there

I've finished incorporating Philip's suggestion and syntax fixes (thank you!) into my litreview and the last thing left is the bibtex formatted bibliography which for some reasons getting misformatted by TEX.
The litreview pre-final draft is here: litreview draft

Literature review draft finished.

I've spent last week working on the literature review. Finished the final draft and submitted it for the review.
While reviewing SAX papers I've got ideas and answers for the algorithmic questions I've hit coding my implementation (concerning the normalization and distribution issues). This week I'm going to embed this special cases into my code and get back on track with the Trajectory software.
Meanwhile I'm transitioning in writing from literature review to thesis proposal. Looks doable (-:.

First draft of litreview.

Last week spent reading and writing literature review. I think that I've put most of the stuff I wanted into the writing and just need to "refactor" the writing unifying all those math terms and making smooth transitions between chapters. Thinking of writing a short conclusion section too.
linky

Working on the literature review and digging some interesting stuff.

I'm keeping up with the last week plan and here is the Sunday evening draft of my literature review, it's not proofreaded or wrapped nicely, but bears all ideas I want to put in so far. I'm a little stuck on the lower bounding of distances, but will get over in couple days hoping to put together this stuff with walk through the time-series decomposition techniques by the beginning of the next week. Aiming the first full draft by the next Monday.

As per findings, checkout following plot and my previous thoughts about the Hackydatat distribution. After all looks pretty similar in some sense, but I've never read this article before.

Started literature review.

I've spent about a week working on the work stuff. I was finishing code for the annual review which happened last Thursday and was pretty favorable. So, very little was done on Trajectory browser or any writing, but starting from last Friday I'm working on the literature review.
My CSDL techreports project for this is ../09-08/.. and this is the plan of the topics for review:

1. Introduction

2. Time-series similarity measurements
2.1 Euclidean distance
2.2. Normalization of time-series
2.3. Transformation rules (smoothening, scales and shifts)
2.4. Dynamic Time Warping (DTW)
2.5. Longest Common Subsequence (LCS)

3. Dimentionality reduction (time-series decomposition) and indexing
3.1. Distance bounding (or why it works)
3.2. Discrete Fourier Transform (DFT)
3.3. Discrete Wavelet Transform (DWT)
3.4. Chebyshev Polynomials (CP)
3.5. Singular Value decomposition (SVD)
3.6. Piecewise Aggregate Approximation (PAA)
3.7. Adaptive Piecewise Constant Approximation (APCA)
3.8. Symbolic Aggregate approXimation (SAX)

Today I'm finishing drafting "Introduction" and "Time-series normalization" and will continue with distances next.

Current build of the Hackystat Trajectory browser.

Current build of the Trajectory browser incorporates a new motif representation for missing values with underscores and allows to select various SAX data models (build, universal and normal on the screenshot):

SQL Schema evolution

Working on the different cuts schema for the SAX moved me to add a new table to the TrajectoryDB; here it is:

Universal cuts for telemetry

Just coded a little function which is taking an array of data and deriving the number of cuts desired. It's purely based on the data, not on the distribution parameters: it simply counts how many data points will be in the each of the bins and makes these numbers equal. This is how cuts are looking for 5 letters alphabet:



From 21501 data values about 4200 ended up in each of the bins.

Picture of the data values distribution in 9 telemetry streams

For each of 9 streams I have in my local DB I've plotted three plots for the raw data (left three) and three plots for the normal data (right three).



I do think that it is really helpful to overlook these plots in order to design SAX cut intervals specific for the Hackystat-type data.

And this is the illustration of the problem I'm working on:


As you can see, the motif finder picked the values of the 1,2 and 29! as the same.

Hackystat telemetry data distribution

I've pulled all of the data available in my local database and plotted normal QQ plots for 9 types of telemetry to get an idea about underlying distribution: and I have no answer right now... it's not normal though

Latest improvements in the index browser tool.

Latest tool improvements:

- user can browse all built indexes;
- regular values chart is displayed along with normal chart, will probably change layout showing both together. *** I have used -1.0 value for N/A data, this is why it's so messed up. Need to solve this issue ASAP. ***



Next thing on the list is to see the distributions of the data and change SAX cutoff intervals favoring Hackystat-type data.

First working build.

Just managed to build and run the Trajectory search tool. Checkout some screenshots:





I'll post more detailed description later, have to run into the "cubicles area".

iBATIS + MySQL + MIGLayout

During last week I've finished improving the TrajectoryDB schema and moved from the tricky JDBC connector business to the iBATIS data mapper. DB things become much easier now and this is a schema I'm currently using:


My current goal is to find out if there are any interesting motifs among the data I have. Using MySQL querying + R scripting turned out to be a quite time-consuming search, so I've ended up hacking a neat GUI tool to conduct the search. Here is a current GUI snapshot:



For now I can choose particular motif (left top scroll table, frequency is the number of motif occurrences over all charts) and set from the projects/charts (second from the top scroll table at left, showing project, chart and frequency) it was found. I am planning to get all of the selection to be interactively rendered at the right panel using JFreeChart tomorrow. So, how cool is that?

SAX code and TrajectoryDB schema evolution

In order to improve the indexing speed and move towards automatic motif discovery I've updated DB schema making it more motif-centric and customized SAX code data structures allowing automatic motif frequency computation.

Due to the fair amount of failing data retrievals from sensorbase I've decided to create a local chart index and rewrite the telemetry data retrieval code.

Current db schema:


As you can see the sax_motif table is the top-level summary of found motifs. It should be easy to count and sort by the frequency and see DISTINCT projects where the particular motif occurs. Just need to write a code and populate the data now.

The confirmation of valid search

The figure depicts the result of unsupervised motif search among all the data available in the local sensorbase. I've run the search with only parameter of length 7 which would correspond a week and this is what I've got:

The first motif search results

Today I was able to index all of the data I currently have locally and there is a screnshot of highest frequences:


I've decided to pull more information about a couple of motif entries and there is a result:


As could be seen, for example, the motif `cgcccgi` was found in projects hackystat at position 23 and Default at position 2945, and motif `ccjjcccc` found at various position in the hackystat-sensor-* projects along with my compilers project at position 105 and so on...

But, the data distribution is still far away from normal :(

Sensorbase Index schema evolution

The new changes in the Sensorbase IndexDB are following the implementation of SAX indexing, currently schema, IMO, is unoptimized, but looks like it works. Currently I'm getting charts over the WAN from dasha and we'll see if I'll be able to find any "motifs" today.

Schema: