Pavel's Software Engineering Log

Some normalization issues.

2009-05-31T13:00:00.004-07:00

While shaping out code for the pilot data set analysis I've figured out browsing motifs and plots that normalization to the zero mean and unit of energy failed to produce "proper" results in the next two cases:
[1] when only single value in the sub-series
[2] when all values are the same, i.e. deviation = 0
which subsequently lead to some weird patterns.

Fixed!

Also still a bit confused by appearance of some "interesting" motifs like here:

The raw values look a bit unaligned to me, last symbol of motif is "c" for both, but this "c" in the red trajectory is less than "a" on blue. But I guess it's all about scale.

Pilot data set.

2009-05-30T20:57:00.004-07:00

I've made plots of pilot data

raw	and Z-normalized

The PDF versions are here: The raw and normalized data.

Working on the Pilot data set.

2009-05-28T09:53:00.004-07:00

Aiming the dissertation thesis proposal I'm working on the Trajectory code right now. The main change here is that the data comes now from the single project and categorized by users instead of simply navigating "anonymous" streams before. This change in the analysis flow pushed me to change a database schema and it looks now as follows:

Also, changes in schema and analysis forced me to rewrite bunch of the iBATIS queries, the coolest query so far is like that:
SELECT sm.id AS motif_id, sm.substring AS motif, sme.id AS entry_id,
(SELECT COUNT(*) FROM sax_motif_offset
JOIN sax_motif_entry ON sax_motif_offset.sax_motif_entry = sax_motif_entry.id
JOIN sax_motif ON sax_motif.id = sax_motif_entry.sax_motif
WHERE sax_motif.id = motif_id) AS entry_frequency
FROM sax_motif sm
JOIN sax_motif_entry sme ON sme.sax_motif=sm.id
JOIN chart ON chart.id = sme.chart
WHERE sm.sax_index = #value#
GROUP BY motif
HAVING entry_frequency > 1
ORDER BY entry_frequency DESC;
and retrieves all motifs for the specific index sorted by frequency.

litreview: almost there

2009-05-11T21:55:00.003-07:00

I've finished incorporating Philip's suggestion and syntax fixes (thank you!) into my litreview and the last thing left is the bibtex formatted bibliography which for some reasons getting misformatted by TEX.
The litreview pre-final draft is here: litreview draft

Literature review draft finished.

2009-05-04T15:14:00.002-07:00

I've spent last week working on the literature review. Finished the final draft and submitted it for the review.
While reviewing SAX papers I've got ideas and answers for the algorithmic questions I've hit coding my implementation (concerning the normalization and distribution issues). This week I'm going to embed this special cases into my code and get back on track with the Trajectory software.
Meanwhile I'm transitioning in writing from literature review to thesis proposal. Looks doable (-:.

First draft of litreview.

2009-04-27T07:51:00.002-07:00

Last week spent reading and writing literature review. I think that I've put most of the stuff I wanted into the writing and just need to "refactor" the writing unifying all those math terms and making smooth transitions between chapters. Thinking of writing a short conclusion section too.
linky

Working on the literature review and digging some interesting stuff.

2009-04-20T14:46:00.005-07:00

I'm keeping up with the last week plan and here is the Sunday evening draft of my literature review, it's not proofreaded or wrapped nicely, but bears all ideas I want to put in so far. I'm a little stuck on the lower bounding of distances, but will get over in couple days hoping to put together this stuff with walk through the time-series decomposition techniques by the beginning of the next week. Aiming the first full draft by the next Monday.

As per findings, checkout following plot and my previous thoughts about the Hackydatat distribution. After all looks pretty similar in some sense, but I've never read this article before.

Started literature review.

2009-04-13T16:52:00.002-07:00

I've spent about a week working on the work stuff. I was finishing code for the annual review which happened last Thursday and was pretty favorable. So, very little was done on Trajectory browser or any writing, but starting from last Friday I'm working on the literature review.
My CSDL techreports project for this is ../09-08/.. and this is the plan of the topics for review:

1. Introduction

2. Time-series similarity measurements
2.1 Euclidean distance
2.2. Normalization of time-series
2.3. Transformation rules (smoothening, scales and shifts)
2.4. Dynamic Time Warping (DTW)
2.5. Longest Common Subsequence (LCS)

3. Dimentionality reduction (time-series decomposition) and indexing
3.1. Distance bounding (or why it works)
3.2. Discrete Fourier Transform (DFT)
3.3. Discrete Wavelet Transform (DWT)
3.4. Chebyshev Polynomials (CP)
3.5. Singular Value decomposition (SVD)
3.6. Piecewise Aggregate Approximation (PAA)
3.7. Adaptive Piecewise Constant Approximation (APCA)
3.8. Symbolic Aggregate approXimation (SAX)

Today I'm finishing drafting "Introduction" and "Time-series normalization" and will continue with distances next.

Current build of the Hackystat Trajectory browser.

2009-03-30T06:38:00.003-07:00

Current build of the Trajectory browser incorporates a new motif representation for missing values with underscores and allows to select various SAX data models (build, universal and normal on the screenshot):

SQL Schema evolution

2009-03-29T17:46:00.001-07:00

Working on the different cuts schema for the SAX moved me to add a new table to the TrajectoryDB; here it is:

Universal cuts for telemetry

2009-03-29T04:12:00.005-07:00

Just coded a little function which is taking an array of data and deriving the number of cuts desired. It's purely based on the data, not on the distribution parameters: it simply counts how many data points will be in the each of the bins and makes these numbers equal. This is how cuts are looking for 5 letters alphabet:

From 21501 data values about 4200 ended up in each of the bins.

Picture of the data values distribution in 9 telemetry streams

2009-03-28T09:02:00.004-07:00

For each of 9 streams I have in my local DB I've plotted three plots for the raw data (left three) and three plots for the normal data (right three).

I do think that it is really helpful to overlook these plots in order to design SAX cut intervals specific for the Hackystat-type data.

And this is the illustration of the problem I'm working on:

As you can see, the motif finder picked the values of the 1,2 and 29! as the same.

Hackystat telemetry data distribution

2009-03-26T19:02:00.002-07:00

I've pulled all of the data available in my local database and plotted normal QQ plots for 9 types of telemetry to get an idea about underlying distribution: and I have no answer right now... it's not normal though

Latest improvements in the index browser tool.

2009-03-25T15:55:00.002-07:00

Latest tool improvements:

- user can browse all built indexes;
- regular values chart is displayed along with normal chart, will probably change layout showing both together. *** I have used -1.0 value for N/A data, this is why it's so messed up. Need to solve this issue ASAP. ***

Next thing on the list is to see the distributions of the data and change SAX cutoff intervals favoring Hackystat-type data.

First working build.

2009-03-23T09:50:00.005-07:00

Just managed to build and run the Trajectory search tool. Checkout some screenshots:

I'll post more detailed description later, have to run into the "cubicles area".

iBATIS + MySQL + MIGLayout

2009-03-22T21:53:00.006-07:00

During last week I've finished improving the TrajectoryDB schema and moved from the tricky JDBC connector business to the iBATIS data mapper. DB things become much easier now and this is a schema I'm currently using:

My current goal is to find out if there are any interesting motifs among the data I have. Using MySQL querying + R scripting turned out to be a quite time-consuming search, so I've ended up hacking a neat GUI tool to conduct the search. Here is a current GUI snapshot:

For now I can choose particular motif (left top scroll table, frequency is the number of motif occurrences over all charts) and set from the projects/charts (second from the top scroll table at left, showing project, chart and frequency) it was found. I am planning to get all of the selection to be interactively rendered at the right panel using JFreeChart tomorrow. So, how cool is that?

SAX code and TrajectoryDB schema evolution

2009-03-16T06:44:00.004-07:00

In order to improve the indexing speed and move towards automatic motif discovery I've updated DB schema making it more motif-centric and customized SAX code data structures allowing automatic motif frequency computation.

Due to the fair amount of failing data retrievals from sensorbase I've decided to create a local chart index and rewrite the telemetry data retrieval code.

Current db schema:

As you can see the sax_motif table is the top-level summary of found motifs. It should be easy to count and sort by the frequency and see DISTINCT projects where the particular motif occurs. Just need to write a code and populate the data now.

The confirmation of valid search

2009-03-12T11:58:00.003-07:00

The figure depicts the result of unsupervised motif search among all the data available in the local sensorbase. I've run the search with only parameter of length 7 which would correspond a week and this is what I've got:

The first motif search results

2009-03-12T06:32:00.007-07:00

Today I was able to index all of the data I currently have locally and there is a screnshot of highest frequences:

I've decided to pull more information about a couple of motif entries and there is a result:

As could be seen, for example, the motif `cgcccgi` was found in projects hackystat at position 23 and Default at position 2945, and motif `ccjjcccc` found at various position in the hackystat-sensor-* projects along with my compilers project at position 105 and so on...

But, the data distribution is still far away from normal :(

Sensorbase Index schema evolution

2009-03-12T05:58:00.002-07:00

The new changes in the Sensorbase IndexDB are following the implementation of SAX indexing, currently schema, IMO, is unoptimized, but looks like it works. Currently I'm getting charts over the WAN from dasha and we'll see if I'll be able to find any "motifs" today.

Schema:

SAX based time series motif search primer

2009-03-11T05:40:00.007-07:00

This post is an illustration of the SAX-based motif search in timeseries.
Let's assume that we are given the next timeseries:(1,1,3,5,8,7,6,2,3,4,3,1,1,5,4,5,2,3,4,6,9,8,5,2,3,4,2,5,3,2,5,6,8,9,0,3,3), the length is 37 points:

Now if we run the SAX algorithm using sliding window of size 7, no PAA and alphabet size 6, this is what the resulting substrings matrix looks like:

[aabdfee, abdfeea, bdfeeab, dfeeabb, ffeabcb, ffbcdca, fbdedaa, bdedaaf, dedaafe, dcaafdf, daafefb, aafefbd, afdfbcd, eceabcf, cdabcef, cabbdff, abbdffc, bbdffca, bdffcab, dffcabb, ffdbbcb, fdabcad, faceafc, bdebfdb, ceafcaf, daebaef, adbadef, cbacdff, bbddffa, bddffab, ddffabb]

if we compute all pairwise distances over this strings we will find next matches (distance between strings is zero, and the second column indicates substings indexes):

ffeabcb - ffdbbcb, [4] - [20]
cabbdff - cbacdff, [15] - [27]

which corresponds to next two motifs:

Indexing the Sensorbase and puzzled with the distribution (for now)

2009-03-09T15:33:00.002-07:00

I'm following the SAX approach for the database indexing and currently pulled some data to run the distribution analysis and this is the current schema for the database I'm using:

What I am doing right now is running the "sensorbase crawler" which pulls all the projects summaries available for the given user and then pulls all the charts possible for the each project (and member). After that data get normalized, transformed into the SAX representation and stored in the database.

While I'm working on the configuring charts retrieval (those parameters), I am worrying about the distribution of the data points from Build and Devtime streams: it looks not normal, - rather exponential. I'm planning to pull other streams from the sensorbase and work a little more on the data distribution analysis and if it'll be the same I guess it will require SAX schema correction.

PAA and SAX R implementations

2009-02-27T11:25:00.006-08:00

Just want to post the figure I've got in R while designing test cases for the Java implementation of the PAA and SAX algorithms. The figure shows original trajectories, their normalized versions, PAA reduction from 14 to 10 points and finally the result of SAX transform into string.

Updated! Linky to the R code

Updated test results:

And the Java-based library I am working on

The lower bounding distance example:

SAX and iSAX

2009-02-23T11:55:00.003-08:00

While I was following experiments from the previous week with similar setup trying to see the performance of DTW on the simpler trajectories I was continuing reviewing literature and hit the Symbolic Aggregate approXimation paper and SAX algorithm. What a find! Next were iSAX and MK. This algorithm for finding motifs (patterns) in time-series looks very good and after studying all this material I've decided to contact authors for Java or R code to try it on the telemetry data, - they replied quick but unfortunately they had no Java/R code in hands, only Matlab/C# BUT they were willing to help with Java code development. Currently I am setting up the project and going to code timeseries motif finding module for the Hackystat.

Euclidean and DTW metrics based sequence (subsequence) matching

2009-02-17T07:19:00.004-08:00

Just did a small experiment with R using Euclidean and DTW metrics.
So, I had two synthetic timeseries in hands - one was a kind of devtime telemetry trend which I was using as the reference and the second one just a "peak pattern" The purpose was to see if this "peak pattern" will match anything on the reference timeseries and if it will, than what exactly.

So, figure 1 represents both timeseries and their normalized versions

Figure 2 shows the same normalized timeseries at the top, found matches for "peak pattern" when using Euclidean distance at the middle, and Euclidean distances at the bottom of the figure (red bars correspond to the set of the minimal found distances (7 in total)).

Figure 3 is very similar to the number 2, but the DTW distance is used.

Figure 4 mimics #3 but threshold is lowered, so more matches shown.

The R code is here. Will follow with more details in the next post.

Working on the DTW implementation and Tests while planning the subsequence search implementation.

2009-02-16T07:22:00.005-08:00

Last week I was working on the DTW package for the Trajectory project. I've spent about 12 hours hacking the Trajectory DTW java package itself: now it performs DTW alignment using most of the step patterns (it was quite interesting to learn about IEEE standards for NaN (not a number)) about the same amount of hours I've spend working in R making test templates and verifying Java methods; hope all the R plots I've made will contribute to litreview/thesis.

I did some research and collected available time-series normalization techniques and typed all in TeX for the literature review and thesis.

Right now I am working on implementing DTW Constraints in Java and on the example of the time-series similarity search. And... it seems like it does work in R for now, I'll post the results tomorrow, today it is too late to process all the plots and clean-up the code.

Progress report

2009-02-09T17:12:00.002-08:00

Last week I've finished the introductory part of the literature review concerned the time-series definition, history and classical analyses development. FYI: the first time-series plot is dated by the 10th century.

Currently I am reviewing literature concerning metrics used in the similarity search, specifically: Euclidian and Edit distances and their applications in DTW and LCS algorithms. While it seems sound somehow trivial, the time-series transformation rules defined in the research literature through years such as a moving averages smoothing, local and global scales and shifts are making life a little bit harder. The further development through the piecewise matching of the time-series without saving the temporal ordering moves it to the next level of complexity while the concept seems to be being valid for the software development which consists of episodes.

I'm drafting the review flow by walking through the time-series similarity measurements approaches while arranging the relevant research within sections chronologically. Think it should work.
Getting ready to move further in the review to the indexing/clustering of time-series methods.

Writing up a literature review.

2009-02-03T00:53:00.002-08:00

Current progress.
I've spent a week working on the literature review. I finished the introductory part and skimmed through couple of classical time-series analysis books refreshing my knowledge of the well established methods of the time-series analysis and forecasting such as autoregressive ARMA/ARIMA models and based on these forecasting, lag analysis and spectrum analysis of the time series.

I think that while all these classical time-series analysis could be potentially used in the trajectory application for finding similarity between time series, it seems to be that this direction would require far more computations to be done (decomposing the time series) and would require a whole bunch of preliminary theoretical work researching specific models suitable for the software development and proving them, which seems to be really unnecessary and moreover impossible.

Interesting that while reading books and walking through the given practical examples from econometrics, I found that interrupted time series and lag analysis could an useful addition to the Hackystat analyzes family. Specifically it would be valuable to implement some kind of such analysis modules to see whether or not some development or managerial events (like (i) stopping regular development activity and switching to test coverage boosting; or (ii) adding/removing a developer to the team; (iii) switching development approach to TDD etc) really impact the development trends and how.

Plans.
This week I will be reviewing the time-series similarity measures and all kinds of applications based on this approach.

It is in my plans to start research on the time-series database indexing after the similarity measurement.

I hope that once these three parts will be done I will cover pretty much all of the stuff that I need to finish the literature review and sketch my path with dissertation proposal. Can't wait to get to this point.

Note on the development
While writing up the review I've started the design of the time-series analysis sub-module, DTW sub-module and thinking on the database (sensorbase) extension for the indexing.

User-account environment variables under Vista

2009-01-27T04:36:00.002-08:00

I had some problems running the TexLive binaries under Vista caused by the conflict between the Cygwin TeX installation and TexLive. TeXLive installer adds the path to binaries as the user-specific environment variable which than added to the very end of the PATH variable and consequently TeXLive binaries are never reached. The way to fix this issue is to reshuffle the variables to make TexLive binaries to be loaded by default. This should help to find your user environment settings

Literature review plans

2009-01-26T12:35:00.005-08:00

Finally I was able (almost) to install the TexLive on my laptop. The installer never finished so I don't really know if everything works (the 00-00 example compiles though). I'm running Windows Vista and it looks like the TexLive installer has permission issues with non-Administrative account + there seems to be a bug with installer when it tries to use system Perl instead of one shipped with installer (which causes some libraries and runtime issues). It took me some time to figure out both issues and I don't really see advantages of TexLive versus MikTEX at this point. The one more disadvantage I see is the lack of the DVI viewer in the TexLive distro, looks like I need to install the viewer, but I'm not sure if I will need it, so we'll see further if I will need one.

From other things I've set up all the Java stuff, updated libraries and Hackystat, checked out all the latest sources etc. and backed up the whole thing, just in case.
So, system is ready to go.

Most of the time I spent on putting together the outline for the literature review. I am seeing the purpose of this writing to be a comprehensive walkthrough through the field of the time-series analysis outlining the milestones and major discoveries and connecting them with my research. I found that I've totally missed some major things in the time-series analysis (funny huh?) and filling these gaps with reading and collecting the literature.

Following is the draft plan, I'm working on the third part and since it is based on the material from the part 2, I am changing its flow too.

Literature review plan

Introduction. (definitions, research field boundaries and common applications)
1. Introduction to time series.
  1. Data sources, time-series representation and common applications
    
    (the time series “origin”, common representation and mainstream applications)
  2. Streaming time-series.
    
    Time series as streams.
  3. Time-series databases and indexing
    
    (examples of existing time-series collections (+ the Hackystat sensorbase) and common time-series databases toolkit for time series data storage, search and retrieval)
2. Classical time series analyses.
  1. General exploration & description
    
    (time series descriptive exploration and common tools used: spectral analysis, autocorrelation, trends, periodicity (+ Hackystat Telemetry, + Hackystat Zorro?, + Hackystat Trajectory))
  2. Prediction and forecasting
    
    (stochastic modeling: AR, MA, ARMA, ARIMA and uses (+ Hackystat Trajectory))
3. Time series similarity (homogeneity) based analyses.
  1. The speech and handwriting recognition.
    
    (pioneering the area of DTW, LCS and HMM)
  2. Sign language, motion and gesture recognition.
    
    (ongoing research)
  3. Trajectory patterns recognition, surveillance applications, shape recognition.
    
    (modern applications)

Time series similarity-based analyses and algorithms

(known research tools, implemented applications and up-to date research directions)
1. Similarity metrics
  1. Euclidean distance.
    
    (application and problem of normalization)
  2. Hamming and Edit distances.
    
    (the formal introduction of edit distance, time-series transformations)
2. Similarity-finding algorithms
  1. DTW
  2. LCS
3. Methods (whole and sub sequence applications)
  1. Clustering
  2. Indexing
  3. Classification
  4. Anomaly detection
4. Known state of the art applications.

Possible application of the algorithms and methods to the Hacvkystat Telemetry Streams
1. Similarity search in the Sensorbase
  
  (the search for similarity using the raw telemetry data stored within the sensorbase)
2. Telemetry Streams data Indexing
  
  (defining the Telemetry patterns, indexing raw telemetry data using definitions and conducting search be means of indices and Edit distance)
3. Live Telemetry Stream analysis and features
  
  (patterns, anomaly detection)

First post in the 2009

2009-01-20T02:17:00.004-08:00

The TechReport for 699 course. The importance of. :)
I've finished the Fall 2008 semester writing a technical report concerning the DTW algorithm, it's existing implementations, uses and extensions and outlined possible application to the software metrics based on my own implementation. While being a little worrying about this before starting I found the report writing as an extremely helpful and interesting activity for the number of reasons:

first of all it forced me to summarize and distill all of the essence from the work done so far and get it graded. By doing so I've not only reassessed my current position in the research but what I found extremely useful - I was able to identify gaps (weak points) in the research I am doing. The goal now is to make my research complete through evenly covering all of the areas of the interest and connecting it with adjacent fields.

secondly I found that in my case writing the "tech report for the 699" is actually more likely writing a draft (or an outline) of the Literature Review and the Thesis Proposal: two pieces of writing which are required for the PhD degree. How cool is that?

in addition to these two items, overviewing the research done so far, seeing things in the ToDo list and having outlined LitReview and Proposal makes ones (my) minds clear and brings confidence of the right track chosen.

Current progress.
This week I am working on the setting up the "working environment" for this semester. Taking in account the amount of the software development and writing ahead, spending some time on selecting technologies, cleaning and organizing the system environment and hardrive, updating tools etc seems to be a reasonable activity.

As in all previous work I am going to use Java, Eclipse, Hackystat infrastructure and the standard set of Hackystat libraries for the core programming. More likely I'll be using MIG Layout for the UI development of the stand-alone Trajectory tool. The R will be used for making figures and fast-scripting when I need to test something before actually implementing.

For my latest report and all other previous LaTeX-based documents I was successfully using a combination of the MikTex and TeXnicCeneter, but the recent changes in the CSDL requirements moving me towards the use of the TexLive and currently I'm setting up tools and environment testing this new for me approach.

The 699 report continued.

2008-12-08T18:36:00.004-08:00

My work last week was focused on the writing 699 report for the Fall'08. The report essentially consists of the next parts:

Introduction

DTW Algorithm
DTW Customization

Weighting

Global path constraints

DTW Optimization

Query by the sample

Software metrics application

Telemetry database indexing

Future work

So far I'm done with first two sections and working on the third one. The progress a bit slow mainly because I was working on the last home assignments for my classes: The Analysis of Algorithms and Engineering Compiler. The finals scheduled at Dec 18 and Dec 19, Hope to finish the most of the report work by this time.

The report is hosted by Google and the permanent link is PDF

The 699 report

2008-12-02T12:05:00.002-08:00

Last week I've tried to evaluate my progress and started the 699 Fall 08 report using TeX. Most of the time was spent collecting and reading articles. After all I've organized them through CiteULike. (Wondering if there any way to share my library with any other person?) Right now I am having around three dozens of DTW algorithm related papers describing basics of DTW, it's customization and optimization along with various applications: data mining (time series databases search), time series clustering, online signature matching, computer vision, computer animation, surveillance, protein sequence alignment, chemical engineering, music and signal processing. After reading I'm truly believe that approach would work applied to software metrics.

Find current report version here.

DTW Step performance in Trajectory

2008-11-20T05:42:00.003-08:00

Today I've hacked in the DTW step implementation in the Trajectory. Take a look on the impact of the step choice on the Euclidean distance after alignment and plots rendering.

Symmetric Pattern 0

Asymmetric Pattern 05

Step functions performance illustrations

2008-11-19T19:45:00.006-08:00

A simple example demonstrating the performance of different step function following "Dynamic programming algorithm optimization for spoken word recognition"
by: H Sakoe, S Chiba and using matrix from T. Giorgino (2007). "dtw: Dynamic Time Warping algorithms in R 1.12-3"

The distance matrix.

The step functions performance, red - the result of DTW alignment, black - the true path of zeroes.

Trajectory is up and running within the ProjectBrowser

2008-11-18T06:44:00.002-08:00

I've fixed CSS styles and did some code changes within the GoogleChart package (due to the precision lost while converting from doubles I wasn't able to see normal and DTW plots at all). The trajectory runs on my workstation, will see today how it deals with bigger datasets. My next step would be to implement constraints, step patterns and open end alignment classes plugging them into UI.

Who broke the build!?

2008-11-10T15:09:00.003-08:00

Last week I've cleaned up the Trajectory code and finally merged my branch with the trunk. I've broke the build. I'm sorry. Thank you Philip for prompt response and help fixing it.

Among other things, since I've forked it long ago, I had some troubles with code layout and Trajectory page is still messed up since the div's name changed within HTML templates. Next week will fix CSS's and HTML's for the trajectory. Good news that it performs as it suppose (at least on my workstation).

Started organizing my literature review using CiteULike.

And after all I'm coding scanner and parser parts for the compiler :P...

Progress report

2008-11-05T05:15:00.002-08:00

I've finished modifying code architecture for the DTW and improved data normalization for the time series. The original Telemetry and GoogleChart modules code which I borrowed for modification appeared to have three or four validation and format routines which were designed the way which prevented normalized data to be rendered properly. I've changed/commented out validations and formatting but not sure if it was right decision, I feel like I need a little more time to see what was behind this formatting and will post results later.

The most current plots:

Progress report

2008-10-27T03:25:00.006-07:00

Last week I was working on two tasks: implementing required code changes in the Trajectory component and second is writing some ideas for the technical report.

Hacking progress:

Time-series normalization: Initially I was normalizing the time series by simply dividing all values by the maximal one, last week I've changed this method to more appropriate one by following the method suggested by Dina Q. Goldin and Paris C. Kanellakis in the "On similarity queries for time-series data: Constraint specification and implementation" chapter of "Principles and Practice of Constraint Programming — CP '95" book.

DTW improvement: I'm trying to move from the original DTW implementation towards the more customizable implementation using step functions suggested by Sakoe-Chiba:
.
Unfortunately hack is no moving fast, - I am working on the architecture design to make step and constraints functions pluggable into the "generic DTW" code. I think this is the most elegant solution. In the future it'll allow to plug any user-configured function in the code. Really want to benchmark the performance of all the options since for the speech and writing recognition it worked well.

Book: Information Retrieval for Music and Motion.

2008-10-26T09:08:00.003-07:00

Want to share one of my findings: while collecting papers for the literature review, I found a very interesting book related to the research I am doing: "Information Retrieval for Music and Motion", Müller, Meinard, 2007, XVI, 318 p. 136 illus., 39 in color., Hardcover. ISBN: 978-3-540-74047-6.
After reviewing I've ordered it, waiting for the stuff to arrive.

A general scenario that has attracted a lot of attention for multimedia information retrieval is based on the query-by-example paradigm: retrieve all documents from a database containing parts or aspects similar to a given data fragment. However, multimedia objects, even though they are similar from a structural or semantic viewpoint, often reveal significant spatial or temporal differences. This makes content-based multimedia retrieval a challenging research field with many unsolved problems.

Meinard Müller details concepts and algorithms for robust and efficient information retrieval by means of two different types of multimedia data: waveform-based music data and human motion data. In Part I, he discusses in depth several approaches in music information retrieval, in particular general strategies as well as efficient algorithms for music synchronization, audio matching, and audio structure analysis. He also shows how the analysis results can be used in an advanced audio player to facilitate additional retrieval and browsing functionality. In Part II, he introduces a general and unified framework for motion analysis, retrieval, and classification, highlighting the design of suitable features, the notion of similarity used to compare data streams, and data organization. The detailed chapters at the beginning of each part give consideration to the interdisciplinary character of this field, covering information science, digital signal processing, audio engineering, musicology, and computer graphics.

This first monograph specializing in music and motion retrieval appeals to a wide audience, from students at the graduate level and lecturers to scientists working in the above mentioned fields in academia or industry. Lecturers and students will benefit from the didactic style, and each unit is suitable for stand-alone use in specialized graduate courses. Researchers will be interested in the detailed description of original research results and their application in real-world browsing and retrieval scenarios.

Working on the Trajectory UI.

2008-10-20T05:04:00.005-07:00

I've enhanced the Trajectory DTW UI over the last week following my initial hand drawing:

It seems to be working fine and next week I am planning to implement the actual DTW algorithm enhancements.

Technical review outline.

2008-10-13T18:59:00.006-07:00

During the Summer and Fall of '08 I was doing research focusing on the "telemetry trajectory" idea which resulted in the three software components, which are "in the development stage" and will constitute the three major chapters of my future report:

Software development activity simulation package. I've developed the sensorbase-sim extension package based on the JADE framework. It is fairly simple agent-based application which generates software metrics data once provided with simulation parameters for the code, such as overall functional points requirements, points per class, LOC's per point etc... and developers data, such as number of developers and their expertize level.

While I've found a large amount of the approach-justifying literature and methods to continue the simulation package was shalved till better times and I focused on the Trajectory-UI development.

Second application I was working on was a stand-alone Java application for the visualizing trajectories in the 3D space.

This work was following Philip's original idea about trajectory visualization.
This application is based on the Java3D package and it's main purpose is the exploration of the trajectory visualization approach. While working on the visualization implementation I was going through the trajectory-visualization related materials review and discovered the vast amount of research done in the area of the complex trajectories matching. Part of the reviewed articles will stuff the report.

The most active development and research I am conducting right now now is on the Trajectory component of the Hackystat-ProjectBrowser application. Currently I am exploring the DTW algorithm application for the software telemetry data. Considering the amount of the work and research I've done this would be the major component of my report describing in the details the DTW application to the telemetry data.

UI design

2008-10-13T18:00:00.006-07:00

Some of the time during last week I've dedicated to the DTW related research and UI design: I was thinking about laying out the workspace for the DTW analysis since after all it suppose to be somehow interactive and iterative. It's very important that UI must be responsive and keep user focused on the analysis aspects. By following the UI-First Software Development principle I've spend about 10-20 minutes just drawing the draft of UI laying out information blocks and parameters at the left screen "control panel".
.
The function which is underlying under UI design is pretty simple, and I outlined three major steps on my drawing:

User sees the input time-series statistics and two accompanying plots: one for the raw timeseries data with gaps and non-normalized peaks representing the data collected from the sensors and second one, with normalized series and gaps filled accordingly to the telemetry type.

Based on the provided information and visualization user selects specific DTW options and actually fires the timeseries alignment action.

Once trajectories aligned, the third plot showing aligned warped timeseries and alignment statistics updated. After evaluating analysis results user makes a decision to alter parameters and run alignment again or just accepts the alignment results.

Currently I'm altering the Wicket application code, HTMLs and CSS styles to implement the UI.

Some issues with DTW I'm working on.

2008-10-06T19:30:00.008-07:00

In couple of previous posts (except one about precision) I was discussing two issues in pattern matching which are apparently similar in the speech recognition (utterance matching) and in the software metrics matching analysis. I read the article by
Rabiner, L.; Rosenberg, A.; Levinson, S., "Considerations in dynamic time warping algorithms for discrete word recognition," Acoustics, Speech and Signal Processing, IEEE Transactions on , vol.26, no.6, pp. 575-582, Dec 1978 and trying to rephrase the reasoning applying it to the software metrics alignment.

Most recently discussed issue: the assumption that the telemetry time series should be the same length. In my opinion this constraint is very strong and we should not consider this in the most cases. Whether or not waterfall or agile processes are considered there are certain activity patterns (phases) already identified. What is not canonized is the length of each of the phases and we should be considering an "open-ended" alignment algorithm to match those process features.

The second issue discussed before is the allowable changes in the warping function. By adjusting this allowances we basically adjusting the sensitivity of the algorithm to recognize peaks of activities as the independent events. Question is what is the range of optimal allowances?

The last issue I was thinking of is whether or not there are some hidden preferences in the choice of the "query" or "template" time series. So far I would say that there are no difference, but I would work on the experimental proof that the warping function is symmetric (or not?).

While researching, I discovered this book, which potentially is the greatest answers source for my questions. Readings in Speech Recognition By Alex Waibel, Kai-Fu Lee

I've seen the following figure in the article and in the book and it seems to be illustrating two first questions I am asking here: first two plots are illustrating the effect of allowances in the warping function, while the third plot (UELM) illustrates the "open-ended alignment" approach.

Open-ended alignment.

2008-10-05T16:08:00.004-07:00

So... precision errors... Once I've made through this I hit another bump: in my previous tests I was always comparing time-series of the exact same length, 15 days. Meanwhile the Trajectory interface allows you to select time series of arbitrary length and moreover to introduce indent before data, which more likely makes time-series lengths different. :). For example like this:

What would happen once I'll hit the Do DTW Analysis button? Right - Exception, due to the difference in length.

While I can restrict users to do only the same length time-series analysis, it is very strong constraints. It would require huge amount of the eye-matching before actually going into the DTW. ...

To be continued later today.

Precision errors....

2008-10-05T15:45:00.004-07:00

If you were following my posts you might have noticed that there is significant difference in the appearance of the plots after DTW between the R implementation and my Java implementation. It was a point of my frustration for the couple of weeks. I spent hours trying to extract the underlying numbers and debug what is going on. Guess what? Precision errors contribute to that.

Example: how DTW works 3 peaks and 2 peaks.

2008-09-28T19:20:00.011-07:00

A simple example of DTW algorithm for two trajectories, one has three peaks, while the second one two peaks. In my opinion two trajectories are distinct enough to consider them non-similar: clearly, trajectory #1 has two minor activity peaks by sides of major activity peak while trajectory #2 has two similar peaks of activity during the time trajectory #1 is idle. The task of this exercise is to explore the DTW toolkit which prevents this trajectories from aligning. In particular, I'd like to try the Sakoe-Chuba Band and Itakura Parallelogram as shown at Figure 4 of Salvador, Stan and Philip Chan. FastDTW: Toward Accurate Dynamic Time Warping in Linear Time and Space, Intelligent Data Analysis, 2007

The original data plotted

Time series

Naive DTW DP in action

Before and After DTW

My Java DTW implementation finally works within the ProjectBrowser

2008-09-28T15:16:00.006-07:00

During last week I've fixed my DTW Algorithm implementation in Java. Next two screenshots are showing the reference R implementation at the left side and my Java implementation at the right side. They both happened to be identical before DTW and after DTW.

And the immediate effect on the Trajectory tool visualization:

Placeholder for the Monday, September, 22

2008-09-22T21:31:00.005-07:00

Basically it's about 10:32PM (MST) and I am late with posting my entry. Just found why my DTW implementation doesn't follows the reference one: I was not averaging points which were merged during the DTW along with NOT interpolating ones which happen at gaps. The bug is understood, but I am not feeling like patching it right now: it requires writing tests and altering some code. Will update this blog entry tomorrow. Meanwhile I went through the "spaghetti code" sections and some telemetry artifacts remaining in my code and cleaned it up, however I think that the sequence of events to get three plots as in the previous entry is still far from the perfect. Also I am working on the DTW panels UI and cleaned up the streams statistics from some junk along with introducing a little DTW statistics panel. See you a little later :).

Naive DTW is up and running.

2008-09-15T17:58:00.007-07:00

So I've made it through the Wicket data models and ended up with a lot of a spaghetti code for this version while refactoring telemetry/trajectory code and my previous DTW code out of trajectory. Look at the amount of debug information I am printing, (haha!):

Voila:

The trajectory convenience in action, "the base page":

If you can find a little "Do DTW analysis" button right to the "Get Chart" button and click it this is what you'll see:

Basically along with the streams statistics at the left you can see the sequence of three plots:
1) the original trajectory/telemetry chart,
2) the normalized timeseries which have Zeroes instead of NA... (i know it'll work for DevTime chart, but I need to figure out what to do with other streams.)
3) the DTW-treated normalized time series.

Nice... However I found that it is somehow different from the R implementation I am using as the reference:

Right now I am debugging what is going wrong with my Dynamic Programming implementation and meanwhile trying to clean-up my code from repeating myself. While debugging, I found an awesome new feature in the Excel 2007: Conditional formatting, just look at this beautiful heat maps:

Bottom line for this week progress is that Naive DTW is implemented and embedded in the ProjectBrowser. You would ask, "So what about "similarity"?", well, I have a code to calculate the Euclidean distances between time series too, and have numbers calculated, just need to find a place where to render those at UI panels.

Next week plan is: code cleanup and optimization, implementation of the R Charting instead of GoogleCharts.

Implementing DTW in trajectory

2008-09-08T19:30:00.004-07:00

In the my previous post about hackystat-trajectory progress I've said that I'm almost there with implementing the indent for the trajectory charts as at the following screenshot:

Yeah, in fact it took me about 3 hours of typing (I type using two fingers :P) since I've got to alter the code for the whole bunch of TelemetryStream dealing pieces... So, basically my initial estimation of the effort needed turned out to be wrong :(. Anyway, the indent is in the place and now I have the ability to choose particular intervals from the telemetry streams for two projects and align them the way I like.

The next research question I'm working on right now is how to quantify the difference (similarity) level of two trajectories? My idea of UI and workflow for this task is that once an user is satisfied with time series (telemetry interval) selection and alignment using the Trajectory page, he (or she) can proceed further to the next level of analyzes using the very new UI page which I call Trajectory DTW analysis page, and which allows to visualize the DTW algorithm implementation and quantify trajectories similarities using Euclidean distance:

Some R color palettes

2008-09-02T07:47:00.003-07:00

Once working on the trajectory plotting procedure I got curious about colors to use in my plots. I did a little research about what are common palettes used in R and there they are:

The R code snippet:


YlOrBr <- c("#FFFFD4", "#FED98E", "#FE9929", "#D95F0E", "#993404")
YlOrBr.Lab <- colorRampPalette(YlOrBr, space = "Lab")
YlOrBr.Lab.bias <- colorRampPalette(YlOrBr, space = "Lab", bias=0.5)

jet.colors <-
  colorRampPalette(c("#00007F", "blue", "#007FFF", "cyan",
                     "#7FFF7F", "yellow", "#FF7F00", "red", "#7F0000"))

rgb.palette <- colorRampPalette(c("blue", "orange", "red"),
                                     space = "rgb")
Lab.palette <- colorRampPalette(c("blue", "orange", "red"),
                                     space = "Lab")

demo.pal <-
   function(n, border = if (n<32) "light gray" else NA,
             main = paste("color palettes;  n=",n),
             ch.col = c("YlOrBr.Lab.bias(n)", "YlOrBr.Lab(n)",
                        "heat.colors(n)", "terrain.colors(n)",
                        "cm.colors(n)", "topo.colors(n)",
                        "rainbow(n, start=.7, end=.1)", "jet.colors(n)",
                        "rgb.palette(n)", "Lab.palette(n)"))
     {
       nt <- length(ch.col)
       i <- 1:n; j <- n / nt; d <- j/6; dy <- 2.2*d
       plot(i,i+d, type="n", yaxt="n", ylab="", main=main)
       for (k in 1:nt) {
         rect(i-.5, (k-1)*j+ dy, i+.4, k*j,
         col = eval(parse(text=ch.col[k])), border = border)
         text(2*j,  k * j +dy/4, ch.col[k])
       }
     }
n <- if(.Device == "postscript") 64 else 18
# Since for screen, larger n may give color allocation problem
demo.pal(n)

Starting the Fall'08.

2008-09-01T10:21:00.014-07:00

Officially starting the Fall'08 semester.

This Fall I'm taking three classes: Compilers, Algorithms, and the independent study. As the part of my independent study, I'll be blogging the progress weekly here.

So, last week I spent coding the Trajectory page for the ProjectBrowser component of the Hackystat. This was my very first experience using Apache Wicket framework as well as Jetty Web Server, since previously in my web-development I was using Google Web Toolkit and Apache Tomcat. It was a fun experience to learn a new framework which is so different from GWT. Can’t say if it’s better or worse, but for sure it’s different. For now it looks a way easier to start coding using Wicket than GWT. When started I didn't follow the Hints for development guide but cloned the telemetry package and started by altering the existing code.

I've had the custom telemetry simulation package I've coded earlier for stand-alone trajectory so data for two trajectories were populated right away into my local sensorbase:

Trajectory1 has the lifecycle from the January, 1, 2008 till February, 1, 2008

while Trajectory2 has the lifecycle from the March, 1, 2008 till April, 1, 2008.

The lack of an overlap between lifecycles for the trajectory1 and trajectory 2 makes it impossible to plot both telemetry curves simultaneously using the Telemetry component of ProjectBrowser.

The Trajectory component "mission" is to overcome this limitation and on the following screenshot you can see the third week of devtime for the trajectory1 and second and third weeks of trajectory2:

And the Trajectory UI screenshot:

The non-cropped version:

There are some changes I've made in the original Telemetry UI and plotting protocol I'd like to get feedback for:

1) The color schema for the plots: I've coded the new "color-picker" method which picks colors from the JetColors palette instead of random color selection. I personally found that some of the random colors which are selected by the original Telemetry color-picker are very hard to read.

2) As you can see, there is no "date ticks" at X axis of the plot... and I don't like it, but didn't explore yet the possibility to print two time-lines along the X axis.

3) Right now I can extend date interval for the one of the two projects to the right, I don't know if it's good or not, but it's rather feature than bug (IMO). I've got an idea that it would be nice to have the ability to introduce the indent before the one of projects interval beginning too... I'm almost there.

4) Should I merge my code with the hackystat-ui-wicket trunk?

So, this is my current progress and right now I'm implementing the Dynamic Time Warping algorithm for trajectories comparison. The current idea is to display the warped curves and warping procedure parameters just below the original trajectory plot.

.

Serving files from your GWT app.

2008-01-21T05:37:00.000-08:00

Also there are other ways to make this trick, I found this implementation easiest.

So. I have a GWT app nicely running but at some point I faced a new requirement - application need to be able create files at server side and serve this files through client to the user, and requirement is that user browser should open "Save as" dialog after the user clicks on the link.

Okay, what I've done first? Since everything coded in GWT, I've immediately created servlet at the server side like:


import java.io.DataInputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;

import javax.servlet.ServletContext;
import javax.servlet.ServletException;
import javax.servlet.ServletOutputStream;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

public class FileServlet extends HttpServlet {

 private static final long serialVersionUID = -4356636877078339046L;

 byte[] bbuf = new byte[1024];

 private static final String textFileName = "files/text.txt";

 public void doGet(HttpServletRequest request, HttpServletResponse response)
     throws ServletException, IOException {

   String filename = textFileName;
  
   try {
     ServletOutputStream out = response.getOutputStream();
     ServletContext context = getServletConfig().getServletContext();

     File file = new File(context.getRealPath("") + "/" + filename);
     String mimetype = context.getMimeType(filename);

     response.setContentType((mimetype != null) ? mimetype : "application/octet-stream");
     response.setContentLength((int) file.length());
     response.setHeader("Content-Disposition", "attachement; filename=\"" + filename + "\"");

     DataInputStream in = new DataInputStream(new FileInputStream(file));

     int length;
     while ((in != null) && ((length = in.read(bbuf)) != -1)) {
       out.write(bbuf, 0, length);
     }

     in.close();
     out.flush();
     out.close();
   }
   catch (Exception e) {
     e.printStackTrace();
   }
 }

}

i.e. servlet which reads the file (hardcoded name here) and streams it to the client side of webapp, and mapped it in the web.xml as independent from GWT servlet named FileServer. In my app I had a button which "generates a file" so i've attached code to the the button listener:


RequestBuilder rb = new RequestBuilder(RequestBuilder.GET, GWT.getModuleBaseURL() + "FileServer");

try {
rb.sendRequest("GetFile", new RequestCallback() {
public void onError(Request request, Throwable exception) {
Window.alert("Error getting file");
}

public void onResponseReceived(Request request, Response response) {
System.out.println("response Recieved");
}
});

} catch (RequestException e) {
e.printStackTrace();
}

while everything looks nice, nothing worked, by debugging I found that server gets the request and generates file, servlet streams the data, but no "Save As..." dialog appears... hmm... Than I wasted _a lot_ of time trying to fix this stuff until I got a point that my GET request is not the same GET request that hyperlink generates, so right solution on the client side is to create a hyperlink like follows:


new HTML("< \a href=\"/file-server-sample/FileServer\">Get a text file")

which queries same servlet and invokes proper dialog.
now it works, everybody happy :). Of course you can add parameters to you hyperlink and see those in servlet to alter file generation behavior.

Great tool for web development: Firebug.

2007-08-20T08:10:00.000-07:00

Just want to share the tool I found: Get Firebug here. I just love it's ability to "Tweak CSS to perfection", "Visualize CSS metrics" along with Javascripts debug.

Installing cygwin on Vista.

2007-08-19T19:07:00.000-07:00

I've got new Dell with Vista installed on and tried to install Cygwin today. It didn't worked... Installer stuck at gnuplot.sh...

In order to move over this point I've had to use the get Sysinternals' Process Explorer to kill this sh process, than it stuck on tetex-tiny.sh, killed again, than post-texmf.sh killed. After that setup finished, I had to do some magic though:


> mkpasswd -l [-d] > /etc/passwd
> mkgroup  -l [-d] > /etc/group

so, command line worked...
I added C:\cygwin\bin to Windows PATH
and had to do magic again to get X client running...


ash from cmd.exe
than ran rebaseall

seems to be working at least so far...

Some Recipes to Improve Your Google Web Toolkit Development

2007-05-28T01:13:00.000-07:00

While struggling with GWT debugging and development I found a pretty good article which has a good reasoning why and how a GWT-based project could be divided into "independent" parts from the development perspective of view: Some Recipes to Improve Your Google Web Toolkit Development. Will try to follow these guidelines right after tomorrow.

Happy Coding Hour with Philip.

2007-05-15T22:01:00.000-07:00

I missed all deadlines for making this post, but had a good reason, that I'll explain further:

So, back to the coding hour: it was just great. We worked on my GWT telemetryviewer, but most of the time we spent discussing and drawing on the board. Philip asks me questions that I wasn't able to answer and which forces me to learn more. I do appreciate this, it reminds me class teaching experience: you seems to know all, but student asks you are very "dumb" (in your opinion) question - ... and you are lost. You just can not answer straight, you are puzzled by a question because it so simple that it really cuts down to the bones of the subject through all the meat. That's it. I've learned the lesson: even if GWT claims that it hides all DOM and client-server interaction from the developer, I must have an idea how all this stuff works.

Happy coding with Hongbing.

2007-05-09T01:54:00.000-07:00

During coding hour (in fact we spent about 90 min) we were working on the UI design today. We tried to follow my instructions about installing the environment and failed since GWT seems to keep absolute paths within its config and datafiles. We had to change hardcoded paths manually.
After stuff was installed and we got mockup working we were trying to understand the GWT hosted and compiled modes along with hosting the mockup within the Tomcat. This task turned out to be pretty difficult since I am using JFreeChart jar and servlet along with JCommon. Basically we gave up and proceeded further.
The last timeframe we spent discussing my UI design. Hongbing was developing HackyStat for a long time and helped me to figure out which way of abstractions would fit previous HackyStat architecture (I want reuse some code instead of hacking by myself). Overall this part was a most productive.

Thinking Linux?

2007-05-05T01:15:00.000-07:00

Okay, I was using SuSe for more than three years now, both Enterprise @HPC and Open @home. I like it, I think Novell rules, and would recommend it for everybody, but boy, there are only 2% females out there in the real life using SuSe??? Something wrong out there, because I have two females and two males in my family using SuSe, i.e. 50/50, WHat's up with others 27458 users taking the survey?

Virtues of Monoculture, Or Why Microsoft Wins.

2007-04-24T23:31:00.000-07:00

I found a very good piece of writing that points out many things I was confused about for a long time. This article precisely and sharply points out all my doubts with Open Source. So, I am now even more confused while struggling with Ruby and AJAX, REST and GWT... But I definitely know, that once you (or I) get to the .Net side "You’re always going to be running on a Windows platform. You got the pretty gold ring, but Sauron gets to pull your strings and make you dance."

Virtues of Monoculture, Or Why Microsoft Wins.

CookBook on Rails

2007-04-23T19:09:00.000-07:00

Ok.

Since Eclipse is downloaded and runs perfectly (see previous post), I went to the Ruby side again.

Let me note that before struggling with Eclipse I had some stuff installed (see below), and I am going to use it now.
1. Ruby 1.8.5
2. Ruby Gems 0.9.2 (extract, then run "ruby setup.rb")
3. Rails ("gem install rails --include-dependencies")
4. MySQL 5.0 which was installed as the service at mysql://localhost:3306

Now we are setting up the Eclipse environment:
The Ruby setup:

The Rails setup:

At this point I am able to create a Rails project through ..->New Project -> Rails Project -> ... It runs and makes some "standard Rails welcome" output in browser.

Time to configure MySQL.
To work with MySQL stuff I've decided to use old good Eclipse Data Tools Platform (DTP) which employs connector/J from MySQL.
To setup Rails MySQL connector stuff I run "gem install mysql" and choose first (1) option.

At this point I have MySQL running as service, and Eclipse DTP and Rails are ready to connect it. In Eclipse I configured MySQL connectivity: ...->Preferences->Connectivity->...

And now we can use SQL Explorer ...

... or Data View to fix our databases and run the example from OnLamp.com: Rolling with Ruby on Rails Revisited

Continuing the Ruby way... (Windows)

2007-04-21T17:08:00.000-07:00

So, I've installed Ruby at my Windows machine along with RDT and RadRails Eclipse plugins and my Eclipse went wrong. It doesn't want to load, it cannot update due to some insane dependencies...

So, I've decided to reinstall Eclipse.

Since I am using WTP, at first I got it downloaded from IBM, but id didn't want to update, I've tried WTP site and this distros didn't work also. I got empty Eclipse and tried to install all my stuff, but it died on update again.

And after I found the best Eclipse configurator ever! Take a look at it at Innoopract. It does has all needed packages and plugins: I use Java, WTP, CDT, EPIC and RDT with dozens of plugins and tools. I got em all as the one zip, unzipped and working on Ruby skills now.

Went for mod_fcgid instead of mod_fastcgi

2007-04-13T22:19:00.000-07:00

Getting the module

 
# wget http://prdownloads.sourceforge.net/mod-fcgid/mod_fcgid.2.1.tar.gz?download

Need to fix Makefile so it point to /usr/share/apache2 and add copy couple of the libraries from /usr/include/apache2-prefork to /usr/include/apache2: mpm.h and mpm_common.h to avoid this error:


# /tmp/mod_fcgid.2.1 # make
/usr/lib64/apr-1/build/libtool --silent --mode=compile gcc -I.  -O2 -fmessage-length=0 -Wall -D_FORTIFY_SOURCE=2 -g -fPIC -Wall -fno-strict-aliasing -DLDAP_DEPRECATED  -DLINUX=2 -D_REENTRANT -D_GNU_SOURCE -DAP_DEBUG  -DSSL_EXPERIMENTAL_ENGINE -DMAX_SERVER_LIMIT=200000 -DLDAP_DEPRECATED -DMAXLINE=4096  -I/usr/include/apache2 -I. -I/usr/include/apr-1 -prefer-pic -c arch/unix/fcgid_proc_unix.c && touch fcgid_proc_unix.slo
In file included from arch/unix/fcgid_proc_unix.c:13:
/usr/include/apache2/mpm_common.h:46:17: error: mpm.h: No such file or directory
In file included from arch/unix/fcgid_proc_unix.c:13:
/usr/include/apache2/mpm_common.h:349: error: expected ')' before '*' token
make: *** [fcgid_proc_unix.slo] Error 1

So it make:


# make
# strip .libs/mod_fcgid.so
# make install

This has to be added to httpd.conf:


This is for fastcgi-mode Ruby on Rails ( Thank Andre Nathan giving me this demo configuration )

LoadModule fcgid_module modules/mod_fcgid.so
SocketPath /tmp/fcgidsock
SharememPath /tmp/fcgidshm
DefaultInitEnv RAILS_ENV production

<Directory /home/sneakymustard/tumble/public>

    Options ExecCGI FollowSymLinks

    AllowOverride AuthConfig Indexes Limit

    Order allow,deny


    Allow from all



    AddHandler fcgid-script .fcgi

    # You need
mod_fcgid version >= 2.1 to support arguments "xxx/dispatch.fcgi" in
FCGIWrapper 

    FCGIWrapper "/usr/bin/ruby
/home/sneakymustard/tumble/public/dispatch.fcgi" .fcgi



    RewriteEngine On


    RewriteRule ^$ index.html [QSA]

    RewriteRule ^([^.]+)$ $1.html [QSA]

    RewriteCond %{REQUEST_FILENAME} !-f

    RewriteRule ^(.*)$ dispatch.fcgi [QSA,L]

</Directory>

along with


AddHandler fastcgi-script fcgi

<Directory /srv/www/htdocs/>
  AllowOverride All
  Options FollowSymLinks
</Directory>

<Directory /srv/www/rails/*/public/>
  Allow from all
</Directory>

and


SocketPath /tmp/fcgidsock
AddHandler fcgid-script fcgi

Edit the

APACHE_MODULES

line in

/etc/sysconfig/apache2

by inserting “fcgid” and removing “fastcgi”.

HOORAY!

The Ruby Way :-), Руби на рельсах

2007-04-13T20:39:00.000-07:00

Thats it. Seeing other people (Robert was the latest victim :-) ) being attracted by Ruby made me thinking that everything is not that bad. So, while running my computations I've decided to see what that beast really is.

1. Using Yast (I have SuSe 10.2) I've installed:


  * apache2
  * apache2-devel
  * apache2-mod_fastcgi
  * ruby
  * ruby-devel
  * ruby-doc-ri
  * rubygems
  * FastCGI
  * FastCGI-devel
  * zlib

2. in shell:


  * # gem install fcgi
  * # gem install mysql
  * # install rails -y
  * # mkdir /srv/www/rails
  * # mkdir /srv/www/fcgi-log
  * # chown wwwrun /srv/www/fcgi-log

3. Do a

 # vi /etc/apache2/httpd.conf

and add


#
# Ruby stuff
#
AddHandler fastcgi-script fcgi

<Directory /srv/www/htdocs/>
  AllowOverride All
  Options FollowSymLinks
</Directory>

<Directory /srv/www/rails/*/public/>
  Allow from all
</Directory>

I've added to

/etc/sysconfig/apache2

in APACHE_MODULES words “rewrite” and “fastcgi”.

4. I think that I am finally done by doing following:


# cd /srv/www/rails
# rails test
# chown -R wwwrun:www test/tmp
# chown -R wwwrun:www test/public
# ln -s /srv/www/rails/test/public /srv/www/htdocs/test
# cd test/public
# vi .htaccess

and adding


RailsFCGIHandler.Process! '/srv/www/fcgi-log/test_fcgi_crash.log'

to the end of .htaccess

AND FINALLY


# rcapache2 start

AND I'VE GOT AN ERROR THAT KEEP ME WORKING FOR 2 HOURS ALREADY:


zhabanet-amd:/srv/www # rcapache2 start
Module "fastcgi" is not installed, ignoring.
Check the APACHE_MODULES setting in /etc/sysconfig/apache2.

I gave up for today...

Some off-site notes.

2007-03-31T01:42:00.000-07:00

While working on my other project, iClouds, found pretty useful thing as CLI package of the Jakarta/Apache commons project, it worked out of the box and I'm very happy about taking care of command-line stuff.

Remote Procedure Calls in Google Web Toolkit

2007-03-29T11:19:00.000-07:00

The information posted here was taken from GWT Beta developer manual, basically they say that their "services" are custom build ones: "the term service in this context isn't the same as the more general "web service" concept. In particular, GWT services are not related to the Simple Object Access Protocol (SOAP)." The diagram of the RPC plumbing suppose to give us some idea about services functionality, but didn't gave much to me personally since I've never touched this technology.
I have downloaded GWT stuff locally and trying to get my lovely dependency telemetry within the rich, GWT based application.

Pavel's Software Engineering Log

Some normalization issues.

Pilot data set.

Working on the Pilot data set.

litreview: almost there

Literature review draft finished.

First draft of litreview.

Working on the literature review and digging some interesting stuff.

Started literature review.

Current build of the Hackystat Trajectory browser.

SQL Schema evolution

Universal cuts for telemetry

Picture of the data values distribution in 9 telemetry streams

Hackystat telemetry data distribution

Latest improvements in the index browser tool.

First working build.

iBATIS + MySQL + MIGLayout

SAX code and TrajectoryDB schema evolution

The confirmation of valid search

The first motif search results

Sensorbase Index schema evolution

SAX based time series motif search primer

Indexing the Sensorbase and puzzled with the distribution (for now)

PAA and SAX R implementations

SAX and iSAX

Euclidean and DTW metrics based sequence (subsequence) matching

Working on the DTW implementation and Tests while planning the subsequence search implementation.

Progress report

Writing up a literature review.

User-account environment variables under Vista

Literature review plans

Introduction. (definitions, research field boundaries and common applications)

Time series similarity-based analyses and algorithms (known research tools, implemented applications and up-to date research directions)

Possible application of the algorithms and methods to the Hacvkystat Telemetry Streams

First post in the 2009

The 699 report continued.

The 699 report

DTW Step performance in Trajectory

Step functions performance illustrations

Trajectory is up and running within the ProjectBrowser

Who broke the build!?

Progress report

Progress report

Book: Information Retrieval for Music and Motion.

Working on the Trajectory UI.

Technical review outline.

UI design

Some issues with DTW I'm working on.

Open-ended alignment.

Precision errors....

Example: how DTW works 3 peaks and 2 peaks.

The original data plotted

Time series

Naive DTW DP in action

Before and After DTW

My Java DTW implementation finally works within the ProjectBrowser

Placeholder for the Monday, September, 22

Naive DTW is up and running.

Implementing DTW in trajectory

Some R color palettes

Starting the Fall'08.

Serving files from your GWT app.

Great tool for web development: Firebug.

Installing cygwin on Vista.

Some Recipes to Improve Your Google Web Toolkit Development

Happy Coding Hour with Philip.

Happy coding with Hongbing.

Thinking Linux?

Virtues of Monoculture, Or Why Microsoft Wins.

CookBook on Rails

Continuing the Ruby way... (Windows)

Went for mod_fcgid instead of mod_fastcgi

The Ruby Way :-), Руби на рельсах

Some off-site notes.

Remote Procedure Calls in Google Web Toolkit

Time series similarity-based analyses and algorithms

(known research tools, implemented applications and up-to date research directions)