Sarah has got us all thinking about the evaluation in while we prepare for show and tell 1. Hopefully this strategy will help minimise the project slippage.
Evaluation discussion stepped around filtering the SnT-1 discovered resource candidates. The list was fairly straight forward:
- Check edna for duplicates
- Check SPAM
- Check blacklists
- Check Scope criteria: language, geographic info.
All very ho hum not very interesting. The most interesting point was the idea - ‘if someone has bookmarked it then it’s relevant - so we can ignore the date criteria.’
What really got us charged up was to Vaughan’s idea to discover which metadata (type and value) is significant as a predictor for evaluation and/or classification of an online resource identified from the previous Discovery phase. To quote Vaughan:
Latent Semantic Analysis (Berry et al) is an example of a technique that maps a document collection to a lower dimensional space to that traditionally used (where each index term effectively forms another dimension in the search space, as used for example in traditional TF.IDF [term frequency–inverse document frequency] similarity measures”.
Getting Vaughan to explain it to me in plainer English:
Documents are usually represented by the terms that occur within them but some of those terms are more significant than others in representing the content. We want to identify those high-information-value terms and throw away the chaff.
So we’re actually identifying both wheat and chaff by valuing the terms used?
Yeah - Lucene, etc do it using a TF.IDF metric.This is a weighting of terms?
Yes. We’re evaluating for both the metadata term and value. The addition of field types adds information but ultimately we still have a bunch of features still, regardless some of them are not going to be significant in separating relevant from non-relevant docs, that is.
How does that relate to using edna as a training set?
We can use existing classifications to generate some more detailed ’signatures’ for representing docs. In a given class/category/sector thus we can build a predictor/evaluator for a given training set (sector/category)we can do this at various levels of granularity, from the collection level to the smallest category so we can train up a series of predictors.
As ed.au’s resident search expert, Vaughan mentions there’s lots of further possibilities with this approach for instance:
- Education Sector prediction
- Meta data suggestion
- lots more…
We all agreed that this Meta Discovery thing sounded like the cool thing to do for evaluation.
Ambitious? Nah - just a walk in the POC.
Post a Comment