[apologies for multiple cross-postings]
Dear NLP-researchers,
We're glad to announce the release of the IQMT Framework for Machine
<http://www.lsi.upc.edu/%7Enlp/IQMT/>
Translation Evaluation <http://www.lsi.upc.edu/%7Enlp/IQMT/>v 1.0
<http://www.lsi.upc.edu/%7Enlp/IQMT/>, an open source software package,
released under
the GNU Lesser <http://www.fsf.org/licenses/lgpl.html> General Public
License (LGPL) <http://www.fsf.org/licenses/lgpl.html> of the Free
Software Foundation <http://www.fsf.org>.
This tool is a joint effort by the UNED NLP & IR <http://nlp.uned.es/>
Research Group <http://nlp.uned.es/> and the
TALP Research Center <http://www.lsi.upc.edu/%7Enlp> NLP group
<http://www.lsi.upc.edu/%7Enlp>.
The IQMT <http://www.lsi.upc.edu/%7Enlp/IQMT/> package (Giménez et
al., IWSLT'2005)
<http://www.lsi.upc.edu/%7Ejgimenez/PUBS/IWSLT05-gah.pdf> is based on
the QARLA
Framework (Amigó et al., <http://acl.ldc.upenn.edu/P/P05/P05-1035.pdf>
ACL'2005 <http://acl.ldc.upenn.edu/P/P05/P05-1035.pdf>). Rather than
defining a new supermetric
'XXX' our tool follows a 'divide and conquer' strategy. You can define a
set of
metrics and then combine them into a single measure of MT quality, in a
robust
and elegant manner, avoiding scaling problems and metric weightings.
Using IQMT <http://www.lsi.upc.edu/%7Enlp/IQMT/> offers a number of
advantages over previous MT evaluation packages.
First, individual metrics improve their level of correlation with human
judgements
when they are applied inside QARLA. Second, it permits to avoid the
'metric bias'
problem, by allowing you to tune your system on a combination of metrics
instead
of on a single metric. Third, it allows you to define a set of subtle
metrics focusing
on partial aspects of MT quality possibly at different linguistic
levels, and then
combine them into a single measure.
Several well-known and freely available current MT evaluation tools
such as
BLEU, GTM, METEOR, NIST, ROUGE, WER and PER, have been incorporated
so far. We plan to incorporate new metrics working at linguistic levels
other than
lexical overlap in a near future.
However, the main feature of the IQMT package is that it allows to
supply
user-defined MT metrics. So, do not hesitate to try your own metrics inside
QARLA!
Please feel free to download IQMT v1.0 at
http://www.lsi.upc.edu/~nlp/IQMT/ <http://www.lsi.upc.edu/%7Enlp/IQMT/> .
Any feedback (comments, suggestions, bug reporting) from the NLP community
members will be highly appreciated.
IQMT developing team.
This archive was generated by hypermail 2b29 : Fri Nov 18 2005 - 13:00:20 MET