ApacheCon NA 2015

DUCC is a new Linux cluster controller designed to scale out any Apache UIMA (tm) pipeline for high throughput collection processing jobs as well as for low latency real-time applications. DUCC stands for Distributed UIMA Cluster Computing. DUCC is running on cluster sizes from 1 to many 100s of machines.

This talk will cover the motivations that led to the creation of DUCC (the IBM Watson Jeopardy! Challenge), DUCC's benefits to developers and to computing cluster administrators, and demos of what you can do with it. It will explain why DUCC is well suited to run large memory Java analytics in multiple threads in ways that fully utilizes modern multi-core machines.

Attendees will leave with an appreciation of where DUCC "fits" in the UIMA set of subprojects, and an understanding of the value and applicability of using DUCC as part of their UIMA infrastructure deployments.

The Video IP Engineering and Research (VIPER) team at Comcast is responsible for HTTP video delivery that exceeds 500M transactions per day. Our DASH VOD Origin is a Java Tomcat application built with Maven. Our Super8 just-in-time packager is an Apache HTTP module written in C that uses Apache Portable Runtime. We implement our forward and reverse caching proxies using Apache Traffic Server, and our browser PlayerPlatformAPI is an Apache Flex application. We ingest and maintain 70,000 hours of VOD content, compress it using H.264/AVC, and store it on a 2PB network attached storage system. Sourcing our content in DASH (Dynamic Adaptive Streaming over HTTP) lets our Super8 packager easily convert video into proprietary formats such as Apple HTTP Live Streaming (HLS) and Adobe HTTP Dynamic Streaming (HDS) for video playback on mobile, browser, and IP set-top devices all across the country.

Text extraction tools are essential for obtaining the textual content and metadata of computer files for use in a wide variety of applications, including search and natural language processing tools. Techniques and tools for evaluating text extraction tools are missing from academia and industry. Apache Tika™ detects file types and extracts metadata and text from many file types. Tika is a crucial component in a wide variety of tools, including Solr™, Nutch™, Alfresco, Elasticsearch and Sleuth Kit®/Autopsy®. In this talk, we will give an overview of a new initiative within Tika to create an evaluation toolkit that allows integrators to evaluate Tika and other content extraction systems on client-specific documents. This talk will end with a brief discussion of a related initiative to take this evaluation methodology public and evaluate Tika on large batches of public domain documents.

Note: This talk was co-authored with Paul M. Herceg, Lead Artificial Intelligence Engineer, The MITRE Corporation. Paul holds an M.S. in Computer Science and a B.S. in Computer Science-Mathematics, both from the State University of New York at Binghamton.