<?xml version="1.0" standalone="yes"?>
<Paper uid="H01-1037">
  <Title>The Integrated Feasibility Experiment (IFE) Process</Title>
  <Section position="3" start_page="0" end_page="0" type="metho">
    <SectionTitle>
2. DESCRIBING THE IFE SIX KEY STEPS
</SectionTitle>
    <Paragraph position="0"> The IFE process consists of six steps that guide development and experimentation. The emphasis placed on each step depends on the maturity of the technology and the involvement of the users.</Paragraph>
    <Paragraph position="1"> The six steps are as follows (note that the six steps are summarized in Figure 1)</Paragraph>
    <Section position="1" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
2.1 Step #1: Scenario
</SectionTitle>
      <Paragraph position="0"> Describe a scenario for employing the information technology that will allow everyone to visualize how the technology is to be used during a real situation. This step places emphasis on making the technology look and behave like a system. This is a critical step for two main reasons: a. For the technology teams that are to integrate technology, the scenario provides a real and accessible description of how the technology should be used. This assists the teams directly in describing the architecture and components needed to build an information system for the given scenario.</Paragraph>
      <Paragraph position="1"> b. The scenario is key in describing the intent of the information system to the operational user. Typically, operational users become involved in this scenario building process to give early and helpful feedback to the technology development teams.</Paragraph>
    </Section>
    <Section position="2" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
2.2 Step #2: Architecture
</SectionTitle>
      <Paragraph position="0"> Many people believe that describing the architecture is the key step in building and information system. However, if the ideas about components and interconnections are vague or incomplete, then the architecture step is actually best developed using a hypothesis and test process. In all cases the architecture must allow plug-and-play concepts that support the inclusion and reuse of mature processing components, plus the inclusion of new components that will be the focus of experimentation.</Paragraph>
    </Section>
    <Section position="3" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
2.3 Step #3: Reuse Components:
</SectionTitle>
      <Paragraph position="0"> The third step is to identify and make plans to reuse components that one will depend on during the IFE. This step is critical for the technology teams because many of the components to be used come from years of development and experimentation. With mature components populating a large share of the architecture, the development teams are then free to experiment with new components that are considered to be necessary for end-to-end processing. Moreover, the developers can experiment with data flow and interconnection strategies. This experimentation step is critical in order to transform into tomorrows' network-centric processing models supported by communication interoperability provide by TCP/IP processing.</Paragraph>
    </Section>
    <Section position="4" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
2.4 Step #4: User Involvement
</SectionTitle>
      <Paragraph position="0"> Obtaining operational user involvement early-on is an important step to support a technology transformation objective. The operational user will have insights and needs that cannot be predicted by the technology developers. Moreover, user involvement improves the interest, understanding and potential commitment to the technology. If user centered final exam metrics are stated clearly then they provide a useful objective to help focus technology development and implementation. This all may sound like motherhood, but it is a critical step that is missing often from technology development projects large and small.</Paragraph>
    </Section>
    <Section position="5" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
2.5 Step #5: Rapid Prototyping
</SectionTitle>
      <Paragraph position="0"> The use of a rapid prototype approach is not new. In the mid 1980s it became the key focus for specifying and building information systems. However, the rapid prototype process must be used in conjunction with other steps of the IFE, or else the development effort will end up as a simple demonstration that does not scale to real user needs. The spiral development model for development that emphasizes the &amp;quot;build a little, test a little&amp;quot; approach, should be used to keep development on track and headed toward the target needs of the user.</Paragraph>
    </Section>
    <Section position="6" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
2.6 Step #6: Evaluation and Feedback
</SectionTitle>
      <Paragraph position="0"> Metric-based evaluation is important for any development process. For an IFE the specification of usable metrics is not easy because the teams are coming together to build a &amp;quot;new&amp;quot; capability. The best approach comes by making an early commitment and following through with the measurement process and then later changing the evaluation process to better represent the emerging information processing capability. One should have measures for  technology accomplishment and such measures should focus on component performance. In addition, one must have an overall &amp;quot;task performance&amp;quot; metric or metrics that reflect the needs of the operational user and the intent of the scenario.</Paragraph>
      <Paragraph position="1"> Integrated Feasibility Experiment Steps 1. Scenario .. Helps to visualize the use of new technology 2. Architecture .. Components, interconnects, data flow, and processing model 3. Reuse components .. Must build on past accomplishments 4. User .. The user provides application pull, as opposed to technology push 5. Rapid prototype .. Build a little, test a little strategy to keep effort on track and on target 6. Evaluation and feedback: Metrics-based evaluations are key to understanding accomplishment</Paragraph>
    </Section>
    <Section position="7" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
2.7 Historical Note
An Integrated Feasibility Development (IFD) process was first
</SectionTitle>
      <Paragraph position="0"> used in 1990 by Steve Cross and his team to guide development of the DARPA and Rome Labs replanning system called DART (Dynamic Adaptive Replanning Technology). DART was developed to assist logistics and transportation planners in scheduling the movement and deliver of people and materials. An operational prototype was actually used during the Persian Gulf conflict in 1990. The IFD name has been changed to IFE by replacing &amp;quot;Development&amp;quot; with &amp;quot;Experimentation&amp;quot; in order to emphasize the experimentation and scientific exploration aspect of the effort, but the steps of the process have remained the same.</Paragraph>
    </Section>
  </Section>
  <Section position="4" start_page="0" end_page="0" type="metho">
    <SectionTitle>
3. WHY THE IFE PROCESS WORKS
</SectionTitle>
    <Paragraph position="0"> Their three good reasons the process works and a forth explanation that deals with the basics of building and implementing information technology.</Paragraph>
    <Section position="1" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
3.1 Application Pull
</SectionTitle>
      <Paragraph position="0"> The scenario and user involvement (steps 1 and 4) work together to provide an &amp;quot;application pull&amp;quot; on the technology. To many efforts fail because they start with a new idea which is pushed and developed and then is found to be in search of (ISO) of a meaningful application. This &amp;quot;application push&amp;quot; model fails in most cases because no user is willing or able to invest in an acquisition follow-on process. Instead, the application pull process will address new information system introduction methods that take full advantage of commercially created information technology, and blend in radically new ideas that provide for scale and success. These steps insure transformation efforts will be based on innovation and speed.</Paragraph>
    </Section>
    <Section position="2" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
3.2 Scalable Baseline
</SectionTitle>
      <Paragraph position="0"> The architecture and reuse-of-components focus (steps 2 and 3) provides a baseline capability that will enable the information technology to scale up to deal with operational needs. Moreover, this investment in the software architecture provides the infrastructure needed to explore new ideas in an affordable and repeatable fashion.</Paragraph>
    </Section>
    <Section position="3" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
3.3 Build A Little, Test A Little
</SectionTitle>
      <Paragraph position="0"> Rapid prototyping and evaluation steps (steps 5 and 6) offer a simple and understandable approach to allow for incremental progress that is informed by failure as much as by achievement.</Paragraph>
      <Paragraph position="1"> This is key. Innovation must be allowed to fail just as long as the process moves forward and is informed in a positive way by the failure. Too many projects fail to provide for the process of managing risk and failure. Such projects are doomed to incremental advancement at best.</Paragraph>
    </Section>
    <Section position="4" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
3.4 A Managed Process That Works
</SectionTitle>
      <Paragraph position="0"> Some observers of the IFE process have said the six steps are necessary and sufficient to provide guidelines for information systems development and implementation. Necessary and sufficient does not guarantee success. It does however provide a small and simple set of steps that can help the technology community to shape information technology, and give it an outstanding shot at success. In most instances the IFE methodology has addressed crisis action and crisis response scenarios that address dynamic problems in the effective use of people, resources, information, and network-centric computing.</Paragraph>
      <Paragraph position="1"> This methodology has been used to increase cooperation between defense and intelligence groups to develop command, control, computing, and intelligence infrastructure fundamental to developing new concepts of operation, and the foundation on which future capabilities are built.</Paragraph>
    </Section>
  </Section>
  <Section position="5" start_page="0" end_page="0" type="metho">
    <SectionTitle>
4. AN EXAMPLE IFE
</SectionTitle>
    <Paragraph position="0"> The following provides an example of the Strong Angel IFE used for the &amp;quot;PacTIDES&amp;quot; exercise in June 2000. This exercise was sponsored by the US Joint Military Command known as CinCPAC and included seven other nations and the United Nations. Both the accomplishments and the lessons learned will be covered. The Strong Angel IFE provided and outstanding framework for learning more about end-to-end language processing.</Paragraph>
    <Section position="1" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
4.1 Strong Angel IFE Overview
</SectionTitle>
      <Paragraph position="0"> Step 1. Scenario: The primary application focus for the IFE was the spread of disease, with special emphasis given to information processing techniques. The operational user was Dr. Eric Rasmussen, MD who was the Third Fleet Surgeon for the United States Navy. Dr.</Paragraph>
      <Paragraph position="1"> Rasmussen was most concerned about providing effective support and relief to people during &amp;quot;Humanitarian Assistance&amp;quot; operations that are becoming common through out the world. Examples like Bosnia and Kosovo come to mind immediately.</Paragraph>
      <Paragraph position="2"> The story line was that refugees were caught in a border location and world organizations were coming together to provide food, shelter, and security. The spread of disease soon became one of the top security risks. The TIDES system was used by the security teams to get timely information about relevant events so they could anticipate critical situations they may face instead of simply reacting to issues.</Paragraph>
      <Paragraph position="3"> Step 2. Technology teams outlined a plug-and-play architecture called the &amp;quot;TIDES Portal&amp;quot; that was used to guide the development and experimentation process. The architecture was built on a client - server model where components for language processing were loosely confederated over the Internet.</Paragraph>
      <Paragraph position="4"> Step 3: Component specification: The three primary information processing components were focused on detection, extraction, and user interaction. There also was a translingual component that provided two way translations to and from Korean. The scenario was expanded to include the treat of a missile launce from North Korea that could carry a biological war-head.</Paragraph>
      <Paragraph position="5"> The translingual component was an add-on rather than a main line processing component. There were seven different sources of news that was being processed to provide information to relief and security personnel.</Paragraph>
      <Paragraph position="6"> These sources included both text and speech information. The speech information was transformed into text and then became input to detection and ext raction processing. The user interface component was the most difficult to construct because the underlying end-to-end processing model was emerging and changing each month. Moreover, the loosely coupled distributed processing model for the TIDES Portal was difficult to realize in a coherent user interface. This issue and other shortcomings are discussed in the lessons learned section of this paper.</Paragraph>
      <Paragraph position="7"> Step 4: Operational User Involvement. The scenario definition process helped Dr Rasmussen and the other relief and security operators understand how the technology would come together to be used. The TIDES Portal and the &amp;quot;PacTIDES&amp;quot; experiments were use by representatives of several of the RIMPAC nations and also by United Nations personnel. For the first time ever the RIMPAC exercises conducted by seven nations: US, Canada, Japan, Chile, Australia, Korea, and UK, included a focus on a humanitarian assistance issues. For the first time users were able to understand in context the kinds of capability an automated information processing system such as TIDES may provide in the future. The potential for TIDES support received strong endorsement from these operators who are literally overwhelmed by data, documents, and email, but who are often starved for actionable information.</Paragraph>
      <Paragraph position="8"> Step 5. The rapid prototype process was used to develop the IFE integrated system called the &amp;quot;TIDES Portal&amp;quot;. Initial TIDES Portal implementation was tested in early 2000, and the final exam for TIDES Portal was conducted during Strong Angel was held In June 2000 on the Parker Ranch in Hawaii. The system was used by military and by UN World Food Program personnel. There was one situation where UN folks needed timely information about a situation in Africa, and the TIDES Portal came through. The UN team was impressed. However, most of the lessons learned at Strong Angel pointed to weaknesses in the TIDES Portal concept of operations. These weaknesses have become the main focus for development of IFE-Bio in 2001.</Paragraph>
      <Paragraph position="9"> Step 6. Metric-based evaluation was used in Strong Angel with limited success. The weaknesses in the end-to-end processing capability of the TIDES Portal dominated the IFE and limited the ability of research groups to conduct full metrics-based evaluations in a meaningful way. This issue will receive more attention in during IFE-Bio final exams in June 2001.</Paragraph>
    </Section>
  </Section>
  <Section position="6" start_page="0" end_page="0" type="metho">
    <SectionTitle>
5. LESSONS LEARNED
</SectionTitle>
    <Paragraph position="0"> The Strong Angel IFE was judged to be a success even though several parts of the effort resulted in failure. The important point is that the TIDES Program learned from both the failures and the accomplishments and the lessons help guide the IFE process in 2001. The following provides an example of the Strong Angel IFE used for the &amp;quot;PacTIDES&amp;quot; exercise in June 2000. This exercise was sponsored by the US Joint Military Command known as CinCPAC and included seven other nations and the United Nations. Both the accomplishments and the lessons learned will be covered. The Strong Angel IFE provided and outstanding framework for learning more about end-to-end language processing.</Paragraph>
    <Section position="1" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
5.1 Lessons Learned From Negative Examples
</SectionTitle>
      <Paragraph position="0"> in Strong Angel IFE A. Process model was too uncoupled. Several groups came together integrated by only the Internet. The processing components were not synchronized and basically had little inter-dependency. Therefore, there was little in the way of information management within the infrastructure to hold the information processing model together.</Paragraph>
      <Paragraph position="1"> B. Late-binding decisions about distributed processing burned up critical development cycles. The situation here is simple: initially the assumption was made that full Internet connectivity at T1 rates would be available, and then the assumption was changed to anticipate NO Internet connectivity outside of the camp. The change in the Internet connectivity and quality of service assumptions were made with two months to go. Most of the time was then spent on building local servers and processes that would simulate that external communications was in place. During the critical last two months critical development and testing was stopped and attention was turned to re-engineering the processing infrastructure.</Paragraph>
      <Paragraph position="2"> C. Collection issues were not properly anticipated: The strengths of TIDES processing comes from end-to-end processing of streams of information from sources such as radio, TV, email, newswire, etc. Unfortunately the language detection and extraction communities are conditioned to processing from training and test sets provided to them in efforts such as TREC and MUC.</Paragraph>
      <Paragraph position="3"> Strong Angel concepts of operation actually required continuous processing of streaming information from multiple sources. These capture and processing priorities were not realized soon enough in the IFE process, and were therefore sorely lacking at the Strong Angel final exam.</Paragraph>
      <Paragraph position="4"> D. TDT processing concepts were not included: The detection, extraction, and summarization process for Strong Angel anticipated Topic Detection and Tracking (TDT) capabilities, but the algorithms were never incorporated. This means that critical front-end filtering and grouping functions were missing.</Paragraph>
    </Section>
    <Section position="2" start_page="0" end_page="0" type="sub_section">
      <SectionTitle>
5.2 Lessons Learned From Positive Examples
</SectionTitle>
      <Paragraph position="0"> in Strong Angel IFE A. The Strong Angel team never imagined how difficult the living and information processing environment could be in a refugee camp. In fact there was fine grain dust everywhere and the power was intermittent. Better understanding of these environmental factors was a positive coming from the Strong Angel effort.</Paragraph>
      <Paragraph position="1"> B. A key positive was developing the understanding for how detection, extraction, and summarization must work together with collection and distribution to provide an end-to-end processing infrastructure.</Paragraph>
      <Paragraph position="2"> C. An important accomplishment occurred when UN folks wanted to know more about a growing crisis in Uganda after a humanitarian incident. It turns out that TIDES processing was able to give the UN contingent information that they needed that was current and multisource. The UN folks were thrilled and amazed. Most amazing to the TIDES folks was the capability only used 10% of what was anticipated for TIDES processing. In other words, a very small and easy product provided significant value. There is great confidence that much more can be accomplished in IFE processing in 2001.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>