Automating for consistency and capacity with the POCDOC Tool

Summary

In the Advanced Technology Center, (or ATC) our Lab Services team is asked to perform Proof of Concepts (or POCs) with our customers every day.  Sometimes our customers require very detailed results documentation that can literally be hundreds of pages long.  The real challenge is trying to take all of the different results from testing in the lab and getting them compiled into a presentable format back to our customers.

The ATC Test Report Generator (POCDOC) tool automates the building of test reports delivered by The ATC Lab Services team by reading through gathered information to quickly generate a consistent and professional final PDF document as a customer deliverable. 

The tool was created to address the need to reduce time invested in formatting customer facing documentation using a solution that would be natural and simple to use.  We explored the test plan writing process and methods to gather and organize information used within the reports.  A prototype was developed and applied to a live customer engagement learning and evolving the tool on the go.  Additional users with different engagement data and perspectives have continued to contribute to the tool's maturation and acceptance in the ATC.

In this ATC Insight, we showcase features that really make the entire process as well as the generated documents shine.  We dive deeper into this tool and describe what's under the hood.  We also highlight the benefits the tool brings to the ATC Lab Services team documentation process for documentation.

ATC Insight

We have used the ATC Test Report Generator (POCDOC) and have proven it is a highly scalable for a variety of ATC Lab Services engagements that require a test plan report deliverable.  By design, the generated document architecture is flexible for a variety of documentation needs while maintaining a consistent look and for a professional result.  

In these next few sections we'll dive into the POCDOC tool and highlight its features.  We give a look around what is under the hood (hidden from users), and explain the top benefits gained by using the POCDOC tool.
 

Before POCDOC...

Before POCDOCS, individuals worked on separate Word docs with no set template.  Documents did not contain a solid versioning standard and change management of the documents were cumbersome.  Hours were spent compiling test results into Word format, and this caused scalability contention within the team.

What POCDOC did for the team...

As you can see in the new flow with POCDOC, there are several improvements to the building results document workflow.  Now, architects and engineers can take their test results in many formats and structure them very simply and efficiently in a Github repository which allows POCDOC to compile the data into a very professional report. 

Digging deeper into the POCDOC Tool...

Centralized and Automated

All information used to create a document is stored in a GitHub repository within the WWT organization.  GitHub lets teams contribute from anywhere with all information centrally stored.  The Git version control system tracks changes to the document and enables document milestones to be marked at that point in time as released.  The document GitHub repository is configured to trigger document generation (build) each time a commit is made to the repo.  

We understand that not all users of the generator are GitHub/Git experts and that even avid users also may not remember all the operational details.  To address this need we've used a centralized Jenkins server to automate the process.  With a few pieces of information from the user, the server automates the initial building and pre-populating of the repo.  The Jenkins server then performs all the configuration needed to trigger report generation on commits, and builds the first document from the new repo using a default sample template.  From that point, only the very basic knowledge of Git is required to be successful.  Jenkins continues to be key for users to download and mark documents as released, and helps make POCDOC a low barrier to entry.   

Organizing a Document Feels Natural

We learned by studying how engineers organize test plans and gather evidence that the trend was to group items into a hierarchy of directories and files.  Keeping that same formula, the generator uses a "natural or human sort" following the structure to define the document order.  Directory names are incorporated into heading values and file names are the keys to processing the files properly.  As an example .png/PNG files are processed and inserted as image files and *CLI.txt files are processed as CLI files.  Markdown text files(.md) are used to add formatted text and may also include inline images and tables.    

Automatic External File Archiving(ZIP)

Occasionally external files can supplement documentation delivered to the customer.  These external files can be organized and committed into the document repository.  The generator then automatically bundles all external files keeping the organization hierarchy intact into a .zip file ready for delivery.  This strategy saves the engineer a bit of time, addresses the need to deterministically deliver information, and to avoid any files being considered malicious by Adobe.    

Consistent Automatic Formatting

The generator let's the user hit the 'easy' button for document formatting.  Generation is tuned to create documents trending toward the same recognizable look and feel without a lot of work by the user.  The following few sections highlight some of the automatic formatting.  

Branding

Branding is important can can tailor the look of a document to a customer.  The cover page and headers may include a customer or WWT image.

Test Case Summary Tables     

Test case planning and results at this time are many times stored using Microsoft Excel.  This process continues but is now done with a standardized format.  From the .csv form of the Excel worksheet the generator creates a custom formatted table.  Formatting also includes the addition of coloring based on test case status for easier status recognition at a glance.  

Tuned CLI Capture Formatting

CLI text files are processed and inserted into the document to assist in reviews and highlight areas of interest.  This includes wrapping lines, adding line numbers, highlighting commands and user prompts, and bordering the text for easier visual tracking especially over multiple pages.    

Image Optimization

JPEG and PNG image files are captured by a variety of tools and the image and file size may vary.  The generator automatically optimizes each image file prior to inserting the image into the document using a consistent size keeping all aspect ratios intact.  Image filenames are used as captions or pseudo-headings.  

The Basics

Typical formatting components are automatically included as part of the generated document.  This include items such as: table-of-contents, PDF bookmarks, heading levels, page numbers, font type and size, and more.

Under-the-Hood of POCDOC

POCDOC uses WWT in-house written Python code to stitch together several popular open source tools not counting GitHub, Git, and Jenkins mentioned above.  This section briefly highlights those tools.

• LaTex - (pronounced LAH-tekh) The core documentation system comprising a collection of macros that uses plain text to format text.  The generator builds a LaTex file (.tex) and then compiles the file using the typesetting engine XeLaTex to create the final .pdf document.
• Pandoc - Is used to convert markdown files into a LaTex file.  The in-house logic then processes the converted information inserting the portions that make sense into the .tex file.
• Pngquant - Is a lossy PNG compression tool used to significantly reduce the file size while maintaining a quality image.  The generator uses this tool to optimize images reducing the overall final PDF file size and to shorten document compile time.
• Docker - The generator is built as a container for seamless deployment to the Jenkins server.  The docker container is pretty much invisible to the users as part of the automated document build process.

Top Benefits

The tool has topped the original goal to reduce document formatting time and has evolved to create even more capacity enabling engineers to perform other tasks.  The following summarizes the top benefits the ATC Test Report Generator tool provides?    

• Final PDF documents built in seconds saving days of time per project - promotes productivity eliminating inefficiency
• Best practices for documentation automated for superior quality
• Consistent and professional-looking documentations
• Collaborate more easily on a project
• Flexibility of design enables documents to be modified for various engagements
• Data consists of files that engineers routinely gather
• Generator processes data and formats for enhanced formatting
• Centralized tool enabling fast tool updates to all

Test Plan

The ATC Test Report Generator(POCDOC) has demonstrated to be a powerful tool and to strengthen that impression this page provides additional data.  We will highlight real customer engagements that have successfully used the generator.  We'll then provide several examples to illustrate top attributes providing a visual of the generators implementation details.

Real Customer Engagements

The generator was developed to be as agnostic as possible in terms of the target documentation generated concentrating on balancing flexibility and consistency.  Organization of the data within a document is ultimately the engineers decision and is key in creating clear and concise results.  Although the generator is relatively new, it has already been successfully used for several POC engagements.   The following describes highlights of those Proof of Concepts (or POCs).  

POC with a Financial Services Company 

The verify first engagement to use the generator was with a major financial services company that requested assistance certifying equipment for their branches.  The test report produced was a typical recipe with basic elements included an introduction section to set the stage and provide global information, logical groupings/modules of test case results, an appendix of detailed results such as screen captures and CLI text, and supplemental external files.  

Doc metrics:  209 pages,  6.07MB .pdf,  323KB .zip of external files, ~45s to generate
Architect Hours saved: 60 hours

POC with an Insurance Provider

A major healthcare insurance provider requested help analyzing a security endpoint in our ATC Lab.  This POC was similar the financial services customer in the previous section as the test report included a more typical test report recipe.  

Doc metrics:  81 pages,   9.19MB .pdf,  11.87MB .zip of external files, ~55s to generate
Architect Hours saved: 40 hours

POC with a Global Service Provider 

The final and most impactful example is one of our Global Service Provider customers who gave us an opportunity to do a POC in our ATC lab to prove our overall testing services capabilities to the service provider business unit.  At a high-level, the goal was to test in a multi-phase approach on a uCPE platform service chaining virtualized network and security services.  For each phase the Lab Services engineer assigned developed the initial test plan using the generator that after an internal review was distributed for customer approval.  

The overall typical test report recipe is present but considerably more complex.  Phase 1 targeted operation and functionality where as Phase 2 targeted performance.  The engineer used multiple test result sections to highlight different logical sections of results with an appendix that reflected the same organization.  For performance testing the choice was to create sections to highlight result evidence earlier in the document moving forward information considered to be more important to the customer.   Each generated document required many screen captures as test evidence adding to the image optimization and compile times and ultimately the final document size.  Professional looking documents played a key role in presenting the WWT brand to the customer.

Phase 1 Doc metrics:  669 pages,  59.06MB .pdf,  no .zip file, ~11min 46s to generate
Architect Hours saved: 60 hours

Phase 2 Doc metrics: 2001 pages, 80.03MB .pdf,  1.08GB .zip file, ~18min 52s to generate
Architect Hours saved: 80+ hours (and prevention of quitting)

What does a doc look like in the GitHub repository?

All information that the POCDOC generator uses to build a document is organized into a single directory file structure.  The document is built by the generator traversing the structure and building the final document in the order of the structure.  This section's image is a capture of a sample document structure used to create a final PDF document.  More detailed information is available in the generator user guide but for now we'll touch on a few highlights.

• Order is forced by the prepended numeric values and are removed prior to insertion into the document
• Forced ordering is unrelated to section header numbering created by the generator
• Directory names become headings within document
• Files within directories become the documents content
• Very few naming convention rules exist and the few that do exist enable the generator to detect processing required

Sample Test Case Summary Results

One of POCDOC's more powerful features is to transform standardized test case data from an Excel .csv file into the generated report.  The first image in this section is an example Excel worksheet containing test case data.  Using an Excel worksheet to track test case progress is a common method used by engineers.

The engineer (when complete with the test results) saves the data into a .csv file for the generator to read.  The next image (below) is a capture of the example subdirectory structure.  Notice the numbering prepended to the directory and filenames to force order in comparison to the final image in this section showing the processed data.  The forced ordering values are unrelated to actual header numbers created by the generator.

The final image (below in this section) is a screen capture of the standardized test case data transformed into the documents test case summary results.  A few interesting formatting points are as follows:

• Example from a single Excel worksheet/.csv file
• Header/section numbering generated automatically with no user work
• Generator detects logical test case groupings(modules) and separates tests into sub-sections
• Sub-section names taken from test case module names
• Status field capitalized, bolded, and colored accordingly
• Expected results if present are aligned with the corresponding steps
• Comments cells enable expansion on status with additional evidence located in appendix sections
• Automatic URL detection (Test case ID TC-3 comments cell)

Sample CLI Processing

Another key feature of the POCDOC report generator is the processing and formatting of CLI text files.  The first image in this section is an example of a CLI text file displayed in an editor.  This example shows several bits of information but a reader must hunt for what is important.

The final image in this section shows how the generator formats the previous CLI text example into something more consumable for the reader.  A few interesting formatting points are as follows:

• Font is monospaced where all characters occupy the same amount of horizontal space
• Line numbers to help direct focus during reviews
• Line wrapping so no information is lost running off the page (see line #12)
• Bordering of CLI text
• Highlighting of commands lines with actual command text present
• Highlighting performed for the entire line to give a cohesive look
• Detection of comment only lines for easier commenting after the file is initially saved(see line #4)